Self-Hosted LLM Gateway: Deployment, Costs & Trade-offs

A self-hosted LLM gateway is middleware that sits between your application and multiple LLM providers (OpenAI, Anthropic, Ollama, local models), unifying their APIs under one endpoint. It manages credentials, routes requests with fallback logic, and tracks spending. Teams choose self-hosting for data sovereignty, lower costs at scale, and reduced latency. But the operational burden is substantial, and most teams underestimate it.

What Is an LLM Gateway, and Why Do Teams Build Them?

An LLM gateway is a routing and abstraction layer that intercepts API calls from your application and forwards them to your chosen LLM providers based on rules you define. It solves four concrete problems: (1) credential sprawl, (2) cost visibility across providers, (3) failover logic when a provider is down or rate-limited, and (4) semantic routing, where requests are sent to different models based on complexity or cost constraints.

Without a gateway, your application code scatters API keys across environment variables, hardcodes provider-specific logic, and has no visibility into which requests cost how much. You switch providers by rewriting code. You handle outages with manual workarounds.

Teams typically realize they need a gateway when one of these happens: (1) they switch LLM providers and don't want to rewrite the application, (2) they hit rate limits on OpenAI and need fallback to Anthropic, (3) they run local Ollama models and want to layer a commercial provider on top, or (4) they need to track spend by team, project, or customer for billing or cost control.

Data sovereignty is a secondary but important reason. If your industry is regulated (finance, healthcare, government), you may need requests to stay within your infrastructure or a specific geography. A self-hosted gateway lets you enforce that. Managed gateways (Vercel, Cloudflare) route through their infrastructure first.

How Self-Hosted LLM Gateways Work

A self-hosted gateway runs on your servers, receiving API calls from your application and routing them to providers. The architecture is simple: listen for HTTP requests in your format, translate them to the provider's format, call the provider, translate the response back, and return it to your app.

Request flow: Your app sends POST /v1/chat/completions to your gateway with an OpenAI-compatible payload. The gateway decrypts the virtual API key, looks up the routing rules, selects a provider (Claude, GPT-4, local Ollama, or a fallback), translates the request to that provider's format, makes the call, translates the response, and streams or returns it.

Key capabilities vary by tool but typically include: (1) multi-provider routing, allowing you to define rules like "if OpenAI is slow, try Anthropic", (2) virtual API keys, so you issue keys to your team without exposing real provider credentials, (3) spend tracking, logging every request with cost estimates, (4) semantic caching, returning stored responses for similar queries, and (5) MCP server support, letting you attach custom tools to the gateway. LibreChat specifically adds a multi-user chat interface on top of this routing layer, plus team and organization management.

State management is essential. The gateway needs to store: (1) virtual keys and which real provider credentials they map to, (2) routing rules (which provider for which request patterns), (3) cost logs (one row per request), and (4) team and user metadata. This requires a database, typically PostgreSQL, and optionally Redis for request caching. Every request hits the database. At scale (millions of requests per month), database latency becomes your bottleneck.

Deploying a Self-Hosted LLM Gateway: Docker and Kubernetes

Deploying a gateway requires containerization, orchestration, state management, and networking. Docker gets a single instance running on a developer's laptop. Kubernetes scales it across multiple nodes and handles failures, but configuration complexity jumps sharply.

For Docker, start with docker-compose. You define the gateway container, a PostgreSQL database container, and optionally a Redis cache. Environment variables hold your real LLM provider keys (OpenAI, Anthropic, etc.) and gateway configuration. The gateway starts, creates database tables on first run, and listens on port 8000 for requests. A sample docker-compose.yml looks like:

version: '3.8'
services:
  gateway:
    image: litellm:latest
    ports:
      - "8000:8000"
    environment:
      - DATABASE_URL=postgresql://user:password@postgres:5432/gateway
      - OPENAI_API_KEY=sk-...
      - ANTHROPIC_API_KEY=sk-ant-...
    depends_on:
      - postgres
  postgres:
    image: postgres:15
    environment:
      POSTGRES_PASSWORD: password
      POSTGRES_DB: gateway
    volumes:
      - postgres_data:/var/lib/postgresql/data
volumes:
  postgres_data:

For Kubernetes, you deploy the gateway as a StatefulSet (multiple replicas behind a Service), PostgreSQL as a StatefulSet with persistent storage, and optionally a ConfigMap for routing rules. Helm charts exist for most popular gateways, reducing boilerplate. A production Kubernetes setup also requires: (1) ingress controller for external routing (nginx-ingress or similar), (2) persistent volumes for the database, (3) resource limits (memory, CPU) per pod, (4) health checks and readiness probes, (5) RBAC for service accounts, and (6) network policies to restrict which pods talk to which.

Database setup is critical. PostgreSQL must run on persistent storage, not ephemeral pod storage, or you lose all data when the pod restarts. You need backup scripts (pg_dump to S3 nightly), monitoring (alerts if replication lag grows), and a disaster recovery plan. For teams under 50 people, a managed PostgreSQL service (AWS RDS, Google Cloud SQL) is simpler than self-hosting Postgres in Kubernetes.

The Real Operational Costs of Self-Hosting

Self-hosting a gateway costs money for infrastructure and time for operations. Most teams underestimate the time cost.

Infrastructure costs include: (1) compute for the gateway containers (typically t3.medium or larger on AWS, ~$30-80/month per node, sometimes just $10-20 on Hetzner or DigitalOcean), (2) PostgreSQL database (managed service $50-200/month, or self-hosted Postgres on Kubernetes adds $30-50/month), (3) Redis cache if you use it ($10-50/month), and (4) storage for logs and backups ($10-30/month on S3). Total infrastructure: roughly $100-350/month for a small team.

But the operational burden dwarfs infrastructure cost. A self-hosted gateway requires: (1) monitoring and alerting (you need dashboards showing request latency, error rates, cost per provider), (2) regular updates (the gateway software gets security patches and bug fixes), (3) incident response (when the gateway is slow or down, you investigate), (4) backup and disaster recovery testing (once a quarter, restore from backup to confirm it works), (5) credential rotation (real provider API keys must be rotated every 90 days), (6) security hardening (network policies, rate limiting on the gateway itself, DDoS protection), (7) on-call support (someone must respond at 3am if the gateway crashes).

Most teams discover they need a dedicated engineer after 1-2 incidents. Early on, a developer maintains it part-time. But as request volume grows and the business depends on the gateway, downtime costs money (delays in AI feature releases, customer complaints, lost revenue). A single engineer costs $100-150K/year in salary. Even at 10% of their time (realistic once the gateway is in production), that's $10-15K/year in labor.

Hidden costs: (1) learning curve (your team must understand the gateway tool, debugging deployment failures takes time), (2) integration (connecting monitoring tools, logging to CloudWatch or Datadog, shipping traces to Jaeger), (3) testing (you need test coverage before deploying updates), and (4) technical debt (every update, the tool drifts from the codebase you built, and eventually a major version breaks your config).

Hetzner or DigitalOcean can lower infrastructure cost by 30-50% compared to AWS, but operational burden stays the same. You still need monitoring, updates, and on-call.

Comparing Self-Hosted LLM Gateways: Open-Source Options

The open-source landscape has fragmented into specialized tools. No single winner serves all use cases.

LiteLLM is the most widely adopted. It supports 100+ LLM providers (OpenAI, Anthropic, Ollama, Mistral, Groq, Azure, Bedrock, etc.), written in Python, and requires PostgreSQL for cost tracking. Strengths: broad provider support, mature codebase, extensive documentation, and a commercial proxy service if you want managed hosting. Considerations: Python is slower than compiled languages (latency matters at high request volumes), and the database becomes a bottleneck at scale. Best for: teams needing maximum provider flexibility with moderate request volumes.

Bifrost is newer and performance-focused. Written in Go, it's compiled, so latency is lower. The OpenZiti blog measured it at 11 microseconds overhead at 5,000 requests per second on a t3.xlarge instance. Bifrost natively supports MCP (Model Context Protocol) servers, so you can attach tools and knowledge bases without custom code. It's Apache 2.0 licensed. Considerations: smaller community than LiteLLM, and documentation is lighter. Best for: teams optimizing for latency and needing MCP integrations.

Portkey emphasizes reliability and fallback logic. It supports 250+ LLMs and includes guardrails (content filtering, prompt injection detection). Written in TypeScript/Node.js, it's faster than Python but slower than Go. Portkey offers both self-hosted and managed options, so you can start self-hosted and migrate to managed later. Considerations: commercial company behind it, so self-hosted features may lag the managed product. Best for: teams needing fallback logic and guardrails.

Kong AI Gateway extends Kong, a popular API gateway. If your team already runs Kong for other APIs, Kong AI Gateway fits naturally. It supports multiple LLM providers and integrates with Kong's existing plugins (rate limiting, authentication, logging). Considerations: Kong has a learning curve, and Kong AI Gateway adds another layer of complexity. Best for: teams already running Kong for API management.

Quick decision matrix:

LibreChat as Your LLM Gateway

LibreChat (45.6K GitHub stars, MIT license) is primarily a multi-user chat interface, but it's built on top of a multi-provider routing engine. Some teams use it as a gateway plus UI combined.

How LibreChat's routing works: You configure custom endpoints in the admin panel, pointing to any OpenAI-compatible API (OpenAI, Anthropic Claude, Ollama, local models, or another gateway). Users can select which endpoint to use per conversation. LibreChat tracks costs per endpoint and per user, and enforces quota limits. It supports fallback routing (if OpenAI is down, retry on Claude), virtual API keys per user, and semantic caching via Redis.

LibreChat also integrates MCP servers, allowing you to attach custom tools (code execution, web search, internal APIs) to any conversation. This is useful for teams building AI agents that need to call internal systems.

Why LibreChat as a gateway: If your team wants a chat interface (for testing, for non-technical users to access LLMs) AND a gateway (for cost tracking, fallback routing, team management), LibreChat bundles both. You don't need to deploy two separate tools.

Limitations: LibreChat's gateway is embedded in the chat interface, so the overhead of serving the web UI (JavaScript, CSS, database queries for UI state) means latency is higher than a dedicated gateway like Bifrost. LibreChat is not API-first; the HTTP API is secondary. If your use case is purely API (your application calls the gateway with no human chat interface), dedicated gateways are leaner.

Best use case for LibreChat: Small to medium teams (under 50 users) who want one tool for both chat testing and production API gateway. The team management features (teams, organizations, roles) make it easy to onboard users. For high-volume API-only workloads, use LiteLLM or Bifrost instead.

Self-Hosted vs. Managed: When Opsily Makes Sense

Self-hosting appeals when you have constraints (data residency, custom routing logic, zero-trust networking) that managed gateways cannot meet. But for most small teams, the operational burden outweighs the benefits.

Managed hosting (like Opsily's hosted LibreChat) runs the gateway for you. You pay a monthly fee instead of building infrastructure. Opsily handles updates, monitoring, backups, and on-call. The gateway scales automatically, databases are managed, and you get 99.9% uptime SLA. For most teams, this eliminates $200-500/month in operational labor cost.

Cost breakeven: Self-hosted costs roughly $100-350/month in infrastructure + $833/month in labor (10% of a $100K engineer = $833/month). Managed hosting with Opsily is typically $200-500/month depending on volume. Breakeven is roughly when your self-hosted labor cost exceeds the managed cost. For a team with one full-time DevOps engineer, managed hosting is cheaper. For a team with 5+ engineers and a dedicated platform team, self-hosting might be cheaper, but only if latency or compliance requirements force self-hosting anyway.

Operational burden: Opsily eliminates: (1) infrastructure decisions (they choose PostgreSQL, Redis, Kubernetes, node sizing), (2) monitoring and alerting (they own it), (3) updates (automatic and tested), (4) backups and disaster recovery (they test daily), (5) credential rotation (they handle it), (6) security hardening (they own DDoS mitigation, rate limiting, network policies).

You still own: (1) API key management (you control which keys go to which providers), (2) routing rules (you define which endpoint for which request pattern), (3) team and user management (you define who can access what).

EU data residency: If your business serves EU customers or is subject to GDPR, data residency is critical. Opsily runs managed LibreChat in ISO 27001-certified German data centers, so customer data never leaves the EU. Self-hosted gateways on AWS Germany or Hetzner also meet this, but you own the infrastructure and compliance responsibility. With Opsily, the data center handles ISO 27001 audits; you only need to confirm your own GDPR compliance (data processing agreement, consent, deletion rights). This shifts compliance work from your team to Opsily.

Decision tree: (1) If your team is under 20 people, choose managed unless you have a specific constraint (air-gapped deployment, custom routing logic, zero-trust network). (2) If you have 20-50 people with a dedicated DevOps or platform team, evaluate both. (3) If you have 50+ people and a mature infrastructure practice, self-hosting might be cost-effective. (4) If you need air-gapped deployment or custom routing, self-hosting is mandatory.

For most Marcus-level founders (10-100 person teams), Opsily's managed hosting is the rational choice. You get to production in days (not weeks of infrastructure work), compliance is handled, and you don't have on-call nights.

Start with Opsily's hosted LibreChat on the /hosting/librechat/llm-proxy page to get a zero-DevOps LLM gateway with full team management, cost tracking, and EU data residency.

Frequently Asked Questions

What exactly is a self-hosted LLM gateway?

A self-hosted LLM gateway is software you run on your own servers (via Docker, Kubernetes, or bare metal) that sits between your application and LLM providers. It translates requests, manages credentials, routes to different providers based on rules, and tracks costs. Your application talks to your gateway instead of to OpenAI or Anthropic directly.

Why would a team choose self-hosting over a managed gateway?

Self-hosting appeals when: (1) data must stay in your infrastructure or a specific geography (GDPR, healthcare compliance, air-gapped networks), (2) you need custom routing logic (your business logic determines which model to use), (3) you already have Kubernetes and DevOps expertise (marginal cost of running a gateway is low), or (4) you handle so many requests that managed pricing is prohibitive (>10 million requests per month, though this is rare).

Can I use Ollama with a self-hosted LLM gateway?

Yes. Most gateways (LiteLLM, Bifrost, Portkey, LibreChat) support OpenAI-compatible APIs, and Ollama exposes an OpenAI-compatible endpoint. You configure your gateway to route requests to your local Ollama instance, and the gateway handles routing, cost tracking, and fallback to commercial providers if needed.

What's the main difference between LiteLLM and Bifrost?

LiteLLM supports 100+ providers and is written in Python, making it broad but slower. Bifrost supports 50+ providers and is written in Go, making it fast (11 microseconds overhead per request at scale). Choose LiteLLM if you need maximum provider flexibility; choose Bifrost if latency is critical or you need MCP integrations.

Does LibreChat work as an LLM gateway?

Yes, but with caveats. LibreChat can route requests to multiple providers, track costs, and manage virtual keys. However, it's optimized for chat (the web interface adds overhead), not pure API performance. For API-only workloads at high volume, dedicated gateways are better. For teams wanting both chat testing and a production gateway, LibreChat is efficient.

How much does it cost to run a self-hosted LLM gateway?

Infrastructure costs roughly $100-350/month (compute, database, storage). Operational labor costs roughly $833/month per 10% of an engineer's time, often growing to $1,500-3,000/month as the gateway scales and incidents occur. Total: $933-3,350/month for a small team. Managed hosting (Opsily) is $200-500/month, often cheaper than self-hosted when labor is included.

What infrastructure do I need for a production self-hosted gateway?

Minimum: (1) compute (1-2 VM instances, t3.medium or larger on AWS, or equivalent on Hetzner), (2) a managed PostgreSQL database (AWS RDS, Google Cloud SQL) or self-hosted Postgres on Kubernetes with persistent storage, (3) optional Redis for caching, (4) monitoring and alerting (Datadog, New Relic, or open-source Prometheus), (5) log aggregation (CloudWatch, ELK Stack, or similar), and (6) backups to S3 or equivalent storage. If you use Kubernetes, add ingress controller, service mesh (optional but recommended), and network policies.

When should I switch to a managed gateway instead of self-hosting?

Switch to managed if: (1) infrastructure maintenance is taking >5 hours per week from your team, (2) you've had two or more incidents (downtime, data loss, security issues) in the past year, (3) your compliance auditor is asking questions about your backup and disaster recovery procedures, or (4) your CEO is asking why you have an engineer on-call for a gateway when other parts of the business need that person.

The Bottom Line

Self-hosted LLM gateways give you control and can lower costs if you already have spare DevOps capacity. But they demand infrastructure, monitoring, updates, and on-call support. The operational burden is the real cost, not infrastructure.

For most teams under 100 people without specific compliance constraints, a managed option like Opsily's hosted LibreChat wins on time-to-value and total cost of ownership. You trade infrastructure ownership for operational peace of mind, and you get EU data residency and GDPR compliance built in.

Start by exploring what a managed LLM gateway can do for your team without building infrastructure.