LibreChat with LiteLLM: Setup, Budgets & Routing

LiteLLM is a proxy layer that lets you point LibreChat at 100+ LLM providers through a single, unified API. Instead of managing OpenAI keys, Anthropic keys, and Azure credentials separately, you configure LiteLLM once and LibreChat talks to it. But setup means running another service, managing virtual keys, and debugging Docker networking. This guide walks you through when it's worth that complexity, how to set it up, and what breaks along the way.

What is LiteLLM and Why Use It with LibreChat?

LiteLLM is an open-source proxy gateway (61.6K GitHub stars, actively maintained as of November 2026) that translates requests to 100+ LLM providers. Think of it as a translator: your app speaks OpenAI-compatible API, LiteLLM translates that to whoever you're calling (OpenAI, Anthropic, Cohere, local Ollama, etc.).

With LibreChat alone, you paste API keys directly into the config. With LiteLLM, you paste keys into LiteLLM, and LibreChat connects to LiteLLM's proxy endpoint. That one extra hop buys you three things:

Multi-provider switching. Use Claude for reasoning tasks, GPT-4o for speed, open-source Llama for cheap classification. Switch models per message without changing LibreChat's config.

Per-team budgets and virtual keys. Create a key for "engineering" capped at $500/month, another for "marketing" at $100/month. Each team calls the same LiteLLM endpoint but hits their own budget limit. Impossible without a proxy.

Automatic failover and cost-based routing. Route to OpenAI first; if it times out, try Anthropic. Or route simple queries to Claude 3.5 Sonnet ($3/1M tokens) and hard reasoning to Claude Opus ($15/1M tokens) based on estimated complexity.

The tradeoff: LiteLLM is another Docker container to run, monitor, and troubleshoot. Latency adds 50-150ms per request. If you have only one team, one provider, and no budget sensitivity, direct keys are simpler.

Architecture: How LiteLLM Fits Into LibreChat

The flow is: Browser LibreChat LiteLLM Provider Response.

LibreChat's config points to LiteLLM's endpoint (e.g., the LiteLLM service address) as a custom endpoint. LibreChat sends OpenAI-compatible POST requests (model name, messages, temperature, etc.). LiteLLM receives that, looks up the model name in its config, fetches the right API key, makes the real request to OpenAI/Anthropic/etc., and streams the response back.

LiteLLM's config file (litellm_config.yaml) defines the model mappings:

• gpt-4o your OpenAI API key + "gpt-4o" model
• claude-opus-4 your Anthropic key + "claude-3-5-opus-20241022" model
• llama2 your local Ollama instance

You can create virtual keys (e.g., sk-team-engineering-12345) and bind them to budgets and routing rules. Every request through that key is tagged, so LiteLLM knows which team made it and whether they've hit their monthly limit.

For setup, LiteLLM runs as a service alongside LibreChat. Both live in the same Docker Compose file. LibreChat's internal DNS resolves the LiteLLM container service name. This is why Docker networking matters: if your container can't reach the other container, the setup breaks.

Step-by-Step: Setting Up LiteLLM with LibreChat (Docker)

Assume you have a working LibreChat Docker setup. Here's what you add.

Step 1: Create litellm_config.yaml

Make a new file in your project root called litellm_config.yaml. This defines every model LiteLLM should know about:

model_list:
  - model_name: gpt-4o
    litellm_params:
      model: gpt-4o
      api_key: ${OPENAI_API_KEY}
      api_base: https://api.openai.com/v1
      timeout: 120

  - model_name: claude-opus-4
    litellm_params:
      model: claude-3-5-opus-20241022
      api_key: ${ANTHROPIC_API_KEY}

  - model_name: llama2-local
    litellm_params:
      model: ollama/llama2
      api_base: http://ollama:11434

general_settings:
  master_key: sk-your-master-key-here
  litellm_settings:
    max_budget: 100
    log_level: INFO
    enable_budget_alerts: true

Every model entry needs a model_name (what LibreChat calls it) and litellm_params (how LiteLLM calls the real provider). The master_key is what LibreChat uses to authenticate to LiteLLM. Store keys in environment variables (${OPENAI_API_KEY}) and load them at runtime.

Step 2: Add LiteLLM to docker-compose.override.yml

Add this service to your compose file:

services:
  litellm:
    image: ghcr.io/berriai/litellm:main
    ports:
      - "4000:4000"
    environment:
      - OPENAI_API_KEY=${OPENAI_API_KEY}
      - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}
    volumes:
      -./litellm_config.yaml:/app/config.yaml:ro
    command: litellm --config /app/config.yaml --port 4000 --num_workers 8
    restart: unless-stopped

This mounts your config file into the container and starts LiteLLM on port 4000 with 8 worker processes. The num_workers flag controls concurrency; 8 is reasonable for a team of 10-20 people.

Step 3: Update librechat.yaml for Custom Endpoint

In your librechat.yaml, add a custom endpoint entry:

endpoints:
  custom:
    baseURL: http://litellm:4000/v1
    apiKey: sk-your-master-key-here
    models:
      default: gpt-4o
      fetch: true

Set fetch: true so LibreChat queries LiteLLM for the list of available models on startup. This saves you from hardcoding every model name in LibreChat's config.

Step 4: Rebuild and Start

Run:

docker-compose down
docker-compose up -d

Check logs:

docker-compose logs litellm -f

You should see LiteLLM starting successfully on port 4000.

Then open LibreChat, select the custom endpoint, and test a message. LiteLLM logs should show successful requests.

Step 5: Verify with a Test Chat

In LibreChat, switch to a model defined in your config (e.g., gpt-4o). Send a message: "Say hello and tell me which model is processing this."

The response should come back and specify the model. If it doesn't, check:

1. Are litellm and librechat containers on the same network? Run docker network ls to confirm both are in the same compose project network.
2. Is the master key correct? Check docker-compose logs litellm for 401 errors.
3. Is fetch: true causing issues? Temporarily hardcode models in librechat.yaml and retry.

Virtual Keys and Team-Based Budgets

LiteLLM's killer feature is per-key budget tracking. Create one key per team, set a monthly spend limit, and LiteLLM enforces it.

Why budgets matter: If you give every team member direct API keys, they can burn through $10K in a weekend by accident (a runaway loop calling the API 100K times). With LiteLLM virtual keys and budgets, the second request after hitting $500 fails with a clear "budget exceeded" error.

Creating a virtual key:

LiteLLM stores keys in a database (Postgres, MongoDB, or in-memory SQLite). To create a key, either use the LiteLLM UI (if you expose the service port, though this is a security risk) or hit the API programmatically. The key generation endpoint accepts parameters for key alias, budget amount, budget duration, and allowed models.

Response includes the generated key, budget limit, duration, and current spend.

That key is now usable. Any app calling LiteLLM with that key in the Authorization header is part of a named team and subject to the configured spend limit.

Enabling persistent budget tracking:

By default, LiteLLM stores budgets in memory, which resets on container restart. To persist them, configure a Postgres database:

  litellm:
    environment:
      - DATABASE_URL=postgresql://user:password@postgres:5432/litellm

Now budgets survive restarts. LiteLLM will also log every request to the database, so you can audit who used what and how much.

Monitoring spend:

Query the LiteLLM API for current usage on any key to see current spend and remaining budget. Export these metrics to Prometheus or your monitoring tool to alert when a team hits 80% of their budget.

Routing and Fallback Strategies

Once you have multiple models configured, you can define routing rules: "try Claude first, fall back to GPT-4o if Claude is down."

Simple failover:

Edit litellm_config.yaml:

model_list:
  - model_name: smart-reasoning
    litellm_params:
      model: claude-opus-4
      api_key: ${ANTHROPIC_API_KEY}
      timeout: 60
    fallback:
      - model_name: gpt-4o
        litellm_params:
          model: gpt-4o
          api_key: ${OPENAI_API_KEY}

Now when a user requests smart-reasoning, LiteLLM tries Claude first. If Claude times out or returns a 5xx error, it automatically retries with GPT-4o. LibreChat sees no difference; the retry is internal.

Cost-based routing:

Route queries to the cheapest model that can handle them:

router_settings:
  - user_id: team-engineering
    routing_strategy: lowest_cost
    models_allowed:
      - llama2-local
      - gpt-4o
      - claude-opus-4

With this rule, "team-engineering" keys route to llama2-local (free, running locally on Ollama) by default. If llama2 fails 3 times in a row, LiteLLM escalates to gpt-4o ($0.003 per 1K input tokens). If that fails, it tries Claude Opus.

You control escalation thresholds (how many failures trigger fallback), retry delays, and cost budgets per routing tier.

Provider-specific routing:

Route different model families to different endpoints:

model_list:
  - model_name: gpt-4o
    litellm_params:
      model: gpt-4o
      api_base: https://api.openai.com/v1
      api_key: ${OPENAI_API_KEY}

  - model_name: claude
    litellm_params:
      model: claude-3-5-opus-20241022
      api_base: https://api.anthropic.com
      api_key: ${ANTHROPIC_API_KEY}

This is the simplest form: one endpoint per provider. LibreChat users pick which model they want; no automatic switching. Good for teams that trust their engineers to choose wisely.

For more complex routing (semantic routing, load balancing, A/B testing), explore LiteLLM's advanced configuration options.

Common Pitfalls and Troubleshooting

Here's what actually breaks in production.

Embeddings and RAG fail with DNS/networking errors:

LibreChat's RAG pipeline (Retrieval Augmented Generation) may use LiteLLM for embeddings. If fetch: true is enabled and your Docker network is misconfigured, the embedding model fails to initialize.

Symptom: LibreChat starts but RAG is grayed out. Logs show connection errors.

Fix: Ensure both containers are on the same network. In docker-compose, use the same compose project (no custom network creation). Or explicitly define the network:

networks:
  librechat:

services:
  librechat:
    networks:
      - librechat
  litellm:
    networks:
      - librechat

API key mismatches (401 errors from provider):

You pass the wrong key to LiteLLM, or the key is expired. LiteLLM logs show authentication errors.

Fix: Double-check every API key in litellm_config.yaml. Use environment variables (${OPENAI_API_KEY}) not hardcoded strings. Verify the key works with the provider's documentation or test tools.

Master key authentication failures (401 from LibreChat to LiteLLM):

LibreChat's master key doesn't match LiteLLM's. LiteLLM logs show authentication token errors.

Fix: Ensure librechat.yaml and litellm_config.yaml use the same master key:

In librechat.yaml:

apiKey: sk-my-secret-master-key

In litellm_config.yaml:

general_settings:
  master_key: sk-my-secret-master-key

They must match exactly.

Function calling doesn't work:

Some models support tool use (function calling); others don't. If you configure LiteLLM to proxy a model that doesn't support tools, requests fail silently or return incomplete responses.

Fix: Verify the downstream model supports the feature. Claude 3.5 Sonnet and GPT-4o both support tools. Older or smaller models may not. Check the provider's documentation, then test with your configuration.

Connection refused or timeout:

LibreChat can't reach LiteLLM. Check:

1. Is LiteLLM running? docker ps | grep litellm
2. Is the port correct? librechat.yaml says 4000; litellm container exposes 4000.
3. Are containers on the same network? docker inspect <container-id> to verify network settings.
4. Is a firewall blocking the connection? (Unlikely in Docker Compose, but possible in Kubernetes.)

LiteLLM vs Direct Provider Keys: When to Choose Each

Direct API keys are simpler. LiteLLM adds power but also complexity. Here's how to choose:

Choose direct keys if:

• You have only one provider (just OpenAI, or just Anthropic).
• You have one team and no budget concerns.
• You want the lowest possible latency (skipping the proxy hop saves 50-150ms).
• You're running LibreChat on a single machine with few users.

Direct is "configure once, forget it." No extra container, no routing logic to maintain.

Choose LiteLLM if:

• You use multiple providers and want to switch models without config changes.
• You need per-team budgets (billing, preventing runaway spend).
• You want automatic failover (provider A down, fall back to B).
• You have cost sensitivity (routing to cheaper models by default).
• You have a central DevOps team managing infrastructure (they can maintain the proxy).

LiteLLM is "more config upfront, more flexibility later."

The scaling argument: At 5 users, direct keys are fine. At 50 users across 5 teams, budgets save money. At 500 users with multi-region deployments, LiteLLM becomes a necessity for cost control.

Next Steps: From Self-Hosted to Managed

Running LiteLLM yourself means:

• Maintaining the Docker setup and networking.
• Managing database backups (if you use Postgres for budgets).
• Monitoring container health and restart policies.
• Scaling workers if request volume grows.

Opsily's managed hosting of LibreChat includes LiteLLM proxy management as part of the deployment. No containers to run. No config files to version-control. Opsily handles backups, monitoring, and scaling. See managed LibreChat hosting for pricing and options.

If you want to explore the LLM proxy concept further or learn about open-source model routing, those guides cover advanced setups.

For now, get the Docker setup working locally, test with one provider, then add complexity as you need it.

Frequently Asked Questions

How do I configure LiteLLM with LibreChat? Create a litellm_config.yaml file with your model definitions, add LiteLLM to docker-compose.yml, update librechat.yaml to point to the LiteLLM endpoint, and restart. Set fetch: true to let LibreChat auto-discover models.

How do I get embeddings and RAG to work with LiteLLM? Ensure both LibreChat and LiteLLM containers are on the same Docker network. If they are on different networks, LibreChat can't reach LiteLLM. Test your Docker network configuration and verify containers can communicate.

How do I set virtual keys and budgets in LiteLLM? Use the LiteLLM key generation API to create keys with budget parameters. Specify budget amount, budget duration, and allowed models. LiteLLM enforces limits at request time. Use a persistent database (Postgres) to survive container restarts.

Does LiteLLM support function calling with LibreChat? LiteLLM passes function calling requests through to the downstream provider. If the provider supports tools (Claude, GPT-4o), they work. Older or smaller models may not support them, returning empty tool responses.

What's the difference between LiteLLM and direct API keys? Direct keys: LibreChat talks straight to OpenAI/Anthropic. Simpler, lower latency, but no multi-provider switching or per-team budgets. LiteLLM proxy: One endpoint for many providers, per-team budgets, automatic failover. Trade-off is extra complexity and ~100ms latency overhead.

How do I route requests between multiple LLM providers in LibreChat? Define fallback chains or routing rules in litellm_config.yaml. Simple failover uses fallback clauses. Cost-based routing uses router_settings to prefer cheaper models. LiteLLM handles escalation automatically.

How do I troubleshoot LiteLLM connection errors in Docker? First, confirm both containers are on the same network. Then test the endpoint directly with standard Docker debugging tools. Common issues: wrong master key, expired API keys, or containers not in the same network.

The Bottom Line

LiteLLM is powerful if you have multiple providers, team-based budgets, or failover needs. If you have just one provider and one team, direct keys are simpler.

The setup is straightforward (4 steps, mostly config), but debugging Docker networking and API key mismatches takes time. Test with one provider first, then add routing and budgets as you scale.

If the operational overhead feels heavy, managed LibreChat hosting through Opsily handles this for you. Explore managed options here.