LibreChat RAG: Complete Setup and Optimization Guide

LibreChat has native RAG (Retrieval Augmented Generation) capability built in. It lets you upload documents, index them into a vector database, and have the LLM retrieve relevant context before answering. This solves a core problem all LLMs face: they hallucinate because they only know facts from their training data. RAG grounds responses in your actual files.

What is RAG and Why Use It in LibreChat?

RAG stands for Retrieval Augmented Generation. It is a two-phase process: first, the system retrieves relevant documents from your knowledge base using semantic search; second, it feeds those documents to the LLM as context while generating the answer. In LibreChat, you upload files (PDFs, Word documents, text files, etc.), the system indexes them into a vector database, and when you ask a question, it finds semantically similar content and passes it to the LLM. This prevents hallucination and grounds answers in your data.

Why does this matter? Standard LLMs like GPT-4 have a knowledge cutoff. They know what their training data contained, but nothing about your internal documents. If you ask a chatbot about your company's support policy, it will guess. RAG solves this by making your proprietary knowledge searchable and retrievable in real time. Every answer the LLM generates is tied to an actual document you control.

LibreChat's RAG implementation is particularly clean because it handles everything in one interface. You upload a file, and the system automatically chunks it, computes embeddings, stores vectors, and makes them searchable. You do not need to script this yourself or integrate separate tools. The file upload and retrieval happen asynchronously, so large batches do not block the chat UI.

Common use cases for LibreChat RAG include: customer support teams answering questions from your documentation; internal wiki systems for employees to ask about company policy; legal teams grounding contract analysis in actual documents; product teams using RAG to answer customer questions about feature details and release notes; and knowledge workers querying archives of meeting notes or research papers.

How LibreChat RAG Works: The Architecture

LibreChat RAG runs on a FastAPI backend paired with LangChain for orchestration and semantic search. The vector database stores embeddings: typically PostgreSQL with the pgvector extension or MongoDB Atlas for managed hosting. When you upload a file, LibreChat does not store it raw. Instead: (1) the file is split into chunks (default 1024 tokens, with 20% overlap); (2) each chunk is converted to a vector embedding via an embedding provider (OpenAI, Ollama, etc.); (3) the vectors and chunk text are stored in the database; (4) metadata (file_id, upload timestamp, user_id) is attached so the system can track which user uploaded which file.

When you ask a question, here is the retrieval flow: (1) your question is converted to a vector using the same embedding model; (2) the vector database runs a semantic similarity search and returns the top K most similar chunks (default K=4); (3) those chunks are formatted as context and injected into the LLM prompt; (4) the LLM generates an answer using both its base knowledge and the retrieved context.

This two-phase design has real advantages. First, retrieval is fast: vector search on a well-indexed database returns results in milliseconds. Second, it is transparent: you can see which documents were retrieved and audit the answer. Third, it is scalable: adding more documents does not slow down the LLM; it only increases database load, which is parallelizable.

LibreChat also implements file-level isolation for multi-user environments. Each file upload is tagged with a file_id and associated with a user or team. When you search, RAG only retrieves from documents you have access to. This is crucial if multiple teams use the same LibreChat instance but need separate knowledge bases.

The system is designed for asynchronous processing. If you upload a 100-page PDF, LibreChat queues the chunking and embedding work and processes it in the background. The UI remains responsive. Large uploads can take minutes, but you are not locked waiting.

Prerequisites: What You Need to Run RAG

To run LibreChat RAG, you need a vector database, an embedding provider, sufficient system resources, and network connectivity. At minimum: a PostgreSQL instance with the pgvector extension enabled (8-16GB storage for medium-sized knowledge bases), an embedding provider API key or local model, at least 4GB RAM on the server running LibreChat, and stable outbound network connectivity if using cloud embedding providers.

The vector database is non-negotiable. PostgreSQL with pgvector is the most common choice for self-hosted setups. A standard PostgreSQL instance (even small cloud instances) can handle millions of vectors. If you prefer managed databases, MongoDB Atlas has vector search built in and is easier to scale. For testing locally, you can use PostgreSQL in Docker.

Embedding providers come in two categories: cloud-hosted and local. Cloud providers (OpenAI, Azure, HuggingFace Inference API) are easiest to set up but cost money per embedding and have API rate limits. Local models (Ollama running an embedding model) cost nothing per embedding but require GPU resources on your server. Ollama can run on modest hardware: an M1 Mac or 8GB Linux server can handle inference for small teams.

System resources scale with knowledge base size. For up to 50 documents (500MB of text): 4GB RAM, 2 CPU cores, and 50GB disk is enough. For 500+ documents: 16GB RAM, 4+ CPU cores, and 500GB disk recommended. The RAG API process itself is not memory-hungry; the load comes from concurrent requests and the database query overhead.

Network connectivity matters if you use cloud embedding providers. If your LibreChat instance sits behind a firewall, ensure outbound HTTPS is allowed to your embedding provider. For on-premise deployments, you may need to use local embeddings only (Ollama) to avoid data leaving your network.

Supported Vector Databases & Embedding Providers

LibreChat officially supports these vector databases:

For embedding providers, LibreChat supports:

The most common setup for small teams is OpenAI + managed PostgreSQL. The most cost-effective setup for teams with DevOps skills is Ollama + self-hosted PostgreSQL. The most enterprise-friendly is Azure OpenAI + managed Azure Database for PostgreSQL.

Core Configuration: Environment Variables & Setup

LibreChat RAG is configured via environment variables. Here are the critical ones:

Database Configuration:

• DB_HOST: Hostname of your PostgreSQL or MongoDB instance (e.g., localhost or postgres.example.com)
• DB_PORT: Port number (default: 5432 for Postgres, 27017 for Mongo)
• POSTGRES_USER / POSTGRES_PASSWORD: Database credentials
• POSTGRES_DB: Database name (default: rag_api)
• PGVECTOR_CREATE_EXTENSION: Set to False if using managed PostgreSQL (e.g., AWS RDS) where you cannot create extensions directly

Embedding Configuration:

• EMBEDDINGS_PROVIDER: Which embedding service (options: openai, azure, huggingface, ollama, bedrock, vertexai, googlegenai)
• EMBEDDINGS_MODEL: The specific model (e.g., text-embedding-3-small for OpenAI, nomic-embed-text:latest for Ollama)
• EMBEDDINGS_API_KEY: API key for cloud providers (not needed for Ollama)
• EMBEDDINGS_BASE_URL: Base URL for the embedding service (required for local Ollama deployments)

RAG Processing:

• CHUNK_SIZE: Number of tokens per chunk (default: 1024). Smaller chunks (512-1024) for dense documents, larger (2048+) for sparse text
• CHUNK_OVERLAP: Overlap between chunks as percentage (default: 20%). Higher overlap improves context continuity but increases storage
• TOP_K: Number of retrieved chunks to pass to the LLM (default: 4). Higher values give more context but risk exceeding token limits

API Configuration:

• RAG_API_URL: URL where RAG API is accessible (e.g., local testing or production server)
• RAG_PORT: Port the RAG API listens on (default: 3090)

A minimal Docker Compose setup looks like this:

version: '3.8'
services:
  postgres:
    image: pgvector/pgvector:pg16
    environment:
      POSTGRES_USER: rag_user
      POSTGRES_PASSWORD: secure_password
      POSTGRES_DB: rag_api
    volumes:
      - postgres_data:/var/lib/postgresql/data
  
  rag-api:
    image: danny-avila/librechat-rag-api:latest
    environment:
      DB_HOST: postgres
      POSTGRES_USER: rag_user
      POSTGRES_PASSWORD: secure_password
      EMBEDDINGS_PROVIDER: ollama
      EMBEDDINGS_MODEL: nomic-embed-text:latest
      TOP_K: 4
      CHUNK_SIZE: 1024
    ports:
      - '3090:3090'
    depends_on:
      - postgres
  
  ollama:
    image: ollama/ollama:latest
    ports:
      - '11434:11434'
    volumes:
      - ollama_data:/root/.ollama
    command: serve

volumes:
  postgres_data:
  ollama_data:

To start, run docker-compose up. The RAG API will be available at port 3090 and LibreChat should point to it.

Configuration best practices:

Chunk size is critical. If your documents are technical specs with lots of tables, use larger chunks (1024-2048 tokens) so context is not split mid-table. If your documents are conversational or have dense facts, use smaller chunks (512 tokens) for precise retrieval. Experiment: if retrieval feels vague, reduce chunk size; if it feels too granular, increase it.

TOP_K balances context quality and token usage. With TOP_K=4 (default), you retrieve approximately 4000 tokens of context, leaving room for the user's question and the LLM's response without hitting token limits. If your documents are sparse, increase TOP_K to 6-8. If you often hit token limits, reduce to 2-3.

Chunk overlap matters less than chunk size but improves continuity. The default 20% overlap ensures that important context is not accidentally split across two separate chunks. For dense documents, increase to 30%; for sparse docs, 10% is fine.

Common Issues & Troubleshooting

Slow file uploads: If uploading a 50-page PDF takes more than 1 minute, the bottleneck is usually the embedding provider. Check your API limits and rate limits. If using local Ollama, the GPU may be under-provisioned; a CPU-only embedding on Ollama can be slow (0.5 seconds per chunk). Solution: use a faster embedding provider (OpenAI is 100x faster than CPU Ollama), or upgrade your hardware. Also check if the database is missing an index on the file_id column; lazy indexing causes slow inserts.

Poor retrieval quality: You ask a question and get irrelevant results. Common cause: your chunk size is too large, mixing multiple topics in one vector. Try reducing CHUNK_SIZE from 1024 to 512. Another cause: embedding model mismatch. If you switch embedding providers mid-deployment (e.g., from Ollama to OpenAI), old vectors do not match new embeddings. You must re-upload all files or recompute vectors. Symptom: very low similarity scores. Solution: clear the database and re-index.

Token limit exceeded: The LLM complains it received too many tokens. This happens when TOP_K is too high and all retrieved chunks exceed the context window. Check your TOP_K setting and reduce it from 4 to 2-3. Also verify that individual chunks are not abnormally large; if CHUNK_SIZE is set to 4096, you will blow token limits quickly.

Connection refused errors: The LibreChat UI cannot reach the RAG API. Common causes: (1) RAG API is not running; (2) firewall blocks the port; (3) RAG_API_URL in LibreChat config points to wrong address. To debug, attempt to connect to the RAG service on its configured port. If that fails, the RAG service is not listening. If it succeeds but LibreChat still fails, check firewall rules and verify RAG_API_URL in your LibreChat env config.

Missing pgvector extension: You see errors like "extension pgvector not found". This happens with managed PostgreSQL (AWS RDS, Azure Database) where you cannot run CREATE EXTENSION. Solution: set PGVECTOR_CREATE_EXTENSION=False in your RAG API config, and ask your database provider to manually enable pgvector for your instance. AWS RDS and Azure Database require you to enable it through the parameter group or server configuration.

Out of memory during processing: The RAG API crashes when processing large files. This is rare but can happen if your server has less than 4GB RAM and you process a 500+ page document. Solution: increase server RAM, or pre-process large files into smaller PDFs before uploading.

To monitor RAG health, check the API logs with your container management tool. Look for errors on file upload and vector insertion. Set LOG_LEVEL=DEBUG for verbose output. For the database, monitor query performance: PostgreSQL slow query logs can reveal if embedding inserts are queuing up.

When to Upgrade to Managed LibreChat RAG Hosting

Self-hosted RAG works well for testing and small teams (1-5 people). But it creates operational burden. You must manage PostgreSQL: patching security updates, planning backups, monitoring disk usage, and handling failover if the server goes down. You must manage embedding API keys and rate limits. You must scale resources as your knowledge base grows. You must monitor and tune performance.

Managed LibreChat hosting (like Opsily's managed LibreChat hosting) handles all of this. You get a fully configured LibreChat instance with RAG pre-enabled, PostgreSQL managed by the hosting provider, automatic backups, security patches applied automatically, and scaling that adapts to your knowledge base size. You also get compliance guarantees: EU data residency, GDPR-compliant processing, SOC 2 auditing, and API access logs for compliance reporting.

When should you switch? If your team is larger than 5 people and uses RAG for production (customer support, internal knowledge base), managed hosting saves money on DevOps time. If you need GDPR compliance or work with EU customers' data, managed hosting is faster than self-hosting with proper configuration. If you do not have a DevOps engineer on staff, managed hosting eliminates operational risk.

The cost math usually works out: a small managed LibreChat instance with RAG costs $50-150/month depending on features. A single junior DevOps engineer costs $5,000-8,000/month. Even if you only save 2-3 hours per month on monitoring and patching, managed hosting pays for itself. For teams prioritizing speed-to-market over cost minimization, managed hosting is the obvious choice.

Opsily's managed LibreChat hosting includes RAG pre-configured with PostgreSQL, automatic embedding provider setup, and scaling that handles knowledge base growth without manual intervention. You deploy with one click and RAG is ready.

Frequently Asked Questions

What is the difference between RAG and fine-tuning?

Fine-tuning modifies the LLM's weights based on training data, making it permanently "remember" facts. RAG retrieves relevant documents at query time without modifying the model. RAG is faster to set up, cheaper, and more transparent (you can see which documents informed the answer). Fine-tuning is more powerful for changing the model's style or reasoning, but requires retraining after every knowledge update.

How do I ensure RAG does not retrieve confidential data?

LibreChat RAG respects file ownership: each file is tagged with a user_id or team_id, and retrieval is filtered by that tag. If you upload a file as user A, user B cannot search it. At the database level, add row-level security (RLS) to the vectors table to enforce this. For stronger isolation, run separate LibreChat instances with separate databases for each customer or team.

Can I use LibreChat RAG offline without an embedding API?

Yes. Run Ollama locally and point EMBEDDINGS_PROVIDER to ollama. This keeps embeddings fully offline. You will sacrifice latency (Ollama on CPU is 500ms per embedding vs 50ms for OpenAI), but zero data leaves your network. This is common in air-gapped environments and security-sensitive organizations.

What file formats does LibreChat RAG support?

LibreChat RAG accepts PDF, DOCX, TXT, and plain text uploads. It does not natively support images or scanned PDFs (OCR). If you need image support, extract text from images first (e.g., using Tesseract) and upload the text.

How long does it take to upload and index a 100-page PDF?

With OpenAI embeddings, typically 30-60 seconds. With Ollama on CPU, 5-10 minutes. The bottleneck is embedding computation. To speed up, use a faster embedding provider or GPU acceleration for Ollama.

Can I delete specific documents from the knowledge base without re-indexing everything?

Yes. LibreChat tracks file_id for each chunk. To delete a file, issue a delete request targeting that file_id, and all associated vectors are removed. You do not need to re-index the entire database.

The Bottom Line

LibreChat RAG is a production-ready feature that prevents LLM hallucination by grounding answers in your documents. It handles chunking, embedding, and semantic search transparently. To run it, you need PostgreSQL with pgvector (or MongoDB Atlas), an embedding provider (OpenAI is easiest; Ollama is cheapest), and 4GB+ RAM. Configuration is straightforward: a handful of environment variables define your database, embedding provider, and chunking strategy.

Self-hosted RAG works for small teams and testing. For production use, teams larger than 5 people should consider managed hosting to eliminate DevOps overhead and ensure compliance. Opsily's managed LibreChat hosting pre-configures RAG, handles scaling, and provides GDPR compliance guarantees.

Start with a local Docker Compose setup to validate that RAG fits your use case. Once you are confident, migrate to managed hosting or a hardened self-hosted setup with backups and monitoring.