How to Use AnythingLLM: Step-by-Step Setup & Tutorial

AnythingLLM is an open-source application you install locally (or on a server) to chat with your own documents using AI. You create a workspace, upload a PDF, and ask questions with cited answers. It uses retrieval-augmented generation (RAG) to answer from your data, not the LLM's training set. No API bills for document storage. No data leaves your machine.

What is AnythingLLM?

AnythingLLM is a desktop and server application (MIT licensed, open-source) built by Mintplex Labs and backed by YCombinator (Summer 2022). It has 65,800+ GitHub stars, 200+ contributors, and 5M+ Docker pulls. At its core, it does three things: chat with your documents, build AI agents that use tools (web, file operations, document search), and run entirely on your machine or server with no data sent to third parties.

You should use AnythingLLM if you need to keep documents private (healthcare, legal, finance), want to avoid API costs for document storage, or need to add AI chat to internal workflows without relying on SaaS. It's not a replacement for ChatGPT. It's a container for your data and your LLM of choice.

System requirements: minimum 2GB RAM for desktop, 4GB for Docker production. CPU and disk depend on your document volume. If you use a cloud LLM (OpenAI, Claude), you only pay the model provider. If you use Ollama (local LLM), you need a modern CPU or GPU.

Installation & Setup (Choose Your Path)

AnythingLLM runs three ways: as a desktop app (Windows, macOS, Linux), as a Docker container, or deployed to cloud platforms (Railway, Render, etc.). For solo use or testing, desktop is simplest. For multi-user or production, Docker is standard. Choose based on who accesses it and what performance you need.

Desktop App:

1. Visit anythingllm.com and download the installer for your OS.
2. Run the installer. First launch opens a setup wizard.
3. It prompts you to choose an LLM provider (Ollama, OpenAI, Anthropic, etc.).
4. Click 'Continue' and you're in.

Desktop runs on your laptop. It's fast for small teams and documents up to a few GB.

Docker:

The most common path for self-hosting. Docker isolates AnythingLLM in a container, makes it portable, and scales better than desktop.

1. Install Docker on your server. (See docker.com.)
2. Pull the image: docker pull mintplexlabs/anythingllm
3. Run the container with persistent storage:

   docker run -d -v anythingllm_storage:/app/server/storage -p 3000:3000 mintplexlabs/anythingllm
   

4. Access it at http://localhost:3000 (or your server IP:3000).
5. On first visit, the setup wizard prompts you to configure an LLM provider and embedding model.

Docker gives you full control: persistent storage, easy updates, and multi-user access if you set up authentication.

Cloud Deployment:

If you don't want to manage a server, Railway and Render offer one-click AnythingLLM deploys with persistent storage. Visit docs.useanything.com for current cloud partner links. Cost varies: Railway starts around $5/month for hobby tier; Render is similar. You still manage LLM provider costs separately.

Create Your First Workspace

A workspace is your isolated data silo in AnythingLLM. You can have multiple workspaces (one per project, per client, or per use case). Documents, chat history, and LLM config are separate per workspace. This matters for privacy, organization, and if multiple people use the same installation.

Create a Workspace:

1. Log in or access AnythingLLM.
2. Click 'New Workspace' (or the + icon).
3. Name it (e.g., 'Product Docs', 'HR Handbook').
4. Select an LLM provider from the dropdown. Options include:
   • Ollama (free, local, requires Ollama running on your machine)
   • OpenAI ($0.02-$0.03 per 1K tokens, depending on model)
   • Anthropic Claude (similar pricing to OpenAI)
   • Others: Gemini, Mistral, LLaMA2
5. Select an embedding model (used to convert your documents into searchable vectors). Default is Ollama embeddings if Ollama is running; otherwise use text-embedding-3-small (OpenAI, $0.02 per 1M tokens).
6. Click 'Create Workspace'.

Why Workspace Isolation Matters:

Each workspace has its own document library and chat history. If you add a 500-page manual to one workspace and a policy document to another, the LLM only searches within the workspace you're chatting in. This prevents confusion and keeps sensitive documents from mixing. It also controls costs if you're using cloud LLMs.

Upload a Document and Chat (RAG Workflow)

Retrieval-augmented generation (RAG) is AnythingLLM's core feature. You upload a document. AnythingLLM converts it into chunks, embeds each chunk into a vector, and stores it in its local database. When you ask a question, it finds the most relevant chunks and feeds them to the LLM with your question. The LLM answers based on what it found, not just its training data.

Upload a Document:

1. Open your workspace.
2. Click 'Add Documents' or drag a file into the left panel.
3. AnythingLLM accepts: PDF, TXT, MD, DOCX, and others. A 50-page PDF typically uploads in under 5 seconds.
4. You'll see 'Processing' briefly as it chunks and embeds. When done, the document appears in your workspace sidebar.

Ask Your First Question:

1. Type a question in the chat box (e.g., 'What are the system requirements?').
2. Hit Enter. AnythingLLM searches your documents for relevant passages, sends them to the LLM, and the LLM generates an answer.
3. Answers include citations: you'll see where the quote came from. Click it to view the source.

Attachment vs. RAG: When to Use Each:

• RAG (default): Upload a document to the workspace. Every chat question searches it. Use for large documents, many files, or when accuracy matters.
• Attachment: Drag a file into a single chat message (without uploading it to workspace). The LLM sees it once for that message. Use for quick one-off files or when you don't want documents stored long-term.

RAG is slower (milliseconds) because it searches a vector database, but accuracy is higher. Attachment is instant but only sees the file you mention.

For a step-by-step example: upload your product manual, ask 'How do I reset the device?', and you get an answer with a page number citation.

Advanced RAG Tuning (for Accuracy):

If answers aren't accurate, start here:

• Raise the similarity threshold in settings (default 0.25). Higher threshold means tighter matches; you get fewer but better results.
• Enable reranking if available (uses a second model to re-score chunks). Slower but more accurate.
• Check chunking strategy: if documents are fragmented, chunks might miss context. Adjust chunk size in workspace settings.
• Use multiple documents: if one manual is vague, add related docs.

For more on document-based AI workflows, see Opsily's managed AI document search hosting.

Using AI Agents

Agents are a step beyond simple chat. An agent can use tools: search your documents, browse the web, run code, write files. In a single prompt, it reasons about which tools to use, calls them, and synthesizes results.

Enable Agent Skills:

1. In workspace settings, under 'Agent Features', toggle on the skills you want:
   • Web Browsing: agent can search the internet and read URLs.
   • File Operations: agent can read/write files on your system.
   • Document Search: agent searches your workspace documents.
   • Code Execution: agent runs Python or Node snippets.
   • Summarization: agent condenses long text.
2. Save settings.

Trigger Agent Mode:

In the chat box, type @agent followed by your prompt. Agents work best for complex questions that need multiple sources. Example:

'@agent: Summarize our Q4 financials from the uploaded PDF and compare them to industry benchmarks you find online.'

The agent then:

1. Decides it needs document search (Q4 financials) and web browsing (benchmarks).
2. Fetches your document, extracts numbers.
3. Searches for industry data.
4. Synthesizes a comparison.
5. Returns a multi-paragraph answer with citations.

Simple Workflow Example:

You have 100 pages of customer feedback PDFs. You ask: '@agent: What are the top 5 customer pain points?' The agent skims your docs, identifies pain points, and ranks them by frequency. No manual reading.

Agents are more expensive (more LLM tokens, more API calls) but handle messy, multi-step questions better than basic chat.

Common Setup Issues & Troubleshooting

Most issues fall into three buckets: connection, embedding, accuracy.

Ollama Connection Errors:

Error: 'Failed to connect to Ollama.' This means AnythingLLM can't reach your local Ollama server.

Fix:

1. Check Ollama is running: ollama serve (in a terminal).
2. Ensure AnythingLLM is set to the default Ollama port, localhost:11434.
3. If Ollama is on a different machine, use its IP address instead of localhost.
4. Restart AnythingLLM.

Embedding Failures:

Error: 'Embedding failed. Out of memory.' Documents won't upload or chunks aren't embedding.

Cause: Your embedding model needs RAM. Ollama embeddings use ~1-2GB. OpenAI remote embeddings use negligible local RAM.

Fix:

1. Switch to OpenAI embeddings (Settings > Embedding Provider). Costs ~$0.02 per 1M tokens, often worth it.
2. If staying local, add RAM to your machine or reduce document size (upload one manual at a time).
3. Close other apps. Embedding is memory-hungry.

Poor Chat Accuracy / Hallucinations:

Problem: The LLM answers your question but the answer isn't in your documents (or it makes up details).

Causes:

1. Documents aren't being retrieved. Check similarity threshold. Default 0.25 is loose; try 0.5 for tighter matches.
2. Chunks are too small (lose context) or too large (noise). Adjust in workspace settings. Typical chunk size: 1000-2000 tokens.
3. Wrong embedding model. Embedding quality varies. OpenAI's text-embedding-3-small is reliable; Ollama's nomic-embed-text is free but less accurate.
4. LLM itself is weak. Ollama's Mistral is better than Llama2 for Q&A. OpenAI's GPT-4 is best but costs more than GPT-3.5.

Fix (in order):

1. Raise similarity threshold to 0.5-0.7.
2. Enable reranking if available.
3. Switch to OpenAI embeddings.
4. Use a stronger LLM (GPT-4 vs. GPT-3.5, or Mistral vs. Llama2).

Vector Database / Storage Issues:

If you lose documents after restarting, it's a storage mount issue in Docker. Ensure -v flag persists storage:

docker run -d -v anythingllm_storage:/app/server/storage...

Without -v, data is lost when the container stops.

Next Steps: Self-Hosting vs. Managed

You now know how to use AnythingLLM. The next decision is: run it yourself or let someone manage it?

Self-Hosting (DIY):

Pros: Full control, no vendor lock-in, meets data residency rules. Cons: You patch security updates, scale storage, handle backups, debug outages.

Minimum setup: $5-10/month (cloud server) plus LLM costs ($0-20/month depending on model). Time commitment: 2-4 hours initial setup, 1-2 hours/month maintenance.

Docker on a VPS (Hetzner, Linode, etc.):

1. Rent a VPS ($5-15/month).
2. SSH in and run the Docker command from the Installation section.
3. Set up a reverse proxy (Nginx) and SSL cert (Let's Encrypt) for security.
4. Monitor disk and RAM monthly.

Managed Hosting (AnythingLLM-as-a-Service):

Pros: No DevOps. Updates happen automatically. Backups are managed. Integrations are simpler (auth, rate limits, webhooks). Cons: Slightly higher cost ($15-50/month depending on plan). You rely on the provider's uptime.

Opsily provides managed AnythingLLM hosting with pre-configured LLM providers, multi-user auth, and scaling. You stop worrying about 'Is disk full?' or 'Is Docker running?'.

For teams of 5+ or document libraries >10GB, managed hosting typically pays for itself in saved ops time.

Learn more at Opsily's AnythingLLM hosting.

Frequently Asked Questions

Is AnythingLLM free to use?

Yes. AnythingLLM itself is MIT licensed and free. You pay only for LLM providers (OpenAI, Claude) or embedding services. Local LLMs (Ollama) are free.

What LLM should I choose?

Trade-off speed vs. accuracy. Ollama's Mistral is fast and free; OpenAI's GPT-4 is slower but more accurate. Start with Mistral or GPT-3.5, upgrade if accuracy isn't sufficient.

How much RAM does AnythingLLM need?

Desktop: 2GB minimum. Docker: 4GB recommended for production. Add 2-4GB if using local LLMs (Ollama).

Can I use ChatGPT or Claude as my LLM?

Yes. In workspace settings, select OpenAI or Anthropic as the provider. Plug in your API key. AnythingLLM will use ChatGPT or Claude to answer questions based on your documents.

How do I improve accuracy?

Raise the similarity threshold to 0.5+. Enable reranking. Use better embeddings (OpenAI vs. Ollama). Provide more documents for context.

What's the best alternative to AnythingLLM?

PrivateGPT and Quivr are similar open-source tools. AnythingLLM has the broadest LLM provider support and easiest UX.

What if I have security concerns?

AnythingLLM is open-source. Code is auditable. If you self-host, no data leaves your network. Use TLS (HTTPS) and auth if exposing over the internet.

How do I run it in production with multiple users?

Use Docker and add authentication via reverse proxy (OAuth2, LDAP). Or use managed hosting that includes multi-user out of the box.

The Bottom Line

AnythingLLM is a straightforward tool. Install it, create a workspace, upload documents, ask questions. Accuracy depends on your documents, embedding model, and LLM. Costs are low (usually $0-20/month) unless you use advanced features heavily.

The main choice is DIY or managed. DIY is cheaper but requires ops time. Managed removes that burden. Start with desktop or Docker. Upload a sample document and test. If it works for your workflow, decide whether to self-host long-term or move to managed hosting.

Ready to deploy? Check out Opsily's managed AnythingLLM hosting.