Open WebUI + vLLM Setup: A Production Guide

vLLM serves LLMs faster than Ollama for teams running multi-user Open WebUI deployments. It batches requests, reduces latency, and scales to higher throughput on the same GPU. This guide walks you through installation, connection, and troubleshooting so you can move beyond Ollama when your team outgrows it.

Why Run vLLM Behind Open WebUI?

vLLM is an inference engine optimized for throughput. Ollama is simpler, and for one user on a MacBook it is fine. For a 10-50 person team sharing one GPU server, vLLM's request batching makes a real difference: it groups multiple chat requests into a single forward pass, cutting latency by 30-60% under load. Open WebUI is the UI; vLLM is the engine that powers it.

Ollama works best for single-machine, interactive workloads. vLLM is built for production inference APIs with concurrent users. Both expose OpenAI-compatible endpoints, so switching is a configuration change, not a rewrite.

vLLM also supports quantization out of the box (AWQ, GPTQ, bfloat16) and lets you pick your serving framework: vLLM's default, or HuggingFace transformers if you need custom code. For teams running internal LLM services, this flexibility matters.

The trade-off is complexity. Ollama is a single binary. vLLM requires Python, pip, and care with dependency versions. If your team lacks DevOps bandwidth, Opsily's managed Open WebUI hosting handles this layer so you focus on prompts and integrations, not GPU memory fragmentation.

Hardware and GPU Requirements

vLLM loads entire models into GPU VRAM. A 7B model (Mistral, Llama 2 7B) needs 14-16 GB. A 13B model needs 24-28 GB. A 70B model needs 80 GB or more. These are minimums with bfloat16; fp32 (full precision) doubles the footprint.

Common hardware setups:

• Single RTX 4090 (24 GB): Run one 7B or 13B model comfortably. Not enough for 70B.
• Single RTX 6000 Ada (48 GB): 13B or 34B models. Start hitting limits with 70B.
• Dual H100 (80 GB each): 70B models, or 2 concurrent smaller models with tensor parallelism.

If your model does not fit in VRAM, vLLM can offload to CPU RAM, but that is 10-100x slower and beats the purpose. Quantization helps: a 70B model in 4-bit GPTQ takes 18-20 GB instead of 140 GB.

CPU matters too. vLLM uses CPU to prepare batches and decode tokens. A modern 16-core CPU (Xeon, EPYC, or equivalent) avoids bottlenecks. If you are on a small cloud VM with 2 vCPUs, you will see the CPU thread become the limiting factor before GPU saturation.

Network is usually not the constraint for internal deployments, but if Open WebUI and vLLM are on different machines, keep latency below 10 ms. Cross-datacenter setups over 100+ ms latency will feel sluggish.

Installing and Starting vLLM

Start with Python 3.10 or 3.11. Python 3.12 support is still rolling out in the vLLM ecosystem.

pip install vllm

Pull a model. Use a 7B model for testing; it is fast to download and runs on modest hardware. The command below downloads Mistral 7B from HuggingFace and starts an OpenAI-compatible server on port 8000:

vllm serve mistralai/Mistral-7B-Instruct-v0.2 --port 8000

That is it. vLLM will auto-detect your GPU, allocate VRAM, and start listening on localhost:8000. You should see logs like:

INFO: Uvicorn running on 0.0.0.0:8000
INFO: Initializing engine with 1 GPU and 32 CPU workers

Test the API:

curl localhost:8000/v1/models

You will get back a list of loaded models. Now Open WebUI can talk to it.

Docker Setup

If you are deploying this to a remote server, Docker is cleaner. Create a Dockerfile:

FROM nvidia/cuda:12.1.1-runtime-ubuntu22.04
RUN apt-get update && apt-get install -y python3-pip
RUN pip install vllm
CMD ["vllm", "serve", "mistralai/Mistral-7B-Instruct-v0.2", "--port", "8000"]

Build and run:

docker build -t vllm-server.
docker run --gpus all -p 8000:8000 vllm-server

The --gpus all flag gives the container access to all GPU devices. Without it, vLLM will not see the GPU and fall back to CPU (it will be very slow).

Connecting Open WebUI to vLLM: Two Methods

Open WebUI can pull models from a vLLM server in two ways. Pick one based on your deployment model.

Method A: Docker Environment Variables (Deployment-Time)

If you control the Open WebUI container, pass the vLLM endpoint at startup:

docker run -e OLLAMA_BASE_URL=vllm-server:8000 -p 3000:8000 ghcr.io/open-webui/open-webui:latest

This tells Open WebUI to fetch models from vllm-server:8000 instead of its default Ollama endpoint. If vLLM and Open WebUI are on the same Docker Compose network, use the service name (vllm-server). If they are on different machines, use the IP address or hostname of the vLLM server (e.g., 192.168.1.100:8000).

Method B: Admin Settings UI (Runtime Configuration)

If Open WebUI is already running, you can reconfigure it in the UI:

1. Log in as admin.
2. Go to Settings > Admin Settings > Connections.
3. Set the Ollama base URL to your vLLM server: vllm-server:8000 or your-vllm-ip:8000.
4. Click Save.
5. The model dropdown should now show models from vLLM instead of Ollama.

This method is slower to change (requires UI interaction) but lets you swap backends without restarting containers.

Network Connectivity Gotchas

If Open WebUI cannot reach vLLM, you will see "Connection refused" or timeouts in the model dropdown. Debug this:

1. From the Open WebUI container, test the endpoint:

   docker exec <open-webui-container> curl vllm-server:8000/v1/models
   

   If this fails, vLLM is not responding or the container cannot route to it.

2. Check the vLLM logs:

   docker logs <vllm-container>
   

   Look for errors about GPU initialization or port binding.

3. If both are on the same Docker Compose network: Make sure both services are defined in the same docker-compose.yml file. The service name is the hostname; no extra config needed.

4. If vLLM is on a remote server: Use the server's external IP. If it is behind a firewall, ensure port 8000 (or your chosen port) is open to the Open WebUI machine. Some cloud providers block inter-VM traffic by default.

5. DNS resolution: If you see "Name or service not known," the hostname is not resolving. Verify the hostname or use an IP address instead.

Performance Tuning and Optimization

Out of the box, vLLM works. Optimizing it takes a few knobs:

Batch Size and GPU Utilization

vLLM defaults to a max batch size of 128. For a single 7B model on a 24 GB GPU, this might be overkill; it will run out of VRAM before batching that many requests. Tune it:

vllm serve mistralai/Mistral-7B-Instruct-v0.2 --max-num-batched-tokens 8192

This caps the total tokens in a batch. A smaller number (e.g., 4096) uses less VRAM but reduces throughput. A larger number (e.g., 16384) uses more VRAM but batches more requests together. Experiment based on your VRAM and concurrency.

Quantization

If your model does not fit in VRAM, quantize it:

vllm serve mistralai/Mistral-7B-Instruct-v0.2 --quantization awq

This loads a 4-bit AWQ version of Mistral (if available on HuggingFace). Token throughput drops by ~5-10%, but VRAM usage is cut by 75%. For teams with tight GPU budgets, this trade-off is worth it.

Token Throughput and Latency

vLLM reports throughput metrics in the logs. Look for lines like:

Completions per second: 125.4
Average token latency: 12.3ms

If latency is high (> 50 ms per token) during inference, either your GPU is saturated or your batch size is too large. Reduce --max-num-batched-tokens and re-run.

Multi-GPU Setup

If you have 2+ GPUs, vLLM can tensor-parallelize large models:

vllm serve meta-llama/Llama-2-70B-hf --tensor-parallel-size 2

This splits the 70B model across 2 GPUs, reducing per-GPU VRAM from 140 GB to 70 GB. It adds latency (inter-GPU communication) but lets you run models that do not fit on a single GPU.

Common Issues and Troubleshooting

Connection Refused

Symptom: "Connection refused" when Open WebUI tries to reach vLLM.

Fix:

• Ensure vLLM is actually running: docker ps | grep vllm or check the process on your machine.
• Ensure the port is correct: default is 8000. If you changed it, update both the vLLM start command and the Open WebUI config.
• If vLLM is on a remote server, verify firewall rules allow inbound traffic to that port.

GPU Out of Memory

Symptom: "CUDA out of memory" error in vLLM logs when starting the server or during inference.

Fix:

• Your model is too large for your GPU. Either switch to a smaller model, quantize it, or add more GPU memory.
• Check VRAM: nvidia-smi shows per-GPU usage. A 7B model in bfloat16 needs ~15 GB; if your GPU has 8 GB total, it will not fit.
• Try quantization: --quantization awq reduces memory by 75%.
• Or reduce batch size: --max-num-batched-tokens 2048.

Slow Token Generation

Symptom: Responses from Open WebUI take 30+ seconds for a short completion, or tokens arrive very slowly.

Fix:

• Check if the GPU is thermal throttling: nvidia-smi under load. If temp is above 85C, throttling may kick in. Improve airflow or reduce batch size.
• Check CPU usage: top or htop. If CPU cores are at 100%, the CPU is the bottleneck. vLLM has many workers; reduce --tensor-parallel-size or the number of CPU workers.
• Check network latency between Open WebUI and vLLM: ping <vllm-server> should be < 5 ms. If it is > 50 ms, network is the issue.

Models Download Very Slowly

Symptom: First inference is very slow as the model downloads from HuggingFace.

Fix:

• This is normal for large models (50+ GB takes time). Pre-download the model before starting vLLM:

  huggingface-cli download mistralai/Mistral-7B-Instruct-v0.2 --cache-dir /path/to/cache
  

• Then start vLLM with that cache directory: HF_HOME=/path/to/cache vllm serve mistralai/Mistral-7B-Instruct-v0.2.

Request Timeout

Symptom: Open WebUI times out waiting for a response from vLLM, especially under load.

Fix:

• If the model is generating very long outputs (500+ tokens), it is just slow, not broken. Increase the timeout in Open WebUI settings if possible.
• If it happens on short outputs, vLLM is overloaded. Reduce batch size or add another GPU.
• Check vLLM logs for errors: docker logs <vllm-container> to see if the server crashed.

When to Consider Managed Hosting

Running vLLM in-house is free after you buy hardware, but the DevOps overhead adds up. You need to:

• Manage GPU drivers and CUDA versions across updates.
• Monitor GPU health and temperature.
• Handle model caching and updates.
• Scale horizontally when a single GPU is not enough.
• Deal with network and Docker debugging when things break.

For a 10-50 person team without a dedicated ML engineer, this is 10-20 hours per month of troubleshooting and maintenance.

Opsily's Open WebUI managed hosting removes this layer. You get a production vLLM instance behind Open WebUI, auto-scaling based on load, model updates handled for you, and a live demo environment to test before committing. You can compare the cost of managed vs DIY and decide if the DevOps hours are worth the savings.

If you are already running Open WebUI on Ollama and need more throughput, vLLM is the next step before managed hosting. If you are starting fresh and your team is small, managed hosting may be faster to value.

Frequently Asked Questions

What is vLLM and how does it differ from Ollama?

vLLM is an inference engine that serves large language models with batching and optimization for high throughput. Ollama is a single-binary tool focused on ease of use and local inference. vLLM is faster under concurrent load; Ollama is simpler to install.

Can I use vLLM with models other than Mistral?

Yes. vLLM supports any model on HuggingFace: Llama, Qwen, Phi, Mixtral, and thousands more. The command is vllm serve <model-id>. If the model is private or fine-tuned, ensure your HuggingFace token is set: huggingface-cli login.

Do I need a GPU to run vLLM?

No, but you should. vLLM on CPU is 50-100x slower than on GPU. For testing, it works, but for production or team use, a GPU is essential.

How much does vLLM cost?

vLLM is open source and free. The cost is hardware: a GPU like an RTX 4090 (1,600 USD) or renting one on a cloud provider (0.50-5.00 USD per hour depending on the GPU).

Can Open WebUI connect to multiple vLLM instances?

Not directly in the UI. Open WebUI connects to one backend at a time. To load-balance across multiple vLLM servers, use a reverse proxy like NGINX or HAProxy in front of them, then point Open WebUI to the proxy.

What quantization format should I use: AWQ, GPTQ, or bfloat16?

AWQ is fastest and smallest. GPTQ is older but works with more models. bfloat16 is slower than both but supports any model. Start with AWQ if the model exists in that format; fall back to bfloat16 otherwise.

How do I update vLLM to a newer version?

Run pip install --upgrade vllm. If you use Docker, rebuild the image: docker build -t vllm-server. && docker run.... Restart the container and check the logs for any breaking changes.

Can I run vLLM on multiple machines for failover?

Yes, but it requires orchestration. A simple approach: run two vLLM instances and put an NGINX reverse proxy in front of them. Configure Open WebUI to hit the NGINX endpoint. If one vLLM server goes down, NGINX routes to the other. More advanced setups use Kubernetes for auto-scaling and health checks.

The Bottom Line

vLLM is faster than Ollama for serving models to multiple users at the same time. It is the right choice when your team outgrows a single Ollama instance. Installation is straightforward: install vLLM, run a model, point Open WebUI to it, and test.

The real work is keeping up with GPU drivers, CUDA versions, model updates, and scaling. If your team lacks DevOps bandwidth, consider whether managed Open WebUI hosting saves more time and money than running it yourself.

For technical teams already comfortable with GPU infrastructure, vLLM is a natural upgrade from Ollama. For everyone else, start with Open WebUI on managed hosting and only run vLLM locally if your specific use case requires it.