Ollama API Python Client: A Developer's Guide

The Ollama Python client is a lightweight library that lets developers interact with Ollama's REST API to run and chat with large language models locally or remotely. You use it to build applications that work with models like Llama, Mistral, and Gemma without relying on paid cloud APIs. This guide covers installation, core patterns, streaming, remote connections, function calling, and integration with Open WebUI.

What Is the Ollama Python Client?

The Ollama Python client is a wrapper around Ollama's REST API (github.com/ollama/ollama-python, with 10.5K GitHub stars). It provides a Pythonic interface to run models on your machine, another server, or managed infrastructure. Unlike cloud APIs, Ollama models run on hardware you control, which is why it appeals to privacy-conscious teams and organizations that want to avoid per-token billing.

Most developers use Ollama Python for three core reasons: privacy (data never leaves your server), cost (no per-token billing like OpenAI or Claude API), and developer velocity (simple API, no rate limits during development). Ollama itself is mature: the main repository has 179.8K GitHub stars and runs models like Llama 3.1, Mistral 8x7B, and Gemma 2 locally in minutes.

The client handles authentication, connection pooling, error retry logic, and request serialization so you don't have to reinvent these patterns. It also supports both synchronous and asynchronous execution, which matters if you're building web services or batch processors that hit the API hard. The library is small (no heavy dependencies like transformers or CUDA), so installation and setup are fast.

You'll find the client most useful if you're building chatbots, document analysis tools, code generation features, RAG pipelines, or any application where you want full control over the model, the data, and the inference stack. It's also popular for internal tools, prototypes, and production services inside organizations that prefer self-hosted infrastructure.

Installation & Setup

You need Python 3.7 or later and an Ollama server running (either locally on your machine or remotely on another server). The setup is straightforward and takes less than five minutes.

Prerequisites

First, install Ollama from ollama.com. Ollama supports macOS, Windows (via WSL2 or Docker), and Linux.

On macOS, download the app and install it. On Linux, use the official installer script. Start the Ollama daemon, which launches a REST server on localhost:11434 by default.

# macOS: brew install ollama (or download from ollama.com)

# Linux: curl -fsSL https://ollama.ai/install.sh | sh

# Then start the server:
ollama serve

Verify Ollama is running by checking curl:

curl localhost:11434/api/tags

If you see a JSON response with an empty models list, you're good. Keep the ollama serve process running in the background (in a terminal, tmux session, or as a systemd service).

Install the Python Client

pip install ollama

That's the entire installation. The library is about 50KB and has no heavy dependencies. You're ready to write code.

Verify It Works with a Test Script

from ollama import Client

client = Client(host='localhost:11434')

# Pull a model (one-time download, ~5-10 minutes for Mistral)
client.pull('mistral')

# Test a simple chat
response = client.chat(model='mistral', messages=[
    {'role': 'user', 'content': 'Explain quantum computing in one sentence.'}
])

print(response['message']['content'])

Run this script. You should see a complete answer in 5-10 seconds depending on your GPU. If it works, your environment is set up correctly.

Core API Patterns: Chat vs. Generate

The Python client exposes two main endpoints: chat() and generate(). Both return text, but they're designed for different use cases and have different performance characteristics.

Chat: Multi-Turn Conversations with History

chat() maintains message history. You pass a list of messages with roles (user, assistant, system). Ollama preserves the conversation context and returns an assistant response that continues the dialogue.

from ollama import Client

client = Client()

messages = [
    {'role': 'system', 'content': 'You are a Python expert teaching beginners.'},
    {'role': 'user', 'content': 'How do I read a CSV file?'},
    {'role': 'assistant', 'content': 'You can use the pandas library...'},
    {'role': 'user', 'content': 'Show me a complete example with error handling.'},
]

response = client.chat(model='mistral', messages=messages)
print(response['message']['content'])

Each message is a dict with role and content. Roles are user, assistant, or system. The system role sets the context or behavior for the model. The model sees the full conversation and can reference earlier exchanges.

Use chat() when you're building conversational experiences: chatbots, Q&A systems, interactive tutors, or any application where context matters. The model has full conversation history and can maintain coherence across multiple turns.

Generate: One-Shot Text Generation

generate() is simpler and stateless. You pass a single prompt string; the model generates a response; there's no history or context. It's faster and uses less memory for single-shot tasks like classification or summarization.

from ollama import Client

client = Client()

prompt = "Write a haiku about Python programming."
response = client.generate(model='mistral', prompt=prompt)

print(response['response'])

The response is a dict with a single response key containing the full output. Unlike chat, there's no message list or role management.

Use generate() for one-off tasks: text classification, summarization, code snippets, data transformation, or batch processing where conversation context is irrelevant.

Key Differences

Feature	Chat	Generate
Input	List of messages with roles	Single prompt string
Context	Maintains conversation history	Stateless, no history
Best for	Dialogue, Q&A, interaction	One-shot classification, summarization
Response field	response['message']['content']	response['response']
Speed	Slightly slower (context overhead)	Slightly faster (no history processing)

Both block until the model finishes (unless you use streaming, covered next). Both return a dict with metadata like model name, tokens used, and generation time.

Streaming Responses & Async Clients

For real-time applications, waiting for the model to finish all at once is slow and poor UX. Streaming returns tokens as they generate. Async handles multiple concurrent requests without threading.

Streaming Chat Responses

With streaming enabled, you get tokens one at a time. This lets you display output to the user immediately instead of waiting 10 seconds for the full response.

from ollama import Client

client = Client()

messages = [
    {'role': 'user', 'content': 'Tell me a 100-word story about a robot.'},
]

stream = client.chat(model='mistral', messages=messages, stream=True)

for chunk in stream:
    # Each chunk is a dict with {'message': {'content': '<token>'},...}
    print(chunk['message']['content'], end='', flush=True)
print()  # newline at the end

Each chunk is a dict. Extract the token from chunk['message']['content'] and print or accumulate it. The flush=True ensures the terminal updates in real time instead of buffering.

Streaming is essential for web UIs. Users expect responses to appear as the model thinks, not all at once after a delay. It also helps with long outputs: you see the first few tokens in under 100ms.

Streaming Generate Responses

generate() also supports streaming:

from ollama import Client

client = Client()
prompt = "Write a 200-word essay on machine learning."

stream = client.generate(model='mistral', prompt=prompt, stream=True)

for chunk in stream:
    print(chunk['response'], end='', flush=True)
print()

With generate(), the streamed token is in chunk['response'] instead of chunk['message']['content'].

Async Client for High Concurrency

If you're handling 10+ simultaneous requests (e.g., a web service with multiple users), blocking I/O will starve your application. Use the async client.

import asyncio
from ollama import AsyncClient

async def query_model(prompt):
    client = AsyncClient(host='localhost:11434')
    response = await client.generate(model='mistral', prompt=prompt)
    return response['response']

async def main():
    # Run 10 queries in parallel without threads
    prompts = [f"What is {i} times {i}?" for i in range(1, 11)]
    results = await asyncio.gather(*[query_model(p) for p in prompts])
    for i, result in enumerate(results):
        print(f"Query {i+1}: {result[:50]}...")  # print first 50 chars

asyncio.run(main())

Async doesn't speed up individual requests; it lets you handle many requests concurrently without creating threads. This is critical for web frameworks like FastAPI or Starlette.

Reuse the same AsyncClient instance for multiple requests. Creating a new client per request adds overhead.

Connecting to Remote Ollama Servers

By default, the client connects to localhost:11434. If your Ollama server runs elsewhere, pass a custom host.

Connect to a Remote Server

from ollama import Client

# Ollama running on another machine
client = Client(host='192.168.1.50:11434')

response = client.chat(model='mistral', messages=[
    {'role': 'user', 'content': 'Hello!'}
])

print(response['message']['content'])

The host can be any IP or hostname where Ollama listens. Verify the port (usually 11434) is open and the machine is reachable from your client.

Connecting Through a Proxy or VPN

If your Ollama server is behind a firewall, configure your HTTP client to route through a proxy:

import os
from ollama import Client

# Set proxy environment variables
os.environ['HTTP_PROXY'] = 'proxy.company.com:8080'
os.environ['HTTPS_PROXY'] = 'proxy.company.com:8080'

client = Client(host='remote-ollama.internal:11434')
response = client.chat(model='mistral', messages=[
    {'role': 'user', 'content': 'Test message'}
])

The underlying HTTP library (requests) respects proxy environment variables. No changes to the Ollama client code are needed.

Error Handling for Remote Connections

Remote connections can timeout, fail, or become unreachable. Always wrap calls in try/except:

from ollama import Client, ResponseError
import socket

client = Client(host='remote-server:11434')

try:
    response = client.chat(
        model='mistral',
        messages=[{'role': 'user', 'content': 'Hello'}],
    )
except ResponseError as e:
    print(f"Ollama API error: {e}")
except socket.timeout:
    print("Request timed out. Ollama server may be slow or unreachable.")
except Exception as e:
    print(f"Connection failed: {e}")

For production, add request timeouts and retry logic:

from ollama import Client
import time

client = Client(host='remote-server:11434')

max_retries = 3
for attempt in range(max_retries):
    try:
        response = client.chat(
            model='mistral',
            messages=[{'role': 'user', 'content': 'Test'}]
        )
        break  # success
    except Exception as e:
        print(f"Attempt {attempt + 1} failed: {e}")
        if attempt < max_retries - 1:
            time.sleep(2 ** attempt)  # exponential backoff

This retries with exponential backoff: 1 second, 2 seconds, 4 seconds.

Function Calling & Tool Integration

Function calling (also called tool calling) lets you give the model access to functions it can request. Instead of just returning text, the model can ask you to run a function and then use the result. This is the foundation for AI agents and augmented applications.

Why Use Function Calling?

Models are text generators, not calculators or web browsers. If you want a model to answer "What's the weather in London?", the model can't look it up on its own. With function calling, the model can request a weather lookup function; your code provides the result; and the model incorporates it into its response.

Ollama's Python client supports function calling via the tools parameter (available in version 0.2+).

Define a Tool

from ollama import Client

client = Client()

# Define a tool the model can call
tools = [
    {
        'type': 'function',
        'function': {
            'name': 'get_weather',
            'description': 'Get the current weather for a city.',
            'parameters': {
                'type': 'object',
                'properties': {
                    'city': {
                        'type': 'string',
                        'description': 'The city name (e.g., London, Tokyo)',
                    }
                },
                'required': ['city'],
            },
        },
    }
]

# Ask the model a question that requires the tool
messages = [{'role': 'user', 'content': 'What is the weather in Paris right now?'}]

response = client.chat(model='mistral', messages=messages, tools=tools)

# Check if the model requested a tool call
if response['message'].get('tool_calls'):
    for tool_call in response['message']['tool_calls']:
        print(f"Model requested: {tool_call['function']['name']}")
        print(f"Arguments: {tool_call['function']['arguments']}")
else:
    print("Model did not request a tool.")

The model sees the tool description and decides whether to use it. If it does, it returns a tool_calls list with function names and arguments.

Implement the Tool Loop (Agent Pattern)

In real applications, you need to handle the tool call, run the function, and feed the result back to the model:

from ollama import Client
import json

client = Client()

# Define the tool
tools = [
    {
        'type': 'function',
        'function': {
            'name': 'get_weather',
            'description': 'Get the weather for a city.',
            'parameters': {
                'type': 'object',
                'properties': {
                    'city': {'type': 'string', 'description': 'City name'}
                },
                'required': ['city'],
            },
        },
    },
    {
        'type': 'function',
        'function': {
            'name': 'get_time',
            'description': 'Get the current time.',
            'parameters': {
                'type': 'object',
                'properties': {},
                'required': [],
            },
        },
    }
]

# Implement the actual functions
def get_weather(city):
    # Stub: in reality, call a weather API
    return f"Clear, 72F in {city}"

def get_time():
    from datetime import datetime
    return datetime.now().isoformat()

# Initial user message
messages = [{'role': 'user', 'content': 'What is the weather in Tokyo and what time is it?'}]

# Loop until the model stops requesting tools
while True:
    response = client.chat(model='mistral', messages=messages, tools=tools)
    
    # Add the assistant's response (which may include tool calls)
    messages.append(response['message'])
    
    # If no tool calls, we're done
    if not response['message'].get('tool_calls'):
        print("Final response:", response['message']['content'])
        break
    
    # Process each tool call
    for tool_call in response['message']['tool_calls']:
        func_name = tool_call['function']['name']
        func_args = tool_call['function']['arguments']
        
        # Call the appropriate function
        if func_name == 'get_weather':
            result = get_weather(func_args['city'])
        elif func_name == 'get_time':
            result = get_time()
        else:
            result = "Unknown function"
        
        # Add the tool result to messages
        messages.append({
            'role': 'tool',
            'content': result,
            'name': func_name,
        })

This is the tool loop pattern: model -> tool call -> result -> model -> final text. It's the foundation for AI agents that can interact with APIs, databases, or external systems.

Integrating with Open WebUI

Open WebUI is a web frontend for Ollama and compatible APIs. It lets non-technical users chat with models through a browser. The Python client and Open WebUI are complementary: Open WebUI provides the UI; the Python client powers custom backends.

How They Work Together

Open WebUI connects to an Ollama API server (the same one your Python client uses). When a user sends a message in the Open WebUI chat, the frontend calls the Ollama REST API. Your Python client makes the same calls programmatically.

Both can run against the same Ollama instance:

[Ollama Server on localhost:11434]
    ^
    |
    +-- [Your Python Client] (scripts, FastAPI backend)
    |
    +-- [Open WebUI Container] (web browser interface)

They share the same models, configuration, and inference engine. This is powerful: you get a user UI for free while retaining full programmatic control via Python.

Use Case: Python Backend + Open WebUI Frontend

Say you're building a document Q&A system. The architecture might be:

1. Your Python backend loads documents, chunks them, embeds them, and stores them in a vector database (Chroma, Weaviate, etc.).
2. When a user asks a question via Open WebUI, you intercept it (or build a custom endpoint) and retrieve relevant chunks.
3. You pass a grounded prompt to Ollama: system message with retrieved context, plus the user question.
4. Ollama generates an answer using the local model.
5. Open WebUI displays the response.

Example Python FastAPI backend:

from fastapi import FastAPI
from ollama import Client

app = FastAPI()
client = Client(host='localhost:11434')

# Simulated document store
documents = {
    'python': 'Python 3.13 includes improved error messages and performance optimizations.',
    'javascript': 'JavaScript ES2024 adds several new features including pattern matching.',
}

@app.post('/chat')
def chat(user_input: str):
    # Retrieve relevant context (in production, use similarity search)
    context = documents.get('python', 'No document found.')
    
    # Build a grounded prompt
    system_message = f"You are a helpful assistant. Answer using this context: {context}"
    
    # Call Ollama
    response = client.chat(model='mistral', messages=[
        {'role': 'system', 'content': system_message},
        {'role': 'user', 'content': user_input}
    ])
    
    return {'response': response['message']['content']}

Connect Open WebUI to your Ollama server and users can ask questions through the UI. Your backend handles retrieval, prompt engineering, and business logic.

Managing Open WebUI and Ollama in Production

For hobby projects, running Ollama and Open WebUI yourself is straightforward. For production workloads (multiple users, SLAs, compliance), self-hosting becomes operationally complex:

• Model updates: Pulling new models blocks inference.
• Security patches: Manual updates to Ollama and Open WebUI.
• Resource contention: Multiple users competing for GPU memory.
• Infrastructure costs: Provisioning and maintaining GPU servers.
• Scaling: Auto-scaling, load balancing, redundancy.

This is where managed services come in. Opsily's managed Ollama hosting offloads these concerns: you get automatic updates, monitoring, backups, and scaling. You focus on your application logic (retrieval, function calling, tool integration), not infrastructure.

Frequently Asked Questions

Can I use Ollama as an API?

Yes. Ollama exposes a REST API on localhost:11434 by default. The Python client wraps it for convenience, but any language (JavaScript, Go, curl) can call the HTTP endpoints directly. You can build web services, microservices, or serverless functions around the Ollama API.

What is the difference between Ollama's /api/generate and /api/chat?

generate() is for one-shot text generation with no message history. chat() is for multi-turn conversations where the model sees the full message history. Use generate() for classification, summarization, or batch processing. Use chat() for chatbots and interactive assistants.

Can I run an LLM locally using Ollama?

Yes. Ollama downloads open-source models (Llama 3.1, Mistral, Gemma, etc.) and runs them on your CPU or GPU. No internet required after the initial model download. Models are stored in ~/.ollama/models (macOS/Linux) or %USERPROFILE%.ollama\models (Windows).

How do I connect to Ollama's API from another machine?

Instantiate the client with the remote host: Client(host='remote-ip:11434'). Ensure the remote machine has Ollama running and the port is open. For security, use a firewall, VPN, or proxy to restrict access.

Is the Ollama API free to use?

Self-hosted Ollama is free (you pay for compute hardware). Ollama also offers a Cloud API at ollama.com, but pricing details are not publicly documented yet.

What is function calling and why should I use it?

Function calling lets you give the model access to functions it can request at runtime. When the model needs to look up information or perform a calculation, it asks your code to run a function, then uses the result. This is essential for building AI agents, RAG systems, and augmented applications.

How do I handle errors in the Python client?

Wrap client calls in try/except and catch ResponseError (API errors) or generic Exception (connection errors). Always set a timeout and implement retry logic for production services. Test against a remote Ollama instance to catch network issues early.

The Bottom Line

The Ollama Python client is a straightforward way to build LLM applications without cloud API dependencies. It's fast to set up, supports streaming and async patterns, and integrates cleanly with Open WebUI and other frontends. Most teams use it for internal tools, prototypes, and production services where they want full control over models and inference.

For teams scaling beyond hobby projects, self-hosting Ollama gets operationally complex. Infrastructure, updates, monitoring, and GPU resource management require significant DevOps effort. If you're ready to focus on application logic instead of infrastructure, explore Opsily's managed Ollama hosting to handle deployment, scaling, and operations for you.