Hermes Agent Memory: Architecture and Self-Hosted Control

Hermes Agent stores conversation history and learned facts in a three-layer memory system: curated facts in MEMORY.md, full-text searchable sessions in SQLite with FTS5 indexing, and optional external providers for advanced retrieval. The key difference: your data lives on your own persistent volume, not extracted to a cloud service.

What Is Hermes Agent Memory? The Three-Layer Model

Unlike simple chat history, Hermes Agent structures memory as three independent layers that work together. Each layer serves a different retrieval pattern: curated facts for broad recall, session indexing for specific conversation recovery, and optional external providers for advanced embedding-based search.

Hermes Agent does not just store every message chronologically. It organizes memory into three distinct layers, each optimized for different access patterns.

The first layer is MEMORY.md and USER.md: curated facts and learned behaviors about the user and domain. These are manually or semi-automatically populated structured knowledge, think "the user prefers short responses" or "integrate with Slack on Fridays." This layer is sparse, high-value, and fast to search. No search overhead; just direct lookup.

The second layer is the FTS5-indexed session store. Every message, every tool call, every decision the agent made goes into SQLite with full-text search indexing enabled. When the agent needs context from a past conversation, it runs a phrase or keyword query against FTS5 and retrieves the most relevant chunks. This is the backbone of session memory: fast, deterministic, and local.

The third layer is optional external providers. Hermes can forward its memory to Hindsight, Mem0, Holographic, or other external systems that apply embedding-based retrieval or other advanced techniques. This layer adds cost and cloud extraction but can improve recall accuracy.

Why three layers? Each has different tradeoffs. Curated facts are fast but require maintenance. FTS5 sessions are local and free but keyword-based, not semantic. External providers are accurate but expensive and move data off your server.

How SQLite Full-Text Search Powers Hermes Memory

Hermes uses SQLite's FTS5 extension to index every message in a conversation, enabling fast phrase and keyword search without extracting data or using language models. FTS5 is a built-in, deterministic full-text engine that ranks results by relevance.

FTS5 is an extension to SQLite that adds full-text search capabilities. Instead of storing raw text, FTS5 creates an inverted index: a map from every word (or phrase) to the messages that contain it. When you search for "deploy to production", FTS5 returns messages in order of relevance, fastest first.

Why FTS5 and not vector embeddings? Vector databases like pgvector (used by ElizaOS) require a language model to turn text into numbers, then measure distance in embedding space. FTS5 does keyword matching directly on text. FTS5 is faster (no embedding step), deterministic (same query always returns same order), and runs entirely in SQLite with zero external dependencies.

For conversation history, especially long sessions, FTS5 is ideal. The agent searches for specific events or user requests by keywords or phrases. "When did the user ask about billing?" FTS5 finds it in milliseconds.

FTS5 is not perfect for semantic search. It won't find synonyms or related concepts without you hand-coding them. But it is perfect for self-hosted memory where low latency and data residency matter more than semantic perfection.

The Self-Hosting Advantage: Data Stays on Your Persistent Volume

Your conversation history and learned facts never leave the server where Hermes runs. Unlike Mem0 or Hindsight, which extract memory to their cloud platforms, Hermes keeps all data in local SQLite files on your persistent volume.

In many agent frameworks, memory is a cloud service. You send every interaction to Mem0's API, Hindsight's managed database, or Holographic's servers. Your data gets extracted, stored, and indexed in someone else's infrastructure.

Hermes takes a different approach: memory lives in SQLite on your persistent volume. If you run Hermes in a container, that volume survives restarts. If you run it on a VPS, that volume is your disk. Your conversation data never leaves your server.

This has three immediate benefits. First, privacy. Your conversation history is not indexed by external vendors. No machine learning on your data, no analytics pipeline, no shared infrastructure. Your data is yours.

Second, compliance. If you operate in regulated industries (healthcare, finance, government), keeping memory local can simplify audits and data residency requirements. GDPR, HIPAA, or FedRAMP all benefit from on-server storage.

Third, cost control. You pay once for storage and compute, not per-API-call. Hindsight and Mem0 charge for queries and uptime. Hermes' SQLite cost is embedded in your infrastructure bill.

For production deployments, explore Opsily's Hermes Agent hosting to see managed persistence options.

Built-In Memory vs. External Providers: The Tradeoff

Built-in SQLite memory is free and local. External providers add cost and cloud extraction but improve retrieval accuracy. Hindsight achieves 94.6% accuracy on LongMemEval benchmarks; Mem0 reaches 67.6% with easier setup.

The question is not "use external providers or not?" It is "when is the added accuracy worth the cost and cloud extraction?"

Built-in FTS5 memory works for most production use cases. It is free, local, and fast. You get keyword recall without API calls or monthly bills. The downside: it does not handle semantic search well. If the user asked "How do I deploy my app?" three months ago, and today they ask "What was the deployment strategy?", FTS5 might miss the connection because the keywords differ.

Hindsight is a managed memory platform built specifically for Hermes. On LongMemEval benchmarks, Hindsight achieves 94.6% accuracy for long-context memory retrieval. It combines multiple retrieval strategies (hybrid search) and caches efficiently. The cost: you run Hindsight's database (PostgreSQL plus embedding models), pay for hosting, and your memory data lives outside your Hermes container. Not ideal for air-gapped deployments.

Mem0 is a freemium memory-as-a-service platform. It achieves 67.6% accuracy on LongMemEval-S (GPT-4o), lower than Hindsight but still useful for semantic search. Mem0 is the easiest to set up (one API key, plug and play) but it extracts all your memory to their cloud servers. Billing is usage-based.

Holographic is a newer option that runs local zero-dependency memory. It stays on your server and adds minimal compute overhead. Documentation is sparse, but community reports good accuracy for medium-length sessions.

Decision framework: Use built-in FTS5 if you need keyword recall, privacy is critical, or you run air-gapped. Use Hindsight if accuracy is critical and you can run PostgreSQL alongside Hermes. Use Mem0 if you want fastest setup and don't mind cloud extraction. Use Holographic if you want zero dependencies and local-only operation.

Hermes Memory vs. ElizaOS pgvector vs. OpenClaw MEMORY.md

Hermes uses FTS5 for keyword/phrase recall on local transcripts. ElizaOS uses pgvector for vector similarity in PostgreSQL. OpenClaw's MEMORY.md is a flat file approach. Hermes is fastest for long sessions; pgvector is better for semantic similarity; MEMORY.md is simplest but does not scale.

Hermes is not the only agent framework with memory. Understanding how it compares helps clarify design tradeoffs.

ElizaOS uses pgvector, an extension to PostgreSQL that stores embeddings as vectors. Every message is embedded into a high-dimensional vector space (e.g., 1536 dimensions for OpenAI's text-embedding-3-small). When the agent searches memory, it embeds the query and finds vectors closest in space. This is powerful for semantic search: "user frustration" and "customer unhappy" map to similar vectors even if keywords differ. The cost is latency and compute: every search requires embedding the query.

OpenClaw uses MEMORY.md, a flat markdown file containing learned facts about the user and domain. It is simple, zero-dependency, and human-readable. But it does not scale. Once you have hundreds of conversations and thousands of learned facts, searching a markdown file becomes slow and noisy. There is no ranking, no indexing, just text search.

Hermes uses FTS5, which is a middle ground. It indexes text efficiently and supports phrase queries and ranking, but it does not embed. It is faster than pgvector for keyword queries and scales better than MEMORY.md flat files. The downside: it misses semantic relationships unless you encode them explicitly.

For a detailed technical comparison between Hermes and OpenClaw, see our Hermes vs OpenClaw guide.

Configuring Hermes Memory: Setup, Limits, and Troubleshooting

Enable memory with a single configuration parameter in your hermes.yaml. Set memory size limits in.env file. Monitor SQLite database growth and flush old sessions before they consume your persistent volume.

Hermes memory is enabled by default in most configurations. To verify it is active, check your hermes.yaml:

memory:
  enabled: true
  provider: "sqlite"
  path: "/data/hermes/memory.db"

Hermes will create the SQLite database and FTS5 indexes automatically on first run.

If you need to tune memory behavior, edit these environment variables:

HERMES_MEMORY_SIZE_LIMIT=10gb
HERMES_MEMORY_SESSION_TTL=2592000
HERMES_MEMORY_FLUSH_INTERVAL=86400

Common issues and fixes:

Memory full error: SQLite has hit HERMES_MEMORY_SIZE_LIMIT. Flush old sessions or increase the limit. Check your persistent volume has space: df -h /data/hermes/.

Slow recall queries: FTS5 indexes are corrupted or your session count is very high (100K+ turns). Rebuild indexes: sqlite3 /data/hermes/memory.db "PRAGMA optimize;" Then restart Hermes.

Disk fills overnight: Session TTL is too high. Reduce HERMES_MEMORY_SESSION_TTL or increase persistent volume allocation.

Hermes forgets past conversations: Check if memory is actually enabled. Run sqlite3 /data/hermes/memory.db ".tables" and confirm fts5_sessions and fts5_facts tables exist.

For production deployments, check Opsily's managed Hermes hosting for persistent volume allocation and monitoring.

Long Sessions and Memory Optimization

As conversation history grows, FTS5 search speed stays constant because indexes are pre-computed. Allocate at least 10 GB persistent volume per active agent; flush sessions older than 90 days to keep search fast.

Hermes can run for weeks or months, accumulating thousands of messages. How does memory performance scale?

FTS5 uses inverted indexes, which are pre-computed. When you add a message to SQLite, FTS5 updates its indexes in the background. Query time does not depend on total message count, only on index size. This is the magic of full-text search: O(log N) instead of O(N).

In practice, a 1-million-message FTS5 index queries in 20-50ms. A 10-million-message index queries in 30-80ms. The difference is small because index lookups are highly optimized.

Storage is linear: roughly 2-3 bytes per message in the FTS5 index (plus 200-500 bytes per message for raw text). A long-running agent with 100,000 messages uses about 100-150 MB SQLite database.

For scale planning:

• Single agent, typical usage (10-100 turns/day for 6 months): 1-2 GB database
• Team of 5 agents, heavy usage (500+ turns/day per agent): 10-20 GB
• Production multi-agent system: 50+ GB persistent volume recommended

Session flushing is key. Old sessions (>90 days) accumulate disk space. Set HERMES_MEMORY_FLUSH_INTERVAL to 86400 seconds (1 day) and sessions older than TTL will be compressed. This keeps search indexes small and disk pressure low.

Monitor SQLite growth:

sqlite3 /data/hermes/memory.db "SELECT page_count * page_size / 1024 / 1024 as size_mb FROM pragma_page_count(), pragma_page_size();"

If it grows linearly month-over-month, increase session TTL or add persistent volume.

Frequently Asked Questions

How much memory does a Hermes Agent need?

A baseline agent needs 1-2 GB for initial indexing. Add 1 GB per 100,000 conversation turns. Long-running agents in production should allocate 10+ GB persistent volume and monitor SQLite growth monthly.

Does Hermes Agent have memory?

Yes. Hermes Agent includes built-in memory via SQLite FTS5 indexing. No setup required; memory is enabled by default in most configurations. You can extend it with external providers if needed.

How do I set up hermes agent memory?

Enable the memory module in your hermes.yaml config file and point it to a persistent volume path. Hermes handles SQLite initialization automatically. Memory is searchable immediately on first run.

What is the best memory system for Hermes agents?

For most production use cases, Hermes' built-in FTS5 memory is sufficient. If you need semantic search (not just keyword match) or multi-agent memory sharing, integrate Hindsight (94.6% LongMemEval accuracy) or Mem0 (67.6% accuracy, freemium).

Why is my Hermes Agent memory full?

SQLite has exceeded allocated storage or your persistent volume is at capacity. Flush old sessions (older than 90 days) to reclaim space with PRAGMA optimize or allocate a larger persistent volume.

What is the best memory provider for Hermes Agent?

Hindsight ranks highest on benchmarks (94.6% LongMemEval accuracy) but requires PostgreSQL and more resources. Mem0 is easiest to set up (freemium) but extracts data to cloud. For air-gapped or high-privacy deployments, use Hermes' built-in memory with no external provider.

The Bottom Line

Hermes Agent's three-layer memory keeps your data local while giving you control over accuracy tradeoffs. Built-in FTS5 memory handles most production workloads; external providers are worth the complexity only if you need semantic search or multi-agent memory sharing. Deploy Hermes with Opsily's managed hosting to simplify persistence and scaling.