Every self-hosted agent project ships with a page on its site claiming "persistent memory." Almost none of them define what they mean by it, and almost none of them agree with each other.
That's the actual state of the art: memory is the least honestly discussed part of the agent stack. It's also the feature that separates an agent that feels intelligent from one that feels like a fresh install every morning. Getting it right is not a features question — it's a design-choice question, and the choices are not compatible.
What agent memory actually is
Break "memory" into three things that get lumped together and you can reason about it.
The first is facts: things you told the agent that it needs to know indefinitely. Your name, your timezone, the fact that "the office" means the address on Elm Street and not the one in Berlin. Facts should survive restarts, model swaps, and version bumps.
The second is skills: reusable procedures the agent has figured out and wants to run again. "When Omer asks for a weekly summary, pull from these five sources in this order and format it like this." Skills are learned, or authored, and they're what makes an agent get better over time without retraining.
The third is session context: the tail of the current conversation. This is the memory people usually mean when they say "context window" — the working scratchpad that gets stuffed into the model on every turn. Session context is by far the largest of the three and by far the least valuable to persist. Most of it is noise.
An agent claiming "memory" without saying which of these three it means is not being helpful. A good design treats them differently because they have different lifespans, different sizes, and different failure modes.
What the pattern actually does
- Writes facts to durable storage — a database, a set of files, an embedded vector store — with an explicit shape. Not "we saved the whole conversation," but "we extracted these three key-value pairs."
- Distills skills from repeated interactions — sometimes automatically, sometimes with an explicit
/learncommand like the one Hermes v0.18.0 added. The distilled skill is a durable artifact you can inspect. - Compresses session context — either by summarization ("here's what happened in the last 20 turns") or by eviction (drop the oldest, keep the most recent), or by retrieval (keep everything, fetch only what's relevant on each turn).
- Surfaces what it remembers on request. A memory system you can't audit is a memory system that will silently lie to you inside three months.
- Survives restarts, redeployments, and model changes. State that lives only in the process is not memory; it's cache.
Why it beats a stateless agent
A stateless agent is a search engine with a personality. Each conversation is a cold start. That's fine for one-off queries and unusable for anything the agent is supposed to do more than once.
The obvious win of a memory system is the "don't make me tell you again" one. The subtler and more important win is that memory turns an agent from a stimulus-response tool into a longitudinal one. It can notice patterns it wouldn't otherwise see: that you always ask for the same three metrics on Sunday, that a specific contact only ever emails you about one topic, that a project has been on your calendar for six weeks without moving.
Longitudinal agents have new failure modes stateless ones don't. Memory can rot. An agent can convince itself of something wrong and cite the wrong belief back at you for weeks. Old context can pollute new decisions. Persistent memory means persistent bugs, which is why the audit surface matters more than the storage backend.
Where this is being built
The most explicit implementation is Letta, formerly MemGPT, which grew out of the MemGPT paper proposing a virtual-context-management OS pattern where the agent itself manages what's in and out of the model's context window. The Letta approach is heavier than most — it treats memory as a first-class subsystem with its own APIs — and it's the most rigorous open-source take on the problem.
Mem0 is a lighter alternative: an open-source memory layer that plugs into any LLM app, with automatic fact extraction and a vector-store retrieval layer. LangMem, from the LangGraph team, is a similar shape aimed at LangChain-native pipelines. OpenAI's ChatGPT memory feature is the hosted-and-opaque version of the same idea — you can see what it remembered, but not how or when it decided to.
Hermes takes a different route. Its memory lives in a memory/ directory of markdown files: a top-level MEMORY.md that indexes everything, plus per-topic files the agent maintains itself. It's plainer than Letta and less magical than Mem0, but it's also the only design in that list where you can cat a file, delete a line, and know exactly what you changed. That property matters more the longer you run the agent.
Claude Code's persistent context is a fourth shape worth naming: a project-scoped CLAUDE.md that the model reads on every session. It's not conversational memory in the Letta sense, but it's a durable, human-editable, version-controlled surface for facts that a coding agent needs, and it's proof that "markdown as memory" is a design pattern, not a Hermes idiosyncrasy.
How to evaluate a solution
Six questions to bring into any evaluation.
What's the eviction policy? When memory fills up, what gets dropped? "Least recently used" is a reasonable default. "Everything older than 30 days" is a reasonable default. "Trust me" is not a policy.
Can you audit the memory surface? For a given fact the agent believes, can you find the exact file, row, or vector entry that stored it? If the answer requires re-running the agent to find out, you don't have an audit surface.
How does corruption recover? A memory system will get poisoned — by a bad extraction, an outdated fact the agent never revised, a hallucination that got saved. Is there a forget operation? A dry-run mode? A way to roll back a bad week of memories without losing the good ones?
Does memory survive model swaps? If you move from GPT-4o to Claude to a local Qwen, does the memory come along, or is it encoded in a way that assumes the previous model? File-based markdown memory is the most portable answer here; embedding-based memory tied to a specific vector space is the least.
What's the cross-session query performance? After a year of use, when the agent needs to look up "the address I gave for the accountant," how long does it take? Some systems degrade sharply as memory grows.
Is persisted context encrypted at rest? Memory means the agent knows things about you. If the disk is stolen, or the backup leaks, what's exposed?
Pick the memory design that answers these cleanly, not the one with the best landing page. Every project claims persistent memory. Very few will still be running the memory system they shipped a year from now.