· UPDATED · ARJUN · WRITING
When an Agent Forgets, Look Beyond the Model
When an Agent forgets yesterday's instruction, the model may not be at fault. The missing piece is often how the system stores and retrieves memory.
You tell a coding Agent, “Use the Go standard library wherever possible in this project. Do not add a framework without a reason.” It follows the instruction for the rest of the conversation. The next day, you open a new thread and it recommends another framework.
The obvious conclusion is that the model has a poor memory.
But the model may never have had a chance to “remember.” When the previous conversation ended, the preference was not written to persistent storage. When the new thread began, the system did not retrieve it. The failure was not that the model forgot. The application never completed the read/write loop that makes memory possible.
This is where explanations of Agent Memory often become muddled. Context windows, chat logs, session state, vector databases, and model weights are all called “memory,” as if connecting a database automatically gave an Agent a durable past.
Separate those concepts first. The rest of the system becomes much easier to reason about.
Three Concepts That Are Often Blurred Together
A practical design starts by separating three scopes:
| Layer | What it is | What it solves | What it does not guarantee |
|---|---|---|---|
| Context Window | The input actually provided to one model call | Gives the model the material needed for the immediate task | Does not automatically persist information across calls |
| Thread State | Persisted state scoped to one conversation or task thread | Stores messages, tool results, task progress, and intermediate artifacts | Does not automatically share information across threads |
| Long-Term Memory | Data stored across conversations and retrieved within an explicit scope | Preserves user preferences, facts, experiences, or rules | Writing something does not guarantee that it will later be retrieved correctly |
The CoALA paper makes an important distinction: language models are stateless across calls, while language Agents can maintain Working Memory and Long-Term Memory in external data structures. It also separates the model’s Context from the Agent’s Working Memory. Working Memory can persist across model calls; each call receives only a subset of it.
In other words, Context is the material handed to the model now. State is the situation the Runtime currently knows about. Memory is a record that may become useful again later.
They are connected, but they are not interchangeable.
A Context Window Is Not a Self-Updating Archive
The Context Window is an input budget. The system prompt, current request, message history, tool definitions, tool results, and retrieved documents all compete for the same space.
A longer window can hold more text, but it does not guarantee that every part of that text will be used equally well. In the multi-document question-answering and key-value retrieval tasks studied in Lost in the Middle, several models performed worse when relevant information appeared near the middle of a long context. The result came from specific models and tasks, so it should not be stretched into “every long-context model ignores the middle.” It does show that fitting information into the window and using it reliably are different problems.
Continually appending the entire conversation to the prompt is therefore a weak memory strategy. It creates three problems at once:
- Irrelevant history consumes space needed by the current task.
- Latency and model cost increase as the input grows.
- Important constraints become buried in old messages and may not be used consistently.
A better approach is to persist the complete State, then let a Context Builder select the information needed for the current step. The model receives a working set, not the whole archive.
Short-Term Memory Is Better Understood as Thread State
Current LangChain and LangGraph documentation defines Short-Term Memory as thread-scoped State. It belongs to one conversation thread, can be persisted through a Checkpointer, and can be restored between graph invocations or execution steps.
Message history is usually part of that State, but it need not be the whole thing. A code-editing Agent might also store:
{
"goal": "Fix duplicate charges on payment retries",
"constraints": ["Do not change the public API", "The change must be reversible"],
"current_step": "Reproduce and locate the broken idempotency path",
"evidence": ["trace-8f21", "test_payment_retry"],
"artifacts": ["diagnosis.md", "patch.diff"]
}
These fields are easier to persist, inspect, and restore than a long paragraph of natural language. After a failure or restart, the Runtime knows where the task stopped. It does not have to ask the model to reread the entire conversation and guess.
When message history grows too long, applications can trim, delete, or summarize earlier messages. Summarization saves tokens, but it is lossy compression. A constraint omitted by the summarizer may be gone from every later model call. Goals, permission boundaries, acceptance criteria, and tool results are safer as structured State than as details entrusted only to an automatic summary.
Long-Term Memory Is Not Another Name for a Vector Database
A vector database solves one kind of retrieval problem: it can find semantically similar text even when the query uses different wording. It does not decide what is worth remembering, whether an old fact is still valid, or which record wins when two memories conflict.
Long-Term Memory can be divided by the kind of information it contains:
- Semantic Memory stores facts about the user, project, or world, such as “this repository uses Go 1.24” or “the user prefers concise answers.”
- Episodic Memory stores what happened before, such as the cause of a failed deployment, the actions taken, and the final result.
- Procedural Memory stores how a class of task should be performed, including system prompts, tool rules, code paths, and reusable Skills.
This taxonomy comes from cognitive architectures. It is not a prescription for three databases. The data structure should follow the query:
- User preferences and project configuration need exact reads and updates. JSON documents or a relational database are often more direct.
- Conversation fragments, issue records, and experience summaries may benefit from full-text, keyword, vector, or hybrid retrieval.
- A graph can help when relationships among entities are the real query, but there is no reason to deploy a knowledge graph merely because the feature is called “memory.”
- Stable rules and procedures belong in versioned prompts, code, or Skills before they belong in a pile of semantically similar text.
LangGraph’s Long-Term Store illustrates another valid design. It can organize memories as JSON documents by Namespace and Key, then add filtering or semantic search where needed. Data ownership and update behavior come first. The retrieval index comes later.
The First Hard Problem: Deciding What to Write
Not every sentence deserves to become Long-Term Memory.
“Use Python for this script” describes the current task. “Use Python by default in future examples” sounds like a preference that should cross conversation boundaries. Persisting the first sentence as a permanent preference may cause the system to recommend Python in the middle of a Go repository next week.
Every candidate write should answer at least five questions:
- Scope: Does it belong to a user, project, organization, or Agent?
- Type: Is it a fact, an experience, or a procedure?
- Source: Did it come from an explicit user statement, a tool result, or a model inference?
- Lifetime: Is it durable, project-scoped, or temporary?
- Identity: Does it create a new record, or replace an existing one?
Memory can be written in the hot path of a request or consolidated later in the background. Hot-path writes make new information immediately available, but add latency and more work to the current request. Background writes keep the response path lean, but introduce scheduling and synchronization questions. Both are real engineering choices. Neither is the universal answer.
I would also keep explicit statements separate from system inferences. “Keep future answers concise” can become a candidate preference. Inferring “this user dislikes explanations” from two short conversations should not silently become a fact. Once an uncertain interpretation is stored with false confidence, every future retrieval can amplify the mistake.
Retrieval Is More Than Dumping the Top K Into a Prompt
An Agent can still “remember” incorrectly after the storage layer works. Sometimes the right record exists, but the retriever surfaces something else. Sometimes a useful record is buried under unrelated results.
A more complete read path usually includes:
- Derive a retrieval intent from the current task instead of copying only the user’s last sentence.
- Apply hard filters for user, project, memory type, and validity status.
- Choose exact lookup, keyword search, vector search, or a hybrid based on the data.
- Rerank and deduplicate candidates, then detect conflicts.
- Place only the information needed for the current step into Context, with provenance attached.
The Generative Agents research prototype combined Recency, Importance, and Relevance in its retrieval score. That is an influential design example, not a universal formula. Customer preferences, code incident records, and the experiences of a simulated character require very different definitions of “important.” The weights have to be evaluated against the actual task.
There is another distinction worth keeping: knowledge-base RAG and user memory may use similar retrieval technology, but they are not the same data. Product documentation answers “what does this system support?” User memory answers “what happened to this user or in this task?” Their permissions, update cadence, and authoritative sources are different.
Memory Must Be Correctable, and It Must Be Able to Forget
Suppose the system has stored this fact:
Deployment method: Docker Compose.
Two months later, the project moves to Kubernetes. If the application simply appends another memory, retrieval may return two incompatible answers.
Deleting all history is not the only option. The old record can be marked Superseded with the period and source for which it was valid, while the new record becomes Current. The Agent then uses Kubernetes when asked about the present deployment, but can still see that Docker Compose existed when investigating an older incident.
Forgetting is also more than lowering a vector-similarity score. Temporary State can expire through a TTL. Stale facts can be invalidated, duplicate episodes can be consolidated, and data the user asks to remove must actually be deleted from persistent storage. Excluding a record from the next prompt does not mean the system has deleted it.
A Minimal Architecture to Start With
You do not need to deploy Redis, a vector database, a graph database, and a memory framework on day one. For many personal Agents, five clear responsibilities are enough:
- A Checkpointer persists messages and task State for each Thread.
- A Profile Store holds a small set of explicit, replaceable user or project facts.
- An Episode Store records past events with sources and outcomes; add full-text or vector indexing when retrieval needs it.
- A Memory Policy decides when to write, update, expire, or delete.
- A Context Builder selects State and Memory for the current step and stays within the input budget.
The pseudocode is deliberately short:
def build_context(user_id, thread_id, request):
state = checkpoints.load(thread_id)
profile = profiles.get(user_id)
episodes = memories.search(
query=request,
filters={"user_id": user_id, "status": "current"},
)
return context_builder.pack(state, profile, episodes)
The database persists. The retriever finds candidates. The Policy makes lifecycle decisions. The Context Builder decides what the model sees. Once those responsibilities are separate, a bad answer becomes much easier to trace to the failing layer.
When Long-Term Memory Is the Wrong Feature
Long-Term Memory is not what makes a system an Agent.
If a tool handles isolated requests and nothing should carry over after completion, there is no reason to retain user data merely to appear more agentic. A deterministic business process may need a reliable database state machine, not model memory. Public-document question answering may need a versioned knowledge index, not a user profile.
Ask one question first: what exactly will the next task lose if this information is unavailable? If the answer is vague, do not store it yet.
References
- Sumers et al., Cognitive Architectures for Language Agents: distinguishes Working, Episodic, Semantic, and Procedural Memory, and explains how Agent Working Memory can persist across LLM calls.
- LangChain, Memory overview: current engineering definitions for Short-Term and Long-Term Memory, write timing, Namespaces, and Stores.
- LangChain, Short-term memory: current Checkpointer, trimming, deletion, and summarization patterns.
- Liu et al., Lost in the Middle: studies how the position of relevant information affects performance for selected models and long-context tasks.
- Park et al., Generative Agents: presents the Memory Stream, Reflection, and a retrieval design using Recency, Importance, and Relevance.
- Packer et al., MemGPT: explores hierarchical storage and data movement for managing limited Context.
- Chhikara et al., Mem0: a system example for memory extraction, consolidation, retrieval, and evaluation.
- Source article: the initial knowledge outline adapted for this article.
When an Agent appears to “remember,” a concrete data flow sits underneath: what was saved, where it was stored, when it was retrieved, which record wins a conflict, and when the data is truly deleted.
The model processes the input in front of it. Making the past useful in the next call is the system’s job.