Agent Memory: Short-Term, Long-Term and What's Just a Database

A model has no memory. It has a context window, which is re-sent in full on every request and forgotten completely between them. Everything marketed as agent memory is a decision about what to put in that window and where the rest is kept.

The metaphor is doing damage

Borrowing “short-term” and “long-term” from cognitive psychology implies a consolidation process that does not exist. Nothing moves from one store to the other on its own. There is no forgetting curve, no rehearsal, no sleep. There is a string you assemble before each API call.

Ask three engineering questions instead, and the categories fall out on their own: when is it written, when is it read back into context, and what makes it go away. Anything called memory that cannot answer all three is a vector database with ambitions.

Three patterns

The third is chronically underrated. An agent that maintains a plan.md in its workspace and re-reads it after every compaction has durable, inspectable, editable memory with no embedding model, no retrieval tuning and no staleness problem — because the file is the state, not a description of it. A human can open it mid-run and see exactly what the agent believes it is doing. No vector store gives you that, and for long-horizon tasks it is usually the difference between a run you can supervise and one you can only watch.

Compaction: what to drop first

Every long run eventually hits the window. What you drop determines what the agent forgets, so it deserves a policy rather than a truncation. In rough order of what to sacrifice:

• Old tool outputs, oldest first. These are almost always the bulk of the transcript — a directory listing, a 6,000-token file, a page of search results — and almost always the least useful twenty steps later. Replace each with a one-line stub: [read_file src/retry.py — 240 lines, elided at step 31]. The stub matters: it preserves the fact that the action happened, so the agent does not repeat it.
• Superseded results. If read_file(x) appears three times, only the last is current. Drop the earlier ones outright.
• Middle reasoning. Summarise steps 5–25 into a paragraph of findings and decisions. This is the lossy one and it is where agents lose the thread, so summarise into a structured form — decisions taken, files touched, hypotheses eliminated — rather than free prose.
• Never the system prompt, the original task, or the last two steps. Dropping the task statement is how an agent finishes confidently doing something adjacent to what it was asked.

One caching note, because it is counter-intuitive: compaction rewrites the prefix, which invalidates the prompt cache from that point on. A compaction is a paid event, not a saving. Compact in large infrequent steps rather than trimming one message at a time, or you will pay full rate for every remaining request in the run.

The fact store, concretely

If you do build pattern 2, build it as a table with the columns that make it maintainable. The retrieval is the easy part; provenance and supersession are what stop it rotting:

CREATE TABLE agent_facts (
  id           TEXT PRIMARY KEY,
  subject      TEXT NOT NULL,        -- 'user:8812'  scope the retrieval
  fact         TEXT NOT NULL,        -- one sentence, self-contained
  embedding    BLOB,
  source_run   TEXT NOT NULL,        -- which run wrote it
  source_step  INTEGER NOT NULL,     -- and which step -- for the trace
  confidence   REAL NOT NULL,        -- from the extractor, not the agent
  created_at   TIMESTAMP NOT NULL,
  expires_at   TIMESTAMP,            -- NULL only for genuinely durable facts
  superseded_by TEXT REFERENCES agent_facts(id)
);
-- retrieval always filters: subject = ? AND superseded_by IS NULL
--                       AND (expires_at IS NULL OR expires_at > now())

• Facts must be self-contained sentences. “Prefers dark mode” is useless once retrieved without its subject. “User 8812 prefers the dark theme in the web app” survives being pasted into a context with no surroundings — which is exactly what retrieval does to it.
• expires_at should be mandatory in practice. A preference expires in months; “is currently debugging the checkout flow” expires in days. A store with no expiry converges on contradicting itself.
• Write on supersession, not on deletion. Keeping the old row with a pointer is what lets a trace explain why the agent believed something last Tuesday.
• Cap what retrieval injects. Five facts, not fifty. The store is unbounded; the context is not, and every fact you inject is one you pay for on every subsequent step of the run.

Memory poisoning

Here is the failure mode that separates a fact store from a cache, and the reason to think hard before building one. A cache being wrong costs you one stale read. A fact store being wrong is permanent and self-reinforcing: a false fact written at step 20 of one run is retrieved into every future run, is treated as established context, and shapes the extraction that writes the next batch of facts.

Three defences, all cheap relative to the cost of discovering the problem in production. Never let the agent write facts about content it did not verify — extract from tool results and user statements, not from its own reasoning. Keep source_run and source_step so any bad fact can be traced to the run that produced it and its cohort deleted together. And make the store inspectable by a human as a plain list, because the only reliable way anyone has found to catch a poisoned memory is to read the memories.

Which returns to the recommendation at the top. Before building pattern 2, check whether pattern 3 covers it. A file the agent reads at the start of each run has no retrieval failure mode, no staleness model, no poisoning story, and can be fixed by editing it.