Memory Architectures for Long-Running Assistants

Nothing in a language model remembers anything. Every “memory feature” is a store outside the model plus a policy for putting some of it back into the window, and the four common policies have very different per-turn costs.

There is no memory

Worth restating precisely because the product language obscures it: a request contains everything the model will know. There is no continuity between calls, no hidden session on the provider’s side, nothing carried over. An assistant that remembers your name from last week remembers it because something in your application read your name out of a database and put it into this request’s input, where it is billed as input tokens like everything else.

Which means memory is not a feature you add; it is a read policy you choose. Four are common. They differ in what is stored, what is selected, and — the number that decides most real arguments — what each costs per turn.

Four architectures

1. Full transcript replay

Store every message; send all of them every turn. Perfect recall within the window, zero selection logic, and the cost grows with the square of the conversation length because turn n re-sends everything from turns 1 to n−1. It is the right answer for short sessions and the wrong one for anything that runs for an hour. It also ends abruptly rather than gracefully: it works flawlessly until the window fills, then breaks.

2. Rolling compaction

Keep a verbatim tail plus a compacted record of everything older, as in the compaction routine. Per-turn cost is bounded by construction — tail plus record, both capped — so cost becomes linear in turns rather than quadratic. What you give up is exact recall of anything the record did not capture. The schema is therefore the design, not an implementation detail.

3. Extracted fact store

Run an extraction pass that writes durable facts to a keyed store — preferences, identifiers, constraints, relationships — and inject the relevant ones on each turn. Facts survive across sessions, not just within one, which is the thing users actually mean by “memory.” The cost is small and roughly constant. The risks are staleness (a fact recorded in March that stopped being true in June) and silent contradiction, so every fact needs a timestamp and a supersession rule.

4. Retrieval over past turns

Index the whole transcript history and retrieve the few most relevant passages for the current turn. Recall is unbounded in principle — a conversation from six months ago is reachable — and the per-turn cost is whatever you allocate to retrieved passages. The failure mode is specific and nasty: retrieval returns a passage that was true at the time and is stale now, presented with the same authority as current state. The retrieval machinery itself belongs to the RAG cluster; what is this cluster’s problem is that the retrieved passage lands in the same window as the current state, and something has to decide which wins.

The per-turn arithmetic

Take a session of N turns. Assume — all of these are assumptions, replace them with your own — an average turn of 400 tokens (user plus assistant), a system block of 1,800, a verbatim tail of 3 turns, a compacted record of 900 tokens, a fact store injection of 250 tokens, and a retrieval budget of 1,500 tokens. Input tokens billed on turn n:

replay      : 1800 + 400*(n-1)          -> grows without bound
compaction  : 1800 + 900 + 400*3        = 3900, flat
facts       : 1800 + 250 + 400*3        = 3250, flat
retrieval   : 1800 + 1500 + 400*3       = 4500, flat

Total input across N turns:
replay      : 1800*N + 400*N*(N-1)/2    -> QUADRATIC in N
the others  : c*N                       -> LINEAR in N

At N = 10, replay bills 1800×10 + 400×45 = 36,000 input tokens against compaction’s 39,000 — replay is cheaper, and it is also simpler and lossless. At N = 50 replay bills 90,000 + 392,000 = 482,000 against 195,000. At N = 200 it is 360,000 + 7,960,000 ≈ 8.3 million against 780,000, a factor of about eleven.

The crossover is the useful output here, not the ratio. Below roughly fifteen turns on these assumptions, the bounded architectures cost more than replay and lose information for the privilege. That is worth knowing, because a great deal of memory machinery gets built for applications whose median session is six turns long. Compute your own crossover from your own average turn size before building any of it — and note that the quadratic term is the whole story of a long chat’s bill.

Choosing, and combining

In practice a mature assistant runs the middle two together and treats the fourth as optional: a fact store for cross-session state, rolling compaction within a session, and retrieval over the archive only when the archive is genuinely large. The combination is also why the precedence question has to be answered explicitly — when the fact store says the region is eu-central-1 and a retrieved passage from March says us-east-1, the assembler must decide, and the safe default is that fresher and more structured wins, with the conflict logged rather than silently resolved.

The write path nobody designs

Every one of these architectures has a read path that gets designed carefully and a write path that gets an afterthought, which is where memory systems actually fail.

• What triggers a write? Every turn is expensive and noisy. A model-judged “is there anything durable here?” gate is cheap and misses things. An explicit user action (“remember that”) is accurate and rare. Most systems need two of the three.
• What supersedes what? A fact store without update semantics accumulates contradictions, and contradictions in the window are worse than absence — the model will pick one, plausibly, and not tell you which.
• Can the user see and delete it? Anything stored about a person that they cannot inspect is a compliance problem waiting for a request, and an inspectable memory list is also the single best debugging tool you will have.
• Is a write reversible? An incorrectly extracted fact — the user quoted someone else’s preference, and it was recorded as theirs — will otherwise be injected into every future session forever.