Context Rot: Why Long Sessions Get Worse

Long sessions get worse, and the usual explanation — the model struggles with long context — is at most half of it. The other half is entirely mechanical, lives in your own assembly code, and will not be fixed by a better model.

What the literature actually says

There is real published work here and it is worth citing precisely rather than gesturing at, because the popular version overstates it.

• Liu et al., “Lost in the Middle: How Language Models Use Long Contexts” (TACL, 2024). The paper places a relevant document at different positions within a long input and reports a U-shaped performance curve: material at the beginning and end of the input is used more reliably than material in the middle. It is a positional finding about a single input, not a claim about conversations over time.
• Needle-in-a-haystack testing, popularised by Greg Kamradt, plants a fact in a long document and asks for it back across depths and lengths. It is a retrieval probe, and models passing it does not establish that they can reason over everything they can retrieve — a distinction the marketing use of these results routinely elides.
• RULER (Hsieh et al., 2024) extends the idea to harder synthetic tasks — multi-hop tracing, aggregation, multiple needles — and reports that models’ effective usable length falls short of their advertised context length. The advertised-versus-effective gap is the tokens cluster’s subject; what it means here is that your allocator should not treat the whole window as equally usable.

None of these studies measures a forty-turn conversation, and no honest page can tell you the shape of degradation over turns because nobody has published it in a form that transfers across models and tasks. What can be stated confidently is the other half: three causes of decay that come from how context accumulates, are visible by inspection, and are fixable in code.

Cause one: superseded state

This is the big one and it is entirely self-inflicted. At turn 4 the user says the deadline is Friday. At turn 19 they say it moved to the following Wednesday. Both statements are now in the window, both are equally authoritative-looking, and nothing marks the first as dead. The model is being asked to infer temporal precedence from message order — which it can often do, and does less reliably as the distance grows and as the number of such pairs multiplies.

The failure is not a probabilistic wobble; it is a genuine ambiguity you created. A long session accumulates dozens of these: rejected approaches still present in full, a file’s contents from before it was edited sitting alongside the version after, a plan that was revised twice with all three versions in the transcript. Ask what fraction of a fifty-turn context is statements that are no longer true, and the answer for most agent transcripts is uncomfortable.

The fix is state, not summarisation. A typed record with supersession holds one current value per constraint, so the window contains the Wednesday deadline and nothing else. This is the strongest single intervention available against long-session decay, and it is deterministic code.

Cause two: instruction dilution

At turn 1 the system prompt is 1,800 tokens out of 2,400 — three quarters of everything in the window is instruction. At turn 40 it is 1,800 out of 90,000: two percent. The instructions have not changed and their share of the input has fallen by a factor of forty.

Whatever the mechanism, the operational consequence is not controversial and is easy to observe in any long-running assistant: formatting rules erode first, then tone, then the more specific behavioural constraints, in roughly the order of how much surrounding content competes with each. The output stops matching the contract long before it stops being reasonable, which is why this is often discovered by a downstream parser rather than by a human.

Two structural mitigations follow, and both are placement decisions rather than wording ones: bound the growth so the ratio cannot fall indefinitely, and restate the parts of the contract that must not erode near the end of the input where the position effect is favourable — where instructions go is a real trade-off rather than a free win, because it also affects what can be cached.

Cause three: tool sludge

In an agent, the majority of a long context is usually not conversation at all. It is accumulated tool output: directory listings, query results, fetched pages, diffs, test runs. Almost all of it was relevant for exactly one step and has been re-sent every step since.

Sludge does three things. It crowds out the material that is still relevant, competing for the same allocation. It pushes the current instruction further from the start of the input, into the region the position literature says is least reliably used. And it re-presents old world-states — the directory listing from before four files were created, the test output from before the fix — as if they were current, which is cause one again in a different costume. Bounding and ageing tool results is the direct treatment.

Mitigations, in order of effect

• Maintain state instead of transcript. One current value per fact, superseded values removed rather than appended. Attacks the largest cause directly.
• Bound every accumulating block. If any block can grow without limit, the session has an expiry date; the only question is when. Give history and tool output explicit ceilings in the allocator.
• Age tool results aggressively. Steepest relevance decay of anything in the window, and the cheapest to evict because the payload is still reachable by handle.
• Restate the contract late. Cheap, uses the position effect the literature actually supports, and costs only the tokens of the restatement — though it does sit outside the cacheable prefix.
• Consider a fresh session with a handoff. Some sessions should end. A handoff document plus a new session resets every one of these causes at once, and is often better than heroic compaction of a context that has become mostly archaeology.