Long-Conversation Degradation

Turn three follows the format. Turn twenty does not. Nothing was truncated, the constraint is still sitting there in the system prompt, and the model will confirm the rule if you ask it. Three separate mechanisms produce this, and only one of them is about context length.

It is not forgetting

A model has no memory between calls. Every turn re-reads the entire transcript from scratch, so nothing is forgotten in any literal sense — if the constraint is in the context, the model read it this turn as surely as it did on turn three.

What changes is weight. The instruction is now one paragraph among forty, competing with thousands of tokens of conversation, most of which the model itself wrote. Attention is a soft, normalised allocation over the whole sequence: adding material does not merely append, it dilutes. “It forgot” is better read as “the instruction lost the competition”, which is a different problem with different fixes.

Position effects

Liu et al.’s Lost in the Middle (TACL 2024) is the foundational result. Placing the relevant document at different positions within a long context, they found a pronounced U-shaped curve: performance is highest when the needed information is at the very beginning or the very end of the context, and drops markedly when it is in the middle. In some settings the middle-position performance fell below what the model achieved with no retrieved documents at all.

Hsieh et al.’s RULER (2024) added the capacity half: models advertising long context windows show substantial degradation well before their advertised limits on tasks harder than needle retrieval. The advertised number is a memory-allocation fact, not a comprehension guarantee.

Together these explain a long conversation directly. Your system prompt starts at the strong beginning position, and every turn pushes it further into the weak middle.

The multi-turn result

Laban, Hayashi, Zhou and Neville’s LLMs Get Lost in Multi-Turn Conversation (2025) isolated the conversational variable specifically. Their design is the clean part: take a fully specified task, then shard it — deliver the same information piece by piece across several user turns instead of all at once — and compare. Same information, same model, different delivery.

They reported a large average performance drop in the sharded setting across the models and tasks they tested — on the order of 39% — and, more interestingly, a large increase in variance. Their framing is worth quoting in substance: models make an assumption early, commit to it, and do not recover when later turns contradict it. Once a model takes a wrong turn, it gets lost.

That reframes the problem. It is not primarily that information decays with distance; it is that an early, under-informed commitment stays in the transcript and conditions everything after it. Which leads directly to the third mechanism.

Self-conditioning

After a few turns, most of the context was written by the model. It is predicting the continuation of a document in which an assistant behaves a particular way, and its own past output is the most locally relevant evidence about how that assistant behaves.

So a single deviation propagates. Drop a required field once and the transcript now contains an example of the assistant omitting it, which raises the probability of omitting it again. The same loop drives language drift and it is a mild version of the mechanism behind repetition loops. It also compounds with sycophancy: an accumulated record of agreeing with the user is itself evidence that agreeing is what this assistant does.

The adherence plot

The measurement that turns this from a complaint into a number. Define a set of invariant constraints that should hold on every single assistant turn — output is valid JSON, never states a price, always cites a source, always replies in the user’s language — and check all of them after every turn of a scripted conversation.

• x-axis: turn index, 1 to N. Not token count, though plotting against cumulative context length as a second series is informative when turn lengths vary.
• y-axis: adherence rate — across many replayed conversations, the fraction that still satisfy every constraint at that turn. Ranges 0 to 1.
• One line per constraint, plus the conjunction. The per-constraint lines are the useful part: formatting rules usually decay far faster than content rules, and the aggregate hides which is which.

def adherence_by_turn(scripts, constraints, n_runs=20):
    """scripts: lists of user turns. constraints: name -> predicate(text).
    Returns {constraint: [rate_at_turn_1, rate_at_turn_2, ...]}"""
    depth = max(len(s) for s in scripts)
    hits = {name: [0] * depth for name in constraints}
    total = [0] * depth
    for _ in range(n_runs):
        for script in scripts:
            history = [("system", SYSTEM_PROMPT)]
            for i, user_turn in enumerate(script):
                history.append(("user", user_turn))
                reply = call_model_with_history(history)
                history.append(("assistant", reply))
                total[i] += 1
                for name, predicate in constraints.items():
                    hits[name][i] += bool(predicate(reply))
    return {n: [h / t if t else None for h, t in zip(v, total)]
            for n, v in hits.items()}

Run it before and after a mitigation. This is the only way to know whether re-anchoring actually helped or whether you moved the failure from turn twelve to turn fifteen.

What actually helps

• Re-anchor at the end. Given the U-shaped curve, the single most effective change is to repeat the critical constraints immediately before the model generates — appended to the last user message or as a short trailing system message. The end position is strong; the middle is not.
• Compact to state, not to a summary. Replace old turns with a structured object — decisions made, constraints active, open questions — rather than a prose summary. Prose re-enters the transcript as more of the same material to be diluted; a state object is short and unambiguous, and it lets you drop the early commitment that the multi-turn result says is doing the damage.
• Consolidate before acting. The direct implication of the sharded-task finding: when a user has specified something across several turns, restate the full specification in one message and confirm it before the model acts. This converts a multi-turn task into a single-turn one, which is the setting where models perform best.
• Validate every turn, not the last one. A per-turn schema check with a single automatic retry stops one deviation from becoming the transcript’s new precedent.
• Start fresh when the task changes. A new task in an old conversation inherits every earlier commitment for no benefit. Carrying forward a state object into a clean context is almost always better than carrying the transcript.