Migrating a Prompt's Handling of Multi-Turn Context Truncation

Your chat service keeps the last twelve turns. That twelve was derived once, from a window you no longer have. The migration target’s window is several times larger, so the obvious move is to keep more turns — and the obvious move degrades answers on exactly the conversations you built the truncation for.

The budget your strategy encoded

“Last twelve turns” is a compiled artefact. Somebody once measured a typical turn, subtracted the fixed costs from the window, and divided. The inputs to that division are all specific to the old model and none of them are written down next to the constant. The full budget equation is:

window
  − system prompt
  − tool schemas
  − retrieved documents
  − reserved output (max_tokens)
  − reserved reasoning (if the model thinks before answering)
  = tokens available for conversation history

Two of those terms are easy to forget and both bite during a migration. Reserved output is not optional: the completion is drawn from the same window as the input on most APIs, so a generous max_tokens takes the space directly out of your history allowance — the distinction is the subject of context window versus max tokens. Reserved reasoning is newer and catches people out: on models that think before answering, thinking tokens are counted against max_tokens together with the visible reply, so a history budget computed for a non-thinking model leaves no room and responses truncate mid-sentence.

Re-deriving it for the new model

Work it with the terms labelled, substituting your own figures. Suppose the new model has a 1M-token window, your system prompt and tool schemas together come to 6,000 tokens under the new tokenizer, you attach up to 20,000 tokens of retrieved documents, and you reserve 16,000 tokens for output including reasoning.

available = 1,000,000 − 6,000 − 20,000 − 16,000 = 958,000 tokens

median turn (measured on your own logs, new tokenizer) = 420 tokens
958,000 / 420  ≈  2,280 turns

Compare against the old model:
  200,000 − 5,000 − 20,000 − 4,000 = 171,000
  171,000 / 350 ≈ 488 turns  — and you kept 12.

Both numbers are absurd, and that is the point of doing the arithmetic: the old constant of twelve was never a capacity limit either. It was a quality and cost decision wearing a capacity limit’s clothes. Somebody found that beyond a dozen turns the answers got worse or the bill got silly, and encoded that as a truncation rule. So the question the migration raises is not “how many turns fit now” but “does the quality-and-cost trade-off sit in the same place”.

A bigger window moves where the model looks

This is the part a capacity calculation misses. Retrieval accuracy within a long context is not uniform across positions. Liu et al., in Lost in the Middle: How Language Models Use Long Contexts (arXiv:2307.03172, published in TACL), showed that performance is highest when the relevant information sits at the beginning or the end of the input and degrades when it sits in the middle — and that the degradation grows as the context lengthens. Later models handle long context better than the ones in that study, and vendors publish their own long-context evaluations, but the shape of the effect is a property of how attention over long sequences behaves rather than a bug that was fixed.

The consequence for a truncation strategy is direct. When you kept twelve turns, everything you kept was near the end of the prompt, in the region the model attends to most reliably. Keep two hundred turns and turn number ninety is in the middle — technically present, practically faint. You have not made that information more available, you have made it available in a worse position, and you have paid for the privilege on every request.

A second effect compounds it. A longer history contains more contradictions: preferences the user stated and then revised, decisions that were reversed, a name that was corrected. With twelve turns the stale version has usually fallen out of the window. With two hundred, both versions are present and the model has to adjudicate. Users experience this as the assistant “remembering” something they took back.

The boundary where retrieval beats stuffing

There is a crossover. Below it, keeping raw turns is simpler, cheaper and better. Above it, you should be selecting from history rather than including it. The crossover is not a token count you can read off a spec sheet, but it is bounded by three observable things.

• Cost per turn. Every request re-sends the whole history. History of length n over a conversation of n turns is quadratic in total tokens billed, and prompt caching only flattens the part of the prefix that is byte-identical across requests. Compute the cost of your 95th-percentile conversation at the new history length before deciding.
• Latency. Prefill is roughly linear in input length. An interactive assistant has a time-to-first-token budget, and a history that pushes past it has failed a product requirement whatever the answer quality.
• Position. The check that matters most and the one nobody runs: take conversations where the answer depends on something said early, and measure whether the model still gets them right at the new history length. This is exactly the shape of a multi-turn context-loss test, and it is the test to write first.

What the new strategy looks like

A strategy that survives a window change has three layers rather than one cutoff, and it keeps the important material at the ends of the prompt where position effects favour it.

A pinned header. Durable facts about the conversation — the user’s stated constraints, decisions taken, entity names — maintained as a compact block immediately after the system prompt. Updated by an explicit summarisation step rather than accumulated. This block is small, it is at the front, and it is the thing you most want the model to attend to.

A verbatim tail. The last k turns in full, where k is chosen for conversational coherence rather than for capacity — pronoun resolution and follow-up questions rarely reach back further than a handful of turns. This sits at the end, the other favourable position.

A retrieved middle. Everything else indexed and selected per request by relevance to the current message, rather than included by recency. Two or three retrieved excerpts beat two hundred raw turns, cost a fraction as much, and put the relevant material next to the tail instead of burying it.

Express all three as token budgets, not as turn counts, and derive the budgets from the equation at the top of this page. Then the next window change is a configuration edit with a recomputed number, and the question “how many turns should we keep” never has to be re-litigated from memory.