What Happens When a Migration Target Has a Smaller Context Window

A smaller window is not automatically a blocker. It is a constraint whose right answer depends entirely on where your tokens come from — and in a majority of real cases a large fraction of them turn out to be repetition, dead tool output, or a retrieval step with no limit on it, none of which you lose anything by removing.

What overflow looks like

Overflow is a hard, deterministic failure at request validation. It arrives as HTTP 400 with a message naming the limit and your count — along the lines of “prompt is too long: 234527 tokens > 200000 maximum”, or “This model’s maximum context length is 128000 tokens. However, your messages resulted in ... tokens.” The exact wording varies; the useful content is always the two numbers.

Three things follow from it being a validation error. It is not retryable, so any generic retry loop around the call is wasting time — classify it as permanent. It fires before any generation, so it costs nothing and the two numbers are free diagnostic information; log both. And it is deterministic in the input, so it will reproduce exactly from a saved request body, which makes it the easiest class of production failure to debug offline.

There is a quieter variant to watch for. Some stacks — a framework, a gateway, your own code — silently truncate an oversized prompt rather than failing. You get an answer, it is subtly wrong because the middle of the document is missing, and nothing anywhere records that anything happened. A hard 400 is the better failure mode, and if your stack is hiding it you should turn the hiding off before migrating.

Triage: where are the tokens?

Do this before choosing a strategy. Take the largest real request that fails and attribute every token in it to one of five buckets, by counting each component separately with the target’s tokenizer.

• Tool schemas. Fixed per request. Frequently a few thousand tokens and frequently forgotten.
• System prompt. Fixed. Usually smaller than people assume, and often containing accreted instructions nobody has read in a year.
• Retrieved documents. Variable, and the bucket most likely to be unbounded because a retriever configured to return “the top results” often has no total-size cap at all.
• Conversation history. Grows monotonically. In an agent loop this is dominated by tool results, not by anything a human wrote.
• The current user input. Usually small, occasionally enormous when somebody pastes a file.

The attribution decides the strategy, and the mapping is fairly mechanical: retrieval-dominated means cap the retriever; history- dominated in an agent means prune tool results; history-dominated in a chat means summarise; single-document-dominated means chunk. Skipping the attribution is how teams end up building a summarisation pipeline for a problem that was one unbounded retrieval call.

The reductions that cost nothing

Exhaust these before considering any strategy that loses information, because in a great many cases they close the gap on their own.

• Cap the retriever by tokens, not by document count. “Top 10 chunks” is unbounded if chunks vary in size. Accumulate until a token budget is reached and stop.
• Drop tools the request cannot use. If the route is answering a billing question, the eight code-execution tools are pure overhead. Selecting tools per route is a small change with a large fixed saving.
• Prune stale tool results. In an agent loop, the full output of a search from twelve steps ago is rarely load-bearing once its conclusion has been used. Several APIs now offer server-side context editing that clears old tool results for you; doing it yourself is also entirely viable.
• Deduplicate retrieval. Overlapping chunks from adjacent regions of the same document are common and are pure repetition.
• Reduce the output reservation. If you reserve 16,000 output tokens for responses that are reliably under 1,000, you are giving away window for nothing.

Truncation, summarisation, chunking

If the gap survives the free reductions, one of these three applies. They are not interchangeable and they fail in different ways.

Truncation drops content by position — oldest turns, or the tail of a document. It is cheap, adds no latency, and is fully deterministic, which matters more than it sounds: it is the only one of the three that does not change the meaning of a repeated request. Its failure mode is silence. The model does not know something was removed, so it answers confidently from a partial picture. Two mitigations are worth the effort: always keep the system prompt and the most recent turns, and insert an explicit marker where content was removed, so the model can say it lacks information rather than inventing it. Truncating from the middle rather than the start preserves both the framing and the recency, which is usually what you want in a conversation.

Summarisation replaces old content with a compressed version, typically by asking a model to write it. It preserves far more than truncation per token retained and is the standard answer for long conversations; several providers now offer it as a server-side feature. Its costs are real: an extra model call in the request path, which adds latency at exactly the moment the conversation is already long; non-determinism, since the same conversation can compress differently on two runs and your outputs stop being reproducible; and lossiness that is unpredictable rather than positional, because a summariser drops what it judges unimportant and it is not always right. Specific identifiers — account numbers, file paths, exact figures — are the classic casualty, and a summariser instruction that names the categories which must be preserved verbatim is worth writing.

Chunking splits the work into several requests and combines the results, rather than compressing anything. It loses nothing and scales to inputs of any size, which makes it the only real answer when a single document genuinely exceeds the window. What it costs is a different shape of program: N requests instead of one, so cost and latency scale with input size, and a combination step that is itself a design problem. Cross-chunk reasoning is where it breaks — a question whose answer depends on relating page 3 to page 300 cannot be answered by any chunk in isolation, and no combining strategy fully recovers that.

Choosing, and what it costs you

The choice follows from the triage, and from one further question: what does the request actually need?

• Long conversation, recent context matters most. Truncate from the middle, keeping the system prompt and the last N turns. Cheapest, deterministic, and adequate far more often than people expect.
• Long conversation, early facts still matter. Summarise the older portion, with explicit instructions to preserve identifiers, decisions and numbers verbatim. Accept the latency and the loss of reproducibility, and cache the summary so you pay for it once rather than on every subsequent turn.
• One document larger than the window. Chunk. Nothing else is honest. If the question is a lookup, retrieval over the chunks beats sending all of them.
• Agent loop with accumulating tool output. Prune tool results rather than summarising the conversation. The reasoning trail is usually worth more than the raw outputs it was derived from.

Whatever you choose, make the reduction observable. Log a counter every time content is dropped or compressed, with how many tokens went and which strategy ran. Without it, quality regressions from over-aggressive reduction are invisible: nobody reports “the answer was worse because the middle of my document was missing”, they report that the new model is not as good.

And re-run the measurement afterwards. The point of auditing the real distribution first is that it gives you a number to compare against once the reductions are in, so you can tell whether you have genuine headroom or merely moved the failure to a slightly larger input.