Context Window Claims vs Usable Context

A model advertised with a million-token context window will accept a million tokens. That is a true claim and it is the only one being made. Whether the model uses information at token 700,000 as effectively as information at token 700 is a separate question, with a separate and less flattering answer.

What the number is a claim about

The advertised context length is a constraint of the serving configuration: the maximum sequence the model will accept without returning an error. It is a statement about acceptance, not about quality, and it is not misleading in the way a false benchmark would be — it is precisely accurate about the thing it describes, and readers import a promise about behaviour that was never made.

The reason this is worth pulling apart is that the two properties came apart quite recently. When windows were short, the maximum length and the usable length were close enough that the distinction did not matter. At the current scale they differ by a lot, and system designs built on the advertised number degrade in ways that are hard to attribute after the fact.

Four numbers, one label

The second row is the one that catches people in production, because it is invisible until a request fails. The fourth is the one that catches people in design, because it never fails — it just gets quietly worse. The input-plus-output arithmetic and effective context length each go through their half in detail.

Where the gap comes from

Three mechanisms, and they compound rather than overlap.

• Training length versus extended length. Long windows are frequently achieved by extending a model trained at a shorter length, using positional-encoding techniques that let it accept longer sequences. Acceptance is not the same as having been trained to use the range, and behaviour in the extended region is where degradation concentrates.
• Positional effects. Retrieval accuracy is reproducibly higher for material at the start and end of a long context than for material in the middle — the lost-in-the-middle effect. This is a property of where you put the information, not of how much you sent.
• Attention dilution. Attention weights are normalised across the whole sequence. More material means the relevant tokens compete with more irrelevant ones, so signal-to-noise falls as the window fills even when nothing is forgotten.

Why the retrieval test flatters the number

The standard evidence for long-context performance is a needle-in-a-haystack test: place one distinctive sentence somewhere in a long document and ask the model to find it. Models score very well on this, and the scores are real.

The test is also close to the easiest possible long-context task, for three specific reasons. The target is lexically distinctive, so it stands out from surrounding text on surface features alone. There is exactly one thing to find. And nothing has to be combined — the answer is a copy, not a synthesis.

Real long-context work is none of those. Summarising a contract requires attending to many passages at once. Answering “which of these clauses conflict” requires holding several distant items simultaneously. Reasoning over a long conversation requires knowing which of many similar earlier statements is the operative one. Multi-fact and aggregation variants of the test show substantially steeper degradation than the single-needle version, and they are the variants that resemble what people actually do. A near-perfect single-needle score is therefore consistent with poor performance on the task you have.

Finding your own limit

The measurement is cheap and nobody can do it for you, because the effective length depends on the task. Take real examples, construct versions at several context lengths holding the question fixed, and score the answers the way you would score them in production. The curve is usually flat and then bends, and the bend is your working limit. It will be well below the advertised figure and well above zero.

That last clause matters, because the sceptical version of this page would stop at “the number is marketing” and that is not true. Long windows are a genuine capability that changed what is buildable: whole documents in one request, long agent transcripts, codebases held in context. The advertised figure understates nothing and overstates only in the reader’s inference. What changed is that context length went from being a hard wall you hit with an error to being a soft gradient you slide down without notice, and soft gradients need measurement where hard walls did not.

Two design consequences follow regardless of where your bend is. Put the material that matters at the start or the end rather than the middle. And prefer selecting a smaller amount of relevant context over sending everything, which is the entire argument for retrieval surviving the arrival of long windows — what a million-token window changes and what it does not works through that trade-off.