What a 1M-Token Context Window Is Actually Good For

Million-token windows arrived with a claim attached: retrieval is obsolete, just put everything in the prompt. The claim is wrong, but the interesting part is why it is wrong, because the tasks where long context genuinely wins are a real and identifiable category rather than an empty set.

What it genuinely solves

The useful distinction is between lookup tasks and global-structure tasks. A lookup task has an answer that lives in one place; retrieval is designed for exactly that and beats long context on cost by orders of magnitude. A global-structure task has an answer that depends on the whole document, and no chunk retriever can construct it because there is no chunk that contains it.

• Consistency and contradiction finding. “Does clause 14 conflict with anything in this contract?” requires having read all of it. A retriever asked for “clause 14 conflicts” has nothing to embed against.
• Whole-artefact summarisation. Summarising a novel, a deposition or a year of incident reports needs proportional representation of the whole, which map-reduce over chunks approximates poorly and expensively.
• Codebase reasoning across files. “Which call-sites break if I change this signature?” is a graph question. Embedding similarity does not find call-sites; it finds text that looks like the function.
• Many-shot prompting. Hundreds of worked examples in the prompt is a legitimate use of the space and behaves differently from a handful — it is closer to lightweight adaptation than to instruction.
• Not having to build a pipeline yet. Genuinely valuable. A long-context prototype answers “is this feature worth building” in a day instead of a fortnight, and you can replace it with retrieval once you know.

What it does not solve

It does not make the model attend evenly. Retrieval accuracy varies with where in the window the relevant passage sits, a result documented in the literature and covered on lost in the middle, and the advertised length is not the length at which quality holds — see effective context length.

It does not scale past the window either, and that is the structural point people skip. A million tokens is perhaps a few thousand pages. A corpus of a hundred thousand documents does not fit in any window that will ever exist, so a retrieval layer is not a stopgap you outgrow — it is the only thing that addresses that problem class at all. Long context changes how big a chunk you can afford to retrieve, not whether you retrieve.

And it does nothing for freshness. Data that changes hourly must be fetched per request; putting it in a long, cached prefix is precisely the wrong shape.

The arithmetic nobody runs first

Take a 400,000-token document and a support agent that answers 5,000 questions a day against it. Stuffing the document in every request, at a hypothetical $3.00 per million input tokens:

stuffed:   400,000 tok x $3.00/M          = $1.20 per question
           x 5,000 questions               = $6,000 per day

retrieved: 6 chunks x 700 tok = 4,200 tok = $0.0126 per question
           x 5,000 questions               = $63 per day

cached:    400,000 tok at a 0.1x read multiplier
                                           = $0.12 per question
           x 5,000 questions               = $600 per day

The prices are illustrative and you should substitute the provider’s current ones, but the ratios are structural and they do not move: retrieval is roughly two orders of magnitude cheaper than stuffing, and caching recovers about one of those two orders without changing the architecture. Which is the real lesson — if you are going to stuff, cache, and if the prefix is stable enough to cache then you have most of the win.

Latency follows the same shape. Prefill is one parallel pass but it is a pass over everything, so time to first token grows with prompt length. A long-context call is not just dearer, it feels slower before a single word appears, and the gap is large enough to change what kind of interface you can build on top of it: a 400,000-token prefill is not a spinner, it is a progress bar and an email when it is done.

There is a third cost that is easy to miss because it does not appear on an invoice. Every token you add is a token the model has to consider, and beyond a certain length that is not free in quality terms either — irrelevant material competes for attention with the relevant material. So stuffing has a failure mode that retrieval does not: adding more context can make the answer worse while making the bill larger, which is the least satisfying combination available.

A decision procedure

The two are not exclusive, and the strongest production pattern uses both: retrieve generously into a large window rather than sparingly into a small one. A 128k budget lets you take twenty candidate passages where an 8k budget forced you to take three, which relaxes the demand on the reranker considerably — recall at twenty is a much easier target than precision at three. That is the quiet, real benefit of long context for most applications, and it is nothing like the claim that retrieval is obsolete.