Is RAG Dead Now That Context Windows Are Huge?

The argument is not about whether a million-token window can hold your corpus. Assume it can. The argument is about paying to re-read the corpus on every single request, and that is arithmetic rather than opinion.

The claim, stated fairly

The strong version of “RAG is dead” goes like this: retrieval exists because context windows were small; windows are no longer small; a retrieval pipeline is chunking decisions, an index to maintain, an embedding model to version and a recall problem you can never fully close, so if you can simply put everything in the prompt, you should, because the model is a better retriever over its own context than your top-k is.

That last clause is the part worth taking seriously. A model attending over the full document does not have a chunk boundary problem and does not miss a passage because its embedding was three degrees off. If your corpus fits and you can afford it, stuffing really is simpler and often more accurate.

The words doing the work are fits and afford.

The ratio that settles it

Per query, stuffing costs you the whole corpus as input tokens. Retrieval costs you the retrieved slice. So, ignoring everything else for a moment, the cost ratio is:

ratio  =  corpus_tokens / retrieved_tokens  ×  (1 - cache_discount)

where cache_discount is whatever fraction the provider takes off a
cached prompt prefix, and 0 if the corpus changes between queries.

That is the entire economic case, and it has an important property: it does not depend on the price per token at all. Whatever you pay, you pay it on that many more tokens. Prices fall for both sides of the comparison simultaneously, which is why “tokens are getting cheaper” is not an argument against retrieval — it is an argument against expensive workloads in general.

The one thing that moves the ratio is prompt caching. A corpus that is byte-identical across requests can be cached as a prefix, and a provider that charges a fraction of the input rate for a cache hit effectively divides the stuffing cost by that fraction. This is the real reason the “RAG is dead” argument got louder: it is not the window size, it is the cache.

A worked instance

Substitute your own numbers; these are assumptions, not measurements. Take a corpus of 2,000,000 tokens — roughly a 4,000-page documentation set — and a retrieval pipeline that puts 3,000 tokens of context in front of the model.

Even with an aggressive cache discount that is a factor of about 67. At 100,000 queries a month the difference between $60,000 and $900 is not a rounding error, and no amount of index-maintenance toil costs $59,000 a month.

Now run it the other way. Corpus of 40,000 tokens — one long contract. Stuffing costs 40,000 tokens; retrieving costs 3,000. The ratio is 13, the absolute difference is a tenth of a cent, and you have spent a week building a pipeline to save nothing. Below roughly a hundred-thousand-token corpus, retrieval is usually not worth the engineering.

Where long context actually wins

• Small corpora. Anything that fits comfortably in the window at a price you would pay anyway. See above.
• Global questions. “What are the recurring themes across these 200 interviews?” has no answer in any k chunks, because the answer is a property of the whole set. Top-k retrieval structurally cannot serve it; either stuff or build the hierarchical summaries that graph-style approaches use.
• One-shot analysis. If each document is queried once or twice, index construction never amortises. Read it, use it, discard it.
• Precision-critical work over a fixed document. Due diligence over one filing, where missing a clause is the failure mode and recall matters more than the bill.

There is also a middle design that gets argued past. Retrieve generously — not five chunks but every document that plausibly matters — and stuff those whole. This is retrieval used as a coarse filter rather than as a precision instrument, and it is often the right shape for a corpus of a few hundred documents where you can cheaply narrow to the three relevant ones and then let the model read them entire. You get the cost profile of retrieval and most of the quality profile of stuffing, and the chunking problem largely disappears because you are selecting documents rather than passages.

The maintenance argument deserves an honest hearing too, because it is the strongest case against retrieval and it never appears in the cost model. An index is a second copy of your data that can be stale, partially deleted, embedded with a superseded model or scoped to the wrong tenant. Stuffing has none of those failure modes: the document you send is the document as it exists right now. That is a real engineering saving, and for a small corpus it can be worth more than the token difference.

The honest summary is that these are not competitors. Retrieval is a cost-control mechanism for large corpora and a precision mechanism for heterogeneous ones. Long context removed the hard constraint that used to force retrieval on everybody, including people with a 40,000-token corpus who never should have built a pipeline. It did not remove the arithmetic.

The quality argument is separate

There is a second claim underneath the cost one: that a model reads a long context as well as it reads a short one. That is not established. Liu et al., “Lost in the Middle” (arXiv:2307.03172), documented a U-shaped accuracy curve in multi-document question answering — models used information best when it sat at the beginning or the end of the input and worst when it sat in the middle. Later models handle this better, but it is a per-model property that has to be checked rather than assumed.

So the decision procedure is: compute the ratio; if it is small, stuff and stop reading; if it is large, retrieve, and treat the retrieved set as something whose ordering matters.