Long-Context RAG: Stuffing 50 Chunks and What Breaks

If retrieval is imperfect and the window is enormous, why not retrieve fifty chunks instead of five and let the model sort it out? Because recall and cost do not grow at the same rate, and because where a chunk sits in the context turns out to matter.

The temptation

It is a reasonable instinct. Every RAG failure traced to a missing document argues for a larger k, and a model with a large context window makes the constraint feel gone. Retrieve fifty, pass them all, and the recall problem is solved by brute force.

Two things go wrong. The bill grows linearly in k while recall flattens, and the model does not attend to a long context uniformly — so the marginal chunk is both the most expensive one and the one least likely to be read.

What the published work found

The reference result is Liu et al., Lost in the Middle: How Language Models Use Long Contexts (arXiv:2307.03172, later in TACL). They constructed multi-document question answering tasks where the answer-bearing document was placed at a controlled position among distractors, and varied that position.

The reported pattern is a U shape: accuracy was highest when the relevant document sat at the very beginning or the very end of the input, and fell substantially when it sat in the middle. They also reported that with enough distractors, performance with the gold document in a middle position could drop below the model’s performance with no retrieved documents at all — the retrieval, in that configuration, made things worse than nothing.

Treat that as a finding about a class of behaviour rather than a constant. It was measured on models of that period, and long-context training has improved since; some newer models show a much flatter curve. What has not changed is that the property is model-specific and worth checking rather than assuming, and that the direction of the effect — ends better than middle — has been robust enough across follow-up work to design around.

Why needle tests do not settle it

A vendor showing 100% on a needle-in-a-haystack evaluation has not refuted the above, and understanding why is the useful part. The needle task inserts one distinctive, out-of-place sentence into filler text and asks the model to find it. It is a lexically trivial search: the needle does not resemble its surroundings.

Retrieved chunks are the opposite. All fifty are on the same topic — they were selected for that — and several are near-misses that discuss the right subject with the wrong specifics. The task is not to locate a distinctive string but to discriminate among plausible candidates and aggregate across a few of them. A perfect needle score is consistent with real degradation on that harder task, so it is not the evidence you want when deciding your k.

Two curves that diverge

Run the numbers on your own retrieval evaluation. Recall@k has a characteristic shape — steep from 1 to about 10, flattening after — because most questions are answered by a chunk the retriever already ranks highly, and the ones that are not usually need a different chunking strategy rather than a longer list. Cost is linear.

Assume 500-token chunks and an input rate of Gin per million tokens.

  k =  5  ->  2,500 context tokens   1.0x cost
  k = 10  ->  5,000                  2.0x
  k = 25  -> 12,500                  5.0x
  k = 50  -> 25,000                 10.0x

Now put your own recall@k beside it. If recall@50 exceeds recall@5 by
a handful of points, you are paying ten times the input cost for that
handful — and adding 45 chunks of distractor to every request that
did not need them.

Latency moves too: prefill is roughly linear in prompt length, so
time to first token grows with k on every single request.

The reranker is what breaks the trade-off, and this is the strongest argument for it. Retrieve 50 so that recall@50 is your ceiling, rerank to 5 so that cost, latency and distraction are all at the k = 5 level. You buy the recall of the long list without paying for its tokens.

Ordering as a free variable

Given the position effect, the order in which you concatenate the retrieved chunks is a real parameter that most pipelines never set. Options, roughly in order of how much they are worth trying:

• Best first. The default and a reasonable one. Puts the strongest evidence in the region models attend to most reliably.
• The sandwich. Best chunk first, second-best last, the remainder in the middle. Directly exploits the U shape, and costs nothing to implement.
• Best last. Immediately before the question. Worth testing, because recency in the context often has an outsized effect and this puts the strongest evidence closest to the generation.
• Source order. If your chunks are sequential parts of one document, ordering by position rather than by score gives the model coherent prose instead of a shuffled one, and can matter more than the score ordering.

Whichever you choose, number the sources and instruct citation, and measure the choice rather than reasoning about it — this is a cheap A/B on a fixed retrieval set, with no index change and no model change, which makes it one of the few genuinely clean experiments available in a RAG pipeline.

One more thing degrades a large context that has nothing to do with attention. Overlapping chunks mean that at k = 50 you are frequently sending the same sentences two or three times, and near-duplicate passages from different documents — a policy quoted in four places — multiply further. The model reads repetition as corroboration, which is the wrong inference when the repetition is an artefact of your splitter. Deduplicate the retrieved set before assembling the prompt: merge chunks that overlap in source range, and drop near-identical text with a cheap similarity check. On a corpus with heavy overlap this recovers a real fraction of the context budget for free.

If you want to know how your model behaves rather than how the 2023 models behaved, the experiment is small enough to run in an afternoon and is worth more than any published number. Take twenty questions from your evaluation set, build a fixed context of k chunks containing the correct one, and permute only its position — first, middle, last — holding everything else identical. Score accuracy per position. That is a clean, honest, single-variable measurement on the corpus you actually serve, and it tells you directly whether ordering is worth engineering.