Structuring Context for Retrieval Within the Window

Two requests containing exactly the same words in a different sequence are different requests, with different costs and different answers. That is uncomfortable, but it is the property everything in this page is built on.

Same tokens, different request

Order is not a presentational choice. It changes three separate things, and it is worth keeping them apart because they suggest opposite designs.

• What the model uses. Position within a long input affects how reliably material is used. This is the effect the published work measures.
• What can be cached. Prompt caching works on prefixes. The first byte that differs from the previous request ends the reusable region, so anything volatile placed early destroys the cacheability of everything after it.
• What survives truncation. Whatever your allocator or the provider drops when the input is too long, it drops from one end. Order decides which block is at that end.

What the position research reports

The load-bearing citation is Liu et al., “Lost in the Middle: How Language Models Use Long Contexts” (TACL, 2024). The experiment inserts a document containing the answer at varying positions among distractors and measures accuracy as a function of that position. The reported shape is a U: performance is highest when the relevant material sits at the very beginning or the very end of the input and lowest when it is in the middle.

Two caveats are part of the finding and are usually dropped when it is repeated. The effect is reported across the models the authors tested, at the lengths they tested, on retrieval-style tasks — it is not a law of nature and later models are not guaranteed to reproduce it in the same shape. And it is about a single input, not about a conversation: nothing in the paper says anything about turn forty. Use it as a prior that motivates an ordering policy, not as a number.

The practical translation is short. Do not put anything you need reliably used in the middle of a long input. The middle is where filler goes.

Two orderings that disagree

Here is the tension that the usual advice steps over. Both of the following are correct, and they are incompatible:

relevance order  : most relevant material near the edges,
                   ideally adjacent to the question at the end
cache order      : least volatile material first, so the longest
                   possible prefix is byte-identical across requests

Retrieved documents are HIGHLY relevant and HIGHLY volatile.
Relevance wants them last. The cache does not care where they go,
but anything placed BEFORE them cannot be cached if they move.

The resolution is not a compromise, it is a layering. Sort by volatility first to define the cacheable prefix, then apply relevance ordering within the volatile suffix. That way the static blocks form a stable prefix and never move, and the ordering decisions that actually matter for quality are all made among the blocks that were going to be uncacheable anyway. The economics of the prefix are worked through separately; here it is just a constraint the ordering function has to respect.

One consequence surprises people: the conversation history should generally come before the retrieved documents, not after. History is append-only, so its prefix is stable turn to turn; retrieval changes completely on every request. Putting retrieval before history means every turn invalidates the cache for the whole history block. Putting history first keeps the growing-but-stable part cacheable and leaves the churning part at the end, where relevance ordering also wants the good material.

The ordering function

type Rank = { volatility: 0 | 1 | 2 | 3; relevance: number };

// volatility: 0 = never changes, 1 = changes rarely, 2 = per session,
//             3 = per request. Ascending order defines the prefix.
const RANK: Record<string, number> = {
  system:    0,   // persona, policy, output contract
  tools:     0,   // schemas, serialised with STABLE key order
  reference: 1,   // static corpus carried on every request
  record:    2,   // compacted session state
  history:   2,   // append-only turns
  retrieved: 3,   // changes every request
  toolresult:3,
  question:  3,   // the current turn, always last
};

function order(blocks: Block[]) {
  return blocks.sort((a, b) => {
    if (RANK[a.kind] !== RANK[b.kind]) return RANK[a.kind] - RANK[b.kind];
    // Within the volatile tail, relevance ascending: weakest material in
    // the middle of the tail, strongest adjacent to the question.
    return a.relevance - b.relevance;
  });
}

// Retrieved documents, ordered within their own block:
//   [3rd best, 5th, 6th, 4th, 2nd, BEST]  <- best nearest the question
function orderDocs(docs: Doc[]) {
  const s = [...docs].sort((a, b) => b.score - a.score);
  const out: Doc[] = [];
  s.forEach((d, i) => (i % 2 === 0 ? out.push(d) : out.unshift(d)));
  return out;                       // strongest at the ends, weakest mid
}

orderDocs is the “lost in the middle” result turned into four lines: it interleaves so that the highest-scoring documents land at the two ends of the block and the lowest-scoring ones sit in the middle, which is exactly the region the paper reports as least reliably used. If you only take one thing from the ordering literature into code, take this.

Note what is not in the function: no reordering of conversation turns. Chronology is semantics in a dialogue, and reordering it produces incoherence rather than improved retrieval. Ordering policies apply between blocks and within retrieval sets, not within a transcript.

Boundaries matter as much as order

Ordering only helps if the model can tell where one block ends and the next begins. Two concatenated documents with no separator are one document with a confusing middle, and a retrieved passage abutting the user’s question can read as part of it. Every block wants an explicit boundary carrying at least its kind and, for retrieved material, its provenance — the encoding choice is a separate decision with its own token cost.

Provenance on retrieved blocks does a second job worth mentioning: it is what makes a citation checkable, and it is what lets you answer “did the model even see this?” when a session goes wrong. A block that carries an id can be searched for in a context dump; an anonymous wall of text cannot.

One last ordering rule that costs nothing: put the actual question last, always. It is the one block whose position is not a trade-off — it is maximally relevant, maximally volatile, and belongs at the end under both orderings. A surprising number of assemblers put the question before the retrieved material because that is the order the user experienced it in, which places the request the model must satisfy in the least favourable position available.

A closing caution about how much to invest here. Ordering effects are real and they are also small compared with the effects of getting eligibility and allocation right. A perfectly ordered context that is missing the relevant document is worse than a badly ordered one that contains it, and the two failures present identically. If a page in this cluster deserves your afternoon before this one does, it is the filter that decides what is in the window at all. Order last, once the contents are right.