Lost in the Middle: Why Long Context Degrades

Put the answer at the top of a long prompt and the model finds it. Put it at the bottom and the model finds it. Put it in the middle and accuracy sags — sometimes below what the same model scores with no documents at all. This is a documented result with a name, and it changes how you order a prompt.

The finding, and whose it is

The canonical reference is Liu, Lin, Hewitt, Paranjape, Bevilacqua, Petroni and Liang, Lost in the Middle: How Language Models Use Long Contexts, published in Transactions of the ACL in 2024. The method was deliberately simple: take a multi-document question-answering task, hold the number of documents fixed, and move the one document that actually contains the answer through every position in the input.

What they reported is a U-shaped curve. Accuracy is highest when the relevant document is first or last and drops in between, and the effect appeared across models with different architectures and different context lengths — including in models explicitly built for long context. They also reported the finding that makes it a design problem rather than a curiosity: on some configurations, performance with relevant information buried mid-context fell below the same model’s closed-book performance, meaning the retrieved context was net harmful.

A second, more informal line of evidence is the needle-in-a-haystack methodology popularised by Greg Kamradt in 2023: hide one sentence at a controlled depth in filler text, ask about it, and plot accuracy against depth and total length. It is less rigorous than the TACL study and far more practical, because you can run it on your own model in an afternoon.

Why position should matter at all

Nothing in the transformer architecture says early tokens are special. Several mechanisms conspire to make them so anyway:

• Attention is a competition with a fixed budget. Softmax weights sum to one. With ten competing passages each gets meaningful weight; with a thousand, the correct passage’s share of attention shrinks even if its score is unchanged.
• Position encodings extrapolate imperfectly. Rotary embeddings produce a relative-distance signal that decays; models served beyond the lengths they were trained at, via interpolation or scaling tricks, are operating where that signal is weakest.
• Training data has a length distribution. Very long sequences are rare in pre-training, so the ability to use position 500,000 is much less practised than the ability to use position 500.
• Recency and primacy are learned. In ordinary text, the beginning states the topic and the end states the conclusion. A model that learned to weight those positions is behaving correctly for its training distribution and wrongly for your prompt.

The method, so you can run it

Nobody here has run this on your model, and results from a paper about other models are not a substitute. This is the whole harness; the numbers it produces are yours:

import csv, itertools

NEEDLE = ("The maintenance code for the Rhodes turbine is ATLAS-4471, "
          "and it must be entered before any restart.")
QUESTION = "What is the maintenance code for the Rhodes turbine?"
ANSWER = "ATLAS-4471"

filler = open("haystack.txt", encoding="utf-8").read()   # any long neutral text

def build(depth_pct, length_tokens, tok):
    ids = tok.encode(filler, add_special_tokens=False)[:length_tokens]
    cut = int(len(ids) * depth_pct / 100)
    head, tail = tok.decode(ids[:cut]), tok.decode(ids[cut:])
    return head + "\n\n" + NEEDLE + "\n\n" + tail

rows = []
for length, depth in itertools.product([4000, 16000, 64000, 128000],
                                       [0, 10, 25, 50, 75, 90, 100]):
    ctx = build(depth, length, tok)
    out = call_model(system="Answer using only the context.",
                     user=ctx + "\n\nQuestion: " + QUESTION)
    rows.append((length, depth, int(ANSWER in out)))

with open("needle.csv", "w", newline="") as f:
    csv.writer(f).writerows([("length", "depth_pct", "hit"), *rows])

Four details decide whether the result means anything. Run each cell several times and record the hit rate, not one binary. Use filler that is topically unrelated to the needle, or you are measuring distractor-resistance instead of position. Vary the needle across runs, because a memorable string like a fictional code is easier to spot than a plausible sentence. And keep the question identical at every depth — one reworded prompt invalidates the comparison.

Reading your own curve

What you are looking for is not a score, it is a knee: the length at which the middle depths stop being reliable. That length is the real design constraint, and it is usually well below the advertised window. Plot hit rate against depth, one line per length. A flat set of lines means position is not your problem at these sizes. Lines that sag in the middle and sag further as length grows is the classic shape, and it tells you exactly how much room you have before ordering matters.

Watch for one confound that invalidates more of these home-grown experiments than any other: the position of the question. If your prompt puts the instruction first and the context after it, then moving the needle to the end also moves it closer to nothing in particular, whereas if the instruction comes last, the deepest needle is adjacent to the question. Those two layouts produce different curves from the same model. Fix the layout, state which one you used, and compare only within it.

One caution about interpretation: a synthetic needle is the easiest possible retrieval task. Real questions need synthesis across several passages, so a model that passes the needle test at 128k has demonstrated the floor, not the ceiling. That gap is what effective context length is about.

Designing around it

• Put the instruction last. If the task description precedes 100,000 tokens of context, it is competing with all of them. Restating it after the documents costs a few dozen tokens and is the single highest-leverage change available.
• Rank, then place at the edges. If you have reranked your retrieved passages, do not lay them out in rank order — put the strongest at the start and the second-strongest at the end, and let the weak ones occupy the middle where they will be under-attended anyway.
• Retrieve fewer, better passages. The paper’s most actionable implication is that more context is not monotonically better. A tighter reranker that returns five passages can beat one that returns fifty.
• Split the call. Two calls over halves of the corpus, each with a short context, followed by a reconciliation call, is often both more accurate and cheaper than one enormous call.