Contextual Retrieval: Adding Context to Chunks Before Embedding

The most common retrieval failure is a chunk that does not say what it is about. Contextual retrieval fixes it in the least clever way available: before embedding, ask a model to write the missing sentence and stick it on the front.

The orphaned chunk, again

Here is a chunk from a quarterly report, split at 500 tokens:

"The company's revenue grew by 3% over the previous quarter."

Which company? Which quarter? Compared with what? The chunk is perfectly clear in context and useless out of it, and the index has only the chunk. A query asking about ACME’s Q2 2023 revenue has no lexical overlap with it beyond “revenue”, and its embedding sits in the generic vicinity of every sentence about revenue growth in the corpus.

Every structural remedy from the chunking page — prepending heading paths, parent-child retrieval, semantic splitting — attacks this problem from the document’s side. Contextual retrieval attacks it from the model’s side, and it works on documents with no usable structure at all, which is most of them.

The method

For each chunk, send the model the whole document plus that chunk, and ask for a short situating sentence. Prepend the result to the chunk text and embed the combination. The chunk above becomes:

"This chunk is from ACME Corp's Q2 2023 SEC filing, in the section
discussing quarter-over-quarter financial performance against Q1 2023.
The company's revenue grew by 3% over the previous quarter."

Now the entity, the period, the document type and the comparison basis are all in the embedded text, and all of them are searchable — by the dense arm because the vector has moved to the right region, and by the lexical arm because “ACME” and “Q2 2023” are now literal tokens in the indexed string. The technique improves both halves of a hybrid retriever, which is unusual.

CONTEXTUALISE = """<document>
{doc}
</document>

Here is a chunk from that document:

<chunk>
{chunk}
</chunk>

Write one or two short sentences situating this chunk within the
document, to improve search retrieval of the chunk. Name the entities,
dates and section it belongs to. Answer with the situating text only
and nothing else."""

def contextualise(doc, chunk):
    prefix = small_model(CONTEXTUALISE.format(doc=doc, chunk=chunk),
                         cache_prefix=True)   # see the cost section
    return prefix + "\n\n" + chunk

Store the contextualised text for the index and the original text for display. The generated prefix is a retrieval aid, not content — if you show it to users, you are showing them a model’s paraphrase of their document and inviting every hallucination in it to be read as source text.

The published numbers, and whose they are

The technique was popularised by Anthropic’s Introducing Contextual Retrieval write-up (September 2024), which is also where the widely-quoted figures come from. Their reported results, on their evaluation, measuring the top-20 retrieval failure rate:

Those are Anthropic’s numbers on Anthropic’s evaluation corpora, not a general law, and they should be read as evidence that the direction is real rather than as a prediction for your corpus. Two things about the shape of the result are worth carrying away regardless of magnitude. The gain is largest where chunks are most context-dependent, so a corpus of self-contained records will see less than a corpus of narrative documents. And the improvements stack with reranking rather than overlapping with it, which makes sense — one fixes what enters the candidate set, the other fixes its order.

Why the ingest pass is affordable

The obvious objection is the cost. One generation call per chunk, with the whole document in the prompt each time, sounds catastrophic: a 50-chunk document means sending that document 50 times.

Prompt caching is what makes it viable, and the arithmetic is worth doing yourself. The document sits at the front of the prompt and is byte-identical across all 50 calls, so it is a cacheable prefix. You pay full price to write it to the cache once, then a small fraction of the input rate on the 49 subsequent reads.

Document D tokens, C chunks, prefix output P tokens per chunk.

no caching   = C * (D * Gin)                    + C * P * Gout
with caching = D * Gwrite + (C-1) * D * Gread   + C * P * Gout

D = 25,000   C = 50   P = 60
uncached input volume  = 1.25 M tokens per document
cached   input volume  = 25 k at write rate + 1.225 M at read rate

If the read rate is a tenth of the input rate, the input cost falls
by roughly 9x. Order the calls per document so the cache stays warm.

Anthropic’s post put the one-time cost at approximately $1.02 per million document tokens using their cheapest model with prompt caching. Whatever your provider’s equivalent, the structure of the calculation is the same, and the operational requirement is identical: process all chunks of one document consecutively, before the cache entry expires. Interleaving documents across workers destroys the entire saving, and it is an easy thing to do by accident when parallelising an ingest job.

When it is not worth it

• Self-contained chunks. Product records, FAQ pairs, support tickets — anything where the retrievable unit already names its own subject. There is no missing context to add.
• Documents that exceed the context window. The method needs the whole document in the prompt. For very long sources, contextualise against a section rather than the whole thing, and accept a narrower prefix.
• Rapidly changing corpora. The prefix is regenerated whenever the chunk changes, and if the document changes the prefixes of unchanged chunks are arguably stale too. High churn turns a one-time cost into a running one.
• Before the cheap fixes. Prepending the heading path costs nothing and captures a good share of the same benefit on structured documents. Do that first and measure what is left.

That last point deserves emphasis before you spend anything. The technique’s benefit is bounded by how much context is missing from your chunks, and a deterministic prefix — document title, heading path, effective date, author, document type — recovers a substantial share of it for the cost of a string concatenation. Build that first, measure recall, and then decide whether an LLM-written prefix adds enough on top to justify a generation pass over the entire corpus. On well-structured documentation the remaining gap is often small; on transcripts, contracts and scanned reports with no usable structure it is where the whole gain lives.

Whichever version you build, store the prefix in its own column rather than concatenating it into the chunk text at ingest. You will want to regenerate prefixes without re-chunking, to compare retrieval with and without them on the same index, and — when a prefix turns out to contain a hallucinated date — to fix it without touching the source text. Joining the two strings at embed time instead of at ingest time costs nothing and keeps all three of those cheap.