Telling a Retrieval Bug From a Model Bug in RAG

When a retrieval-augmented system answers wrongly, the visible artefact is a wrong sentence from a model, so the model is what gets debugged. In most cases the model was given the wrong documents and answered them faithfully. Three measurements settle which it is, and they take an afternoon rather than two weeks.

Why the model gets blamed

The failure surfaces at the last stage of the pipeline. The user sees a confident wrong answer; the team sees hallucination; the response is prompt engineering, then a bigger model, then a fine-tune. All three can produce a small improvement, which reinforces the diagnosis, because a stronger model is somewhat better at answering “I do not know” when the context is useless.

Meanwhile the actual fault — a chunker that split the table from its header, a metadata filter silently excluding the right document, an index three weeks stale — is untouched, and every improvement is bounded by it. The asymmetry is stark: retrieval faults are cheap to fix and hard to see, generation faults are easy to see and expensive to fix.

Three measurements that localise the fault

You need a set of question-and-answer pairs where you know which document contains the answer. Fifty is enough. Everything below is computed from that set.

The third is the decisive one and is almost never run, because it requires deliberately disabling the system under test. It converts an argument into arithmetic: if oracle-context accuracy is 92% and end-to-end accuracy is 61%, then 31 points of the gap are retrieval and 8 are generation, and you know exactly which project to fund.

The procedure

1. Log the assembled context, not the prompt template. For twenty failing cases, capture the exact chunk text passed to the model, with chunk ids and scores. Most teams cannot do this on demand, and that gap is the actual root cause of the two weeks.
2. Read them. Before any measurement, read twenty failing contexts end to end. In practice this alone identifies the fault class in the majority of cases, and it costs an hour.
3. Compute recall@k. If the answer-bearing document is missing from the context in a meaningful share of failures, stop. The problem is retrieval and the model is irrelevant.
4. Run the oracle-context test. Paste the correct passage in by hand. High accuracy here plus low end-to-end accuracy proves retrieval; low accuracy here proves generation.
5. Only now change something. And change one thing, because the pipeline has enough interacting stages that two simultaneous changes produce an uninterpretable result.

Seven retrieval faults that look like model faults

Each of these produces a confident, fluent, wrong answer — the exact signature of hallucination.

• The chunk boundary cut the answer in half. A fixed window split a table from its header row, or a definition from its term. The retrieved chunk is topically right and factually useless. See chunking strategies.
• A metadata filter is excluding the right document. A filter on tenant, date or document type that is subtly wrong removes the answer before ranking, and there is no error — the query simply returns the next best thing. Metadata filtering is where this lives.
• The index is stale. The document was updated and the index was not. The system answers from a version that no longer exists, which reads exactly like a model recalling outdated training data. See index freshness.
• The embedding model changed and the index did not. Queries embedded with a new model against vectors from an old one return results that are nearly random while remaining plausible.
• The wrong text was embedded. Boilerplate, navigation chrome or an entire document’s worth of footer text dominating the vector. Retrieval returns documents that match the boilerplate rather than the content.
• k is too small for the question type. Comparative and aggregative questions need several documents; a k tuned for lookup questions structurally cannot answer them, and the model answers from the subset it has.
• The answer is in the middle of a long context. With a large k, relevant material placed centrally is attended to less reliably than material at either end — lost in the middle. Retrieval succeeded and the answer was still not used.

The last one is the awkward case: it is a retrieval-configuration fault with a generation-side mechanism, and it is why re-ranking to a small k often beats increasing k.

When it really is generation

Sometimes the oracle test comes back low, and then the fault genuinely is in generation. The common forms:

• The model answers from parametric knowledge over the context. Detectable by planting a deliberately wrong fact in the context and seeing which one comes out. If the model prefers what it knows, the instruction to prefer the context is not strong enough — or the context is not clearly delimited from the instruction.
• It will not say “not in the documents”. Abstention has to be an explicitly allowed and demonstrated output, with an example. See abstention.
• Citations point at the wrong chunk. The answer is right and the attribution is invented, which is worse than a wrong answer because it survives spot-checking. Citations in RAG covers enforcing them structurally.
• Conflicting sources. Two retrieved documents disagree and the model silently picks one. This needs a product decision, not a prompt fix.

The wider set of generation-side failures is catalogued in RAG generation failures.

Keeping the diagnosis cheap

Everything above is expensive exactly once, and then nearly free forever, if three things exist:

1. Every answer records its retrieved chunk ids and scores. Not the text — the ids, which are small — plus a way to fetch the text as it was at that moment.
2. A labelled set of fifty question-document pairs is kept in the repository and grows whenever a real failure is diagnosed. This is the asset; RAG evaluation is what you do with it.
3. The oracle-context run is a command, not a project. One flag that bypasses retrieval and substitutes the labelled passage. Ten lines, and it is the single highest-leverage thing on this page.