Evaluating a RAG System: Retrieval and Generation Separately

A RAG system is two systems in a trench coat, and almost every evaluation setup scores the coat. When the number goes down you learn that something got worse, which you already knew from the complaint that prompted you to look.

The one-number trap

End-to-end answer quality is the metric that matters to the user and the least useful metric to an engineer, because it is a product of two independent probabilities. If retrieval finds the right chunk 80% of the time and generation uses it correctly 90% of the time, end-to-end correctness is about 72%. Improving the second factor to 95% moves the product to 76%; improving the first to 90% moves it to 81%. Same effort, different payoff, and the end-to-end number tells you nothing about which one to spend it on.

Worse, the two failures can cancel in a way that flatters you. A model that answers correctly from its own weights while the retrieval returned rubbish scores as a success and is a latent disaster: the day the question moves to your proprietary domain, it fails silently and the evaluation never warned you.

Four cells, four different bugs

Score each evaluation question on two binary axes: did retrieval return a chunk that contains the answer, and did the generated answer match the reference. Four cells, four unrelated action items.

Building this requires an evaluation set where each question is labelled with the chunk id that should answer it, not just the expected answer text. That labelling is the expensive part of RAG evaluation and the part that is worth doing by hand.

Measuring the retrieval half

Retrieval is ordinary information retrieval and the metrics are fifty years old. Use two, for different decisions:

• Recall@k — the fraction of questions whose relevant chunk appears anywhere in the top k. This is the ceiling on everything downstream, and it is the number to watch when you change chunking, the embedding model, or the hybrid weighting. Plot it at k = 5, 10, 25, 50; the shape tells you where to set the reranker candidate count.
• MRR or nDCG@k — how high up the relevant chunk lands. This is the number a reranker moves. Recall can be flat while MRR improves substantially, and that improvement is real, because position within the context affects how the model uses it.

Both are computed offline against fixed labels, cost nothing to run, and are deterministic. Run them in CI. A retrieval regression from a changed splitter is exactly the kind of thing that ships unnoticed and is trivially caught here.

Measuring the generation half

Here you are stuck with fuzzier instruments, and the useful move is to ask narrow questions rather than “is this a good answer”. The RAGAS framework (Es et al., arXiv:2309.15217) formalises a set of these; the ones worth implementing even if you use nothing else:

• Faithfulness / groundedness. Decompose the answer into claims and ask, per claim, whether the provided context supports it. This is a judgement a model makes reliably, because it is a small verification task rather than an open-ended assessment.
• Context relevance. What proportion of the retrieved text was actually needed? Low values mean you are paying for tokens that do nothing and adding distractors.
• Answer relevance. Does the answer address the question that was asked, as opposed to a related one? Catches the case where the model summarises the retrieved chunk instead of answering.

If you use a model as a judge, hold the judge fixed when comparing two systems, and periodically check it against human labels on a subsample. A judge that drifts turns your entire time series into noise, and a judge that scores its own family’s outputs is a known source of bias.

How many questions you need

This is the part that is almost always missing, and it decides whether your evaluation can detect anything at all. Recall@k is a proportion, so the standard error on an estimate p from n questions is the usual one:

se = sqrt(p * (1 - p) / n)      95% interval ≈ p ± 1.96 * se

p = 0.80,  n = 50    ->  se = 0.057   ->  ±11.1 points
p = 0.80,  n = 100   ->  se = 0.040   ->  ± 7.8 points
p = 0.80,  n = 300   ->  se = 0.023   ->  ± 4.5 points
p = 0.80,  n = 1000  ->  se = 0.013   ->  ± 2.5 points

So a fifty-question evaluation set measures recall to within about eleven points. A change that genuinely improves recall by five points is invisible; a run-to-run wobble of eight points is expected and means nothing. Teams celebrate and revert on that noise constantly.

Two mitigations. First, paired comparison: run both systems on the same questions and count only the questions where they disagree. Discordant-pair tests are far more sensitive than comparing two independent proportions, because the shared variance from question difficulty cancels. Second, build the set incrementally — every real failure a user reports becomes a labelled question, which both grows n and biases the set toward the cases that actually break.

A hundred hand-labelled questions is a realistic first target and enough to see large changes. Three hundred is enough to run a pipeline on. Fifty is a smoke test, and should be described as one.

Building the set is less painful than it sounds if you invert the usual direction. Rather than writing questions and hunting for the chunk that answers each, sample chunks from the corpus and ask a model to write the question each one answers. That gives you the label for free — the chunk is known by construction — and a human only has to review and discard, which is perhaps ten times faster than authoring. The generated questions will be too easy and too closely paraphrased from the chunk, so treat that set as a floor, and mix in real user questions, which are the ones that break things.

One structural warning about the labels. A question often has several acceptable supporting chunks, and a set that records only one will under-report recall whenever your retriever finds a different valid one. Allow a list of acceptable chunk ids per question and score a hit if any of them appears. Systems have been declared regressions on the strength of a label set that simply had not enumerated the alternatives.