Building an Eval Set From Scratch

A small eval set built from your own traffic will answer questions a public benchmark cannot, because it is drawn from your distribution. What it cannot do is resolve a small difference — and the arithmetic for which differences it can resolve takes about five minutes.

Start from a decision, not a dataset

The failure mode of most first eval sets is that they were built to “measure quality”, which is not a question and therefore has no answer. Every useful eval set exists to support a specific decision that is about to be made:

• Can we ship this prompt change, or does it regress the hard cases?
• Is the cheaper model good enough for this step of the pipeline?
• Did last week’s provider-side model update break anything?
• Does the extraction pipeline handle scanned documents, or only clean ones?

Write the decision at the top of the file. It determines what goes in the set — a set for “can the cheap model do this” is weighted toward the boundary cases where it plausibly cannot — and it gives you grounds to refuse examples that do not bear on it. A set that measures everything measures nothing at a useful resolution.

What fifty examples can resolve

Suppose your grader gives each example a pass or a fail, and 45 of 50 pass. Your point estimate is 90%. The Wilson score interval — the one to use for small samples, rather than the normal approximation, which misbehaves near the boundaries — puts the 95% confidence interval at roughly [77%, 94%].

That is the number to internalise before building anything. A 50-example set that scores 90% is consistent with a true rate of 78% and with a true rate of 94%. It cannot tell you whether a change that moved the score from 88% to 92% did anything at all.

Two more results from the same arithmetic, both useful:

• The rule of three. If zero of n examples fail, the upper bound of the 95% interval on the failure rate is approximately 3/n. Fifty clean passes are consistent with a 6% real failure rate. A hundred are consistent with 3%. This is why “it passed all of them” is much weaker evidence than it feels like.
• Sizing for a difference. Resolving a 5-percentage-point difference between two independent samples at conventional power needs several hundred examples per side, not fifty. If that is the question, either get the examples or accept that you are making a judgement call and say so.

from statsmodels.stats.proportion import proportion_confint

# Wilson interval — correct behaviour at small n and near 0 or 1.
lo, hi = proportion_confint(45, 50, alpha=0.05, method="wilson")
print(f"{45/50:.0%}  95% CI [{lo:.0%}, {hi:.0%}]")   # 90%  95% CI [77%, 94%]

# Rule of three: zero failures in n trials.
for n in (30, 50, 100, 300):
    print(n, f"failure rate could still be up to ~{3/n:.1%}")

None of this is an argument against a small set. It is an argument for knowing which claims it supports. Fifty examples are excellent at catching a change that breaks a whole category, and useless at ranking two models that are close — and the majority of real decisions are the first kind. The statistics of eval results goes further into this.

Comparing two models is a different sum

When both systems are run on the same examples, you are no longer comparing two independent proportions, and the paired comparison is much more sensitive. The relevant quantity is the discordant pairs: the examples where one system passed and the other failed. McNemar’s test operates on exactly those two counts and ignores the examples both systems got right, which is what makes a fifty-example set more informative than the unpaired arithmetic suggests.

The practical consequence: always run every candidate against the identical set, keep per-example results rather than only the total, and look at the disagreements by hand. Twelve examples where the new model failed and the old one passed is a finding you can act on immediately, whatever the aggregate score did.

Where the fifty come from

A rough allocation that produces a set worth having. The proportions matter less than the fact that all four sources are represented.

Store expected outputs, not just inputs, and store why each example is in the set. Six months later, the note explaining that this example exists because of the March incident is what prevents somebody deleting it for being weird.

Grading, and the part you must not automate yet

Grade the first version by hand. Not as a matter of rigour — as the only way to discover that your rubric is ambiguous. You will find that two people disagree about a third of the borderline cases, and the disagreement is information about the task specification rather than about the model.

Then automate in the order that keeps the graders trustworthy: deterministic checks first (exact match, schema validity, a required field is present, the number is right), then a model judge only for what is left, and only after you have measured the judge’s agreement with your hand grades on the same examples. A judge that agrees with you 80% of the time is applying a 20% error rate to every result it produces, and that error is not random.

Keeping it honest over time

• Never put the eval set in a prompt. Few-shot examples drawn from the eval set turn it into training data and the score becomes meaningless. Decontamination applies to your own sets, not only to public benchmarks.
• Add every new failure. The set should grow by one example each time production surprises you. This is the discipline that turns it into an asset over a year.
• Re-grade a sample periodically. Labels rot when the product changes and yesterday’s correct answer becomes wrong. An eval set nobody has re-read in a year is measuring an old product.
• Keep a slice out of the loop. If you iterate on prompts against the set often enough, you are fitting to it. A held-out portion, opened rarely, is the only defence.