Skip to content
Multigrid

Sample Size Calculator for Evals

Work out how many examples an eval set needs before a difference between two prompts or two models means anything.

Examples needed in the eval set
434

Both variants run on all 434. This is a paired design, so that is 434 examples, not 868.

z for the confidence level (two-sided)
1.960
z for the power (one-sided)
0.842
Disagreements favouring the new variant (π₁₀)
13.00%
Disagreements favouring the old one (π₀₁)
7.00%
Difference being detected (δ = π₁₀ − π₀₁)
6.00%
Expected disagreements at that sample size
87
What this checked: this computes a sample size from the numbers you typed, using the normal approximation in both modes. The paired mode uses the standard McNemar sizing, n = [zα/2√πd + zβ√(πd− δ²)]² / δ², where πd is the disagreement rate. The single-rate mode uses n = z²p(1−p)/e², with the finite-population correction applied when you give a pool size. It does not know your eval set, your variance across repeated runs, or whether your examples are independent — and it cannot tell you whether the difference you are sizing for is one worth having.
What this assumes: examples are independent and each is graded pass or fail. The paired mode assumes both variants are run on the same examples, which is what makes it far cheaper than two independent samples — if you cannot do that, this number is too small. Re-running one example several times to average out non-determinism reduces the noise on that example but does not reduce n: the example is the sampling unit, and 50 examples run 10 times each is still a 50-example eval. Normal approximations get shaky below about 20 expected disagreements, so treat a result under 100 as a floor rather than a target.

Everything on this page runs in your browser. Nothing you paste is uploaded, logged, or put in the URL — only the settings above the input are, so a configured tool can be linked to.

Why the paired number is so much smaller

The instinct with two model versions is to score each on the eval set and compare the two accuracies with a two-proportion test. That throws away the strongest fact you have: both variants saw the same examples. Every example where they agree — both right, or both wrong — tells you nothing about which is better, and a paired test simply ignores them. What is left is the disagreements, and the question shrinks to whether they split evenly.

That is why the disagreement rate is the field that moves this calculator most. Two variants that disagree on 2% of examples need an enormous eval to separate; two that disagree on 30% need a small one, even if their headline accuracies are close. If you do not know your disagreement rate, run both on whatever you have and count — it is a cheaper measurement than the one you are sizing.

The number this cannot give you

Statistical significance is not importance. An eval large enough to prove a 1-point difference is real does not tell you a 1-point difference is worth a migration, and the reverse trap is worse: an eval set of 40 hand-written examples will happily show a 10-point swing that is entirely noise. If your result sits near the boundary, the honest move is more examples, not a different test. And a difference you found by testing twenty variants against one eval set will not survive contact with production — that is the multiple-comparisons problem, and no sample size fixes it.

The reasoning behind this

Related tools

Sample Size Calculator for Evals · Multigrid