Skip to content
Multigrid

Statistical Significance Calculator for Evals

Decide whether one variant really beat another on a small, noisy eval set, using exact tests rather than large-sample approximations.

Everything on this page runs in your browser, and this one is a handful of counts rather than a payload — so all of it is in the URL, and the result you are looking at can be linked to exactly as it stands.

p-value
0.1153

Above your 5% threshold. That is not evidence the two are the same; it means this eval was not able to tell them apart. The interval on the difference below is the useful number.

Examples
100
Disagreements — the only ones that count
20
Old variant pass rate
76.0%
New variant pass rate
68.0%
Difference
-8.0%
95% interval on the difference
-16.6% to 0.6%
Exact McNemar p (binomial on the disagreements)
0.1153
Chi-square approximation with continuity correction
0.1175
  • notep = 0.115 is not significant at your threshold, and it is also not evidence that the two are the same — "we could not tell them apart with this eval" is the honest reading

    Expected: more examples if the difference matters. The interval on the difference below shows how much room is left

  • noteevery example where both variants agreed — 80 of 100 — carries no information about which is better and is correctly ignored by this test

    Expected: no action. This is why a paired eval needs far fewer examples than two independent ones

What this checked: this computes an exact test on the counts you typed and reports the normal approximation beside it so you can see when the two diverge — which is exactly when the eval was too small to use the approximation. Paired mode uses the exact binomial McNemar test on the disagreements only, with the continuity-corrected chi-square shown for comparison. Unpaired mode uses Fisher's exact test, with Wilson intervals on each rate and a Newcombe interval on the difference. It knows nothing about your eval set: whether the examples are independent, whether they are representative, whether the grader is consistent, or whether you ran this comparison twenty times and are looking at the one that came out significant. All four of those break a p-value more thoroughly than any choice of test.
What this assumes: each example is graded pass or fail, once, and the examples are independent of one another. Paired mode assumes both variants saw the same examples — if they did not, the p-value is far too small. Re-running one example several times to smooth out non-determinism reduces the noise on that example but does not increase n: the example is the unit, and 50 examples run ten times each is still a 50-example eval. The interval on the paired difference is a Wald interval, which is the standard choice and is optimistic when the disagreement count is very small.

Why the exact test is the headline here

Eval sets are small — fifty examples, a hundred, occasionally a thousand — and the normal approximations that everyone reaches for are derived for large samples. On a 40-example eval they can be out by a factor of two in either direction, which is enough to move a decision. The exact tests make no such assumption; they enumerate the possibilities. They used to be avoided because they were expensive to compute, which stopped being true decades ago. Both numbers are shown above: when they agree, the sample was big enough for either, and when they disagree, the exact one is right.

The three ways a significant result is still wrong

First, multiple comparisons. Test twenty prompt variants against one eval set at a 5% threshold and one of them will look significant by construction — that is what 5% means. Decide which comparison matters before you run it, or divide your threshold by the number of things you tried. Second, non-determinism: run the same variant twice and it will not score the same, so some of the difference you are testing is the model disagreeing with itself, and a p-value computed from a single run of each does not know that. Third, and the one that actually costs money: statistical significance is not importance. A one-point improvement proven beyond doubt on a thousand examples may still not be worth a migration, and the interval on the difference — not the p-value — is the number to take to that conversation.

The reasoning behind this

Related tools

Statistical Significance Calculator for Evals · Multigrid