Confidence Intervals on a Benchmark Score

A model scoring 72% on a 200-example benchmark has a 95% confidence interval of roughly 66% to 78%. That is a twelve-point window, which is wider than almost every difference reported between models, and this page computes it from scratch so the claim is arithmetic rather than an opinion.

The interval on one score

A benchmark score is a proportion: k correct out of n items. The standard error of a proportion is:

SE = sqrt( p * (1 - p) / n )

Model A: 144 correct out of 200, p = 0.72

  SE = sqrt(0.72 * 0.28 / 200)
     = sqrt(0.2016 / 200)
     = sqrt(0.001008)
     = 0.031749
     = 3.17 percentage points

95% interval, Wald:  p +/- 1.96 * SE
                  =  0.72 +/- 0.0622
                  =  [0.6578, 0.7822]
                  =  [65.8%, 78.2%]

A twelve-and-a-half-point window. That is what 200 examples buys, and it does not depend on the model, the task, or how carefully the benchmark was built — only on n.

Half-width of the 95% interval at p = 0.72:

  n =    50   ->  +/- 12.4 points
  n =   200   ->  +/-  6.2 points
  n =   500   ->  +/-  3.9 points
  n = 1,000   ->  +/-  2.8 points
  n = 5,000   ->  +/-  1.2 points
  n = 20,000  ->  +/-  0.6 points

The width falls as 1/sqrt(n). Halving it costs 4x the data.

A 200-item benchmark cannot report 1.2 points

Before any statistics, an arithmetic observation that rules out a large class of reported results. On n items, every possible score is a multiple of 100/n points.

n = 200  ->  resolution 0.5 points
             possible scores: ..., 71.5, 72.0, 72.5, 73.0, ...

A 1.2-point difference is not reachable.
The differences you can observe are 0.5, 1.0, 1.5, 2.0, ...

n = 500  ->  resolution 0.2 points
n = 1000 ->  resolution 0.1 points

So a reported 1.2-point gap came from a benchmark of at least 500 items, or from averaging across sub-tasks, or from multiple runs, or from a number that was rounded on its way to the table. Each of those is a different measurement with different error properties, and none of them is stated. It is worth asking which, and the question is usually more revealing than the gap.

The paired comparison, which is the right test

Two models evaluated on the same 200 items are not two independent samples. Most items are easy for both or hard for both, and that shared difficulty is not evidence about either model. The correct analysis looks only at items where they disagreed.

Cross-tabulate the 200 items:

                       B correct   B wrong
  A correct               133         11      -> A total 144 (72.0%)
  A wrong                  13         43      -> A wrong  56
                          ---        ---
  B total                 146         54         B total 146 (73.0%)

Concordant (both same):  133 + 43 = 176   -- carry no information
Discordant:               11 + 13 =  24   -- the entire evidence base

b = 11  (A right, B wrong)
c = 13  (B right, A wrong)

Twenty-four items decide it. The question becomes: if the two models were equally good, would 13 versus 11 be surprising? It is a coin flip repeated 24 times, and 13 heads is not surprising at all.

Difference: (c - b) / n = 2 / 200 = 0.010 = 1.0 point

SE of the paired difference:

  SE = sqrt( b + c - (b - c)^2 / n ) / n
     = sqrt( 24 - 4/200 ) / 200
     = sqrt( 23.98 ) / 200
     = 4.8969 / 200
     = 0.024485 = 2.45 points

95% CI: 1.0 +/- 1.96 * 2.45 = 1.0 +/- 4.80
                            = [-3.80, +5.80]

Contains zero. Not significant.

McNemar's test with continuity correction:
  chi2 = (|b - c| - 1)^2 / (b + c)
       = (2 - 1)^2 / 24
       = 0.0417,  1 degree of freedom
  p = 0.84

A p-value of 0.84. If the two models were identical, you would see a gap this large or larger 84% of the time. There is no evidence here of any difference at all.

It is worth noting that the paired test is the generous one. Treating the two scores as independent samples gives a wider interval:

Unpaired SE of the difference:

  sqrt( 0.72*0.28/200 + 0.73*0.27/200 )
  = sqrt( 0.001008 + 0.000986 )
  = sqrt( 0.001994 )
  = 0.044648 = 4.46 points

95% CI: 1.0 +/- 8.75 points

Paired:    +/- 4.80
Unpaired:  +/- 8.75

Pairing nearly halves the interval, and still
cannot make a 1-point win significant.

How many examples a one-point win would need

Turn the question round. To detect a true 1-point difference with 80% power at the 5% level, how large must the benchmark be?

Unpaired, two proportions near 0.72:

  n per arm = (z_alpha/2 + z_beta)^2 * 2p(1-p) / delta^2

  z_0.025 = 1.960,  z_0.20 = 0.8416
  (1.960 + 0.8416)^2 = 7.849
  2 * 0.72 * 0.28    = 0.4032
  delta = 0.01,  delta^2 = 0.0001

  n = 7.849 * 0.4032 / 0.0001 = 31,646

Paired (McNemar), with a 12% discordance rate:

  n ~= (1.96*sqrt(0.12) + 0.8416*sqrt(0.12 - 0.0001))^2 / 0.0001
     = (0.678964 + 0.291418)^2 / 0.0001
     = 0.941641 / 0.0001
     = 9,416

Roughly 9,400 paired examples to establish a one-point difference, and around 32,000 per arm without pairing. Public benchmarks are typically a few hundred to a few thousand items. The conclusion is not that benchmarks are useless — it is that they resolve differences of five to ten points and not differences of one, and a leaderboard ordering models by tenths of a point is ordering them by noise.

The variance nobody puts an interval on

Sampling error is only the source of variance that is easy to compute. Three others are usually larger and almost never reported:

• Run-to-run variance. At any temperature above 0 the same model on the same benchmark produces different answers on different runs, and even temperature 0 is not fully deterministic. Run the benchmark five times, take the standard deviation of the five scores, and you have measured this directly — it costs five times the evaluation budget and is the single most informative number you can add to a report.
• Prompt-format variance. The same benchmark with options labelled A/B/C/D instead of 1/2/3/4, or with the instruction moved from the start to the end, moves scores by more than most model differences. This sensitivity is well documented and rarely controlled for.
• Grading variance. Where a model grades the outputs, the grader has its own error rate, its own biases about length and style, and its own run-to-run variance on top of everything above.
• Contamination. Not variance but bias, and unbounded. If the test items are in the training data the score measures memorisation, and no interval describes that.

The practical instruction is short: report the interval, run the evaluation more than once, and put both numbers next to the score. If you would rather not derive the formula each time, the bootstrap gives the same interval by resampling the per-item results, works for any metric, and is twenty lines.

Reading a leaderboard after this

1. Find n. If the item count is not published, no interval can be computed and no comparison on the table means anything. This is the first thing to look for and it is missing surprisingly often.
2. Compute 1.96 * sqrt(p(1-p)/n) in your head, roughly. At p near 0.7, the half-width is about 90/sqrt(n) points. For n = 200 that is 6.4; for n = 1000, 2.8.
3. Ignore every gap smaller than that. Not “treat with caution” — ignore. Two models within the interval are tied on the evidence available.
4. Prefer a paired comparison on your own data. Two hundred examples from your actual traffic, both models run on all two hundred, cross-tabulated as above, will tell you more about your application than any public leaderboard. That is also the cheapest evaluation set you will ever build.