The Benchmark Scores People Quote, and What Must Be Attached
10 min read · updated August 4, 2026
A benchmark score is not a property of a model. It is the result of running a specific version of a specific dataset through a specific harness with specific decoding settings, and changing any one of those moves the number by more than the gaps people cite as evidence.
A score alone is not a fact
“Model X scores 87 on MMLU” is roughly as informative as “the car did 40”. Forty what, over what distance, with what load, measured how? The same model evaluated by two competent teams on the same benchmark routinely produces different numbers, and the difference is not a mistake by either — it is the settings.
This matters commercially because benchmark scores are used to justify purchase decisions and marketing claims, and it matters technically because the same score gap that looks decisive in a table is often smaller than the benchmark’s own noise floor.
The eight settings a score must carry
| Setting | Description |
|---|---|
| dataset version | MMLU and MMLU-Pro are different benchmarks. SWE-bench and SWE-bench Verified are different benchmarks. The suffix is not a detail. |
| split and subset | Test, validation, or a sampled subset. Reporting on a 200-item sample of a 14,000-item benchmark is legitimate and must be stated, because the noise floor changes completely. |
| shots | Zero-shot, five-shot, or few-shot with a chosen exemplar set. The exemplars themselves change the result, and which ones were used is almost never published. |
| prompt format | Whether the options were presented as a list, whether the model answered with a letter or with text, whether a system prompt was used. Multiple-choice scoring by comparing per-option log-likelihoods and scoring by parsing a generated letter are different measurements of the same model. |
| reasoning | Chain-of-thought allowed or not, and if a reasoning model, what effort setting. This moves scores on reasoning-heavy benchmarks by large margins and is the single most common unstated variable. |
| decoding parameters | Temperature, top-p, and maximum output tokens. Greedy decoding and temperature 0.8 are different experiments, and a truncated answer scores zero regardless of whether the model knew. |
| scoring rule | Exact match, normalised match, an extraction regex, or an LLM judge. An LLM judge introduces a second model’s biases into the measurement — see judge bias. |
| harness and version | EleutherAI's lm-evaluation-harness, HELM, a vendor's internal script. Harness versions change prompt templates between releases, which silently changes historical comparability. |
A score reported with all eight is reproducible. A score reported with none of them is a marketing figure, and benchmark marketing covers what is usually being done with it.
When a gap is noise, derived
Benchmarks are finite samples of questions, so a score has a sampling error, and that error is calculable from the item count alone. Almost no leaderboard shows it.
For a benchmark scored as a proportion correct over n independent items: standard error = sqrt( p * (1 - p) / n ) 95% confidence interval ~= p +/- 1.96 * SE Worked at the worst case p = 0.5, which maximises the variance: n = 200: SE = sqrt(0.25/200) = 0.0354 -> 95% CI = +/- 6.9 points n = 500: SE = sqrt(0.25/500) = 0.0224 -> 95% CI = +/- 4.4 points n = 5,000: SE = sqrt(0.25/5000) = 0.0071 -> 95% CI = +/- 1.4 points n = 14,000: SE = sqrt(0.25/14000) = 0.0042 -> 95% CI = +/- 0.8 points Comparing two models needs the standard error of the *difference*. For independent runs: SE_diff = sqrt( SE_a^2 + SE_b^2 ) On a 500-item benchmark with both models near 50%: SE_diff = sqrt(0.0224^2 + 0.0224^2) = 0.0317 95% CI on the difference = +/- 6.2 points So on a 500-item benchmark, a 3-point lead is not evidence of anything.
Two refinements make the real situation slightly better and substantially worse in different directions. Better: when both models are evaluated on the same items, a paired test is far more sensitive than the independent comparison above, and any serious evaluation should use one. Worse: the formula assumes items are independent, and benchmark items are clustered by topic, so the effective sample size is smaller than the item count.
The statistics of this are covered further in evaluation statistics. The practical rule: if a leaderboard does not publish item counts and intervals, compute the interval yourself from the item count before believing any ranking it implies.
pass@k is not one number either
Code benchmarks report pass@k, the probability that at least one of k samples passes the tests. It is an estimator, not a measurement, and the way it is estimated changes the answer.
Unbiased estimator (Chen et al., 2021, the HumanEval paper), from n samples of which c pass: pass@k = 1 - C(n - c, k) / C(n, k) Worked for one problem, n = 10 samples, c = 2 passing, k = 1: 1 - C(8,1)/C(10,1) = 1 - 8/10 = 0.20 And for k = 5: 1 - C(8,5)/C(10,5) = 1 - 56/252 = 0.778 Same model, same problem, same samples. pass@1 is 20% and pass@5 is 78%, and both are correctly labelled "the HumanEval score".
Two further wrinkles. A greedy pass@1 at temperature 0 and an estimated pass@1 from many samples at temperature 0.8 are different quantities and the second is usually higher. And the tests themselves are the scoring rule: a solution that is correct but formats output differently fails, which is a property of the benchmark rather than of the model. See coding benchmarks.
The benchmarks worth knowing by name
These are the ones you will meet in model announcements. Each is named with its originating publication so you can read what it actually measures rather than what a table implies.
- MMLU (Hendrycks et al., 2020) — multiple-choice questions across 57 academic subjects. The default general-knowledge number for years, now saturated at the top end and superseded by MMLU-Pro for harder discrimination.
- GPQA (Rein et al., 2023) — graduate-level science questions written to be resistant to web search, which is the design decision that makes it interesting.
- GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021) — grade-school and competition mathematics. Both heavily affected by whether chain-of-thought is allowed.
- HumanEval (Chen et al., 2021) — small self-contained Python functions with unit tests. Widely reported and widely acknowledged as too easy and too narrow to represent software work.
- SWE-bench (Jimenez et al., 2023) — real GitHub issues in real repositories, resolved by producing a patch that passes the project’s own tests. Far more representative, and its Verified subset (a human-validated selection released in 2024) is a separate benchmark whose scores are not comparable to the full set.
- HELM (Stanford CRFM) — not a benchmark but a standardised evaluation framework that runs many benchmarks with documented, fixed prompts. When you want comparability rather than a headline, this is the shape of thing to look for.
- Pairwise human-preference arenas — crowd voting between anonymised model responses, aggregated into a rating. Real information about which outputs people prefer, and not a capability measurement: it rewards formatting, length and tone alongside correctness. Read it as a preference ranking and nothing more.
Contamination and label error
Two documented problems undermine benchmark scores in ways no amount of careful reporting fixes.
Contamination. If benchmark items appeared in training data, the score measures memorisation. The strongest public evidence for this comes from building a fresh benchmark to match an existing one’s distribution and comparing: Scale AI’s 2024 work constructing GSM1k as a like-for-like replacement for GSM8K found performance drops for some models and not others, which is exactly the signature contamination predicts. See benchmark contamination.
Label error. Benchmarks contain wrong answers. The 2024 paper Are We Done with MMLU? (Gema et al.) re-annotated MMLU questions and documented errors concentrated in particular subsets, meaning a portion of the remaining headroom on that benchmark is unreachable by a correct model. When scores approach the ceiling, the label error rate becomes the ceiling.
What to do instead
- Use public benchmarks to shortlist, never to decide. They are evidence that a model is in the right class, not that it will do your task.
- Build a small evaluation set from your own data — a hundred real inputs with known-good outputs beats any public leaderboard for your decision. See building an eval set.
- Compute the confidence interval on your own set with the formula above before concluding that one model beat another. A hundred items gives roughly plus or minus ten points, which is sobering and worth knowing before a migration.
- Evaluate both models on the same items and use a paired comparison. The gain in sensitivity is large and free.
- Re-run when the model changes. Scores are attached to a snapshot, and model lifespans are short enough that a benchmark result outlives its subject.