Skip to content
Multigrid

Benchmarks, one at a time

What each named LLM benchmark actually contains, the exact arithmetic that turns answers into a score, and what that score does and does not license you to conclude.

Almost everything written about benchmarks is either a leaderboard screenshot or an argument that leaderboards are worthless. Both skip the part that would let you judge for yourself: what is actually in the test. MMLU is fourteen thousand four-option questions scraped from practice exams. GPQA Diamond is 198 questions written by PhD students. HumanEval is 164 hand-written Python functions. Once you know that, a great deal of the commentary answers itself.

Each page here takes one benchmark and describes four things: how it was constructed, how it is scored — with the arithmetic worked through where the metric is not obvious — what a score on it can support, and what it is a bad proxy for. No model scores appear anywhere in this cluster. They change weekly, they are the easiest thing on a page like this to get wrong, and they are not what these pages are for.

The argument that public benchmarks do not predict your results is made elsewhere and made well: why benchmark rank does not transfer to your task and what contamination does to a leaderboard. These pages assume you have read that and want to know what the tests contain. The last two are the practical ones: how to run a public benchmark yourself, and the eight fields that must accompany any score you publish.

MMLU: What It Tests and What a Score Means

MMLU is about 14,000 four-option exam questions across 57 subjects. What it measures, its label-error problem, and why a two-point gap near the top is noise.

9 min read

GPQA and Google-Proof Questions

GPQA is a few hundred expert-written science questions designed so that skilled non-experts with a search engine still fail them. What that design buys, and what it costs.

9 min read

HumanEval and the pass@k Metric

164 hand-written Python problems graded by unit tests. The unbiased pass@k estimator worked through with real arithmetic, and why pass@1 and pass@10 answer different questions.

10 min read

ARC-AGI: The Test Built to Resist Memorisation

A grid-puzzle benchmark whose design goal is that no amount of training data helps. How the tasks are built, how the private set works, and what each version changed.

9 min read

MATH, AIME and Competition Maths Benchmarks

Competition maths sets grade the final answer, not the working. What that hides, why the answer-equivalence checker is a scoring decision, and why AIME numbers are so noisy.

10 min read

How Arena Rankings and Elo Are Computed

Arena ratings come from pairwise votes fitted with a Bradley-Terry model. The formula, the arithmetic that turns a rating gap into a win probability, and what the number cannot mean.

10 min read

MTEB and How Embedding Models Are Ranked

MTEB averages eight kinds of task scored by five different metrics. What each task type measures, why the headline average is not a meaningful number, and which column to read instead.

9 min read

BEIR and Zero-Shot Retrieval Evaluation

BEIR tests retrieval models on domains they were not trained on, scored with nDCG@10. The metric worked through by hand, and the judgement-pool problem that biases it.

10 min read

HELM and Multi-Metric Evaluation

HELM measures seven things about every scenario, not one. What calibration, robustness, fairness, bias, toxicity and efficiency each mean operationally, and how they are computed.

10 min read

BIG-bench, BBH and the Hard Subsets

BIG-bench is 200-odd community-contributed tasks; BBH is the subset models failed. Why filtering on failure selects for broken tasks as well as hard ones.

9 min read

LongBench, RULER and Testing Long Context

Needle tests prove a model can find a fact in a haystack. RULER and LongBench test whether it can use one. What each design can and cannot establish about a context window.

10 min read

Tool-Use and Function-Calling Benchmarks

How function-calling benchmarks decide a call was correct — AST comparison, execution, or final state — and which production failures they systematically do not test.

10 min read

TruthfulQA and Measuring Honest Answers

TruthfulQA tests whether a model repeats common human misconceptions. How its questions were selected, why truthfulness and informativeness must both be reported, and the standing critique.

9 min read

Safety and Refusal Benchmarks

Safety suites measure two opposite errors: complying with harmful requests and refusing harmless ones. Which suites measure which, how attack success is judged, and why both numbers are required.

10 min read

MMMU, DocVQA and ChartQA

Three multimodal benchmarks that demand three different things. Their image sources, their answer formats, and the two forgiving metrics — ANLS and relaxed accuracy — that make scores look better than exact match would.

10 min read

LibriSpeech, FLEURS and Common Voice

The standard ASR benchmarks are read speech. Why word error rate rises sharply on spontaneous speech, and why text normalisation moves a WER more than the model does.

10 min read

WebArena, GAIA and Agentic Benchmarks

Agentic benchmarks score the end state of an environment rather than a string. What that buys, and why the resulting number belongs to a model-plus-scaffold pair rather than to a model.

10 min read

Cost- and Latency-Aware Leaderboards

How score-per-dollar and score-per-second are computed, why the ratio is the wrong summary, and what a Pareto frontier or a cost-to-fixed-quality curve shows instead.

10 min read

Running a Public Benchmark Yourself

How to run a public evaluation harness so the number means something: the nine settings that move a score, and a small harness that measures the difference between two configurations.

12 min read

Reporting a Benchmark Result Honestly

Eight fields that must accompany any published score, a machine-readable manifest to record them in, and five questions to ask of somebody else's number.

12 min read

Other topics