MMLU, GPQA, SWE-bench: What Each Actually Measures

Model cards quote these names as if everyone knows what they contain. Most people know the acronym and the number. Here is the construction of each one, and the specific conclusion each does not support.

Three shapes of benchmark

Before the individual tests, the taxonomy — because the shape determines what the score can mean.

• Closed-form. One correct answer, checkable by string or number comparison. Cheap, perfectly reproducible, and biased towards tasks that happen to have a single answer.
• Execution-based. The output is a program and the grader runs it. Much harder to game by pattern-matching, because the code either passes the tests or it does not — but the score is only as good as the test suite’s coverage.
• Preference-based. Two outputs, a judge picks one. The only shape that handles genuinely open-ended work, and the only one whose grader has opinions of its own.

Knowledge: MMLU and GPQA

MMLU

Massive Multitask Language Understanding (Hendrycks et al., 2020) is roughly 16,000 four-option multiple-choice questions spread over 57 subjects, from elementary mathematics to professional law, largely drawn from exam-preparation material. It became the default headline number because it is broad, easy to run, and correlated with general model progress for several years.

What it cannot tell you: whether the model can produce anything. Every MMLU item is a selection among four given options, so the score says nothing about generation quality, format compliance, or the ability to say “I do not know.” It is also a benchmark with known label noise — the MMLU-Redux re-annotation effort (2024) documented incorrect gold answers concentrated in particular subjects — which puts a hard ceiling on what a top score can mean and makes small differences near the top uninterpretable. MMLU-Pro (2024) exists precisely because of the saturation: it raises the option count to ten and filters for items that require reasoning rather than recall.

GPQA

GPQA (Rein et al., 2023) is a deliberately small, deliberately hard set: 448 multiple-choice questions in biology, physics and chemistry, written by people with or working towards PhDs in those fields, and validated so that the questions are hard to answer by searching. The paper reports domain experts scoring around 65% and skilled non-experts with web access around 34% — the gap is the whole design. The 198-item “Diamond” subset is the one usually quoted.

What it cannot tell you: anything about your workload, unless your workload is graduate-level natural science. It is a capability probe for frontier reasoning, not a proxy for general usefulness, and its small size means the confidence interval on a Diamond score is wide enough to swallow most of the gaps people quote from it.

Reasoning: GSM8K, BBH, ARC-AGI

GSM8K (Cobbe et al., 2021) is 8,500 grade-school maths word problems, of which 1,319 are the test split, each solvable in a handful of arithmetic steps. It was the benchmark that made chain-of-thought prompting famous. Modern frontier models score near the top of it, which means its remaining value is as a floor test: a low score is informative, a high one is not. What it cannot tell you: whether the model can do multi-step reasoning over material it has not effectively seen — which is the reason the GSM1k replication effort (Scale AI, 2024) built a fresh set matched to GSM8K’s distribution and found some model families dropping substantially on it while others showed almost no gap.

BIG-bench Hard is the 23-task subset of BIG-bench on which language models had underperformed the human baseline. It is useful precisely because it was selected for difficulty. What it cannot tell you: anything about tasks not in those 23, and the tasks are synthetic puzzles — tracking shuffled objects, disambiguating pronouns, evaluating boolean expressions — rather than work.

ARC-AGI (Chollet, 2019) is a set of visual grid puzzles where each task supplies a few input/output examples and the solver must infer the transformation. It is explicitly designed to resist memorisation, with a private held-out set. What it cannot tell you: anything about language work at all. It is a research instrument about abstraction and sample efficiency, and treating it as a general capability score is a category error.

Code: HumanEval and SWE-bench

HumanEval

164 hand-written Python problems from the Codex paper (Chen et al., 2021). Each gives a function signature and docstring; the model writes the body; hidden unit tests decide. The paper also introduced the unbiased pass@k estimator that the whole field now uses.

What it cannot tell you: whether the model can work in a codebase. Every problem is a self-contained function of a few lines with no imports of consequence, no existing conventions to follow, and shallow test suites — the associated MBPP-style critiques and the EvalPlus effort both found that adding more tests to these problems changes the picture, because solutions that pass the original tests are not always correct. It is also small enough that a handful of items is a whole percentage point.

SWE-bench

SWE-bench (Jimenez et al., 2023) is the closest thing to real work in common use: 2,294 task instances, each built from a genuine GitHub issue and its merged pull request across twelve popular Python repositories. The model gets the repository at the parent commit and the issue text; success means producing a patch that makes the repository’s own tests pass. OpenAI’s SWE-bench Verified (2024) is a 500-instance subset that human annotators confirmed to be solvable and correctly specified, after it became clear that a portion of the original instances had underspecified issues or broken tests.

What it cannot tell you: anything about languages other than Python, about greenfield code, or about design quality — the tests were written for the human’s patch, so a passing patch is a patch that satisfies one specific existing test suite. It also conflates the agent scaffold with the model: a SWE-bench figure is always a score for a (model, harness) pair, and the harness can be worth a lot. Comparing two models’ SWE-bench numbers from different scaffolds compares mostly the scaffolds.

Open-ended: MT-Bench and arenas

MT-Bench (Zheng et al., 2023) is 80 multi-turn questions across eight categories — writing, roleplay, extraction, reasoning, maths, coding, STEM, humanities — scored by a strong model acting as judge. Its companion, Chatbot Arena, collects pairwise human votes on live traffic and fits a Bradley-Terry model to produce ratings. Both come from the same paper, and that paper is also the standard citation for the biases of model judges.

What they cannot tell you: for MT-Bench, anything reliable at fine resolution — 80 questions graded on a 1-10 scale by a model is a small, noisy instrument, and the paper itself documents the judge’s position and verbosity biases. For arena ratings, what the population of voters wanted: they are a preference aggregate over whoever showed up and whatever they typed, which weights style, formatting and agreeableness heavily. A rating is a real measurement of a real thing that is not your task.

How to read a model card

Three habits. First, look for the harness and the prompt format before the number; a score without them is not reproducible, and different harnesses genuinely disagree. Second, check whether the benchmark predates the model’s training cutoff, which decides whether contamination is on the table. Third, ignore any gap the benchmark’s own size cannot support — on a 198-item set, two points is three items.