MMLU: What It Tests and What a Score Means

MMLU is a set of roughly 14,000 four-option multiple-choice questions spread across 57 subjects, collected from freely available practice exams and evaluated five-shot. It measures whether a model can pick the right letter on exam-style questions. That is a narrower claim than “massive multitask language understanding” suggests, and the gap between the two is most of what you need to know to read a score.

What MMLU actually is

MMLU — Massive Multitask Language Understanding — was introduced by Hendrycks and colleagues in the 2020 paper Measuring Massive Multitask Language Understanding. The questions were gathered by graduate and undergraduate students from material already free on the web: practice questions for exams like the GRE and the US medical licensing exam, quiz banks, and textbook problem sets.

It is organised as 57 separate subject tasks, conventionally grouped into four categories: STEM, humanities, social sciences, and an “other” bucket that holds the professional and applied subjects such as clinical knowledge, accounting and marketing. The subjects are wildly uneven in size — some have a few hundred test questions, others barely a hundred — which matters later.

The question format, and why it matters

Every item is a question stem with exactly four options, labelled A to D, and exactly one is keyed correct. There is no partial credit, no working shown, and no way for a model to signal uncertainty. The answer space is four tokens wide.

Two consequences follow immediately. First, the floor is 25 per cent, not zero — a model that has learned nothing but the format scores a quarter. Any headline number should be read against that floor rather than against zero. Second, because the model only has to rank four candidates rather than produce an answer, MMLU rewards recognition rather than generation. A model can pick the right option for a question it could not have answered on a blank page.

There is also a well-documented positional bias in multiple-choice evaluation: models are not indifferent to whether the correct answer sits at A or at D. Permuting the options changes scores. Some harnesses now evaluate each item under several option orderings and average; most do not, and the two are not comparable.

How it is scored

The metric is plain accuracy: the fraction of items where the model selected the keyed option. The headline number is an average over the 57 subjects, and there are two ways to take that average which give different answers.

Micro-average  = (total correct across all items) / (total items)
Macro-average  = mean over the 57 subjects of (correct in subject / items in subject)

Toy example, two subjects:
  Subject A: 900 items,  700 correct  -> 77.8%
  Subject B: 100 items,   40 correct  -> 40.0%

  Micro = (700 + 40) / (900 + 100) = 740 / 1000 = 74.0%
  Macro = (77.8 + 40.0) / 2                    = 58.9%

Same answers. 15 points apart.

Because MMLU’s subjects differ in size by roughly an order of magnitude, this is not a hypothetical gap. A published number that does not say which average it used is missing a load-bearing fact. Most modern harnesses report the micro-average over all test items; the original paper reported a macro-average over the 57 tasks.

The answer-extraction problem

This is where MMLU numbers most often diverge, and it has nothing to do with the model’s knowledge. There are two families of implementation.

• Log-probability ranking. Present the question, then score the log-probability the model assigns to each of the four continuations. Pick the highest. Nothing is generated. This is fast, deterministic, and requires access to logprobs. It also has a sub-choice with real consequences: whether to compare the probability of the letter (“A”) or of the full answer text, and if the latter, whether to normalise by length. Longer options are less probable purely because they have more tokens, so unnormalised scoring quietly favours short options.
• Generate and parse. Ask the model to answer, let it emit text, and extract a letter with a regular expression or a second-stage parser. This is the only option for an API that does not return logprobs, and it is closer to how the model is used in practice. It is also fragile: a model that writes “The answer is (C)” when the parser expects a bare letter is scored wrong for a formatting mismatch.

The two families do not agree, and the disagreement is not small. HuggingFace published a detailed write-up in June 2023 explaining why the Open LLM Leaderboard’s MMLU numbers did not match the published ones, tracing it to exactly this: three implementations of “MMLU” making different prompting and extraction choices. If two numbers disagree, check this before concluding anything about the models. There is more on this in running a public benchmark yourself.

Known label errors

MMLU’s questions were harvested at scale from web sources, and a proportion of them are wrong: mis-keyed answers, questions whose correct option is absent, ambiguous stems, and items whose source material was mangled in extraction.

This has been studied. The MMLU-Redux work (Gema and colleagues, 2024) manually re-annotated a sample of MMLU across a subset of its subjects and found a meaningful error rate overall, concentrated very unevenly — some subjects were close to clean, while at least one subject had errors in a large fraction of the sampled items. Treat the per-subject rates as approximate unless you read the paper; treat the headline finding as solid, because it is the one everybody who has looked at MMLU by hand reports.

MMLU-Pro (2024) is the response most people reach for. It expands the option count from four to ten, filters out items answerable without the question, and leans harder on reasoning. Ten options drops the guessing floor from 25 per cent to 10 per cent and spreads the top of the range back out. It is a different benchmark with a different distribution; scores are not interchangeable with MMLU’s.

The ceiling problem

MMLU was published in 2020 as a test that models of the day found very hard. It is no longer that. The frontier now sits in a narrow band near the top, which changes what the number can do.

The arithmetic is the reason. Take a subject with 100 test items. One item is one percentage point in that subject’s score. Now consider the whole test at around 14,000 items: the standard error of a proportion is the square root of p(1−p)/n, so at p = 0.85 and n = 14,000 that is about 0.3 points, giving a 95 per cent interval of roughly plus or minus 0.7 points from sampling alone — before you add label error, prompt sensitivity, or run-to-run variance from sampling.

SE = sqrt( p * (1 - p) / n )

Whole test:      p = 0.85, n = 14000 -> SE = sqrt(0.1275/14000) = 0.0030 -> ±0.6 pts (95%)
One subject:     p = 0.85, n =   100 -> SE = sqrt(0.1275/  100) = 0.0357 -> ±7.0 pts (95%)

So a 1-point gap on the whole test is at the edge of sampling noise,
and a 5-point gap on a single subject is inside it.

A one-point difference in a launch post is not a difference. This is why GPQA and similar harder sets exist, and it is why reading a launch post critically starts with asking how many items produced the number.

What MMLU is a bad proxy for

• Anything with a free-form answer. MMLU never asks a model to write a paragraph, produce a structured object, or make a judgement call. Ranking four options and composing an answer are different tasks and the second is what most applications need.
• Reasoning over long inputs. Every stem is short. Nothing here says anything about behaviour at 50,000 tokens — for that see long-context benchmarks.
• Calibration. The format has no place to express “I do not know”, so nothing about a score tells you whether the model would have been confidently wrong. Suites that measure that exist — see HELM.
• Your domain, even if it is one of the 57. The professional-subject tasks are exam questions, not work. A model that scores well on professional medicine has demonstrated performance on licensing-exam items, which is a real thing and not the same thing as clinical usefulness.
• Uncontaminated recall. MMLU has been fully public since 2020 and appears verbatim across the open web. Contamination exposure is at its maximum. What that means and how it is detected is a page of its own.

None of that makes the number useless. A model far below the pack on MMLU has told you something real about its breadth of factual recall. It is the small differences at the top that carry no information, and those are exactly the ones that get quoted.