MATH, AIME and Competition Maths Benchmarks

Competition maths benchmarks share one design decision that explains nearly all their behaviour: they grade the final answer and ignore the working. That makes them cheap and unambiguous to score, and it means a model can reach the right number by a wrong route and be marked correct, while a correct derivation formatted unexpectedly is marked wrong.

The family, and what distinguishes them

The one property that matters most is the answer format, because it decides how hard grading is. AIME’s integers from 0 to 999 are the easiest case in all of automated evaluation: exact integer match, no ambiguity, and a guessing floor of one in a thousand. MATH’s LaTeX expressions are the hard case, and the next two sections are about why.

They grade the answer, not the working

A MATH item ships with a full worked solution, but the grader does not read the model’s reasoning. It extracts the final answer — conventionally whatever is inside a boxed expression — and compares it to the key. Everything before that is ignored.

This has three consequences, in descending order of how often they bite.

• Right answer, wrong reasoning counts as correct. On a problem whose answer is 0, 1, 2 or a small integer, guessing has a non-trivial hit rate. On multiple-choice-derived items with a constrained answer space it is worse. The published research on process supervision exists precisely because outcome grading cannot see this — see process versus outcome supervision.
• The model is rewarded for arriving, not for arguing. Nothing about a score reflects whether the derivation would survive a marker. If you care about the proof, these benchmarks are not measuring it, and Putnam-style proof problems adapted to answer-only grading have quietly stopped testing what they were designed for.
• Truncation scores as wrong. A reasoning model that hits its token limit mid-derivation never emits a boxed answer, and an unparseable output is conventionally scored zero. Set a token limit too low and you measure the limit rather than the model. This is one of the settings covered in running a benchmark.

The equivalence checker is a scoring decision

MATH answers are mathematical expressions, and the same value has many correct spellings. The grader has to decide which spellings match, and that decision changes scores.

Gold answer:  \frac{1}{2}

Candidate            String match?   Sympy-style equivalence?
--------------------------------------------------------------
\frac{1}{2}          yes             yes
1/2                  no              yes
0.5                  no              yes
\dfrac{1}{2}         no              yes
\frac{2}{4}          no              yes
50\%                 no              depends on the parser
2^{-1}               no              yes

Gold answer:  2\sqrt{3}

\sqrt{12}            no              yes
3.464                no              only with a tolerance rule
2 \sqrt 3            no              yes (whitespace normalised)

A strict string comparison marks most of the right-hand column wrong. A symbolic checker — parse both sides, simplify, test equality — marks nearly all of them right. Between those two extremes sit a dozen hand-written normalisation rules that different harnesses implement differently: strip units, normalise fractions, handle intervals, handle sets, handle multiple answers separated by commas.

Two harnesses running the same model on the same 500 problems can disagree by several points purely on this. If you publish a MATH number, name the checker. If you read one that does not, you cannot compare it to another.

Fifteen questions is not a measurement

AIME is the extreme case, and it is quoted constantly. One paper is 15 questions. Both papers in a year is 30.

One AIME paper: n = 15
  One question is worth 100/15 = 6.67 percentage points.
  SE at p = 0.6 : sqrt(0.24/15)  = 0.1265 -> 95% CI = ±24.8 points

Both papers, one year: n = 30
  One question is worth 3.33 points.
  SE at p = 0.6 : sqrt(0.24/30)  = 0.0894 -> 95% CI = ±17.5 points

MATH-500: n = 500
  SE at p = 0.6 : sqrt(0.24/500) = 0.0219 -> 95% CI = ±4.3 points

Full MATH test: n = 5000
  SE at p = 0.6 : sqrt(0.24/5000)= 0.0069 -> 95% CI = ±1.4 points

A ten-point gap on a single AIME paper is one and a half questions and means nothing. Even the widely used MATH-500 cannot resolve gaps smaller than about four points from a single run. Anyone reporting AIME should be running the paper many times and reporting the mean with a spread across runs — and most do not, which is why AIME figures bounce around between sources for the same model.

The interval above is sampling error over questions only. For a reasoning model run at non-zero temperature there is a second variance component across repeats of the same question, and on a 15-item set it can dominate. The right procedure is in statistical significance with non-deterministic models.

The contamination clock

Competition maths has an unusual contamination profile, and it is worth understanding as a timeline rather than a yes-or-no.

1. The exam is sat. For a few hours, the problems exist only on paper.
2. Within a day, the problems and full solutions are posted to competition forums, discussed, and indexed.
3. Within months, they are in web crawls, in solution compilations, and in the training data of anything trained afterwards.
4. From then on, a score on that paper measures an unknown mixture of reasoning and recall, and there is no way from the outside to tell the proportions.

So an AIME score is only clean for models whose training cutoff predates the exam, and the window is measured in months. This is why the freshest paper is the one everyone reports, and why a model released a year after a paper scoring well on it tells you much less than the same score would have told you the week the paper was sat. MATH, published in 2021, has had five years of exposure.

The detection methods, and what they can and cannot establish, are in benchmark contamination. The decontamination side — checking your own data for overlap before you train on it — is in deduplicating and decontaminating a dataset.

What they are a bad proxy for

• Arithmetic reliability. Competition problems reward insight; they rarely require long, error-free calculation. A model can be strong here and still make arithmetic slips on a spreadsheet — the failure modes are catalogued in numerical reasoning failures.
• Applied or statistical mathematics. No modelling, no estimation, no data, no units, no deciding which method applies.
• Proof. Answer-only grading structurally cannot see a proof. Formal theorem-proving benchmarks are a separate genre with a separate scoring method.
• Cost. A high score obtained by generating tens of thousands of reasoning tokens per problem is a different product from the same score obtained in a few hundred. Tokens spent belongs next to the score — see cost-aware leaderboards.