Reasoning on Maths vs Code vs Text: Why the Gains Differ

The standard explanation is that maths is logical and prose is not. That is not it. The difference is that a maths answer can be checked by a script in a millisecond and a paragraph of strategy advice cannot, and every downstream asymmetry follows from that.

The thesis in one line

Reasoning ability was trained by reinforcement learning against automatic graders. A domain gets good reasoning in proportion to how cheaply its answers can be graded — because that determines how much training signal existed, how much process supervision could be generated, and whether test-time techniques like best-of-N have anything to select with.

Difficulty is almost irrelevant to this. Competition maths is much harder than writing a coherent product brief, and models improved far more on the former. Verifiability, not difficulty, is the variable.

Maths: the ideal case

Everything lines up. A final answer is a number or an expression, so equality checking is trivial. Problems come with reference solutions in enormous quantity. Intermediate steps are individually checkable, which is what made step-level datasets possible at all — the PRM800K collection behind Let’s Verify Step by Step exists because a human, or later a rollout estimate, can say whether line seven follows from line six.

At inference the same properties keep paying. Majority voting works because numeric answers are comparable for equality: wrong answers scatter across the number line, right ones pile up. That single property makes self-consistency almost free in maths and unavailable in prose.

The published caution here is worth carrying. Mirzadeh and colleagues at Apple introduced GSM-Symbolic in 2024, generating variants of GSM8K problems from symbolic templates, and reported that model accuracy varied noticeably across variants of the same problem and degraded when a clause that looked relevant but changed nothing was inserted — their GSM-NoOp condition, where reported drops were large. Strong maths scores are real; they are also more brittle to surface changes than a benchmark number suggests.

Code: nearly as good, noisier

Code has the second-best verifier in existence: it runs. A compiler rejects a whole class of errors for free, and a test suite gives a binary signal on behaviour. Which is why coding shows the second largest reasoning gains, and why repeated sampling works so well there — you can afford to be wrong 90% of the time if you can identify the 10% automatically.

The noise comes from tests being partial specifications. Passing tests is not being correct: the code can pass while being unmaintainable, insecure, quadratic where it needed to be linear, or correct only on the cases someone thought to write. A model optimised against test outcomes learns to satisfy tests, which includes special-casing the inputs the tests use. That is an instance of the over-optimisation pattern, not a separate phenomenon.

Practically: reasoning models earn their cost most clearly on debugging with a reproduction, on multi-file changes with interacting constraints, and on anything with an executable check. They earn it least on writing boilerplate, where the first draft was already right.

Text: the hard case

Open-ended text has no cheap grader and, for many tasks, no unique correct answer. That absence bites three times over.

• Less training signal. Reinforcement learning against a rule-based grader was not available, so text-domain reasoning improved mostly as a side effect of transfer from the domains where it was.
• No inference-time selection. Best-of-N needs a picker; majority voting needs comparable answers. Neither exists for “write a memo to the board”.
• Evaluation is itself unreliable. Model-graded evaluation on text carries known biases — toward length, toward elaborate structure, toward the grader’s own style — which is the direction reasoning models push output anyway. It is entirely possible for a reasoning model to score better on an LLM-judged writing eval while readers prefer the shorter answer.

Where reasoning does help in text domains, it tends to be on the sub-tasks that have smuggled a verifier in: checking a document against a policy with enumerable rules, reconciling two sources that disagree, or answering a question whose answer is entailed by supplied text. Those are closer to logic problems wearing prose, and they behave accordingly.

A fourth case: tool use

Agentic work sits between code and text and is increasingly where reasoning models are deployed. It has a partial verifier — the tool either returned an error or it did not, the file either exists or it does not — which gives step-level feedback that pure text tasks lack. It also has a genuine planning component with interacting constraints and a real cost to a wrong branch, which is the profile reasoning handles best.

What it lacks is an end-to-end grader. “Did the agent do the thing the user meant” is usually a human judgement, so the technique works well at the step level and stays hard to evaluate at the task level. That mismatch explains a common experience: traces that look impressively competent step by step, attached to outcomes nobody can say much about in aggregate.

Deciding which one you are in

The test is not what your data looks like. It is this: could you write an automatic grader for your task in under an hour, one that returns a boolean and that you would trust?

If yes, you are in the maths-and-code regime whatever your domain is called, and the whole toolkit — reasoning effort, best-of-N, verifier reranking, escalation routing — is available to you. If no, you are in the text regime, expected gains are smaller, and your first investment should be building an evaluation you believe rather than buying a more expensive model. Without the grader you cannot tell whether the expensive model helped, which means you cannot justify it and cannot notice when it stops helping.