Do Reasoning Models Actually Reason? The Honest Answer

Two camps present largely the same evidence and reach opposite conclusions, which is a reliable sign that the disagreement is about a word. It is worth reading both sets of results carefully anyway, because the specific failures they document are the ones you will hit.

Why the argument does not resolve

“Reasoning” is being used for at least three different claims: that the system produces correct answers to novel multi-step problems; that it does so by manipulating internal representations with a structure resembling inference; and that it possesses understanding in whatever sense a person does. The first is measurable and largely settled in the affirmative for some domains. The second is an open empirical question about mechanisms. The third is not an empirical question at all in its current formulation.

Most public disagreements consist of one party arguing the first and another denying the third. Neither is wrong; they are answering different questions with the same sentence.

The question is not merely academic, which is why it is worth an hour rather than a shrug. What you believe about it decides how far you extrapolate. If these systems perform something like general inference, a strong benchmark score is evidence about performance on your problem too. If they perform sophisticated pattern completion within the neighbourhood of their training distribution, a benchmark score is evidence about that benchmark and very little else, and the correct engineering response is to test on your own inputs and distrust every transfer argument. That is a practical disagreement about how much a number carries, and it has budget consequences.

The case that they do not

The sceptical case rests on brittleness under changes that would not trouble anyone who understood the problem, and it has real results behind it.

• Surface sensitivity. Mirzadeh et al.’s GSM-Symbolic work (2024) generated variants of grade-school maths problems from templates — same structure, different names and numbers — and reported that accuracy varied across variants of the same problem in ways a robust procedure would not produce. The GSM-NoOp condition, which adds a clause that looks relevant but changes nothing, produced large reported drops.
• Complexity cliffs. Apple’s The Illusion of Thinking (2025) found that on controllable puzzles, accuracy fell to near zero past a complexity threshold, and that models reduced their thinking effort as they approached it. The paper argues this is inconsistent with a general reasoning procedure, which would simply take longer.
• Unfaithful traces. If the stated reasoning is not the cause of the answer — the finding in the faithfulness literature — then the visible “reasoning” is not evidence of reasoning, whatever else is going on.
• Contamination. Benchmark problems and their solutions circulate on the web. Some fraction of any impressive score is retrieval, and separating the fractions is genuinely hard.

The case that they do

The other side is not merely enthusiasm, and its strongest points are specific.

• Performance on problems that postdate training. Competition problems released after a model’s cutoff cannot be memorised. Scores on such sets are lower than on contaminated ones and are not zero, which is the single most direct rebuttal to the pure-retrieval account.
• The intervention works. Spending more tokens improves accuracy on hard problems and not on easy ones, in the way a computation-limited process would and a lookup would not. Retrieval does not get better because you gave it more time to think.
• The failures are human-shaped. Arithmetic slips, dropped constraints, sign errors, plausible wrong turns. A lookup table fails by returning nothing or something unrelated; these systems fail the way a tired person doing the problem fails.
• The sceptical results are contested on their methods. A 2025 response to The Illusion of Thinking argued that several of the reported collapses coincided with output token limits being hit and that at least one puzzle family included provably unsolvable instances scored as failures. Whether or not you accept the rebuttal, it illustrates how much of this debate turns on experimental design rather than on model behaviour.

A definition you can test

Replace the word with a property you can measure: performance survives perturbations that preserve the problem. That is checkable, it is what you actually need from a production system, and it does not require anyone to agree on what understanding is.

For each item in your eval set, produce four variants:

  rename      swap entity names, keep the structure
  renumber    change the numbers, recompute the expected answer
  reorder     permute independent clauses and answer options
  distract    add a true but irrelevant sentence

Score all four. Report the WORST, not the mean.

  worst / original  > 0.9   robust on this task
                    ~ 0.7   usable with a verifier
                    < 0.5   the original score was measuring
                            something other than the task

The ratio is the number to carry around. It converts an unresolvable philosophical question into a per-task measurement, and it is the thing that predicts whether a benchmark score will survive contact with your inputs. Run it once and the abstract debate stops being load-bearing for your decisions.

What follows for engineering

Whichever camp you find more convincing, the operational conclusions are the same, which is itself informative.

Verify outputs where correctness matters, because neither camp claims reliability. Test on perturbed versions of your own data, because benchmark numbers are the most contaminated and least representative measurements available. Expect degradation as problems grow, and find your own cliff rather than assuming it is beyond your range. And treat the visible trace as a debugging aid rather than as a justification.

The reasonable position, and the one most working practitioners converge on, is that these systems perform a genuine computation that is genuinely useful and genuinely unlike deduction — powerful in a band around what the weights already nearly know, and sharply limited outside it. That is not a compromise for the sake of politeness. It is what the evidence in both directions actually supports, and it is enough to build on.