Reasoning Models vs Chain-of-Thought Prompting

Both put intermediate steps between the question and the answer. That is where the similarity stops. One is a prompt you own and can edit; the other is a trained policy you cannot see, cannot cache and are billed for at the output rate.

The thing they have in common

The underlying mechanism is identical, and it is worth stating plainly because it demystifies both. A transformer does a fixed amount of computation per token. If the answer requires more computation than one token’s worth, the only way to get it is to produce more tokens — the intermediate tokens are the scratch space, and they enter the context so later steps can attend to them. Chain-of-thought prompting discovers this behaviour by asking; a reasoning model has it trained in.

Wei et al.’s 2022 paper Chain-of-Thought Prompting Elicits Reasoning in Large Language Models is the origin of the first, and Kojima et al.’s 2022 zero-shot result showed the same effect could be triggered by the phrase “let’s think step by step” alone. Reasoning models are what happened when someone trained the behaviour with reinforcement learning against checkable answers instead of asking for it politely.

Six differences that do not move

Read that table as a statement about ownership. Chain of thought is engineering you do; a reasoning model is capability you rent. The second is far less work and gives up the ability to fix a specific recurring mistake by editing four words.

Why combining them usually backfires

The instinct is to put “think step by step, showing your work” into the system prompt of a reasoning model. It is redundant at best. At worst it produces a documented set of annoyances: the model does its trained thinking, then dutifully restates a second visible chain of thought in the answer, so you pay twice for the same reasoning and your users get a wall of working they did not ask for.

Vendor guidance on this has been unusually consistent — the recommendation for reasoning models is to state the goal and the constraints and stop, rather than to prescribe a method. Prompts that impose a rigid decomposition can also cut across whatever decomposition the policy learned, which is the one part of these models you have no visibility into and therefore no ability to debug.

Few-shot exemplars deserve a specific warning. On non-reasoning models, worked examples are among the highest-leverage prompt techniques available. On reasoning models they are frequently neutral or harmful, because the exemplar trace competes with the trained one. If you carry a prompt library across from an older model, the exemplars are the first thing to try removing.

Self-consistency sits between them

Worth knowing about as a third option, because it is often the right one and rarely considered. Wang et al. (2022) showed that sampling several chain-of-thought answers and taking the majority final answer outperformed a single greedy chain. It buys reliability with parallel calls rather than with serial thinking, which means it costs tokens without costing latency, and it works on any model — including one with no reasoning variant at all.

The requirement is that final answers be comparable for equality, so it applies to numbers, labels and short spans and not to prose. Where it applies, it is frequently the cheapest quality improvement on the table, and it composes with either of the other two; best-of-N sampling covers the arithmetic of how far it goes.

Choosing

• Take chain of thought when you need a specific procedure followed — a compliance checklist, a rubric, a fixed order of checks — or when you need the working itself as an artefact, or when your model of choice has no reasoning variant, or when the visible steps are the product (tutoring, explanations, worked solutions).
• Take a reasoning model when the task is open-ended enough that you do not know the right decomposition, when you have an automatic grader and care only about the final answer, or when you have tried to write the chain-of-thought prompt and found it needs a different shape for every input.
• Take neither when the task is extraction, classification or reformatting. Both are overhead on work that has no intermediate steps — the taxonomy of tasks that need nothing applies to both techniques equally.

What the published record supports

It is fair to say the direction is settled and the size is not. Chain-of-thought prompting was shown to produce large gains on multi-step arithmetic and symbolic tasks in the 2022 literature, with the crucial finding that the effect was strongly dependent on model scale — small models got worse, not better. Trained reasoning policies, from the o1 announcement in 2024 onwards, have posted large gains on competition maths and code benchmarks relative to their non-reasoning siblings.

What nobody has published, and what you should be suspicious of anyone claiming, is a general cost-per-correct-answer comparison between the two. It depends on the task, the model pair, the effort level and the prompt quality, and every one of those changes the answer. The procedure in calibrating reasoning effort is the same procedure that answers it for your workload, with one extra arm for the chain-of-thought prompt.

One asymmetry does generalise, and it is about maintenance rather than performance. A chain-of-thought prompt is a thing you own: it keeps working when the vendor updates the model, it can be diffed, reviewed and reverted, and when it breaks you can see why. A reasoning policy is a thing you rent, and it changes underneath you on the vendor’s schedule with no diff to read. Neither property decides the choice by itself, but the second is the one teams consistently fail to price in, and it is the reason to keep a non-reasoning arm in your evaluation permanently even after you have chosen.