Chain-of-Thought: The Paper and What It Actually Found

Almost every prompt-engineering guide contains the sentence “ask the model to think step by step”. It comes from a real paper with a real result, and that result carried two conditions which the advice dropped somewhere on the way.

The finding

Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (2022) makes a minimal change to few-shot prompting. Instead of exemplars that pair a question with its answer, the exemplars pair a question with a worked chain of intermediate steps and then the answer. Nothing about the model changes; no fine-tuning is involved. The model then produces its own intermediate steps for the new question before answering.

On multi-step arithmetic word problems the improvement was large — the kind of gap that does not require statistics to see. Similar gains appeared on commonsense and symbolic reasoning tasks. The paper is a clean demonstration: one variable changed, several benchmarks, several model families, results reported across model sizes.

That last choice is what makes it a good paper rather than a good trick, because the size sweep is where the actual finding lives.

The scope conditions

It is a property of large models

Plotted against model size, the benefit is roughly absent for small models and appears at the large end of the range the paper tested. For the smaller models, chain-of-thought prompting produced fluent chains that were logically wrong, and it could perform worse than answering directly. The paper describes the ability as emergent with scale and is careful to present it that way.

That has a direct practical consequence people still hit: applying the technique to a small model can make it worse. The prompt makes the model generate more text, and more text from a model that cannot carry the reasoning is more opportunity to go wrong. If you are choosing a small model to save money, this is one of the things you are buying.

It is a property of certain tasks

The tasks are ones with a decomposable procedure: arithmetic word problems, symbolic manipulation, multi-hop questions. There is a sequence of intermediate states, and writing them down helps because the model conditions on its own intermediate results — which is the whole mechanism, since a single forward pass has a fixed amount of computation available and generating tokens is the only way to buy more.

For tasks with no such structure — classifying sentiment, extracting a field from a document, rewriting a paragraph in a different register — there is no intermediate state to externalise. Adding a reasoning preamble there costs latency and output tokens and buys, at best, nothing. At worst it introduces a plausible rationalisation that the final answer then has to agree with.

The one-line sequel

A companion result, from Large Language Models are Zero-Shot Reasoners (2022), showed that much of the effect could be obtained without any exemplars at all — appending an instruction along the lines of “let us think step by step” before the model answers. That is where the famous phrasing comes from, and it is a separate paper from the one usually cited for it.

Worth noticing as a reading exercise: the zero-shot result is both more useful and weaker evidence. More useful because it needs no exemplars; weaker because a single magic phrase that improves benchmark scores is the sort of finding most exposed to prompt sensitivity — a nearby phrasing can behave differently, and the effect size depends on the model. That fragility is a general property of prompt-level results, and it is why small wording changes moving outputs deserves to be treated as a measurement problem rather than a curiosity.

Where the generalisation went wrong

The claim in the paper is roughly: on decomposable reasoning tasks, at sufficient scale, prompting for intermediate steps substantially improves accuracy. The claim that spread is roughly: telling a model to think step by step makes it smarter. Three specific errors follow from the difference.

• Applying it everywhere by default. System prompts that instruct a model to reason before every reply, including for extraction and classification tasks, pay output tokens and latency on every request for a benefit that only exists on a subset. That is a real bill, and it is the sort of change catalogued in output length as a cost lever.
• Confusing the prompt technique with a reasoning model. Models trained to spend inference compute on internal deliberation are a different thing from a prompting pattern applied to a standard model, and instructing the former to think step by step is at best redundant. The distinction is set out in reasoning models versus chain-of-thought prompting.
• Treating the chain as an explanation. The most consequential error, and it gets its own section.

The question the paper does not answer

The paper demonstrates that producing intermediate steps improves final answers. It does not demonstrate that the steps produced are the process by which the model reached the answer. Those are different claims and only the first one is supported.

This matters wherever the chain is being used as a justification: shown to a user as an explanation, logged for an audit, or graded by another model as a proxy for correctness. A chain can be wrong while the answer is right, right while the answer is wrong, or a post-hoc account that does not correspond to the computation at all. Subsequent work on whether stated reasoning reflects the actual determinants of the output treats this as an open and non-trivial problem, and the honest summary is that legibility and faithfulness are not the same property. There is more in whether the reasoning trace tells you anything.

The reading lesson generalises past this paper. When a result is stated as “X improves Y”, check whether the paper also claims a mechanism, and whether the mechanism is demonstrated or assumed. An awful lot of downstream advice is built on the assumed half.