Chain of Thought Is Now Built In. Should You Still Prompt for It?

Adding think step by step to a prompt was the highest-leverage five words in prompting for about two years. On a model that already reasons before it answers, the same five words buy nothing and may cost you something. The question is how to tell which model you have.

Why it worked

A model produces one token per forward pass, and each pass does a fixed amount of computation. A question whose answer needs six steps of arithmetic cannot be answered correctly in one token unless the model has memorised the answer, because there is nowhere to put the intermediate work. Chain of thought creates that somewhere: the intermediate steps become tokens in the context, and each subsequent token gets to attend to them.

That is the whole mechanism, and it explains both published results. Wei et al. (2022) showed that prompting with worked exemplars produced large gains on multi-step arithmetic and symbolic tasks, and that the effect appeared only at sufficient model scale. Kojima et al. (2022) showed you did not even need the exemplars — appending Let’s think step by step was enough to elicit the same behaviour zero-shot, which is why that exact sentence ended up in every prompt template on the internet.

A concrete version of the constraint: ask for the product of two four-digit numbers. The model has one forward pass to produce the first digit of the answer, and at that moment no partial product has been computed anywhere, so the digit is a guess conditioned on the shape of the question. Let it write the partial products first and every later digit attends to numbers that are now literally present in the context. That is why the technique helps exactly where there is intermediate state, and why it does so little where there is none.

What changed

Reasoning models moved the loop inside the product. Rather than hoping a prompt elicits intermediate tokens, the model generates them by construction, often hidden or summarised, and bills them as their own category — reasoning tokens, thinking tokens, whatever the vendor calls them. The controls moved too: instead of wording, you get a reasoning-effort setting or an explicit thinking-token budget.

On such a model an explicit CoT instruction is at best redundant. OpenAI’s published prompting guidance for its reasoning series has advised against chain-of-thought instructions on exactly these grounds — the model is already doing it, and the instruction can interfere with how it does it. Treat that as the strongest available evidence about a model’s internals, because it comes from the people who trained it.

Where the gains actually are

The most useful published result for planning is Sprague et al. (2024), To CoT or not to CoT?, a meta-analysis over a large body of prior CoT papers plus their own evaluations. Their headline conclusion is that the benefit of chain-of-thought prompting is concentrated in mathematical and symbolic reasoning — tasks with a formal intermediate state — and is small on average elsewhere.

That maps onto the mechanism cleanly. If a task genuinely has intermediate state (carry digits, a partial parse, a filtered set), writing that state down helps because the model can then read it. If the task is classification, extraction, tone judgement or retrieval from provided text, there is little intermediate state to externalise, and the reasoning tokens are mostly narration.

The cost of that narration is not small, because output tokens are the expensive side. Turning a 20-token answer into a 400-token reason-then-answer, at an illustrative $15 per million output tokens, moves a call from $0.0003 to $0.006 — a factor of twenty — and adds the wall-clock time of 380 sequential forward passes on top.

The decision

One mechanical rule survives everywhere, and it is the detail most often got wrong: the answer must come after the reasoning. Tokens are generated left to right, so if your output format puts answer before reasoning, the answer token was produced before any of the reasoning existed. What follows is a post-hoc justification of a committed answer, and you have paid for reasoning that could not possibly have informed it.

On a reasoning model the lever that remains is the size of the budget, not its existence. Effort controls are coarse — a few named levels, or a token ceiling — and the honest way to choose one is to run your eval set at each setting and plot accuracy against cost. The point where that curve flattens is a property of your task and cannot be inherited from somebody’s benchmark; expect it to arrive early for extraction and retrieval work, and late for anything with arithmetic in it.

The visible reasoning is not the computation

If you keep the reasoning for audit purposes, be precise about what you have. Turpin et al. (2023), Language Models Don’t Always Say What They Think, demonstrated that a model’s stated reasoning can systematically omit the factor actually driving its answer — they biased answers via features of the prompt and found explanations that never mentioned those features.

So a chain of thought is a plausible account, not a trace. It is genuinely useful for debugging, because it shows you which interpretation of the question the model took. It is not evidence of why a decision was made, and it should not be shown to an end user as if it were.

There is a corollary for prompts that ask a model to explain its refusal or justify its confidence. Those explanations are produced by the same next-token process as everything else and inherit the same unfaithfulness. Use them as a debugging aid and, if you must, as a UX affordance. Do not treat them as an audit record, and do not build a rule engine that parses them.