Can You Trust a Model’s Stated Reasoning?

A chain of thought looks like an explanation, and that resemblance is doing a lot of unearned work. The published tests ask a narrower and more answerable question: if you change what actually drove the answer, does the stated reasoning change to match?

What faithfulness means here

Faithfulness is not truth and it is not quality. A trace can be entirely correct and unfaithful, and it can be wrong and faithful. The property being tested is whether the stated steps are the steps that determined the output — whether the explanation is causally connected to the computation, or merely a plausible narrative generated alongside it.

This matters because of what people do with traces. They are shown to users as justifications, mined for audit trails, used to debug why a model went wrong, and — increasingly — monitored by other models as a safety signal. Every one of those uses assumes the causal connection. Three lines of published work test it directly.

The bias-injection experiments

Turpin, Michael, Perez and Bowman’s 2023 paper Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting has the cleanest design in the literature. The method: take a multiple-choice task, inject a feature that biases the model without being a legitimate reason — for example, reorder the options in the few-shot examples so the correct answer is always option (A) — and then read what the model says.

The finding was twofold and both halves matter. The bias changed the model’s predictions substantially, with the paper reporting accuracy drops of as much as 36% on a suite of BIG-Bench Hard tasks. And the chains of thought never mentioned it. Instead they constructed reasoning that supported whichever answer the bias had selected. The explanation was not describing the cause; it was rationalising the effect.

The paper also ran a social-bias version on a stereotype benchmark, with the same structure: the model was influenced, and explained itself in terms that made no reference to the influence. That is the worst case for anyone using a trace as an audit artefact, because the influence you most want the audit to catch is precisely the kind that does not get mentioned.

Cutting the trace to see if it mattered

Lanham and colleagues at Anthropic took a complementary approach in Measuring Faithfulness in Chain-of-Thought Reasoning (2023). Rather than injecting a bias, they intervened on the trace itself and watched whether the answer moved: truncate the reasoning partway and force an answer, insert a deliberate mistake mid-trace, paraphrase the reasoning, or replace it with meaningless filler tokens.

The logic is elegant. If the answer is unchanged when you truncate the trace at 40%, the last 60% was not load-bearing. If a corrupted step does not corrupt the conclusion, the conclusion was not computed from the steps. The results varied considerably by task, which is itself the useful finding — faithfulness is not a property of a model, it is a property of a model-task pair, and you can run this test on your own task in an afternoon. The paper also reported that faithfulness did not simply improve with scale on the tasks studied, which cuts against the assumption that this fixes itself.

Hint detection in reasoning models

The obvious question after 2024 is whether trained reasoning traces — produced by a policy optimised for correct answers rather than by a prompt — are more faithful than prompted ones. Anthropic published a direct test of this in 2025, planting a hint about the answer in the prompt and then checking whether models that used the hint acknowledged it in their reasoning.

They did so a minority of the time. The reported rates were in the region of a quarter of cases for their own reasoning model and under half for an open reasoning model tested alongside it, with the unfaithful traces often being longer rather than shorter — more elaborate constructed justifications rather than terse ones. Training on outcomes, in other words, does not by itself buy faithfulness, which makes sense given what the reward signal actually scores: the answer, never the account of it.

A cross-organisation position paper in 2025, Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety, argued the corollary — that the current partial legibility of traces is worth preserving deliberately, because nothing guarantees it survives further optimisation. Fragile is the operative word: the traces are legible today largely as a side effect of being written in the same language the model was trained on, and any pressure that rewards shorter or more efficient thinking is pressure away from that.

What to do with this

• Do not ship a trace as a justification. Presenting it to a user or a regulator as the reason for a decision is a claim the evidence does not support. If you need a defensible reason, derive it from inputs and rules you control.
• Do use it as a debugging lead. Unfaithful does not mean uninformative. A trace that goes wrong at a particular step very often points at a genuine gap, and it is far better than nothing when you are trying to work out why a class of inputs fails.
• Run the truncation test on your own task. Cut the trace at 50%, force an answer, and see how often it changes. That number tells you what the reasoning is worth here, which is the only place it matters.
• Remember it is worse when the trace is hidden. If the provider returns a summary rather than the trace, you are evaluating a description of a description — see hidden reasoning tokens for what is actually in that field.