Process Supervision vs Outcome Supervision

A twenty-step solution with one bad step in the middle gets the same score as a solution that was wrong from line one, if all you grade is the final answer. Process supervision is the decision to stop doing that, and it is the single training change most responsible for modern reasoning behaviour.

The credit assignment problem

Suppose a model produces a chain of twenty steps and the final answer is wrong. An outcome-supervised signal tells you: this trajectory was bad. It does not tell you that steps one to twelve were impeccable and step thirteen dropped a minus sign. Every token in the trajectory receives the same discouragement, including the good ones.

The reverse case is worse. A trajectory with a wrong step that happens to reach the right answer — through a compensating second error, or because the answer space was small enough to guess into — receives full credit, and the bad step is reinforced. Outcome supervision cannot distinguish a sound derivation from a lucky one, and reasoning traces contain a great many lucky ones.

Process supervision assigns a label per step. Now the gradient can be specific: this prefix was fine, this step was the error. It is the same argument that makes a failing unit test more useful than a bug report saying the program is wrong.

The size of the problem scales with trace length, which is why this became urgent exactly when traces got long. A five-step solution has little credit to misassign. A three-hundred-step trace that explores, backtracks and recovers contains a great many good decisions inside a trajectory that failed, and an outcome-only signal throws all of them away with the ending. Long-trace reasoning is the regime in which outcome supervision is weakest, and it arrived at the same time as the models that needed it most.

Two reward models

The distinction matters at inference time as much as at training time. An ORM can only rank complete candidates, which makes it a best-of-N reranker. A PRM can score a half-finished solution, which is what makes guided search possible at all — you cannot prune a branch you cannot evaluate, and that is precisely the requirement in tree search over reasoning steps.

The published comparison

Uesato and colleagues at DeepMind published the first careful head-to-head in 2022, Solving Math Word Problems with Process- and Outcome-Based Feedback, reporting comparable final-answer accuracy between the two but a substantially lower rate of reasoning errors under process feedback — that is, the outcome-supervised model was right about as often and right for the wrong reasons more often.

The result that moved the field was Lightman et al.’s Let’s Verify Step by Step (2023). They collected step-level human feedback at scale — released as the PRM800K dataset, around 800,000 step-level labels — and reported that a process- supervised reward model used to rerank solutions solved 78.2% of a representative subset of the MATH test set, beating outcome supervision under the same sampling budget. The paper also argued a point that has aged well: process supervision is more directly aligned with what you want, because it rewards a chain a human would endorse rather than an answer that happens to match.

One methodological detail from that paper is worth carrying, because it recurs whenever these results are compared. The advantage of process supervision grew with the number of candidate solutions being reranked — the two approaches were close at small sample counts and separated as the budget increased. Any comparison of ORM against PRM therefore has to state the sampling budget, and a comparison that does not state one is not really reporting a result.

Right answer, wrong reasoning

This is the practical consequence for anyone consuming these models rather than training them. Outcome-supervised training produces models that are optimised for the final token being correct, which is exactly the objective under which an unfaithful but well-formed trace is free. The faithfulness results in the faithfulness literature are not a separate mystery; they are the predictable output of a reward that never scored the account of the reasoning.

It also explains a specific irritation: a model that gets the answer right while presenting a derivation with a visible error in it. Under outcome supervision that trajectory was rewarded. There was never a gradient that said the middle should be right too.

Where the labels come from

Human step labelling does not scale past a research budget, which is why the interesting work went into producing process labels automatically. The dominant technique estimates the value of a prefix by rollout: from a given step, sample many completions, and score that step by the fraction that reach a correct final answer. A step after which 80% of continuations succeed is a good step; one after which 5% do is where the solution died. Math-Shepherd (2023) is the widely cited instance of this construction.

The cost is obvious once stated — it is a per-step Monte Carlo estimate, so labelling one solution costs many completions rather than one. That expense is why process supervision arrived on maths and code first, and why domains without an automatic final-answer check have largely not received it. The same asymmetry shows up again in reasoning by domain, and it is the most reliable predictor in this cluster of where a technique will and will not transfer.

For a reader consuming these models rather than training them, the takeaway is a heuristic about trust. A model whose reasoning was shaped by step-level feedback in a domain is likelier to have a trace whose middle is worth reading; one shaped only by final-answer rewards is likelier to have a trace that is decorative. You will rarely be told which you have — but the domain is a decent proxy, because process labels exist where automatic graders do.