Verifiers and Reward Models at Inference Time

Generating a correct proof is hard; checking one is easy. Every test-time scaling technique that works is leaning on some version of that asymmetry, and the size of the asymmetry in your domain decides how much any of them can help.

The asymmetry that makes it work

The reason a verifier is worth building is that it can be far smaller and far cheaper than the generator while still being right often enough to be useful. A model with a fraction of the parameters can score candidates when it could not have produced them, because recognising a good answer requires less of the model than constructing one.

Cobbe et al.’s 2021 GSM8K paper is the canonical demonstration. They trained a verifier on model-generated solutions labelled by whether they reached the right answer, sampled a hundred candidate solutions per problem at test time, and reranked. The paper reported that verification scaled better with more data than finetuning did, and framed the finding as evidence that verification is the more favourable place to spend. Everything since — outcome reward models, process reward models, escalation routers — is a variation on that shape.

Three kinds of verifier

The strong recommendation is to exhaust the first row before touching the second. A schema validator and a unit test are not glamorous, but they are the only components in this entire cluster that do not have an error rate. A pipeline that runs the tests and only then asks a reward model to break ties is strictly better than one that starts with the reward model.

It is worth being precise about what a sound verifier is sound about. A JSON schema guarantees shape, never truth. A test suite guarantees the behaviours somebody thought to test. A solver guarantees the constraints you encoded. In each case the verifier is complete with respect to a specification and silent about everything outside it, and the gap between that specification and what you actually wanted is where your remaining errors live. Selecting hard against a partial specification pushes candidates into that gap — the same over-optimisation dynamic as with a learned verifier, arriving by a different route.

The third row deserves a specific caution because it is free and therefore over-used. Agreement measures confidence, not correctness, and a model that misreads a question misreads it consistently. When all eight samples agree on the same wrong answer, a consistency selector reports maximum confidence — which makes it not merely useless on that item but actively misleading if you surface the agreement rate to users as a certainty score.

How good does it have to be?

Worth doing the arithmetic before you build. Suppose your generator is correct 40% of the time and you draw eight samples, so coverage — the chance at least one candidate is right — is about 0.98. Now suppose your verifier ranks a wrong candidate above the right one some fraction of the time. What you ship is coverage times the chance the verifier picks correctly given that a correct candidate is present.

p          = 0.40      per-sample correctness
N          = 8         candidates
coverage   = 1 - 0.6^8 = 0.983

verifier picks the correct one, given one exists:
  1.00  ->  shipped accuracy 0.983     (sound verifier)
  0.80  ->                   0.786
  0.60  ->                   0.590
  0.45  ->                   0.442     barely above p = 0.40
  0.40  ->                   0.393     WORSE than one sample, 8x the cost

That last row is the point of the table. A verifier that is not meaningfully better than chance at discriminating among your candidates turns a best-of-N pipeline into an expensive random-selection pipeline. Measure the discrimination rate directly — give it candidate sets where you know which are correct, and count how often it ranks a correct one first — before you build anything on top of it.

Wiring it up

async function verifiedAnswer(task, n = 8) {
  const candidates = await Promise.all(
    Array.from({ length: n }, () => call(GENERATOR, task.prompt, { temperature: 0.8 }))
  );

  // 1. Sound checks first. Free correctness, no error rate.
  const valid = candidates.filter(c => runTests(task, c).ok);
  if (valid.length === 1) return valid[0];

  const pool = valid.length ? valid : candidates;   // never end up empty

  // 2. Consistency, if answers are comparable at all.
  const grouped = groupBy(pool, normaliseAnswer);
  const top = maxBy([...grouped.values()], g => g.length);
  if (top.length > pool.length / 2) return top[0];

  // 3. Learned verifier only to break a genuine tie.
  const scored = await scoreAll(VERIFIER, task, pool);
  return maxBy(scored, s => s.score).candidate;
}

The ordering is the design. Each stage is cheaper and more trustworthy than the next, and the expensive fallible one only sees the cases the reliable ones could not resolve. Note the pool line: if every candidate fails the sound check you must not return nothing, and you must not silently pretend one passed — rank the failures and mark the result as unverified so the caller can escalate.

Keeping it honest

A learned verifier used as a selection target is a reward model being optimised against, and the over-optimisation results named in best-of-N sampling apply in full. The failure is quiet: your verifier score climbs, your user satisfaction does not, and nothing in the pipeline reports a problem.

• Hold out a human-judged set and check periodically that verifier score still correlates with it. This is the only instrument that detects the failure.
• Cap N when the verifier is learned. The strength of optimisation against it scales with N, so a large N is precisely the condition under which it is most exploited.
• Retrain when the generator changes. A verifier is trained on one model’s error distribution; swapping the generator changes the distribution and quietly invalidates it.
• Log disagreements between stages. Cases where the sound check and the learned verifier disagree are the highest-value examples you will find for improving either.

There is one more thing to decide, and it is a product decision rather than a technical one: what happens when nothing verifies. Every pipeline eventually produces a request where no candidate passes, and the three available answers are to ship the best failure with a caveat, to escalate to a more capable model, or to decline. All three are defensible; silently shipping the best failure without marking it is not, and it is the default if nobody chooses. Make the unverified state explicit in your return type so that the caller has to handle it rather than discovering it from a complaint.

Finally, keep the verifier’s cost in view. A learned verifier scoring eight candidates is eight additional calls, and if it is a large model that is most of the saving gone. The technique works because the verifier is small and the generator is not. If yours has drifted towards the same size, you no longer have a verification pipeline — you have two generators and a preference between them.