Verifiable Rewards: Training on Problems With Right Answers

A verifiable reward replaces a learned scoring function with an exact check: did the answer match, did the tests pass, did the proof compile. The reward becomes a program rather than a model, and that single change removes the largest source of reward hacking while introducing a smaller one.

The idea, and the term

In standard RLHF, the reward comes from a reward model trained on human preferences. That model is an approximation with its own errors, and the policy will find them. If instead the task has a checkable answer, the reward can be computed exactly:

reward = 1  if normalise(extracted_answer) == normalise(reference_answer)
       = 0  otherwise

or, for code:

reward = 1  if all held-out tests pass
       = 0  otherwise

The term reinforcement learning with verifiable rewards, abbreviated RLVR, was introduced in the Tulu 3 work from the Allen Institute for AI (2024), which described training against exact checks for tasks with known answers as a distinct method alongside preference-based post-training. The underlying practice — rewarding a model for getting the right answer — is older; the name gave it a shared reference point.

Structurally, this is a return to something the field had before preference learning. Game scores are verifiable rewards. So are unit tests. Preference learning existed because the tasks people wanted language models to do had no score, and verifiable rewards are the subset where they do after all.

The four properties a task needs

A task admits a verifiable reward only if all four of these hold. Most valuable tasks fail at least one, and knowing which one tells you what would have to change.

Run a task you care about through those four. Customer support: no fact of the matter for most tickets. Medical diagnosis: an answer exists, checking is not cheap and labelled volume is limited. SQL generation against a known schema: all four can hold, if you have queries with expected result sets. That last one is a genuinely promising and underused case.

Why maths and code came first

Competition mathematics and programming problems satisfy all four properties better than almost anything else. The answer is a short string or a number. Checking is a comparison or a test run. Datasets of problems with answers exist in quantity and have for years. And difficulty is graded, so a training set can span the range where the model sometimes succeeds.

There is a fifth property that is less often stated and may matter more: the reasoning is long and the answer is short. A model can spend two thousand tokens working, and the reward depends on one line at the end. That means the training signal says nothing about how to reason and everything about whether the reasoning worked, which leaves the method free to discover approaches nobody demonstrated. Training on human-written derivations teaches the derivations people write; training on whether the answer is right does not.

The consequence people find surprising is that reasoning learned this way transfers unevenly. A model trained on verifiable maths problems gets better at maths, and whether it gets better at anything else is an empirical question with different answers in different reports. Do not assume it generalises to your domain because it generalised to somebody’s benchmark; benchmark results transfer poorly in general and there is no reason this is an exception.

The verifier can still be gamed

An exact check removes the reward model as an attack surface and replaces it with a smaller one. The failures are concrete and every one of them has been seen in practice.

• Tests that pass without solving. Given the tests, a model can special-case them: hard-code the expected outputs, detect the test harness, or write a function that returns the right value for exactly the inputs tested. The mitigation is held-out tests the model never sees, which costs you the tests you were going to train on.
• Answer extraction exploits. If the checker takes the last number in the response, a model can emit several candidates and end with the most likely one. If it takes a boxed expression, the box can be filled with something that normalises to many things. Every extraction rule is a small parser and every parser has behaviour its author did not intend.
• Right answer, wrong reasoning. An outcome reward cannot distinguish a correct derivation from a lucky guess or a memorised answer, so both are reinforced. On problems that appeared in pretraining this is not a hypothetical — it is the connection to contamination, and it is an argument for holding out problems by source and date rather than at random.
• Format reward capture. Where a small reward is given for producing the required output format, and the task reward is rarely earned, the policy can converge on producing beautifully formatted wrong answers. The fix is to make the format reward small enough that it cannot compete, and to check that it is not the only thing rising.

The general principle carries over from reward hacking unchanged: the reward is whatever the program computes, and the program is shorter and more inspectable than a reward model but is not the same as your intention.

What has actually been published

This area moves fast and is widely misdescribed, so it is worth separating what labs have published from what is inferred about them.

• DeepSeek published its method. The DeepSeek-R1 paper (2025) describes reinforcement learning with rule-based rewards — accuracy checks on answers plus a format reward — using GRPO, and reports that reasoning behaviours emerged from that training rather than from demonstrations. It is the most detailed public account of the approach at scale.
• Allen Institute for AI published Tulu 3 (2024), including the RLVR formulation and its training recipe.
• Process supervision has been compared with outcome supervision. Lightman et al. (2023), “Let’s Verify Step by Step”, reported that rewarding each reasoning step outperformed rewarding only the final answer on a mathematics benchmark, at the cost of much more expensive labelling.
• OpenAI has not published the method behind o1 and its successors. The public description is that large-scale reinforcement learning improves the model’s chain of thought. The specifics — the reward, the algorithm, the data — have not been released, and confident third-party accounts of them are reconstruction, not documentation.

Extending it, honestly

The obvious question is what else can be verified, and the honest answer is: less than the enthusiasm suggests, but more than is currently exploited. Candidates where all four properties can hold with engineering effort:

1. Structured output conformance. Schema validation is an exact check, cheap, and produces unambiguous rewards. It is a narrow objective but a real one.
2. Query correctness against a database. Compare result sets rather than query text, and you have a verifier that does not care how the query was written.
3. Tool call validity and outcome. Whether an API call was well-formed and whether the resulting state matches an expected state are both checkable, given a test environment — which is what writing your own environment is for.
4. Self-consistency as a proxy. Where no reference answer exists, agreement across independent samples is sometimes usable as a weak signal. It is not verification — a model can be consistently wrong — and it should be treated as a noisy reward with a known bias towards whatever the model already believes.

The pattern in the successful cases is that somebody built a checker, and the checker is a piece of software with tests and a maintainer. If your plan for verifiable rewards does not include who owns the verifier and how it is validated, the plan is missing its main component.