Reward Models: Turning Preferences Into a Number

A reward model is a learned scoring function: text in, one number out, trained so that the number is higher for responses people preferred. It exists because people can reliably say which of two answers is better and cannot reliably write the best one.

What a reward model is, structurally

Take a language model, remove the head that projects the final hidden state onto the vocabulary, and replace it with a linear layer that projects onto a single number. Read that number off the final token of the response. That is the whole architecture.

Two consequences follow immediately. It is the same size and cost as the model it was built from, so a 70B reward model is a 70B model in your memory budget during training. And it inherits that model’s representations, which is why reward models are almost always initialised from a capable model rather than trained from scratch — the hard part, understanding the text, is already done.

The training data is pairs. Preference data is collected by showing an annotator one prompt and two responses and asking which is better. No score is ever assigned by a human. The scores are entirely an invention of the model, fitted so that their differences reproduce the choices.

The Bradley-Terry loss, with numbers

The standard model, from Bradley and Terry (1952), says the probability the winner is preferred is a logistic function of the score difference:

P(y_win preferred over y_lose)  =  sigmoid( r(y_win) - r(y_lose) )

loss for one pair  =  -log sigmoid( r(y_win) - r(y_lose) )

The only quantity that appears is the margin. Working the loss at several margins makes the shape of the gradient obvious:

margin   sigmoid(margin)   loss = -ln(sigmoid)
 +3.0        0.9526             0.049      already right; almost no gradient
 +2.0        0.8808             0.127
 +1.0        0.7311             0.313
  0.0        0.5000             0.693      no opinion; maximum uncertainty
 -1.0        0.2689             1.313
 -3.0        0.0474             3.049      confidently wrong; large gradient

Note where the gradient lives. A pair the model already gets right by a margin of 3 contributes almost nothing; a pair it gets confidently wrong dominates. That is usually what you want, and it is also why a small number of mislabelled pairs can do disproportionate damage — a wrong label looks exactly like a confidently-wrong prediction and receives the largest gradient in the batch. This is the reward model’s version of label noise.

Why the absolute number means nothing

Add 5 to every score the model produces. Every margin is unchanged, so every loss term is unchanged, so the training objective cannot tell the difference. The reward scale is unidentified: only differences are constrained by the data.

Several practical rules fall out of that one fact, and each of them is routinely broken.

• A reward of 2.3 is not good. It is not anything. Comparisons are only meaningful between responses to the same prompt, and only within one trained model.
• Two reward models are not comparable. Retrain with a different seed and the same responses get different numbers. A threshold tuned against one checkpoint is meaningless against the next.
• Cross-prompt comparison is undefined. An easy prompt and a hard prompt may sit on entirely different parts of the scale, so averaging reward over a mixed evaluation set measures the mix as much as the model. This is one reason GRPO normalises within a group of samples that share a prompt.

Some training setups add a small regularisation term to pin the mean reward near zero. That makes the numbers look stable; it does not make them mean anything more than they did.

The accuracy ceiling your data imposes

Reward models are usually reported by their accuracy on held-out preference pairs. That number has a ceiling well below 100 per cent, and you can derive it from how often two annotators agree with each other.

Assume a simple model of the data: a fraction q of pairs have a genuine consensus answer that any careful annotator would give, and the remaining 1 - q are effectively ties on which annotators choose at random. Then two independent annotators agree when the pair is a consensus one, or when it is a tie and they happen to coincide:

agreement  =  q  +  (1 - q) * 0.5

if agreement is measured at 0.75:
  0.75 = q + 0.5 - 0.5q   ->   0.5q = 0.25   ->   q = 0.50

a perfect predictor scores 1.0 on the consensus half and 0.5 on the ties:
  ceiling = 0.50 * 1.0  +  0.50 * 0.5  =  0.75

Under these assumptions the ceiling equals the agreement rate. So if your annotators agree with each other 75 per cent of the time, a reward model scoring 72 per cent on held-out pairs has captured almost all the available signal, and a model scoring 85 per cent has learned something about your annotation pipeline rather than about quality. Measure the agreement rate first; it is the only thing that tells you what a reward model number means. It is also the argument for spending on annotation quality before spending on reward model capacity.

Where the scale silently drifts

The reward model is trained on responses from one distribution — the starting policy’s. During RL the policy moves, and the further it moves the more it is asking the reward model to score text unlike anything it was fitted on. The scores keep coming back, confidently, and they stop tracking quality.

This is over-optimisation, and it has been characterised directly. Gao, Schulman and Hilton (2023), “Scaling Laws for Reward Model Overoptimization”, used a synthetic setup with a known “gold” reward to measure both at once: as the policy moves away from its starting point, the proxy reward from the trained model keeps rising while the gold reward peaks and then falls. Stiennon et al. (2020) had reported the same shape earlier on summarisation, where over-optimised policies scored well under the reward model and worse under human raters.

The KL penalty in PPO-based RLHF is the direct response: it holds the policy near the distribution the reward model understands. That makes the KL coefficient a strange hyperparameter — it is not tuning the objective, it is bounding how far you are allowed to trust your own measuring instrument.

The bias everybody finds first

Longer responses get higher reward. Singhal et al. (2023), “A Long Way to Go: Investigating Length Correlations in RLHF”, examined this directly and found reward strongly correlated with length, with a substantial share of the apparent gain from RLHF reproducible by simply making outputs longer.

Mechanically it is not mysterious. Annotators shown two answers, one thorough and one terse, often prefer the thorough one, and length is the easiest feature in the input correlated with that preference. The reward model learns the easy feature. The policy then optimises it, because that is its job.

The diagnostic is cheap and worth running on any reward model before it is used: score a held-out set, plot reward against token count, and compute the correlation. If it is strong, you know what your RL run is going to produce. Partial mitigations include length-balancing the preference pairs, subtracting a length term from the reward, and regressing reward on length and using the residual — none of them is a full fix, and all of them are better than discovering it after a training run.

What to check before trusting one

1. Held-out accuracy against the agreement ceiling. Not against 100 per cent. If you have not measured annotator agreement, you cannot interpret the accuracy at all.
2. Correlation with length, on held-out data. Then with any other cheap surface feature: markdown headings, bullet counts, hedging phrases, the presence of an apology.
3. Accuracy sliced by prompt category. An aggregate of 72 per cent can hide 90 per cent on chitchat and 55 per cent — barely above chance — on the technical prompts you actually care about.
4. Behaviour on out-of-distribution text. Score some deliberately strange inputs: empty responses, repeated tokens, text in another language. High confidence on nonsense predicts what happens when the policy discovers that nonsense.
5. Score drift over a training run. Log the mean reward of the policy’s samples and the KL from the reference together. Reward rising while KL rises steeply is the over-optimisation shape, and it is visible long before the outputs look wrong to a human.

Every one of those is an hour of work and each one has caught real problems. The alternative is finding out from the trained policy, by which point you have paid for the run and the failure is baked into a set of weights — the subject of reward hacking.