RLHF Explained Without the Jargon

Reinforcement learning from human feedback exists because of one gap: people can reliably say which of two answers is better, and cannot reliably write the best answer. RLHF is the machinery for turning the first ability into training signal.

The problem SFT cannot solve

Supervised fine-tuning needs a target. For “extract these fields as JSON” that is easy. For “be helpful without being sycophantic”, “refuse this politely but do not moralise”, or “explain this at the right level of detail”, writing the target is the hard part, and different annotators will write different targets that both look right.

Comparison is much easier than composition. Shown two answers, an annotator picks the better one quickly and with decent agreement. RLHF is built on that asymmetry: collect comparisons, learn a function that predicts them, and optimise the model against that function.

The three stages, in order: supervised fine-tuning to get a usable starting policy; a reward model trained on pairwise preferences; and a reinforcement learning phase that updates the policy to score higher under the reward model. Christiano et al. (2017) established the approach on control tasks, Stiennon et al. (2020) applied it to summarisation, and Ouyang et al. (2022) — the InstructGPT paper — is the standard reference for the language model pipeline.

Stage two: the reward model

The reward model is a language model with its next-token head replaced by a scalar head. Feed it a prompt and a response, get one number. It is initialised from a model of the same family, often the SFT model itself, so it starts out understanding the text it is scoring.

The training data is pairs: one prompt, two responses, a human judgement of which is better. Note what is not collected — no absolute scores, no ratings out of ten. Absolute ratings drift between annotators and within one annotator over a day. Pairwise comparisons are stable, and the model recovers a consistent scale from them.

The preference loss, written out

The reward model is trained under the Bradley-Terry model of pairwise comparison, which says the probability a human prefers response y_w to y_l is the logistic function of the difference in their rewards:

P(y_w  preferred over  y_l | x)  =  σ( r(x, y_w) − r(x, y_l) )

loss  L(r)  =  − E[ log σ( r(x, y_w) − r(x, y_l) ) ]

  σ      logistic sigmoid
  r      the reward model being trained
  y_w    the response the human chose ("won")
  y_l    the response the human did not ("lost")

Two consequences fall straight out of that expression, and both matter more than the formula itself.

First, only the difference in rewards is constrained. Add a constant to every reward and the loss is unchanged, so the absolute scale is arbitrary — a reward of 3.2 means nothing on its own, and comparing reward values across two separately trained reward models is meaningless.

Second, the reward model is only trained on the distribution of responses it was shown. Ask it to score something far outside that distribution and it will confidently return a number that is not grounded in anything. That is the seed of reward hacking, and it is why stage three needs a leash.

Stage three: the policy update

Now the loop. Sample a prompt, have the current policy generate a response, score it with the reward model, and update the policy to make high-scoring responses more likely. The algorithm used in the InstructGPT pipeline is Proximal Policy Optimization (Schulman et al., 2017), which constrains how far the policy can move in a single update so that one high-variance reward estimate cannot wreck it.

This is the expensive stage, and the reason is structural rather than incidental. Four models are involved:

• The policy — being trained, needs gradients and optimiser state.
• The reference policy — a frozen copy of the SFT model, used for the KL term below.
• The reward model — frozen, run on every sample.
• The value model — a learned baseline that estimates expected return, so the advantage estimate is not pure noise. Also trained, so also carrying optimiser state.

And every step requires actually generating text, which is the slow autoregressive path rather than a single teacher-forced forward pass. Four models resident plus online sampling is the cost that motivated everything in the direct-preference methods.

Why the KL penalty is not optional

Optimise a policy against a learned reward function hard enough and it finds the places where the reward function is wrong. The reward model is an approximation fitted to a finite sample; somewhere in the space of possible strings there is text that scores 9.8 and is gibberish, and gradient ascent is extremely good at finding it. The classic manifestations are degenerate: the model discovers that longer answers score higher, or that a particular opening phrase does, and produces nothing else.

The fix is to penalise the policy for drifting from the reference:

reward used by PPO
  R(x, y)  =  r(x, y)  −  β · KL( π_θ(y|x) ‖ π_ref(y|x) )

  r       the learned reward model
  π_θ     the policy being trained
  π_ref   the frozen SFT model
  β       how hard the leash pulls

β is the central trade-off of the whole method. Too small and the policy wanders into the reward model’s blind spots. Too large and it cannot move far enough to learn anything, and you have spent four models’ worth of memory to reproduce the SFT checkpoint. In practice teams watch the KL divergence as a first-class training metric alongside the reward, because reward rising while KL rises faster is the signature of hacking rather than improvement.

What it bought, in published terms

The single most-cited result from the InstructGPT paper is a size comparison: human labellers preferred outputs from the 1.3B-parameter InstructGPT model to outputs from the 175B-parameter GPT-3, despite the former having over a hundred times fewer parameters. Whatever RLHF is doing, it is not something more pretraining compute obviously substitutes for.

The same paper is candid about the cost, and names it the alignment tax: performance on some standard NLP benchmarks regressed after the RL stage. Their mitigation was to mix pretraining gradients back into the RL updates — the variant they call PPO-ptx. That regression is the general phenomenon covered in catastrophic forgetting, showing up in the place it is best documented.