PPO: The Algorithm Behind Most RLHF

PPO is an actor-critic method with one addition: a clip that stops a single update from moving the policy further than the collected data can justify. Every design decision in it follows from that one concern, and the objective is much easier to read once you know which disaster it is guarding against.

The failure the clip exists to prevent

Policy gradient data is perishable. You run the current policy, collect a batch, and every advantage in that batch describes the world as seen by that policy. Update, and the batch is now a description of a policy you no longer have.

Usually that is a small inaccuracy. Occasionally it is a catastrophe. Suppose one action in the batch got a large positive advantage by luck. The gradient pushes its probability up hard; the step is large enough that the policy becomes something quite different; the new policy visits states that never appeared in the batch, so the critic’s values there are guesses; the next batch is collected under the damaged policy and reflects its behaviour rather than the good one you had.

There is no path back. Supervised learning recovers from a bad step because the dataset is unchanged and the next epoch sees the same correct labels. Reinforcement learning does not, because the data source is the thing that was damaged. This is the single most characteristic failure mode of the field, and it is why the useful question is not “which direction improves the objective” but “how far can we move before the estimate stops being trustworthy”.

The ratio, and why data goes stale

To reuse a batch collected under pi_old for a policy that has since moved, importance sampling reweights each sample by how much more or less likely the new policy is to have produced it:

r_t(theta)  =  pi_theta(a_t | s_t)  /  pi_old(a_t | s_t)

r = 1.0   the new policy would act identically here
r = 1.2   the new policy is 20% more likely to take this action
r = 5.0   the new policy is five times more likely -- this sample is now
          being counted five times, and it was only observed once

Importance sampling is exactly correct in expectation and increasingly useless in practice as the ratio moves away from 1, because the variance of a reweighted estimate grows with the weights. A handful of samples with large ratios come to dominate the batch. The estimate is unbiased and worthless — which is a specific, recognisable statistical situation and not a vague warning.

The clipped objective

PPO’s answer, from Schulman et al. (2017, arXiv 1707.06347), is to take the pessimistic view of every sample:

L(theta) = E[ min( r_t(theta) * A_t ,
                   clip(r_t(theta), 1 - eps, 1 + eps) * A_t ) ]

with eps typically 0.2

The min is doing something subtler than it looks. It is not symmetric clipping. It removes the incentive to keep moving in the direction you are already moving, once you are outside the trust region, while leaving the incentive to move back untouched. The result is a soft trust region that costs one comparison per sample — no second-order optimisation, no KL constraint solved per step, which is what made PPO practical where its predecessor TRPO was not.

The four cases, with numbers

Take eps = 0.2 and an advantage of magnitude 2, and evaluate both arms of the min.

Read the four rows together and the rule is: clipping only ever switches off the gradient for a sample the update has already moved far enough in the right direction. Nothing that needs correcting is ever clipped away. A reasonable sanity check while training is the fraction of samples being clipped — commonly reported as a clip fraction, and typically between a few per cent and about 0.3. Near zero means the steps are too small to need the mechanism; near one means almost every gradient is being switched off and the learning rate is far too high.

The hyperparameters that actually move things

PPO inside an RLHF pipeline

The version used for RLHF is the same algorithm in an unusual environment. The episode is one completion. The state is the prompt plus the tokens so far, the action is the next token, and the environment is a text buffer with no dynamics of its own.

Four models are resident. The policy is being trained. The reference is a frozen copy of the starting model. The reward model is frozen and scores completed responses. The value model is trained alongside the policy. That is roughly four times the memory of a fine-tune, plus a generation loop in the middle of training, and it is the practical reason lighter methods attract so much interest.

The reward is sparse in a specific way: the reward model scores the whole response, so the environment returns zero at every token except the last. On top of that, a per-token KL penalty against the reference model is added to the reward. This is the point most often confused, so it is worth stating flatly:

• The clip is a constraint against the previous iterate. It stops one optimisation step from being too large. It resets every time a new batch is collected.
• The KL penalty is an anchor to the starting model. It stops the policy drifting arbitrarily far from the supervised fine-tuned model over the whole run, which is what keeps the reward model inside the distribution it was trained on. It accumulates across the entire training run.

They solve different problems and you need both. Remove the clip and individual updates go wild; remove the KL and the policy walks somewhere the reward model scores highly and humans do not, which is reward hacking in its most predictable form.

Reading a PPO run from its metrics

PPO exposes an unusually informative set of diagnostics, and knowing what each one says turns a mysterious run into a readable one. These are the six worth plotting, and what a bad value of each means.

The pairing that resolves most confusion is entropy against clip fraction. High clip fraction with falling entropy is a policy being pushed too hard and collapsing; low clip fraction with flat entropy and flat reward is a policy that is barely moving, which is a learning rate problem and not a trust region one. The two look identical on a reward curve.

What PPO does not fix

PPO is a variance and step-size solution. It has nothing to say about three things that will decide whether your run works.

1. A wrong reward. PPO will optimise a bad reward function very effectively. The clip constrains how fast the policy moves, not where it is going.
2. Exploration. Nothing in PPO seeks out unvisited states. It improves what the current policy already does occasionally, which is fine for text and inadequate for problems where the reward is behind a door the agent has never opened.
3. Seed variance. Two identical runs with different random seeds can end at meaningfully different performance. Henderson et al. (2018), “Deep Reinforcement Learning that Matters”, documented this across algorithms and it has not stopped being true. Report several seeds or report nothing.

The direction the field moved after PPO is towards dropping pieces of it. DPO drops the reward model, the value model and the sampling loop. GRPO keeps the sampling and the clip and drops the value model, replacing the learned baseline with the mean reward of a group of samples for the same prompt.