DPO vs PPO vs GRPO: Preference Tuning Compared

All three take preference data and produce a model that behaves more like the preferred half. They differ in how much machinery stands between the preference and the gradient, and that difference is the entire practical decision.

The comparison, up front

The column that predicts your experience is the model count together with the sampling requirement. Two frozen-plus-trained models with no generation loop is a job an SFT harness can almost run. Four models plus generation is a distributed systems project.

PPO: the full apparatus

PPO learns a separate reward model from preferences, then optimises the policy against it with an on-policy RL algorithm, with a KL penalty holding the policy near its starting point. The value model exists to reduce variance: without a baseline, the advantage estimate for a given sample is just its raw reward, and the gradient is dominated by the fact that some prompts are simply easier than others.

What you get for the complexity is a reward signal that can be reused and interrogated. The reward model is a separate artefact — you can evaluate it on held-out preferences, look at what it scores highly, and reuse it across policy runs. You can also generate unlimited on-policy samples, which means the policy is being corrected on the mistakes it currently makes rather than on mistakes some earlier model made.

The costs are real: four models in memory, a generation loop in the training step, and a hyperparameter surface — clip range, KL coefficient, value loss coefficient, GAE lambda, rollout batch size — each of which can produce a run that trains smoothly to a worse model.

DPO: deleting the reward model

The DPO paper’s subtitle is the whole idea: Your Language Model is Secretly a Reward Model. The authors show that for the KL-constrained objective RLHF optimises, the optimal policy and the reward function are related in closed form — so the reward can be written in terms of the policy and reference log-probabilities, and substituted into the preference loss. The reward model cancels out.

implicit reward
  r̂(x, y)  =  β · log[ π_θ(y|x) / π_ref(y|x) ]

DPO loss
  L  =  − E[ log σ( r̂(x, y_w) − r̂(x, y_l) ) ]

...which is the Bradley-Terry preference loss from RLHF with the
learned reward model replaced by a ratio of log-probabilities the
policy already computes.

Everything follows from that substitution. No reward model to train, no value model to stabilise, no generation during training — the loss needs four forward passes over fixed text (policy and reference, on the chosen and rejected responses). It runs on the same infrastructure as SFT with roughly twice the memory.

What you lose is on-policy correction. DPO trains on a fixed dataset of pairs, so as the policy moves during training it drifts away from the distribution its data came from, and the pairs stop describing the mistakes it is currently making. The best-documented symptom is length: DPO-trained models have a well-known tendency to become more verbose, because in most preference datasets the longer response is more often the preferred one and the method has no counterweight. Park et al. (2024, arXiv 2403.19159) analyse exactly this and propose a length-regularised variant.

GRPO: deleting the value model

Group Relative Policy Optimization keeps the online RL structure and removes the value network. Instead of learning a baseline, sample G responses to the same prompt, score them all, and use the group’s own mean and standard deviation to normalise:

for one prompt x, sample  y_1 ... y_G  from the current policy
score each                r_1 ... r_G

advantage of sample i
  A_i  =  ( r_i − mean(r_1..r_G) ) / std(r_1..r_G)

no value network. the baseline is the group.

The introducing paper is DeepSeekMath, and the setting is the reason the method took hold: mathematical reasoning, where the reward can come from a verifier rather than a learned model. Checking whether a final answer is correct is a function, not a neural network, and a rule-based reward cannot be hacked in the way a learned one can. GRPO plus a verifier is a much shorter causal chain from “this answer is right” to a gradient than PPO plus a preference-trained reward model.

It still needs generation, and G samples per prompt rather than one, so the sampling cost per optimiser step goes up even as the memory for the value model goes away. It is best suited to tasks with a checkable answer, and it is not a drop-in replacement for preference tuning on open-ended helpfulness.

The data each one needs

• PPO — a preference dataset to train the reward model, plus a large set of prompts to generate against. The prompts need no labels, which is why the prompt set can be much bigger than the preference set.
• DPO — preference pairs only, and quality matters more than volume because there is no reward model to average out annotation noise. Pairs generated by the model you are about to train are worth substantially more than pairs from a different model.
• GRPO — prompts plus a scoring function. If that function is a verifier, you need no human preference data at all, which is the property that makes it attractive for maths, code and anything with a test suite.

Choosing