Group-Relative Policy Optimisation
10 min read · updated August 4, 2026
GRPO is PPO with the value network deleted. Instead of learning a baseline, it samples several responses to the same prompt and uses their mean reward as the baseline. That one substitution removes a model from memory and introduces a constraint on which prompts can teach anything.
The one change
Group-relative policy optimisation was introduced in Shao et al. (2024), the DeepSeekMath paper, arXiv 2402.03300, and used in DeepSeek-R1 (2025) for reasoning training with rule-based rewards. The procedure per prompt:
- Sample a group of
Gcompletions from the current policy for the same prompt. Group sizes in published work are typically in the range of 8 to 64. - Score each with the reward function — a reward model, a verifier, or a rule.
- Compute each completion’s advantage relative to the group:
A_i = (r_i - mean(r)) / std(r). - Apply that advantage to every token of that completion, and run a PPO-style clipped update with a KL term against the reference model.
No value network is trained and none is loaded. The baseline that actor-critic methods spend a whole network learning is instead estimated from samples you were already generating.
Why a group mean is a valid baseline
The proof on the policy gradient page shows that subtracting any quantity that does not depend on the action leaves the gradient unbiased. The mean reward of a group of samples for the same prompt is a property of the prompt, not of any individual completion, so it qualifies.
It is also a better-matched baseline than a learned value function in this particular setting. The critic in RLHF has to predict, from a partial completion, the reward the finished response will receive — a hard regression problem on a moving target. The group mean measures the same quantity directly, from the same policy, at the same moment. Where samples are cheap relative to training a second network, that is a good trade.
It also sidesteps the cross-prompt scale problem. Because rewards are only ever compared within a group that shares a prompt, an easy prompt scoring 0.9 across the board and a hard prompt scoring 0.1 across the board both produce advantages centred on zero. The reward scale can be arbitrary per prompt and the algorithm does not care.
What it costs and what it saves
PPO for RLHF, models resident: policy trained full weights + optimiser state reference frozen weights only reward frozen weights only value trained full weights + optimiser state GRPO, models resident: policy trained full weights + optimiser state reference frozen weights only reward frozen weights only (or a verifier: no weights at all)
The saving is one trained model. That is more than a quarter of the memory, because a trained model carries optimiser state — with Adam, two additional tensors the size of the weights — while a frozen one does not. Dropping a trained model of the same size as the policy removes roughly as much memory as the policy’s own weights and optimiser state, and with a verifier instead of a reward model there is no third set of weights either.
The cost is generation. Every prompt now requires G completions instead of one, so the sampling phase is G times larger. The trade is therefore specific rather than universal: GRPO exchanges training memory and a hard regression problem for inference compute. It is a good exchange when generation is well optimised and memory is the binding constraint, which describes most large-model training clusters, and a poor one when generation dominates the step time.
When a group produces no signal at all
Here is the constraint that follows directly from the design and is usually left out. If every completion in a group gets the same reward, the mean equals every value, all advantages are zero, and the prompt contributes nothing to the gradient. With a binary verifier this happens whenever all G samples pass or all G fail.
Treating each sample as an independent draw with pass probability p, the probability that a group of G is unanimous — and therefore wasted — is p^G + (1-p)^G. At G = 8:
pass rate p P(all pass) P(all fail) wasted prompts
0.50 0.0039 0.0039 0.8%
0.70 0.0576 0.0001 5.8%
0.90 0.4305 0.0000 43.1%
0.95 0.6634 0.0000 66.3%
0.99 0.9227 0.0000 92.3%
0.10 0.0000 0.4305 43.1%
0.01 0.0000 0.9227 92.3%At a 99 per cent pass rate, 92 per cent of your generation budget produces no gradient. At a 1 per cent pass rate, the same. The useful range is narrow and centred on prompts the model gets right about half the time, which is a direct argument for difficulty-aware prompt selection — the subject of curriculum learning, and the reason it matters more here than in most training setups.
Two practical consequences. First, filtering out unanimous groups before the update saves nothing on generation but does keep the effective batch size honest, and a batch that looks like 256 prompts but contains 30 useful ones has a much noisier gradient than the number suggests. Second, a training set that was well matched at the start of a run becomes too easy as the policy improves, so the wasted fraction climbs over the course of training unless the prompt mix is refreshed.
Choosing the group size
G controls two things that pull in opposite directions, and both are calculable. A larger group makes the baseline less noisy — the standard error of a mean over G samples falls as 1 / sqrt(G) — and makes unanimous groups rarer. It also multiplies generation cost linearly.
Take a task the policy passes 80 per cent of the time and tabulate the wasted fraction and the cost:
p = 0.8 G P(unanimous) = p^G + (1-p)^G useful groups samples per useful group 4 0.4112 58.9% 6.8 8 0.1678 83.2% 9.6 16 0.0281 97.2% 16.5 32 0.0008 99.9% 32.0 64 0.0000 100.0% 64.0
The last column is the honest one: samples spent per group that actually produces a gradient. It is minimised at small G and rises steadily, so if all you want is some signal from as many prompts as possible, small groups are more compute-efficient. What larger groups buy is a lower-variance advantage on the groups you do use, and the freedom to train on prompts near the edges of the difficulty range.
The practical reading. Below about 4 the baseline is too noisy to be worth having — with G = 2 the group mean is the midpoint of two samples and the advantages are always plus and minus the same number, which discards the magnitude entirely. Above about 32 the returns are small and the generation bill is not. Published work sits in between for this reason rather than by convention, and the right value for you depends on your pass-rate distribution, which you can measure in an afternoon.
The objective, and what stayed from PPO
for each completion i in the group, for each token t: ratio = pi_theta(token) / pi_old(token) surrogate = min( ratio * A_i , clip(ratio, 1-eps, 1+eps) * A_i ) objective = mean over i, t of surrogate - beta * KL(pi_theta || pi_ref)
The clipped surrogate is unchanged from PPO and does the same job: it allows a batch of samples to be reused for several gradient passes without the policy drifting outside the region the samples describe. The KL term against the reference is also unchanged in purpose, though GRPO as published applies it as an explicit term in the objective rather than folding it into the per-token reward, which is a real difference in the gradient even though the intent is the same.
The advantage A_i is a single number per completion applied to every token in it. That is a strong assumption — every token of a correct answer is credited equally, including the ones that were irrelevant — and it is the same coarse credit assignment that outcome-supervised training has in general.
What is still being argued about
GRPO is recent enough that its details are actively contested, and this page is marked for refresh accordingly. Two normalisations in the original formulation drew scrutiny in follow-up work during 2025: dividing the advantage by the group standard deviation, which weights low-variance prompts more heavily and may bias the objective; and normalising by response length, which interacts with how long completions are credited. Several variants have been proposed that drop one or both.
What is not contested is the structural observation, and it is the one worth carrying away: a value network is one way to get a baseline, and if you can afford to sample several times from the same start state, you can measure the baseline instead of learning it. That idea is older than GRPO and applies well beyond language models.