Actor-Critic Methods: Two Networks, One Loop

An actor-critic method trains two things at once: a policy that acts, and a value function that predicts how well things are going. The value function is not there to choose actions. It is there to tell the policy which of its results were better than expected.

The problem the critic is hired to solve

A policy gradient scales each action’s gradient by the return that followed it. That return is a single sample of a random quantity, and its variance grows with the length of the episode and with the randomness of the environment. In a task with 400 steps and stochastic transitions, the same action in the same state can be followed by returns that differ by more than the effect of the action itself.

The consequence is not bias — the estimator is still correct on average — but a sample requirement that can be orders of magnitude larger than it needs to be. The critic attacks this directly, by answering the question the raw return answers badly: was this outcome better or worse than what we should have expected from here?

What the critic predicts, exactly

The critic estimates V(s): the expected discounted return from state s under the current policy. Three things about that definition are easy to get wrong.

• It is not a reward predictor. It predicts the sum of all future discounted rewards, which is why a state one move from a goal has high value and near-zero immediate reward.
• It is tied to the policy, not to the environment. Improve the actor and every value the critic has learned becomes wrong. The critic is chasing a target that the actor keeps moving, which is why the two learning rates interact.
• It never selects an action. Unlike Q-learning, where the value function is the policy through an arg-max, here the actor decides and the critic only comments.

Architecturally the two are usually one network with two heads sharing a trunk, which halves the compute and couples their gradients — a trade-off with a real failure mode, covered below.

The advantage, on a three-step episode

The advantage of an action is how much better the outcome was than the critic expected. The one-step version is the temporal-difference error:

delta_t  =  r_t  +  gamma * V(s_t+1)  -  V(s_t)

Take a concrete episode. Three steps, reward only at the end, gamma = 0.99, and a critic that currently predicts the values shown. The terminal state has value 0 by definition.

step   state    reward    V(state)
  0      s0       0.0       0.50
  1      s1       0.0       0.70
  2      s2       1.0       0.90
  -      s3        -        0.00   (terminal)

delta_0 = 0.0 + 0.99*0.70 - 0.50 = 0.693 - 0.50 = 0.193
delta_1 = 0.0 + 0.99*0.90 - 0.70 = 0.891 - 0.70 = 0.191
delta_2 = 1.0 + 0.99*0.00 - 0.90 = 1.000 - 0.90 = 0.100

Every delta is positive, which says the episode went slightly better than the critic expected at each step. Notice the scale: the raw returns are around 0.98, and the advantages are around 0.15. The critic has removed the common-mode signal, which is exactly what a baseline is for.

One knob between two extremes

The one-step delta is low variance and biased — it trusts V(s_t+1), which is wrong early in training. The full Monte Carlo return is unbiased and high variance. Generalised advantage estimation (Schulman et al., 2015, arXiv 1506.02438) interpolates with a single parameter, computed by a backwards recursion:

A_t  =  delta_t  +  gamma * lambda * A_(t+1)         (A after the last step is 0)

Running that on the episode above, at three values of lambda:

lambda = 0     (one-step TD)
  A_2 = 0.100
  A_1 = 0.191
  A_0 = 0.193

lambda = 0.95  (the usual default; gamma*lambda = 0.9405)
  A_2 = 0.100
  A_1 = 0.191 + 0.9405*0.100 = 0.285
  A_0 = 0.193 + 0.9405*0.285 = 0.461

lambda = 1     (Monte Carlo; gamma*lambda = 0.99)
  A_2 = 0.100
  A_1 = 0.191 + 0.99*0.100  = 0.290
  A_0 = 0.193 + 0.99*0.290  = 0.480

check against the raw return at lambda = 1:
  G_0 = 0 + 0.99*0 + 0.99^2 * 1 = 0.9801
  G_0 - V(s0) = 0.9801 - 0.50   = 0.4801   -- matches

The spread at step 0 is 0.193 to 0.480 — a factor of two and a half, from one hyperparameter, on the same episode with the same critic. At lambda = 0 the first action gets almost no credit for the reward three steps later, because all of that credit is being carried by the critic’s estimate of V(s1). At lambda = 1 the critic is used only as a baseline and the actual reward does the work.

The default of 0.95 is not arbitrary: it discounts the credit chain by about five per cent per step on top of gamma, giving an effective credit horizon of roughly 1 / (1 - gamma*lambda), which here is about 17 steps. If your episodes are much longer than that, the critic is doing most of the credit assignment and its quality matters more than the reward signal’s.

The loop, and the two losses

1. Run the current policy for a batch of steps — typically 2,048 or more, from several environments in parallel — recording states, actions, rewards and the critic’s value at each step.
2. Compute the advantages backwards through the batch with the recursion above, then the value targets as A_t + V(s_t).
3. Update the actor with the policy gradient, using the advantages as the scaling term. Detach them: they are data.
4. Update the critic by regressing V(s_t) onto the value targets, usually with mean squared error.
5. Optionally add an entropy bonus to the actor loss, which penalises the policy for becoming too confident too early and is often the difference between learning and immediate collapse.

Advantage normalisation — subtracting the batch mean and dividing by the batch standard deviation — is applied almost universally at step 3. It makes the effective learning rate independent of the reward scale, which is why you can change a reward from 1 to 100 and see almost no difference in training dynamics.

How it fails, and what the symptom looks like

Two of those four are diagnosable only if you log policy entropy and the critic’s explained variance alongside the return. Explained variance is the single most useful critic diagnostic and it is one line: 1 - Var(returns - values) / Var(returns). At 1 the critic predicts the returns perfectly; at 0 it is no better than predicting the mean; below 0 it is actively misleading and your advantages are noise with a sign. Logging the return alone tells you that something is wrong and nothing about what, which is the most common way an actor-critic debugging session becomes a week long.

Where the critic goes in language model training

In RLHF with PPO, the critic is the value model: a fourth network alongside the policy, the frozen reference and the reward model. Its job is to predict, from a half-written response, the reward the finished response will receive.

That is a harder regression problem than it sounds, for a reason specific to text. The critic must judge a partial completion, and the quality of a partial completion is often not determined yet — a response that has written two sentences of preamble could still go either way. So the critic is being asked to predict a quantity that genuinely has high variance at the point it is asked, and it is being asked at every token.

It is also expensive in an unusually visible way. The value model is typically initialised from a model of comparable size to the policy and is trained, so it carries optimiser state as well as weights. Dropping it is therefore worth roughly as much memory as the policy itself, which is exactly the trade GRPO makes: sample several completions for the same prompt and use their mean reward as the baseline, so the number the critic was learning is measured instead.

The general lesson holds outside language models. A learned baseline is worth its cost when samples are expensive and states are rarely revisited. When you can cheaply generate several episodes from the same start state, a sample mean is an unbiased baseline that needs no training and cannot lag the policy. Ask which of those two situations you are in before adding a second network.

The direct descendant of all this is PPO, which is an actor-critic method plus one modification to the actor loss that lets a batch be reused for several gradient steps instead of one.