KL Divergence, and What Distance Between Distributions Means

KL divergence measures how many extra nats you pay for encoding data from distribution P using a code built for distribution Q. It is often described as a distance between distributions, and it is not one: computed both ways round on the same two distributions it gives different answers, which is demonstrated below with numbers rather than asserted.

What it measures

KL(P || Q) = sum_i P(i) * log( P(i) / Q(i) )

Read it as an expectation. For every outcome, take the log ratio of how likely P thinks it is to how likely Q thinks it is, and average those log ratios weighting by P. If the two distributions agree everywhere, every ratio is 1, every log is 0, and the divergence is 0. It is never negative.

There is a second reading that connects it to cross entropy directly, and it is the one worth remembering:

KL(P || Q) = H(P, Q) - H(P)

             cross entropy   minus   entropy of P
             of P under Q            (the unavoidable part)

Cross entropy is the total cost of encoding P with Q’s code. Entropy is the irreducible cost of encoding P at all. KL is the waste — the part that is your model’s fault. This also explains why minimising cross entropy and minimising KL are the same optimisation: H(P) does not depend on the model, so it is a constant that the gradient ignores.

Worked on two distributions

Three outcomes. P = [0.5, 0.3, 0.2] and Q = [0.4, 0.4, 0.2].

KL(P || Q), term by term:

i=1: 0.5 * ln(0.5 / 0.4) = 0.5 * ln(1.25)   = 0.5 * ( 0.223144) =  0.111572
i=2: 0.3 * ln(0.3 / 0.4) = 0.3 * ln(0.75)   = 0.3 * (-0.287682) = -0.086305
i=3: 0.2 * ln(0.2 / 0.2) = 0.2 * ln(1.00)   = 0.2 * ( 0.000000) =  0.000000
                                                                  ----------
                                                                   0.025267 nats

in bits: 0.025267 / 0.693147 = 0.036452 bits

Two things to notice. Individual terms can be negative — the second one is — but the total is guaranteed non-negative, which is Gibbs’ inequality. And the magnitude is small: 0.025 nats means these two distributions are very nearly the same, and encoding data from P with Q’s code wastes about a thirtieth of a bit per symbol.

Both ways round, and why they differ

KL(Q || P), same two distributions, reversed:

i=1: 0.4 * ln(0.4 / 0.5) = 0.4 * (-0.223144) = -0.089258
i=2: 0.4 * ln(0.4 / 0.3) = 0.4 * ( 0.287682) =  0.115073
i=3: 0.2 * ln(0.2 / 0.2) = 0.2 * ( 0.000000) =  0.000000
                                                ---------
                                                 0.025815 nats

KL(P || Q) = 0.025267
KL(Q || P) = 0.025815     different

The gap is small here because the distributions are close. Make them far apart and the asymmetry becomes dramatic:

P = [0.9, 0.1]   Q = [0.5, 0.5]

KL(P || Q) = 0.9*ln(0.9/0.5) + 0.1*ln(0.1/0.5)
           = 0.9*( 0.587787) + 0.1*(-1.609438)
           = 0.529008 - 0.160944 = 0.368064 nats

KL(Q || P) = 0.5*ln(0.5/0.9) + 0.5*ln(0.5/0.1)
           = 0.5*(-0.587787) + 0.5*( 1.609438)
           = -0.293893 + 0.804719 = 0.510826 nats

Ratio: 0.510826 / 0.368064 = 1.39x

The reason is structural, not accidental. The weighting is by the first distribution, so KL(P || Q) only cares about outcomes P considers likely. Where P is near zero, the term is near zero no matter how wrong Q is there. This gives the two directions their nicknames:

• KL(P || Q), forward, mode-covering. Punishes Q for assigning low probability anywhere P is high. A Q minimising this spreads out to cover all of P’s mass, including the gaps between modes.
• KL(Q || P), reverse, mode-seeking. Punishes Q for putting mass where P has none. A Q minimising this collapses onto one mode of P and ignores the rest.

That choice is a real design decision. Training a model on data is forward KL, which is why a language model will happily produce low-quality outputs that were rare in the data rather than refuse — it was optimised to cover the distribution, not to concentrate on its best parts.

The case where it is infinite

If Q(i) = 0 while P(i) > 0, the term is P(i) * ln(P(i) / 0) = P(i) * infinity. The whole divergence is infinite.

P = [0.5, 0.5]   Q = [1.0, 0.0]

KL(P || Q) = 0.5*ln(0.5/1.0) + 0.5*ln(0.5/0.0)
           = 0.5*(-0.6931)   + 0.5*(+inf)
           = inf

Reverse direction is finite:
KL(Q || P) = 1.0*ln(1.0/0.5) + 0.0*ln(0.0/0.5)
           = 0.6931 + 0        (0 * log 0 is defined as 0)
           = 0.6931 nats

One direction infinite, the other 0.69. This is not a curiosity; it is the source of a large fraction of NaN losses in real training code. The standard guards are to mix a small uniform component into Q, or to compute in log space so that a -inf logprob multiplied by a zero probability is caught explicitly rather than becoming 0 * -inf = nan.

Where it appears: RLHF, distillation, drift

Preference optimisation

When a model is tuned against a reward model, the objective is not the reward alone. It is the reward minus a penalty for moving away from the model you started with:

objective = E[ reward(x, y) ] - beta * KL( policy || reference )

Without that term the policy discovers whatever degenerate output the reward model overrates and produces it forever. The beta coefficient is the leash length: small and the model drifts far from fluent text, large and it barely changes at all. The KL here is computed per token between the tuned model’s distribution and the frozen reference model’s, over the same context. This is the structure of RLHF as normally described, and it is also what DPO reformulates away by folding the constraint into a closed-form loss.

Distillation

A student trained to match a teacher minimises KL(teacher || student) over the full output distribution rather than cross entropy against a one-hot label. The extra signal is in the 127,999 tokens the one-hot target throws away: the teacher saying “the answer is mat, but rug was nearly as good and Thursday was absurd” teaches more per example than the label alone.

Drift monitoring

Given two sets of production outputs — last month and this month — you can bin them into categories and compute the KL between the two histograms. It is a single number that says how much the output distribution moved. The same technique applies to embedding drift, though for symmetric monitoring most people prefer Jensen-Shannon divergence: the average of both KL directions against the mixture, which is symmetric and bounded above by ln(2).

Choosing a direction, and when not to use it

1. Decide which distribution is the reference. The reference goes on the left if you want to cover it, on the right if you want to stay inside it. RLHF puts the policy on the left precisely so the penalty grows when the policy puts mass where the reference does not.
2. Check for zeros in the right-hand distribution. Any zero where the left has mass gives infinity. Smooth, clip, or restrict to the shared support.
3. Fix the units. Nats or bits, and say which. A reported KL of 0.03 with no unit is uninterpretable.
4. Use a different measure if you need a metric. KL fails the triangle inequality and fails symmetry, so it cannot be used where a distance is required — clustering, nearest-neighbour search, embedding into a metric space. Use Jensen-Shannon, or total variation distance, or Wasserstein, and be explicit about which.