Cross-Entropy: What Training Actually Minimises

Cross-entropy loss is one number: the average surprise of the correct answer, measured in nats. If the model gave the right token a probability of 0.659, the loss on that token is -ln(0.659) = 0.417. Everything else — the summation sign, the one-hot vectors, the perplexity conversion — is bookkeeping around that.

Loss is surprise, and surprise is a logarithm

Start from what a good measure of surprise must do. Something certain should be worth zero surprise. Something impossible should be worth infinite surprise. And the surprise of two independent things happening should be the sum of their individual surprises, not the product, because that is how information adds up.

Exactly one family of functions has all three properties: -log(p). It is 0 at p = 1, unbounded as p approaches 0, and turns the product of independent probabilities into a sum.

p = 1.0   ->  -ln(1.0)  = 0.000     no surprise
p = 0.9   ->  -ln(0.9)  = 0.105
p = 0.5   ->  -ln(0.5)  = 0.693     one bit, in nats
p = 0.1   ->  -ln(0.1)  = 2.303
p = 0.01  ->  -ln(0.01) = 4.605
p = 0.001 ->  -ln(0.001)= 6.908

The base of the logarithm is a unit choice, not a mathematical one. Natural log gives nats, base 2 gives bits. Divide nats by ln(2) = 0.6931 to get bits. Frameworks report nats; papers often report bits; they are the same number in different currencies.

The full formula, and why it collapses

Written in general, cross-entropy between a true distribution y and a predicted distribution p over a vocabulary of size V is:

H(y, p) = - sum_{i=1..V} y_i * log(p_i)

In language model training, y is one-hot: the actual next token has y = 1 and every other token has y = 0. Every term where y_i = 0 contributes nothing. The sum over 128,000 terms collapses to one:

H(y, p) = - log(p_correct)

This is why you will see the same quantity called cross-entropy, negative log-likelihood and log loss depending on who is writing. With a one-hot target they are the same arithmetic. The general form matters again when the target is not one-hot — label smoothing, and distillation against a teacher’s full distribution — where the other 127,999 terms come back.

Three tokens, worked

Suppose a model reads “the cat sat on the” and the next three actual tokens get these probabilities from the model:

token      p(correct)   -ln(p)
-------    ----------   ------
" mat"       0.900      0.10536
" and"       0.500      0.69315
" Thursday"  0.100      2.30259
                        -------
sum                     3.10110
mean (3 tokens)         1.03370   <- the reported loss

perplexity = exp(1.03370) = 2.8115

Three things are visible here that a formula alone hides. First, the reported loss is a mean over tokens, so a long sequence of easy tokens can hide a few catastrophic ones. Second, one very surprising token dominates: the “Thursday” token contributes 74% of the total. Third, the loss is not bounded above, so a single token the model assigned 1e-9 would contribute 20.7 to the sum on its own.

That third point is why log(0) handling matters. An implementation that computes log(softmax(z)) in two steps can underflow to log(0) = -inf; one that computes log_softmax(z) directly — z_i - max(z) - log(sum(exp(z - max(z)))) — never forms the small probability at all. This is the same stability guard as the max-subtraction in softmax, one step later.

Why loss starts at 11.76

An untrained model has no information, so its best available answer is a uniform distribution over the vocabulary. If the vocabulary has V tokens, every token gets probability 1/V, and the loss is:

loss = -ln(1/V) = ln(V)

V = 32,000    ->  ln(32000)  = 10.373
V = 50,257    ->  ln(50257)  = 10.825
V = 128,000   ->  ln(128000) = 11.760
V = 256,000   ->  ln(256000) = 12.453

So the opening loss of a training run tells you the vocabulary size and nothing else. If your run starts at 11.76 with a 128k vocabulary, the initialisation is doing what it should. If it starts much lower, something is leaking the answer — a shifted target, a mask bug, an embedding accidentally sharing weights with the output head in a way that helps. If it starts much higher, the initial distribution is worse than uniform, which usually means the initialisation scale is wrong.

This is the single most useful sanity check in the whole of training, it costs one line, and it is derived rather than remembered.

The mean is over tokens, not sequences

Almost every implementation divides the summed loss by the number of tokens in the batch, not by the number of sequences. That choice has consequences most people never see stated, and one of them is a bug class that makes a training run look successful.

A batch of two documents:

  doc A:   10 tokens, mean loss 4.0  ->  sum  40
  doc B: 1000 tokens, mean loss 2.0  ->  sum 2000

Token-weighted (the normal choice):
  (40 + 2000) / 1010 = 2.0198

Sequence-weighted:
  (4.0 + 2.0) / 2 = 3.0

Same batch, two very different numbers. Under the
token-weighted mean, doc B contributes 99% of the
gradient, because it contributed 99% of the tokens.

That is usually what you want for pretraining, where each token is an equally valuable prediction. It is often not what you want for instruction tuning, where one long example should not count as a hundred short ones. Where it matters, the fix is to weight per sequence explicitly, and it should be a decision rather than a default nobody looked at.

The bug is in the denominator. Batches are padded to a common length, and padded positions must be excluded from both the sum and the count. Excluding them from the sum but not the count is a single missing mask and it is completely silent:

Batch padded to 500 positions per row, 100 of which are real.

  correct:  sum_real_loss / 100  = 2.00
  buggy:    sum_real_loss / 500  = 0.40

The loss curve looks superb. The gradients are
scaled down 5x, so the effective learning rate is
one fifth of what the config says, and training is
five times slower than it appears.

Sanity check: at step zero the loss must be ln(V).
0.40 against an expected 11.76 finds this instantly.

Which is why the step-zero check above is worth more than its one line suggests. Almost every masking, shifting and normalisation error in a training loop moves the initial loss away from ln(V), and nothing else in the run reports them.

Reading a training log

The conversion in the second column is perplexity, which is just exp(loss) and is easier to reason about because it has units of “how many options.” The conversion to bits is loss / 0.6931.

What a low loss does not tell you

• It is not comparable across tokenizers. A model with a tokenizer that produces fewer, longer tokens has a higher loss per token on identical text while being no worse. This is not a subtlety, it is a factor that regularly exceeds the differences people draw conclusions from, and it is worth a page of its own.
• It is not comparable across datasets. Loss on code is lower than loss on prose for almost every model, because code is more predictable. Two loss numbers from two corpora are two different quantities.
• It measures next-token prediction, not usefulness. Instruction tuning and preference optimisation frequently increase cross-entropy on the pretraining distribution while making the model far more useful. Loss and helpfulness stopped being the same axis the moment RLHF existed.
• The mean hides the tail. A model with a good average loss can be catastrophically surprised on a specific document type. Look at the distribution of per-token losses, not just their mean, if you want to know what the model does not know.