Perplexity: Reading the Number Properly

Perplexity is exp(mean negative log-likelihood) — literally the exponential of the cross-entropy loss you already have. It is a useful number for tracking one model on one dataset and a meaningless one for comparing two models with different tokenizers, and the second half of that sentence is what this page proves with arithmetic.

The definition, and its exact relation to loss

PPL = exp( -(1/N) * sum_{i=1..N} ln p(token_i | context_i) )

which is exactly

PPL = exp(cross_entropy_loss)          when loss is in nats
PPL = 2 ^ (cross_entropy_loss_in_bits) when it is in bits

There is nothing else to it. If your training log says the loss is 2.0, the perplexity is e^2.0 = 7.389. If a paper reports perplexity 12.18, the loss was ln(12.18) = 2.500. The two numbers carry identical information and the conversion is one call.

loss 1.50  ->  PPL  4.482
loss 1.80  ->  PPL  6.050
loss 2.00  ->  PPL  7.389
loss 2.50  ->  PPL 12.182
loss 3.00  ->  PPL 20.086
loss 11.76 ->  PPL 128,000    (uniform over a 128k vocabulary)

That last line is the anchor. A model that has learned nothing has a perplexity equal to its vocabulary size, which follows directly from the loss of a uniform distribution being ln(V).

Why it is called a branching factor

A uniform distribution over k options has perplexity exactly k. So perplexity answers: “this model is as uncertain as somebody choosing uniformly among how many options?”

Uniform over 4 options:
  p = 0.25 for each
  loss = -ln(0.25) = 1.3863
  PPL  = exp(1.3863) = 4.0

A perplexity of 7.39 means the model is, on average, as uncertain as a fair seven-and-a-bit-sided die at each token. That is a genuinely useful intuition, and it is why perplexity survives alongside loss despite carrying no extra information: humans reason better about “seven options” than about “2.0 nats.”

The intuition also warns you when a number is implausible. A perplexity below about 2 on natural language held-out text means the model is choosing between fewer than two options per token on average, which prose does not permit — so the evaluation set is contaminated, or the loss is being computed over the wrong positions.

The same model, two tokenizers, two answers

This is the part almost every explanation omits, and it is the reason perplexity comparisons across model families are worthless.

Take one document. A model assigns it some total log-probability. That total is a property of the string: whatever way you cut the string into tokens, a well-formed model is assigning probability mass to the same sequence of characters, so the total negative log-likelihood of the document is approximately the same either way. Call it 2,000 nats.

Now tokenise it two ways. Tokenizer A produces 1,000 tokens. Tokenizer B has a better-fitted vocabulary and produces 800.

Document total NLL: 2000 nats (same model, same string)

Tokenizer A:  N = 1000 tokens
  mean loss = 2000 / 1000 = 2.000
  PPL       = exp(2.000)  = 7.389

Tokenizer B:  N = 800 tokens
  mean loss = 2000 / 800  = 2.500
  PPL       = exp(2.500)  = 12.182

Same model. Same text. Same total uncertainty.
Perplexity is 65% higher for the better tokenizer.

Read that last line again. Tokenizer B compresses the text more efficiently, which is unambiguously the better tokenizer, and it makes the perplexity look 65% worse. The denominator changed and the numerator did not.

This is not a small effect at the margins. Token counts for identical text differ substantially between model families, and for non-English text the differences are larger still. A perplexity table comparing models with different tokenizers is measuring tokenizer vocabulary design at least as much as it is measuring modelling quality, and there is no way to tell the two apart from the table.

Bits per byte, which does compare

The fix is to normalise by something both models agree on. The text has a fixed number of bytes (or characters) regardless of how anyone chose to tokenise it, so divide by that instead.

bits_per_byte = total_NLL_in_nats / ln(2) / n_bytes

Document: 2000 nats, 5000 bytes.

  2000 / 0.693147 = 2885.39 bits
  2885.39 / 5000  = 0.5771 bits per byte

Tokenizer A: same 2000 nats, same 5000 bytes -> 0.5771
Tokenizer B: same 2000 nats, same 5000 bytes -> 0.5771

Identical, which is the point.

Bits per byte is comparable across tokenizers, across vocabularies, and across languages. It is also directly interpretable: it is the size of the file if you compressed it optimally with this model as the probability source, and a value of 0.5771 bits per byte means an 8x compression ratio against raw bytes.

1. Score the evaluation text and accumulate the total negative log-likelihood in nats. Do not average yet.
2. Count the bytes of the original text, before tokenisation, in the same encoding for every model you are comparing. UTF-8 unless you have a reason.
3. Divide the total by ln(2) to get bits, then by the byte count.
4. Report bits per byte alongside perplexity, never perplexity alone, whenever more than one tokenizer is involved.

The other four ways a perplexity comparison lies

• Different evaluation corpora. Perplexity on code is far lower than on prose for nearly every model, because code is more predictable. Two perplexity numbers from two corpora are two different quantities with the same name.
• Different context lengths. Scoring a document in 512-token windows gives a higher perplexity than scoring it in 4,096-token windows, because tokens near the start of each window have less context. The stride used for the sliding window is a parameter of the measurement and is almost never reported.
• Contamination. If the evaluation text was in the training data, the perplexity is a measurement of memorisation. This is the single most common reason for an implausibly good number, and it is difficult to rule out for any public corpus.
• It measures the pretraining objective, not the product. An instruction-tuned model typically has worse perplexity on raw web text than the base model it came from, while being far more useful. Perplexity stopped tracking usefulness around the same time chat models did.

What perplexity is still good for

One model, one fixed evaluation set, one fixed tokenizer, tracked over time. In that setting every confound above is held constant and the number does exactly what you want: it goes down when the model gets better at predicting that text, and it is sensitive enough to notice small changes.

It is also a good regression alarm. Quantise a model and re-measure perplexity on the same held-out set with the same harness: a jump from 7.39 to 7.6 is a small quality cost, a jump to 11 means something broke. That comparison is valid precisely because everything except the weights stayed the same, and it is the honest version of the claim quantisation write-ups usually make. For anything user-facing you still want a task benchmark, because a quantised model can hold its perplexity and lose instruction-following badly.