Entropy and Information, With Tokens as the Example

Entropy is the average number of yes-or-no questions you need to pin down an outcome, given what you already know about how likely each outcome is. A fair coin is 1 bit. A coin that lands heads 90% of the time is 0.469 bits. Every number on this page is computed from those two sentences.

A bit is an answer to a yes-or-no question

Suppose there are eight equally likely outcomes. Ask “is it in the first four?”, then “is it in the first two of those?”, then one more. Three questions, eight outcomes, log2(8) = 3 bits. That is the whole intuition, and it generalises to unequal probabilities by letting the questions be uneven too: put the likely outcomes behind fewer questions and the average drops.

Surprise of one outcome:  -log2(p)

p = 1.00  ->  0.000 bits   (told you nothing)
p = 0.50  ->  1.000 bits
p = 0.25  ->  2.000 bits
p = 0.10  ->  3.322 bits
p = 0.01  ->  6.644 bits

Entropy is the average of that column, weighted by how often each outcome actually happens.

The formula, and the one-symbol case

H(X) = - sum_i p_i * log2(p_i)      bits
H(X) = - sum_i p_i * ln(p_i)        nats

Fair coin:      -0.5*log2(0.5) - 0.5*log2(0.5)
                = 0.5 + 0.5 = 1.000 bits

Biased 90/10:   -0.9*log2(0.9) - 0.1*log2(0.1)
                = 0.9*0.152003 + 0.1*3.321928
                = 0.136803 + 0.332193
                = 0.468996 bits

The biased coin carries less than half the information of the fair one, because most of the time it tells you what you already expected. A coin that always lands heads has entropy 0: 1*log2(1) = 0, and the convention 0 * log(0) = 0 handles the impossible outcome. Uncertainty is maximised by the uniform distribution, always, for any number of outcomes.

Worked on eleven characters

Take the string abracadabra. Eleven characters, five distinct.

counts:  a=5  b=2  r=2  c=1  d=1     total 11

p(a) = 5/11 = 0.454545    log2(5/11) = -1.137504
p(b) = 2/11 = 0.181818    log2(2/11) = -2.459432
p(r) = 2/11 = 0.181818    log2(2/11) = -2.459432
p(c) = 1/11 = 0.090909    log2(1/11) = -3.459432
p(d) = 1/11 = 0.090909    log2(1/11) = -3.459432

H = 0.454545 * 1.137504 = 0.517047
  + 0.181818 * 2.459432 = 0.447169
  + 0.181818 * 2.459432 = 0.447169
  + 0.090909 * 3.459432 = 0.314494
  + 0.090909 * 3.459432 = 0.314494
                          --------
                          2.040373 bits per character

Two point zero four bits per character. Check it against the boundaries: five distinct symbols used uniformly would be log2(5) = 2.322 bits, and this string is below that because a is over-represented. Using only one symbol would be 0. 2.04 sits where it should.

Consequence: the whole string needs at least

  11 characters * 2.040373 bits = 22.44 bits = 2.81 bytes

An optimal code stores "abracadabra" in 3 bytes.
ASCII stores it in 11.

That is the operational meaning of entropy: a lower bound on the size of any lossless encoding. No compressor can beat it, and this is why compression ratio and modelling quality are the same problem viewed from two sides — which is the connection bits per byte exploits.

Entropy of a token distribution

At each generation step a model produces a distribution over its whole vocabulary. The entropy of that distribution is a per-step measure of how uncertain the model is, and it is computable from the top few logprobs that most APIs will return.

Model's top-4 probabilities at one position:

  " mat"   0.6590    -log2 = 0.6017    contribution 0.3965
  " rug"   0.2424    -log2 = 2.0446    contribution 0.4956
  " sofa"  0.0986    -log2 = 3.3427    contribution 0.3296
  " floor" 0.0000+   (tail)            approx 0
                                       ------
                                       1.2217 bits

Compare against the ceiling for a 128,000-token vocabulary:
  log2(128000) = 16.966 bits

The model has removed about 15.7 bits of uncertainty
from this position using its context.

Per-token entropy is a directly useful signal. High entropy means the model has many plausible continuations, which is where sampling parameters actually change the output and where a confident-sounding fabrication is most likely. Low entropy means the continuation was forced and temperature is doing nothing. Averaging entropy across a generated response gives a cheap, model-internal confidence proxy that needs no second call.

The caveat: entropy over the top-k returned logprobs underestimates the true entropy, because the truncated tail carries real mass. With k = 5 on a flat distribution the underestimate is large. Renormalise the returned probabilities and say in your metric name that it is a top-k entropy, not the entropy.

Conditional entropy and mutual information

Everything above measured one variable in isolation. The quantities that matter for a model measure how much knowing one thing tells you about another, and they are built from the same sum.

A joint distribution over two binary variables.
X: does the context contain a date?
Y: is the next token a number?

              Y=0    Y=1     P(X)
      X=0     0.40   0.10    0.50
      X=1     0.10   0.40    0.50
      P(Y)    0.50   0.50

Marginals are both uniform:
  H(X) = 1.000000 bits
  H(Y) = 1.000000 bits

Joint entropy, over all four cells:
  H(X,Y) = -2*(0.4 * log2 0.4) - 2*(0.1 * log2 0.1)
         =  2*(0.4 * 1.321928) + 2*(0.1 * 3.321928)
         =  1.057542 + 0.664386
         =  1.721928 bits

If the two were independent the joint entropy would be 1 + 1 = 2 bits. It is 1.72, and the missing 0.28 bits is exactly the amount that knowing one variable saves you on the other.

Conditional entropy:
  H(Y | X) = H(X, Y) - H(X) = 1.721928 - 1.000000 = 0.721928 bits

Check it directly. Given X=0, the conditional
distribution over Y is [0.8, 0.2]:
  H = 0.8*log2(1/0.8) + 0.2*log2(1/0.2)
    = 0.8*0.321928 + 0.2*2.321928
    = 0.257542 + 0.464386
    = 0.721928 bits

Same for X=1 by symmetry, so the average is 0.721928.
Both routes agree.

Mutual information:
  I(X;Y) = H(Y) - H(Y|X) = 1.000000 - 0.721928 = 0.278072 bits
         = H(X) + H(Y) - H(X,Y) = 1 + 1 - 1.721928   same number

Mutual information is symmetric, non-negative, and zero exactly when the two variables are independent. It is the amount of uncertainty about one that the other removes, and 0.278 bits out of 1 means the context resolves about 28% of the question.

This is what a language model does, restated: it drives H(next token | context) as low as it can. The vocabulary ceiling is log2(128,000) = 16.97 bits, and a competent model operates around 2.5 bits, so it is extracting roughly 14.5 bits of mutual information between the context and the next token at each step. Every architectural improvement is a claim about increasing that number.

Mutual information is also directly useful as a feature-selection tool: compute it between a candidate metadata field and the outcome you care about, and a value near zero says the field carries no information about the outcome, no matter how sensible it looks. The caution is that it is a measure of dependence in general, not of a usable relationship, and it is biased upward on small samples — with few examples per cell, unrelated variables show positive mutual information simply from sampling noise.

Computing it on your own text

Character-level entropy of any file, with no dependencies:

import sys, math
from collections import Counter

data = open(sys.argv[1], "rb").read()
counts = Counter(data)
n = len(data)

H = -sum((c / n) * math.log2(c / n) for c in counts.values())

print(f"bytes          : {n}")
print(f"distinct       : {len(counts)}")
print(f"entropy        : {H:.4f} bits per byte")
print(f"lower bound    : {n * H / 8:.0f} bytes compressed")
print(f"ratio          : {8 / H:.2f}x")

Run it on English prose and you will see roughly 4.0 to 4.5 bits per byte. Run it on JSON and you will see less, because the punctuation repeats. Run it on a compressed archive and you will see close to 8.0, because a good compressor’s output looks random by construction — which is a neat way to verify the tool is measuring what it claims to.

For per-token entropy from a model, request logprobs with a top-k of at least 5, convert each with p = exp(logprob), renormalise so they sum to 1, and apply the same sum. The arithmetic is identical; only the source of the probabilities changed.

Entropy, cross entropy and the gap between them

Three quantities, one relationship, and it is the sentence that makes all three make sense:

H(P)     entropy of the true distribution
           the irreducible uncertainty

H(P, Q)  cross entropy: cost of encoding P using Q's code
           what a model's loss actually measures

KL(P||Q) = H(P, Q) - H(P)
           the waste, which is the model's fault

Therefore:  H(P, Q) >= H(P),  always.

A model’s cross-entropy loss can never go below the entropy of the language it is modelling. There is a floor, and it is not zero. Training runs that appear to approach zero loss on held-out text have not discovered something about language; they have discovered that the held-out text was in the training set.

The gap between the loss and that floor is exactly KL divergence, which is the only part any amount of engineering can remove.