Logits, Odds and Log-Odds

A logit is the raw score a model produces for a token before anything normalises it. The name is not decoration: a logit is a log-odds, and taking that literally explains why an individual logit value tells you nothing while a difference of two logits tells you exactly how many times more likely one token is than another.

What a logit is, and what it is not

The final layer of a language model produces one real number per vocabulary entry. For a 128,000-token vocabulary that is 128,000 numbers per position, typically in the range roughly -20 to +20. They are unbounded, they can be negative, and they do not sum to anything. Softmax turns them into probabilities; until then they are just scores.

Logit means log-odds, and it means it literally

In the two-outcome case the relationship is exact and invertible:

logit(p) = ln( p / (1 - p) )
sigmoid(z) = 1 / (1 + exp(-z))

p = 0.500  ->  odds  1.000  ->  logit  0.000000
p = 0.800  ->  odds  4.000  ->  logit  1.386294
p = 0.900  ->  odds  9.000  ->  logit  2.197225
p = 0.990  ->  odds 99.000  ->  logit  4.595120
p = 0.999  ->  odds 999.00  ->  logit  6.906755
p = 0.100  ->  odds  0.111  ->  logit -2.197225

Check the inverse:
  sigmoid(1.386294) = 1 / (1 + exp(-1.386294))
                    = 1 / (1 + 0.25)
                    = 0.800

Two things the table shows. The scale is symmetric: p = 0.9 gives +2.197 and p = 0.1 gives -2.197. And it expands enormously near the ends — the distance from 0.99 to 0.999 is 2.3 in logit space, the same as the distance from 0.5 to 0.9. That is precisely what makes logits a good space to do arithmetic in: probabilities bunch up at the extremes and logits do not.

Only differences carry meaning

Softmax is unchanged by adding a constant to every logit, which is proved on the softmax page. So the absolute value of a logit is arbitrary: shift every logit at a position by +1000 and the model’s output distribution is identical.

What survives the shift is the difference, and it has a clean reading:

p_a / p_b = exp(z_a) / exp(z_b) = exp(z_a - z_b)

So a logit difference is a log odds ratio between two tokens.

  z_a - z_b = 0.0  ->  equally likely
  z_a - z_b = 1.0  ->  a is  2.72x as likely as b
  z_a - z_b = 2.0  ->  a is  7.39x
  z_a - z_b = 2.5  ->  a is 12.18x
  z_a - z_b = 5.0  ->  a is  148x
  z_a - z_b =10.0  ->  a is 22,026x

This is the number to reach for when comparing two candidate continuations. “The logit for yes was 12.3 and for no was 9.8” is not two facts, it is one: yes is 12.2 times as likely, and neither 12.3 nor 9.8 means anything on its own.

What a logit bias of +2 does, exactly

Logit bias adds a constant to one token’s logit before softmax. Because a logit is a log-odds, adding B multiplies that token’s odds by exp(B) — and that is checkable.

Start: z = [3.0, 2.0, 1.0, 0.0] for A, B, C, D

  exp = [20.0855, 7.3891, 2.7183, 1.0000]  sum 31.1929
  p   = [0.643914, 0.236883, 0.087144, 0.032059]

  Odds of C = 0.087144 / (1 - 0.087144)
            = 0.087144 / 0.912856
            = 0.095463

Apply logit_bias = +2 on token C: z = [3.0, 2.0, 3.0, 0.0]

  exp = [20.0855, 7.3891, 20.0855, 1.0000]  sum 48.5601
  p   = [0.413620, 0.152162, 0.413620, 0.020593]

  Odds of C = 0.413620 / 0.586380
            = 0.705378

  Ratio of odds: 0.705378 / 0.095463 = 7.3891
  And exp(2) = 7.3891.

The odds multiplied by exactly e^2. Not approximately.

So logit bias is an odds multiplier, and the useful mental conversion is: +1 is a 2.7-fold odds increase, +2 is 7.4-fold, +5 is 148-fold. Large negative values are how a token is suppressed — -100 makes it effectively impossible, and bans on specific tokens are usually implemented this way rather than by post-filtering.

The limit worth knowing: bias applies to single tokens, not to words or phrases. Suppressing a word that tokenises into three pieces requires biasing whichever token starts it, and that token also starts other words. Getting a token-level intervention to implement a word-level intention is harder than it first appears, and constrained decoding is the tool for anything more than one token.

Logits, logprobs and what an API returns

Almost no hosted API returns logits. They return logprobs — log(p) after softmax — which is a different quantity and a strictly less informative one, because the shift has already been resolved.

logprob -> probability:  p = exp(logprob)

  -0.0001  ->  0.99990
  -0.1054  ->  0.90000
  -0.6931  ->  0.50000
  -2.3026  ->  0.10000
  -4.6052  ->  0.01000
  -9.2103  ->  0.00010

Sequence probability is a sum in log space:

  20 tokens at -0.1 each
  total = -2.0
  P(sequence) = exp(-2.0) = 0.1353

  500 tokens at -0.1 each
  total = -50.0
  P(sequence) = exp(-50) = 1.9e-22

That last line is why sequence scores must be length-normalised before being compared. The mean logprob per token — here -0.1 regardless of length — is comparable across lengths; the total is not, and a search that maximises the total will always prefer the shortest candidate.

For the practical business of getting logprobs out of an API, converting them, and using them as a confidence signal, there is a page on exactly that. The caution that belongs here is a calibration one: a token probability of 0.99 says the model finds that continuation overwhelmingly likely under its training distribution, which is not the same as a 99% chance the sentence is true.

Calibration, and temperature scaling

A model is calibrated if, among the predictions it makes with confidence 0.8, about 80% turn out correct. Most trained classifiers are not: they are systematically overconfident, and the standard fix operates directly on the logits.

The measurement first. Bin predictions by confidence, compare the mean confidence in each bin to the actual accuracy in that bin, and average the gaps weighted by bin size. That is the expected calibration error.

bin        n     mean conf   accuracy   |gap|
--------  ----  ----------  ---------  ------
0.5-0.6    100     0.55       0.53      0.02
0.6-0.7    150     0.65       0.60      0.05
0.7-0.8    200     0.75       0.66      0.09
0.8-0.9    300     0.85       0.72      0.13
0.9-1.0    250     0.95       0.80      0.15
          ----
           1000

ECE = sum over bins of (n_b / n) * |gap|
    = 0.100*0.02 + 0.150*0.05 + 0.200*0.09
    + 0.300*0.13 + 0.250*0.15
    = 0.002 + 0.0075 + 0.018 + 0.039 + 0.0375
    = 0.1040

The model is overconfident by about 10 points on average,
and the error grows with confidence — the usual pattern.

Temperature scaling fixes this with one fitted parameter. Divide every logit by T before the softmax or sigmoid, choosing T on a held-out set. Because it divides all logits equally, it cannot change the ranking — accuracy is untouched, only the probabilities move.

A prediction at p = 0.90 that is right only 75% of the time.

  logit(0.90) = ln(0.9/0.1) = 2.197225
  target: logit(0.75) = ln(0.75/0.25) = ln(3) = 1.098612

  T = 2.197225 / 1.098612 = 2.0

Apply T = 2.0 across the range:

  p = 0.99  ->  z 4.5951 -> /2 = 2.2976 -> sigmoid = 0.9087
  p = 0.90  ->  z 2.1972 -> /2 = 1.0986 -> sigmoid = 0.7500
  p = 0.60  ->  z 0.4055 -> /2 = 0.2027 -> sigmoid = 0.5505
  p = 0.50  ->  z 0.0000 -> /2 = 0.0000 -> sigmoid = 0.5000

Everything moves toward 0.5, ordering unchanged,
and 0.5 is a fixed point because its logit is 0.

Three caveats worth stating. T is fitted on a validation set and only transfers to data of the same kind — a temperature fitted on one domain will not calibrate another. Temperature scaling corrects the overall confidence level and cannot fix a model that is overconfident on one class and underconfident on another; that needs a per-class or a more flexible mapping. And it does nothing whatever about a model that is confidently wrong in a correlated way, which is the failure mode behind a fluent, confident fabrication — there the probability is not miscalibrated so much as answering a different question from the one you wanted.

Note also that this is exactly the same arithmetic as the temperature parameter on a generation request. The operation is identical; the purpose is not. One is fitted to make probabilities honest, the other is set by hand to change how adventurous the sampling is.

Converting in both directions

import numpy as np

def softmax(z):
    z = np.asarray(z, dtype=np.float64)
    z = z - z.max()
    e = np.exp(z)
    return e / e.sum()

def log_softmax(z):
    z = np.asarray(z, dtype=np.float64)
    z = z - z.max()
    return z - np.log(np.exp(z).sum())

def logit(p):
    p = np.asarray(p, dtype=np.float64)
    return np.log(p / (1.0 - p))

def sigmoid(z):
    return 1.0 / (1.0 + np.exp(-np.asarray(z, dtype=np.float64)))

z = [3.0, 2.0, 1.0, 0.0]
p = softmax(z)
print(p)                       # [0.643914 0.236883 0.087144 0.032059]
print(np.exp(log_softmax(z)))  # identical
print(logit(0.8), sigmoid(1.3862944))   # 1.3862944 0.8

# recover logits from probabilities, up to a constant
recovered = np.log(p)
print(recovered - recovered.max() + max(z))   # [3. 2. 1. 0.]

The last two lines are the point of the section. You can recover the logits from the probabilities only up to an additive constant, because the constant was never in the probabilities to begin with. If you pin it by assuming the maximum logit was 3.0, you get the original vector back exactly — and if you cannot pin it, you have lost nothing that mattered.