Sampling From a Probability Distribution, Concretely

Sampling takes a probability distribution and one random number between 0 and 1, and returns a token. Every parameter you can set — temperature, top-k, top-p, min-p — modifies the distribution before that draw happens, and this page applies all four to the same four tokens so you can follow each one.

How a random number becomes a token

Build the cumulative sum of the probabilities, draw a uniform random number, and take the first token whose cumulative total exceeds it.

Tokens and probabilities:

  A  0.50    cumulative 0.50
  B  0.30    cumulative 0.80
  C  0.15    cumulative 0.95
  D  0.05    cumulative 1.00

Draw u ~ Uniform(0, 1):

  u = 0.12  ->  0.12 < 0.50           -> A
  u = 0.63  ->  0.50 < 0.63 < 0.80    -> B
  u = 0.91  ->  0.80 < 0.91 < 0.95    -> C
  u = 0.97  ->  0.95 < 0.97 < 1.00    -> D

Each token occupies an interval of [0, 1) exactly as wide as its probability, so a uniform draw lands in it exactly that often. The method is called inverse-CDF sampling and it is the whole mechanism. Everything else is preprocessing.

The random number comes from a pseudorandom generator with a seed. Fixing the seed fixes the sequence of draws, which is why a seeded request is reproducible in principle — and why it is often not reproducible in practice, since batching changes the floating-point reduction order upstream of the draw.

Temperature, applied first

Start from logits rather than probabilities, because temperature acts on logits. Take z = [3.0, 2.0, 1.0, 0.0] for tokens A, B, C, D.

T = 1.0   (divide by 1, i.e. do nothing)
  exp(3.0) = 20.0855
  exp(2.0) =  7.3891
  exp(1.0) =  2.7183
  exp(0.0) =  1.0000
  sum      = 31.1929
  p = [0.6439, 0.2369, 0.0871, 0.0321]

T = 0.5   logits become [6.0, 4.0, 2.0, 0.0]
  exp = [403.4288, 54.5982, 7.3891, 1.0000]  sum 466.4161
  p   = [0.8650, 0.1171, 0.0158, 0.0021]

T = 2.0   logits become [1.5, 1.0, 0.5, 0.0]
  exp = [4.4817, 2.7183, 1.6487, 1.0000]  sum 9.8487
  p   = [0.4551, 0.2760, 0.1674, 0.1015]

The top token went from 64% to 87% at T = 0.5, and down to 46% at T = 2.0. Note what temperature does not do: it never changes the ranking, and it never makes an impossible token possible. Token D keeps a nonzero probability at every temperature, which is why temperature alone cannot stop a model producing rubbish — it can only make it rarer.

Top-k, and renormalisation

Keep the k highest-probability tokens, discard the rest, rescale what remains so it sums to 1.

p = [0.6439, 0.2369, 0.0871, 0.0321]   (T = 1.0)

k = 2:
  keep A, B          0.6439 + 0.2369 = 0.8808
  renormalise:
    A: 0.6439 / 0.8808 = 0.7311
    B: 0.2369 / 0.8808 = 0.2689
  C and D now have probability exactly 0.

k = 3:
  keep A, B, C       sum 0.9679
    A: 0.6439 / 0.9679 = 0.6652
    B: 0.2369 / 0.9679 = 0.2447
    C: 0.0871 / 0.9679 = 0.0900

The flaw in top-k is that k is fixed while the distribution is not. At a position where the model is confident, k = 50 admits 49 tokens it had all but ruled out. At a position where it is genuinely torn between two hundred plausible continuations, k = 50 cuts off 150 reasonable ones. A fixed count cannot adapt, which is what top-p was invented to fix.

Top-p, and the token that crosses the line

Sort descending, accumulate, and keep tokens until the cumulative probability reaches p. The number kept varies with how peaked the distribution is.

p = [0.6439, 0.2369, 0.0871, 0.0321]
top_p = 0.9

  A  0.6439   cumulative 0.6439   < 0.9, keep
  B  0.2369   cumulative 0.8808   < 0.9, keep
  C  0.0871   cumulative 0.9679   >= 0.9, keep and stop
  D  0.0321   discarded

Renormalise over {A, B, C}, sum 0.9679:
  A: 0.6652   B: 0.2447   C: 0.0900

Note the boundary rule: the token that crosses the threshold is included, not excluded. Otherwise a distribution whose top token already exceeds p would leave an empty candidate set. This differs between implementations at the margins — some include the crossing token, some do not — and it is one of the reasons two runtimes with identical settings can produce different output.

The adaptivity is the point. Apply the same top_p = 0.9 to the sharp T = 0.5 distribution above:

p = [0.8650, 0.1171, 0.0158, 0.0021]

  A  cumulative 0.8650   < 0.9, keep
  B  cumulative 0.9821   >= 0.9, keep and stop

Only two tokens survive, against three before.
Same setting, different width, because the model
was more confident.

Min-p, which scales with confidence

Min-p sets a floor relative to the most likely token: keep every token whose probability is at least min_p * p_max.

p = [0.6439, 0.2369, 0.0871, 0.0321]
min_p = 0.10

  threshold = 0.10 * 0.6439 = 0.06439

  A  0.6439 >= 0.06439   keep
  B  0.2369 >= 0.06439   keep
  C  0.0871 >= 0.06439   keep
  D  0.0321 <  0.06439   drop

On the sharper T = 0.5 distribution:
  threshold = 0.10 * 0.8650 = 0.08650
  A keep, B keep (0.1171), C drop (0.0158), D drop

The threshold moves with the model’s confidence rather than with an absolute cumulative mass, which makes it behave more predictably at high temperature. It is a newer parameter than the others and is not available everywhere; the parameter set a given endpoint accepts varies and is worth checking rather than assuming.

Repetition and frequency penalties, arithmetically

The penalties are a third family, and they act on the logits of tokens that already appeared. There are two distinct formulations and they are not interchangeable, which is a common source of confusion because the parameter names look similar.

Additive: frequency and presence penalties

z_i <- z_i - frequency_penalty * count_i
             - presence_penalty * (1 if count_i > 0 else 0)

Token A has logit 3.0 and has already appeared 3 times.
frequency_penalty = 0.5:

  z_A = 3.0 - 0.5 * 3 = 1.5

New logits [1.5, 2.0, 1.0, 0.0]:
  exp = [4.4817, 7.3891, 2.7183, 1.0000]  sum 15.5891
  p   = [0.2875, 0.4740, 0.1744, 0.0642]

A was the most likely token at 0.6439. It is now
second at 0.2875, and B has taken the lead.

The frequency penalty scales with the count, so it grows without bound as a token repeats. The presence penalty is a single flat subtraction the first time a token appears and never grows. Use the presence penalty to nudge toward new vocabulary; use the frequency penalty to break an actual loop, and be aware that a large one eventually suppresses ordinary words like the, which degrades fluency in a way that reads as the model getting worse.

Multiplicative: repetition penalty

For any token that has already appeared:

  z_i <- z_i / penalty   if z_i > 0
  z_i <- z_i * penalty   if z_i < 0

Token A, logit 3.0, penalty 1.2:
  3.0 / 1.2 = 2.5

New logits [2.5, 2.0, 1.0, 0.0]:
  exp = [12.1825, 7.3891, 2.7183, 1.0000]  sum 23.2898
  p   = [0.5231, 0.3173, 0.1167, 0.0429]

A drops from 0.6439 to 0.5231. A gentler intervention
than the additive version at these settings.

The sign test is the part worth understanding, because it looks like a quirk and is not. If the rule simply divided, a repeated token with a logit of -2.0 would become -1.667 — larger, so more likely, which is the opposite of a penalty. Multiplying negatives instead gives -2.4, and the penalty pushes in the right direction on both sides of zero.

The consequence of that asymmetry: a multiplicative repetition penalty has an effect that depends on the magnitude of the logit, and since logits are only meaningful up to an additive constant, this is a parameter whose behaviour is not invariant to a shift that should not matter. It works in practice because implementations apply it to raw logits with a conventional scale. It is also why a repetition penalty tuned on one model transfers poorly to another, and why settings copied between models are worth re-checking.

The order of operations changes the answer

Truncation and temperature do not commute. Apply the same two settings in the two possible orders and the surviving candidate set differs.

Settings: T = 2.0, top_p = 0.9. Logits [3, 2, 1, 0].

Order A: temperature, then top-p
  p after T=2.0:  [0.4551, 0.2760, 0.1674, 0.1015]
  cumulative:      0.4551  0.7311  0.8985  1.0000
  0.8985 < 0.9, so the fourth token is needed to cross.
  Candidate set: {A, B, C, D}  -- all four survive.

Order B: top-p at T=1, then temperature
  p at T=1.0:     [0.6439, 0.2369, 0.0871, 0.0321]
  cumulative:      0.6439  0.8808  0.9679
  Candidate set: {A, B, C}  -- D is gone.
  Then temperature spreads the remaining three.

Same two numbers on the request. Different tokens available.

Most implementations apply temperature first and then truncate, which is order A, but this is a convention rather than a specification. It is one more reason that porting a prompt and its sampling settings between runtimes changes the output, and that class of difference is easy to misattribute to the model.

The whole sampler, in twenty lines

import numpy as np

def sample(logits, temperature=1.0, top_k=0, top_p=1.0, min_p=0.0, rng=None):
    rng = rng or np.random.default_rng()
    z = np.asarray(logits, dtype=np.float64)

    if temperature <= 0:
        return int(z.argmax())              # greedy
    z = z / temperature

    z = z - z.max()                         # stability, see the softmax page
    p = np.exp(z)
    p = p / p.sum()

    order = np.argsort(-p)                  # descending
    ps = p[order]

    keep = np.ones(len(ps), dtype=bool)
    if top_k > 0:
        keep &= np.arange(len(ps)) < top_k
    if top_p < 1.0:
        cum = np.cumsum(ps)
        keep &= np.concatenate(([True], cum[:-1] < top_p))
    if min_p > 0.0:
        keep &= ps >= min_p * ps[0]

    ps = np.where(keep, ps, 0.0)
    ps = ps / ps.sum()

    u = rng.random()                        # the one random number
    idx = int(np.searchsorted(np.cumsum(ps), u))
    return int(order[min(idx, len(ps) - 1)])

Run it on [3.0, 2.0, 1.0, 0.0] with a fixed seed and every number in this page comes back out. The only genuinely random line is rng.random(); everything above it is the deterministic reshaping of a distribution, and everything about how a decoding strategy behaves is decided in those fifteen lines.