Temperature, Top-p and Top-k: A Practical Sampling Guide
7 min read · updated August 3, 2026
Most advice about these parameters is a table of recommended values with no account of what they do. Here is the arithmetic instead — small enough to check on paper — followed by a script that produces the side-by-side comparison for your prompt, because the only comparison worth trusting is the one run on the work you actually do.
What the sampler receives
The model emits one logit per vocabulary item and stops. Everything after that is not the model — it is a small piece of code turning tens of thousands of numbers into one choice. These parameters configure that code. Nothing here changes what the model believes; it changes which of its beliefs get acted on.
To keep the arithmetic visible, pretend the vocabulary has three tokens with logits 4.0, 3.0 and 1.0.
Temperature, by hand
Temperature divides the logits before the softmax: pi ∝ exp(zi / T). Work the three cases:
logits 4.0 3.0 1.0 T = 1.0 exp(z/T) 54.598 20.086 2.718 sum = 77.402 p 0.705 0.260 0.035 T = 0.5 (z/T = 8, 6, 2) exp(z/T) 2980.958 403.429 7.389 sum = 3391.776 p 0.879 0.119 0.002 T = 2.0 (z/T = 2, 1.5, 0.5) exp(z/T) 7.389 4.482 1.649 sum = 13.520 p 0.547 0.331 0.122
Three things are now visible that prose alone hides. Temperature is monotone — the ranking never changes, so the most likely token stays the most likely token at every temperature. What changes is the ratio between neighbours: at T = 0.5 the second token is a seventh as likely as the first, at T = 2.0 it is six-tenths as likely. And the effect on the tail is violent: the third token goes from 3.5% to 0.2% when you halve T. Low temperature does not make the model “more accurate”; it makes rare continuations rare.
T = 0 is a special case, implemented as argmax rather than as a division by zero. It is not quite deterministic in practice, for reasons that have nothing to do with sampling — see why temperature zero still varies.
Top-k and top-p
Both delete tokens before sampling, and differ in how they choose what to delete.
Top-k keeps the k highest-probability tokens and renormalises. It is a fixed count, applied to a distribution whose shape varies enormously between steps. After “the capital of France is” the distribution is a spike and k = 50 admits 49 tokens that should never be considered. Mid-sentence in open prose the distribution is broad and k = 50 may cut off perfectly good continuations.
Top-p — nucleus sampling, from Holtzman et al. (2020) — fixes that by choosing the smallest set whose cumulative probability reaches p, then renormalising. On the example above at T = 1 with p = 0.9: sort to 0.705, 0.260, 0.035; the running total hits 0.705, then 0.965, which clears 0.9. So the nucleus is two tokens, renormalised to 0.730 and 0.270, and the third is gone. The size of the kept set adapts to the shape of the distribution, which is the entire point.
A third member of the family is min_p, supported by several open runtimes: keep tokens whose probability is at least min_p times the probability of the top token. It is relative to the peak rather than to a cumulative total, which makes it robust in a different way. Note that frequency_penalty, presence_penalty and repetition penalties are not in this family — they modify logits based on what has already been generated, not on the shape of the current distribution.
Why stacking them is usually a mistake
Two independent reasons, and the first is the one that surprises people.
They are not independent knobs. Where a runtime applies temperature before the top-p cut — a common order — temperature changes the cumulative probabilities, and therefore changes which tokens fall inside the nucleus. Take the example again with p = 0.9. At T = 1 the nucleus held two tokens. At T = 0.5 the first token alone carries 0.879, still short of 0.9, so the nucleus is two tokens; at T = 0.4 it would be one. Lowering temperature quietly narrowed top-p as well. So “I turned the temperature down and it got much more repetitive than I expected” is often two truncations moving together, and the order of operations differs between runtimes.
The binding constraint is invisible. With temperature, top-k and top-p all set away from their neutral values, the output is shaped by whichever is tightest at each step, and nothing in the response tells you which. You cannot attribute a change to the parameter you moved. The discipline that works: set the others to neutral (T = 1, top_p = 1, top_k disabled), move exactly one, and keep a note of it alongside the prompt version.
What each is actually for
| Setting | Description |
|---|---|
| T = 0 | Extraction, classification, structured output, code edits — anything with one right answer, where run-to-run variation is a defect and reproducibility is worth more than variety. |
| T around 0.7-1.0, top_p = 1 | Open-ended prose, brainstorming, anything where you want the model's own distribution. The default in most chat products for a reason. |
| top_p below 1, T = 1 | Trimming the unreliable tail without flattening the head. Preferred over top-k because it adapts to the shape of each step. |
| top_k | Largely legacy for hosted models. Useful in local runtimes as a cheap guard, and as a hard cap alongside min_p. |
| Sampling n times | A different lever entirely: keep T high enough to get diversity, generate several candidates, and select with a verifier rather than with the sampler. |
Run the comparison yourself
This page shows no model outputs, and that is deliberate: any table of “what temperature 0.9 produced” is about one prompt on one model on one day, and it is exactly the thing you should generate rather than read. The script below sweeps a grid on your prompt against any OpenAI-compatible endpoint and reports how much the output varies at each setting. What you get is the side-by-side table, produced by you, on the work you care about.
import itertools, os
from collections import Counter
from openai import OpenAI
client = OpenAI(base_url=os.environ["BASE_URL"], api_key=os.environ["API_KEY"])
MODEL = os.environ["MODEL"]
PROMPT = "your real prompt here"
TEMPS = [0.0, 0.3, 0.7, 1.0]
TOP_PS = [1.0, 0.9, 0.5]
N = 8 # samples per cell
print(f"{'T':>4} {'top_p':>6} {'unique':>7} most common output")
for t, p in itertools.product(TEMPS, TOP_PS):
outs = []
for _ in range(N):
r = client.chat.completions.create(
model=MODEL,
messages=[{"role": "user", "content": PROMPT}],
temperature=t, top_p=p, max_tokens=120,
)
outs.append(r.choices[0].message.content.strip())
common, count = Counter(outs).most_common(1)[0]
print(f"{t:>4} {p:>6} {len(set(outs)):>7} ({count}/{N}) {common[:60]}")Read it for two things. The unique column is your reproducibility budget: if a cell returns eight distinct answers and your product needs one, that cell is not a candidate however good the answers look. And the outputs themselves are the only evidence about quality, which no variability metric substitutes for — a setting that is perfectly stable and stably wrong scores well on the first column.
If you want a continuous rather than a categorical view of the same thing, request logprobs and inspect the distribution directly instead of sampling from it — what logprobs are and what you can do with them covers the mechanics. And if the question behind the sweep is really “should I be sampling at all”, that is the decoding-strategy question, which sits one level above these parameters.