What Logprobs Are and What You Can Actually Do With Them

Logprobs are the only channel through which a chat API tells you anything quantitative about its own output. Used carefully they turn a generative model into a scorer. Used carelessly they produce a confidence number that means less than it appears to.

What a logprob is

For each generated token, the natural logarithm of the probability the sampler assigned it. Since probabilities lie in (0, 1], logprobs are zero or negative: −0.01 is near-certainty, −0.7 is about a coin flip, −4.6 is one percent. Recover the probability by exponentiating, and note the useful mental anchors — e^−0.69 ≈ 0.5, e^−2.3 ≈ 0.1, e^−4.6 ≈ 0.01.

They are logs rather than probabilities for two reasons that matter in practice. Probabilities for rare tokens underflow to zero in floating point; logs do not. And the probability of a sequence is a product of per-token probabilities, which becomes a sum of logprobs — numerically stable and much easier to work with.

What the API returns

Typically two things: the logprob of each token the model actually emitted, and — if you ask for top_logprobs — the top few alternatives at each position with their logprobs. The alternatives are the interesting part, because they tell you what the model nearly said.

One detail to establish before you build on them, because it differs between providers and is rarely documented prominently: whether the returned values are the raw distribution or the distribution after your temperature and truncation were applied. If they are post-transform, then setting temperature to 0.2 makes every reported confidence look higher without anything having changed in the model. Check it by requesting the same prompt at two temperatures and comparing the returned logprobs for an identical token; if they move, they are post-transform, and any threshold you calibrate is tied to that setting. Also expect that many reasoning models do not return logprobs at all.

Classification with a probability

This is the highest-value use, and the trick is to make the answer exactly one token so the token’s probability is the answer’s probability.

import math, os
from openai import OpenAI

client = OpenAI(base_url=os.environ["BASE_URL"], api_key=os.environ["API_KEY"])

SYSTEM = "Answer with exactly one letter: Y if the review is positive, N if not."

def classify(text, model):
    r = client.chat.completions.create(
        model=model,
        messages=[{"role": "system", "content": SYSTEM},
                  {"role": "user",   "content": text}],
        max_tokens=1, temperature=0,
        logprobs=True, top_logprobs=20,
    )
    top = r.choices[0].logprobs.content[0].top_logprobs
    scores = {t.token.strip(): math.exp(t.logprob) for t in top}
    y, n = scores.get("Y", 0.0), scores.get("N", 0.0)
    if y + n == 0:
        return None, 0.0            # model answered off-menu; treat as abstain
    return ("Y" if y > n else "N"), max(y, n) / (y + n)

The renormalisation on the last line is what makes the number usable. Suppose the API returns logprob −0.12 for “Y” and −2.30 for “N”. Exponentiating gives 0.887 and 0.100, which sum to 0.987 rather than 1 because the remaining 1.3% sits on tokens that are not either class. Dividing by the sum gives 0.899 and 0.101 — a probability conditional on answering the question at all, which is the quantity you want to threshold on.

Two implementation traps. Tokenizers usually treat a leading space as part of the token, so “Y” and “ Y” are different entries — strip and compare, as above. And if a class needs several tokens, its probability is the product of theirs (the sum of logprobs), which systematically penalises longer labels; single-token labels avoid the whole problem.

Three more uses

• Scoring a candidate you did not generate. Sum the logprobs of a supplied continuation and divide by its token count to get an average — that is a per-token cross-entropy, and exponentiating it gives perplexity. Useful for ranking rewrites or for choosing between two phrasings, provided both are scored under the same model and prompt.
• Routing to human review. Threshold the renormalised class probability and send the low-confidence tail to a person. This is the single most reliable production use, because it does not need the number to be well-calibrated in absolute terms — only to be monotone enough that the bottom decile is genuinely worse than the top.
• Detecting off-menu answers. If neither class token appears anywhere in the top alternatives, the model is not answering the question you asked. That is a distinct failure from answering it wrongly and deserves distinct handling — see abstention.

Where they mislead

A logprob is the model’s estimate of how likely that token is to follow, given its training. It is not an estimate of whether the claim is true. A confidently-worded fabrication has high token probabilities throughout — that is why it is fluent — so token confidence and factual correctness are different quantities, and the gap between them is the subject of calibration.

Two more limits. Confidence on the first token of a long answer says little about the rest of it, so per-token numbers do not aggregate into an answer-level confidence in any principled way. And the whole distribution these numbers come from lives inside the subspace described by the softmax bottleneck, which is a reminder that they are properties of a model’s output layer rather than measurements of the world.

The practical rule that survives all of this: use logprobs to rank and to triage, not to certify. A threshold you calibrated on your own labelled examples, for one model at one setting, is a real tool. The raw number quoted as “the model was 89% confident” is not.