logprobs and top_logprobs in the OpenAI API: Which Models Return Them

The model already computes a score for every token in its vocabulary before it picks one. logprobs asks it to hand some of those scores back, which is the only direct signal of model confidence the API offers — and the only alternative to asking the model how sure it is, which produces a number it made up.

Two parameters, one of which needs the other

• logprobs is a boolean. Set it to true and the response includes the log probability of each token the model actually emitted.
• top_logprobs is an integer from 0 to 20. It asks, for each emitted position, for that many of the highest-scoring alternatives at that position — the tokens that could have come out instead. It is only accepted when logprobs is true; sending it alone is a 400 telling you so.

Neither costs extra tokens. The scores exist regardless; the parameter only controls whether they are serialised into the response. What it does cost is response size, and that is not trivial: at top_logprobs: 20 every generated token carries twenty alternatives with their tokens, logprobs and byte arrays, which can multiply the size of a long response by an order of magnitude. Ask for 5 unless you have a reason to ask for more.

{
  "model": "gpt-4o-2024-08-06",
  "logprobs": true,
  "top_logprobs": 3,
  "max_tokens": 1,
  "messages": [
    {"role": "system", "content": "Answer with exactly one word: yes or no."},
    {"role": "user", "content": "Is a tomato a fruit?"}
  ]
}

The response object

The scores hang off the choice, not off the message, in a logprobs.content array with one entry per generated token:

{
  "choices": [
    {
      "index": 0,
      "message": { "role": "assistant", "content": "Yes" },
      "finish_reason": "length",
      "logprobs": {
        "content": [
          {
            "token": "Yes",
            "logprob": -0.0004201,
            "bytes": [89, 101, 115],
            "top_logprobs": [
              { "token": "Yes", "logprob": -0.0004201, "bytes": [89, 101, 115] },
              { "token": "yes", "logprob": -8.1254,   "bytes": [121, 101, 115] },
              { "token": "No",  "logprob": -9.7761,   "bytes": [78, 111] }
            ]
          }
        ]
      }
    }
  ]
}

Field by field. token is the token as a string. logprob is its natural log probability, so it is always negative and closer to zero means more likely. bytes is the raw UTF-8 bytes, and it exists because a token can be half a character — for multi-byte scripts and emoji, token may contain a replacement character while bytes is exact. If you are reassembling text from tokens, use the bytes.

The first entry of top_logprobs is not guaranteed to be the emitted token. With temperature above 0 the sampler can pick something other than the top-scoring candidate, and then the emitted token appears further down the list — or, in principle, not in a short list at all. Code that assumes top_logprobs[0].token === token is correct at temperature 0 and wrong the moment somebody turns temperature up.

Which models return them

The parameter is a property of the model, not of the endpoint, and the split at the time of writing is clean:

• The GPT chat models return them — the GPT-4o family including gpt-4o-mini, GPT-4 Turbo, and GPT-3.5 Turbo. This is the case the documentation describes.
• The o-series reasoning models are the exception to check. Support arrived on those models later than on the GPT family, and the early snapshots reject the parameter outright with an unsupported_parameter error — the same family of rejections that covers the system role and the sampling parameters. Verify against the model page for your exact id.
• Embedding, audio, image and moderation models do not have them, which follows from what they return: there is no token stream to score.

From logprob to probability

Exponentiate. A logprob of -0.0004201 is a probability of exp(-0.0004201) = 0.99958, so the model was about 99.96% sure of that token. A logprob of -8.1254 is 0.000295, or roughly three in ten thousand.

const p = (lp) => Math.exp(lp);

p(-0.0004201);   // 0.9995800
p(-0.6931);      // 0.5000    (log 0.5)
p(-2.3026);      // 0.1000    (log 0.1)
p(-8.1254);      // 0.0002951

For a multi-token answer, the probability of the whole sequence is the product of the token probabilities, which is the sum of the logprobs — which is exactly why they are given as logs. Sum, do not multiply exponentiated values; a twenty-token sequence multiplied out underflows towards zero and tells you nothing.

const seq = res.choices[0].logprobs.content;
const total = seq.reduce((a, t) => a + t.logprob, 0);
const perToken = Math.exp(total / seq.length);   // geometric mean, comparable
                                                 // across different lengths

Use the per-token geometric mean rather than the raw sum when comparing answers of different lengths. The raw sum always favours the shorter answer, because every additional token multiplies in a number below 1.

One warning about interpreting these as calibrated probabilities. The logprobs returned are the scores after whatever the sampling configuration does to them, so a request with a modified temperature or top_p is not reporting the model’s raw beliefs — it is reporting a reshaped distribution. If you are using logprobs as a confidence signal, hold the sampling parameters fixed across everything you compare, and prefer temperature 0 so that the number means one thing.

Where they appear when streaming

With stream: true, the scores arrive alongside the tokens rather than in one block at the end. Each chunk carries a delta and, when logprobs are enabled, a logprobs.content array covering only the tokens in that chunk:

data: {"choices":[{"index":0,"delta":{"content":"Yes"},
  "logprobs":{"content":[{"token":"Yes","logprob":-0.00042,"bytes":[89,101,115],
  "top_logprobs":[...]}]},"finish_reason":null}]}

data: {"choices":[{"index":0,"delta":{},"logprobs":null,"finish_reason":"stop"}]}

Two consequences. You can act on confidence before the answer finishes — abandoning a generation whose first label token came out at a low probability saves the rest of the tokens, which is the one case where logprobs reduce cost rather than merely informing you. And you must accumulate the arrays yourself if you want a whole-response figure; there is no summary chunk that repeats them. The final chunk carries the finish reason and a null logprobs, which is not an error.

What this is actually good for

• Confidence gating on classification. Constrain the model to a single-token label, read the logprob of that token, and route anything below a threshold to a human or to a larger model. This is the strongest use of the feature: a calibrated abstain, costing nothing extra, on a task where the alternative is a confidently wrong label.
• Seeing near-ties. Two candidates within a few tenths of a logprob means the model was genuinely undecided, and no amount of asking “how confident are you?” would have surfaced it. This is also how you discover that your prompt has made two labels functionally synonymous.
• Debugging format failures. When output drifts from a requested shape, the logprobs at the divergence point show whether the model was confidently wrong or whether the correct token was a close second — which tells you whether the fix is a better prompt or a hard constraint via Structured Outputs.

What it is not good for is factuality. A high logprob means the token is a likely continuation given the context, not that the claim is true; a fluent hallucination is high-probability by construction. Confidence here is confidence in the text, not in the world.

One further limitation is worth stating plainly, because it is why people abandon the feature after a promising start. The scores are per position, conditioned on everything already emitted. They do not give you the probability of an answer; they give you the probability of a token given the answer so far. Once the model has committed to the first token of a wrong claim, the remaining tokens of that claim are often extremely high probability, because they are the likely continuation of what it just said. That is exactly why the technique is strong for single-token classification and weak as a general hallucination detector: with one token, position and answer are the same thing; with fifty, they are not.