What to Do When the Target Provider Has No Logprobs Support

Token log-probabilities are the only signal an API gives you about how confident the model was, and several providers do not expose them. Before you build an approximation, find out whether the number you are losing was ever separating anything.

Who exposes what

OpenAI’s Chat Completions API takes a boolean logprobs and an integer top_logprobs, and returns choices[].logprobs.content: one entry per output token, each with token, logprob, bytes, and a top_logprobs array of the highest-scoring alternatives at that position. Google’s Gemini generation config documents a response-logprobs switch and a count of alternatives to return. Anthropic’s Messages API documents no logprobs parameter or response field.

So the migration cases are: the field is present under another name and you rename it, or the field does not exist and the feature that consumed it has to be rebuilt from something else. This page is about the second. The general treatment of what the numbers mean is in the logprobs page.

First: was the threshold doing anything?

Systems accumulate confidence gates that nobody has evaluated since the day they were added. Before spending a week rebuilding one, measure it against the labelled data you already have. The procedure, on your own traffic:

1. Take a few hundred logged requests from the source provider where you know whether the output was correct — from human review, from a downstream success signal, from a golden set.
2. For each, compute the statistic your gate uses. Usually that is the mean or the minimum logprob across output tokens, or the logprob of the single decisive token in a classification.
3. Sort by that statistic and compute, at your current threshold, how many correct outputs it rejects and how many wrong ones it lets through.
4. Compare against a random threshold on the same data. If the gate is not meaningfully better than chance at separating correct from incorrect, you are not losing a capability. Delete the gate and the migration problem disappears.

This is worth doing honestly because the result is frequently “no”, particularly for gates on long free-text outputs where the mean logprob is dominated by grammatical filler tokens that are high-probability regardless of whether the content is right. Gates on a single decisive token in a constrained classification are the ones that usually survive the test. See model calibration for why the difference exists.

Rebuilding confidence from sampling

If the gate does earn its place, the replacement that most closely preserves its meaning is self-consistency: ask the same question n times at non-zero temperature and use the agreement rate as the confidence. For a classification with a small label set this is a direct substitute — the fraction of samples choosing the modal label is an estimate of the probability mass on that label, which is close to what you were reading off the logprob.

from collections import Counter

def classify_with_confidence(text, n=5, temperature=0.7):
    votes = [classify_once(text, temperature) for _ in range(n)]
    label, count = Counter(votes).most_common(1)[0]
    return label, count / n          # in {0.2, 0.4, ..., 1.0} for n=5

Three properties of this substitute have to be stated plainly. It costs n times the tokens and, unless you parallelise, roughly n times the latency. Its resolution is coarse: with five samples the only values you can observe are multiples of 0.2, so a threshold at 0.85 is not expressible and any threshold between 0.8 and 1.0 behaves identically. And it requires the output to be constrained enough that two samples can be compared for equality — which means pairing it with schema-constrained output, not free text.

Choose n from the resolution you need rather than by feel: to distinguish confidence bands of width w you need at least 1/w samples, so bands of 0.1 need ten calls per decision. That arithmetic is usually what kills the approach for high-volume paths and leaves it viable for the small fraction of requests that a cheaper first-pass gate has already flagged as uncertain.

Asking the model, and calibrating the answer

The cheaper option is to make confidence part of the output: a schema with a label and a confidence number, filled in by the model. This costs one call rather than n and it works on any provider with structured output.

{
  "type": "object",
  "properties": {
    "label": { "type": "string", "enum": ["refund", "billing", "other"] },
    "confidence": { "type": "number", "minimum": 0, "maximum": 1 }
  },
  "required": ["label", "confidence"],
  "additionalProperties": false
}

The catch is that a self-reported number is not a probability. It is text the model produced, and there is no mechanism forcing it to correspond to the frequency with which that answer is right. Models tend to cluster their self-reports in a narrow high band, which makes a naive threshold at 0.9 reject almost nothing.

That does not make it useless — it makes it a raw score that needs calibrating. Collect a few hundred labelled examples, bucket them by the reported confidence, and compute the actual accuracy per bucket. What you get is a lookup from reported value to observed accuracy, and you set your gate on the second number, not the first. It is the same work you would have had to do for the logprob threshold; nobody did it because the logprob felt like it meant something on its own.

Why a proxy model does not substitute

A tempting move is to keep a smaller open-weights model available, score the target model’s output with it, and use that as the confidence signal. Be clear about what this measures. The proxy’s logprob for a string is the proxy’s estimate of how likely that string is under the proxy’s distribution. It is not the target model’s confidence, and the two can diverge exactly where it matters — a confidently wrong hallucination is usually fluent, so it scores well under any reasonable language model, including the proxy.

There is a narrow case where it is legitimate: detecting output that is malformed, off-distribution or degenerate, where low fluency is precisely the signal you want. Call that a fluency check and gate on it for that purpose. Do not report it to users or to downstream systems as the model’s confidence, and do not carry over a threshold that was tuned on the target’s own logprobs — the numbers are on different scales and there is no conversion.

The uses you have to give up

Some things genuinely do not survive, and planning for them is better than discovering them.

• Per-token analysis of the answer. Highlighting the specific spans a model was unsure about, for a human reviewer, requires per-token numbers. There is no reconstruction from sampling that gives you token-level attribution at usable cost.
• Perplexity of a supplied string. Scoring text the model did not generate — for detection, for ranking candidates, for evaluation — needs the ability to evaluate rather than generate. Chat-shaped APIs without logprobs cannot do it at all.
• Cheap reranking by likelihood. Ordering k candidate answers by their sequence logprob is one extra call under a logprobs-capable API and is a judge call or a reranker model without one. That is a real cost increase; the library covers the alternative in the reranking page.

For each of these the decision is the same shape as the one in the logit bias mapping: keep the hosted API and rebuild the feature at the application level, drop the feature, or serve a model yourself where the primitive still exists. Write down which of the three you chose and why, next to the code that used to read the field, because the next person to look will otherwise assume it was an oversight.