Confidence Scores in Structured Extraction

Adding confidence: number to a schema is the most popular non-solution in extraction. You will get a number. It will be 0.95. It will be 0.95 when the model is right and when it is wrong, and it will be 0.95 on a blank page.

The confidence field is a token sequence

The model does have an internal quantity that behaves like a probability — the distribution over the next token. A confidence field is not that quantity. It is a number generated the same way as every other output token, conditioned on what confidence values look like in text the model was trained on. Those skew high, round, and reassuring.

Three symptoms that identify a useless confidence field, all cheap to check on data you already have:

• Clustering. Histogram the values. If almost everything is 0.9, 0.95 and 0.99, you have captured a style, not a belief.
• No discrimination. Compare the mean score on records you know are right against records you know are wrong. If the two are close, the field carries no information regardless of how it is distributed.
• Position sensitivity. Move the field from after the value to before it and watch the numbers change. A real belief would not depend on where you asked for it — and emitted first, it is not even a judgement about an answer that exists yet.

What the literature actually found

It is worth being precise here, because the honest position is more interesting than either “models know when they are wrong” or “confidence is meaningless”.

Kadavath et al., Language Models (Mostly) Know What They Know (arXiv:2207.05221, Anthropic, 2022), found that large models can be reasonably calibrated on multiple-choice questions when the probability is read off the token distribution, and that a model asked to evaluate the probability that its own answer is true — their P(True) setup — carries real signal. The elicitation method mattered a great deal.

OpenAI’s GPT-4 technical report published a calibration plot showing the pre-trained model well calibrated on MMLU and the post-RLHF model noticeably less so. That is the finding to carry around: alignment training, which is what makes a model pleasant to talk to, appears to degrade the calibration of its stated confidence. The models you call through an API are the post-training ones.

Neither result licenses trusting a self-reported number in an extraction schema. Both suggest that a probability read from the distribution is a different and better object than one read from the text.

Three signals with something behind them

1. Token logprobs

Set logprobs: true and you get the log-probability of each emitted token. For a field constrained to an enum this is close to what you want, especially combined with top_logprobs to see the runner-up — the margin between the top two labels is more informative than the top probability alone. The single-token version of this is the cleanest case.

For free-text fields, be careful. The joint probability of a ten-token value is mechanically lower than that of a two-token value, so raw joint scores rank by length. Use the mean per-token logprob, or the minimum over the tokens of the value — the minimum is often better because one uncertain token in the middle of a reference number is exactly the failure you are hunting. And note the structural tokens are near-certain under constrained decoding, so include only the tokens inside the value.

2. Self-consistency

Sample the same extraction n times at a non-zero temperature and measure per-field agreement: the fraction of samples that produced the modal value. This needs no logprobs, works on any endpoint including ones that expose nothing, and is straightforwardly interpretable. It costs n times as much, which restricts it to fields where a mistake is expensive, or to a sampled audit rather than every record.

3. Verification

The strongest and the cheapest. Require a verbatim source_quote and check it appears in the input; check the total equals the sum of the line items; check the date is inside the contract period. These are not confidence scores — they are binary and they are grounded in something outside the model. A failed check is worth more than any number in the 0.7 to 0.95 band, and it costs no extra tokens.

Checking calibration yourself

Whatever signal you pick, do not assume it is calibrated — test it. Take a few hundred records you have verified, bucket the scores, and compare the mean score in each bucket to the actual accuracy in that bucket. Expected calibration error is the weighted average gap:

def ece(scores, correct, bins=10):
    """scores: predicted confidence in [0,1]. correct: 0/1 ground truth.
    Returns (ece, table) -- print the table, it is more use than the number."""
    n = len(scores)
    total, table = 0.0, []
    for b in range(bins):
        lo, hi = b / bins, (b + 1) / bins
        idx = [i for i, s in enumerate(scores)
               if (s > lo or b == 0) and s <= hi]
        if not idx:
            continue
        conf = sum(scores[i] for i in idx) / len(idx)
        acc  = sum(correct[i] for i in idx) / len(idx)
        total += (len(idx) / n) * abs(acc - conf)
        table.append((f"{lo:.1f}-{hi:.1f}", len(idx), round(conf, 3), round(acc, 3)))
    return round(total, 4), table

# A model-written confidence field typically produces a table with one
# crowded row near 0.9-1.0 whose accuracy is far below its confidence.
# A useful signal produces rows spread across the range with acc ~ conf.

The table matters more than the scalar. A single ECE number hides whether you are overconfident everywhere or badly wrong in one bucket, and the second is fixable by moving a threshold.

Using a score once you trust it

Do not surface raw confidence to users. Nobody knows what 0.82 means, and putting it on a screen invites people to treat it as a probability it has not earned. Use it internally, for three things:

• Routing. Below a threshold, re-run on a larger model. The threshold comes from the calibration table above and from what a mistake costs you, not from a round number.
• Review queues. Sort human review by ascending confidence. Even a poorly calibrated score with some discrimination beats random order, and this is the use with the lowest bar.
• Drift detection. Track the distribution over time. The mean confidence moving is a signal about your inputs or your model version that arrives before any accuracy metric, because it needs no labels.