Confidence and Calibration: Does the Model Know It's Wrong?

A model can be wrong and know it, wrong and not know it, or right for reasons that make its confidence meaningless. Calibration is the statistical machinery for telling these apart, and it is worth learning properly because the sloppy version — treating a logprob as a probability of being correct — fails in a specific and predictable way.

The definition

A predictor is calibrated if, among all the predictions it made with stated confidence p, a fraction p turn out correct. Say it makes a thousand predictions at 70% confidence; about seven hundred should be right. That is the whole property, and note what it is not: it is not accuracy. A weather model that says “30% chance of rain” every single day in a climate where it rains 30% of days is perfectly calibrated and completely useless. Calibration and discrimination are separate axes, and you want both.

The relevance to hallucination is direct. If a model were well calibrated on its own answers, you would not need to detect hallucination at all — you would threshold on confidence and route the low-confidence cases to a human or to a search. The reason that does not work out of the box is the subject of the rest of this page.

Reading a reliability diagram

The plot everyone shows and few label. Both axes run from 0 to 1.

• x-axis: predicted confidence. Predictions are sorted into bins — conventionally ten equal-width bins, [0.0, 0.1), [0.1, 0.2) and so on — by the confidence the model stated. For a multiple-choice answer that confidence is the softmax probability of the chosen option.
• y-axis: observed accuracy. Within each bin, the fraction of predictions that were actually correct.
• The diagonal. y = x is perfect calibration. Points below the diagonal mean the model was more confident than it deserved: overconfidence. Points above mean it was underconfident.
• The bin counts. Almost always drawn as a histogram underneath, and they matter — a bin holding twelve predictions can sit anywhere, and a diagram without them invites you to read noise as a finding.

Guo et al.’s On Calibration of Modern Neural Networks (2017) is the paper that made this standard equipment, and its central observation transfers: modern networks trained to high accuracy tend to be badly overconfident, and a single scalar — temperature scaling on the logits, fitted on held-out data — recovers most of the calibration without touching accuracy at all. If you have logits and labels, try that before anything more elaborate.

ECE, and what it hides

Expected calibration error is the reliability diagram collapsed to one number: the weighted average distance from the diagonal, weighted by how many predictions fell in each bin.

ECE = sum over bins b of  (n_b / N) * | accuracy(b) - mean_confidence(b) |

Useful, and freely abused. It depends on the binning scheme — equal width versus equal mass changes the number, sometimes a lot. It is insensitive to direction, so a model that is overconfident on hard questions and underconfident on easy ones can score well by cancellation. And it says nothing about discrimination, so it can be driven to near zero by a model that predicts the base rate for everything. Report the diagram; use ECE as a scalar for tracking, not as the finding.

What the published curves show

Two results are worth knowing before you spend a week on this, because together they explain why the naive approach disappoints.

Kadavath et al., Language Models (Mostly) Know What They Know (Anthropic, 2022). On multiple-choice tasks, large models’ answer probabilities were found to be reasonably well calibrated, and the paper introduced a self-evaluation move — ask the model to output P(True), the probability that its own proposed answer is correct — which carried real signal, improving when the model was allowed to see several of its own candidate answers first. The signal is there. It is not nothing.

The GPT-4 system card (OpenAI, 2023). It contains two reliability diagrams on MMLU, side by side, and they are the single most instructive pair of plots in this subject. The pretrained model sits close to the diagonal. The post-RLHF model — the one you can actually call — is visibly further from it, piling probability mass near the top of the confidence range. The alignment step that made the model useful degraded the calibration that made its confidence readable.

That is the same mechanism described on why models hallucinate: optimising against binary-graded preferences pushes towards stating answers confidently, and confidence that is always high is confidence that carries no information. Whether a given current model behaves this way is a question about that model, which is why this page is marked for refresh and why the code below matters more than any figure quoted here.

Free-form answers break the naive method

Multiple choice is the easy case: one token, one distribution, done. Free-form generation is where the naive approach quietly fails, for a reason worth internalising.

Sum the logprobs of a generated sequence and you get the probability of that exact string. But “Paris”, “It is Paris” and “The capital is Paris” are the same answer wearing three costumes, and the model’s probability mass is split across all of them. Low sequence probability can mean genuine uncertainty about the answer, or complete certainty about the answer combined with indifference about the phrasing — and the number cannot tell you which.

Kuhn, Gal and Farquhar’s Semantic Uncertainty (ICLR 2023) fixed this by clustering: sample several answers, group them by bidirectional entailment so that paraphrases land in one cluster, and compute the entropy over clusters rather than over strings. Farquhar, Kossen, Kuhn and Gal took the method to Nature in 2024, reporting that semantic entropy detects the subclass of hallucinations they call confabulations — arbitrary, wrong answers that vary between samples — substantially better than sequence-likelihood baselines. Manakul et al.’s SelfCheckGPT (2023) arrives at the same territory from a different direction: sample several times, and measure how much the samples contradict each other, with no external knowledge base required at all.

The shared intuition is worth stating plainly. A fabricated detail is not stable across samples, because it was drawn from a flat region of the distribution. A memorised fact is. Disagreement between samples is therefore a usable proxy for “this was made up”, and unlike a logprob it survives paraphrase.

Producing your own curve

For a task with checkable answers, a reliability diagram is about forty lines. Use a multiple-choice or short-extraction framing so that the confidence is well defined.

import math

def answer_with_confidence(prompt):
    """Request logprobs. For an A/B/C/D question the answer is one token,
    so the confidence is just the probability of the chosen token."""
    r = call_model(prompt, max_tokens=1, logprobs=True, top_logprobs=5)
    tok = r.choices[0].logprobs.content[0]
    return tok.token.strip(), math.exp(tok.logprob)

def reliability(records, n_bins=10):
    """records: [(confidence, correct_bool), ...]"""
    bins = [[] for _ in range(n_bins)]
    for conf, correct in records:
        i = min(int(conf * n_bins), n_bins - 1)   # equal-width bins
        bins[i].append((conf, correct))
    rows, N, ece = [], len(records), 0.0
    for i, b in enumerate(bins):
        if not b:
            rows.append((i / n_bins, None, None, 0)); continue
        acc = sum(c for _, c in b) / len(b)       # y-axis
        avg = sum(p for p, _ in b) / len(b)       # x-axis
        ece += (len(b) / N) * abs(acc - avg)
        rows.append((i / n_bins, avg, acc, len(b)))
    return rows, ece

# rows -> plot avg (x) against acc (y), draw y=x, and print the bin counts.
# A bin with fewer than ~30 items is decoration, not evidence.

Two cautions. Any confidence you derive is conditional on the exact prompt, so re-derive it when the prompt changes — the prompt sensitivity page explains how large that effect is. And a verbalised confidence — asking the model to say “I am 80% sure” — is a different quantity from the logprob and needs its own curve; Lin, Hilton and Evans (2022) showed models can be trained to express calibrated uncertainty in words, but an untuned model’s verbalised numbers cluster on round figures and are typically far worse calibrated than its logits.