Scoring Confidence Per Field Instead of Per Document

A document-level confidence of 0.96 sounds like a document you can post without looking at it. It is compatible with twelve fields at 0.99 and one field — the payment amount — at 0.55. The score did not lie; it answered a question nobody asked.

Two different objects

Per-field and per-document confidence are not the same quantity at two resolutions. They are estimates of two different events.

A per-field score estimates P(this value is correct). It is about one region of one page and one extracted string, and it is the number every downstream decision actually needs: whether to route that field to review, whether to trust it in a cross-field rule, whether to let it drive a payment.

A per-document score, if it means anything, estimates P(every field on this record is correct). That is a joint probability over a conjunction of events. It is a strictly harder thing to be right about, it is much smaller than any of its parts, and it is not recoverable by averaging.

Once you see them as different events, the pathology of the document-level score is obvious: it is a summary statistic over a distribution whose only interesting feature is its worst element. Averaging destroys exactly the information you wanted.

The mean is the wrong rollup

Suppose a record has 20 fields and each is correct with probability 0.98, independently. The mean confidence is 0.98, which reads as a very good document. The probability that the whole record is correct is 0.98^20, which is about 0.67. One record in three has at least one wrong field.

Push it a little: 40 fields at 0.98 gives 0.98^40 ≈ 0.45. The mean has not moved at all and the document-level truth has crossed below a coin flip. This is the single most useful piece of arithmetic in extraction engineering, because it explains why per-field accuracy numbers that sound excellent produce a document error rate that everybody finds shocking.

Independence is an assumption and usually a generous one — errors cluster, because the same bad scan degrades every field on the page. Correlated errors make the product a pessimistic bound and the true document-correct probability somewhat higher. That does not rescue the mean; it just means the honest rollups are:

• The minimum, when the decision is “can this record be auto-accepted”. Acceptance is a conjunction, so it is governed by the weakest field, and the minimum is an upper bound on the joint probability that requires no independence assumption.
• The product, when you want an estimate rather than a bound, and you have some evidence that field errors are not strongly coupled. Compute it in log space if the record is wide.
• A weighted minimum over the fields that matter, which is what most systems actually want. Nobody should block a payment run because a free-text delivery note is uncertain. Define a set of critical fields and roll up over that set only.

The mean survives in one legitimate role: as a monitoring metric over time, where it is a cheap indicator that something in the pipeline has shifted. It is a health signal. It is not a routing input.

Where a per-field score comes from

You have three plausible sources and they are not interchangeable.

Token log-probabilities over the value span

When the provider returns per-token log-probabilities, the value of a field is a contiguous run of generated tokens and you can turn that run into a score. Two aggregations are common: the sum of the log probabilities (which is the sequence probability, and is length-dependent, so a long value scores lower purely for being long) and the mean per token (which is length-normalised and is usually what you want for comparing fields against each other). The minimum per-token probability inside the span is a third and it is often the most diagnostic, because a single uncertain character in an otherwise confident identifier is exactly the failure you are hunting.

An upstream recogniser’s own confidence

If a classical OCR stage runs before the model, it emits per-word and often per-character confidences of its own, and these are grounded in image evidence in a way a language model’s scores are not. They are also the only source that can tell you which character was doubtful, which is what makes checksum recovery of a single unreadable digit possible. The general pipeline is covered in the OCR pipeline page.

Agreement between independent extractions

Run the field through two extractors that fail differently — a different model, or a regex over the OCR text against the model’s structured answer — and use agreement as the signal. This is the most expensive option and by some distance the most trustworthy for high-value fields, because the two sources have to be wrong in the same way to fool it. Reserve it for the handful of fields where C_wrong justifies a second call.

A number the model typed is not a score

Asking the model to include "confidence": 0.93 in its own JSON output is the most common implementation and the weakest. The value is generated the same way every other token is: it is the continuation the model finds natural, conditioned on a prompt that asked for a confidence. It is not a readout of any internal state, and nothing in the training objective makes it correspond to an error rate.

The characteristic symptom is a distribution with almost no mass below 0.8 and enormous spikes at round numbers — 0.85, 0.9, 0.95 — because those are the strings that follow “confidence” in text. Plot a histogram of self-reported confidences from a real run and you will see them immediately. A grounded score has a long, messy left tail; a self-reported one usually does not.

This does not make self-reported confidence useless. It makes it an uncalibrated ordinal signal: the model’s 0.6 really is less reliable than its 0.95, often enough to be worth ranking by. What you cannot do is compare it to a threshold as if it were a probability, because the mapping from that number to an error rate is unknown until you measure it. Measuring it is the whole of calibrating extraction confidence, and after that measurement a self-reported score is a legitimate input.

Designing the field record

Per-field confidence only exists if the record has a place to put it, which means the extracted document is a collection of field objects rather than a flat map of scalars. That shape costs you some convenience at every read site and buys you every decision in this cluster.

type FieldValue<T> = {
  value: T | null;
  status: "present" | "absent" | "illegible" | "not_extracted";
  confidence: number | null;        // calibrated, in [0,1]
  confidenceSource: "logprob" | "ocr" | "agreement" | "self_report";
  source?: { page: number; bbox: [number, number, number, number] };
  extractedBy: string;              // pipeline version stamp
};

type Invoice = {
  invoiceNumber: FieldValue<string>;
  invoiceDate: FieldValue<string>;
  totalAmount: FieldValue<number>;
  // ...
};

// Acceptance is a conjunction over the fields that matter.
const CRITICAL = ["invoiceNumber", "invoiceDate", "totalAmount"] as const;

function autoAcceptable(doc: Invoice, thresholds: Record<string, number>) {
  return CRITICAL.every((k) => {
    const f = doc[k] as FieldValue<unknown>;
    return f.status === "present" && (f.confidence ?? 0) >= thresholds[k];
  });
}

Note that thresholds is a map and not a constant. Different fields have different costs of being wrong, so they get different cut-offs — the derivation is in setting a confidence threshold that sends a field to review. And note confidenceSource: when you later find that one class of field is badly calibrated, this column is what lets you find out whether the problem is a model, a recogniser, or the fact that somebody wired a self-reported number into a threshold.