Setting a Confidence Threshold That Sends a Field to Human Review

Almost every extraction system routes to human review below 0.8, and almost nobody can say why 0.8. It is a number that reads as “quite sure” in English. The correct threshold is a break-even between two costs you can write down, it is different for every field, and for the fields that matter it is usually far higher than 0.8.

Where 0.8 comes from

It comes from the score looking like a percentage. Once a number is in [0, 1] it invites a grade boundary, and 0.8 is where most people put the line between “fine” and “check it”. Two things are wrong with that instinct beyond its arbitrariness.

The first is that a threshold is only meaningful on a scale where the number is a probability. On an uncalibrated score, 0.8 might correspond to a 60% chance of being right or a 99% chance, and both happen. Derive nothing until you have measured what your scores mean; the rest of this page assumes p is calibrated.

The second is that one threshold across all fields asserts that every field costs the same to get wrong. On an invoice, the payee bank account and the delivery-note reference sit in the same record and differ by three orders of magnitude in what a wrong value does. A single cut-off is therefore too loose for one and absurdly tight for the other, simultaneously.

The three costs

Everything follows from three quantities, and the work is in estimating them honestly rather than in the algebra.

• C_review — the fully loaded cost of one human looking at one field. Not the hourly rate: the rate times the handling time, plus the fixed overhead of the item existing in a queue. Handling time is the number that moves most, and it is the number showing the source location attacks directly — a review that starts with a crop of the right region can be several times cheaper than one that starts with a 40-page PDF.
• C_error — the cost of a wrong value that nobody catches. This includes the direct loss, the cost of the eventual discovery and correction, and any regulatory or contractual exposure. It is field-specific and it is usually badly underestimated, because the discovery cost lands on a different team than the one doing this arithmetic.
• r — the probability that a reviewer actually catches the error when one is present. Not 1. Reviewers miss things, especially on a queue where most items turn out to be fine, and a realistic figure sits somewhere below 0.95 for tedious numeric fields. Leaving this out of the model is the most common way the derivation overstates the value of review.

Deriving the break-even

For a single field with calibrated correctness probability p, compare the expected cost of the two actions.

Auto-accept. With probability 1 − p the value is wrong and escapes. Expected cost: (1 − p) × C_error.

Send to review. You pay C_review regardless. With probability 1 − p the value is wrong, and of those the reviewer misses a fraction 1 − r, which still escapes. Expected cost: C_review + (1 − p)(1 − r) × C_error.

Review is worth it when its expected cost is lower:

C_review + (1 - p)(1 - r) * C_error  <  (1 - p) * C_error

C_review  <  (1 - p) * C_error * [1 - (1 - r)]
C_review  <  (1 - p) * r * C_error

(1 - p)   >  C_review / (r * C_error)

so:  review whenever   p  <  t*
     where             t* = 1 - C_review / (r * C_error)

Three readings of that result are worth spelling out. With a perfect reviewer, r = 1, it collapses to the familiar t* = 1 − C_review/C_error. A fallible reviewer raises the ratio and therefore lowers the threshold — if review only catches half the errors, it is worth half as much, so fewer fields justify it. And if C_review ≥ r × C_error the threshold is negative, which is the model telling you that reviewing this field is never economic at any confidence. That is a legitimate answer and you should let it stand rather than inventing a floor.

Two fields, two thresholds

Take one invoice and two of its fields. The inputs below are labelled assumptions — substitute your own, because the point is the method, not these numbers.

Payee bank account number. Assume a reviewer costs £26/hour fully loaded and the field takes 90 seconds with a source crop, so C_review = £0.65. Assume a misdirected payment costs £250 on average once recall, fees and reconciliation are counted, and assume reviewers catch 95% of wrong account numbers, so r = 0.95. Then t* = 1 − 0.65 / (0.95 × 250) = 1 − 0.00274 = 0.9973. Anything below about 99.7% goes to a human. That is a long way from 0.8, and it is why a naive threshold on a payment field feels “surprisingly noisy in production”.

Delivery-note reference. Same reviewer cost. Assume a wrong value costs £4 in downstream confusion and the reviewer catch rate is 0.9. Then t* = 1 − 0.65 / (0.9 × 4) = 1 − 0.18 = 0.82. Here 0.8 was nearly right — which is presumably why the habit survives. It is right for cheap fields and dangerously wrong for expensive ones.

Notice what the two thresholds do to volume. If your calibrated confidence on account numbers is above 0.997 on 88% of documents, the strict threshold sends 12% to review. On the reference field, 0.82 might send 2%. The strict threshold is not a small adjustment; it is the dominant driver of queue size, and that is the constraint the next section is about.

Capacity is a different problem

The derived thresholds will often flag more fields than your reviewers can process. There are two bad responses and one correct one.

The first bad response is to raise the thresholds until the volume fits. This silently converts a cost-based decision into a capacity-based one and hides the trade: you are now auto-accepting fields you have computed are not safe to auto-accept, and no number anywhere records that you did. The second bad response is to keep the thresholds and let the queue grow unboundedly, which produces the same outcome with a delay and a backlog.

The correct response is to separate the two questions. The threshold decides eligibility: which fields are, on cost grounds, not safe to accept unreviewed. Capacity decides ordering: given fewer reviewer-minutes than eligible fields, which ones get the minutes. Keeping them separate means the shortfall is visible and measurable — you can report the expected loss of the tail you did not get to, which is the number that justifies hiring or justifies improving the model. Collapsing them into a raised threshold makes that number unobtainable.

What to watch after it ships

1. Flag precision. Of the fields sent to review, what fraction were actually changed? If it is 3%, reviewers spend 97% of their time confirming correct values, and they will start rubber-stamping — which drops r, which by the derivation above should have lowered your threshold in the first place. A precision below roughly one in ten is a signal that the score has no discrimination in that region, not that the threshold is wrong.
2. Escape rate on the audit stratum. The randomly sampled auto-accepted fields are the only unbiased estimate of what you are letting through. Compare the measured escape rate against 1 − t*; a persistent gap means the calibration has drifted.
3. Flag rate per field, over time. A step change on one field on one day is almost always upstream — a new document template, a different scanner, a provider model update. Alert on the rate, not on the average confidence, because the rate moves first.
4. Re-derive when a cost changes. The threshold is a function of three numbers that belong to the business, not to the model. When the reviewer cost or the loss estimate changes, the threshold changes, and it should be stored as a derived value with its inputs recorded next to it rather than as a constant somebody typed.