Routing Only Low-Confidence Outputs to a Human Reviewer

If reviewers can read 200 outputs a day out of 10,000, a uniform random sample spends most of that capacity confirming that fine output is fine. Triage concentrates it on the outputs most likely to be wrong — which requires a usable notion of “likely wrong”, and the obvious one is only trustworthy for some tasks.

The signals that exist

Five sources of a confidence score, in rough order of how much they cost you.

• Token log probabilities. Free, in the sense that they come back with the response if you ask for them. On OpenAI-compatible chat completions this is logprobs: true and optionally top_logprobs: n, documented in the OpenAI chat completions API reference; not every provider or model exposes them, and several reasoning models do not.
• Validator and schema outcomes. Free and deterministic. An output that failed a repair step, needed a retry, or came back with finish_reason: "length" is low-confidence by construction and needs no model to tell you so.
• Retrieval evidence. If the answer is supposed to be grounded, the retrieval score of the best supporting chunk and whether the answer’s claims overlap it at all. Cheap, and for a RAG system this is usually the strongest available signal.
• Self-consistency. Sample the same input k times at nonzero temperature and measure agreement across the samples. This is the technique Wang and colleagues introduced for improving reasoning accuracy in Self-Consistency Improves Chain of Thought Reasoning in Language Models, used here for its by-product: disagreement is uncertainty. Costs k times the inference.
• A judge model. A second call that scores the first. Most expensive, most flexible, and the least calibrated — judges have well-documented position and verbosity biases and their scores are not probabilities.

Where logprobs mean something

This is the distinction the whole design turns on, and it is not about how good the model is.

A language model’s output distribution is over tokens, trained to predict what text follows. When the task is classification and you have constrained the answer to be one token from a small set — a label, a yes or no, a category — then the probability mass on that token genuinely is the model’s posterior over the label. It is a real, usable confidence, and thresholding on it works.

When the output is free-form prose, the same numbers measure something else entirely: how predictable the wording was. A fluent, confident, entirely fabricated citation has high token probabilities, because the shape of a citation is extremely predictable. Low token probability in prose correlates with unusual phrasing, proper nouns and rare formatting, none of which is the same as being wrong. Using mean logprob to triage summaries will hand your reviewers the outputs with the most unusual vocabulary.

The practical rule that follows:

task shape                        useful confidence signal
--------------------------------  ----------------------------------
single-label classification       logprob of the label token
extraction into a fixed schema    min logprob over the extracted spans
tool selection                    logprob of the tool-name token
short constrained generation      logprob, with care
free-form prose                   NOT logprob - use retrieval overlap,
                                  self-consistency, or a judge

For extraction, note the minimum rather than the mean. Mean logprob over an output averages away the one uncertain field, which is exactly the field you wanted to find. Take the minimum over the tokens that carry the extracted values, and ignore the tokens that are punctuation and schema scaffolding — those are near-certain and only dilute the score.

Turning a score into a queue

Do not pick a threshold. Pick a budget, sort, and take the tail. The reason is the same one that governs any screening filter: a threshold calibrated last month produces an unpredictable queue length this month, and a queue that overflows is a queue whose contents are arbitrary.

The gain from triage is worth quantifying so you know whether it is worth building. Suppose 10,000 outputs a day, a true defect rate of 2% (200 defects), and a reviewer budget of 200. Under uniform sampling the expected yield is 200 × 0.02 = 4 defects found. Now suppose the confidence score is informative enough that the bottom 2% of outputs by confidence contains 30% of all defects:

uniform 200 reviews    :  200 * 0.02          =  4 defects found
triaged bottom 200     :  200 defects * 0.30  = 60 defects found

lift = 60 / 4 = 15x

The 30% figure is an assumption, not a measurement — it is the number you have to establish for your own signal before believing any of this. Establish it by taking one week of uniform random reviews, recording each reviewed item’s confidence score, and computing what fraction of the defects fell in the bottom 2% by score. If the answer is close to 2%, your signal carries no information and the router is not worth building.

Combine signals additively on normalised scales rather than by multiplying raw values, and keep the deterministic failures out of the scored pool entirely: a schema failure goes to the front of the queue as a verdict, not as a low score competing with everything else.

Why this cannot also be your measurement

The most consequential mistake in this design is reusing the triage queue as an estimate of quality. If you review the bottom 2% by confidence and 40% of them are defective, the defect rate of your system is not 40%. It is not anything you can compute from that sample, because the sample was selected on a variable correlated with the outcome. The bias is large and it always points the same way: triaged samples make the system look far worse than it is.

Worse, the bias is not stable over time. Improve the router and the measured rate goes up while quality is unchanged. A team that reports the triage queue’s hit rate as a quality metric has built an instrument that penalises them for improving their instrument.

Run two queues. The triage queue finds and fixes defects and its metric is defects found per reviewer-hour. The audit queue is a uniform random sample, sized by the arithmetic in sampling rate for human review, and its metric is the defect rate with an interval. The audit queue is smaller, more boring, and the only one you can quote. It also doubles as the labelled data you need to know whether the router works.

Building the router

1. Ask for the confidence signal on every request, not just the ones you suspect. Requesting logprobs adds response size but not latency in any meaningful sense; deciding after the fact that you wish you had them means re-running the request against a model that may have changed underneath you.
2. Record the score alongside the request id, prompt version and task type. The score is only interpretable within a task type — a mean logprob from a classification call and one from a summarisation call are not on the same scale and must never be ranked against each other.
3. Apply the deterministic verdicts first. Schema failure, empty output, truncation, refusal flag, tool-argument validation failure. These bypass scoring.
4. Rank the remainder within task type and allocate the day’s reviewer budget across types in proportion to how much you care, not in proportion to volume.
5. Feed dispositions back in two directions. Confirmed defects become cases in the golden dataset; reviewed-and-fine items become the negative labels you need to re-estimate the router’s lift next quarter.
6. Re-check the lift when the model changes. A provider version bump changes the logprob distribution, so a threshold or a normalisation constant tuned on the old model is stale on the new one. This is one more consequence of silent model updates.