Pattern: Verify With a Second Call

Checking generated output with a second model call is either one of the highest-return things in this cluster or a way to double your bill for a rubber stamp. Which one you get is decided almost entirely by whether the second call is independent of the first in a way that matters.

What this is not

Three things are commonly called verification and only one of them is.

• Not self-critique in the same conversation. Asking the model whether its own answer is correct, with that answer in context, mostly produces agreement. The prior answer is context, and continuing consistently with context is the machine’s entire function. This is the same mechanism as sycophancy and it does not turn off because you framed the turn as a review.
• Not a confidence score from the generator. A model asked how sure it is produces a number that reads like a probability. Whether it behaves like one is a calibration question you must answer with your own data before gating on it.
• Not deterministic validation. Schema checks, arithmetic checks, database lookups and permission checks are cheaper, total and certain. Do all of them first. A model verifier is what remains for claims that cannot be checked mechanically, and using one where a schema would do is expensive theatre.

Real verification is a separate evaluation of a specific claim, made with information the generator did not have or in a framing the generator was not in, by something that has not already committed to an answer.

When it pays

The arithmetic is short and it hinges on one term most teams have never estimated: what an undetected error costs. Let g be the cost of the generation, v the cost of the verification, e the error rate without it, d the fraction of those errors the verifier catches, and K the cost of one error reaching a user.

cost without verification per request:  g + e * K
cost with verification per request:     g + v + e * (1 - d) * K + e * d * R
                                                                  ^ cost of
                                                                    handling a
                                                                    caught error
                                                                    (retry, or a
                                                                    human, or a
                                                                    degraded answer)

worth it when:   v + e * d * R  <  e * d * K
        <=>      v              <  e * d * (K - R)

  Verification pays when its price is less than the error rate times the
  catch rate times the AMOUNT BY WHICH catching an error is cheaper than
  letting it through.

Two readings of that:
  · If K is small — a wrong tag on an internal record — no verifier is worth
    it, however cheap, because (K - R) is near zero.
  · If K is large — a wrong number in a customer's invoice, a wrong medical
    or legal statement — verification pays even at a poor catch rate d,
    which is why the pattern belongs on high-stakes outputs and nowhere else.

The practical consequence is that verification is a per-feature decision, not a house style. Applying it everywhere is how a three-call feature becomes a six-call feature with no change in outcomes; applying it to the two outputs where K is genuinely large is one of the better trades available.

Note also that v need not be comparable to g. A verifier answering a narrow yes/no question with a short output can run on a much cheaper model than the generator, because checking is an easier task than producing — which is what makes the inequality satisfiable at all.

Independence is the whole trick

A verifier that shares the generator’s blind spots catches nothing and costs full price. Independence comes in degrees, and they are worth ranking because the cheap ones are also the weak ones.

The single most common implementation error is leaving the generation transcript in the verifier’s context because it was convenient. That one decision converts the pattern into self-critique and removes most of its value while keeping all of its cost.

Three shapes that work

Grounding check

Give the verifier the source and the output and ask it to list claims in the output not supported by the source. Narrow, enumerable, checkable — and the answer is a list you can act on rather than a score. This is the highest-yield verification for anything retrieval-backed, and it composes with grounding as an architecture rather than replacing it.

Constraint check

Where the requirements are explicit — must mention the deadline, must not give financial advice, must be under 200 words, must use the customer’s name — hand the verifier the requirement list and ask which are violated. Several of these will turn out to be expressible in code once written down, which is a good outcome: every constraint you move into code is one you stop paying to check.

Independent redo and compare

Generate the answer twice, independently, and compare. Disagreement is a strong signal of an unreliable answer even when you cannot tell which one is right. This is the cheapest to implement and the most expensive to run, and it works best where answers are short and comparable — a number, a label, an extracted field — rather than prose, where two different phrasings are not disagreement. Sampling more than twice and taking the majority is self-consistency, with the same trade at a higher multiple.

What to do when they disagree

The design decision people postpone is what a failed verification actually triggers, and the wrong default — regenerate and hope — turns a verifier into an expensive random number generator.

• Regenerate at most once, with the objection attached. The second attempt should know what the verifier said; a blind resample has roughly the same error rate as the first. If the second attempt also fails, stop — you are now paying repeatedly to draw from a distribution that is not producing what you need.
• Degrade rather than loop. Return the part of the answer that verified, or a deterministic fallback, or an honest absence. A feature that can express “I could not verify this” is more valuable than one that returns something unverified with the same confidence as everything else.
• Escalate when the stakes justified the verifier in the first place. If K was large enough to pay for verification, it is large enough to pay for a person. A failed verification is the highest-quality routing signal you will ever have for a review queue, because it is precisely the subset of traffic where human attention has the most leverage.
• Record the disagreement either way. The rate at which the verifier objects is a live quality metric that needs no labelling, and a step change in it is one of the earliest signals that something upstream — a prompt, a model version, an input distribution — has moved.