Anti-Pattern: Trusting Output Without Validation
5 min read · updated August 3, 2026
Nobody decides to trust model output. What happens is that the output is well-formed, the parse succeeds, the tests pass, and there is no moment at which anyone is prompted to ask whether the content is acceptable — because in every other part of the system, a value that parses is a value that is fine.
Why the missing check passes review
Consider what a reviewer sees. A function calls an API, parses JSON, and returns a typed object. There is error handling for the network call and a schema type on the result. This is, by the standards of every other integration in the codebase, a careful piece of work.
The reason it is not careful enough is a property of this dependency that no other one has: a well-formed response carries no information about whether it is correct. A payment API that returns a successful charge object has charged the card. A model that returns a syntactically perfect object describing an invoice has asserted something, and the confidence of the assertion is uncorrelated with its truth. The type system checks the shape of the claim. Nothing checks the claim.
The second reason is that validation feels redundant when the model is good. If the failure rate is a few per cent, every manual test passes, and the check you would have written appears to be dead code. It is not dead code; it is code whose trigger rate is a few per cent of a large number.
Three different questions
Most implementations collapse three unrelated checks into one “validate” step and then find that fixing one breaks the others. They fail differently and should be separate.
| Question | Description |
|---|---|
| Is it well-formed? | Does it parse, match the schema, contain the required fields, use permitted enum values? Cheap, deterministic, total. This is the only one most code does, and the only one a provider's structured-output mode helps with. |
| Is it permissible? | Is this an action or a value the system is allowed to accept — an amount within limits, a recipient on the allow-list, a status transition the state machine permits, a link to a domain you trust? Also deterministic, and entirely your business logic. A model has no idea what your rules are. |
| Is it true? | Does it correspond to reality — does the cited row exist, does the total match the line items, is the quoted sentence actually in the source document? Sometimes checkable, sometimes only by a person, never by asking the same model again. |
Sorting a bug into one of these three tells you where the fix belongs. A malformed field is a schema and decoding problem. A payment to an unknown account is a permission problem and would have happened regardless of how the value was produced. A confident fabrication is a truth problem, and prompt changes will move its rate around without ever removing it.
A layer per output type
Validation is not one technique. What is available depends entirely on what the output is, and the good news is that the cheapest cases are also the most common.
Structured data
Validate against your own schema, always, regardless of what the provider promised — a strict-mode guarantee from one model is not a guarantee on the fallback path. Beyond shape, check the invariants only you know: line items summing to the total, dates within a plausible range, an identifier that must resolve to an existing row. Cross-field arithmetic is where extraction errors surface, and it costs nothing. Repair-versus-retry when it fails is a decision with its own economics.
Classifications and enums
The check is that the label is in the set — and the interesting part is what you do with a label that is not. Silently coercing to a default is how a taxonomy quietly acquires a bucket that means “the model said something else”. Keep an explicit unclassified outcome, count it, and route it, because its rate is your earliest indicator that inputs have shifted.
Code and queries
Parse it. A syntax check is free and catches a real share of failures before execution. Beyond that, the check is capability-based rather than semantic: run it with the permissions it should have, not the permissions you have. A generated query executed read-only against a restricted role cannot do damage regardless of what it says, which is a far stronger property than any inspection of the query text.
Free text shown to a user
The hardest case, because there is no schema. What is still checkable: length bounds, language, the absence of prompt fragments or system text, the absence of markup that renders as a link or an image to an arbitrary host — that last one is not cosmetic, it is the exfiltration channel and it belongs in the renderer. Where the text is supposed to be grounded in a source, quoted spans can be verified against that source exactly.
Actions and tool calls
Validate the call, not the sentence that motivated it. Parameters against a schema, then the action against policy: this recipient, this amount, this resource, this user’s permissions. Treat model output as an untrusted request to your own API, because that is exactly what it is — the same posture tool security starts from.
What a failed validation should do
The default reflex is to retry, and it is wrong often enough to need a rule. Retry is appropriate when the failure is plausibly a draw from the distribution — a malformed JSON object, a truncated array, a missing optional field. It is inappropriate when the failure is about the request or the world, because a second sample will fail the same way and you will have paid twice to learn it.
- Well-formedness failure — retry once, ideally with the validation error appended so the second attempt has more information than the first. Then stop. A second identical failure is a schema or prompt problem, not luck.
- Permission failure — never retry. The model proposed something you do not allow; sampling again is asking a different way for permission you already denied. Reject, log the attempted action, and alert if the rate rises, because a climbing rate of impermissible proposals is what indirect injection looks like from inside your own system.
- Truth failure — do not retry, escalate. Either degrade to a path that does not require the claim, or hand it to a person. A model that fabricated a citation once will fabricate a different one on the next sample, and you will have no way to know which.
Whatever the outcome, count validation failures by type as a first-class metric next to latency and cost. It is the cheapest quality signal you will ever have, it needs no labelling, and it moves before users complain — which makes it the leading indicator that a provider changed something underneath you.
Validating with a model, carefully
When the check cannot be deterministic, a second model call is the remaining option, and it has one hard requirement: it must be independent of the first in a way that matters. Asking the same model, in the same conversation, whether its answer was correct is close to worthless — the answer is in the context, and agreement with context is what the machine does. That is the mechanism behind sycophancy, and it does not switch off because you phrased the question as a check.
Useful independence means a fresh context, a different framing, and ideally a different model. Give the checker the source and the claim and ask a narrow, verifiable question — is every fact in this summary present in this document — rather than a general one. Narrow questions about grounding are the case where a model judge is reliable enough to gate on; broad quality judgements are not, and the failure modes to expect are catalogued in judge bias.
The economics of when a second call is worth its price is a separate question with a clean answer, and it is worked through on the verification-pattern page below.