Writing Invariants for LLM Output You Can Actually Test

Every property-based testing guide starts with an example where the invariant is obvious — reversing a list twice gives the list back. Nothing about model output is obvious in that way, and that is the actual problem. This page is about finding the property, not about the framework that runs it.

What counts as an invariant here

An invariant is a predicate over the pair (input, output) that must be true for every input your strategy can generate, with no reference to any particular input. If you cannot write it without mentioning a specific example, it is a test case, not a property, and it belongs in a golden file instead.

The second requirement is that it must be checkable by code you trust. “The answer is correct” fails this: it needs an oracle, and the absence of an oracle is why you are here. “The answer cites only documents that were in the context” passes: it is a set membership test on strings you already have.

Five families that hold up

• Structural. The output parses. It validates against the schema. Every required key is present. Every enum-typed field holds a value from its enum. The array has between one and five elements. These are the easiest to write and the first to fire, and they stay valuable even under constrained decoding, because constrained decoding guarantees the shape and not the enum membership of a field the schema typed as a free string.
• Conservation and provenance. Every entity in the output appears in the input. Every number in a summary is a number that occurs in the source. Every quoted span is an exact substring of a supplied document. Every cited id is in the retrieved set. This family is the strongest cheap hallucination check that exists, it needs no second model, and it is the one most teams have never written down.
• Bounds. The output is at most 200 characters. At most one tool call is emitted. The total cost of the call is under a ceiling. The response arrives within an SLO. A refusal is never accompanied by a tool call. Bounds are how a property test catches the failure mode where the model is right but ruinous, which assertions about content never see.
• Negative and redaction. The output never contains the API key that was in the system prompt. It never contains a string from the deny list. It never contains an email address that was not in the input. Negative invariants are unusually well suited to property testing because they hold for every input by construction, so a generator exploring weird inputs is doing exactly the work you want — this is the shape a prompt injection defence is testable as.
• Relational. Instead of a property of one output, a relation between two: the same input phrased differently gives the same label; adding a disqualifying fact never moves the decision toward approval. These are metamorphic relations, and they are the family that reaches correctness-adjacent questions the other four cannot touch.

Properties that only look like invariants

“The output equals this string.” A pinned example, and a brittle one. It fails on a reworded system prompt, a model point release and a different sampling seed, all of which are changes you do not care about. It is the failure this whole cluster exists to correct.

“The output is polite / helpful / on-brand.” You can assert this, but only by putting a second model in the assertion. That is a legitimate technique and it has a cost that must be stated: the judge has its own error rate and its own drift, so a failure now means “the pair of models disagreed”. Use it where nothing cheaper exists, keep the judge’s version pinned, and never use it for something a regex could have decided.

“The reasoning mentions the discount rule.” An assertion on prose, dressed as an assertion on behaviour. It fails when the model says the same thing in different words and passes when the model recites the rule and then ignores it. If the rule matters, make the model emit a structured field that names which rule applied and assert on the field.

“Two runs give the same output.” Not an invariant even at temperature zero. Greedy decoding is deterministic in theory; in practice batching changes the order of floating-point reductions on the provider’s hardware, and different batch compositions can produce different tokens from identical inputs. Assert on the extracted decision, which is stable, rather than the token stream, which is not.

How strong to make it

There is a dial here and both ends are useless. An invariant so weak that it can never fail — “the output is a string” — costs a call per example and tells you nothing. An invariant so strong that it fails on five per cent of valid inputs is a flaky test, and a flaky test gets muted, which is worse than not having written it.

The calibration procedure is: write the strongest form you can state, run it over your existing labelled set rather than over generated data, and weaken it only where it fails on a case you agree is correct. Weaken by widening the predicate, not by adding a retry. “At most 200 characters” becomes “at most 260”; it does not become “at most 200, retrying three times”, which converts a real constraint into a cost multiplier and hides the regression it was written to catch.

Deriving one from a spec you already have

Three sources, in increasing order of how productive they are.

The output schema is the first. Every required, every enum, every maxLength, every minItems is an invariant somebody already wrote down; a validator turns the whole schema into one assertion. The prompt is the second: any instruction of the form “answer in at most 40 words”, “reply only in the language of the question” or “never mention a competitor” is a testable predicate that you wrote and are not currently checking.

The third and best is the consumer. Read the code that receives the model’s output and list everything it would crash on, silently mishandle or route wrongly. Every if and every unchecked index in that code is an assumption about the producer, and each assumption is an invariant with a known blast radius, because you can see exactly what breaks when it does not hold. Properties derived this way are the ones whose failures are worth waking somebody for.