Reducing Flakiness by Testing Properties Instead of Exact Text

Most flaky model tests are not flaky because of the model. They are flaky because the assertion asks a question the model was never promised to answer the same way twice. Change the question and the test stops flaking — not as a mitigation, as a fix.

Why an exact-text assertion cannot be stable

The model returns a probability distribution over the next token, and a sampler draws from it. Setting temperature to zero makes the draw greedy but does not make the pipeline deterministic: batching, hardware, kernel selection and floating-point non-associativity all move the low-order bits of the logits, and two tokens with nearly equal probability can swap places between runs. OpenAI is explicit about the limit — its documentation describes seeded sampling as a best-effort mechanism, points at system_fingerprint as the signal that the backend configuration changed, and notes that responses can still differ even when parameters and fingerprint match.

So assert response == "The refund was processed." is asserting on something nobody guaranteed. It will pass for weeks and then fail on a Tuesday because the model wrote “Your refund has been processed.”, which is not a defect by any definition your product uses. The test is not detecting a change in behaviour; it is detecting a change in wording, and it cannot tell you which it found.

The replacement is to assert on something that must hold for every acceptable output. That is a property, and the set of useful properties is smaller and more concrete than it sounds.

Four families of stable assertion

• Structural. The output parses, and it validates against a schema. With a provider’s structured-output mode this is close to guaranteed, but assert it anyway — the assertion is what tells you the mode silently stopped being applied. Validate with the same schema object your production code uses, so the test cannot drift from the contract. See testing structured output.
• Referential. Every identifier in the output exists in the input. This is the single highest-value assertion available against a model, because it catches fabrication mechanically: if the model returns an order id, a product code or a citation key, it must appear in the context you supplied. No judgment call, no rubric.
• Behavioural. The right tool was called, with arguments that validate against the tool’s schema, the right number of times. Assert the tool name and the argument structure, never the model’s explanation of why it called it. When a tool call does not fire covers the failure side of this.
• Negative. Nothing forbidden appears: no unredacted email address or card number, no key from the system prompt, no reference to a document outside the retrieved set. These are the assertions worth keeping strict, because a false positive costs you a rerun and a false negative costs you an incident.

import re
from pydantic import BaseModel, ValidationError

class Extraction(BaseModel):
    order_id: str
    reason: str
    refund_cents: int

EMAIL = re.compile(r"[\w.+-]+@[\w-]+\.[\w.]+")

def test_extraction_properties(model_client, transcript):
    raw = model_client.extract(transcript)

    # structural: it parses and validates
    parsed = Extraction.model_validate_json(raw)

    # referential: nothing invented
    assert parsed.order_id in transcript

    # domain invariant: a refund is a non-negative amount
    assert 0 <= parsed.refund_cents <= 1_000_00

    # negative: no address leaked into the free-text field
    assert not EMAIL.search(parsed.reason)

Not one of those assertions depends on the sampler. Each of them fails only when something is genuinely wrong, which is the property that makes a test worth keeping.

Metamorphic relations

The families above test one output in isolation. A metamorphic relation tests two outputs against each other, and it is how you assert on behaviour you cannot specify absolutely. The pattern: transform the input in a way whose effect on the output you can state, then assert that relation rather than either output.

• Permutation. Shuffle the order of the retrieved documents. The set of extracted entities should be unchanged. If it is not, you have found position sensitivity, which is a real defect and one that no single-output assertion can see.
• Irrelevant addition. Append a paragraph about something unrelated. A classification should not move. If it does, the prompt is not robust to the retrieval quality you will have in production.
• Paraphrase. Restate the user’s question in different words. The routing decision, the tool chosen, or the selected category should be the same, even though the prose will not be.
• Monotonicity. Add one more piece of evidence for the correct answer. A confidence field should not decrease, and an extracted set should not shrink.

import random

def test_entity_extraction_is_order_invariant(model_client, docs):
    baseline = set(model_client.entities(docs))

    shuffled = docs[:]
    random.Random(1234).shuffle(shuffled)   # seeded: the test is reproducible
    assert set(model_client.entities(shuffled)) == baseline

Seed the shuffle. An unseeded permutation makes the test itself a source of variance, and you will spend an afternoon deciding whether the model or the test was responsible — the exact confusion isolating provider flakiness from your own exists to resolve.

Generating the inputs

Property assertions pair naturally with generated inputs, and Hypothesis is the mature tool in Python. Its decorators are @given, which fills arguments from strategies, and @settings, which controls the run — including max_examples (100 by default) and deadline, a per-example soft time limit that defaults to 200 milliseconds and which any real model call will blow straight through.

from hypothesis import given, settings, strategies as st

@settings(max_examples=20, deadline=None)   # deadline=None: a network call is slow
@given(order_id=st.from_regex(r"[A-Z]{2}-\d{4,6}", fullmatch=True),
       noise=st.text(min_size=0, max_size=200))
def test_order_id_is_never_invented(model_client, order_id, noise):
    transcript = f"Customer: my order {order_id} arrived broken. {noise}"
    parsed = model_client.extract(transcript)
    assert parsed.order_id == order_id

Two warnings. Set max_examples deliberately: the default of 100 means 100 billed model calls per test, which is a real cost decision and not one to leave to a default. And Hypothesis shrinks a failing case to a minimal example by re-running it, so a nondeterministic assertion makes shrinking unreliable — another reason the assertion must be a genuine property rather than a coin flip.

What you give up

Property assertions are strictly weaker than exact matching, and it is worth being honest about the gap. A schema-valid, referentially sound answer can still be unhelpful, badly formatted for your UI, rude, or wrong in a way no invariant captures. Nothing on this page catches a summary that is technically accurate and useless.

That gap is real and it is not filled by tightening the assertion back up — it is filled by a different mechanism at a different cadence: a scored evaluation set with a rubric, run on a schedule, reporting a distribution rather than a boolean. See evaluation rubrics and building a golden dataset. The division of labour is the point: properties belong in CI, where a binary answer is required in ninety seconds, and quality scoring belongs in an eval run, where it can afford to be slow, sampled and statistical.