Property-Based Testing for LLM Output With Hypothesis

Hypothesis generates inputs and checks that a property holds for all of them. Applied to a language model, the model is the system under test and the property is whatever you can assert about its output without knowing what the output will say. That second half is the part worth getting right.

You generate the input, not the output

The mistake that kills these suites on day one is trying to generate an expected answer. There is no expected answer — if you could compute one you would not be calling a model. What you generate is the request: the order id, the currency, the free-text note, the locale, the number of line items. What you assert is a property of the pair.

This is the same relationship a property test has with any function too complicated to reimplement in the test. You do not check sort(xs) == [expected]; you check that the result is a permutation of the input and is ordered. For a model, finding that pair of assertions is the prerequisite, and it is worth settling before you write a line of Hypothesis.

A strategy that produces plausible requests

Build the request out of strategies whose shapes match your real traffic. st.from_regex with fullmatch=True gives you identifiers in your own format; st.sampled_from gives you the closed sets; st.text gives you the free-text field where the interesting failures live.

from hypothesis import strategies as st

@st.composite
def refund_requests(draw):
    return {
        "order_id": draw(st.from_regex(r"[A-Z]{2}-[0-9]{6}", fullmatch=True)),
        "currency": draw(st.sampled_from(["EUR", "USD", "GBP", "JPY"])),
        "amount_cents": draw(st.integers(min_value=1, max_value=1_000_000)),
        "locale": draw(st.sampled_from(["en-GB", "nl-NL", "de-DE", "ja-JP"])),
        "customer_note": draw(st.text(min_size=0, max_size=120)),
    }

st.text by default draws from the full Unicode range, which is a feature here rather than a nuisance: the note field is where a customer pastes an emoji, a right-to-left mark, a stray closing brace or a newline, and each of those has broken somebody’s structured output. If you find you need to exclude something, narrow the alphabet with st.characters rather than filtering after the fact — a .filter() that rejects most draws will eventually trip HealthCheck.filter_too_much.

The property, and the assertion inside it

The body of the test makes one model call and asserts things that must be true for every input the strategy can produce.

from hypothesis import HealthCheck, given, settings

@settings(
    max_examples=25,
    deadline=None,
    suppress_health_check=[HealthCheck.too_slow],
    derandomize=True,
)
@given(refund_requests())
def test_extraction_preserves_the_request(request):
    out = classify_refund(request)          # one model call, returns parsed JSON

    # structural
    assert set(out) == {"decision", "currency", "amount_cents", "reason"}
    assert out["decision"] in {"approve", "review", "decline"}

    # conservation: nothing invented, nothing dropped
    assert out["currency"] == request["currency"]
    assert out["amount_cents"] == request["amount_cents"]

    # bounds
    assert len(out["reason"]) <= 200

Notice what is absent. There is no assertion about which decision is correct, because the strategy generates inputs for which nobody knows. The three assertions that are there hold for every generated request, and the conservation one — the model returned the amount it was given rather than one it composed — is the cheapest hallucination check available and the one that fires most often in practice.

Settings that matter when examples cost money

Four @settings parameters change from their defaults for this kind of test, and all four are load-bearing.

• deadline=None — Hypothesis’s documented default deadline is 200 milliseconds per example. A model call is never that, so without this every example fails on time before it fails on substance.
• max_examples — the default is 100. With a network call inside the property that is 100 paid requests per property per run, before shrinking. Set it explicitly and read it as a budget line: examples multiplied by properties multiplied by CI runs per day.
• suppress_health_check=[HealthCheck.too_slow] — the health check exists to tell you your data generation is slow. Here the slowness is the model, not the generation, so the check is reporting something true and useless.
• derandomize=True — makes the generated examples a deterministic function of the test, so two CI runs on the same commit make the same calls. Without it you are also paying for a different random sample every run, which makes a spend spike impossible to attribute.

One more, which is not a setting but bites in CI: Hypothesis’s example database is a directory on disk, so a failure it found yesterday is replayed on your laptop and is not replayed on a fresh CI runner. When a counterexample matters, pin it with @example(...) above the @given. Explicit examples run in their own phase before generation, so a pinned regression case is checked first and costs one call.

assume() discards an input as invalid rather than failing on it, and target() lets you hand Hypothesis a number to maximise — output length, or a similarity score you want driven low — so generation steers toward the region where the property is fragile instead of sampling uniformly.

One structural warning about where the model call goes. Put it in the property body, not in a function-scoped pytest fixture. Hypothesis runs the body once per generated example while a function-scoped fixture is set up once for the whole test, which is precisely why HealthCheck.function_scoped_fixture exists. Hoisting a cached client is fine; anything that holds per-example state is not, and a suite that accumulates results into a fixture-held list will quietly mix examples together and report on the wrong one.

Putting it together

1. Install the dependencies: pip install hypothesis pytest. No extras are needed for this test.
2. Write the strategy first and check it alone, before any model is involved: refund_requests().example() in a REPL, or a test that just asserts the generated request validates against your own request schema. A strategy bug looks exactly like a model bug from inside the property.
3. Pin the model version in the client, not a floating alias. A property test whose system under test can change without a commit reports failures you cannot bisect — see what happens when the model behind an alias moves.
4. Start at max_examples=10 and raise it once the property is stable. The first run of a new property almost always fails on the assertion being wrong rather than the model being wrong, and discovering that costs ten calls instead of a hundred.
5. Run it: pytest -q tests/test_refund_properties.py. On failure, copy the falsifying example Hypothesis prints into an @example(...) decorator so it is checked on every future run for the price of one call.