Property-Based Testing a JSON Schema an LLM Must Satisfy

A JSON schema sits between a model and the code that consumes it, and there are two independent ways for that pairing to fail: the consumer mishandles something the schema permits, or the model cannot produce something the schema demands. Only the second needs a paid call, and most suites test only the second.

Two directions, one schema

Direction one runs schema to instance to consumer, with no model anywhere. It answers: does my code handle every value this schema allows? Direction two runs input to model to instance to validator. It answers: can the model actually satisfy this schema on my traffic?

Separating them matters because they have opposite economics. Direction one is free, fast and exhaustive-ish, so run thousands of examples in unit-test time. Direction two costs a call per example, so run twenty and choose them well. A suite that conflates the two ends up paying model prices to discover that a nullable field was dereferenced.

There is a third thing the schema is doing that neither direction tests, and it is worth naming so you do not go looking for it here: the schema is also part of the prompt. Its field names and descriptions are read by the model, so renaming reason to justification changes the output even though nothing about validation changed. That makes the schema a versioned artefact with the same care requirements as the prompt itself, and it is why the two should move through review together.

Generating instances from the schema

hypothesis-jsonschema exists for exactly this. Its public surface is essentially one function, from_schema, which takes a schema and returns a Hypothesis strategy for values that satisfy it.

from hypothesis import given, settings
from hypothesis_jsonschema import from_schema

SCHEMA = {
    "type": "object",
    "required": ["decision", "amount_cents", "currency"],
    "additionalProperties": False,
    "properties": {
        "decision": {"enum": ["approve", "review", "decline"]},
        "amount_cents": {"type": "integer", "minimum": 0},
        "currency": {"enum": ["EUR", "USD", "GBP", "JPY"]},
        "reason": {"type": ["string", "null"], "maxLength": 200},
        "evidence_ids": {"type": "array", "items": {"type": "string"}},
    },
}

@settings(max_examples=500)
@given(from_schema(SCHEMA))
def test_consumer_handles_everything_the_schema_allows(instance):
    result = apply_refund_decision(instance)   # no model call
    assert result.status in {"queued", "held", "rejected"}

What this finds is a specific and very common bug: the schema types reason as string-or-null, and the consumer calls .strip() on it. Or evidence_ids has no minItems, so the empty array is legal, and the consumer indexes element zero. The model has probably never returned either — until the day it does, and then the failure is in production rather than in the suite.

Generating the inputs that break it

The second direction generates the prompt inputs and validates whatever comes back. The interesting engineering is in choosing input shapes with a reason to break structured output, rather than sampling uniformly:

• Free text containing the delimiters of the output format. A customer note with a brace, a quote, a backslash or a code fence in it. Without constrained decoding this is the single most productive input class.
• Text in a script the schema’s enum is not in. A Japanese note with an English enum tends to produce a translated enum value, which is valid JSON and an invalid instance.
• Inputs where the honest answer is “not enough information” and the schema has no way to say so. This is the important one, and the finding is a design bug rather than a model bug: a required decision enum with no insufficient_data member forces the model to invent one of the three. The fix is in the schema.
• Inputs at the boundary of a numeric constraint — an amount of zero against minimum: 1, a note that is exactly maxLength. Generators reach these; hand-written examples do not.
• Long inputs that push the schema toward the truncation boundary. A response cut off by the output token limit is invalid JSON, and it is worth asserting on the finish reason separately so this failure is labelled rather than reported as a parse error.

Validating, and what to assert on a failure

Use a real validator, not a try/except around json.loads. Python’s jsonschema package gives you iter_errors, which returns every violation rather than the first, and each error carries a json_path pointing at the offending location. That difference decides whether a failure message is actionable.

from jsonschema import Draft202012Validator

VALIDATOR = Draft202012Validator(SCHEMA)

@settings(max_examples=20, deadline=None)
@given(refund_requests())
def test_model_output_validates(request):
    raw = extract_decision(request)          # one model call, parsed JSON
    errors = sorted(VALIDATOR.iter_errors(raw), key=lambda e: e.json_path)
    assert not errors, "schema violations:\n" + "\n".join(
        f"  {e.json_path}: {e.message}" for e in errors
    )

One caveat about the whole exercise. If your provider supports strict structured output — constrained decoding against the schema rather than an instruction to please return JSON — then the structural assertions become trivially true and stop earning their keep. That is a good outcome and it means the test has to move: assert on cross-field consistency (a decline decision must carry a non-null reason), on provenance (every evidence_ids entry was in the retrieved set), and on the semantics of enum choice, none of which constrained decoding touches. The distinction between the two provider modes is worth understanding first — see JSON mode versus structured outputs and testing structured output.

The loop

1. pip install hypothesis hypothesis-jsonschema jsonschema pytest.
2. Put the schema in one file and import it into both the client and the tests. A schema that exists twice will differ within a month, and the test will be validating against the copy the model was not given.
3. Write the direction-one test first, with a high max_examples. It runs in seconds and it will find something.
4. Write the direction-two test with max_examples=20, deadline=None, and an input strategy that deliberately includes the shapes listed above rather than uniform text.
5. When direction two fails, decide first whether the schema or the model is wrong. If the schema cannot express the correct answer for the failing input, fix the schema; adding a retry to make the model guess harder is the wrong repair and it will hold until the same input arrives in production.