Testing Structured Output: A Test Suite Design

A test suite is a set of assertions about a function. Yours is non-deterministic, changes under you when a provider updates a snapshot, and costs money to call. Almost every habit from ordinary unit testing needs adjusting, and the ones that survive are the property-based ones.

You cannot assert equality

assert extract(doc) == expected fails for reasons that are not bugs: a different but equally correct quote span, a reordered array, a date normalised two valid ways. Loosen it enough to stop the false alarms and it stops catching anything.

Temperature 0 does not rescue this. It removes sampling non-determinism, not all of it — batching, mixed-precision arithmetic and expert routing in sparse models all make bit-identical output across calls something no provider promises. Treat exact reproducibility as unavailable and design assertions that do not need it.

Four layers of assertion

The first three need no labels at all, which is what makes them practical. Most teams build only the fourth, find labelling expensive, build fifty examples, and end up with a suite too small to detect anything smaller than a catastrophe.

import re, pytest

def norm(s): return re.sub(r"\s+", " ", s).strip()

def check_invariants(doc: str, rec: dict):
    hay = norm(doc)
    for item in rec["line_items"]:
        assert norm(item["quote"]) in hay, f"ungrounded quote: {item['quote']!r}"
    total = round(sum(i["line_total"] for i in rec["line_items"]), 2)
    assert abs(total - rec["total"]) < 0.01, f"{total} != {rec['total']}"
    assert rec["issue_date"] <= rec["due_date"], "due before issue"
    assert len({i["source_line"] for i in rec["line_items"]}) == len(rec["line_items"])

Inverting the problem

Property-based testing needs generated inputs and a known answer. Extraction seems to offer neither — until you notice you can generate the record first and render the document from it. Then ground truth is free, arbitrarily plentiful, and covers shapes no real corpus contains:

from hypothesis import given, settings, strategies as st

money   = st.decimals(min_value=0, max_value=9999, places=2)
words   = st.text(alphabet="abcdefghijklmnopqrstuvwxyz ", min_size=3, max_size=18)
line    = st.tuples(words, st.integers(1, 40), money)

@st.composite
def invoice(draw):
    lines  = draw(st.lists(line, min_size=1, max_size=12))
    number = "INV-" + str(draw(st.integers(10000, 99999)))
    return {"invoice_number": number,
            "line_items": [{"description": d.strip(), "qty": q, "unit_price": float(p),
                            "line_total": round(q * float(p), 2)} for d, q, p in lines]}

def render(rec) -> str:
    """The inverse of extraction. Vary the layout here, not the data."""
    rows = "\n".join(f"{i['description']:<24} {i['qty']:>4} {i['unit_price']:>9.2f}"
                     f" {i['line_total']:>10.2f}" for i in rec["line_items"])
    total = round(sum(i["line_total"] for i in rec["line_items"]), 2)
    return (f"INVOICE {rec['invoice_number']}\n\n"
            f"{'Description':<24} {'Qty':>4} {'Unit':>9} {'Amount':>10}\n{rows}\n"
            f"{'TOTAL':<24} {'':>4} {'':>9} {total:>10.2f}\n")

@settings(max_examples=25, deadline=None)     # every example is a paid API call
@given(invoice())
def test_recovers_every_line(rec):
    got = extract(render(rec))
    assert got["invoice_number"] == rec["invoice_number"]
    assert len(got["line_items"]) == len(rec["line_items"])      # the count bug
    for want, have in zip(rec["line_items"], got["line_items"]):
        assert abs(have["line_total"] - want["line_total"]) < 0.01

What this catches that a hand-written corpus does not: the twelve-line invoice when your fixtures all have three, the zero-value line, the description that happens to contain a number, the quantity of 1 that reads as a currency symbol in a fixed-width column. Hypothesis will also shrink a failure to the smallest example that still fails, which usually names the cause without any debugging.

Two constraints to design around. Cap max_examples hard and set deadline=None, because each example is a network call, not a microsecond. And keep the renderer honest — if it only ever produces one layout, you are testing one layout thoroughly. Vary column widths, separators, currency placement and stray header text there, since that is the axis your real documents vary on.

Metamorphic relations

A metamorphic test asserts a relation between outputs on two related inputs rather than a value for one. It is the standard technique for testing systems with no oracle, and extraction has several natural relations:

• Permutation. Reorder independent sections. The extracted set should be equal as a set. A failure here means position is influencing extraction, which is a real and common defect.
• Irrelevant addition. Append a paragraph of unrelated text. No new entities should appear. This catches the model treating proximity as relevance.
• Deletion. Remove the line containing a field. That field must become null. This is the strongest single test for invention, and the one most likely to fail on a schema whose fields are all required and non-nullable.
• Formatting neutrality. Change whitespace, wrap lines, or convert a table to pipe-delimited. Values should not change. Failures here usually point at your document preparation rather than the model.

Each of these is a handful of lines and needs no labels. The deletion test in particular is worth writing on day one; it is the difference between a pipeline that reports absence and one that fills the gap with something plausible.

Metamorphic tests also fail informatively, which invariants often do not. An invariant failure tells you the record is wrong; a permutation failure tells you why — position is leaking into the answer, which points at your chunking or at a prompt that mentions “the first” something. The name comes from the software testing literature, where metamorphic testing was developed precisely for systems without an oracle, and extraction is a textbook instance of that shape: nobody can say what the right answer is, but everyone can say which pairs of answers are inconsistent.

A flakiness budget

• Assert on a rate, not on a run. For anything genuinely stochastic, run k times and require m passes — k = 5, m = 4 is a reasonable start. Record m/k as a metric so a slide from 5/5 to 4/5 is visible before it becomes 3/5.
• Separate the fast suite from the paid suite. Schema validation, invariant checks and migrations run offline against recorded responses on every commit, in milliseconds. Live calls run nightly. A test suite that costs twenty dollars per push gets disabled within a month.
• Record and replay. Store real responses as fixtures and run the parsing, validation and repair paths against them offline. Most of your code is not the model, and none of it needs the network to be tested.
• Pin the model id, and test the pin. An alias that resolves to a new snapshot is a silent dependency upgrade. Pin the dated id and have one nightly job that runs the suite against the alias, so you learn about the change before it is forced on you.
• A failing test must name the record. Print the input, the raw response and the validator message. A red build that says assert 7 == 10 against a model call is an hour of work; one that prints the document and the response is five minutes.