Generating Test Inputs With a Grammar Instead of Examples

A file of forty hand-written test prompts covers the forty situations somebody thought of. A grammar with six rules and four alternatives each covers thousands, and the combinations nobody thought of are precisely the ones that break things. The cost is writing the grammar, which is about an hour.

Why a grammar beats a list of examples

Random text is useless as a model input — it is out of distribution, the model will handle it strangely, and you will learn nothing you can act on. What you want is text that is realistic in shape and varied in combination, and a grammar is exactly the object that expresses “realistic in shape” formally.

The combinatorics do the work. Four subjects, four complaint verbs, three optional constraints, an optional greeting and two closings is several hundred distinct tickets from about twenty lines of grammar. A human writing examples produces variety along one axis at a time; a grammar produces the cross product, which is where the interaction bugs live — the ticket that is both a multi-item complaint and phrased as a negation, which nobody wrote by hand because nobody thinks in cross products.

There is a second benefit that outlasts the tests. The grammar is a written-down model of what your system is supposed to accept, in a form somebody can review. An example file cannot be reviewed for completeness, because nothing about a list of forty strings tells you what the forty-first would have been. A grammar makes the gap visible: a reviewer reads the problem rule, notices that “arrived damaged” is missing, and adds one alternative instead of writing ten new examples. The input space becomes something you maintain rather than a pile you accumulate.

Writing the grammar

Lark is the usual choice in Python because Hypothesis integrates with it directly. The grammar is EBNF; terminals are quoted strings or regular expressions, and alternatives are separated by a pipe.

GRAMMAR = r"""
start:      greeting? complaint constraint? closing?
greeting:   "Hi, " | "Hello, " | "Hey there. "
complaint:  subject " " problem
subject:    "my order" | "the payment" | "my subscription" | "the last delivery"
problem:    "never arrived" | "was charged twice" | "is stuck in transit"
          | "was cancelled without warning"
constraint: " I need this resolved before " DAY "."
          | " I've already contacted support twice."
          | " I do not want a replacement, only a refund."
closing:    " Thanks." | " Please advise." | " Regards, a customer."
DAY:        "Friday" | "the weekend" | "the end of the month"
"""

Two things about this grammar are deliberate. Every alternative is a shape you would recognise in your ticket queue, because the point is realism rather than coverage of the character set. And the recursive rule is absent: nothing here refers to itself, so every derivation terminates quickly and the generated strings stay ticket-sized.

Driving it from a property test

Hypothesis ships a Lark integration in its lark extra. The entry point is from_lark, whose current signature takes the compiled grammar positionally and everything else by keyword:

from hypothesis import given, settings, strategies as st
from hypothesis.extra.lark import from_lark
from lark import Lark

tickets = from_lark(Lark(GRAMMAR), start="start")

@settings(max_examples=30, deadline=None)
@given(tickets)
def test_every_ticket_gets_a_valid_triage(ticket):
    out = triage(ticket)
    assert out["category"] in CATEGORIES
    assert out["priority"] in {"low", "normal", "urgent"}
    assert len(out["summary"]) <= 200

The explicit keyword argument is the one worth knowing about. A terminal declared in the grammar with %declare has no definition, and explicit maps its name to a Hypothesis strategy — which is how you inject a realistic order id from st.from_regex, or a date, or a value sampled from a real catalogue, without expressing it in EBNF. The grammar handles structure; strategies handle the leaves that need to look like your data.

Outside Python the same idea has tooling: Grammarinator, presented by Hodován and Kiss at A-TEST 2018, generates test inputs from ANTLR grammars, which is the pragmatic route when a grammar for your format already exists. In JavaScript, fast-check’s fc.letrec plays the same role — you define named arbitraries that reference each other through the tie function, which is a grammar written as arbitraries instead of as EBNF.

A grammar also composes with metamorphic relations more neatly than a fixed example set does, and this is where it earns the most. The grammar supplies the source input; the transformation supplies the follow-up. Better still, some transformations can be expressed inside the grammar: generate a derivation, then re-render it with the greeting removed, or with the constraint clause moved before the complaint, and you have a mechanically meaning-preserving pair with no paraphrase model involved. That gives you hundreds of invariance pairs for the cost of writing one renderer.

Keep the derivation, not just the string

This is the part that turns grammar generation from a novelty into a diagnostic. A failing string on its own tells you almost nothing: you get one 90-word ticket and have to guess which of its features mattered. What you want is the class of failure — “every failing case used the was cancelled without warning alternative together with a constraint clause” — and that is the derivation, not the output.

Hypothesis gives you two tools for this. event() records a labelled occurrence and the run summary reports how often each label appeared, so you can see whether generation is actually reaching every alternative or spending ninety per cent of its budget on one branch. note() attaches a value to the test case and prints it only on failure, which is where the derivation belongs. If you build the ticket from strategies yourself rather than through from_lark, you can keep both the string and the list of chosen alternatives in the drawn value and assert on the string while reporting both.

Where grammars go wrong

• Unbounded recursion. A rule that refers to itself with no bias toward the base case produces mostly enormous strings, which burns your token budget on inputs no user would send. If you need recursion, make the terminating alternative more likely, or use st.recursive, whose max_leaves parameter bounds the size explicitly.
• Uniform generation is not uniform coverage. A rule with four alternatives nested inside one that is optional is reached far less often than the top-level rules. Measure with event() before you conclude that thirty examples covered the grammar.
• The grammar has your blind spots. It contains only the shapes you wrote down — combinatorially expanded, but still yours. It cannot generate the phrasing you never imagined, which is the same limit the example file had; the grammar just fails at it more slowly. Derive the alternatives from real tickets, not from imagination.
• Generated text is grammatical but not idiomatic. Real users write fragments, misspell things, paste order confirmation emails and switch language mid-sentence. Treat grammar output as a structured stress test and keep a sample of real traffic as the other half of the suite. Neither half covers the other.
• Every example is a paid call. Thirty derivations is thirty requests. The grammar makes it trivial to ask for ten thousand; nothing in the tooling will stop you, and the bill arrives regardless.