Shrinking a Failing Property Test Case to Its Minimal Prompt

In ordinary property testing, shrinking is a convenience: it saves you reading a 200-element list. Here it is the product. The artefact you hand to whoever owns the prompt is itself a prompt, and a 400-word failing prompt is a haystack while a six-word one is a hypothesis about the mechanism.

Why shrinking matters more here

Consider the two forms of the same finding. The unshrunk version is a generated support ticket containing a greeting, four sentences, a Japanese locale, an emoji, a 120-character note with a brace in it and an order id — and the assertion that the returned currency did not match the requested one. Any of those six features could be the cause. Somebody now runs experiments to find out which.

The shrunk version is: locale ja-JP, note empty, everything else at its minimum, same assertion. That is not a smaller test case, it is a different kind of object — it names the mechanism. The experiments that would have taken an afternoon were done by the shrinker, at machine speed, as a side effect of the run that found the bug. That is why the phase is worth paying for even though it is the expensive one.

How the two shrinkers actually work

The two libraries reach the same goal by different routes, and the difference shows up in what you can rely on.

Hypothesis performs internal reduction: rather than reducing the generated value, it reduces the sequence of random choices that produced it and re-runs generation. MacIver and Donaldson describe this in Test-Case Reduction via Test-Case Generation: Insights from the Hypothesis Reducer, published at ECOOP 2020. The consequence that matters to you is that every candidate the shrinker tries is a value the strategy could have produced in the first place. A shrunk st.from_regex string still matches the regex; a shrunk value from a composite strategy still satisfies whatever that strategy guarantees. You get this without writing a reducer, and it is why a shrunk case is never “invalid input” noise.

fast-check shrinks per arbitrary along paths each arbitrary declares: integers toward their minimum, strings toward empty, constantFrom toward its first argument, and oneof within the chosen branch but not across branches unless withCrossShrink is set. The practical upshot is that you influence the reported failure by how you order and structure your arbitraries, which is a real lever and a real footgun.

Both descend from the same two ideas: Claessen and Hughes’s QuickCheck, which introduced shrinking alongside property-based testing at ICFP 2000, and Zeller and Hildebrandt’s delta debugging (Simplifying and Isolating Failure-Inducing Input, IEEE Transactions on Software Engineering, 2002), which is the general algorithm for minimising a failing input.

Shrinking assumes a deterministic predicate

Every shrinker works by asking one question repeatedly: does this smaller input still fail? It assumes the answer is a fact about the input. With a model call inside the predicate the answer is a sample, and that breaks the search in two distinct ways.

A false negative — the smaller input does fail in general but happened to pass this time — ends that branch of the search early. You get a valid but unnecessarily large minimal case. Annoying, not dangerous.

A false positive — the smaller input does not really fail but happened to this time — is worse. The shrinker walks confidently toward it, keeps going, and reports a minimal case that passes when you re-run it. To the person reading the CI log this looks exactly like a flaky test, and the usual response to a flaky test is to mute it, taking the real bug with it. If your shrunk cases are often not reproducible, this is the mechanism, and the fix is in the predicate rather than in the shrinker.

Four mitigations and what each costs

• Make the predicate as deterministic as it can be. Temperature zero, a fixed seed if the provider exposes one, and a pinned model version. The last is the one people forget: a floating alias means the system under test can change between the run that found the failure and the run that reproduces it. Cost: nothing, and it should be the default.
• Define failure as “fails at least once in k”. Run the predicate k times per shrink step and treat any failure as a failure. This makes false positives much rarer and false negatives rarer still. Cost: every shrink step now costs k calls, and shrinking is already the phase with the most steps. At k=3 a shrink that took forty steps costs a hundred and twenty requests.
• Turn shrinking off while iterating. Hypothesis takes a phases setting you can supply without Phase.shrink; fast-check has endOnFailure. Use these while you are still fixing the assertion itself, then turn shrinking back on for the run that produces the report. Cost: you get an unminimised case, which is fine when you only need to know that it fails.
• Never pay for the same search twice. Once you have a minimal case, pin it: @example(...) in Hypothesis, the examples parameter in fast-check. Hypothesis also has print_blob, which prints a @reproduce_failure blob you can paste to replay the exact case — useful for a hand-off, but an explicit example is the better permanent home because it is readable. Cost: one call per pinned example per run, forever, which is the cheapest regression test in the file.

There is a fifth thing that is not a mitigation but resolves a common confusion. Shrinking toward the empty string will often hand you a degenerate case — an empty note, a zero amount — where failing is arguably correct behaviour. That is not a shrinker problem; it is the shrinker telling you the strategy generates inputs outside your spec. Fix it by constraining the strategy at the source (min_size=1, fc.integer({ min: 1 })) rather than by adding a filter, because a filter that rejects a large share of draws wastes generation and will eventually trip a health check.

From minimal case to bug report

The minimal case is the headline; six other things make it actionable. Include the exact model identifier and version, the assertion that fired with its message, the observed output verbatim, the k you used and how many of the k failed, the seed and path or the reproduce blob, and the strategy definition — because “this is the smallest input my generator can produce that fails” is a different claim from “this is the smallest input that fails”, and the reader needs to know which one you are making.

Two cautions before it leaves your machine. The failure count out of k is not a measurement of a rate — three out of three at one input says nothing about frequency across your traffic, and writing it as a percentage will be read as one. And if the shrunk case originated from replayed production data rather than from a generator, check it for personal data before pasting it into a tracker; a shrinker minimises for failure, not for anonymity, and the usual rules about what leaves your system apply to test artefacts too.