Mutation Testing a Prompt Test Suite to Check It Actually Catches Failures

A green prompt suite tells you nothing on its own. It is equally consistent with a suite that guards the behaviour you care about and one whose assertions are so loose that no realistic breakage could fail them. Mutation testing is the standard way to tell those two apart, and it transfers to prompts almost unchanged.

What mutation testing borrows

In ordinary mutation testing a tool rewrites your source — flips > to >=, deletes a statement, replaces a return value with null — and reruns the suite against each rewrite. Each rewrite is a mutant. If the suite fails, the mutant is killed. If the suite still passes, the mutant survived, and a surviving mutant is a proof that some real change to behaviour goes unnoticed. The mutation score is killed divided by total.

The reason this is worth importing is that it measures a property of the suite rather than a property of the model. Model quality moves under you every time a provider ships a point release; the question “would this suite notice if the output-format instruction vanished” has a stable answer. That makes the score something you can compare against itself over quarters, which is more than most numbers in this area manage.

What does not transfer is the tooling. Stryker and mutmut mutate an abstract syntax tree, and a prompt has no semantics for them to perturb. You supply the operator set yourself, and it is small enough to write by hand.

Mutation operators for a prompt

A useful operator is one that a careless human could plausibly commit. Mutating a prompt into gibberish proves nothing, because any suite catches that. The interesting ones are the edits that look harmless in a diff:

• Delete the output-format instruction. Remove the sentence that says to answer with JSON matching a schema. If nothing goes red, your suite is not validating structure — see asserting JSON validity.
• Delete one constraint sentence. Any single “never mention” or “always include” line. Each constraint in a system prompt should have exactly one test that dies without it. Constraints with no such test are decoration.
• Remove every few-shot example. Examples are usually carrying format and tone. A suite that cannot tell they are gone is not testing what the examples are for.
• Invert a polarity word. must not to must, exclude to include. This is the mutation closest to a real typo and the one most likely to survive.
• Downgrade the model. Swap the configured model for the cheapest one in your catalogue. If every case still passes, your cases are easier than your product.
• Move the sampling temperature. Set it to 1.5. A suite that only ever exercised one sample per case will often not notice.
• Truncate the system prompt after its first paragraph. This simulates the common real bug where a template render or a context trim silently drops the tail.
• Shuffle the tool descriptions. Order should not matter; if it does, that is worth knowing before a refactor discovers it for you.

Running mutants against a stochastic model

The one adaptation the technique genuinely needs is that a single run cannot decide whether a mutant was killed. A case that fails once may have failed to sampling noise, and crediting that as a kill inflates the score with luck.

Establish the baseline first: run the unmutated suite n times and require it green in all n. If the baseline is flaky, stop — a mutation score computed on a flaky baseline is uninterpretable, and the flakiness is the more urgent bug. Then run each mutant n times and count it killed only if a majority of runs fail. Five runs is a reasonable starting point and the number matters less than fixing it, because the score is only comparable against itself.

# one mutant, five samples, majority rules
for i in $(seq 1 5); do
  PROMPT_MUTANT=drop-format-instruction \
  pytest tests/prompts -q --tb=no >/dev/null 2>&1 \
    && echo pass || echo fail
done | sort | uniq -c

The mutant is applied through configuration rather than by editing the prompt file in place, so that a crashed run cannot leave a damaged prompt on disk. Load the prompt through one function, and let that function apply a named mutation when an environment variable asks for it. That function should be unavailable in production builds.

A survivor names a missing assertion

Survivors are the entire output. Each one is a sentence of the form “we can delete this and every test still passes”, and there are exactly three honest responses.

• Add the assertion. The usual case. The constraint matters, nothing was checking it, and now something does. This is where a suite gains teeth.
• Delete the instruction. Also common and much less popular. If no test depends on a sentence and no one can say what it buys, it is occupying context and tokens for nothing. Prompts accumulate this kind of sediment — see how prompts decay.
• Record it as untestable. Some constraints are about tone or judgement and have no cheap predicate. Write that down next to the constraint rather than leaving a silent gap that the next mutation run rediscovers.

What you should not do is chase a mutation score of 100%. Past a point, the only way to kill the remaining mutants is to assert on model prose, and an assertion on an exact sentence is a test that fails on the next provider point release while catching nothing.

What it costs and how often to run it

Cost multiplies by mutants times samples. Eight operators at five samples is forty full suite runs, which is why this is a quarterly or pre-release exercise and not a per-commit gate. Three things bring it down without changing what it tells you: run the mutants against the cheapest model you support rather than the production one, since you are measuring the suite and not the model; sample a stratified subset of cases rather than the whole golden set; and run operators in parallel, because they are independent by construction.

Keep the score and the survivor list in the repository next to the suite. The number in isolation means little, but a score that fell between two quarters means somebody added prompt surface without adding coverage, which is the exact drift this is for. The related drill in injecting one known bad prompt is the manual, single-shot version of the same question and is worth running on the months you do not run this.