Testing the Tests: Injecting a Known Bad Prompt to Verify Your Suite Catches It

Prompt suites rot in a particular direction: assertions get loosened to stop a flaky failure, the loosening is never tightened, and two quarters later the suite is green against a prompt that would embarrass you in production. The cheapest way to find out is to break the prompt on purpose and see who notices.

Why a green suite decays

Nobody sets out to weaken a test. The sequence is always the same. A case fails intermittently because the model phrased something differently. Somebody under deadline changes the assertion from “the summary names the account owner” to “the summary is at least 40 characters”. The build goes green, the ticket closes, and nothing ever revisits it. Repeat across a year and the suite asserts that the model returned some text.

Coverage tooling cannot see this, because there is no line of your code left uncovered — the assertion ran. Only an actual regression can distinguish a strong suite from a weak one, and if you wait for a real regression you learn the answer from a customer. So you manufacture one.

Choosing the bad prompt

The quality of the drill is entirely in this step. A bad prompt that is obviously bad proves nothing. Pick from breakages your team has actually shipped or nearly shipped, which you can usually recover from incident notes and reverted commits:

• The schema instruction removed, so the model returns prose that happens to look structured.
• A placeholder left unfilled, so the literal template variable reaches the model — the failure mode in unfilled placeholders.
• One tool removed from the tool list, so the model narrates the action instead of calling it.
• The refusal or safety clause deleted from the system prompt.
• The few-shot block reordered so the last example is now an edge case rather than the canonical one.
• The model id changed to the previous generation, which is what happens when a config default is not overridden in one environment.

Write the chosen breakage down before you run anything, in a sealed note, along with which test you predict will catch it. The prediction is half the value: a drill where the suite catches a different thing than you expected has told you something about both.

The drill

1. One person injects, on a branch. Not on main, and not through a mechanism that could be merged. A branch named drill/2026-q3 with a single commit touching only the prompt file.
2. Nobody else is told what changed. The rest of the team knows a drill is running this week; they do not know which edit. This is the part that makes it a test of the suite rather than a test of the reviewer.
3. Open a pull request and let CI run exactly as it would for a real change. Same gates, same eval budget, same reviewers on rotation. If your eval job is normally skipped on prompt-only diffs, that fact is now a finding.
4. Record what happens and in what order. Which job failed, how long it took to fail, what the failure message said, and whether a human could tell from the message what was wrong.
5. Let the review happen. If CI is green, does the reviewer catch it by reading the diff? That is a second, separate line of defence and it is worth measuring separately.
6. Close the pull request without merging, and post the sealed note.
7. Turn every miss into a test in the same week. A drill whose findings sit in a document is a drill you will stop running.

Keep the injected prompt as a fixture afterwards. A committed fixtures/known-bad/2026-q3-missing-schema.txt plus a test that asserts the suite fails against it turns the one-off drill into a permanent guard, and next quarter’s drill can pick something new.

# tests/meta/known_bad_test.py
import pytest
from evals import run_suite, load_prompt

@pytest.mark.parametrize("fixture", [
    "known-bad/2026-q3-missing-schema.txt",
    "known-bad/2026-q1-tool-removed.txt",
])
def test_suite_rejects_known_bad_prompt(fixture):
    result = run_suite(prompt=load_prompt(fixture))
    assert result.failed > 0, (
        f"{fixture} is a prompt we know is broken and the suite passed it"
    )

Note the direction of that assertion. It is a test that fails when the suite succeeds, which reads backwards the first time and is the only shape that can express “this must be caught”.

Reading the result

Three outcomes, and they mean different things. The suite fails on the test you predicted: the drill passed and you have a calibrated understanding of your coverage. The suite fails on a different test: worth investigating, because it usually means one test is doing more work than its name suggests, and that test is now load-bearing without anyone knowing.

The suite fails but the message is useless — a diff of two long strings, or a score below a threshold with no indication which case — counts as a partial miss. A failure that nobody can act on gets rerun until it passes. That is the same failure mode as no failure at all, and it belongs in the same list of fixes as a genuine gap. Whoever gets paged should be able to read the first screen of output and name the broken constraint.

When nothing fails

Resist the instinct to add one test for the specific injected edit and call it done. The injected prompt was a sample from a family of breakages, and patching the sample leaves the family. Ask instead which class of change your suite is blind to, then add the assertion that covers the class: structure validation over every case rather than one, a tool-selection check on every case that has an expected tool, a check that the rendered prompt contains no residual template syntax at all.

The systematic complement to this drill is mutation testing the whole suite, which runs the same idea across a defined operator set and gives you a number. This drill is cheaper, is done by a human who knows your product, and catches things an operator list would not have thought of. Both, alternating, is a reasonable cadence.