Alerting the Right Person When a Specific Prompt Test Fails

A prompt test suite is shared infrastructure covering many teams’ prompts. Routing its failures by who owns the test file sends every alert to the platform team, who cannot fix any of them.

Routing on the wrong thing

The default in most CI setups is to notify the person who pushed the commit. That is right for a unit test and wrong here, for a specific reason: prompt tests fail for causes that have nothing to do with the commit. A model default changed, a provider started refusing a category of input, a shared retrieval index was rebuilt. The pusher is the correct recipient in maybe half of cases, and in the other half they spend an hour discovering it was not them.

Route on the artefact instead. A failing test loaded some prompt file; that file has an owner; the owner is who to tell. This survives the cases the pusher-based rule gets wrong, because it does not care why the test failed — the person who owns the billing refund prompt is the right person to look at a billing refund failure regardless of cause. It also survives a shared suite: one test file parameterised over two hundred cases can route to fifteen different teams.

This requires the join described in building a report that shows which prompt version broke what. If the run does not record which prompt each case loaded, there is nothing to route on and everything below is unavailable.

Reusing the ownership file you have

Most repositories already carry an ownership map in CODEOWNERS, maintained because it gates review. Reusing it means one map rather than two, and the second map is always the stale one.

# CODEOWNERS
prompts/                    @acme/ai-platform
prompts/billing/            @acme/payments
prompts/billing/refund.md   @acme/refunds-squad
prompts/support/            @acme/support-eng

One rule matters when you implement the lookup: in GitHub’s CODEOWNERS the last matching pattern wins, which is the opposite of the first-match precedence used by .gitignore and by most routing tables people write from memory. A lookup that returns the first match will send every billing failure to the platform team, because prompts/ is listed first. Walk the file in order and keep the last hit.

Fall back up the tree deliberately: if prompts/billing/refund.md has no entry, the nearest ancestor directory that does is the owner, and if nothing matches, the suite owner gets it along with a message saying the file is unowned. An unowned prompt is a finding in itself and should be visible rather than silently dropped.

When there is no path to route on

Some failures do not implicate a prompt file: a harness error, a fixture that will not load, a provider outage, a cassette that no longer matches. Give the test itself an owner tag for those. In pytest, a custom marker registered in the project configuration under markers is the supported mechanism, and it reads cleanly at the call site:

# pyproject.toml
# [tool.pytest.ini_options]
# markers = ["owner(team): the team paged when this test fails"]

@pytest.mark.owner("ai-platform")
def test_cassette_index_is_loadable(): ...

Read the marker in a reporting hook and attach it to the result. In a JavaScript suite there is no single stable equivalent, so the cheaper route is a naming convention: if every test id begins with a feature segment, as described in naming conventions for a prompt test suite, the routing table is a map from that segment to a team and needs no framework support at all.

What the message has to contain

Assume the recipient is holding a coffee and has ten seconds. Four things decide whether they act now or close the tab:

• Which prompt, and what changed about it. The path and the two content hashes, with the commit and author if it moved. “Unchanged” is important information, not an omission.
• The assertion that failed, in words. Not a stack trace. “9 cases expected key currency in the output object; it was absent” is triage. A traceback is homework.
• The exact command to reproduce. Including the model id and any seed or cassette name the run used, so the reader is not reproducing a different thing.
• How long it has been failing. First failure on this run reads differently from failing for six runs, and the second deserves an escalation the first does not.

What it must not contain is the rendered prompt or the model output. Both routinely contain customer text, and a chat channel is a far wider audience than the datastore that text came from.

Not sending the other nineteen

The failure mode of any alerting scheme is volume, and prompt suites generate volume beautifully: one bad edit fails forty parameterised cases at once. Three rules keep it survivable.

1. One message per cause, not per case. Group by (prompt file, failure message) and send one message listing the count and up to three example case ids. Forty messages get muted; one message with a count of forty gets read.
2. Only newly failing. A case that was already failing at the merge base is not news. It belongs in a weekly digest to the owning team, not in a real-time alert.
3. Quarantine before you mute. A test that fails intermittently will eventually be ignored, and an ignored alert is worse than none because it trains people to ignore the channel. Move it to a quarantined set that still runs and still reports, but does not page, and give the quarantine an expiry — a test quarantined for more than a fortnight is either fixed or deleted.

Escalation is the mirror image. If the same (prompt, cause) pair fires on three consecutive runs of the default branch, the group is not being picked up and the message should widen — to the team channel, then to whoever owns the release. Tie the threshold to consecutive runs rather than elapsed time, so a quiet weekend does not escalate a failure nobody could have seen.