Naming Conventions for a Prompt Test Suite That Scales Past 500 Cases

At fifty cases any naming works. At five hundred the name is the only interface most people have to the suite — in a failure report, in a selector, in an alert — and a name that answers no question makes the suite unusable long before it makes it untidy.

Three segments, three questions

The scheme is feature.failure-mode.case, lowercase, dot-separated, hyphens inside a segment. Each segment exists because somebody asks that question about a failure, and no segment is there for symmetry.

• feature — who owns this? Matches the product area and, ideally, the directory the prompt lives in, so it can drive the routing described in alerting the right person. Examples: billing, support, onboarding.
• failure-mode — is this the same bug as those other eleven? The most valuable segment and the one usually missing. It groups failures by symptom across features, which is what turns forty red lines into three causes.
• case — which input? Named after the input, never numbered. two-currencies tells a reader what to look at; case-7 tells them to open the file, and forces a renumbering the day case 3 is deleted.

Two segments is the common compromise and it fails specifically at triage time: billing.two-currencies cannot be grouped with support.missing-locale even when both are the same schema bug. Four segments is worse, because the third and fourth are never applied consistently and half the suite ends up with an empty one.

The other property worth insisting on is that the name describes the case rather than the expectation. billing.schema-invalid.two-currencies names an input and a symptom; billing.returns-eur names an answer, and answers change. When the expected currency for that input is revised, the first name still fits and the second has to be renamed or, more likely, quietly left lying about what it checks.

The middle segment is a closed set

This is the rule that makes the scheme work rather than just look tidy. If the failure-mode segment is free text you get bad-json, invalid-json, json-broken and schema-fail in the same suite, and grouping by it produces four groups of one. Fix the vocabulary in a file, keep it short, and reject new values at review:

# failure-modes.txt  — the complete list; adding one is a review decision
schema-invalid      output did not validate against the declared schema
empty-output        no content, or content that is only whitespace
truncated           stopped early: finish reason length, or an unclosed structure
wrong-tool          called a tool other than the expected one
missing-tool-call   answered in prose where a tool call was required
hallucinated-field  invented a key or an entity not present in the input
refusal             declined a legitimate request
wrong-language      replied in a language other than the input's
over-budget         exceeded the token, cost or step budget for the task
unstable            same input, materially different output across samples

Ten to fifteen values is the right size. Fewer and the segment stops discriminating; more and nobody remembers them and free text returns through the back door. A case that genuinely does not fit is a signal worth discussing — that is usually a real new failure mode, and adding it deliberately keeps the list meaningful.

What the name buys you

Every one of these is a thing somebody wants to do on a bad afternoon, and each is a one-liner only if the name carries the structure:

1. Run everything for one team before merging their prompt change: pytest -k "billing", or vitest -t "billing.".
2. Check whether a fix generalised, by running one failure mode across all features: pytest -k "schema-invalid". This is the query that two-segment naming makes impossible.
3. Group a report by failure mode by splitting the id on the first two dots — no metadata, no parser, no extra file to keep in sync (building a report that shows what broke).
4. Route an alert by taking the first segment and looking it up in a table of ten rows.
5. Search a Slack message back to a file, because the id in the alert is literally greppable in the repository.

Letting the file tree carry it

Do not repeat the segments inside a long function name. Let the directory and file provide the first two and the parametrisation id provide the third, and the runner assembles the full id for you:

tests/prompts/billing/test_schema_invalid.py

@pytest.mark.parametrize("case", CASES, ids=lambda c: c.id)
def test_case(case, guardrail): ...

# pytest prints:
# tests/prompts/billing/test_schema_invalid.py::test_case[two-currencies]

That printed id already contains feature, failure mode and case, is copy-pasteable back into the runner to reproduce exactly one case, and needs no convention beyond where the file sits. It also means adding a case is adding one row to a list rather than writing a function, which is what keeps a suite at five hundred cases readable.

The same layout gives you a cheap coverage view for free. A directory listing shows which features have which failure modes covered, and the holes are visible without tooling: support has no wrong-tool file, onboarding has no truncated one. That table is the artefact people actually use when deciding what to write next, and it exists only because the first two segments were promoted out of the function name and into the tree.

Names that age badly

• The ticket number. test_jira_4471 is meaningless the week the ticket closes, and the ticket system will be replaced before the test is.
• The model name. test_gpt4_summary becomes a lie the first time you switch model, and the test almost never actually depends on the model — see testing prompt portability across models.
• The date. Version control has the date and it is correct.
• “works” and “should”. test_summarize_works does not say what would make it not work, so a failure tells you nothing and nobody can judge whether a new case duplicates it.
• The author. People move teams; prompts do not.

Adopting it without a rename

A suite that already has five hundred badly named cases does not need a five-hundred-file rename, and proposing one is how the convention gets rejected. Enforce it on new cases only, with a lint that checks ids against a pattern and a vocabulary file:

^[a-z0-9-]+\.[a-z0-9-]+\.[a-z0-9-]+$

Run it over the ids the runner collects, allow-list the existing names in a file that only ever shrinks, and rename opportunistically when somebody is already editing a case. The allow-list shrinking is the progress metric, and it is one number, which is the only kind of migration metric anybody actually watches.