Building a Test Report That Shows Which Prompt Version Broke What

Forty tests fail after a merge that touched six prompts. The report tells you which tests failed. It does not tell you which of the six prompts to revert, and that is the only question anybody is asking.

The join key nobody records

The universal test report format is JUnit XML, which every CI product reads. Its schema has testsuite and testcase elements with classname, name, time and a failure child carrying a message. There is no field for “which prompt version produced this”. So it has to be carried in one of the fields that does exist, and the choice of what to put there is the whole design.

Use the content hash of the prompt file, not a hand-maintained version number. A number in a file header goes stale the moment someone edits the prompt without bumping it, and that is exactly the commit you are trying to identify. Git already computes a content hash for every file:

# The blob hash changes if and only if the bytes change.
git hash-object prompts/billing/refund.md
# -> 9f2c1a7b6d0e4f83a1c5b2e7d4f6a8c0b3e1d7f2

# And the commit that last touched it, for the triage link.
git log -1 --format=%h,%an,%ad -- prompts/billing/refund.md

Compute this once at the start of the run for every prompt file, keep it in a dictionary, and have the loader that reads a prompt record which one it used. A test that never loads a prompt records nothing, which is correct — it cannot have been broken by a prompt change.

Emitting it from the test run

In pytest, the record_property fixture writes a property element into that test’s testcase node in the XML produced by --junitxml. That is the supported route and it survives into most CI parsers:

import pytest
from prompts import load_prompt, blob_hash

@pytest.fixture
def prompt(record_property):
    def _load(name: str) -> str:
        text = load_prompt(name)
        record_property("prompt_file", name)
        record_property("prompt_hash", blob_hash(name))
        return text
    return _load


def test_refund_returns_valid_schema(prompt, client):
    system = prompt("billing/refund.md")
    out = client.complete(system=system, user="refund order 4417")
    assert_valid(REFUND_SCHEMA, out)

Run it with pytest --junitxml=report.xml. In a JavaScript suite the equivalent is Vitest’s junit reporter, configured with reporters: ["junit"] and an outputFile entry; it also accepts a classname option and reads VITEST_JUNIT_CLASSNAME from the environment, which gives a second place to put a grouping key if properties are inconvenient.

If your CI product drops property elements — some parsers do — fall back to a sidecar. Write one JSON line per test containing the test id, the prompt file, the hash and the outcome, upload it as a build artifact, and join it to the XML afterwards on the test id. The sidecar is more work but it is not at the mercy of a parser.

One design choice is worth making deliberately: record both the prompt file and the hash, not just the hash. The hash alone is unreadable and cannot be grouped by anything a human recognises; the path alone cannot tell you whether the file moved. Together they give you the two axes the report needs — group by path, colour by whether the hash changed — and they cost one extra property per test.

Newly failing, not failing

A prompt suite of any size has a standing set of known failures. A report listing all of them after every merge is unreadable, so the report must be a diff, and the thing it diffs against has to be chosen carefully. The right baseline is the last run on the branch this change merges into, at the merge base — not the previous run on the feature branch, which contains your own earlier attempts, and not the last nightly, which may be against a different model.

Store the baseline as an artifact keyed by commit. Then each test lands in one of four buckets, and only the first two need a human:

• Newly failing — passed at the merge base, fails now. This is the report.
• Newly passing — worth surfacing, because it is sometimes a case that was silently disabled rather than fixed.
• Still failing — collapse to a count with a link.
• Still passing — a count.

The report a human can triage

Invert the usual grouping. Group by prompt file first and test second, because the unit of action is a revert of a prompt, not a revert of a test. A useful report reads:

Newly failing: 12 cases across 2 prompts

prompts/billing/refund.md  9f2c1a7 -> 4b81e0d  (PR #482, a.kaur)
  9 newly failing
    billing.schema-invalid[two-currencies]      missing key "currency"
    billing.schema-invalid[zero-amount]         missing key "currency"
    billing.wrong-tool[partial-refund]          called issue_credit, expected issue_refund
    ... 6 more, all missing key "currency"

prompts/support/triage.md   c3d9f10 -> c3d9f10  (unchanged)
  3 newly failing                                <- not a prompt change
    support.empty-output[arabic]
    support.empty-output[thai]
    support.empty-output[hebrew]

The second group is the payoff. Three tests failed against a prompt whose hash did not change, which means the cause is somewhere else — a model default, an SDK bump, a dependency, an infrastructure change. Splitting those out of the list stops nine people investigating a prompt that is innocent. If you see that pattern, go to testing whether an SDK upgrade changed output or silent model updates.

Collapsing identical failure messages, as in the “6 more, all missing key” line, matters more than it looks. Nine failures with one cause is a five-minute fix; nine failures with nine causes is an afternoon, and the report should say which it is before anyone opens a file.

Where the run is non-deterministic, add one more column: the pass rate across attempts. A case that passed four times out of five at the merge base and one time out of five now is a regression the binary pass-or-fail view records as a single failure, indistinguishable from a case that was always broken. Recording attempts also lets the report separate “this got worse” from “this was always marginal and today it landed on the other side”, which is the distinction that decides whether anyone should be woken up.

Three ways this goes wrong

1. The hash is recorded at the wrong moment. If you hash the file on disk at the end of the run rather than the string the test actually loaded, a test that renders a template with variables, or picks a prompt variant at runtime, records a hash it never used. Hash what the loader returned.
2. Retries hide the signal. A suite that retries failing tests twice reports the third outcome. For a non-deterministic system that turns a two-in-three failure into a pass. Record every attempt and report the pass rate, not the last result; see mutation testing a prompt suite for the related question of whether the suite can detect anything at all.
3. The report quotes the prompt. Failure messages that embed the rendered system prompt leak customer data into CI logs that have a much wider audience than your database. Print the hash and the path; keep the content out (keeping prompt content out of CI logs).