Building a Review Queue for Failed Prompt Test Cases

Forty cases fail in a regression run against a new prompt. The CI log shows forty stack traces in the order they finished, with the model output truncated to eighty characters. Nobody can tell whether this is one bug or nine, whether any of it failed yesterday, or which failures are the test being wrong. That is a tooling problem and it is a small one.

Why raw CI output does not work

Three specific things go wrong, and each maps onto a feature the queue needs.

There is no identity across runs. Test names in an LLM suite are often generated from the case data, and the assertion messages contain the model output, which differs every run. So the same failing case looks like a different failure each time, and you cannot answer “is this new?” — which is the only question that matters when triaging forty of them.

There is no grouping. Thirty of the forty are one missing field in one schema. Presented as thirty entries they consume thirty times the attention of one entry with a count of thirty.

There is not enough context to judge. An LLM test failure cannot be assessed from the assertion message, because the question is usually “is this output actually wrong, or is the assertion too strict?” and answering it needs the full input, the full output, and the expected value side by side. A truncated diff in a terminal is not that.

Case identity, which must be stable

Derive the case id from the input and the test, never from the output or the message.

case_id = sha256( suite_name + "|" + case_name + "|" + canonical_json(input) )

Stable across: model version, temperature, sampling, prompt version,
               reordering of the suite, retries.
Changes when:  the case's input changes, which is a new case.

Canonicalise the JSON before hashing — sorted keys, no insignificant whitespace — or a serialiser upgrade silently reissues every id in the suite and the queue loses its entire history in one commit. This is the single most load-bearing decision in the tool, and it is four lines of code.

With a stable id, three states become computable and each drives a different response: new (failed this run, not the last), persistent (failed both), and flaky (failed in some runs of the same commit and passed in others). Flaky detection needs the suite run more than once per commit, which for an LLM suite is worth doing anyway — a case that passes four times in five is a case whose assertion is measuring sampling noise, and it should be quarantined rather than debugged.

Grouping by failure signature

The signature is what makes forty entries into nine. Build it from the parts of the failure that describe the kind of problem, having stripped everything that varies:

signature = sha256(
    assertion_name          // "expected schema to validate"
  + "|" + error_class       // "SchemaValidationError"
  + "|" + normalise(message)
)

normalise() replaces, in order:
  - JSON pointers to a stable form:  /items/3/name  ->  /items/N/name
  - any run of digits                ->  N
  - UUIDs and hex ids                ->  ID
  - quoted model output              ->  removed entirely
  - absolute paths                   ->  relative

Removing the quoted model output is the step people leave out, and without it nothing ever groups, because the output is different every time. The signature is deliberately lossy: two genuinely different bugs occasionally collide into one group, and that is a much cheaper error than forty ungrouped entries.

Three dispositions, and why the third matters

A reviewer looking at a group picks one of exactly three, and the tool should not offer a fourth.

• Real regression. The output is worse. The prompt change is at fault. Block the rollout.
• Bad test. The assertion was wrong, over-specific, or encoded an accident of the previous model’s phrasing. Fix the assertion.
• Acceptable variation. The output changed, it is not worse, and the assertion is legitimately unable to express that. The expected value is updated.

The third one is what makes the queue converge instead of growing forever, and it is also the one that is dangerous, because “acceptable variation” is how a suite is gradually rewritten to accept whatever the model currently does. Two guards keep it honest. Require a reason string, so that the audit trail records why it was acceptable. And count the disposition rate: if more than about a third of dispositions in a quarter are “acceptable variation”, the assertions are testing prose rather than invariants, and the fix is to rewrite them to assert on something that does not change — a schema, a tool name, a numeric field, a metamorphic relation.

Building it

1. Get structured output from the test runner. Vitest emits a Jest-compatible JSON report, documented under Vitest’s reporters guide, with a testResults array of files each containing an assertionResults array of cases.

   npx vitest run --reporter=json --outputFile=.artifacts/results.json

2. Have the test itself write the evidence. The runner report contains the failure message and not the model output, so the suite must record the artefact separately, keyed by case id. Do this in a fixture or an afterEach, not by parsing messages.

   import { afterEach, expect } from "vitest";
   import { writeFileSync, mkdirSync } from "node:fs";
   
   export const evidence = new Map();
   
   afterEach((ctx) => {
     const record = evidence.get(ctx.task.id);
     if (!record || ctx.task.result?.state !== "fail") return;
     mkdirSync(".artifacts/evidence", { recursive: true });
     writeFileSync(
       ".artifacts/evidence/" + record.caseId + ".json",
       JSON.stringify(record, null, 2),
     );
   });

3. Build the queue file. Read the runner report, join it to the evidence directory by case id, compute signatures, group, and diff against the previous run’s queue to assign new / persistent / flaky.

   import { readFileSync, writeFileSync, existsSync } from "node:fs";
   
   const report = JSON.parse(readFileSync(".artifacts/results.json", "utf8"));
   const previous = existsSync("queue.json")
     ? JSON.parse(readFileSync("queue.json", "utf8"))
     : { groups: [] };
   
   const seenBefore = new Set(previous.groups.flatMap((g) => g.caseIds));
   
   const failures = report.testResults.flatMap((file) =>
     file.assertionResults
       .filter((a) => a.status === "failed")
       .map((a) => ({
         caseId: caseIdFor(file.name, a.fullName),
         signature: signatureFor(a),
         title: a.fullName,
         message: (a.failureMessages ?? [])[0] ?? "",
       })),
   );
   
   const groups = new Map();
   for (const f of failures) {
     const g = groups.get(f.signature) ?? { signature: f.signature, caseIds: [], sample: f };
     g.caseIds.push(f.caseId);
     groups.set(f.signature, g);
   }
   
   writeFileSync(
     "queue.json",
     JSON.stringify(
       {
         generatedAt: new Date().toISOString(),
         groups: [...groups.values()]
           .map((g) => ({
             ...g,
             count: g.caseIds.length,
             state: g.caseIds.every((id) => seenBefore.has(id)) ? "persistent" : "new",
           }))
           .sort((a, b) => b.count - a.count),
       },
       null,
       2,
     ),
   );

4. Render it as one static page. Groups sorted by count, new ones first, each expandable to show the input, the baseline output, the new output and the assertion. This does not need a framework or a database; a generated HTML file published as a CI artefact is enough, and being an artefact means it is versioned with the run that produced it.
5. Take dispositions as a checked-in file. Write them to dispositions.json keyed by signature, with reviewer, date and reason. Keeping them in the repository rather than a database means a disposition arrives in the same pull request as the fix it justifies, which is where a reviewer can see both.
6. Fail the build on undisposed new groups. The queue is only useful if not clearing it blocks something. Persistent groups with a recorded disposition do not block; anything new does.

Making the queue shrink

Two habits decide whether this becomes infrastructure or abandonware.

Quarantine aggressively. A case that flips between pass and fail on the same commit is not giving you information, and leaving it in poisons every group count. Move it to a quarantined list with the date and revisit monthly; most quarantined LLM cases turn out to be assertions on prose, and the fix is to rewrite the assertion rather than to stabilise the model.

And store the evidence properly, because the evidence is the entire value of the tool and it is also the risky part. A failing case’s recorded request contains whatever was in the input, which for cases derived from production traffic means real user data landing in a repository or a CI artefact store. That is a separate job with its own failure modes, covered in scrubbing PII from recorded fixtures.