Writing a Regression Test From a Support Ticket

A support ticket is the highest-value source of regression cases you have, because somebody who was not thinking about your test suite found an input you would never have written. It is also a paraphrase, and converting a paraphrase into a case is where most of the work is.

What the ticket gives you and what it omits

A typical ticket says the assistant quoted the wrong refund amount, and attaches a screenshot of the reply. What it does not contain is everything you need: the exact message array that was sent, the model ID that served it, the sampling parameters, the prompt revision in force that day, the tool definitions attached to the call, the retrieved documents in context, and whether it happened once or every time.

Skip that recovery and you will write a case from the screenshot, run it, watch it pass, and conclude the problem is fixed or was never real. Both conclusions are wrong for the same reason: you did not reproduce the conditions, so the run told you about a different situation.

There is one hard prerequisite, and it is worth being blunt about. If you do not log the rendered prompt — the actual messages, after templating and retrieval — you cannot do this at all, and no technique in this page substitutes for it. Log the request payload with secrets redacted, the model ID, the parameters, the prompt hash and the response, keyed by a request ID the support tool can carry. What to log for an LLM call is the general treatment; the specific ask here is that a ticket ID must be resolvable to one request record.

Recovering the exact input

Pull the record, then confirm the failure still reproduces before doing anything else. Run the recovered payload against the same model ID and parameters, five to ten times rather than once, and count. Three outcomes, each meaning something different:

• Fails every time. Deterministic given this input. The easiest case, and the one where minimisation is safe.
• Fails sometimes. A stochastic mode. Record the rate you observed and carry it forward — it decides how many repetitions the eventual case needs, per the arithmetic in why a passing suite misses regressions.
• Never fails. Something else differed. The prompt has changed since; the model ID has changed; a retrieved document is no longer in the corpus; a feature flag altered the tool set. Find the difference before writing anything, because a case built on the wrong input is a case that will never catch its own bug.

Then redact, before the payload goes anywhere near the repository. A fixture file is a permanent, widely-readable copy of whatever you put in it, and a customer’s order history in a public repository is a worse incident than the one you are fixing. Replace real values with synthetic ones that preserve the properties that might matter: the same length, the same script, the same embedded newlines, the same unusual punctuation, the same null-versus-empty-string distinction. Then re-run the redacted version and confirm it still fails. If redaction fixed it, the thing you removed was the cause, which is itself the finding.

Minimising without deleting the cause

A four-thousand-token fixture is a bad permanent case: slow, expensive, and impossible to reason about when it fails in eight months. Cut it down — carefully.

The naive approach is bisection. Halve the context, run once, keep the half that still fails. For deterministic programs this is correct and it is what every test-case reducer does. For a model it is a trap, because “run once and it passed” does not mean the half you just discarded contained the cause. If the underlying rate is 30 per cent, a single passing run discards the right half 70 per cent of the time, and you converge confidently on a minimal input that reproduces nothing.

So minimise against the rate rather than against a single run. If the full input failed six times in ten, a candidate reduction has to fail at a comparable rate over the same number of draws before you accept it. That makes minimisation cost k calls per step instead of one, which is why you should reduce along structural boundaries — whole retrieved documents, whole conversation turns, whole fields — and stop early rather than chasing a truly minimal input. Getting from four thousand tokens to six hundred is worth the calls. Getting from six hundred to five hundred and eighty is not.

Stop when removing any remaining piece makes the failure rate collapse. Then record the rate you ended at in a comment next to the case, because the next person to see it go red needs to know whether one failure in twenty is news.

Choosing the assertion

This is the step that decides whether the case is worth keeping. The ticket complains about a sentence, so the instinct is to assert on the sentence — that the reply contains “41.99”, or that it equals the corrected text. Both are wrong, and the second is worse: it will go red the next time the model rephrases anything, and it will be deleted within a quarter.

Ask instead what property the output violated, stated so it applies to every input rather than to this one. “It said 41.90 instead of 41.99” becomes “the quoted refund equals the sum of the refundable line items”. “It cited an order the customer does not have” becomes “every order identifier in the output appears in the input”. “It answered in English when the customer wrote in Dutch” becomes “the detected reply language matches the detected input language”. Each of those is an assertion you can apply to your entire fixture set on the same afternoon, which is where the leverage is.

# tests/regression/fabrication_test.py
import re
import pytest

ID_RE = re.compile(r"\bORD-\d{6}\b")

@pytest.mark.fabrication
@pytest.mark.parametrize("case", load_cases("fabrication"), ids=lambda c: c.id)
def test_no_invented_order_ids(case, run_prompt):
    out = run_prompt(case.input)
    quoted = set(ID_RE.findall(out.text))
    supplied = set(ID_RE.findall(case.input_text))
    assert quoted <= supplied, (
        f"{case.id} (written against {case.written_against}): "
        f"invented {sorted(quoted - supplied)}"
    )

The failure message carries the case ID, the prompt revision it was written against and the offending values — enough to triage without opening the file.

The procedure end to end

1. Resolve the ticket to a request record: exact messages, model ID, sampling parameters, tool definitions, retrieved documents, prompt hash. If you cannot, stop and fix logging first; everything after this step is guesswork without it.
2. Re-run the recovered payload ten times against the same model ID and parameters. Record how many failed. If none did, find what changed before continuing.
3. Redact every real value, preserving length, script and structure, and re-run to confirm the redacted version still fails at a similar rate.
4. Reduce along structural boundaries, running k draws per candidate and comparing failure rates rather than single outcomes. Stop when further removal collapses the rate.
5. Write the assertion as a property of the output relative to the input, not as a comparison to a stored string. Put the case in the directory for its failure mode.
6. Tag it with the current prompt hash, the mode and origin: ticket:SUP-4182, so a future reader knows a real customer hit this rather than someone imagining it.
7. Run the new assertion across every existing fixture, not only the new one. If it fails on cases nobody complained about, you have found more than the ticket did — and that is the usual outcome, which is the argument for doing this at all.