Debugging a Bad Response Six Hours Later

A support ticket says the assistant gave a customer bad pricing information yesterday afternoon. You have a name, an approximate time, and a screenshot. Whether that is a ten-minute investigation or an unanswerable question was decided months ago, by which columns you wrote.

Replayable is a property, not a feeling

Here is a definition worth holding yourself to: a request is replayable if you can reconstruct the exact bytes that were sent to the provider, without consulting anything that has since changed.

That last clause is the whole difficulty. Most systems can reconstruct most of a request — but they do it by re-running the code, which has been deployed twice, against the prompt template, which has been edited, using retrieval, which now returns different documents. What you replay is a request that resembles the original. If the bug was caused by any of the three things that changed, resemblance is exactly no help.

The property is testable, which is the nice part. Take a request from a week ago, reconstruct it from the log, hash it, and compare to the stored input_hash. If those match, your logging is complete. If they do not, the diff tells you which field is missing. Run that as a scheduled job over a handful of random rows and you have a continuous check on your own observability, which is otherwise the one system nobody tests until they need it.

There is a weaker form worth naming, because it is what most teams actually have and it is not useless. Diagnostic replay reconstructs enough of the request to understand the failure without reproducing it exactly — the prompt version, the model, the retrieved document ids, the parameters — and is usually sufficient to answer “what went wrong”. Exact replay is what you need to answer “will the fix work”, because that requires running the same input through a changed system. Aim for exact; accept diagnostic for the paths where storing the resolved body is genuinely impractical, and know which paths those are before an incident rather than during one.

The fields replay actually needs

Beyond the identity and timing columns in the request log schema, replay needs these, and they are all things that are cheap at write time and impossible to recover later:

• The resolved request body, or a hash plus a pointer to it. Resolved means after template substitution, after retrieval, after truncation — the bytes, not the recipe.
• The tool definitions as sent. A JSON-schema change to one tool alters model behaviour as much as a prompt edit and is far less likely to be noticed.
• Retrieved document ids and versions. Not the text (that is in the body) but the identifiers, so you can answer “did we retrieve the wrong thing, or the right thing at the wrong version?” These are different bugs with different fixes.
• prompt_id and prompt_version resolved at call time, from the registry rather than inferred from the deploy.
• Every sampling parameter, including the ones you left default. “Default” is a value that the SDK chooses and SDK defaults change across major versions.
• The SDK and gateway versions. A one-line resource attribute that has resolved more “it worked last week” arguments than any other field on this list.
• A user-facing correlation id. The single highest- leverage item here: surface the trace_id in the UI on any error or feedback control, so a ticket arrives with a key instead of a time range.

The six-hours-later walk

With the schema above, the investigation is four queries. Start from whatever the ticket gives you — ideally a correlation id, realistically a tenant and a time window.

-- 1. Find the candidates.
select request_id, trace_id, started_at, served_model, prompt_version,
       finish_reason, attempt, duration_ms, cost_usd, output_ref
from llm_request
where tenant_id = $1
  and feature = 'pricing_assistant'
  and started_at between $2 and $3
order by started_at;

-- 2. Was this request unusual, or is the whole feature unusual?
select prompt_version, served_model,
       count(*)                                             as n,
       avg(output_tokens)::int                              as avg_out,
       avg((finish_reason = 'length')::int)::numeric(5,4)   as truncated_rate,
       avg((error_type is not null)::int)::numeric(5,4)     as error_rate
from llm_request
where feature = 'pricing_assistant'
  and started_at > now() - interval '7 days'
group by 1, 2
order by n desc;

-- 3. Did the served model change under us during the window?
select date_trunc('hour', started_at) as hour, requested_model, served_model,
       count(*)
from llm_request
where started_at > now() - interval '3 days'
group by 1, 2, 3
having count(*) > 0
order by 1;

-- 4. Pull the exact bytes and replay them.
select input_ref, input_hash, temperature, seed, provider_request_id
from llm_request where request_id = $1;

Query 2 is the one people skip and the one that most often ends the investigation. A single bad answer from a model that produces 2% bad answers is not a bug; it is the base rate, and the fix is an eval and a guardrail, not a code change. A single bad answer from a prompt_version that appeared on Tuesday and has a truncation rate five times the previous version is a regression with a name.

Query 3 is the second cheapest thing on this page. It costs one scan and it eliminates an entire hypothesis — either the model that served your traffic changed during the window or it did not, and knowing which takes seconds rather than the twenty minutes usually spent arguing about it. Keep it as a saved view; it is the same query the silent-update detector runs on a schedule.

The habit worth building from all four is to establish the base rate before reproducing anything. Reproduction is satisfying and slow, and roughly half the time the aggregate view has already answered the question: this is normal for this feature, or this started at 11:04 when a named thing changed. Reproduce when you need to test a fix, not to find out whether there is a problem.

When the answer is “ask the provider”

Sometimes the request is fine, the parameters are fine, and the response is nonsense or the latency was 40 seconds. At that point you are escalating, and the only currency a provider’s support team accepts is their own request id — the opaque string returned on the response, typically in a header such as x-request-id or a vendor-prefixed equivalent, and also present as gen_ai.response.id if you follow the semantic conventions.

Log it unconditionally, including on failures, where it is most needed and most often dropped because the error path did not go through the same code as the success path. A useful test: force a 500 in staging and check that the row still has a provider_request_id.

Four things that break replay