Tracking Which Prompt Version Produced a Given Production Response

Somebody forwards a screenshot of a bad answer from eleven days ago. To act on it you need the exact prompt text that produced it — not the current text, not the version number in the config, the actual rendered string. If you did not record it at the time, that information no longer exists.

What a support ticket actually gives you

A ticket arrives with, at best, a timestamp, a user identifier, and the model’s output. Everything else has to be recovered from your own records, and the chain has several links that break independently: which prompt template was in force, which version of it, what was substituted into it, which model and parameters served it, and what came back.

The tempting shortcut is to log a version number. It is not enough, for reasons that all show up in the same week:

• The number identifies the template, not the rendered prompt. Retrieved context, user text and locale all vary per request and all change behaviour.
• Version numbers are maintained by hand and therefore sometimes are not. A prompt edited without bumping the number gives two different behaviours the same identity, which is the exact failure the number was supposed to prevent.
• A number tells you nothing during an incident unless you can also get the text it referred to at that moment. If the file has been edited since and the version was not bumped, the record points at text that never ran.

A content hash has none of these problems, because it is derived from the bytes rather than asserted about them.

What to record

Seven fields on every request, and none of them is expensive:

• request_id — yours, surfaced to the user somewhere they can copy. This is what turns a screenshot into a lookup.
• prompt_id — the logical prompt, e.g. support.triage.
• prompt_template_hash — hash of the template file’s body before substitution. Constant per deploy, and the field you group by when comparing versions.
• prompt_rendered_hash — hash of the exact string sent to the provider, after substitution and after any truncation. This is the one that answers “what did the model actually see”.
• model and the sampling parameters that were in force. A prompt is only half of the behaviour.
• provider_request_id — whatever identifier the provider returns, so a vendor support conversation can start from their side.
• finish_reason and token counts. A complaint about a truncated answer is usually answered by finish_reason alone.

Recording the rendered text as well is a separate decision with a retention and privacy cost, since it contains user content. A reasonable middle path is to store the hash on every request and the full text only for a sampled fraction plus every request already flagged as a failure. That is the same trade-off human review sampling makes from the quality side, and the sampling rate can be the same number.

Computing the hash, and what to hash

The rule that makes hashes useful is that identical behaviour must produce identical hashes, so normalise deliberately and then never change the normalisation without renaming the field.

import hashlib, json

def prompt_hash(*, template_body: str, messages: list[dict], params: dict) -> dict:
    """Two hashes: the template as deployed, and what was actually sent."""
    template = hashlib.sha256(
        template_body.strip().encode("utf-8")
    ).hexdigest()[:12]

    # Hash the exact message array, in order, plus the parameters that
    # change behaviour. sort_keys makes this stable across dict ordering.
    payload = json.dumps(
        {
            "messages": messages,
            "model": params["model"],
            "temperature": params.get("temperature"),
            "top_p": params.get("top_p"),
            "seed": params.get("seed"),
            "tools": [t["function"]["name"] for t in params.get("tools", [])],
        },
        sort_keys=True,
        separators=(",", ":"),
        ensure_ascii=False,
    )
    rendered = hashlib.sha256(payload.encode("utf-8")).hexdigest()[:12]
    return {"prompt_template_hash": template, "prompt_rendered_hash": rendered}

Three choices in there are deliberate. Truncating to twelve hex characters keeps the field readable in a log line and a support conversation while leaving a collision space far larger than the number of distinct prompts any team will produce. Including the sampling parameters means a temperature change shows up as a new rendered hash, which is correct — the behaviour changed. And hashing tool names rather than full tool schemas is a judgement call: it keeps the hash stable across cosmetic description edits, at the price of not distinguishing them. If your tool descriptions are load-bearing, hash them in full and accept the churn.

The log line and the response field

Two surfaces. The log line is for you; the response field is for the user, and it is the one people forget.

{"ts":"2026-08-11T09:14:02Z","level":"info","event":"llm.completion",
 "request_id":"req_01J9Z6","user_id":"u_88214",
 "prompt_id":"support.triage","prompt_template_hash":"9f2c81ad0b41",
 "prompt_rendered_hash":"3d70b6c11e0a","model":"your-model",
 "temperature":0,"provider":"primary","provider_request_id":"chatcmpl-abc123",
 "finish_reason":"stop","prompt_tokens":812,"completion_tokens":96,
 "latency_ms":1840,"flag_variant":"triage_v7"}

Return request_id to the client on every response — a response header is enough — and show it in the interface somewhere copyable, next to whatever feedback control you have. Every screenshot then arrives with the key to its own record, and the twenty-minute archaeology of matching a timestamp against a user id against a timezone disappears.

Keep the field names identical across services. A hash logged as prompt_hash in one service and promptVersion in another cannot be joined without somebody remembering the mapping, and during an incident nobody does.

Joining a complaint back to a prompt

With the two hashes recorded, the questions that were previously unanswerable become straightforward queries.

1. What text produced this answer? Look up the request by id, take prompt_template_hash, and find the commit whose prompt file hashes to it. Emit the hash of every prompt file into your build metadata so this is a lookup rather than a bisect.
2. Is this complaint about the new prompt? Group complaint volume by prompt_template_hash and compare the windows before and after the change. Grouping by deploy time instead gets this wrong whenever a rollout was gradual, which is exactly when you most need it right.
3. Did the rollback take effect? After reverting, watch for the old prompt_template_hash reappearing in live requests. This is the only direct evidence that the change reached every instance, and it is much better than watching a deploy dashboard.
4. Which requests are still on the bad version? Filter on the hash. If in-flight or cached requests are still using it, that is the awkward middle state covered in what to do about in-flight requests.