Rolling Back a Bad Prompt Deploy in Under a Minute

The reason a bad prompt stays live for twenty minutes is almost never that nobody noticed. It is that the only way to change the prompt is to build an artefact and roll it out, and that pipeline was designed for correctness rather than for speed. Fixing the rollback time means taking the prompt off that path entirely.

Why reverting the commit is not the fast path

Time the steps honestly. Revert the commit; wait for CI, which on most teams is several minutes and cannot be safely skipped because skipping it is how you ship a second incident; build the image; push it; roll it out gradually, because a fast rollout is its own risk; wait for the old instances to drain. Ten to thirty minutes is ordinary, and every minute of it is serving the bad prompt.

Worse, that path degrades exactly when you need it. Under an incident, the CI queue is busy with the fix somebody else is pushing, and the person doing the revert is the person who least wants to be reading pipeline logs. A rollback mechanism that depends on your slowest system working perfectly is not a rollback mechanism.

The structural fix is a single invariant: the prompt text must not be an immutable part of the deployed artefact. Everything else in this page follows from that.

Read the version at request time

Instead of baking the prompt in, the running process holds a pointer: an identifier saying which version is active. That pointer is read from somewhere that can be changed in seconds, and the version it points at is fetched and cached.

“At request time” does not mean a network call per request. That would add latency and make your prompt store a hard dependency of every completion. The workable shape is a short cache with a bounded age:

# prompts/active.py
import time, threading

TTL_SECONDS = 15           # worst-case propagation delay for a rollback
_lock = threading.Lock()
_cache: dict[str, tuple[float, dict]] = {}

def active_prompt(prompt_id: str) -> dict:
    """Returns {'version': int, 'body': str}. Never raises on store failure."""
    now = time.monotonic()
    with _lock:
        entry = _cache.get(prompt_id)
    if entry and now - entry[0] < TTL_SECONDS:
        return entry[1]
    try:
        version = store.get_pointer(prompt_id)          # small, fast read
        body = store.get_version(prompt_id, version)    # immutable, cacheable
        value = {"version": version, "body": body}
        with _lock:
            _cache[prompt_id] = (now, value)
        return value
    except Exception:
        if entry:
            return entry[1]      # serve stale rather than fail the request
        raise

Two properties are doing the work. The TTL is your worst-case rollback propagation time, so choosing it is choosing your recovery time — 15 seconds means every instance is serving the reverted prompt within 15 seconds of the pointer write, without anyone deploying anything. And the fallback to a stale entry means a prompt-store outage degrades to “prompts are slightly out of date” rather than to an outage of your own.

Serve stale, but bound it and alarm on it. An instance that has been serving a stale prompt for an hour because the store is unreachable is an instance your rollback will not reach, which is the worst possible state to discover during an incident.

The store: immutable versions, one mutable pointer

The data model is the part that makes this safe, and it is small:

• Versions are immutable. Writing version 7 writes it once. Nothing ever edits a published version, so a version identifier permanently means one exact text and the hash you recorded on a response last month still resolves.
• The pointer is the only mutable thing. One row, one key, one integer or content hash. Rolling back is writing an older value into it.
• The pointer’s history is retained. Every write records who, when and why. That history is the timeline you will be reconstructing afterwards, and it is free at write time and impossible to recover later.

Almost any durable store will do — a database table, a key in a configuration service, an object in blob storage. What matters is that the read is fast and highly available and the write is auditable. What does not work is a store whose only writer is your deploy pipeline, because then you have moved the prompt but kept the deploy on the rollback path.

Version numbers should be published from your repository, not typed into a console. The prompt file remains the source of truth and CI publishes each change as a new immutable version on merge; the pointer is what humans move. That keeps review, blame and readable diffs intact while removing the deploy from the emergency path.

Performing the switch

1. Identify the version to return to. From the pointer history, not from memory. The previous value is recorded; use it rather than reasoning about which change was which.
2. Write the pointer. One command, with a reason attached: promptctl set support.triage 6 --reason "INC-412 quality drop". Whatever your tooling, it should be one command, because a multi-step console procedure at 2 a.m. is where the wrong environment gets edited.
3. Watch the version field in live logs. You recorded prompt_template_hash on every response for exactly this — see tracing a prompt version to a response. Within the TTL, live traffic should show only the old hash. If some instances still show the new one after twice the TTL, you have found a cache bug and it is the most important thing in the incident.
4. Then, unhurriedly, revert the pointer’s source. Open the revert PR so the repository and the live pointer agree again. This is a normal change at a normal pace; the incident ended at step 2.

Proving the rollback path works before you need it

A rollback mechanism that has never been exercised is a hypothesis. Two tests and one drill.

The unit test asserts the cache honours the TTL and the stale fallback: patch the clock, change the pointer, assert the next request after the TTL sees the new version and the one before it does not; then make the store raise and assert the previous value is still served rather than an exception propagating. Both of these are cheap and both catch real regressions, because caches are where somebody eventually adds an optimisation that breaks propagation.

The integration test asserts the whole loop against a real store: write pointer, issue request, assert the response metadata names the expected version. Run it in staging on every deploy. It is the only check that the mechanism still works after somebody refactors the request path.

The drill is the one that finds the real problems. Once a quarter, in production, move the pointer to the previous version and back, and time it end to end. What that exercise usually turns up is not a bug in the code: it is that the person on call does not have write access to the store, or that the command lives in a runbook nobody can find, or that the confirmation prompt requires a ticket reference that does not exist yet. Those are the failures that turn a 30-second rollback into a 20-minute one, and no amount of unit testing surfaces them.