Versioning Prompts Like Code

The test of a prompt workflow is a single question, asked three months later about a bad answer in a support ticket: which exact prompt produced this, and what did it look like that day? Most teams cannot answer it.

The question you cannot answer

Prompts drift into places git does not see: a database row edited through an admin panel, a provider’s playground, an environment variable set during an incident and never reverted. Each is reasonable on its own and together they mean the deployed prompt has no history.

Treating prompts as source is not about ceremony. It is about three properties you already get for code and silently gave up for the part of the system that decides what your product says: a diff, an owner, and a test that runs before merge.

The usual objection is that non-engineers need to change prompts without waiting for a deploy, and it is a fair one. The resolution is not to abandon git but to make the store a projection of it: edits go through a surface that writes a commit, or the runtime reads from a store that is exported back into the repository on every change. What you must not give up is the pair of properties that make an incident tractable — every version is diffable, and every version is reachable from a log line.

Repository layout

prompts/
  triage/
    v3.ts              # the current prompt, as data + a render function
    CHANGELOG.md       # one line per version: what changed and why
evals/
  triage/
    cases.jsonl        # {"input": ..., "expect": ..., "tags": ["refund"]}
    metric.ts          # exact-match on category; f1 across the label set
    baseline.json      # scores of the version currently in production

Version in the filename, not only in git history, because two versions have to coexist during a rollout. Numbers rather than semver: the only distinction that matters in practice is whether the output contract changed, since that breaks parsers downstream, and that deserves a note in the changelog rather than a numbering scheme nobody applies consistently.

The eval cases live next to the prompt they test and are appended to from production failures. That is the flywheel: every incident adds a case, and the case makes the regression impossible to reintroduce quietly.

Make every generation attributable

Log a fingerprint of the rendered prompt with every call, alongside the model id and the sampling parameters:

prompt_id      "triage"
prompt_version 3
prompt_sha     sha256(rendered_system + rendered_user)[:12]   -> "9c1af4d0e2b7"
model          the exact model string the request was routed to
params         {temperature, top_p, max_tokens, seed}

The hash matters more than the version number, because it catches what the number misses: a template edit that shipped without a version bump, a variable that rendered differently, a config that injected an extra line in staging only. Two requests with the same version and different hashes is a bug you would otherwise never see.

Log the sampling parameters for the same reason. “The prompt got worse” is very often a temperature default that changed in a client-library upgrade.

Hash the rendered prompt, not the template. A template hash tells you which code produced the call; a rendered hash tells you what the model actually read, which is the thing you are trying to explain. If the rendered prompt contains personal data, keep the hash as the identifier and store the reconstruction inputs separately under your normal retention rules — the hash exists to make two requests comparable, and it does not need to be reversible to do that.

The eval gate

The rule is that a prompt change cannot merge without an eval run, and it has to be cheap enough that nobody wants to skip it. Run only the suites whose prompts changed:

# .github/workflows/prompt-eval.yml
on:
  pull_request:
    paths: ["prompts/**", "evals/**"]

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      # only the suites whose prompt files changed in this PR
      - run: npm run eval -- --changed-since origin/main --temperature 0
      # fails the job if any suite drops below evals/*/baseline.json
      - run: npm run eval:compare -- --max-regression 0.02
      - uses: actions/upload-artifact@v4
        with: { name: eval-report, path: .eval/report.html }

• Temperature zero in CI. You are testing a prompt change, so remove the other source of variance. Keep one nightly run at production settings to catch what determinism hides.
• Allow a small regression band. A hard “never worse” rule on a 200-case suite blocks merges on noise. Two points of tolerance plus a per-tag check — no tag may collapse — is a more honest gate than a single average.
• Post the diff of failures, not the score. The useful artefact is the list of cases that flipped in each direction. A change that fixes twelve and breaks nine is a conversation, not a number.
• Cap the spend. An eval suite in CI is an API bill with a merge button. Cache by prompt hash plus case id so an unchanged pair is never re-run.

Two honest caveats about the gate. Temperature zero is not a guarantee of identical output — batching and floating-point non-determinism mean identical requests can still differ — so a suite that flips on one case between runs is normal and should not be treated as a regression. And a gate defends only what it measures: it will not notice a tone change, a new refusal pattern or a fifty percent increase in output length unless somebody wrote a check for those, which is the argument for keeping guardrail metrics beside the accuracy metric rather than in a dashboard nobody opens.

Rolling one out

Merging is not deploying. Three stages, each answering a different question: shadow the new version on a sample of live traffic and compare outputs without serving them, which catches format breakage on real inputs; serve it to a small percentage and watch the operational metrics that the eval cannot see — latency, output length, retry rate, truncation; then ramp.

Keep the previous version renderable for as long as any logged generation might be investigated. A prompt you cannot reconstruct is a log line you cannot explain.

Keep the shadow stage cheap by sampling rather than mirroring everything — a few percent of traffic usually surfaces a format break within the hour — and by comparing structurally before comparing substance: parse rate, field presence, length distribution. Reading outputs side by side is valuable and does not scale, so spend it on the cases where the structural comparison already flagged a difference.