Migrating a Prompt Versioning System Between Providers

A prompt versioning scheme built for one provider has an implicit assumption in it: that a version number identifies a behaviour. During a migration that stops being true, because the same version now runs on two models and produces two behaviours.

Why the existing scheme stops working

Most working schemes are some form of “prompt id plus version number, stored in the repository, referenced by the calling code, and recorded on every request”. That is a good design and prompt versioning covers it. The migration breaks it in three specific places.

• A version no longer identifies an output distribution. When you look at a bad response and see that it came from summarise@v7, you can no longer reproduce it without also knowing which provider served it. Every reproduction path, every bug report and every regression triage now needs a second coordinate.
• Some text genuinely has to differ. Not the instructions — the scaffolding. Where a system instruction lives, whether examples are expressed as alternating turns or as a block, how a tool schema is declared, and whether a JSON-output constraint is enforced by a request parameter or by prose all differ between providers. A single string that works well on both is usually a string that works optimally on neither.
• Rollback has to include the prompt. If you tuned v8 to fix something on the new provider and then roll traffic back, the old provider is now serving a prompt that was optimised against a different model. A rollback that reverts routing but not the prompt version is a partial rollback, and partial rollbacks are how a migration produces an incident on the system it was rolling back to.

Widening the version key

The instinct is to branch: one prompt directory per provider. Resist it, because a branch is permanent by default and nobody ever merges it back — you end up maintaining two libraries of prompts long after the migration ended, and the second one silently rots.

The alternative is to widen the key temporarily. A prompt is addressed by (id, version, target), where target names a provider or model family and is allowed to be absent. Resolution falls back: ask for a target-specific variant, and if there is not one, use the shared one. The overwhelming majority of prompts never grow a variant.

prompts/
  summarise/
    v7.md              # shared body: instructions, constraints, examples
    v7.target-b.md     # only where behaviour genuinely required a variant
    meta.yaml          # owner, created, supersedes, targets: [default, b]

resolve(id="summarise", version=7, target="b")
  -> v7.target-b.md if present, else v7.md

Every request records all three: prompt_id, prompt_version, prompt_target.

Recording all three on the request is the part that pays for itself immediately. It is what lets you answer “did the regression come from the provider change or from the prompt change”, which during a ramp is the question you will ask most often and cannot answer any other way — because both changed in the same week. The field goes in the same normalised log record as everything else in the logging field mapping.

One discipline makes the whole scheme hold: a variant may never change the instructions, only the packaging. If a variant needs to say something different about the task, that is not a variant, it is a different prompt, and the difference will eventually produce two systems with two behaviours that nobody intended. The practices for holding variants together are in keeping prompt versions in sync during a migration.

What legitimately diverges, and what must not

A short list of what is properly a packaging difference, so that “this needs a variant” is a decision with a test attached rather than a preference.

Legitimate

• Where the system instruction goes. One provider takes it as a message with a designated role inside the messages array; another takes it as a separate top-level request parameter. The text can be identical; the placement cannot.
• How output structure is requested. Where structured output is enforced by a request-level schema parameter, the prompt should not also spend two hundred tokens describing the JSON shape in prose — that is duplicated instruction and a source of conflict. Where it is not enforced, the prose is load-bearing. This is a real divergence in prompt length and content, and it is the most common legitimate variant. Background in structured output support.
• Few-shot formatting. Examples as alternating user/assistant turns versus examples inline in a single instruction block. Both work; which works better is model-dependent and cheap to check on your replay corpus.
• Stop sequence and length handling. Different defaults for maximum output length mean a prompt that says “answer in at most three sentences” may need reinforcing on one side and not the other.

Not legitimate

• Any change to what the task is, what the output must contain, or what the model must refuse. These are product decisions and must be identical across targets or your two providers are running two products.
• Silent divergence — a variant edited to fix a live issue without the shared version being updated or the difference being recorded. This is how the temporary widening becomes a permanent fork.

Evaluation results are versioned too

The consequence people forget: every stored evaluation result was produced by a prompt version on a model, and adding the target dimension to prompts adds it to results as well. A results table keyed only by prompt version will quietly overwrite the old provider’s numbers with the new one’s, and the baseline you were going to compare against disappears at exactly the wrong moment.

Key evaluation runs on the full tuple — prompt id, prompt version, target, model string as returned, dataset version, and the date. Then a regression during the ramp can be attributed: same prompt and different model is a provider effect, same model and different prompt version is your own change, and both changing at once is a run you cannot learn from, which is itself a useful thing to detect before you spend money on it. The broader treatment of comparing runs across model versions is in regression suites across model versions.

Collapsing the dimension when the migration ends

The extra dimension is temporary, and saying when it ends is what stops it becoming permanent. Put this in the closing section of the migration runbook alongside the removal of the routing flag, because they expire together and for the same reason: an untested alternative is not an alternative.

1. When the soak completes and the old provider is decommissioned, promote each surviving target-specific variant to be the shared body for a new version, and delete the variant file. The new version number is the record that the change happened; the variant mechanism does not need to persist to preserve the history.
2. Delete the variants for the retired target in the same change, not later. A variant for a provider you no longer call is dead text that the next person will assume is live.
3. Keep the prompt_target field in your logs and your evaluation keys even after collapsing. It costs one column and it means the next migration widens an existing dimension rather than adding one to a schema that is already in production.
4. If the migration ends in a deliberate multi-provider steady state — routing by cost, by capability or by tenant — then the dimension does not collapse, and that is fine. But say so explicitly, and put the variants under the same review process as the shared bodies, because from that point they are permanent product surface rather than migration scaffolding.