Migrating an Internal Prompt Approval Workflow

The workflow survives the migration unchanged. What does not survive is the reviewer’s intuition, which was the load-bearing part: they knew what the old model did with an ambiguous instruction, and the checklist was a formality on top of that knowledge.

What the gate is actually for

A prompt review gate exists to catch changes whose effects are not visible in the diff. A one-word edit can change a refusal rate, a formatting convention or an output length distribution, and none of that is inferable from reading the words. Before a migration, the reviewer bridged that gap from experience. After it, they cannot, and the gate degrades to a spelling check unless the knowledge moves into the process.

That gives the design rule for the migrated checklist. Every item must be either automated, or answerable in under a minute by looking at the diff and its attached evidence. An item that requires a judgement about model behaviour will be answered “looks fine” by the third review, and a checklist with one rubber-stamped item on it teaches everyone that the others are optional too.

The checks a new provider adds

Four new items, each tied to a mechanism rather than to a preference.

• Instruction literalness. Model families differ in how much intent they infer. OpenAI’s GPT-4.1 prompting guide states plainly that the model follows instructions more closely and more literally than predecessors that inferred intent more liberally, and that existing prompts may not carry over unchanged — see the guide. The reviewable form of this is not “is the prompt clear”. It is: does every conditional instruction state what to do when the condition does not hold. That is a yes-or-no question about the text.
• House delimiters. If the migration changed how prompts are structured — sections, example wrappers, output markers — then consistency is now a correctness property, not a style preference, because the model uses those markers to segment the prompt. This item belongs in a linter, not in a human’s head; see migrating prompt linting rules.
• Refusal surface. Content policies differ between providers, so a prompt that was never refused on the source can be refused on the target, and a prompt that was refused can now pass. The check is procedural: did this change run against the refusal probe set, and is the result attached. The policy differences themselves belong in the page on content policy differences.
• Parameter ranges and defaults. A sampling parameter copied across from another API may be out of range, or in range and meaning something different at the same number. The reviewable form: does the diff change any generation parameter, and if so is the new value inside the target’s documented range and justified in one sentence.

What a reviewer approves on

Not the prompt. The evidence attached to it. A migrated workflow should require every prompt change to carry three things, produced by CI rather than pasted by the author:

• the identifier of the eval run, and the prompt version it was run against, so the result is reproducible six months later;
• the pass rate on the held-out set before and after, on the same inputs and the same scoring function;
• the list of items that newly fail. Not the count — the list. An aggregate that improves while three specific cases regress is a change worth having a conversation about, and the count hides exactly that.

Making the CI job that produces this a blocking check is the entire reform. Everything else on the checklist is secondary to the fact that the reviewer now has something to look at that is not their memory of how a different model behaved. The held-out set itself is the artifact the workflow depends on; if you do not have one, building a golden dataset precedes all of this.

One rule about the scoring function: fix it before the migration and do not change it during. A scoring change and a provider change landing together produce numbers that cannot be attributed to either.

The dual-provider period needs an end condition

For a window, every prompt change ships to both providers and is reviewed against both. That is the right call and it is expensive: review effort roughly doubles, and the two prompts diverge as soon as somebody makes a target-specific fix. Teams enter this period without writing down how it ends, and a year later it is simply how the team works.

Write the end condition when you start, as a conjunction of things you can check. For example: the target has carried the full traffic share for a defined number of consecutive days; the last N prompt changes have required no source-specific variant; and the source’s eval suite has not caught a failure the target’s suite missed. When all three hold, the source path is deleted from the workflow the same week — not marked deprecated, deleted, because a review step that exists but is optional is a review step everyone skips inconsistently.

A subtlety worth planning for: during dual-running, the two prompt variants must be versioned separately and both must be reachable from one logical prompt identity, otherwise the eval results cannot be compared and the registry cannot answer “what is in production”. That is a branching problem, and it is the same one as in branching prompt version control through a migration.

Retiring checks so the list stays read

Checklists only grow, and a checklist that has grown past about a dozen items stops being read line by line and starts being scanned. So each item carries a review date and a purpose recorded in one sentence. At the review date, ask what it has caught. An item that has caught nothing in ninety days is either wrong or should be automated; either way it leaves the human list.

This is also the mechanism that lets the migration-specific items disappear. Half the checks above exist because the team is new to a provider; a year in, they are either encoded in a linter or they are folklore, and the review date is what forces that decision instead of letting the list calcify. Who holds that decision for each prompt is the subject of prompt ownership, and it should be a named person rather than a team, because a team cannot decline to renew a check.