Migrating a Team's Prompt Review Checklist for a New Model's Quirks

A prompt review checklist is a list of questions someone asks before approving a prompt change. After a migration it needs a handful of new questions, and the first one is the one nobody asks: what is each few-shot example in this prompt actually doing?

What each example is actually for

Few-shot examples get written for two entirely different reasons, and teams almost never label which is which. The first is format anchoring: the example exists to pin an output shape. Two lines, no preamble, keys in this order, a citation marker that looks like this. The content of the example is arbitrary; only its silhouette matters.

The second is task teaching: the example exists to demonstrate a judgment the instructions cannot state. Which of five labels applies to a genuinely ambiguous case. When to abstain rather than guess. What counts as a claim needing a citation and what is background. The shape of these examples is incidental; their content is the whole point.

These two survive a model change very differently, which is why the distinction earns a checklist item. A format-anchoring example is competing with the target model’s structured-output machinery. If the target enforces a schema at decode time — a JSON schema in response_format on an OpenAI-shaped API, or an output_config.format on the Messages API — then the example is duplicating a constraint the decoder now guarantees. That is wasted input tokens at best. At worst the example and the schema disagree on one field name or one optional key, and the model has been handed two specifications for the same output.

A task-teaching example, by contrast, must be kept, and must be re-read. Models match the register of their examples: length, tone and structure of the demonstrations propagate into the output. An example written verbosely for a terse model will make a naturally verbose model more verbose. So the checklist item is not “keep or delete” but “keep, and check that its style is still the style you want”.

A test that tells the two apart

You cannot classify examples by looking at them, because a good example does both jobs at once. The discriminator is behavioural: delete the example and rerun the evaluation, then read which metric moved.

• Only the format checks regress — schema validity, key ordering, a length assertion, presence of a required marker. The example was anchoring format. If the target enforces that format structurally, delete it permanently and rely on the schema.
• Accuracy on the hard cases regresses — disagreement rate against your labels rises, or the abstention rate collapses. The example was teaching the task. Keep it, and treat it as prompt content that needs the same review as an instruction.
• Nothing moves. The example was carrying its own weight in tokens and nothing else. This is more common than teams expect in prompts that grew one example per incident.

Run this once per example, not once per prompt. It is a few evaluation runs and it converts an argument about taste into a table. Bundle the result into the migration pull request so the reviewer is reading evidence rather than adjudicating opinions.

Where the system prompt physically goes

The second migration item is structural. Ask where the system prompt lands in the request the adapter now builds. The Messages API takes it as a top-level system parameter, outside the messages array. OpenAI-shaped APIs take it as a message inside the array, and newer models there use a developer role in place of the older system role for instructions with operator authority.

A translation layer that flattens the system prompt into the first user message will work — the model will read it — and it will quietly change two things. Instructions delivered through the privileged channel are treated differently from instructions inside user content, which is precisely the property prompt-injection defences rely on. And the caching prefix changes shape, because the render order the provider hashes puts tools and system content ahead of messages. The checklist item is: does the target expose a distinct privileged channel, and are we in it?

A related item worth adding while you are here: prompts written to overcome an older model’s reluctance tend to over-apply on a more literal one. Anthropic’s own migration guidance is explicit that instructions of the form “CRITICAL: you MUST use this tool” were written for models that under-triggered and now cause over-triggering, and recommends restating them plainly. Add a checklist line for shouting: if a rule is in capitals and has no reason attached, ask what failure it was patching and whether that failure still reproduces.

Parameter items that produce real errors

The last group of items catches the failures that show up as a 400 rather than as degraded output. Each is a single line in the checklist with the error string next to it, so the reviewer recognises it in a log rather than deducing it.

Unsupported parameter: 'max_tokens' is not supported with this model.
Use 'max_completion_tokens' instead.

That one appears whenever a request built for an older chat model is sent to a reasoning-tier model on an OpenAI-shaped API; the output cap moved to max_completion_tokens, and the cap now covers reasoning tokens as well as visible ones. Sampling parameters are the other reliable source: some current models reject a non-default temperature or top_p outright rather than ignoring it, so a default carried across from an old client configuration becomes a hard failure on every call. Stop sequences differ in the permitted count and in the parameter name (stop against stop_sequences), and a prompt that relied on four of them may not fit.

Turning it into a reviewable checklist

1. Add a section to the pull request template titled “model migration” that only appears when the diff touches a prompt or a model binding.
2. For each few-shot example added, changed or retained, require a one-word classification — anchoring or teaching — and, for anchoring examples, a note on whether the target enforces the shape structurally.
3. Require the delete-and-rerun table for any prompt where examples were removed. An assertion that removal is safe is not the same artefact as a run showing it.
4. Require the system prompt’s destination to be named: which parameter or which role. One line.
5. Require a link to the request as actually serialised on the new provider — not the code that builds it — so the reviewer can see the parameter names that were sent.

Every item asks for an artefact rather than a yes. That is the difference between a checklist that catches the four failures above and one that gets ticked in eight seconds. The style rules that survive between migrations belong in the prompt style guide instead; the items here are the ones that only earn their place while a migration is in flight.