Few-Shot Examples Stop Working After Switching Models

The prompt is identical, the examples are identical, and the classification accuracy dropped four points. Few-shot examples are not instructions; they are a demonstration the model imitates, and what it picks up from a demonstration is a property of how it was trained. Change the family and you change what gets imitated.

What the failure looks like

The characteristic signatures are worth learning, because each points at a different mechanism below.

• The output format regressed — extra preamble, a code fence around JSON that used to be bare, a different key order. This is usually the turn-structure problem.
• Accuracy fell but the format held. Look at the confusion pattern: if errors concentrate in one class, suspect label balance.
• Outputs got longer or more hedged. Stylistic residue: the examples were written to correct the old model and are now pushing in the wrong direction.
• The model started explaining its answers when the examples show bare answers. Common when the target is reasoning-tuned.
• Cost per call rose sharply with no quality change. Your example block tokenizes to more tokens under the new vocabulary, which is a real effect on any set with heavy delimiters or non-Latin text.

Examples as text versus examples as turns

There are two ways to supply few-shot examples on a chat API and they are not equivalent.

As text, all the examples live inside one user message, separated by labels or delimiters. This is portable, cheap to template, and easy to select from dynamically. Its weakness is that the model sees a block of text describing a task, not a record of the task being performed.

As turns, each example becomes a real user message followed by a real assistant message, and only then does the actual input arrive as the final user turn. Instruction-tuned models are trained overwhelmingly on turn-structured data, so a demonstration in that shape sits directly on the prior the model uses to decide what an assistant message looks like. In practice this is the single largest lever on format compliance.

// as text — one message
messages: [
  { role: "system", content: SYSTEM },
  { role: "user", content:
      "Classify the ticket.\n\n" +
      "Input: My card was charged twice.\nOutput: billing\n" +
      "Input: The export button does nothing.\nOutput: technical\n\n" +
      "Input: " + ticket + "\nOutput:" },
]

// as turns — the same examples, structurally
messages: [
  { role: "system", content: SYSTEM },
  { role: "user", content: "My card was charged twice." },
  { role: "assistant", content: "billing" },
  { role: "user", content: "The export button does nothing." },
  { role: "assistant", content: "technical" },
  { role: "user", content: ticket },
]

The turn form has two costs to know about. It interacts with prompt caching — a long fixed example prefix caches well, a dynamically selected one does not — and it complicates any code that assumes the message list is history. Tag the synthetic turns in your own store so the example messages never get persisted into the conversation record; that is a real bug people ship, and it makes the thread grow by the example set on every turn.

Stylistic residue from the old model

Few-shot examples are almost never written from scratch. They are edited from outputs the previous model produced, or written iteratively until that model behaved. Both processes bake in corrections that are specific to it.

If the old model was verbose, your examples are probably unnaturally terse, because you shortened them until the output came out right against a verbose prior. Give those examples to a naturally terse model and it becomes clipped to the point of dropping required content. The same runs in reverse: examples padded with an explanatory clause to coax a laconic model produce a target that now explains everything.

Then there are the artefacts nobody intended to teach. Whatever is consistent across your examples is a pattern, whether you meant it or not: every example answer ending in a full stop, every one being under twelve words, every one starting with a verb, an American spelling throughout. The model imitates all of it. If your examples happen to share a superficial feature that does not generalise — say, every example input is short — the model will underperform on the long inputs that dominate your real traffic.

The check is mechanical: compute the distribution of output length, first token, and punctuation across your example set, and compare it with the distribution of the outputs you actually want. Divergence there is the residue.

Count, order and balance

• Count is not monotonic. More examples help until they compete with the instruction, and where that turns over differs per model. A stronger instruction-follower often does better with three examples than with twenty, because twenty examples that each differ slightly from the stated rule teach the model that the rule is approximate. Sweep the count as a parameter — 0, 2, 4, 8, 16 — rather than assuming the old number transfers.
• Order carries weight, and the weighting differs. Positional effects are real and family-specific: the last example is the most recent thing the model saw before your input. If your examples are dynamically selected and sorted by similarity, the sort direction is a parameter you have set without deciding, and it is worth flipping once to see. Selection strategy in general is covered in testing few-shot selection logic.
• Label balance sets a prior. This is the one that causes the accuracy-fell-in-one-class signature. If six of your eight classification examples are labelled technical, the example set is telling the model that most tickets are technical. Some models largely ignore that and rely on the instruction; others copy the distribution. A set that was harmless on the old model can be actively misleading on the new one. Balance the classes, or make the imbalance deliberate and match it to your real base rate.
• Reasoning targets change the calculus again. Against a reasoning-tuned model, examples that show a bare answer with no working can conflict with the model’s own process, and examples that show worked reasoning add tokens for something it does anyway. Try the zero-shot baseline first on those models; it is often competitive and it removes the whole problem.

The repair pass

1. Get a baseline before you change anything. Run your evaluation set against the new model with the existing examples, and also with zero examples. If zero-shot is close, the examples were doing less than you thought and the cheapest fix is to delete most of them.
2. Convert to turns if you were using the text form, and re-measure. This is the highest-yield single change for format regressions and it does not require rewriting any example content.
3. Rebalance the labels to match your intended prior, and re-measure per class rather than in aggregate.
4. Sweep the count at 0, 2, 4, 8 and 16 with the set held otherwise constant. Record the cost per call alongside the quality figure — half the value of this sweep is discovering you can drop twelve examples for no loss.
5. Rewrite the example outputs against the new model. Not the inputs — the inputs are real data and should stay. Regenerate candidate outputs with the new model, correct them by hand, and use those. This removes the old model’s residue at the source and is quicker than editing the old ones.
6. Move hard constraints out of the examples. Anything that must always hold — a field must be present, a value must be from an enum — belongs in the instruction and, where the API supports it, in a schema. Examples are for demonstrating judgement, and a constraint expressed only as a pattern across eight samples is a constraint you are hoping for rather than one you have.
7. Freeze the final set as a versioned artefact next to the prompt, so the next migration starts from a known state rather than from a string literal somebody edited in a hurry.