Testing Whether a Prompt Is Portable Before You Switch Model Families

Most prompt portability work is written as though the risk were quality — will the new model be as good. The migrations that go badly rarely fail on quality. They fail because the request itself is not accepted, or is accepted and means something different.

What actually breaks, in order of likelihood

• Where the system prompt goes. Some APIs take it as a top-level system parameter, others as the first message with a system role, and a few models treat a long system prompt with markedly different weight from the same text in a user turn. A translation layer that quietly moves it changes the prompt.
• Assistant prefill. Putting words in the assistant’s mouth to force a format is a widespread trick and it is not universally supported. Anthropic’s errors documentation records that Claude 4.6 and later models reject a request whose last message is a prefilled assistant turn, with a 400 invalid_request_error whose message says the model does not support assistant message prefill and that the conversation must end with a user message. A prompt built around prefill is not portable to those models at all, and the replacement is a structured output constraint rather than a reworded instruction.
• The tool schema dialect. The same tool is input_schema in one API and parameters nested under a function object in another, and the surrounding fields differ in whether a description is required, how strictly the schema is enforced, and what happens to an unsupported JSON Schema keyword.
• Message alternation and empty content. Some APIs reject two consecutive user messages, or a message with empty content. A transcript that a lenient provider accepted for a year fails on the first request to a stricter one.
• Token accounting. A prompt that fits comfortably can stop fitting on a model with a different tokenizer even at the same advertised context length, which is the subject of the pre-deploy context check.
• Stop sequences and reasoning. A stop sequence that worked as a delimiter can interact badly with a model that emits reasoning content, and a model that always reasons will not honour a request to answer with a single token in the way a non-reasoning one does.

Assertions that transfer between families

The reason a portability suite is possible at all is that most of what you need from a prompt is checkable without comparing prose. Write the suite entirely out of assertions that are true of a correct answer from any model:

• Shape. The output parses and validates against your schema. This is the single highest-value assertion because it is binary and it is the thing that most often breaks.
• Tool choice. For an input that unambiguously requires a lookup, assert that some tool was called and that its name is in the expected set — not which arguments were chosen.
• Invariants over the input. Every identifier that appeared in the source document appears in the extraction. No date in the output that was not in the input. The summary contains no numeric token absent from the source.
• Metamorphic relations. Reorder the retrieved chunks and the extracted fields should not change. Rename an entity consistently throughout the input and the output should change in exactly the same way. These hold for a correct system regardless of wording, which is precisely why they survive a model change when an exact-match assertion does not.
• Refusal and abstention. For an input where the right answer is “not enough information”, assert the system takes the abstention path. This is the assertion most likely to regress on a model swap and the one nobody writes.

What does not transfer is any assertion on a sentence. If a test expects the string “I could not find that record”, it is a test of one model’s phrasing and it will fail on every candidate including a better one.

The matrix test

Parameterise the same cases over the candidate models and run them as one suite. In Vitest that is a table driving describe.each; in pytest it is @pytest.mark.parametrize on a model fixture. Keep the model list in a config file so adding a candidate is a one-line change and so the CI job can run a subset.

import { describe, expect, it } from "vitest";
import { extractInvoice } from "../src/pipeline";
import { cases } from "./fixtures/invoices";

const CANDIDATES = ["provider-a/model-1", "provider-b/model-2", "provider-b/model-3"];

describe.each(CANDIDATES)("invoice extraction on %s", (model) => {
  it.each(cases)("$name: output validates and preserves identifiers", async (c) => {
    const out = await extractInvoice({ model, document: c.document });

    expect(() => InvoiceSchema.parse(out)).not.toThrow();
    for (const id of c.identifiersInSource) {
      expect(JSON.stringify(out)).toContain(id);
    }
    expect(out.total_cents).toBeTypeOf("number");
  });

  it("abstains when the document has no total", async () => {
    const out = await extractInvoice({ model, document: cases.noTotal.document });
    expect(out.total_cents).toBeNull();
    expect(out.confidence).toBe("insufficient");
  });
});

Run it against a fixed, small case set — twenty or thirty documents chosen to cover the awkward shapes, not a thousand sampled at random. The suite exists to answer a yes-or-no question about a migration, and a case set small enough to read is a case set somebody will actually look at when a cell goes red.

Hard failures you want at request time

Separate the request-shape checks from the behaviour checks and run them first, because they are cheap, deterministic and they explain the rest. A single test per candidate that sends the smallest valid request — with your real tool definitions attached and your real system prompt — and asserts a 200 will catch prefill rejection, schema dialect mismatches and alternation errors in under a second each, before any of the behavioural cases run and confuse the picture by failing for the same underlying reason.

Reading the matrix

A cell that fails on shape is a blocker and is usually a day’s work. A cell that fails an invariant is a real behavioural difference and needs a decision. A cell that fails only the abstention case is the most dangerous result of the three, because the system will look fine in every demo and will confidently answer questions it has no basis for in production.

Do not average the matrix into a score. The point of running it per case is that you can see which case broke, and a single number turns a migration decision back into the vibe check the suite was written to replace. Keep the matrix from the last run committed alongside the fixtures so the next person can see what changed rather than what is true today — the same reasoning behind watching for model updates you were not told about.