Testing the Fallback Prompt Your App Uses When the Primary Model Errors

Your fallback prompt is shorter, plainer, and aimed at a smaller model. It has run in production perhaps four times. Every one of those times was during an outage, and at least one of them produced output your parser rejected.

Why this path is always broken

Fallback prompts rot for a structural reason: they are edited when the primary is edited, which is never, because nobody remembers the fallback exists. The primary prompt gains a new output field over six months of ordinary work. The fallback still emits the four fields it had at birth. The parser downstream was updated to require five. The fallback path is now guaranteed to fail, and the only condition under which anyone finds out is the condition under which the primary is already down.

There is a second reason, which is that the fallback usually targets a different model. A prompt tuned for a large model frequently does not transfer — instructions it followed implicitly need stating, few-shot examples it did not need become necessary, and JSON it emitted reliably arrives wrapped in a code fence. That is the subject of prompt portability; the consequence here is that the fallback prompt must be evaluated against the fallback model, not against the primary.

The contract is the assertion

The fallback is allowed to be worse. It is not allowed to be different in shape, because the code that consumes it does not know which prompt produced it. So the primary assertion is a schema assertion, and it is the same schema in both tests.

import { describe, it, expect } from "vitest";
import { z } from "zod";
import { primaryPrompt, fallbackPrompt } from "../prompts";
import { complete } from "../client";

const Answer = z.object({
  summary: z.string().min(1),
  confidence: z.enum(["low", "medium", "high"]),
  sources: z.array(z.string().url()),
  needs_human: z.boolean(),
});

describe.each([
  ["primary", primaryPrompt, "provider/model-large"],
  ["fallback", fallbackPrompt, "provider/model-small"],
])("%s prompt satisfies the answer contract", (_name, prompt, model) => {
  it.each(goldenCases)("case %#", async (input) => {
    const raw = await complete({ model, prompt, input, temperature: 0 });
    const parsed = Answer.safeParse(JSON.parse(stripFence(raw)));
    expect(parsed.success, parsed.error?.message).toBe(true);
  });
});

Writing it as describe.each over both prompts is the whole trick. One schema, one set of cases, two prompts, and the fallback can no longer drift silently because the same contract test covers it. The cost is that the fallback half calls a model, so this suite belongs with your nightly evaluation run rather than on every commit — the same placement argument as a golden dataset.

The stripFence helper in that snippet is not incidental. Smaller models wrap JSON in a fenced block far more often, so the parser your production code uses must already handle it — and if it does not, this test is where you find out, which is the correct place.

Testing the trigger separately

The second thing to prove is that the fallback is reached. This is a control-flow test, it needs no model, and it should be exhaustive over the error conditions you claim to handle. Stub the transport and drive it through each failure.

• HTTP 429 — assert the primary is retried according to your policy before falling back. Falling back on the first 429 wastes the cheaper capacity you were rate-limited out of and doubles your load on the small model.
• HTTP 500 and 503 — assert immediate fallback, with the attempt counted.
• A timeout — drive it with fake timers and assert the fallback started at the deadline, not after the transport eventually gave up. Assert the total wall time budget is respected: a fallback that begins after a 60-second primary timeout has already lost the user.
• HTTP 400 and 422 — assert there is no fallback. A malformed request will be malformed for the second model too, and falling back on a 400 turns one bad request into two.
• A valid response that fails schema validation — decide whether this triggers fallback and assert the decision. It is a legitimate design either way, and an unasserted one is a coin flip.

Assert the fallback was called with the fallback prompt, not merely that a second call happened. A retry against the primary model and a fallback to a different one are different behaviours, and a call-count assertion cannot tell them apart.

A floor, not parity

Do not gate the fallback on matching the primary’s quality; it will not, and a test that demands it gets disabled. Gate it on a floor you would be willing to serve during an incident, expressed in the same terms as your primary evaluation so the two numbers are comparable.

A practical shape: the fallback must satisfy the schema on 100% of the golden cases, must retain the must-keep facts on a stated fraction of them, and must never set a field that triggers an automated action — if needs_human exists, degraded mode is allowed to over-use it and not allowed to under-use it. That asymmetry is the point of a degraded path, and it is assertable: assert the fallback escalates at least as often as the primary on the cases the primary escalates.

Keeping the two prompts from drifting apart

Two mechanical habits keep the fallback alive between incidents.

1. Derive what you can. If the output schema is generated from one definition and injected into both prompts, a new field cannot appear in one and not the other. Assert in a test that both rendered prompts contain every field name in the schema — a cheap string check that runs in milliseconds and catches the exact drift described above.
2. Version them together. One version identifier covering the pair, so a change to the primary is visibly a change to an artefact that includes the fallback.
3. Exercise it deliberately. A scheduled job that routes a small, constant share of traffic through the fallback path keeps it warm and gives you a production signal, in the same spirit as a canary release. A path exercised weekly cannot be four fields behind.