Using a Cheap Model in CI and the Expensive One in Production

Swapping the production model for a cheaper sibling in CI can cut the suite’s bill by most of itself. It also changes what the suite is testing, in two opposite directions at once, and only one of them is the risk people talk about.

Two failure modes, not one

The discussed risk is the masked regression: a prompt change breaks something the expensive model would have caught, the cheap model was never good enough at that capability to notice, and the suite stays green. This is real, and it is the reason a cheap tier cannot be the only gate on a prompt change.

The undiscussed risk is the phantom failure, and in practice it does more damage. The cheap model fails a case the production model passes comfortably. The build goes red. Somebody spends an afternoon on a defect that does not exist in production, and then loosens the assertion until the cheap model passes — which permanently weakens a test that was correct. A tier substitution that produces phantom failures does not merely waste time; it ratchets the suite down to the weaker model’s capability and leaves it there.

Both modes come from the same cause: the assertion depends on a capability that differs between tiers. The fix is not to pick a different cheap model. It is to sort the assertions by whether they depend on capability at all.

What transfers between tiers

These assertions are about the wiring, and the wiring is the same regardless of which model answered.

• Request construction. The messages array is built correctly, the tool definitions serialise, the parameters are set.
• Response handling. Parsing, streaming assembly, finish_reason branching, usage accounting, error translation, retry on rate limits.
• Schema mechanics. That a structured-output request with a valid schema returns something the validator accepts — provided both tiers support the same structured-output mode, which is the thing to check first.
• Cost and token accounting. That your meter adds up, that budgets are enforced, that a cap trips. The absolute numbers differ; the arithmetic does not.
• Absence assertions. That no secret appears in the output, that redaction happens before logging. These are properties of your pipeline, not of the model.

If most of your suite is in this list, the honest conclusion is not “use a cheap model” but “use no model” — all of it runs faster and free against a recorded response.

What does not transfer

• Instruction adherence under pressure. A long system prompt with several competing constraints is where tiers separate most sharply. The cheap model drops the fourth constraint; the expensive one holds it.
• Tool selection among many tools. With two tools, most models choose correctly. With fifteen, selection accuracy is precisely the capability you are paying for, and a cheap-tier test measures a different quantity.
• Long-context recall. Both tiers may advertise a large window; retrieval from the middle of it is not the same skill, and a test built on a short prompt cannot see the difference.
• Format adherence without a schema. If your prompt asks for a format in prose rather than enforcing it with structured output, adherence is a capability and it does not port.
• Refusal and safety boundaries. Tiers are tuned differently. A test asserting a refusal, or asserting the absence of one, is testing that tier’s tuning.
• Feature support. The dangerous one, because it fails silently in the passing direction: if the cheap model does not support a feature your test exercises — a structured-output mode, parallel tool calls, a reasoning parameter — the test may pass by never reaching the branch.

Thresholds do not port at all

This deserves separating out because it is the most common quiet mistake. Any number you tuned — a rubric score floor, a similarity cutoff, a maximum allowed refusal rate, a pass rate on a golden set — was calibrated against one model’s output distribution. Move to another tier and the same threshold means something else entirely.

A cutoff calibrated on the expensive model will fail constantly on the cheap one; recalibrated for the cheap one, it will pass everything the expensive one produces including genuine regressions. There is no conversion factor, because the two distributions differ in shape and not only in location. A threshold belongs to a model, and moving the model means the threshold has to be re-derived from data, not scaled.

The practical consequence: never share a threshold constant between a cheap tier and a production tier. Two constants, each labelled with the model it was derived against, and a comment saying when.

A rule that survives contact

Sort every test into three buckets and route them.

1. No model. Anything in the transfers list runs against a recorded response with the network disabled. This is most of the suite and it is free.
2. Cheap model, non-gating. Smoke tests that the real API path works end to end — credentials, routing, tool plumbing, a schema round-trip. Cheap because these do not depend on capability, and non-gating because a phantom failure here must not block a merge.
3. Production model, gating, small. Every assertion about capability, and every threshold. Keep this set deliberately small — a dozen cases, not a hundred — because it is the expensive one and because a small set that runs on the real model catches more than a large set that runs on the wrong one.

The rule with the sharpest edge: a prompt change must never be gated only by the cheap tier. Prompt changes are exactly the class of change whose effect differs between tiers, and gating them on the weaker model is the masked regression by construction. Everything else can negotiate; that one should not.