Model Cascading: Cheap Model First, Expensive on Failure

A cascade tries a cheap model, checks the answer, and escalates to an expensive one only when the check fails. Whether it saves money is not a matter of opinion: it depends on one rate, and that rate has a threshold you can compute before writing any code.

The shape of a cascade

Three components, and the middle one is the one people get wrong.

• The cheap stage. A smaller or older model, or the same model with reasoning disabled, handling the request normally.
• The verifier. Something that decides whether the cheap answer is acceptable. Could be a schema validation, a confidence signal, a set of assertions, a retrieval-grounding check or another model call.
• The expensive stage. Runs only on failure. Usually given the original request rather than the failed attempt, because showing a model a bad answer tends to anchor it.

It is worth being clear about what a cascade is not. It is not a fallback for provider errors — that is failover, and its cost behaves differently. And it is not a router: a router picks one model up front from the request, and pays for exactly one call. A cascade always pays for at least two things.

The arithmetic

Let C_big be the cost of one call to the expensive model, C_small the cheap one, C_v the verifier, and p the probability that the verifier rejects the cheap answer. The baseline is calling the expensive model always. The cascade is:

E[cascade]  = C_small + C_v + p * C_big
E[baseline] = C_big

Note that C_small and C_v are paid on every request, including the ones that escalate. That is the part omitted from most descriptions of the technique, and it is the part that decides the answer.

The break-even escalation rate

Set the cascade below the baseline and solve for p:

C_small + C_v + p * C_big  <  C_big

                       p  <  1 - (C_small + C_v) / C_big

  p*  =  1 - (C_small + C_v) / C_big      the break-even rate

Everything about cascade design is in that expression. The threshold depends only on the ratio of the cheap path to the expensive one, which is convenient because ratios survive price changes far better than absolute figures do.

Worked, with the three costs as assumptions: C_big = $0.0100, C_small = $0.0008, and a cheap programmatic verifier at C_v = $0.0004.

p* = 1 - (0.0008 + 0.0004) / 0.0100
   = 1 - 0.12
   = 0.88

So the cascade wins unless it escalates on more than 88%
of requests. At p = 0.30:

E[cascade] = 0.0008 + 0.0004 + 0.30*0.0100 = $0.0042
saving vs baseline = 1 - 0.0042/0.0100 = 58%

Now change one input to see how fragile that is. Make the verifier a call to the expensive model itself, so C_v = $0.0100:

p* = 1 - (0.0008 + 0.0100) / 0.0100 = -0.08

A negative threshold means there is no escalation rate at which this cascade is cheaper. It cannot win, ever, and it is a design people build regularly because “have the strong model check the weak model’s work” sounds like the obviously correct architecture. If the verifier costs as much as the thing it is verifying, you have built a slower way to call the expensive model.

The verifier is the whole design

Two properties matter and they pull in opposite directions. The verifier must be cheap — from the formula, C_v eats the savings directly — and it must be accurate, because its errors are not symmetric.

A false accept ships a bad answer, which is a quality cost, not a money one. A false reject escalates unnecessarily, which raises p toward the threshold. So the useful ordering, cheapest and most reliable first:

• Deterministic checks. Schema validation, JSON parses, required fields present, the cited document id exists, the arithmetic in the answer is correct. Cost is effectively zero and the verdict is certain.
• Signals the model already gave you. A refusal, finish_reason indicating truncation, an explicit “I do not know”, or a confidence field you asked for in the schema. Free, and surprisingly effective for the retrieval case where the failure mode is “the context did not contain the answer”.
• A small-model judge. Costs about what the cheap stage costs, so it roughly doubles the fixed part of the cascade. Worth it only when the deterministic checks cannot express the failure you care about.
• The expensive model as judge. Ruled out by the arithmetic above for cost purposes. It has uses — offline evaluation, for one — but not inside the hot path of a cascade whose justification is money.

What a cascade costs you elsewhere

The mean falls; other things rise, and a design review should name them.

• Worst-case cost goes up, always. An escalating request costs C_small + C_v + C_big, which exceeds the baseline by construction. In the worked example that is $0.0112 against $0.0100 — the p99 of your cost distribution rises by 12% while the mean falls by 58%. If a per-request spend ceiling is enforced anywhere, it has to be raised to accommodate the cascade.
• Latency behaves the same way. Escalated requests wait for all three stages sequentially. For a user-facing route, compute the p95 latency of the cascade, not the mean, and check it against whatever the interface promises.
• p is not stable. It drifts with the traffic mix, a prompt change, or a silent model update on either stage. Track it as a metric with an alert, because a cascade whose p has crept past p* looks exactly like a working cascade from the outside.
• Complexity is a real denominator. Two models, a verifier and an escalation path is three times the surface area for bugs, and the w term in the lever scoring formula is not decoration.

The honest summary: a cascade is worth building when the price ratio between the stages is large, the verifier is deterministic, and you have a reason to believe p is well below p* rather than just below it. Measure p by running the cheap stage and the verifier in shadow mode against real traffic for a day before committing to the design — that costs C_small + C_v per request and answers the question exactly.