When a Small Model Is Genuinely Good Enough

“Is the small model good enough” has no answer, because “good enough” is not a property of a model. It is a relationship between how often each model fails, how much a failure costs, and how much money the swap saves. All three are numbers.

The question is not which is better

The larger model is better. That is not in dispute and it is not the decision. The decision is whether the quality difference, valued in money, exceeds the price difference, and a benchmark leaderboard cannot tell you that because it does not know your task and has no opinion about what your failures cost.

So the procedure starts by refusing to look at benchmarks. Take 300 to 500 real requests out of your logs, freeze them as an evaluation set, and define — in advance, in writing — what makes a response a failure for your product. “Failure” must be binary and it must be checkable by something other than vibes: schema invalid, cited document does not support the claim, the extracted field is wrong, a human reviewer marks it unusable. If you cannot write the definition down, you cannot make this decision quantitatively and everything below is unavailable to you.

The break-even failure cost

Run both models over the frozen set. You now have four numbers:

c_big, c_small     cost per request of each model
q_big, q_small     failure rate of each model  (0..1)

dC = c_big - c_small        money saved per request by switching
dQ = q_small - q_big        extra failures per request from switching

Switching is worth it when   dC  >  dQ * L
where L is what one failure costs you.

Rearranged, the break-even failure cost:   L* = dC / dQ

L* is the useful form. It converts an argument about model quality into a single question that a product owner can actually answer: is one bad answer worth more or less than this?

Worked, with all four inputs as assumptions:

c_big   = $0.0100      q_big   = 4%
c_small = $0.0010      q_small = 9%

dC = $0.0090
dQ = 0.05

L* = 0.0090 / 0.05 = $0.18

If a bad answer costs less than 18 cents, switch. For a “suggest three tags for this note” feature where the user simply ignores a bad suggestion, the cost of a failure is a fraction of a cent of annoyance and the answer is obvious. For a feature that generates a support reply sent without review, one bad answer might cost a ticket escalation, and a support ticket costs several dollars of somebody’s time — well above L*, so keep the large model. Same models, same prices, opposite decisions, and the thing that decided it was never the model.

Two refinements worth adding once the basic form is in place. First, if failures are caught before they reach a user — by a validator, a human review queue or a cascade — then L is the cost of the catch, not the cost of the consequence, and it is usually far smaller. That is precisely why cascading changes the calculus. Second, if a caught failure is retried on the large model, the small model’s effective cost is c_small + q_small × c_big, which is the cascade formula again.

How many examples you need

A 300-request evaluation set that shows 4% versus 9% is a real signal. A 40-request set that shows 5% versus 10% is two failures against four and means nothing. The standard rule of thumb for the number of examples per model, to detect a difference d between two rates with roughly 80% power at the usual significance level:

n  ~=  16 * p_bar * (1 - p_bar) / d^2

  p_bar  the average of the two failure rates
  d      the difference you want to be able to detect

For p_bar = 0.065 and d = 0.05:
  n ~= 16 * 0.065 * 0.935 / 0.0025 = 389 per model

Roughly 400 each. That is a useful number to know before anyone proposes deciding from a demo of six examples, and it is also reassuring: 400 requests through both models, at the prices above, costs about $4.40 in total. The evaluation is cheap. The thing that is expensive is the labelling, which is why the failure definition needs to be programmatically checkable wherever possible.

Note what the formula says about small differences: detecting d = 0.01 instead of 0.05 needs 25× the examples. If your two candidates are genuinely within a point of each other on your task, stop trying to prove it and take the cheaper one.

Where the answer usually goes each way

Not a ranking and not a measurement — a description of the task properties that push dQ toward zero or away from it, which is the thing that actually generalises across model releases.

• dQ tends to be small when the answer is present in the context and the job is to find it, reformat it or extract it; when the output space is closed (classification into known labels, routing, yes/no); when a schema constrains the shape; and when the task is short and single-step.
• dQ tends to be large when the task needs knowledge not in the context; when it requires several dependent reasoning steps; when it involves long-range consistency across a large document; when instructions are numerous and must all be obeyed simultaneously; and when the task involves code that has to run.

The general pattern behind both lists is that small models have lost capacity, not information-processing ability. Give a small model the facts and a narrow job and the gap closes; ask it to supply the facts or to hold a long chain together and the gap opens. That framing has survived several generations of releases and is more durable than any specific comparison, which is the reason there is not one here.

The costs the formula leaves out

• Prompt portability. Prompts tuned for one model frequently underperform on another, so a fair comparison includes a few hours of adapting the prompt to the small model. An unadapted prompt overstates dQ and can talk you out of a good switch.
• The evaluation is perishable. Both models will be updated. Keep the frozen set and the runner in the repository so re-running the decision is an afternoon rather than a project.
• Latency is a separate axis. Smaller models are usually faster, which is a benefit the cost formula does not capture. If the route is user-facing, that may matter more than the money.
• The failure distribution changes shape, not just size. A small model does not fail on a random 9% of requests; it fails on the hard ones, which may be concentrated in one customer or one document type. Check whether the failures cluster before accepting an aggregate rate.