Anti-Pattern: One Model for Everything

Picking the best available model and using it everywhere is a genuinely good first decision. It is also the decision most likely to still be in place two years later, unexamined, quietly paying frontier prices for work a much smaller model would do identically.

The case for one model, honestly

Standardising has four real benefits and they are not trivial. Prompts, which are tuned to a model’s habits, only have to be tuned once — and prompts do not port cleanly. There is one set of quirks to learn, one error dialect, one set of capability flags. Any evaluation you build applies everywhere rather than being fragmented per route. And it removes an entire category of decision from every future feature, which is worth more than it sounds when a team is moving fast.

Against that, the cost of the anti-pattern is invisible by construction: nothing breaks. The bill is higher than it needs to be and the latency is worse than it needs to be, but neither shows up as a defect, so the decision is never revisited. This is the defining property of the whole pattern — it fails by not failing.

Where it stops being true

The premise of one-model-for-everything is that your workload is homogeneous. It is not, and it becomes less so with every feature. A mature application has at least four distinguishable classes of call, and their requirements have almost nothing in common.

The mechanical class is where the argument is won. A classification into six buckets, evaluated with a single token and logprobs, is a task where the difference between a frontier model and a small one is frequently within the noise of your own labelling — and it is a task you can test in an hour, because the answer space is closed and the ground truth is cheap to produce.

The arithmetic, on your numbers

Whether routing is worth doing is a calculation, not an opinion, and it needs four inputs from you. There are no numbers on this page because there are no numbers that would be about your traffic.

YOU SUPPLY:
  share_i   fraction of calls in class i          (from a week of logs)
  tokens_i  mean input and output tokens for i    (from the same logs)
  price(m)  input and output price of model m     (from the catalogue)
  delta_i   accuracy of the cheap model minus the expensive one, on class i
            (from YOUR eval set — this is the term nobody can give you)

DECIDE PER CLASS:
  saving_i  = share_i * tokens_i * (price(expensive) - price(cheap))
  risk_i    = share_i * delta_i * cost_of_one_error_i

  route class i to the cheap model  <=>  saving_i > risk_i

THEN CHECK THE SHAPE OF YOUR TRAFFIC:
  if one class dominates share_i, routing the others changes nothing.
  if one class dominates share_i * tokens_i * price, that class is your bill,
  and every optimisation elsewhere is decoration.

Two things fall out of writing it this way. The first is that delta_i is the only term you cannot look up, and it is the term the decision turns on — which is why a routing project that starts with a benchmark leaderboard rather than with an evaluation of your own classes is starting at the wrong end. Public benchmark rank is a poor predictor of task performance, and the closer your task is to mechanical, the less it predicts.

The second is the last block. Cost is almost always concentrated. Rank your classes by share × tokens × price before optimising anything, because the usual finding is that one or two of them are the bill and the rest are rounding. Routing the rounding is work that produces no result and consumes the appetite for the work that would have.

A routing rule you can defend

A rule survives contact with a team only if it is simple enough to apply without a meeting. This one is three questions per call site, in order.

• Is the output space closed? An enum, a boolean, a small set of labels, a number in a range. If yes, start with the cheapest model that supports the interface, and promote only if your eval says otherwise. Closed answer spaces are where cheap models are most competitive and where you can prove it fastest.
• Is the output verified before it is used? If a schema check, a database lookup or a downstream test will catch a bad answer, the cost of being wrong is a retry rather than an incident, and that changes cost_of_one_error by orders of magnitude. Verified outputs tolerate cheaper models.
• Does a human read this specific output? If yes, and if their judgement of the product depends on it, this is the class to spend on. If no — an embedding, a routing decision, an internal tag — the quality bar is “good enough for the consumer of it”, which is code.

Two further notes. Routing by task is not the same as cascading, where a cheap model attempts everything and escalates on failure; cascading is a runtime decision and this is a design-time one, and they compose. And the decision belongs in configuration rather than in code, because the right answer changes when a provider ships a new model — which is often, and which is precisely when nobody wants to open a pull request.

What the fix costs you

Routing is not free, and pretending otherwise produces the opposite anti-pattern: eleven models, four of which are load-bearing and none of which anyone has evaluated since they were added.

• Prompts multiply. Each routed class needs its prompt validated on its model. That is real work and it recurs on every model change, which is an argument for keeping the number of distinct routes small — three or four, not ten.
• Evaluation multiplies. Each route needs its own cases, or you have no basis for the routing decision you made. This is the actual cost of the pattern and it is why the rule above says to route the classes that dominate the bill and leave the rest.
• Failure modes diversify. Different models refuse different things, truncate differently, and disagree about JSON. Everything in normalising errors across vendors and keeping the call site provider-agnostic becomes load-bearing the moment you have more than one route.
• Routes rot silently. A route chosen eighteen months ago against models that have both been superseded is not obviously wrong, and nothing will tell you. Put a review date on the routing table.