Routing Between a Reasoning Model and a Fast Model

Most workloads are a mixture of trivial requests and a small tail of genuinely hard ones. Picking one model for the mixture means either paying reasoning prices for the trivia or failing the tail. Routing is how you stop choosing.

Why per-request beats per-product

Take a support assistant. Eighty per cent of the traffic is “where is my order”, “how do I reset my password”, “cancel my subscription” — retrieval and templating. Fifteen per cent needs a document read carefully. Five per cent is a billing dispute where four rules interact and the wrong answer costs a refund and a complaint.

If a reasoning call costs roughly five times a fast call for this traffic shape — the multiplier derived in test-time compute — then routing perfectly gives you 0.8 + 0.15 + 0.05 × 5 = 1.2 units against 5 units for sending everything to the reasoning model. A little over four times cheaper, with the hard five per cent handled exactly as well. That gap is the entire argument, and it is large enough to survive a fairly bad router.

Escalation: the one to build first

The most reliable router does not predict difficulty at all. It tries the cheap model, checks the result, and escalates when the check fails. This works because verification is very often easier than generation — and because the check is grounded in the actual output rather than in a guess about the input.

async function answer(task) {
  const fast = await call(FAST_MODEL, task.prompt);

  const check = verify(task, fast);        // your checker, see below
  if (check.ok) return { result: fast, tier: "fast" };

  metrics.increment("escalation", { reason: check.reason });

  const slow = await call(REASONING_MODEL, task.prompt, {
    reasoning: { effort: "medium" },
  });
  return { result: slow, tier: "reasoning", firstAttempt: fast };
}

// verify() must be cheap and must not be another LLM call if you can
// help it. In rough order of preference:
//   - run the tests / execute the code
//   - validate against a JSON schema or a solver
//   - check arithmetic by recomputing it
//   - check the answer is entailed by the retrieved documents
//   - only then, a small model as a judge

The cost of escalation is the wasted first attempt, which is small: if the fast call is a fifth of the reasoning call, escalating everything would cost 1.2 times sending everything to the reasoning model directly. So the strategy is nearly free even in the worst case, and in the expected case it is the four-fold saving above. That asymmetry is why this is the default recommendation.

It has one hard requirement: a checker. If you cannot write verify() for your task, escalation is not available to you and you need the classifier instead.

A classifier that decides up front

When the answer cannot be checked, or when the latency of a failed first attempt is unacceptable, decide from the input. Start with features rather than with a model — cheap, inspectable, and often sufficient.

function needsReasoning(task) {
  let score = 0;

  // Structural signals about the work, not about the wording.
  if (task.constraints.length >= 3)          score += 2;
  if (task.requiresArithmetic)               score += 2;
  if (task.documents.length > 1)             score += 1;
  if (task.type === "extract")               score -= 3;
  if (task.type === "classify")              score -= 3;
  if (task.expectedOutputTokens < 50)        score -= 1;

  // Business signals: what a wrong answer costs.
  if (task.value > 500)                      score += 2;
  if (task.userTier === "enterprise")        score += 1;
  if (task.latencyBudgetMs < 3000)           score -= 5;   // hard veto

  return score >= 2;
}

Two design choices in there are deliberate. The latency veto is a subtraction large enough to override everything else, because no amount of difficulty makes a reasoning model fit inside three seconds. And the value features are business inputs, not difficulty inputs — routing should escalate a hard question and also an ordinary question where being wrong is expensive. Those are different reasons and both are legitimate.

A learned classifier is the natural next step once you have logs, and the training data is free: every escalation your rules produced, labelled by whether the reasoning model actually did better. Keep the model small and keep it off the critical path — if your router adds 300 ms to every fast request, it has eaten a meaningful part of what it saved.

The break-even arithmetic

Whether a router pays depends on four numbers you can obtain: the fast cost c_f, the reasoning cost c_r, the fraction p of traffic that genuinely needs reasoning, and the router accuracy. With perfect routing the expected cost is (1 − p)·c_f + p·c_r. A false negative — routing a hard request to the fast model — costs you an error rather than money, and a false positive costs c_r − c_f.

Which means the sensible bias is asymmetric and depends on your error budget, not on your instincts. If a wrong answer costs more than c_r − c_f — which for most business tasks it does, by orders of magnitude — you should tune the router to over-escalate. Set the threshold to catch the tail, accept the false positives, and revisit only when the bill says to.

Resist adding tiers. A three- or four-stage cascade looks like it should capture more of the saving, and in practice each extra stage adds a decision that can be wrong, a failed attempt you pay for, and latency on every request that ends up at the bottom. Two tiers capture most of the available gain because the traffic distribution is bimodal to begin with — a large mass of trivial requests and a small tail of hard ones, with relatively little in between. Add a third only when you can point at that middle mass in your own logs.

Operating it

• Log the decision and the counterfactual. Sample a small percentage of fast-routed requests, run them through the reasoning model too, and compare. Without this you never learn that your p was wrong.
• Alert on the escalation rate, not on cost. A rate that jumps means either your traffic changed or the fast model regressed, and both are things you want to hear about on the day.
• Keep the routing decision in the response metadata. When someone reports a bad answer, the first question is which tier served it, and reconstructing that later is miserable.
• Re-tune after any model change. A cheaper model getting better is the most common reason a router is quietly escalating traffic that no longer needs it.

One organisational note. A router is a piece of business logic disguised as infrastructure — its thresholds encode what your company thinks a wrong answer costs. Keep those thresholds in configuration with a comment explaining the reasoning, not buried in a scoring function, because the person who needs to change them in six months will be reacting to a cost review or an incident rather than reading the code.