Testing That a Routing Rule Sends a Request to the Right Model

Routing bugs are quiet. Nothing errors; requests just go to a model that costs four times as much, or to one that cannot call tools. The test that catches it is a table, and it never opens a socket.

Make the decision a pure function

Most routing code is untestable for one reason: the decision and the call live in the same function. It reads the request, checks a flag, maybe consults a feature store, then awaits a completion. To test the decision you have to mock the provider, and now the test is about the mock.

Split it. A function that takes everything the decision depends on and returns a target — no I/O, no clock, no environment reads inside. Anything dynamic becomes a parameter.

// routing.ts
export type RouteInput = {
  task: "chat" | "summarise" | "classify" | "code";
  tokensIn: number;
  needsTools: boolean;
  tier: "free" | "pro" | "enterprise";
  region: "eu" | "us";
};

export type Target = { provider: string; model: string; reason: string };

export function selectModel(req: RouteInput, cfg: RoutingConfig): Target {
  for (const rule of cfg.rules) {
    if (matches(rule.when, req)) {
      return { provider: rule.provider, model: rule.model, reason: rule.id };
    }
  }
  return { ...cfg.default, reason: "default" };
}

The reason field is not decoration. Without it a test can only assert which model was chosen, so two rules that happen to name the same model are indistinguishable — and when one of them is deleted, the test still passes. Returning the rule id that fired makes the assertion specific to the decision, and it gives you the same string to put in a log line so a production route can be explained after the fact.

The case table

Each row is a condition somebody actually cares about and the target they expect. Vitest’s it.each and pytest’s @pytest.mark.parametrize both give you one named failure per row, which matters when eleven of twelve pass.

import { describe, it, expect } from "vitest";
import { selectModel } from "./routing";
import { config } from "./routing.config";

const base = {
  task: "chat", tokensIn: 500, needsTools: false, tier: "pro", region: "us",
} as const;

describe("selectModel", () => {
  it.each([
    ["short chat on pro", base, "big-chat"],
    ["classify is cheap", { ...base, task: "classify" }, "small-fast"],
    ["long input escalates", { ...base, tokensIn: 180_000 }, "long-context"],
    ["tools force a tool-capable model", { ...base, needsTools: true }, "big-chat"],
    ["eu stays in eu", { ...base, region: "eu" }, "eu-resident"],
    ["free tier is capped", { ...base, tier: "free" }, "small-fast"],
  ])("%s", (_name, req, expectedRuleId) => {
    expect(selectModel(req, config).reason).toBe(expectedRuleId);
  });
});

One more property belongs in this table and is usually discovered later: the decision must not depend on anything the caller cannot see. If selectModel reads an environment variable, a feature flag client or the clock, the same request routes differently on two machines and no table can express it. Pass those in as fields on the config or the request, and the tests you already have cover the flagged case for free — a row with the flag on and a row with it off.

Building each case by spreading one base request is deliberate. It makes each row state exactly one difference, so a failure tells you which condition is mis-handled rather than which of six fields might be. It also means a new field added to RouteInput does not require editing twelve literals.

Precedence is the part that breaks

A single-condition table proves very little, because real routing configs are ordered and the interesting failures are conflicts. A request that is long and needs tools matches two rules. A free user in the EU matches two. The config resolves these by order, and the order is invisible in the rule definitions.

So the table needs conflict rows, and they should be written from the policy rather than from the code:

• Residency beats cost. An EU free-tier request goes to the EU-resident model even though the cheap model is cheaper. Getting this backwards is a compliance incident, not a bug.
• Capability beats cost. A request needing tools never lands on a model that cannot call them, whatever the tier.
• Hard limits beat everything. A 180k-token input on the free tier does not silently route to a 32k model — it either escalates or is rejected with a message, and the test asserts which.

Write these three as their own assertions with names that state the policy. When someone reorders the config next quarter, the failure message is the reason the order was that way.

Dead rules and unrouted requests

Two structural properties are worth asserting once, over the whole config, and they catch things no individual case does.

First, no dead rules. Run every case in the table, collect the set of reason values that fired, and assert it covers every rule id in the config. A rule that never fires in any case is either untested or shadowed by an earlier rule that matches a superset of its condition — and shadowing is the single most common routing config bug, because it looks correct when you read the file top to bottom.

it("has no unreachable rules", () => {
  const fired = new Set(cases.map(([, req]) => selectModel(req, config).reason));
  const declared = config.rules.map((r) => r.id);
  expect(declared.filter((id) => !fired.has(id))).toEqual([]);
});

Second, no unrouted requests. If your rule predicates are anything more than equality checks, generate the cross-product of the enum-valued fields with a couple of numeric boundaries, and assert every combination returns a target whose model appears in your model catalogue. It is a few hundred cases and runs in milliseconds. This is also the test that catches a typo in a model identifier, which otherwise surfaces as a 404 from the provider at three in the morning.

One test for the dispatch, and no more

Everything above is offline. You still want exactly one test that the chosen target is the one actually called, or the decision is provably right and provably ignored. Mock at the HTTP boundary, make one request, and assert the outbound URL and body carry the model you expect — then stop. Do not re-test the table through the network layer.

import { setupServer } from "msw/node";
import { http, HttpResponse } from "msw";

const seen: string[] = [];
const server = setupServer(
  http.post("https://api.example.com/v1/chat/completions", async ({ request }) => {
    const body = (await request.json()) as { model: string };
    seen.push(body.model);
    return HttpResponse.json({ choices: [{ message: { content: "ok" }, finish_reason: "stop" }] });
  }),
);

The setupServer helper comes from msw/node and the request handlers from msw; the Mock Service Worker documentation covers the lifecycle calls you need around it. Starting the server with the unhandled-request behaviour set to error is worth doing here, so a route that escapes to a provider you did not expect fails the test rather than reaching the internet.

Related routing behaviour is tested separately: what happens when the chosen model errors and keeping a user on one side of a split are different functions with different invariants, and folding them into the routing table makes all three harder to read.