Migrating an Internal Model Router's Config Format

Almost every homegrown router reaches the same failure: adding a provider means editing the routing logic. That happens because two different things were written into one config file, and this is the rewrite that separates them.

Two layers wearing one coat

A router config that grew around one provider usually looks like a list of rules whose conditions and whose actions are both written in that provider’s vocabulary. A rule says “if the request is a summarisation job, use this model string, with max_tokens set to 1024”. Both halves of that sentence are provider-specific: the model string obviously, and max_tokens less obviously, because the same concept is spelled max_completion_tokens on OpenAI’s newer reasoning-capable endpoints and is a required rather than optional field on Anthropic’s Messages API.

There are really two layers here. The decision layer answers “given this request, what kind of model do I want?” — and the honest answer is expressed in facts about the workload: task class, tenant tier, latency budget, cost ceiling, required capabilities. None of those mention a vendor. The binding layer answers “given that description, which endpoint do I call and how do I spell the request?” — and all of that is vendor-specific, all of it changes when a provider renames something, and none of it should be able to change a routing outcome.

When the two are merged, adding a provider forces you to touch rules, and touching rules risks changing where existing traffic goes. That is the actual cost of the merged format, and it is why this migration is worth doing before the one you are actually planning.

A provider-neutral vocabulary

The decision layer needs a fixed set of terms that every provider entry is described in. Keep it small and keep every term falsifiable — a capability flag you cannot test is a lie waiting to happen.

• Capabilities — booleans the adapter can prove: streaming, tool calling, strict JSON schema output, image input, a usable stop-sequence list, prompt or context caching, a seed parameter. Each one corresponds to a real request field, so each one can be verified by a smoke test rather than asserted in YAML.
• Class — a coarse tier (small, standard, reasoning) that rules match on instead of naming models.
• Cost ceiling — input and output price per million tokens, held as data you own and dated, because a rule that says “cheapest model that can do tools” needs a number and must not read it from a comment.
• Limits — context window and maximum output tokens, because a rule that routes a 200k-token request must be able to eliminate candidates that cannot hold it.

The rules then stop mentioning providers entirely. A rule becomes a predicate over the vocabulary plus an ordered preference, which is what you actually meant all along. The general shape of those predicates is covered in the library’s page on routing algorithms; this page is only about the config that feeds them.

The binding table

Everything vendor-shaped moves into one table, one entry per provider-model pair. The entry holds the endpoint, the auth header name, the model string, and — the part people forget — the field mapping and the response paths.

{
  "id": "vendor-a:standard",
  "class": "standard",
  "endpoint": "https://api.vendor-a.example/v1/chat/completions",
  "auth": { "header": "Authorization", "scheme": "Bearer", "secret": "VENDOR_A_KEY" },
  "model": "vendor-a-standard-2026-05",
  "capabilities": ["stream", "tools", "json_schema", "cache"],
  "limits": { "context": 200000, "max_output": 8192 },
  "price_per_mtok": { "input": null, "output": null },
  "request_map": {
    "max_output_tokens": "max_completion_tokens",
    "stop": "stop",
    "system": "messages[0]"
  },
  "response_map": {
    "text": "choices[0].message.content",
    "finish": "choices[0].finish_reason",
    "input_tokens": "usage.prompt_tokens",
    "output_tokens": "usage.completion_tokens",
    "cached_input_tokens": "usage.prompt_tokens_details.cached_tokens"
  }
}

The four mappings that reliably differ, and that a router built against one provider will have hard-coded somewhere:

• The output cap. max_tokens on OpenAI’s original chat completions field and on Anthropic’s Messages API, where it is required; max_completion_tokens on OpenAI’s newer surface, where the older name is deprecated. A router that omits the field entirely because “the default is fine” breaks on the provider that requires it.
• The system prompt. A message with a system role in the array on one side; a top-level system parameter alongside the message array on the other. A mapping that appends it as a message where a top-level field is expected does not error — it just changes behaviour.
• Stop sequences. stop against stop_sequences, with different documented list-length limits on either side. Truncating a list silently to fit is the wrong answer; failing the binding validation at config-load time is the right one.
• The usage object. usage.prompt_tokens and usage.completion_tokens against usage.input_tokens and usage.output_tokens. Cost tracking that reads one path and gets undefined from the other reports a spend of zero, which is the failure mode that does not page anyone.

Terminal reasons are the fifth, and they deserve their own enum rather than a path: finish_reason values and a top-level stop_reason use different vocabularies for overlapping ideas. Map both into your own set at the adapter boundary.

The rewrite, step by step

1. Freeze current behaviour first. Take a sample of real request metadata from your logs — a few thousand rows of the fields the rules read — and record the model each one resolves to today. This file is the only thing that will tell you the rewrite was faithful.
2. Write the vocabulary down as a schema, and validate every provider entry against it at process start. A router that discovers a missing mapping at request time discovers it in production.
3. Move names out of rules mechanically. For each rule that names a model, create a binding entry with that model and replace the rule’s action with the class and capability set the model satisfies. Do this one rule at a time and re-run the frozen sample after each.
4. Add the new provider as data only. One binding entry, no rule change. At this point the new provider is reachable by id and is chosen by nothing.
5. Give it preference in one rule, gated by whatever fraction control you already have. The rule change and the provider addition are now two separate commits, which means they are two separate rollbacks.
6. Delete the old names. Only after the frozen sample still resolves identically and the new entry has served real traffic.

Proving the routing did not move

The check is a replay, not a unit test. Feed the frozen sample through the new resolver and diff the resolved binding id against the recorded one. A faithful rewrite produces a zero-row diff; anything else is a behaviour change you did not intend, and the diff tells you which rule moved.

Do the same for the request body. Build the outbound payload for each sampled request under both the old and new code paths and compare them field by field, ignoring only fields you deliberately renamed. This catches the class of bug where the routing is right and the mapping quietly dropped a stop sequence or a tool definition. It pairs with the library’s routing-rule test rather than replacing it: that one asserts intent, this one asserts that intent did not change during a refactor.

Keep the replay in CI afterwards. The config is now data, and data changes are exactly the changes nobody reviews carefully.