Migrating an Internal Model Selection Decision Tree

Somewhere in your codebase is a function that decides which model gets a request. It grew one branch at a time, each branch is a model name typed as a string literal, and the migration means every one of those literals is now wrong. Search-and-replace is the obvious move and it is the wrong one.

The tree you have

It looks approximately like this, whatever language it is written in.

def pick_model(task, text, user):
    if task == "classify":
        return "vendor-a-small"
    if task == "summarize" and len(text) > 400_000:
        return "vendor-a-long-context"
    if task == "extract" and user.tier == "enterprise":
        return "vendor-a-large"          # strict JSON needed
    if task == "chat" and user.locale != "en":
        return "vendor-a-large"          # small one is weak here
    if task == "vision":
        return "vendor-a-vision"
    return "vendor-a-medium"

Three things are entangled in each branch: a business rule (enterprise tier gets the better model), a capability requirement (this path needs strict schema enforcement, or a long window, or image input), and an answer (this specific model id). Only the third one changed. But because all three live on the same line, you cannot change the third without re-deriving the other two from a comment.

Why renaming the leaves does not work

A rename assumes a one-to-one correspondence between the old lineup and the new one, and there almost never is one. The new provider may have four tiers where the old had three. The long-context branch may be unnecessary because every model in the new lineup has a window large enough. The vision branch may collapse because vision is no longer a separate model. And a capability your branch silently depended on — strict schema enforcement on tool arguments, say, or a seed parameter for reproducibility — may exist on a different subset of the lineup than before.

The worst case is a branch that keeps working but for a different reason. If your extract path routed to the large model because it was the only one with strict schema support, and the new large model happens to be the right answer anyway, you have preserved the behaviour and lost the reason. The next person to touch the tree will move that branch for cost reasons and reintroduce a schema bug that nobody can trace.

Turning branches into capability predicates

The rebuild has one rule: a leaf is never a model name. A leaf is a set of requirements, and a resolver turns requirements into a name at startup. Go through each branch and write down what it actually needs.

• Input capacity. How many tokens of context does this path need in the worst case? Compute it, do not guess: system prompt plus tools plus the largest document you accept plus reserved output. See context window versus max tokens for why those are two budgets, not one.
• Output capacity. The largest completion this path is allowed to produce, which caps which models can serve it.
• Modality. Does the path pass images, PDFs or audio? This is a hard filter, not a preference.
• Output discipline. Does it need schema-constrained output, or strict tool-argument validation? Both are capabilities that vary across a lineup.
• Quality floor. The only genuinely subjective one, and the only one that should be expressed as a tier rather than a boolean. Give it a name like tier: "frontier" | "standard" | "fast" and resolve it from a table.
• Cost ceiling. An optional per-request budget, applied after the hard filters.

Building the new tree

The tree now returns a requirements object, and a separate resolver maps requirements to a model. Crucially, the resolver’s capability data comes from the provider rather than from a comment.

# routing.py

@dataclass(frozen=True)
class Need:
    tier: str = "standard"          # frontier | standard | fast
    min_input_tokens: int = 32_000
    min_output_tokens: int = 4_000
    vision: bool = False
    structured_output: bool = False

def needs_for(task, text, user) -> Need:
    if task == "classify":
        return Need(tier="fast", min_input_tokens=8_000, min_output_tokens=64)
    if task == "summarize":
        return Need(min_input_tokens=estimate_tokens(text) + 8_000)
    if task == "extract":
        return Need(
            tier="frontier" if user.tier == "enterprise" else "standard",
            structured_output=True,
        )
    if task == "vision":
        return Need(vision=True)
    return Need()

# --- resolver: built once at startup, not per request ---

class Registry:
    def __init__(self, client, tiers: dict[str, list[str]]):
        self._tiers = tiers                      # tier -> ordered candidate ids
        self._caps = {m.id: m for m in client.models.list()}

    def resolve(self, need: Need) -> str:
        for model_id in self._tiers[need.tier]:
            m = self._caps.get(model_id)
            if m is None:
                continue                          # id retired since the table was written
            c = m.capabilities
            if m.max_input_tokens < need.min_input_tokens:
                continue
            if m.max_tokens < need.min_output_tokens:
                continue
            if need.vision and not c["image_input"]["supported"]:
                continue
            if need.structured_output and not c["structured_outputs"]["supported"]:
                continue
            return model_id
        raise NoEligibleModel(need)

Resolving once at startup rather than per request is deliberate. A capability lookup on the request path adds a network call to every inference, and a provider outage on that endpoint would take down your routing rather than merely your inference. Fetch the capability table at boot, keep it in memory, and refresh it on a timer or on a deliberate signal — with the previous table retained if the refresh fails, so a transient error never empties your registry.

The capability fields above come from Anthropic’s Models API, which returns max_input_tokens, max_tokens and a nested capabilities object per model from GET /v1/models (platform.claude.com, models overview). Other providers expose different shapes; the pattern is the same whichever it is — a small adapter that normalises whatever the provider publishes into your Need vocabulary, so the rest of the resolver never learns a vendor’s field names.

Verifying it before it routes traffic

A routing rewrite is exactly the kind of change that looks correct and is not. Run these four checks before it takes a request.

1. Replay the old tree against the new one. Take a day of production request metadata, run both functions, and diff the chosen model per request. Every difference should be explainable by a deliberate decision. Unexplained differences are bugs.
2. Assert the registry resolves at startup. Call resolve once for every Need your tree can produce, as a startup health check. A NoEligibleModel at boot is infinitely better than one on a Friday evening request.
3. Test the requirement, not the name. Your unit tests should assert needs_for(...).structured_output is True for the extract path, never that it returns a particular model id. A test that pins a model id has to be edited on every migration, which is how test suites end up rubber-stamping the change they were meant to catch.
4. Route a canary slice first. Send a small percentage of live traffic through the new tree with per-path cost and quality metrics broken out, and keep the old tree behind a flag you can flip back within one deploy.