Migrating an Internal Model Zoo Reference Doc Across a Provider Swap

Every team of more than about eight people has a page listing the models it runs, who owns each one, and what it is for. Every one of those pages is wrong. It is wrong because nothing breaks when it is wrong, and a migration is the moment that stops being harmless.

Why it rots, specifically

The model zoo doc answers a different question from the capability matrix, and confusing them is the first mistake. The matrix answers “what can this model do” and is refreshed by probing — that is the capability matrix page. The zoo answers “what are we running, where, and who owns it”. It is an inventory, and inventories rot for a reason capability tables do not: an inventory can only be verified against reality by somebody going and looking.

Three specific decay paths show up in every audit. A service ships a new call site and nobody adds a row. A model is swapped in a config file during an incident and the doc keeps the old string. And an owner leaves, so a row names a team that no longer exists and nobody feels entitled to delete it. None of these produces an error. The doc keeps rendering, and its wrongness surfaces only when somebody plans a migration from it and discovers three call sites nobody knew about, usually mid-cutover.

Generate it from something the code reads

The structural fix is to stop authoring the document. Put the inventory in a machine-readable file that the calling code depends on at runtime or at build time, and render the document from that file. The properties follow immediately: an entry that is missing breaks a service that needs it, an entry that is wrong breaks a deploy, and a stale entry is impossible because there is only one copy.

# models.yaml — the source of truth the code reads, and the doc renders
summarizer-v3:
  provider: acme
  provider_model_id: "<vendor model string>"
  owner: platform-content
  purpose: "Ticket summarisation for the agent console"
  bound_prompts: [prompt/ticket-summary@7]
  eval: eval/ticket-summary-goldens
  fallback: summarizer-v2
  deprecate_after: 2027-02-01
  last_verified: 2026-08-11

The file above is small enough that people will maintain it and rich enough to generate a readable page. The fields that earn their place are the ones somebody looks up under pressure: which prompt version is bound to this entry, which evaluation decides whether a change is acceptable, what it falls back to, and when it must be revisited.

Make the generated page obviously generated — a header saying so, a link to the file, and no edit button. Half the rot in wiki-based inventories comes from someone helpfully correcting the page instead of the source, after which the two disagree and both become untrusted.

The alias is the whole migration

The single most consequential field is the key: an internal alias like summarizer-v3 rather than a vendor model string. If services name the alias and the registry maps alias to provider, a migration is one edit in one file plus a canary. If services name the vendor string directly, the migration is a grep across every repository, a set of pull requests owned by different teams on different schedules, and a document written afterwards by whoever remembers what happened.

This is the difference between a migration that can be rolled back in one commit and one that cannot be rolled back at all. It is also what makes the doc self-maintaining: because the alias is the thing code references, the registry cannot drift from what is deployed without something failing. The related problem of what to do when a provider retires a model out from under an alias is custom aliases after a retirement.

One caution: an alias is a routing decision wearing a name. Resist aliases that encode the vendor (acme-fast) or the tier (cheap-model), because both become lies. Name the alias after the job it does, which is the thing that does not change when you migrate.

What the registry cannot hold

Two things belong in prose alongside the generated table, and they are the two most often missing when a migration starts.

• Why this model and not the cheaper one. Somebody once ran a comparison and chose. Without the reason recorded, the migration re-litigates it from scratch, or worse, silently drops to a cheaper model because nobody remembers the constraint that ruled it out. Two sentences and a link to the evaluation run is enough.
• Who agreed to it, and under what conditions. Some model choices carry commitments — a data residency requirement, a retention constraint, a customer contract clause, an approval from a review board. Those attach to the use, not to the vendor, and they are the constraints most likely to be violated by a migration nobody routed past the same people.

Keep both in the repository next to the registry, not in a ticketing system, because tickets are not read by the person doing the migration two years later. The rest of the surrounding documentation practice — where prompts, evaluations and runbooks live relative to each other — is migrating internal prompt docs.

Representing the dual-run, and ending it

During a migration each entry temporarily has two provider bindings and a traffic split between them. If the registry cannot represent that state, the split lives in a feature flag system, a config map, and a Slack thread, and no document describes the system as it actually is for the several weeks that state persists — which is exactly the period during which somebody will be paged and need to know.

So add the ability to express it: a second provider binding and a weight, on the same entry, with the same generated rendering. The document then tells the truth continuously rather than describing a before state and an after state with a gap in the middle.

It also gives the migration a completion criterion that is checkable by a script rather than declared in a meeting: the migration is finished when no entry in the registry has two bindings. Until then the generated page carries a banner saying so, the on-call runbook keeps its provider-comparison branch, and the old provider’s credentials stay live. When the count reaches zero, all three come out in one change — and the last commit of the migration is a diff against a file, which is a much better artefact to hand the next person than a memory.