What Happens to a Custom Model Alias After the Model Is Retired

A 404 with an error code of model_not_found, on a call that worked yesterday, on a model string nobody edited. That is the cheap version of this bug. The expensive version returns 200.

Two failures, one of which has no error

The loud failure is a 404 whose body carries a code such as model_not_found and a message saying the model does not exist or you do not have access to it. On Azure the equivalent 404 names the deployment rather than the model, which is its own diagnostic trap: the deployment name is a string you invented, so the error contains no evidence about which underlying model version expired. The general treatment of that response is the model not found error; this page is about why an alias produced it.

The quiet failure is the one that costs money. The alias did not break — it moved. Your requests still succeed, still return 200, and now run against a different model version, with different formatting habits, a different refusal surface and a different token bill. Nobody gets paged. Three weeks later someone notices that the summaries got longer. That is the same mechanism as a silent model update, with the difference that here you chose the indirection yourself.

What an alias actually resolves

An alias is a name resolved on the server at request time. There are two kinds and both are in most estates.

• A vendor-maintained floating pointer. A bare family name that resolves to whichever dated snapshot the vendor currently considers default. Convenient, and by design it changes under you.
• A deployment name you created. On Azure and similar surfaces, the string in your code is your own alias for a (model, version) pair held in the deployment’s configuration. Your code is now insulated from the model name entirely, which is the point, and also means the model your code runs is not visible anywhere in your code.

A third case hides in fine-tuning: a tuned model identifier is an alias over a base model. When the base is retired the derivative goes with it, and the inventory that lists your models by name will not tell you that, because the tuned identifier is still perfectly valid-looking.

The audit: requested against resolved

The response already tells you what ran. Chat-style responses carry a model identifier, and for an alias that identifier is the resolved snapshot rather than the string you sent. So log both, on every request, as two separate fields.

# at the call site, not in a wrapper somebody can bypass
resp = client.chat.completions.create(model=REQUESTED, messages=msgs)

log.info("llm_call",
         model_requested=REQUESTED,       # what the code asked for
         model_resolved=resp.model,       # what actually ran
         request_id=resp.id)

One sanity check first: if model_resolved is always exactly equal to model_requested for a name you believe is an alias, you are probably echoing your own input somewhere in the wrapper rather than reading the response. Verify that against a call you know is aliased before trusting the audit.

Then the audit is a single scheduled query: group yesterday’s traffic by the pair, and diff against the previous day.

SELECT model_requested, model_resolved, count(*)
FROM llm_calls
WHERE ts >= current_date - 1
GROUP BY 1, 2;

-- alert when a model_requested that had exactly one model_resolved
-- yesterday has a different or additional one today.

That alert fires on the day the pointer moves. Compare that to the alternative, which is noticing when someone complains about output quality — by which time you have weeks of mixed traffic and no clean before-and-after to reason from. If you keep an eval suite, the alert is also the correct trigger to run it: this is exactly the scenario in catching a regression after a silent model update.

The upgrade policy picks your failure

Where deployments are configurable, the policy you chose decides which of the two failures you get. Microsoft’s documentation for Foundry Models describes the options as: deployments set to opt out of automatic model version upgrades, which require a manual upgrade and stop working when the model is retired; deployments set to upgrade once a new default version becomes available, which move automatically to the new default; and deployments set to upgrade once the current version expires, which update when their current version is retired. See Microsoft Learn on model versioning.

Read that as a choice between failure modes rather than a configuration detail. Opting out buys reproducibility and hands you an outage on a date you must now track yourself. Auto-upgrading buys continuity and hands you silent behaviour change. There is no third option in which nothing happens, and the common mistake is choosing one per deployment by accident and discovering the mixture during an incident.

The defensible position is: pin, and pair the pin with a calendar entry for the retirement date and the resolved-identifier alert. Pinning alone converts a slow problem into a hard deadline you have not written down.

The fix, in order

1. Inventory every model string in the estate. Application code, config files, environment variables, infrastructure templates, the prompt registry, notebooks, and anything on a cron. The one that fails is always the quarterly job.
2. Resolve each string against the provider’s models listing to find out whether it is an alias, and against the provider’s published retirement table to find out how long it has.
3. Add the requested/resolved logging and let it run for a week before changing anything, so you have a baseline of what each alias currently resolves to.
4. Pin each alias to a dated snapshot, one service at a time, and record the pin somewhere the retirement calendar reads — a single constants module is the usual answer, and it is the same module the documentation should be generated from.
5. Schedule the un-pin. A pin with no scheduled review is how you arrive at the retirement date with no plan; a feature-flagged model version, as in flagging a model version through deprecation, makes the eventual move a config change rather than a deploy.