Why Your Staging Tests Pass but Production Fails With a Different Model

Everything passes in staging. In production the same input produces a worse answer, a different format, or a tool call that never fires. The most likely explanation is the least interesting one: the two environments are not talking to the same model, and nothing in your configuration says so.

The symptom

It has a recognisable shape. The failure is not intermittent — it is consistent in production and absent in staging. It often involves a capability rather than a quality: structured output that is not enforced, a tool that is never called, a longer prompt that is silently truncated. And redeploying does not help, because the code is not what differs.

The reason this diagnosis is missed is that everyone checks the configuration and the configuration agrees. Both environments say they use the same model. What they mean by that string is the question nobody asks.

Confirm it in one request

Do not reason about it. Every chat completion response carries the model that answered, in response.model, and for OpenAI-shaped APIs that value is frequently more specific than what you sent — you ask for an alias and get back a dated snapshot. Log it in both environments and compare.

const res = await client.chat.completions.create({ model, messages });

logger.info("inference", {
  requested: model,
  answered: res.model,             // often a dated snapshot, not your alias
  fingerprint: (res as any).system_fingerprint ?? null,
  promptVersion,
  temperature,
});

If answered differs between environments, you are done diagnosing and the rest of this page tells you where to look. If it matches, the substitution is not happening at the model level and you should compare the resolved sampling parameters and the rendered system prompt next — the same log line should carry both, which is why it is worth adding permanently rather than temporarily.

Aliases are not model ids

The central mechanism deserves stating plainly. A name without a version suffix is a pointer, and a pointer can differ by account, by region, by date and by the provider’s own rollout schedule. Two environments can both request the same alias, on the same day, and receive different weights — because one account is in a gradual rollout cohort and the other is not.

This makes the alias a perfectly reasonable default for a hobby project and a bad idea for anything with a test suite, because it removes your ability to say what you tested. Pin the dated snapshot in both environments, and treat moving that pin as a deliberate change with its own evaluation — which is the whole subject of models changing underneath you.

Five places the substitution happens

• An environment variable with a default in code. process.env.MODEL ?? "gpt-4o-mini" is the classic: staging sets the variable, production’s deployment manifest was written before the variable existed, and the fallback wins silently. Grep for every default beside a model name; there should be none.
• A cost control nobody documented. Somebody added a cheaper model for production traffic during a spend scare and it was never revisited. This is the specific cause the row for this page names, and it is common precisely because it was a responsible decision at the time.
• A routing rule outside the repository. A gateway or proxy that maps a model name per environment, per tenant or per traffic percentage. Nothing in your codebase shows it, and the only evidence is response.model.
• Account-level differences. The production organisation may not have access to the model staging uses, and some setups fall back rather than fail. A response that arrives from a model you did not name is the tell.
• A stale deployment. The staging deployment is newer. This is boring and it is worth ruling out first by comparing the build identifier in both environments before you go looking for anything cleverer.

Making it impossible to recur

A fix that only corrects today’s value leaves the mechanism intact. Three changes remove it. First, fail fast at boot: if the resolved model id is not in an explicit allowlist for that environment, refuse to start. A process that will not start is a better outcome than one that quietly serves a cheaper model for three weeks.

Second, assert the identity at runtime rather than trusting the request. On the first response after boot, compare response.model against what was requested and emit a metric when the pinned prefix does not match. Third, add a smoke test that runs against production immediately after deploy and asserts the same thing — the model that answered is the model you configured. It is one request and it converts this entire class of failure from a customer report into a deploy that goes red.

The preventative structure for all of it is a configuration where the environments can only differ where you allowed them to. This page is what you do when that structure was not there yet.

One last thing worth checking before you close the incident: whether the staging suite could have caught this at all. Frequently the answer is no, and not because of the model — because the staging tests assert on things that hold for any competent model, and the production failure was a capability difference that only shows up under a specific input. If your suite has no test that fails when the model is swapped for a weaker one, then it is not testing the model, and the cheapest way to fix that is to run the existing suite deliberately against a smaller model once and see how much of it stays green. The part that stays green is the part that was never going to warn you.