Pinning Model Version in Tests to Avoid Silent Drift

The single largest source of drift in a model test suite is not the sampler. It is that the string you put in the model field is an alias, and the thing behind it changed while you were not looking.

What an alias actually resolves to

Providers publish two kinds of model identifier. One is a dated snapshot — a family name with a release date appended, in the shape most vendors use, such as a -2024-08-06 or -20250929 suffix — and it names one fixed set of weights. The other is an alias: the bare family name, or one carrying a -latest suffix, which resolves to whichever snapshot the provider currently considers current.

An alias is the right default for production in some setups and it is almost never right for a test suite, because it means the assertion you wrote in March is being evaluated against a model that arrived in June. When that suite goes red, the diff that broke it is not in your repository, and people will spend a day looking for it there. The library’s page on silent model updates covers the production side of the same problem; the testing side is narrower and easier to fix.

The cost of the alias is not only that the model changes. It is that the change is invisible in every artefact a team normally uses to explain a regression. Nothing in the diff moved, the deploy log is empty, the dependency lockfile is untouched, and the suite that passed on Friday fails on Monday. Because every available piece of evidence points inward, the investigation starts by bisecting your own commits, which cannot find anything because the cause is not there.

One pin, read from configuration

The pin belongs in one file that the whole suite reads, not scattered across call sites and not hard-coded inside a helper. Two properties matter: a single grep finds it, and a live tier can override it without editing test code.

# tests/models.py
import os

# The model under test. Dated snapshot, never an alias.
# Bump this deliberately; see the upgrade procedure in the README.
DEFAULT_MODEL = "<provider-family>-<yyyy-mm-dd>"

MODEL = os.environ.get("TEST_MODEL", DEFAULT_MODEL)

# A separate, explicit pin for the judge, if the suite uses one. Changing the
# judge changes every score in the suite, so it must not ride on the same pin.
JUDGE_MODEL = os.environ.get("JUDGE_MODEL", DEFAULT_MODEL)

Keeping the judge model on its own pin is worth the extra line. An evaluator model and a model under test drift for different reasons and on different schedules, and a single constant makes it impossible to change one without silently changing the other — which invalidates every stored score at once.

Assert the model you were served

Setting the field is not proof that it was honoured. Providers echo the resolved model back on the response, and that echo is the cheapest assertion in the whole suite. Add it once, at the fixture level, and every test inherits it.

# tests/conftest.py
import pytest
from tests.models import MODEL

@pytest.fixture
def complete(client):
    def _complete(messages, **kw):
        resp = client.chat.completions.create(
            model=MODEL, messages=messages, temperature=0, **kw
        )
        # A gateway, a proxy or an alias expansion all show up here.
        assert resp.model == MODEL, f"served {resp.model}, expected {MODEL}"
        return resp
    return _complete

The assertion catches three separate things: an alias you thought you had pinned and had not, a gateway routing you to a fallback model because the primary was unavailable, and a provider expanding a shorthand to a snapshot you did not choose. All three are invisible without it, and each one produces a suite whose results describe a model you are not shipping.

Where the fallback is deliberate — a routing layer that fails over on purpose — the assertion should be a warning in the mocked tier and a hard failure in the live tier, because a failover during an eval run means the numbers came from two different models.

The pin also has to reach the places that are not the test client. An embedding model used to build a retrieval index, a judge model, a small model used for a routing decision inside the pipeline — each of these is a separate model identifier and each can be an alias somebody set once. An embedding model in particular changes the meaning of a stored index rather than just the next response, so an alias there produces a suite whose retrieval results drift without any model call in the test appearing to change at all.

A pin has an expiry date

The uncomfortable half of pinning is that snapshots are retired. Providers publish deprecation dates, and a suite pinned to a retired snapshot fails with a 404-shaped error naming a model that no longer exists — typically on the morning of the retirement, in everyone’s pipeline at once.

• Put the retirement date in a comment next to the pin, taken from the provider’s deprecation page, so it is visible to whoever reads the file.
• Add one scheduled job that calls the pinned model once a week and alerts on failure. It costs a few tokens a week and converts an outage into a warning.
• Distinguish the two failures in the error message. “Model not found” and “assertion failed” are read very differently at 9am.

Treat the retirement as a scheduling problem rather than a surprise. The gap between a snapshot’s deprecation announcement and its shutdown is usually months, and the work of moving is a day — but only if the day is chosen. Left to the deadline it becomes an emergency in which the prompt changes and the model changes at once, and nobody can attribute the resulting differences to either.

Upgrading deliberately

1. Set TEST_MODEL to the new snapshot on a branch and run the full suite without changing anything else. Every failure is now attributable to the model, because nothing else moved.
2. Triage the failures into three piles: the prompt genuinely needs adjusting, the assertion was over-specified and should have been a property, and the new model is worse at this task.
3. Re-record any fixtures that were captured against the old snapshot. A cassette recorded on one model and replayed while the suite claims to test another is a lie the suite tells about itself.
4. Change the default pin in one commit that touches only that file, so the change is greppable in history and revertable on its own.
5. Keep the old pin available as an environment override for one release, so a regression found after the switch can be bisected against the model rather than against your code.