Pinning an API Version in Tests So a Provider Update Cannot Break You Silently

Pinning is three separate things that get talked about as one: the API contract, the model weights, and your client library. Only one of them has a version header, and the other two cause most of the breakages.

What can actually be pinned

• A dated API version, where the provider has one. The Claude API requires an anthropic-version header on every request; 2023-06-01 is the value that appears throughout Anthropic’s own documentation examples. The versioning policy attached to it is the part that matters for testing: Anthropic notes on its errors page that the values inside its error objects may expand and that new type values may appear over time. So a pinned version protects you from breaking changes, not from additive ones, and code that switches exhaustively over a set of error types must have a default branch.
• A dated model snapshot. A model alias points at whatever the provider currently considers current, and it moves. A dated snapshot id does not. This is the pin that most affects output, and it is the one most often left unset because the alias is shorter and reads better in a config file.
• The client library. A lockfile pins it. What a lockfile does not pin is the library’s behaviour when you do update it, which is the subject of the snapshot below.
• Beta opt-ins. Where a provider gates a feature behind a header, that header is part of your request contract, and a feature graduating out of beta can change what the header does. Treat it exactly like the version.

Assert it on the wire

Setting a version in a config file proves nothing; the interesting question is whether it survives the client, the retry wrapper, any proxy you route through, and whatever helper somebody wrote last month that constructs its own client. So assert on the outgoing request, captured by an interceptor, in a test that runs against every call site rather than against one.

import { http, HttpResponse } from "msw";
import { setupServer } from "msw/node";
import { afterAll, afterEach, beforeAll, expect, it } from "vitest";
import { PINNED_API_VERSION, PINNED_MODEL } from "../src/config";
import { summarise, extract, classify } from "../src/calls";

const seen: Request[] = [];
const server = setupServer(
  http.post("https://api.anthropic.com/v1/messages", ({ request }) => {
    seen.push(request.clone());
    return HttpResponse.json({ type: "message", content: [{ type: "text", text: "ok" }] });
  }),
);

beforeAll(() => server.listen());
afterEach(() => { seen.length = 0; server.resetHandlers(); });
afterAll(() => server.close());

it.each([summarise, extract, classify])("%o pins version and model", async (call) => {
  await call({ input: "x" });

  expect(seen).toHaveLength(1);
  expect(seen[0].headers.get("anthropic-version")).toBe(PINNED_API_VERSION);

  const body = await seen[0].json();
  expect(body.model).toBe(PINNED_MODEL);
  expect(body.model).toMatch(/-\d{8}$|^claude-[a-z]+-\d/);  // a snapshot, not a bare alias
});

The list of call sites is the load-bearing part. A single test on a single wrapper passes forever while the third helper somebody added quietly constructs a default client with no pin at all.

When there is no version to pin

Several providers date nothing. The API evolves additively, features arrive behind flags, and there is no header to send. In that case the pin has to be reconstructed from the pieces you do control, and the test changes shape accordingly:

• Assert the model id is a dated snapshot rather than an alias, with a regular expression, so that switching to an alias fails a test rather than passing review.
• Assert the exact set of parameters you send. A parameter you never set is a parameter whose default belongs to the provider, and a default can move. Sending temperature explicitly is a pin; omitting it is a subscription to whatever the default becomes.
• Assert on the response envelope’s required fields rather than on its exact key set, so an additive change does not fail the suite while a removal does. This distinction is the whole practical content of a versioning policy, and you have to encode it yourself when the provider has not.

There is one more thing to pin that has no header anywhere: the tokenizer. A dated model snapshot fixes it implicitly, but a pipeline that counts tokens with a separate library has pinned the model and left the counter free to move, and the two can disagree after a routine dependency update. Assert a known string produces a known count, with the model id named in the test, and the pair stays consistent.

The request body snapshot

The best single test for an SDK upgrade is a snapshot of the serialised request. Build a request through your normal code path, capture the exact JSON the client would send, and store it. When you bump the SDK, the diff on that snapshot is a precise, readable statement of what the new version changed: a renamed field, a new default, a parameter that moved from top level into a nested object, an encoding change in how tool schemas are serialised.

Two rules make this snapshot useful rather than noisy. Normalise anything genuinely nondeterministic — idempotency keys, timestamps, request ids — before snapshotting, or the test fails on every run for no reason and gets deleted. And keep the snapshot minimal: one per distinct call shape, not one per test case, because twenty near-identical snapshots produce twenty near-identical diffs and nobody reads the twentieth.

What to do when it changes

A failing pin test is information, not an incident, and the response should be routine. Read the diff, decide whether the change is additive or behavioural, and update the snapshot in a commit that does nothing else, so that the change is visible in history rather than buried in a feature branch. If the change is behavioural — a default moved, a field is now serialised differently — that is the trigger for a behavioural run, not just an updated file.

The pins themselves need a review cadence, because a pin held too long is its own risk: the version you froze eventually stops being supported, and you upgrade three years of changes at once. Put a calendar reminder against the version constants, roll them deliberately, and route the roll through a canary release rather than a deploy. Pinning is what makes a silent model update impossible; it is not what makes an update unnecessary.

Finally, keep the pin values in one module that everything imports, rather than as literals scattered through call sites. The test above is only meaningful because it compares the wire against the same constant production reads; if the test hard-codes the version string separately, it will pass while the application sends something else, which is the precise failure mode a pin test exists to rule out. One constant, one import, one assertion per call site — and a regression run behind it for the changes a header cannot describe.