Testing for Default Parameter Changes After a Major SDK Version Bump

You upgraded a provider SDK across a major version. Your code did not change, the tests pass, and outputs are subtly different — longer, or less deterministic, or no longer valid JSON. A default moved, and nothing you own recorded what it used to be.

Same code, different behaviour

SDK defaults are invisible by construction. You do not set max_tokens, so the SDK or the API supplies one; you do not set a timeout, so the SDK supplies one; you do not set a retry count, a sampling parameter, an API version header, a base URL or a streaming flag, and each of those has a value your code never mentions. A major version is allowed to change any of them, and changelogs record the ones the maintainers considered notable.

The candidates that most often change behaviour rather than just performance:

• Output token cap. The single most consequential. Raising it costs money on verbose responses; lowering it truncates them, which surfaces as invalid JSON rather than as an error.
• Timeout and retry policy. A default timeout that drops from ten minutes to one turns long generations into failures; a retry default that rises turns one slow request into three and multiplies your spend on the exact requests that were already expensive.
• Sampling parameters. Any change to a default temperature or top-p changes the output distribution across your whole application at once.
• Version and beta headers. These select behaviour server-side, so a header the SDK now sends by default can change responses without a single byte of the body changing.
• Serialisation details. Whether an absent field is omitted or sent as null, whether content is a string or an array of blocks. These break server-side validation and schema handling.

Capture the request, not the response

The instinct is to snapshot the model’s output and diff it. That does not work: the output is non-deterministic, so the snapshot is either always stale or pinned at temperature zero and still fragile against a server-side model update. It also answers the wrong question, because it cannot distinguish an SDK change from a model change.

The request is deterministic. Given the same code and the same arguments, the bytes your process puts on the wire are a pure function of the SDK, and that is exactly the thing that changed. Intercept at the HTTP layer — below the SDK, above the network — and record the method, URL, headers and body.

// request-snapshot.test.ts
import { setupServer } from "msw/node";
import { http, HttpResponse } from "msw";
import { beforeAll, afterAll, it, expect } from "vitest";
import { askModel } from "../src/ask";

let captured;

const server = setupServer(
  http.post("https://api.example.com/v1/*", async ({ request }) => {
    captured = {
      path: new URL(request.url).pathname,
      headers: Object.fromEntries(request.headers.entries()),
      body: await request.json(),
    };
    return HttpResponse.json({
      choices: [{ message: { content: "{}" }, finish_reason: "stop" }],
      usage: { prompt_tokens: 10, completion_tokens: 2 },
    });
  }),
);

beforeAll(() => server.listen({ onUnhandledRequest: "error" }));
afterAll(() => server.close());

it("sends the request we think it sends", async () => {
  await askModel({ question: "what is the status of order 88421?" });
  expect(normalise(captured)).toMatchSnapshot();
});

The Python equivalent registers a handler with a request-mocking library such as responses or respx and reads the recorded request off the mock; the mechanism is identical and only the recording API differs. Whichever you use, configure it to fail on an unmatched request, so an SDK that starts calling a new endpoint — a token-counting call, a capability probe — is a test failure and not a silent extra request in production.

The snapshot and what goes in it

Include the path, the body, and a filtered set of headers. Two things people leave out and should not: the API version or beta headers, because those select server behaviour, and the user-agent, because it is where the SDK version is recorded — which makes the snapshot diff self-documenting about what changed.

Two things must not go in: the authorization header, and anything derived from a real credential. A committed snapshot is a committed file; redact those in the normaliser before the assertion, not after, and assert in a separate test that the authorization header was present and non-empty so the redaction cannot hide a missing key.

Normalising the volatile parts

A raw capture changes on every run and the snapshot becomes noise everyone re-approves without reading, which is worse than not having it. Normalise before snapshotting.

1. Sort object keys, so a serialisation reordering is not a diff.
2. Replace values that are volatile by nature — idempotency keys, trace ids, timestamps, the version segment of the user-agent — with stable placeholders. Keep the field names: their disappearance is exactly what you want to see.
3. Redact secrets, then assert separately that they were present.
4. Leave every default alone. It is tempting to strip fields you did not set; those are the entire point of the snapshot.

Running it across the upgrade

The procedure is three commands and it is the whole payoff.

1. On the current version, run the suite so the snapshots are written and committed. Do this before the upgrade branch exists.
2. Bump the SDK and run the suite again with snapshot updating off. Any default that moved is now a diff with a field name in it, and you can decide per field whether to accept it or to pin the old value explicitly in your call.
3. Pin what matters. For every field whose default you depend on, set it explicitly in your code rather than accepting the new snapshot. A default you have written down is no longer a default. This is the durable fix; the snapshot only tells you which ones to write down.
4. Keep the snapshots in the repository afterwards. They catch the second upgrade too, and they catch a transitive bump of the SDK by a framework you did not upgrade deliberately — which is how most people meet this bug.

Cover more than one call shape: a plain completion, a tool-calling request, a structured-output request, a streaming request, and a multimodal one if you send images. Defaults differ by path, and the serialisation changes in particular tend to show up on the compound shapes first. The multimodal fixture for that is built in testing an image prompt without a real image.