Testing That Upgrading the Provider's SDK Doesn't Change Output Silently

“We bumped the SDK and the answers changed.” Nothing in your prompt moved and the model id is the same, so the change is in the request the SDK built — and that is a thing you can diff.

The symptom

It arrives in one of three shapes. Outputs got shorter or longer with no prompt change. A field that used to be present in the response is now missing or renamed, and your parser throws. Or nothing visibly broke and an evaluation score moved a few points, which is the worst one because it gets attributed to the model.

In all three the useful reframing is the same: the SDK is a request builder and a response parser. Between two versions, the model did not change; the bytes you sent did, or the way you read the bytes back did. Both are deterministic and therefore testable, which is more than can be said for the output itself.

What actually changed

Six candidates cover nearly everything, and knowing the list is half the triage:

• A default parameter appeared or disappeared. A default max_tokens, a default temperature, a default for a structured-output flag. Sending a parameter you previously omitted is a behaviour change even when the value looks harmless.
• A version header changed. The same SDK against a newer server-side API version. This is the one that changes output with an otherwise byte-identical body.
• The default endpoint moved. Major versions have moved clients between API surfaces, which changes the request shape wholesale.
• Serialisation of absent values. Whether an unset optional is omitted or sent as null. Some APIs treat the two differently.
• Retry and timeout defaults. These do not change a single response but change what happens under load, which shows up as latency and cost rather than as text.
• Response parsing. A renamed field, a changed type, a streaming event shape. Your code sees it as missing data.

Snapshot the request

Intercept at the HTTP boundary, serialise the request into a canonical form, and commit it. The fixture returned does not matter — you are not asserting on the answer — so return the same recorded response every time and let the snapshot be entirely about what went out.

import json, responses, pytest

def canonical(request) -> str:
    body = json.loads(request.body)
    headers = {
        k.lower(): v
        for k, v in request.headers.items()
        if k.lower() in {"content-type", "anthropic-version", "openai-beta"}
    }
    return json.dumps(
        {"method": request.method, "url": request.url,
         "headers": headers, "body": body},
        sort_keys=True, indent=2,
    )

@responses.activate
def test_request_shape_is_unchanged(snapshot):
    responses.add(
        responses.POST,
        "https://api.example.com/v1/messages",
        json=RECORDED_RESPONSE,
        status=200,
    )

    app.summarise("the quarterly report")

    assert len(responses.calls) == 1
    assert canonical(responses.calls[0].request) == snapshot

Two details make the difference between a useful snapshot and a noisy one. Sort the keys, or a dictionary ordering change produces a diff with no meaning. And allow-list the headers rather than excluding a few: an allow-list will not accidentally commit an authorisation header, and it will not churn when the SDK adds a telemetry header you do not care about. The equivalent in a JavaScript suite is an interceptor that captures the request body and headers before returning a fixture; the shape of the assertion is identical.

Running the bump

1. Confirm the suite is green on the pinned version, and that the lockfile is actually respected in CI — an install that resolves fresh makes this whole exercise decorative.
2. Bump only the SDK, in its own branch, with nothing else in the diff.
3. Run the request-snapshot suite. Every changed line is a candidate cause, and there are usually two or three, not fifty.
4. Triage them by kind. A header-only diff points at an API version change. A body diff naming a parameter you never set is a default change. A URL diff is an endpoint move.
5. Decide each one explicitly and pin what you want. If the SDK now sends a default you would rather not have, set the parameter yourself so the next bump cannot move it again. Then accept the new snapshot in the same commit as the bump, so the diff and the decision are reviewed together.
6. Only then look at outputs. With the request pinned, any remaining change is server-side and belongs to silent model updates, not to the SDK.

Wire the same suite into automated dependency-update pull requests. A bot that bumps the SDK will otherwise merge on a green unit suite that never inspected a single outbound byte.

What this cannot catch

Be clear about the boundary, because a snapshot suite invites overconfidence. It catches everything the client sends and nothing the server decides. A model updated behind a stable alias, a change in default sampling on the provider’s side, a safety filter tuned last night — all invisible here, and all requiring the periodic behavioural baseline described in production quality regression.

It also will not catch a change in how the SDK parses a response unless you assert on the parsed object too. Add one test per response variant you care about — a tool call, a refusal, a truncated completion, a streaming sequence — that feeds a recorded raw payload through the SDK and asserts on the object your code receives. Those recorded payloads are worth keeping in the repository for exactly this reason (recording provider error fixtures).