Skip to content
Multigrid

Contract and streaming tests

Proving a provider's API is shaped the way your code assumes, including the streaming path where most of the surprises live.

A contract test asks one question: is the thing on the other end of this URL still shaped the way my parser assumes? It is not an evaluation. It says nothing about whether the answer was good, and it must not try — the moment a test asserts on a sentence the model produced, it starts failing for reasons that are nobody’s fault and gets deleted within a month. What it can assert on is everything around the sentence: the envelope, the field names, the types, the enum values, the order of the streamed events, the shape of a tool call, the presence of a token count. Those are promises, and promises can be broken.

The streaming path is where most of them break, because it is the part that is least often specified and most often reimplemented. Two endpoints can agree perfectly on a buffered response and disagree about whether the first chunk carries a role, whether an empty choices array is legal, whether usage arrives at all, and whether the stream ends with a sentinel. These pages work through the assertions that catch each of those, and through the harness that lets you point the same suite at a provider you are considering, a provider you already run, and a gateway of your own.

Contract Testing an OpenAI-Compatible API Before You Switch Providers

The suite to run against a candidate endpoint before any production traffic reaches it, and the specific places compatible implementations stop agreeing.

11 min read

Writing a Contract Test Suite for Your Own LLM Gateway

The assertions that belong to the proxy layer rather than to the model behind it, tested against a stub upstream so they run in milliseconds.

11 min read

Contract Tests for the /v1/chat/completions Response Shape

Field-by-field assertions on the completion envelope, including the ones that matter for billing and truncation.

10 min read

Contract Tests for /v1/embeddings Dimension and Format

Assertions that catch a dimension or encoding change before it quietly poisons a vector index.

10 min read

Using Pact for Consumer-Driven Contract Tests Against an LLM API

Where Pact's consumer-provider model fits an LLM API, where it stops, and how to write the pact so it does not assert on generated text.

11 min read

Catching a Provider's Breaking API Change Before Production Does

Why the contract suite has to run against the live API on a schedule, and how to do that without making it a deploy-blocking flake.

10 min read

Contract Tests for Streaming Chunk Format

Assertions on the SSE wire format and the chunk envelope, written against the raw byte stream rather than the SDK's assembled result.

11 min read

Contract Tests for Function-Calling Schema Compliance

Assertions that the provider still returns tool calls in the documented shape, including the streamed form where fragments are keyed by index.

11 min read

Running the Same Contract Suite Against Every Provider You Support

Turning the suite into the gate for adding or keeping a provider, including what to do when a provider legitimately cannot do something.

11 min read

What a Contract Test Catches That a Mocked Unit Test Misses

Why a green mocked suite is not evidence that the real API still matches it, and which layer owns which question.

10 min read

Writing an Integration Test for a Streaming Chat Endpoint

A test that drives your own SSE endpoint with a faked upstream, so the proxy layer between the provider and your client is covered rather than assumed.

10 min read

Asserting on a Stream of Chunks Instead of a Final String

Assertion patterns for chunk shape, ordering and delta content that survive a model rewording its answer, and that a final-string comparison cannot express.

10 min read

Testing That a Stream Closes Cleanly on the Happy Path

What a clean close consists of in each provider's frame format, and the four assertions that prove your endpoint produced one.

9 min read

Testing a Client's Behaviour When a Stream Drops Mid-Response

How to truncate a connection on purpose, and the invariant that separates a dropped stream from a short but complete one.

10 min read

Testing Reassembly of Streamed Tokens Into a Full Message

A test that splits the byte stream at every possible offset, so a multi-byte character or a frame boundary landing in the wrong place fails in CI instead of production.

10 min read

Testing Backpressure Handling on a Slow Consumer of a Stream

A deterministic assertion that a fast upstream stops producing while a slow client is behind, instead of buffering the whole response in memory.

10 min read

End-to-End Testing a Chat UI That Renders Tokens as They Arrive

A Playwright test driven by a stream the test itself controls frame by frame, with an assertion that the rendered text only ever grew.

11 min read

Testing Cancellation of an In-Flight Streaming Request

Asserting that an abort reaches the provider connection rather than only the UI, and what happens to your token accounting when it does.

11 min read

Testing That Partial JSON Mid-Stream Doesn't Crash the Parser

Feeding a JSON document to a streaming parser one prefix at a time, and asserting a monotonicity property rather than a sequence of exact values.

11 min read

Unit Testing a Tool-Calling Loop Without Calling the Model

Scripting the model's side of the conversation so the loop's control flow, message construction and termination are tested deterministically.

11 min read

Other topics