What a Contract Test Catches That a Mocked Unit Test Misses

A mocked test proves your code behaves correctly given a response. It cannot prove the response is one the provider still sends, because the response came from you.

The gap between a fixture and a service

Every mock is a claim about somebody else’s behaviour, written by you, verified by nobody. That claim was true when it was written. It is checked into your repository, it is never re-verified, and it will still be sitting there passing on the day the provider changes the field it describes.

This is not an argument against mocks. Mocking is the right tool for the thing it is for: exercising your branches deterministically and fast. What matters is being precise about the question each layer answers, because the failure that hurts is not a missing test — it is a team believing a green suite answered a question it never asked.

The compact way to say it: a mocked test asks “does my code handle this shape correctly?” and a contract test asks “is this still the shape?”. Both are necessary. Neither substitutes. A repository with only the first has no mechanism by which it could ever learn it is wrong.

Which layer answers which question

• Unit tests with mocks. Your parsing, your retry logic, your accumulator, your error branches. Fast, deterministic, run on every commit. Owns: does my code do the right thing given input X? See testing without the model.
• Contract tests against the live API. Shape, types, enum values, framing, status codes. Slower, needs credentials, runs on a schedule. Owns: is input X still what arrives?
• Evaluations. Whether the model’s answers are good. Non-deterministic by nature and scored rather than asserted. Owns a question neither of the others touches; see evaluation frameworks.
• Integration and end-to-end tests. Whether the pieces are wired together. Overlaps with contract tests and is frequently confused with them, but it exercises your composition rather than the provider’s promises — an end-to-end test can pass while relying on a stubbed provider.

The confusion that causes real damage is between the second and the fourth. “We have integration tests” is often said to mean the API contract is covered, when the integration tests run against a stub. Ask which tests make a real outbound call; the answer is frequently none.

What a mock structurally cannot catch

These are not gaps you can close by writing better mocks. They are things a fixture is the wrong kind of object to represent.

• Fixture drift. The provider changed and your fixture did not. This is the whole category and it is unfixable from inside the mock, because the mock has no channel to the truth.
• Client-library defaults. The SDK sends something you did not write. The base64 embedding default is the canonical example: the SDK sets encoding_format for you, your mock never sees the request, and you discover it against a provider that handles only float.
• Transport behaviour. Content encoding, chunked transfer, HTTP/2 versus HTTP/1.1, connection reuse, proxy buffering, TLS. A mock at the client-object level bypasses the network stack entirely, so a proxy that buffers your event stream is invisible to every test you have.
• Authentication and headers. Whether your key format is accepted, whether an organisation header is required, whether an expired key returns 401 or 403. Your mock returns what you told it to.
• Rate limits and their headers. The actual limit, the actual header names, the actual behaviour when you cross it. Backoff logic tested only against a synthetic 429 has never met the real one; the general treatment is in rate limits explained.
• Timing. An instant mock response hides every timeout bug you have. Streaming makes this acute: the difference between a proxied stream and a buffered one is entirely temporal and entirely invisible to a fixture.
• Error body shapes. Providers return several different error envelopes depending on where the request died — the model, the router, a CDN in front of both. Nobody mocks the CDN’s HTML error page, and it is the one that breaks the JSON parser.

The cassette that pretends to be a contract test

Record-and-replay libraries sit between the two layers and are often mistaken for the second one. You record real traffic once, commit the recording, and every subsequent run replays it. The recording came from the real provider, which feels like it settles the question.

It settles it for one day. From the moment the cassette is committed it is a fixture with better provenance, and it ages exactly like a hand-written mock — slightly more slowly, because it started accurate. This is worth stating plainly because the recorded-once cassette is the single most common thing teams point at when asked whether their contract is verified.

Cassettes are genuinely valuable in their proper role: making expensive, non-deterministic calls cheap and repeatable inside your unit layer, with a recording that is more realistic than anything you would type by hand. The rule that keeps them honest is that a cassette must be re-recordable on demand and re-recorded on a schedule, and the re-record must be diffed rather than blindly accepted. A cassette refresh that quietly overwrites a changed shape has destroyed the only evidence that anything changed.

What this looks like in a repository

Three directories, three schedules, three owners of three questions. Unit tests with mocks run on every commit and must be fast enough that nobody thinks about them. The contract suite lives separately, needs credentials, and runs on a timer rather than on the merge button — the reasoning for that separation is in catching a provider’s breaking change. Evaluations run when the prompt, the model or the retrieval changes, and are scored rather than passed.

One shared artefact ties the first two together and is worth the effort: define the response schema once, and use it both to validate your fixtures and to validate live responses. A fixture that no longer satisfies the schema fails immediately, which closes the most common drift path — somebody hand-edits a mock to make a test pass and invents a shape the provider never sends. If the schema is the same object in both places, that edit cannot survive.

The last thing worth writing down is the failure protocol, because the two layers mean different things when they go red. A red unit test is your bug and blocks the merge. A red contract test is somebody else’s change and blocks nothing automatically — it opens an investigation into whether your code needs to adapt. Teams that treat them identically end up either ignoring provider breakages or blocking deploys on a vendor’s outage, and both of those are worse than having no suite at all, because they cost trust in the one you have.