Splitting Tests Into a Mocked Tier and a Paid Live Tier

Every team that calls a model from tests arrives at two tiers eventually. Arriving on purpose is cheaper, and the design decision that matters is not which tests go where — it is what happens when the live tier goes red.

What each tier can prove

The two tiers are not fast tests and slow tests. They answer different questions, and neither question is optional.

The mocked tier proves that your code is correct with respect to a response. Given this JSON, does the parser produce the right object; given a 429, does the retry back off; given a truncation, does the handler notice. It is deterministic, free, instant, and it tests your beliefs about the provider rather than the provider.

The live tier proves that your beliefs about the provider are still true. That the pinned model still exists, that the schema still validates, that the tool still fires, that the response shape has not gained a wrapper. It is slow, costs money, is flaky for reasons outside your control, and is the only thing that can catch a vendor change.

Running only the mocked tier means a provider change reaches production before it reaches a test. Running only the live tier means a vendor outage blocks every merge in the repository, the suite is slow enough that people stop running it locally, and the bill grows with your commit rate. The failure modes are not symmetric but both are real.

The mocked tier

• Runs on every commit and blocks the merge. This is the gate. It must be fast enough that nobody minds and reliable enough that a red build always means a real defect.
• No network at all. Enforced, not agreed — block sockets, unset the credential, refuse new recordings. An accidental live call must fail rather than succeed.
• Responses come from recordings or hand-written doubles. Recordings for the happy paths, because they are real; hand-written for the awkward ones, because you cannot reliably provoke a provider into returning a malformed tool call on demand. Both belong in the repository.
• Includes the ugly cases. Truncation, refusal, invalid JSON, an unknown tool name, a 429 with a retry-after header, a 500, a stream that ends mid-object. These are where production incidents come from and they are all free to test here.

The live tier

• Small and deliberate. Ten to thirty cases, chosen because each one can only be answered by the real provider. If a case would pass against a recording, it belongs in the other tier.
• Scheduled, plus a manual trigger. Nightly is usually right, with a label or workflow dispatch so a risky change can request it. Not on every commit, which is what makes the cost arithmetic in the suite estimate manageable.
• Asserts contracts, not quality. The model resolves, the response validates, the tool fires, the fields are present. Score tracking is a third thing — an eval, reported as numbers over time, with a golden dataset behind it — and conflating it with a pass/fail tier is how teams end up gating merges on a rubric score.
• Capped. A hard spend cap on the job, because this is the tier that can spend.
• Records what it saw. Model string, fingerprint where available, token counts and latency, stored per run. The live tier is also your monitoring of the provider, and that only works if it keeps a history.

Expressing the split so it cannot rot

The split has to be a property of the test, selectable from the command line, with a default that excludes the expensive tier. Directory conventions rot because people put files where the other files are; a marker rots much more slowly because running the wrong set requires typing something.

# pyproject.toml or pytest.ini
[tool.pytest.ini_options]
markers = [
  "live: calls a real provider; costs money; excluded by default",
]
addopts = "-m 'not live'"

# tests/test_contracts.py
import pytest

@pytest.mark.live
def test_pinned_model_still_resolves(complete):
    resp = complete([{"role": "user", "content": "ping"}], max_tokens=5)
    assert resp.model == MODEL

The default in addopts is the load-bearing part: a developer who runs the bare test command gets the free tier, and CI has to ask for the other one explicitly. In a JavaScript runner the equivalent is a conditional skip driven by an environment variable, or a separate project entry in the config with its own include pattern — check your runner’s current configuration format, since this is an area that has changed.

One more guard worth adding: a test in the mocked tier that fails if any unmarked test makes a network call. With sockets blocked that happens automatically, which is the argument for blocking them rather than trusting the marker.

Where a live failure goes

This is the design decision that determines whether the arrangement survives six months. A live-tier failure usually means the provider changed something, and the person whose pull request is open did not do it. Route that failure onto their pull request and one of two things happens: they wait for someone else to fix a vendor problem, or the check gets marked non-required and quietly ignored.

• Never a required check on a pull request. The live tier runs on a schedule and reports to a channel or an issue tracker, with an owner.
• The failure message must name the suspect. Which model, which fingerprint, what changed since the last successful run. “Live contract tests failed” with no context gets triaged into the backlog.
• Distinguish an outage from a change. A 503 and a schema mismatch are different events with different owners. Classify in the reporter, retry the transient class once, and only alert on the second failure.
• Give it a pre-release gate. The one place a live failure should block is a release, not a merge. That preserves its teeth without putting it in anyone’s way daily.