Why Your CI Bill Jumped After Adding a Prompt Test Suite

The suite is forty tests and the bill went up by a factor of twenty. The arithmetic does not obviously work, and it does not need to: the multiplier is almost never the number of tests. It is how many times each one runs, and how many calls each one makes that you did not count.

Attribute the spend before you guess

Do this first, because every remaining step is faster once you know which environment the money went to, and because the most common outcome of guessing is a week spent optimising the wrong thing.

• A separate API key or project for CI. The single highest-value change here, and it takes minutes. Provider dashboards break spend down by key or project, so a CI-only credential turns “the bill went up” into a number you can watch daily. Do this even if you fix nothing else.
• A per-request tag. Where the provider supports a metadata or user field, set it to the pipeline, the repository and the run. It survives into the usage export, so you can attribute to a workflow rather than to an environment.
• Your own meter. The usage object on every response, summed per run and emitted as a job summary. This gives you per-run and per-test numbers the provider cannot, because the provider does not know what a test is.

With those three in place the diagnosis is usually immediate: either one run is far more expensive than you thought, or the runs are far more numerous. The causes below split along exactly that line.

The six causes, in order of frequency

• Every test hits the live API on every commit. The base case. Forty tests times forty commits a week is 1,600 paid calls, and none of them needed a live model to assert what they assert.
• The trigger fires more than once per change. A workflow on both push and pull_request runs twice for every commit on a branch with an open pull request; a merge queue adds a third. Check the workflow triggers before believing your commit count.
• A build matrix multiplies it. Three Python versions, or three models, means three full suites. Matrices are usually added for reasons that predate the model calls, and nobody revisits them afterwards.
• Job-level retries. A retried job re-runs every test in it, not the failing one. One flaky assertion with a job-level retry doubles the cost of the whole suite each time it fires, which is why flakiness and cost are the same problem here.
• An LLM judge on every case. Doubles the calls and can more than double the tokens, since a judge prompt carries the rubric, the input and the candidate answer. The arithmetic works this through.
• A fixture that re-runs per test. The one that surprises people, and the next section.

The fixture-scope bug

A setup step that calls the model — building an index, priming a conversation, generating test data, warming a cache — written as a fixture without an explicit scope runs once per test that requests it, not once per session. Forty tests then make forty setup calls, and because the setup prompt is usually the long one, those forty calls can dominate the bill while the forty tests you were thinking about are a rounding error.

# Wrong: default scope is per-test. 40 tests, 40 model calls.
@pytest.fixture
def seeded_index():
    return build_index_with_model(CORPUS)

# Right: one call for the whole session.
@pytest.fixture(scope="session")
def seeded_index():
    return build_index_with_model(CORPUS)

The same shape appears in JavaScript runners as a call inside beforeEach that belonged in beforeAll, and in both cases it is invisible in review because the diff is one word. It is worth grepping for: any fixture or hook whose body reaches a model client and whose scope is the default is a candidate.

Parametrisation compounds it. A fixture that is both per-test and parametrised over five values runs five times per test, and a suite with forty tests has just made two hundred setup calls to assert forty things.

Make the network unreachable

Fixing the six causes is not the fix. The fix is making the failure impossible to reintroduce, because the next person to add a test will copy the nearest example and the nearest example may be a live one. Enforcement means the mocked tier physically cannot reach the provider.

1. Split the suite into a mocked tier and a live tier, with the split expressed as a marker or a tag rather than a directory convention people can forget — see the two-tier design.
2. In the mocked tier, block outbound sockets. In Python, pytest-socket’s disable-socket option turns any real network call into an immediate error; in Node, MSW can be configured to treat an unhandled request as an error rather than passing it through. Either way an accidental live call fails loudly instead of billing quietly.
3. Set the replay record mode to refuse new recordings in CI, so a missing cassette is an error rather than a live request.
4. Remove the production API key from the mocked tier’s environment entirely. A tier that cannot authenticate cannot spend, and this catches the paths the socket block misses.
5. Move retries from job level to test level, so a rerun re-issues one test’s calls rather than the suite’s.
6. Add the spend cap from setting a hard spend cap to the live tier, so the remaining paid path has a ceiling.

Verifying the fix held

Two checks, both cheap and both worth automating. The first is a per-run token total emitted as a job summary on every run, with the mocked tier expected to report zero. A number that is supposed to be zero is the easiest thing in the world to monitor, and a non-zero value names the run that reintroduced a live call.

The second is a weekly comparison of the provider’s reported spend on the CI key against the sum of your own per-run totals. If the provider says more than you counted, something is calling the API outside your wrapper — a subprocess, a second client, a developer using the CI key locally. That gap is the one your own instrumentation is structurally unable to see, which is exactly why the separate key from the first section earns its keep long after the incident is over.