Setting a Hard Spend Cap on a CI Pipeline That Calls an LLM

The incident this prevents is specific: a retry loop with no ceiling, or a parametrised fixture that multiplied, running unattended on a branch overnight. The guard that works is a counter inside the process that raises before the next call, backed by a provider-side limit that does not depend on your code being correct.

Five layers, cheapest first

• A max_tokens on every request. The cheapest guard there is, and the one most often omitted. Without it a single call can run to the model’s full output limit, and a loop of those is how a small bug becomes a large invoice.
• A job timeout. Every CI system has one and most teams leave it at the default. A suite that normally takes four minutes should have a timeout of fifteen, not six hours.
• A concurrency limit. Both on your test runner and on the pipeline. Cancel in-progress runs for the same branch when a new commit arrives; otherwise a fast pusher has six copies of the suite running.
• An in-process token counter. The subject of this page: accumulate usage, compare against a cap, refuse to make the next call. Precise, immediate, and dependent on your code running.
• A provider-side limit on a CI-only key. The backstop. A separate API key or project for CI with its own budget ceiling, so that a failure of every layer above still terminates at a number you chose. This is the only layer that survives your process being killed, forked or bypassed.

The layers are not alternatives. The counter gives you a good error message and stops the run in seconds; the provider-side limit gives you a guarantee. Neither substitutes for the other.

The in-process counter

Wrap the client once, accumulate usage from every response, convert to a cost estimate with the same labelled prices you used to estimate the suite, and raise before the call that would exceed the cap rather than after.

# tests/spend_guard.py
import threading

class BudgetExceeded(RuntimeError):
    pass

class SpendGuard:
    """Accumulates token usage and refuses the next call past the cap.

    Prices are per million tokens and belong in configuration, not here.
    """

    def __init__(self, cap_usd, price_in, price_out):
        self.cap = cap_usd
        self.price_in = price_in
        self.price_out = price_out
        self.spent = 0.0
        self.calls = 0
        self._lock = threading.Lock()

    def check(self):
        if self.spent >= self.cap:
            raise BudgetExceeded(
                f"CI spend cap reached: est. ${self.spent:.2f} of ${self.cap:.2f} "
                f"after {self.calls} calls"
            )

    def record(self, usage):
        with self._lock:
            self.spent += (
                usage.prompt_tokens / 1e6 * self.price_in
                + usage.completion_tokens / 1e6 * self.price_out
            )
            self.calls += 1

    def wrap(self, create):
        def _create(**kwargs):
            self.check()                 # before, not after
            resp = create(**kwargs)
            if resp.usage is not None:
                self.record(resp.usage)
            return resp
        return _create

Three details do the work. The check happens before the call, so the cap is a ceiling rather than a description of what you already spent. The lock is there because test runners parallelise by thread as well as by process. And usage is treated as possibly absent, because it frequently is — which is the next section.

Streaming hides the usage

A streamed response is a sequence of chunks, and by default the usage totals are not among them. A guard that only reads resp.usage will therefore record zero for every streamed call and cheerfully allow an unbounded run — which is precisely the shape of the incident it was built to stop, since streaming is what long-running agent loops use.

On the OpenAI-compatible surface the fix is to opt in explicitly, with stream_options set to include usage; the totals then arrive on a final chunk after the content is complete, and your accumulator has to read that chunk rather than stopping at the last content delta. Other providers surface the same information differently — Anthropic’s streaming events carry usage on the message-level events — so a multi-provider harness needs one adapter per provider whose only job is to answer “how many tokens was that?”.

Where a provider genuinely gives you nothing, fall back to counting request tokens locally and estimating output from the character count, and mark the estimate as such in the error message. An approximate cap that fires is worth far more than an exact one that never runs.

Where a provider reports cached and uncached input separately, the accumulator should price them separately too. A guard that charges every input token at the full rate will fire early on a cache-heavy suite, which is the annoying direction — a cap that stops a run which was not actually expensive teaches people to raise the cap rather than to trust it.

Fail the job, not the test

When the cap trips, the correct behaviour is to stop the entire run immediately, not to fail one test and continue. A budget exception that surfaces as an ordinary test failure means the remaining two hundred tests each make their own call, discover the cap, and fail — and if any of them are wrapped in a retry, they make several calls each doing it.

1. Install the guard as a session-scoped fixture so one instance covers the whole run.
2. Raise a distinct exception type, not AssertionError, so nothing mistakes it for a test result.
3. Trigger the runner’s stop-on-first-failure behaviour when that exception appears — in pytest, calling session.shouldstop or exiting from a hook; in a JS runner, the bail option.
4. Print the running total, the call count and the cap in the message. “Budget exceeded” with no numbers tells the next person nothing about whether the cap is too low or the suite too greedy.
5. Emit the total as a job summary or annotation even on success, so the number is visible before it becomes a problem.

Where the counter cannot help

Two cases, both worth knowing before you rely on it. The first is a process that never returns: a call that hangs holds tokens the provider is already generating, and an in-process counter that is blocked on the response cannot fire. Request-level timeouts, set on the client rather than only on the job, are the guard for that.

The second is anything that runs outside your wrapper — a subprocess, a tool the agent shells out to, a second library with its own client, a developer running the suite locally with the same key. The counter is per process and knows only about calls that went through it. This is exactly why the provider-side limit on a CI-only key is the last layer rather than an optional extra, and why the per-run cap is not sufficient on its own: per-pull-request token budgets address the case where forty compliant runs add up to a bill no single run could have produced.