Budget Alerts and Hard Spend Caps

Most “spend limits” are notifications. They tell a human that money has already left, which is a useful thing to know and is not a limit. A limit refuses the request.

An alert is not a cap

The distinction is whether the mechanism sits in the request path. An alert reads spend after the fact and pages someone. A cap is a check before the call that can return an error instead of an answer. Only one of them bounds your loss, and the gap between them is measured in the time it takes a person to wake up, understand, and deploy a fix.

The failure this protects against is rarely a gradual overrun. It is a loop: an agent that retries forever, a webhook that reprocesses the same document, a bug that resubmits a queue, a scraper that found an unauthenticated endpoint. These do not creep. They run at whatever rate your concurrency allows, which is usually thousands of times your normal rate, and they are indistinguishable from healthy traffic on every dashboard except the cost one.

What the lag costs

max_loss = burn_rate * detection_lag

  burn_rate      dollars per minute during the incident
  detection_lag  alert delay + notice + diagnosis + deploy

Compute burn_rate for your own worst case rather than guessing it: it is concurrency × requests_per_second_per_worker × cost_per_request × 60. With an assumed 50 concurrent workers each managing 2 requests per second at $0.004 a request, that is 50 × 2 × 0.004 × 60 = $24 per minute.

burn = $24/min

usage dashboards refresh hourly ......  60 min
alert fires, engineer notices ........  15 min
diagnose, decide .....................  20 min
ship the fix .........................  15 min
                             total ... 110 min

max_loss = 24 * 110 = $2,640   from a single loop bug,
                               with alerting working perfectly.

The dominant term is the first one. If your spend data is an hour stale, no amount of alerting discipline gets the loss below an hour’s burn — which is the argument for a cap in the request path, where the lag is zero by construction.

The race at the heart of a cap

Here is why this is harder than a rate limit. A request’s cost is not known until it has completed, because output length is decided by the model. So a cap that checks spend before the call is checking a number that excludes every request currently in flight.

The fix is a reservation, borrowed from ordinary accounting: charge a conservative estimate up front, then reconcile with the real figure.

ceiling = ( T_in * P_in  +  max_tokens * P_out ) / 1e6

  the maximum this request can possibly cost, computable
  before sending it -- which is why max_tokens must be set

on_request:
    reserved = atomic_add(key_spend, ceiling)
    if reserved > cap:
        atomic_add(key_spend, -ceiling)
        reject(402)

on_response:
    atomic_add(key_spend, actual_cost - ceiling)   # refund the slack

on_error_or_timeout:
    atomic_add(key_spend, -ceiling)                # release

Three properties make this work. The reservation is atomic, so concurrent requests cannot both see room that only one of them has. The ceiling is an over-estimate, so the cap errs toward refusing early rather than overshooting. And the reconciliation returns the difference, so a cap set at $100 does not behave like $40 because every request reserved 2.5× what it used.

Even so, the overshoot is bounded rather than zero. With k requests in flight when the cap is reached, the maximum overshoot is k × ceiling — the reservations already granted. Size your cap with that headroom in mind, and note that this is one more reason to keep max_tokens from being enormous: it appears directly in the overshoot bound.

Four layers, cheapest first

Each layer catches a different failure and none of them substitutes for the others.

• 1. Per-request ceiling. Set max_tokens on every call, always. It is the only thing that bounds a single runaway generation, it costs nothing, and it makes the reservation arithmetic above possible at all.
• 2. Per-logical-operation budget. One user action may involve an agent loop, retries and a fallback. Give the whole operation a budget in micro-dollars and a step cap, and abort when either is exhausted. This catches the loop that no single-request limit can see, and it is the same accumulator described under retry cost.
• 3. Per-key or per-tenant daily cap. The blast radius control. One compromised key, one abusive customer, one misconfigured integration is contained to its own cap rather than to your account balance. Daily rather than monthly, because a monthly cap lets a single bad day consume a month.
• 4. Account-level cap at the provider. The last resort, and the only one that still works when the bug is in your own limiter. It fails closed and it takes the whole product down, so it belongs well above your expected spend — as a circuit breaker, not as a budget.

A useful test of whether the layers are real: pick each one and ask what specific incident it stops that the layer below does not. If you cannot answer, it is a duplicate.

What to do when the cap is hit

Returning a 402 to everyone is a decision, not a default, and it is usually the wrong one. There is a ladder between full service and no service, and each rung is cheaper than the last:

• Route to a cheaper model. Often an order of magnitude less per request, and for many features a smaller model is a degradation the user will not notice.
• Shorten the output. Tighten max_tokens and drop optional generated sections. Cuts the expensive half.
• Disable the expensive features first. Turn off reasoning, drop to fewer retrieval chunks, disable the summariser. Named in advance, in priority order, so the decision is not made during the incident.
• Queue instead of refuse. If the work is not interactive, defer it to the next budget period rather than dropping it.
• Then refuse, with a specific error. A 402 that says which cap was hit and when it resets is an operable error. A generic 500 turns a working cost control into a support ticket.

Whichever rungs you implement, test them. A cap that has never been exercised is a cap that fails on the day it matters, and the cheapest way to find out is to set it to a trivially small value in staging and watch what the product does.