What a Provider's Usage Reporting Delay Means for Real-Time Budgets

A budget cap built on a provider’s usage dashboard is enforcing a number that describes the past. How far in the past, and how much you can spend inside that window, is arithmetic — and it is worth doing before you promise anyone a hard cap.

Two numbers, and which one is real

Providers expose consumption twice, and the two are not versions of each other.

The inline usage object comes back attached to the response: token counts for the request that just finished, available in your process microseconds after the call ends. It is exact for that request and it exists nowhere else until you record it.

The aggregate reporting surface — a usage or cost API, and the dashboard drawn from it — is a rollup. It is bucketed, usually by hour or by day in UTC, and it is published after a delay while events are collected and priced. Both properties are inherent to the design rather than defects: an organisation-wide cost figure has to collect from every endpoint, every project and every key before it can be correct, and that collection takes time.

The consequence is a rule with no exceptions. Enforcement runs on the inline object; reporting runs on the aggregate. If your cap reads the aggregate, then in the interval between an event happening and the aggregate knowing about it, your cap does not exist.

Deriving the exposure window

Name the inputs. Every one of these is yours to measure or to choose; none of them is quoted here as a fact about any provider.

• L — the reporting lag: bucket granularity plus publication delay, in hours. Measure it by making one known request and timing how long until it appears in the aggregate.
• P — your polling interval of that surface, in hours.
• R — the peak spend rate in dollars per hour that your traffic can reach. Not the average; the cap only matters when something has gone wrong.
• N — the maximum number of requests that can be in flight at once, which is your concurrency limit if you have one and is unbounded if you do not.
• m — the worst-case cost of one request, computed as the context window times the input price plus max_tokens times the output price, per million tokens. This is a bound you can compute exactly from your own configuration.

For a cap enforced from the aggregate surface, the overshoot past the cap is everything spent that the enforcer could not see. Spending continues at R for the lag L, plus up to one more poll interval P before you look again, plus whatever is already in flight when you finally do stop admitting:

overshoot_aggregate  =  R * (L + P)  +  N * m

worked example, all inputs assumed and none of them
a claim about any provider:

  R = $40/hour        peak spend under a runaway loop
  L = 1 hour          measured lag of the aggregate surface
  P = 5 minutes       = 0.083 hours
  N = 50              concurrency limit
  m = $0.12           128k context in, 4k out, at assumed prices

  overshoot = 40 * 1.083 + 50 * 0.12
            = 43.33 + 6.00
            = $49.33  past a cap you told someone was hard

Now enforce from your own meter instead. Every completed request writes its inline usage into a counter you own, so the lag is the time to write a row and the poll interval is zero. Both terms vanish:

overshoot_self_metered  =  N * m  =  50 * 0.12  =  $6.00

That is the result worth carrying away. Self-metering does not make the cap exact — it reduces the exposure to exactly one term, the in-flight set, and that term is the product of two knobs you already control. Halve the concurrency limit or halve max_tokens and you halve the overshoot. Neither is available to you at all when the cap is built on a dashboard, because the R * (L + P) term dominates and is set by the provider.

Reserving instead of counting

The N * m term exists because a request’s cost is unknown until it finishes, so a counter incremented on completion is always behind by the in-flight set. You can remove it by debiting at admission instead, which is the pattern airlines and payment systems both use and which is worth naming explicitly: reserve, then settle.

1. At admission, compute the request’s worst-case cost — the tokens you are about to send, priced as input, plus max_tokens priced as output — and debit that from the budget atomically. If the debit would exceed the cap, reject before calling the provider.
2. At completion, read the inline usage object, compute the actual cost, and credit back the difference. Almost every request finishes well short of max_tokens, so most of the reservation is returned within seconds.
3. On failure or cancellation, settle with whatever usage you have, estimated from the deltas received if the stream was truncated, and credit the rest back. Never simply release the whole reservation — the provider bills for tokens it generated before you disconnected.
4. Expire orphans. A worker that dies between reservation and settlement leaks the reservation forever. Give every reservation a TTL equal to your request timeout plus a margin, and sweep expired ones into an estimated settlement rather than deleting them, so a crash does not create free spend.

The trade is explicit and worth stating: reservations make the cap strictly conservative, so you will refuse some requests that would in fact have fit under the budget. The size of that error is the gap between max_tokens and typical output length, which means the same parameter that bounds your overshoot also bounds your false refusals, in opposite directions. Setting max_tokens to a realistic ceiling rather than the model’s maximum improves both at once.

Making the counter correct across replicas

A cap that reads a total, compares it, and then writes an increment is wrong the moment two replicas do it concurrently. Both read a total below the cap, both admit, and the cap is exceeded by however many replicas raced. This is not a rare interleaving: a runaway loop produces exactly the burst of simultaneous admissions that triggers it.

Increment first, then decide. An atomic increment that returns the new value — a single Redis INCRBYFLOAT, a database UPDATE ... RETURNING — lets each replica see its own position in the sequence. If the returned total is over the cap, that replica refunds its own increment and rejects. The refund is safe because it only ever undoes an increment the same replica made.

// admission check: increment, inspect, refund if over
const reserved = worstCaseCost(req);
const total = await redis.incrbyfloat(key(tenant, window), reserved);
if (total > cap) {
  await redis.incrbyfloat(key(tenant, window), -reserved);
  throw new BudgetExceeded({ cap, total });
}
// ... call provider, then settle:
await redis.incrbyfloat(key(tenant, window), actualCost - reserved);

Two details that decide whether this survives contact with production. The window key must encode the budget period, so that a monthly cap uses a key that changes at the period boundary and old keys expire on their own rather than needing a reset job. And the counter must be as durable as the promise: an in-memory counter that resets when the process restarts turns a hard cap into a cap per deploy, which is exactly the kind of thing nobody notices until a bad week coincides with a rollout.

What you are left exposed to

Even with reservations, atomic counters and inline metering, three gaps remain, and it is better to name them than to describe the cap as absolute.

• Pricing drift. Your cap is enforced against your price table. If the provider changes a price and your table lags, the cap holds in your currency and not in theirs. Date the table and alert on the drift between your computed spend and the aggregate surface, which is what the aggregate is genuinely good for — see rebuilding usage alerts.
• Spend that never passes through your gateway. A batch job with its own key, a notebook, a fine-tuning run, a per-tenant key issued under a bring-your-own-key arrangement. The cap covers the path it sits on and nothing else, which is an argument for having one path.
• Non-token charges. Storage for fine-tuned models, cache write premiums, image or audio units priced differently from text. A meter that models only input and output tokens will underread by whatever fraction of your bill those represent.

None of these is a reason to fall back to the dashboard. They are the list of things to reconcile, on a cadence, against a bound you can state — and the bound, derived above, is N * m plus your reconciliation tolerance rather than an unbounded window of a provider’s choosing.