What a Provider Migration Means for an Existing Prompt Cost Ceiling Alert

A cost ceiling alert is a small piece of code that multiplies token counts by prices and compares the result to a number. A migration breaks both halves at once, and the failure is not an exception — it is an alert that goes quiet while spend goes up.

It fails by computing zero

The instinct is that a threshold set in dollars against one price table will misfire against another: too low and it pages constantly, too high and it never fires. That is true and it is the easier problem, because somebody notices within a day.

The harder problem is upstream. Cost calculators read token counts out of the response, and almost every implementation reads them defensively, in the shape of usage.get("prompt_tokens", 0) or an optional chain that coalesces to zero. When the field name changes, that code does not raise. It computes a cost of zero for every request, the alert never fires, and the cost dashboard shows a sharp drop the day of the cutover — which reads as a migration win and gets screenshotted into a slide. The first correction usually arrives from the provider’s own billing page a month later.

Rule one, therefore, before anything about thresholds: the usage extractor must fail loudly on an unknown response shape. A missing usage object is a bug, not a zero, and a cost of exactly zero on a request that returned text is never a valid answer.

The usage fields, per API shape

Three shapes are in play, and none of them share a field name with the others for the same quantity.

• OpenAI Chat Completions. usage.prompt_tokens, usage.completion_tokens, usage.total_tokens. Breakdowns sit under usage.prompt_tokens_details (including cached_tokens) and usage.completion_tokens_details (including reasoning_tokens).
• OpenAI Responses API. The same quantities renamed: usage.input_tokens and usage.output_tokens, with the cached count under usage.input_tokens_details.cached_tokens. Moving between the two OpenAI surfaces breaks the calculator just as thoroughly as changing vendor does.
• Anthropic Messages API. usage.input_tokens and usage.output_tokens, plus two fields with no counterpart elsewhere: usage.cache_creation_input_tokens and usage.cache_read_input_tokens. Note that input_tokens here is the uncached remainder, not the total — the total prompt size is the sum of all three. A calculator that treats input_tokens as the whole prompt undercounts every cached request.

Two categories deserve their own line in the normalizer because they are priced differently from what they superficially resemble. Reasoning tokens are billed as output but never appear in the text, so any ceiling derived from the length of the visible answer undercounts them. Cached input is billed at a fraction of the uncached rate, so a ceiling that multiplies the whole prompt by the uncached price overcounts — and on a heavily cached workload it overcounts enough to make the alert useless.

Streaming reports nothing unless you ask

A ceiling alert on a streaming endpoint has a second way to compute zero. On the Messages API the usage totals arrive on the terminal message-delta event, so a consumer that stops reading once it has all the text it needs never sees them. On OpenAI-shaped streaming, the usage object is omitted entirely unless you opt in by sending stream_options with include_usage set true, in which case it arrives on a final chunk after the content chunks.

So a migration that moved a route from non-streaming to streaming — a common accompaniment, since streaming is often adopted at the same time — silently removes cost accounting from that route. The test to write is not about thresholds at all: assert that every completed request, streaming or not, produced a usage record, and alert on the count of requests with no usage record rather than only on cost.

Three layers, only one of which migrates

Rebuild the alert as three separate pieces with a defined boundary between them, because only the first is provider-specific and only the first should ever need touching again.

# 1. normalizer: provider response  ->  one shape. The only provider-aware code.
Usage = {
  "input_uncached":   int,
  "input_cache_read": int,
  "input_cache_write":int,
  "output_visible":   int,
  "output_reasoning": int,
}

# 2. price table: data, versioned, with an effective date. No logic.
#    rate per million tokens, per model, per usage kind.

# 3. policy: threshold + comparison + what to do. Never provider-aware.
cost = sum(usage[k] * rate(model, k) for k in usage)
if cost > CEILING_PER_REQUEST: alert(...)

The normalizer is where the field names live and the only place a migration edits. The price table is data, and treating it as data — versioned in the repository with an effective date rather than hardcoded in the calculator — is what lets you recompute history under a new table when you want to compare like with like. The policy layer holds the threshold and the escalation, and if it needs changing because you switched providers, the boundary is in the wrong place.

Write one test per provider that feeds a recorded response into the normalizer and asserts every field of the output. Those fixtures are the thing that would have caught the silent zero, and they cost about twenty minutes.

Re-deriving the threshold

1. Pull the last few weeks of normalized usage records — token counts, not costs. If you only stored costs, this is the migration that teaches you to store counts; costs are derived and can always be recomputed, counts cannot.
2. Recompute each historical request’s cost under the new price table. You now have the distribution your alert would have seen if you had been on the new provider all along.
3. Find the percentile your old threshold sat at in the old distribution. This is the step everyone skips. The threshold’s meaning was never a dollar figure; it was “rarer than one request in a thousand” or “the top half percent”.
4. Set the new threshold at the same percentile of the new distribution, and record both the percentile and the resulting figure in the alert definition, so the next migration re-derives it rather than guessing again.
5. Run the new threshold in shadow for a week — computing and logging but not paging — before it wakes anyone up. A cost alert that pages incorrectly twice gets muted, and a muted alert is worse than none.

Do the same for a per-session ceiling, with one addition: sessions accumulate, and a migration that changed the tokenizer changed how fast they accumulate for the same conversation. Re-derive the session threshold from the distribution of completed sessions, not by multiplying the per-request figure by an average turn count. The per-tenant version of this problem, and what to do when the answer is “charge it back”, is unified cost attribution.