Cost Tracking Is Wrong After Switching LLM Provider

KeyError: 'prompt_tokens' is the good outcome, because it stops. The bad outcome is a dashboard that keeps drawing, shows spend dropping by a third the week you migrated, and gets reported upward as a saving — until the invoice arrives and does not agree with it.

The symptom and the first check

The presenting symptom is one of three: cost per request drops sharply and implausibly at the migration boundary, cost reads exactly zero for the new provider, or the total is plausible but does not reconcile to the invoice. Before diagnosing, run the check that distinguishes them — take one hour of requests to the new provider and count how many have a non-null input token count and a non-null output token count. If that share is near zero, you have a rename or a streaming problem. If it is near 100% but the money is wrong, you have a sub-field or a rate problem.

That check is worth automating permanently, because this failure recurs on every provider addition and on some SDK upgrades. A daily assertion that token counts are present on at least, say, 99% of completed requests catches it within a day rather than within a billing cycle.

Cause 1: the usage object was renamed

The field names differ across the three surfaces most migrations touch, and they differ in a way that is easy to miss because the objects are all called usage.

• OpenAI Chat Completions reports usage.prompt_tokens, usage.completion_tokens and usage.total_tokens, per OpenAI’s Chat Completions reference.
• OpenAI’s Responses API reports usage.input_tokens, usage.output_tokens and usage.total_tokens — a different vocabulary on the same vendor, which is why a migration between two of that vendor’s own APIs breaks cost tracking just as thoroughly as a migration between vendors. The naming and the reason for it are covered in the Responses API naming.
• Anthropic’s Messages API reports usage.input_tokens and usage.output_tokens, and does not report a total — a pipeline that reads total_tokens and multiplies by one blended rate gets nothing at all here, and would have been wrong anyway, since input and output are priced differently.

The lesson in that last point generalises. A cost pipeline that stores a single total is unfixable by mapping, because the information needed to price it was discarded at write time. Store input and output separately from the beginning, whatever the provider hands you.

Streaming adds the same trap described in migrating logging fields: on Chat Completions, usage is omitted from a streamed response unless you request it with stream_options: {"include_usage": true}, and on the Messages API the output token count arrives on the message_delta event rather than at the start. If your old integration was non-streaming and the new one streams, the cost pipeline can break for that reason alone, with no provider change involved.

Cause 2: cached tokens are counted differently

This is the cause that produces a plausible-but-wrong number, which is the hardest kind to notice.

Both major surfaces report cache activity, and they report it with opposite conventions. On OpenAI’s objects the cached portion is reported as a detail of the prompt or input tokens — usage.prompt_tokens_details.cached_tokens on Chat Completions, usage.input_tokens_details.cached_tokens on Responses — meaning the cached tokens are included in the headline input count and the detail tells you how many of them were discounted. On Anthropic’s Messages API the cache figures are separate top-level usage fields, usage.cache_creation_input_tokens and usage.cache_read_input_tokens, alongside usage.input_tokens.

A pipeline that assumes the first convention on a provider using the second undercounts the cached portion entirely. A pipeline that assumes the second on a provider using the first double-counts it. Both produce a number that looks like money and is not, and both are invisible without reconciliation.

There is a pricing dimension on top of the counting one. Cache writes and cache reads are typically priced differently from ordinary input tokens, and the multipliers differ by provider, so a correct cost calculation needs three rates on the input side rather than one. Do not fold them together; keep the token categories separate in storage and apply the rates at query time, so that a rate change is a configuration edit rather than a backfill.

Cause 3: reasoning tokens are billed but hidden

If either side of the migration involves a reasoning model, there is a category of output token you are charged for and never see in the response text. OpenAI reports these under usage.completion_tokens_details.reasoning_tokens on Chat Completions and under usage.output_tokens_details.reasoning_tokens on Responses; these are counted inside the output total, not in addition to it.

The migration-specific failure is a cost model that estimates output tokens from the length of the returned text — a shortcut that is roughly defensible for a non-reasoning model and badly wrong for a reasoning one, since the visible answer can be a small fraction of the billed output. If your dashboard derives tokens from text length anywhere, that derivation is the bug; the accounting in reasoning token billing explains why. Always read the reported usage rather than measuring the string.

A fourth, duller cause deserves a line: the model identifier. Rates are per model, and the model string a provider returns in the response is not always the string you sent — an alias can resolve to a dated snapshot. A rate table keyed on the string you sent will silently miss on the string that comes back, and a lookup miss that defaults to zero rather than raising is how a dashboard reaches zero without an error. Key on the returned model string, and make a missing rate an alert.

The mapping, and reconciling against the invoice

Normalise into token categories that are provider-neutral, and price at query time.

type UsageRecord = {
  provider: string;
  model: string;            // as returned, not as requested

  input_uncached: number;   // input tokens charged at the full rate
  input_cache_read: number; // charged at the cache-read rate
  input_cache_write: number;// charged at the cache-write rate, if any
  output_visible: number;   // output tokens excluding reasoning
  output_reasoning: number; // billed output you cannot see

  usage_source: "response" | "stream_final" | "absent";
};

// Provider A (details nested inside the headline count):
//   input_cache_read = usage.prompt_tokens_details?.cached_tokens ?? 0
//   input_uncached   = usage.prompt_tokens - input_cache_read
//   output_reasoning = usage.completion_tokens_details?.reasoning_tokens ?? 0
//   output_visible   = usage.completion_tokens - output_reasoning
//
// Provider B (cache counts are separate top-level fields):
//   input_uncached    = usage.input_tokens
//   input_cache_read  = usage.cache_read_input_tokens ?? 0
//   input_cache_write = usage.cache_creation_input_tokens ?? 0
//   output_visible    = usage.output_tokens

The usage_source field is the part that earns its place. It records whether the numbers came from a normal response body, from the final event of a stream, or were absent — and a dashboard that plots the share of rows with usage_source: "absent" makes the entire class of failure on this page visible on the day it starts rather than on the day the invoice arrives.

Then reconcile. Once per billing period, sum your computed spend by model and compare it against the provider’s own reported usage for the same window. Do not aim for exactness — rounding, minimum billing units and free-tier allowances make small gaps normal — but do set a tolerance and alert on it. A persistent one-directional gap is almost always one of the four causes above, and the sign tells you which side to look at: undercounting points at a missing category, overcounting at a double-counted one. The general practice is in testing that token counting matches the bill.