Migrating a Cost-Per-Conversation Metric Between Providers

Cost per request survives a migration. Cost per conversation does not, because it is built out of a token sum whose fields are named differently, nested differently, and — on one common shape — overlap in a way that makes the obvious addition wrong.

Define the unit before the price

Write the metric down as a formula before touching any provider. For a conversation consisting of turns 1..T:

cost(conversation) = SUM over turns of
      in_billable[k]  * P_in
    + cache_write[k]  * P_write
    + cache_read[k]   * P_read
    + out[k]          * P_out

P_in, P_write, P_read, P_out are per-token prices you
substitute from the provider's current price page. Every
number below is worked with placeholders; none of the
arithmetic depends on their values.

The four price terms are placeholders on purpose. Per-token prices change, they change per model, and a page that quotes them is wrong within a quarter. What does not change is the structure: four quantities, four prices, and a token count that grows with the conversation rather than with the message.

The arithmetic of a multi-turn conversation

The mistake that makes a cost-per-conversation metric wrong before any provider is involved is treating it as turns multiplied by cost per turn. Chat APIs are stateless: every turn resends the entire transcript, so input grows linearly across turns and the sum of inputs grows quadratically.

Let S be the system prompt in tokens, u the tokens in each user message and a the tokens in each assistant reply, both treated as constant for the derivation. At turn k the request carries the system prompt, all k-1 completed exchanges, and the current user message:

in[k]      = S + (k-1)*(u + a) + u

total_in   = T*S + T*u + (u + a) * T*(T-1)/2
total_out  = T*a

Worked with labelled assumptions — S = 800 tokens, u = 60, a = 220, T = 8 turns, all four chosen as illustrative round numbers rather than measured from any system:

total_in  = 8*800 + 8*60 + 280 * (8*7/2)
          = 6400  + 480   + 280 * 28
          = 6400  + 480   + 7840
          = 14,720 input tokens

total_out = 8 * 220 = 1,760 output tokens

The naive per-turn estimate — eight turns each carrying the system prompt and one user message, 8 * 860 = 6,880 — understates input by a factor of 2.1 at eight turns. At sixteen turns with the same assumptions the quadratic term alone is 280 * 120 = 33,600 tokens, and the understatement grows with every turn. Any budget or alert built on cost per request rather than cost per conversation will therefore be systematically optimistic for exactly the sessions that cost most.

The usage fields, and the subset trap

Now the migration part. Each provider reports usage under its own names, and they are not merely renamed — they are structured differently.

• OpenAI, Chat Completions. usage.prompt_tokens, usage.completion_tokens, usage.total_tokens, with a nested usage.prompt_tokens_details.cached_tokens and usage.completion_tokens_details.reasoning_tokens.
• OpenAI, Responses API. The same information under input_tokens, output_tokens and total_tokens, with the details objects renamed to match. One provider, two endpoints, two vocabularies — a detail that catches teams migrating between endpoints rather than between vendors.
• Anthropic, Messages. usage.input_tokens, usage.output_tokens, usage.cache_creation_input_tokens and usage.cache_read_input_tokens as siblings, per Anthropic’s Messages API reference.
• Google, Gemini. A usageMetadata object with promptTokenCount, candidatesTokenCount, cachedContentTokenCount, thoughtsTokenCount and totalTokenCount, per Google’s generateContent reference.

The trap is the relationship between the cached figure and the headline input figure, and it goes in opposite directions on the two common shapes. Where the cached count is nested inside a details object it is generally a breakdown of the headline number: the cached tokens are already inside prompt_tokens, so adding them again double-counts. Where the cached counts are siblings of input_tokens they are generally additional: the headline figure excludes them, so omitting them under-counts the tokens processed.

Both mistakes produce a metric that looks plausible and is wrong by the size of your cached prefix, which for a system-prompt-heavy application is most of the input. So do not infer the relationship from field names. Send one request with a large cached prefix, print the whole usage object, and check by arithmetic whether the parts sum to the total the provider reports. That single check, repeated per provider and per endpoint, is the entire defence, and it belongs in a test rather than in a notebook — the library’s general treatment is testing that your token counting matches the bill.

Two more fields change the answer without changing the transcript. Reasoning or thinking tokens are billed as output on the providers that expose them but are not present in the text you received, so a metric computed by counting the characters you got back will under-report; take the number from the usage object, never from the response body. And on providers that report a service tier or batch indicator in usage, the same token counts carry different prices, so the tier has to be carried into the cost record as a dimension rather than folded away.

What caching does to the same conversation

Apply prompt caching to the worked example. Assume the whole transcript up to the current user message is served from cache, and that a cached read is priced at a fraction r of the normal input price — r is a placeholder you substitute from the provider’s price page, and it is the single most important input to this calculation.

cached[k]  = S + (k-1)*(u + a)
fresh[k]   = u

total_cached = 8*800 + 280*28 = 14,240 tokens
total_fresh  = 8*60             =    480 tokens

effective input, in units of uncached input tokens:
  14,240 * r + 480

  at r = 0.1  ->  1,904   (7.7x cheaper than 14,720)
  at r = 0.25 ->  4,040   (3.6x cheaper)
  at r = 0.5  ->  7,600   (1.9x cheaper)

The sensitivity is the deliverable here. Because the quadratic term is almost entirely cacheable, the ratio between two providers’ cost per conversation depends far more on their cached-read discount and their cache lifetime than on their headline input price. A provider that is nominally dearer per input token can be cheaper per conversation, and a comparison built from price-page headline numbers will get the sign wrong. Cache write cost and cache expiry matter too: if the cache lifetime is shorter than the gap between turns in your real traffic, you pay to write a cache you never read.

Comparing across the switch

Recompute history rather than splicing two series together. Keep the raw per-turn token counts as recorded by each provider, keep the price table as a dated, versioned artefact, and derive cost at query time. Then the same conversation can be costed under both providers’ price tables, which is the comparison you actually want and the one a stored-cost column can never give you.

Report the metric as a distribution, not a mean. Conversation cost is heavy-tailed by construction — the quadratic term means the longest five per cent of sessions dominate the total — so a median and a 95th percentile per provider tell you something a mean hides, especially when a migration changes the length distribution because the new model is terser or more verbose. That change in output length is a real cost effect of the migration, and it belongs in the same report rather than in a footnote about model behaviour.