Why Token Counts Differ Between Your Count and the Bill

You counted 1,400 tokens. The response says 2,900. Nothing is broken — there are at least eight ways a token gets into a request without passing your counter, and each one is identifiable from the numbers you already have.

Where the tokens come from

The usage you never received

One gap deserves singling out because it produces a total of zero rather than a wrong number. On OpenAI-shaped streaming endpoints, usage is not included in the stream by default — you must ask:

stream = client.chat.completions.create(
    model=MODEL, messages=messages, stream=True,
    stream_options={"include_usage": True},     # without this: no usage at all
)

usage = None
for chunk in stream:
    if chunk.usage:                 # arrives in a final chunk with empty choices
        usage = chunk.usage
    elif chunk.choices:
        yield chunk.choices[0].delta.content or ""

Teams that stream in production and batch in evaluation frequently end up with cost dashboards that only account for the evaluation traffic, and a monthly invoice several times larger than the dashboard. The final chunk also carries no choices, so a loop that indexes choices[0] unconditionally will throw on it — which is the usual reason the option gets reverted rather than fixed.

A worked gap

A four-tool research agent, twelve turns to complete a task. Counting only message content:

what you counted (message content only)
  system 350 + growing history, averaging ~2,000 tok/turn
  x 12 turns                                        = 24,000 tok

what was actually sent
  tool schemas      900 tok  x 12 turns             = 10,800 tok
  template overhead  ~8 tok  x ~40 messages         =    320 tok
  provider preamble  unknown, assume 100 x 12       =  1,200 tok
                                                      ----------
  total unaccounted                                 = 12,320 tok

  actual / counted = 36,320 / 24,000                = 1.51x

Fifty-one per cent over, and every token of it structural — the same gap on every request of this shape, forever. The tool schemas alone are nearly a third of the true prompt, which is also the argument for trimming tool descriptions and for not attaching all twenty tools to every call.

The other systematic gap is the double application of a chat template. A self-hosted server applies the checkpoint’s template to the messages you send; if your client also applied it before sending, the scaffolding appears twice. Token counts inflate by a predictable amount per message, output quality drops, and neither symptom points at the cause. The tell is a count that is high by a constant multiple of your message count rather than by a constant fraction of your text.

Rounding and aggregation account for the rest. Invoices are usually summarised per model per day in whole units, so a small discrepancy between your per-request sum and a dashboard figure is expected and uninteresting; a discrepancy that scales with volume is not. And if a gateway can route the same model name to more than one upstream host, the counts can differ per route even for identical text, because the hosts may template or count slightly differently. Record which provider served each request, or you will be reconciling an average of two distributions.

Reconciling properly

• Log the returned usage, never your estimate. The response object is the authoritative number. Store it per request alongside your estimate, and alert when the ratio between them moves — a prompt change that adds a tool will show up here days before it shows up on an invoice.
• Store the buckets separately. Input, output, cache read, cache write and reasoning are five different prices. A single “tokens” column cannot be reconciled against a bill and cannot be attributed to a feature.
• Reconcile a day, not a month. Sum one full UTC day of logged usage and compare it to the provider’s own figure for that day. If it matches, your accounting is sound and you can trust the forecast. If it does not, the gap has one of the eight causes above and a day is a small enough haystack to search.
• Count retries as requests. Instrument at the HTTP layer, below your SDK’s retry logic, or you will be blind to precisely the traffic that costs the most.

The wider point is that the discrepancy is a feature of the measurement, not a defect in it. Your counter measures the text you composed; the invoice measures the request that was served, and those are genuinely different objects with a well-defined set of terms between them. Once you have identified which terms apply to your workload the ratio becomes a constant, and a constant you know is a forecast. Teams that do this once tend to end up predicting the monthly bill within a few per cent; teams that do not tend to describe inference cost as unpredictable, which it is not.