Testing That Your Token-Counting Code Matches the Provider's Actual Bill

Your dashboard says the request used 1,842 input tokens. The provider’s usage object on the same response says 2,106, and the monthly invoice agrees with the provider. A 14% gap is not rounding, and it is almost never the provider being wrong.

The symptom

It presents in one of three ways. Cost projections that are consistently under the invoice by a stable percentage, which is the benign version because the ratio is at least predictable. A context-fit precheck that passes and is then rejected by the API with a context-length error, because your count said the prompt fits and the provider’s count says it does not. Or, worst, a gap that was small for a year and jumped after a model change — the version where nothing in your code changed and every number moved.

All three come from the same root: you are computing a number the provider also computes, using a different method, and treating yours as authoritative because it is the one you can see before the request.

Eight reasons the two numbers differ

• You counted the message text, not the request. A chat request is serialised into a format with per-message overhead — role markers, separators, a trailing prime for the assistant turn. Counting the concatenated content strings undercounts by a few tokens per message, which is invisible on one long message and large on a hundred short ones.
• Tool schemas are input tokens. Every tool definition, including every description and every JSON Schema property, is serialised into the prompt and billed. This is frequently the single biggest missing term, and it is the subject of what a tool schema costs in tokens.
• The system prompt was not in the count. Often because it is injected by a layer below the code doing the counting.
• Images and documents. These are billed as input tokens by a formula based on dimensions or page count, and a text tokenizer will count the base64 blob or nothing at all. Both are wrong.
• The wrong encoding. With tiktoken, the encoding is per model family — cl100k_base and o200k_base give materially different counts on the same string. Hard-coding one and then adding a model from another family is a silent desync.
• The provider changed tokenizer. This is not hypothetical. Anthropic’s token-counting documentation states that Claude 4.7 and later models use a newer tokenizer under which the same input text produces approximately 30% more tokens than on earlier models, and explicitly warns against reusing counts measured against earlier models. Anthropic’s token counting page is the primary source for that.
• Cached tokens are reported separately. A cache read is billed differently from a fresh input token and appears in its own field. Summing only the headline input figure across a caching-enabled workload will not reconcile with the invoice in either direction.
• Reasoning tokens are output tokens. On models that think before answering, the thinking is billed as output and does not appear in the text you received, so any count derived from the visible response undercounts output badly.

Stop estimating things you can be told

The fix is a change of authority, not a better tokenizer. There are three sources of truth, in descending order of trustworthiness, and your local count is not one of them.

The usage object on the response is the best available: it is what you were billed for that request. On the OpenAI chat completions shape the fields are prompt_tokens, completion_tokens and total_tokens, with cached and reasoning components in nested detail objects. On Anthropic’s Messages API they are input_tokens and output_tokens, with cache_creation_input_tokens and cache_read_input_tokens alongside. Log all of them, per request, from day one. This costs almost nothing and it is the only data that makes the rest of this possible.

Where you need a count before sending — a context-fit precheck, a budget gate — use the provider’s own counting endpoint rather than a local library. Anthropic documents client.messages.count_tokens(...) in Python and client.messages.countTokens(...) in TypeScript, taking the same arguments as a message creation call including system, tools, images and PDFs, and returning input_tokens. It is documented as free and rate-limited separately from message creation. That endpoint solves six of the eight causes above by construction, because it serialises the request exactly as the real call would.

A local tokenizer still earns its place for hot-path decisions where a network round trip is not acceptable — truncating a retrieval set, say. The point is that it becomes an approximation you monitor, rather than a figure you report.

The drift test

Assert your estimator against the authority on a fixed corpus, and let it fail when the relationship changes. The corpus should be real requests, redacted, spanning your actual shapes: short chat turns, a long document, a tool-heavy request, a multi-turn conversation, and one request in a non-Latin script.

# tests/tokens/test_estimator_drift.py
import json, pytest, anthropic
from app.tokens import estimate_input_tokens

client = anthropic.Anthropic()
CORPUS = json.load(open("fixtures/token-corpus.json"))  # list of request kwargs

@pytest.mark.parametrize("req", CORPUS, ids=lambda r: r["id"])
def test_local_estimate_tracks_provider_count(req):
    authoritative = client.messages.count_tokens(
        model=req["model"], system=req["system"],
        tools=req.get("tools", []), messages=req["messages"],
    ).input_tokens

    local = estimate_input_tokens(req)
    drift = abs(local - authoritative) / authoritative

    assert drift <= 0.03, (
        f"{req['id']}: local={local} provider={authoritative} "
        f"drift={drift:.1%} — re-derive the estimator or update the encoding"
    )

Two properties of that test matter. It is parameterised per fixture, so a failure names which shape broke — if only the tool-heavy request fails, you know the tool serialisation is the term that changed, and you have skipped the whole investigation. And it calls a network endpoint, so it belongs in a scheduled job rather than on every commit; nightly is right, because the thing it detects is a vendor change and vendor changes do not correlate with your commits.

Add the reconciliation test alongside it, which needs no fixtures at all: over the last day of production traffic, compare the sum of your recorded estimates against the sum of the logged usage values. That covers the requests you never thought to put in a corpus, which is where the surprise usually lives.

Choosing the tolerance

Do not pick a tolerance from a feeling. Derive it from what the number is used for. If the count drives a context-fit precheck, the tolerance must be smaller than your safety margin below the context window — a 3% underestimate on a 200,000-token window is 6,000 tokens, so a 4,000-token margin is not protecting you. If it drives cost reporting, the tolerance is whatever error your finance reporting can absorb, and the assertion belongs on an aggregate rather than per request, because per-request percentage error is dominated by tiny requests where a four-token overhead is 10%.