Why Your Token Count Changed After Switching Providers

The same string, the same request body, and the usage object comes back with a different input_tokens than the provider you left. This is expected behaviour and there are four separate mechanisms behind it. The fix is to find out which one accounts for your gap, because only two of them are worth doing anything about.

Nothing is broken

A token is not a unit of text. It is an entry in one model family’s vocabulary, and that vocabulary was learned from that family’s training corpus by a byte-pair-encoding procedure that merges frequent adjacent pairs until it reaches a target size. Two labs running that procedure on different corpora with different target sizes end up with different vocabularies, so “how many tokens is this sentence” has no provider-independent answer. There is no canonical count that one of them is getting wrong.

What you should expect: for ordinary English prose, the counts differ by a modest percentage. For code, for text with unusual formatting, and especially for languages written in scripts under-represented in the vocabulary, the gap can be large. If your gap is small and your text is English, stop worrying and re-baseline the number. If your gap is large, the sections below tell you why.

Cause one: a different learned vocabulary

Vocabulary size is the first-order effect, and it is observable from the encoding names. OpenAI’s tiktoken library exposes cl100k_base, used by the GPT-4 and GPT-3.5-turbo generation, and o200k_base, used by the GPT-4o generation. The numbers in the names are the approximate vocabulary sizes: about 100,000 entries and about 200,000. A larger vocabulary has room for more multi-character and multi-word sequences as single entries, so the same text generally segments into fewer pieces under the larger one.

You can watch this happen rather than take it on trust. This runs and prints two numbers for whatever string you give it:

import tiktoken

text = open("sample.txt", encoding="utf-8").read()

for name in ("cl100k_base", "o200k_base"):
    enc = tiktoken.get_encoding(name)
    ids = enc.encode(text)
    print(name, len(ids))

# And to see *where* the difference is, print the pieces:
enc = tiktoken.get_encoding("o200k_base")
print([enc.decode([i]) for i in enc.encode(text)[:40]])

That last line is the diagnostic worth running. Printing the individual pieces shows you exactly which parts of your text are being shattered into many small tokens — and it is almost always the same suspects: identifiers in camelCase or snake_case, long runs of whitespace, base64 or hex blobs, emoji, and any script whose characters fall back to per-byte tokens. If your prompt embeds a serialised structure or a UUID per line, you will see it immediately.

For providers that do not publish an offline tokenizer, count through their API instead: Anthropic exposes a token-counting endpoint on the Messages API returning input_tokens for a request body without running the model, and Google’s Gemini API exposes a countTokens method returning totalTokens. Do not substitute a third-party tokenizer that claims to approximate a closed model’s; an approximation is exactly the thing you were trying to stop relying on.

Cause two: per-message overhead

This is the cause of the most common version of the complaint, which is not “two providers disagree” but “my local count disagrees with the bill on both of them”.

You are not billed for the concatenation of your message strings. You are billed for the serialised conversation the provider builds, which includes markers delimiting each turn and identifying each role, plus a few tokens priming the assistant’s reply. OpenAI’s cookbook documents this for its chat models as a fixed number of tokens added per message plus a fixed number for the reply priming. The consequence is that overhead scales with the number of messages, not their length — so a long conversation of many short turns carries proportionally far more of it than one long turn with the same character count.

Two providers that structure conversations differently therefore differ on overhead even where their vocabularies would agree. If your workload is a chat with a long history, and your gap grew when you moved rather than staying proportional, look here before looking at the vocabulary. Tool definitions are the same story: a tools array is serialised into the prompt and billed as input, and two providers serialise schemas differently. See what a tool schema costs in tokens.

Causes three and four: reasoning and cache fields

These two produce a changed count that has nothing to do with tokenization at all, and mistaking them for a tokenizer problem sends you down a long dead end.

Reasoning tokens. On models that generate internal reasoning before answering, those tokens are counted as output and billed as output, but they are not in the text you received. If your output count jumped by a multiple after a switch and the visible replies are the same length, this is it, and no tokenizer comparison will explain it. The fix is to account for them explicitly and to check whether the output cap you are sending is large enough to cover reasoning plus the answer.

Cache fields. Where prompt caching is active, the input count reported in the ordinary field may exclude the cached portion, which is reported separately — on Anthropic’s Messages API as cache_creation_input_tokens and cache_read_input_tokens. A dashboard that reads only the ordinary input field will show input tokens apparently collapsing after a switch, when in fact the same tokens are being counted in a different field at a different rate. Summing all input-bearing fields is the correction.

Attributing your own gap

1. Take one real request. Count its raw message text locally under each provider’s available counting mechanism, ignoring roles and structure. Call these L_A and L_B. The ratio is the pure vocabulary effect.
2. Send the same request to each and record the reported input tokens, U_A and U_B. The difference between U and L on each side is that provider’s structural overhead.
3. Repeat with a request that has the same total characters split across ten turns instead of one. If the overhead figure grows, it is per-message; if it does not, it is a fixed preamble.
4. Sum every input-bearing usage field, including cache fields, before comparing anything. A missing field looks exactly like a tokenizer difference and is not one.
5. Compare output counts only on requests where reasoning is disabled or absent on both sides, or your output comparison is measuring two different things.

Once attributed, most of the gap is not actionable — you cannot negotiate with a vocabulary. What is actionable is the part caused by your own text being tokenizer-hostile: dense identifiers, embedded JSON with heavy whitespace, repeated boilerplate. Those you can rewrite. And the whole exercise feeds directly into what the difference does to the monthly bill, which is the number anyone else will ask you for.