Gemini's Tokenizer: Why Token Counts Differ From GPT's

Paste the same paragraph into a GPT token counter and into Gemini’s countTokens endpoint and you get two different integers. Neither is wrong. They are two different segmentations of the same bytes, and the difference is large enough that a budget built on one of them will be wrong about the other.

What Gemini’s tokenizer is

Gemini tokenizes with SentencePiece over a vocabulary of 256,000 pieces. That number is not folklore: Google ships the actual SentencePiece model file to clients that do local counting, and the Gemma model cards — Gemma being the open-weights family built from the same research — document the 256k vocabulary directly. The pieces are learned subword units, and the model has byte fallback, meaning any byte sequence the vocabulary does not cover is still representable, one token per byte, rather than becoming an unknown token.

A GPT model uses a byte-pair-encoding vocabulary from OpenAI’s tiktoken. GPT-4o and later use o200k_base, roughly 200,000 pieces; GPT-4 and GPT-3.5 use cl100k_base, roughly 100,000. Those are different training corpora, different merge rules and different sizes, so the segmentation of any given string is different by construction.

Google’s own rule of thumb, published in the Gemini API tokens guide, is that a token is about four characters and 100 tokens is about 60 to 80 English words. That is a planning figure for English prose and nothing more. It is not a conversion factor between providers, and it is close to useless for code, JSON, non-Latin scripts or anything with long identifiers.

Why the counts diverge

Three mechanisms produce nearly all of the gap.

• Vocabulary size sets the ceiling on merging. A larger vocabulary can afford to keep whole words, and whole common multi-word fragments, as single pieces. A smaller one has to split them. All else equal, a 256k vocabulary segments ordinary text into fewer pieces than a 100k one, and the advantage narrows against a 200k one.
• What the vocabulary was trained on decides which text is cheap. A tokenizer trained with heavy multilingual coverage keeps common Hindi, Arabic or Japanese sequences as single pieces; a more English-weighted one falls back toward bytes and the count for the same sentence can multiply. This is the largest source of divergence between providers, and it runs in different directions for different languages — which is exactly why a single ratio does not exist.
• Whitespace and punctuation conventions differ. SentencePiece marks word boundaries with a meta-symbol on the leading space, so  the and the are usually distinct pieces. Byte-pair vocabularies encode the leading space too, but with different merges. Indentation-heavy code, which is mostly repeated spaces, is where you see this most.

The consequence worth internalising: the ratio between two providers is not a constant. It is a function of your text. Publishing a single multiplier for your workload is fine; publishing one for “Gemini versus GPT” is not, and any page that gives you one without saying what it counted has told you nothing.

Getting both numbers for your own text

This takes about a minute and settles the question for the text you actually send, which is the only text whose ratio matters. For Gemini, the API counts it for you and the call is free of generation cost:

curl "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash:countTokens" \
  -H "x-goog-api-key: $GEMINI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "contents": [{"parts": [{"text": "The quick brown fox jumps over the lazy dog."}]}]
  }'

# -> {"totalTokens": N, "promptTokensDetails": [...]}

For an OpenAI model, count locally with tiktoken and the encoding that model uses:

import tiktoken

enc = tiktoken.encoding_for_model("gpt-4o")   # o200k_base
print(len(enc.encode("The quick brown fox jumps over the lazy dog.")))

Run both over a representative sample of your real traffic, not a sentence. Fifty prompts drawn from a log will give you a ratio you can budget with; one pangram will give you a number that does not generalise past itself. See the countTokens method in detail for the response shape and for counting a multi-turn conversation rather than a single string.

The part GPT tokenizers cannot count at all

For Gemini the token count of a request is frequently dominated by things a text tokenizer has no opinion about. Google documents fixed rates for the non-text modalities in the tokens guide: images below a threshold size count as a flat number of tokens and larger ones are tiled, with each tile counting the same flat amount; video and audio count at a fixed number of tokens per second of media.

This is why countTokens accepts a full contents array including inlineData and fileData parts rather than a bare string. If your prompts carry media, a local text tokenizer is not an approximation of your bill — it is measuring a different thing. Send the real request body to countTokens and read totalTokens.

Where the gap actually bites

• Cost comparisons. Per-million-token prices are only comparable if the token means the same thing, and it does not. A provider that is 10% cheaper per token and segments your text 15% more finely is more expensive. Compare cost per request on your own corpus, not per token.
• Context-window headroom. A prompt assembler that trims to fit using a GPT tokenizer’s count will systematically mis-estimate against a Gemini limit. Trim using the count from the provider you are about to call.
• Chunking for retrieval. Chunk sizes tuned in one tokenizer drift when the same corpus is embedded or read by another. If chunks are sized in tokens, they are sized in a specific tokenizer’s tokens.
• Cache thresholds. Caching minimums are expressed in the provider’s own tokens. A prefix that clears the threshold under one count may sit just under it in another — see the documented minimum for Gemini caching.