Grok's Tokenizer: Why Token Counts Differ From GPT's

Counting your prompt with tiktoken and sending it to Grok gives you a number from the wrong vocabulary. xAI exposes an endpoint that gives you the right one, and a usage field that gives you the one you were actually billed for.

Why two tokenizers disagree

A tokenizer is a learned artefact, not a standard. Byte-pair encoding starts from bytes and repeatedly merges the most frequent adjacent pair, recording each merge, until the vocabulary reaches its target size. The merge list depends entirely on the corpus it was fitted to and the size it was fitted for — so two labs training on different data with different vocabulary sizes end up with different segmentations of the same sentence, by construction. Nothing is being done wrong by either. There is simply no canonical answer to “how many tokens is this string” independent of a model.

The disagreement is not uniform, which is what makes a single correction factor unreliable. English prose is the case every vocabulary is best fitted to, so counts tend to be closest there. The gaps widen for anything the corpora treated differently: source code, where indentation runs and language keywords may or may not have dedicated tokens; non-Latin scripts, where a vocabulary with little coverage falls back toward per-byte encoding; identifiers, snake_case and camelCase splits; long digit strings; emoji; and unusual whitespace. Those are exactly the inputs that show up in structured extraction and code assistance, which are exactly the workloads with the tightest budgets.

tiktoken is the wrong instrument

tiktoken implements OpenAI’s encodings — o200k_base for the GPT-4o family, cl100k_base before it. It is fast, offline, and correct for the models it was built for. It has no knowledge of Grok’s vocabulary, and it does not fail when pointed at Grok text — it returns a confident number computed from the wrong merge table.

That is the property that makes it dangerous rather than merely unhelpful. A wrong count that raises an exception gets fixed; a wrong count that returns cleanly becomes a constant in a truncation function. This page will not give you a Grok-to-GPT ratio, because publishing one would recreate exactly that problem: a number with no measurement behind it, used with confidence, applied to text that does not resemble whatever it was derived from.

The same argument applies to every cross-model estimate — an Anthropic count is its own vocabulary and a Gemini count is another again. Use each provider’s own counter.

The tokenize endpoint

xAI exposes tokenization as a service, so you do not have to estimate. It takes text and a model name — the model matters, because the tokenizer belongs to the model — and returns the token sequence, including token ids and their string representations.

from xai_sdk import Client

client = Client(api_key=os.environ["XAI_API_KEY"])
tokens = client.tokenize.tokenize_text(text="Hello, world!", model="grok-4.5")
print(len(tokens))   # the count
for t in tokens[:5]:
    print(t)         # token id and its string form

It is also reachable over gRPC at https://api.x.ai/xai_api.Tokenize/TokenizeText with the same two parameters. Being able to see the individual pieces, rather than only a total, is what makes this worth using during development: it shows you where a template is spending tokens, and it is how you find out that some formatting choice you made for readability costs a fifth of your prompt.

Two limits. It is a network call, so it is not something to run per-request in a hot path — cache counts for static content such as system prompts and tool definitions. And it counts text. It will not tell you what an image costs; only the usage field does that.

The number that was billed

The authoritative count is the one on the response. Every completion returns usage with prompt_tokens, completion_tokens, total_tokens and a prompt_tokens_details breakdown into text, audio, image and cached tokens. On reasoning models it also carries reasoning_tokens.

That is the number the meter used, and it includes things a tokenizer run over your prompt string cannot see: chat template scaffolding around each message, tool definitions, retrieved search results, and image tokens. If you count only the text you wrote, you will always be low, and the gap grows with exactly the features that make the request interesting.

If you stream, ask for it explicitly — stream_options with include_usage: true — or the usage chunk is never sent and you have no count at all for that request.

Budgeting across providers

The practical consequence for anyone comparing costs is that price per million tokens is not directly comparable between providers, because the tokens are not the same unit. A model at $2.00 per million that segments your particular text 15% more finely than a model at $2.20 is the more expensive of the two on that text.

• Compare on cost per request, not per token. Take a representative sample of real traffic, get a real count from each provider’s own counter, multiply by that provider’s price. That is the only comparison that survives contact with the tokenizer difference.
• Leave headroom in truncation logic. Any code that fits input to a window using an estimate needs a margin, and the margin should be larger for code and non-Latin text than for prose.
• Count static content once. System prompts and tool schemas do not change per request. Tokenize them at build time, store the number, and spend the network call on nothing.
• Reconcile against usage. Log your estimate and the billed prompt_tokens side by side for a week. The ratio between them, on your own text, is the only correction factor worth having — and unlike a published one, it is measured.