What Is a Token? The Unit You’re Actually Billed In

Every model price you will ever be quoted is per million tokens. Before you can predict a bill, or explain one, you need to know what one of those is — and it is not a word, not a character, and not the same thing from one model family to the next.

What a token actually is

A token is an entry in a fixed vocabulary that was built once, before training, and frozen. The tokenizer takes your string, splits it into pieces that all exist in that vocabulary, and replaces each piece with its integer id. The model never sees text. It sees a list of integers, and it emits integers, which the same vocabulary turns back into text on the way out.

Vocabularies in current model families run from about 32,000 entries to about 256,000. The entries are not words. Common English words are usually a single entry; rarer ones are assembled from two or three fragments; and — the detail that surprises people first — the leading space is normally part of the token. In most byte-level BPE vocabularies " free" and "free" are two different tokens with two different ids, which is why a stray double space is not free and why prompts that end with a trailing space sometimes behave oddly.

Seeing the split

The fastest way to stop guessing is to print the pieces. Every tokenizer on Hugging Face will show you them:

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("<any model repo id>")
s = "Tokenization isn't free, and unhelpfully it isn't uniform."

ids = tok.encode(s, add_special_tokens=False)
print(len(ids), "tokens for", len(s), "characters")
print(tok.convert_ids_to_tokens(ids))

Run that on the same sentence against two different families and you will get two different counts. Run it on a paragraph of code, then on a paragraph of prose of identical length, and you will get two different counts again. This is the single most useful ten seconds of work available before you write a cost model.

From characters to cents

Here is one request, costed end to end. The shape is a retrieval-backed support assistant; the token counts are the assumption, and you would replace them with your own:

Input is 2,050 tokens, output is 250. At a hypothetical $3.00 per million input tokens and $15.00 per million output tokens — a ratio in the range large models are commonly priced at, though you must take the actual figures from the provider’s own pricing page on the day — that is:

input   2,050 / 1,000,000 x $3.00  = $0.00615
output    250 / 1,000,000 x $15.00 = $0.00375
                                     --------
per request                          $0.00990

40,000 requests / month              $396.00

Two things fall out of that table immediately and neither is obvious from the price sheet. First, the 250 tokens of output cost nearly as much as the 2,050 tokens of input — output is where the money goes, and “be concise” is a budget instruction. Second, 1,500 of the 2,050 input tokens are byte-identical on every single request, which is exactly the shape prompt caching was built for.

Why not words, why not characters

Both alternatives were tried and both lose. Character-level models have a tiny vocabulary and no unknown inputs, but they make sequences four to five times longer, and since attention cost grows with the square of sequence length that is a catastrophic multiplier. Word-level models make sequences short but need an unbounded vocabulary: every typo, every product code, every name you have not seen becomes an unknown token, and the model is blind to it.

Subword tokenization is the compromise that won. Frequent strings get their own entry so common text is compact; anything unseen decomposes into smaller pieces, and in byte-level schemes it decomposes all the way down to raw bytes, so there is no such thing as an out-of-vocabulary input. Feed a modern tokenizer an emoji, a Klingon transliteration or a base64 blob and it will encode all of them — expensively, but without failing.

One consequence of the vocabulary being frozen before training is that it cannot learn new words afterwards. A term coined after the tokenizer was built — a product name, a new framework, a recent piece of jargon — has no entry, so it is assembled from fragments every time it appears. If that term is central to your domain you are paying a small tax on every mention of it, and there is nothing to be done about it except notice it when you compare families.

The frozen vocabulary also explains an oddity of streaming. A single token can be a fragment of a UTF-8 character, so a streaming client cannot always decode a token the instant it arrives — it has to buffer until the bytes form a valid character. That is why streamed output sometimes appears in ragged chunks rather than smoothly, and why a naive client that decodes each token independently emits replacement characters in the middle of non-Latin text.

Five things that follow

• Counts are model-specific. “How many tokens is this?” has no answer without naming the tokenizer. A migration between families changes your bill even at identical per-token prices.
• Formatting costs money. Pretty-printed JSON, markdown tables and deep indentation are all paid for at the same rate as content. Minified JSON in a prompt is a real, if small, discount.
• Numbers tokenize badly. Long digit strings get split into groups that do not line up with place value, which is part of why models are unreliable at arithmetic they were not given a tool for.
• You cannot count with len(). Not on characters, not on split(). Estimating by bytes is fine as a pre-filter and useless as an admission check.
• The limit and the price use the same unit. Which means every cost decision and every truncation decision are the same decision, made against the same budget.