How Many Tokens Is a Word? Real Ratios by Language

The internet’s answer is “a token is about 0.75 words, or about four characters”. That number is real, but it is a measurement of English prose through one family of tokenizers, and almost every way you can differ from that description makes it wrong in the expensive direction.

Where 0.75 came from

The figure originates in OpenAI’s own developer documentation, which has long offered “roughly 4 characters per token” and “roughly ¾ of a word” as helpers for English text, alongside a tokenizer playground for checking properly. It is a decent rule of thumb for what it describes: running English prose, ordinary punctuation, no code, no markup, encoded with a byte-pair vocabulary whose merges were learned mostly from English.

The reason it holds at all is that BPE spends its vocabulary budget on whatever was frequent in its training corpus. If that corpus is mostly English web text, most common English words earn a dedicated entry, and the compression ratio settles around four bytes per token. Change the corpus, the language, or the tokenizer and there is no mechanism keeping the ratio anywhere near four.

It is also worth noticing that “words” is a poor denominator to begin with. A word, operationally, is whatever falls between two spaces — which makes the quantity undefined for Chinese, Japanese and Thai, all of which are written without word spacing, and misleading for German or Finnish, where a single orthographic word can carry what English spreads across five. Characters per token is the more portable statistic, and bytes per token is the most portable of all, because it is the quantity the tokenizer is actually compressing. Quote the ratio you measured in the unit you measured it in.

What moves the ratio

For the size of the cross-language spread, the honest source to cite is the published work rather than anyone’s blog estimate. Petrov, La Malfa, Torr and Bibi’s Language Model Tokenizers Introduce Unfairness Between Languages (NeurIPS 2023) compared tokenized lengths of parallel corpora across many languages and reported that the same content, translated, can differ in token length by more than an order of magnitude between the best- and worst-served languages. That is the number to plan around: not “a bit more”, but potentially several times more.

Measuring it on your own corpus

The ratio you need is not a published constant, it is a property of your text and your tokenizer, and it takes about a minute to get:

import glob, statistics
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("<the model you will actually call>")

ratios = []
for path in glob.glob("samples/*.txt"):
    text = open(path, encoding="utf-8").read()
    n_tok = len(tok.encode(text, add_special_tokens=False))
    n_word = len(text.split())
    ratios.append((path, n_tok, n_word, n_tok / max(n_word, 1),
                   len(text) / max(n_tok, 1)))

for path, n_tok, n_word, tpw, cpt in sorted(ratios, key=lambda r: -r[3]):
    print(f"{path:40s} {n_tok:7d} tok  {tpw:5.2f} tok/word  {cpt:5.2f} char/tok")

print("p95 tokens per word:",
      statistics.quantiles([r[3] for r in ratios], n=20)[-1])

Sample per language and per document type, not in aggregate. The average across a mixed corpus is the least useful statistic available, because the thing you are protecting against is the worst document, not the median one. Take the p95 and budget with that.

Include the shapes you would not think to sample. Code blocks inside prose, tables pasted from a spreadsheet, a support transcript full of order numbers, a document that arrived as OCR with stray characters throughout — these are the documents that break a ratio, and they are all real inputs in a product that accepts uploads. If your corpus sample contains only clean articles you have measured a corpus you will never receive.

Planning with a ratio you do not trust

Two rules make the uncertainty survivable. First, use a ratio for capacity planning and an exact count for admission control. Estimating is fine when you are deciding how much context a feature can afford in principle; it is not fine as the check immediately before a request, where you should tokenize properly and reject or trim on the real number.

The size of the mistake is easy to underestimate, so it is worth costing once. Suppose you size a summarisation feature on the rule of thumb: 90,000 words of source material, at 0.75 words per token, is a predicted 120,000 tokens, which fits a 128k window with room to spare. If the real corpus turns out to encode at 2.0 tokens per word — a plausible figure for an inflected non-English language through an English-heavy tokenizer — the same material is 180,000 tokens. The feature does not degrade, it does not cost 50% more; it fails outright with a context-length error on the documents that matter most, and it does so only for the locales you tested least.

Second, keep the margin proportional to how far your text is from the rule’s conditions. English marketing prose through the tokenizer it was tuned on needs very little. A multilingual product where the same 128k-token window has to hold Hindi or Japanese source documents needs a lot, and the failure is not a rounding error — it is a request that will not fit at all. That mechanism is the subject of the language tax page, and the family-by-family differences are on the tokenizer comparison.