Why Chinese Text Costs More Tokens Than English

The commonly repeated figure is that Chinese costs two to three times more tokens than English. That number is either right or badly wrong depending on whether you measure per character or per sentence, and nobody who quotes it says which.

Start with the bytes, which are exact

Take one sentence and label it, because an unlabelled example is what makes these ratios untrustworthy.

人工智能模型的定价通常按照令牌数量计算，而不是按照字符数量计算。
"AI model pricing is usually calculated by token count rather than by character count."

The Chinese line is 32 characters: 30 Han ideographs plus a fullwidth comma (U+FF0C) and an ideographic full stop (U+3002). Every one of those code points sits in the range U+0800–U+FFFF, which UTF-8 encodes in three bytes. The sentence is therefore exactly 96 bytes. The English line is 85 characters, all ASCII, so exactly 85 bytes. Those two numbers are not estimates and you can verify them with len(s.encode("utf-8")).

Now the floor. In a byte-level BPE tokenizer, no string can ever cost more than one token per byte, because the base vocabulary already contains all 256 bytes. So the Chinese sentence is somewhere between 1 and 96 tokens and the English is somewhere between 1 and 85. The interesting question is where in that range each lands, and the answer is decided entirely by the merge table.

What the merge table can and cannot do

For English, the merge table is doing almost all the work. Common words like model, pricing and count appeared often enough in the training corpus that their bytes were merged all the way up into one token each, usually with the leading space attached. OpenAI’s own guidance has long used the rule of thumb that English averages about four characters per token, which for 85 ASCII characters puts this sentence near 21 tokens — roughly four bytes per token.

For Chinese, the merge table faces a much harder problem. A Han ideograph is three bytes, and those three bytes have to be merged twice before the character is one token: byte one with byte two, then the pair with byte three. That happens only for ideographs frequent enough in the corpus to earn two merge slots. Frequent characters such as 的, 是 and 不 are essentially certain to have them. Rarer ones do not, and fall back to two tokens or three. A handful of extremely common two-character words — 中国, 可以, 因为 — may be merged further into a single token.

So the derivation, with the assumption stated plainly: if you assume most characters in ordinary modern Mandarin prose are common enough to be one token and a minority cost two, the 32-character sentence lands somewhere in the region of 32 to 48 tokens. That is a derived range from a stated assumption about merge coverage, not a measurement. The bound that holds unconditionally is the one from the bytes: it cannot exceed 96.

The multiplier that is not on your invoice

Here is where the popular figure goes wrong. Per character, Chinese is dramatically more expensive: about one token per character against English’s one token per four characters, so a factor of roughly three to four. If you have a fixed number of characters — a column in a database, a UI string budget — that is the number that matters.

But you are not billed per character. You are billed per token for a piece of meaning, and Chinese needs far fewer characters to carry the same meaning. The sentence above says the same thing in 32 characters that English needs 85 characters to say. Divide it out: if the Chinese is around 32–48 tokens and the English around 21, the sentence-level multiplier is roughly 1.5× to 2.3× — much less alarming than the per-character figure, and the one that appears on the bill.

The gap between those two numbers is the entire content of this page. A per-character multiplier of four and a per-sentence multiplier of two are both true statements about the same sentence, and citing either without the other produces a wrong budget. Chinese is the language where the two diverge most, because logographic writing is unusually dense per character and byte-level BPE is unusually bad at it. Korean, which is phonetic and therefore uses several syllable blocks where Chinese uses one character, does not get the same discount — see why Hangul costs more than Han despite both being three-byte scripts.

Measuring it on your own corpus

The ratio for your text depends on your text: classical quotations, proper nouns, technical terminology and traditional characters all shift it, because all of them are rarer in the tokenizer’s training corpus than everyday prose. Run it rather than assuming it.

import tiktoken

enc = tiktoken.get_encoding("o200k_base")

zh = "人工智能模型的定价通常按照令牌数量计算，而不是按照字符数量计算。"
en = "AI model pricing is usually calculated by token count rather than by character count."

for label, s in (("zh", zh), ("en", en)):
    n = len(enc.encode(s))
    b = len(s.encode("utf-8"))
    print(label, "chars", len(s), "bytes", b, "tokens", n,
          "bytes/token", round(b / n, 2), "chars/token", round(len(s) / n, 2))

Two numbers to read from that output. chars/token tells you how well the merge table covers your character inventory — anything near 1.0 means most characters are single tokens, anything near 0.4 means many are falling back to two or three. bytes/token tells you how far you are from the byte floor: a value of 1.0 means the tokenizer has given up entirely and is emitting raw bytes, which is what you would see for rare or archaic characters.

What this does to a 4,000-token budget

Concretely, for retrieval. If you size RAG chunks at 4,000 tokens and assume the English rule of four characters per token, you have budgeted for about 16,000 characters. Feed Chinese into the same setting and at roughly one token per character you get about 4,000 characters — a quarter of the text you expected, in a script where 4,000 characters is nevertheless a substantial passage. The chunk is not too small in meaning; it is too small relative to what the pipeline was tuned for, and if the chunker splits on character count rather than token count you will overflow the budget instead.

• Chunk on tokens, never characters. A character-based splitter tuned on English will cut Chinese chunks that are four times over budget. See chunk sizing in tokens for CJK.
• Do not split on spaces. Written Chinese has no inter-word spaces, so every whitespace-based splitter degenerates to splitting on punctuation only, producing wildly uneven chunks.
• Expect the output side to be worse. Output tokens are priced several times higher than input tokens by most providers, so a multiplier on generated Chinese hits harder than the same multiplier on prompt text.
• Traditional characters are not free. They occupy the same three-byte ranges but appear far less often in most training corpora, so merge coverage is thinner and the per-character token count runs higher than for simplified text.