What Actually Changes for AI Between Logographic and Alphabetic Scripts

“Logographic scripts are different” is not actionable. Two mechanisms are: one character of Chinese carries roughly a morpheme where one character of English carries a phoneme, and the encoding and tokenization stack prices those two characters very differently. Almost every practical consequence falls out of those two facts.

Mapping density: what one character buys you

In an alphabetic script, a character maps to a sound, and meaning emerges only at the word level. In a logographic script, a character maps to a morpheme — a unit of meaning — and usually also to a syllable. The Chinese character for “electricity” is one character; it is also the first element of the words for telephone, computer, film, lift, battery and email. Meaning is compositional at the character level in a way it simply is not in English.

The measurable effect is that the same content takes far fewer characters in Chinese than in English. A sentence that runs sixty Latin characters commonly runs fifteen to twenty Chinese characters. This is why character limits transfer so badly: a 280-character limit is a couple of sentences in English and a paragraph in Chinese, and a 140-character SMS field holds substantially more information in Chinese — but only if the transport is Unicode, because at 16 bits per character the SMS segment length drops to 70.

It is also why “logographic” is a slight overstatement for Chinese. Most characters are phono-semantic compounds: one component hints at meaning, the other at pronunciation. That structure is invisible to a model that sees only code points, which is one reason rare-character handling is weaker than character frequency alone would predict — the internal structure a human reader uses to guess an unfamiliar character is not available in the representation.

The UTF-8 tax, and why it is not a rounding error

Modern tokenizers are byte-level: they run byte-pair encoding over UTF-8 bytes rather than over characters, which is what lets them represent any text at all without an unknown-token escape hatch. UTF-8 encodes ASCII in one byte per character and CJK ideographs in three.

"cat"  -> 63 61 74                     3 bytes
"猫"    -> E7 8C AB                     3 bytes
"electricity" -> 11 ASCII bytes        11 bytes
"电"    -> E7 94 B5                     3 bytes

So before any merge rules are applied, a Chinese character starts at three bytes against an English character’s one. The merges are what decide whether that becomes a penalty or not. A tokenizer trained on a corpus with substantial Chinese learns merges that fold common characters and character pairs into single tokens, and the density advantage from the previous section can then outweigh the byte cost. A tokenizer trained on a corpus with little of a given script learns no such merges, and every character in that script costs its full three bytes as three separate tokens.

That is the actual variable, and it is a property of the tokenizer rather than of the script. It is why the same Chinese paragraph can cost meaningfully different token counts across two providers, and why per-script token cost has to be measured against the specific tokenizer you are billed by rather than assumed. The per-language numbers are worked through on the Chinese token cost and Japanese token cost pages.

Tokenization granularity is where they diverge

Put the two mechanisms together and the granularity mismatch appears. For English, a well-trained BPE vocabulary lands at roughly a token per short word: the token boundary sits somewhere near the word boundary, which is a linguistically meaningful place. For Chinese, a token often lands at one character — which is a morpheme, so also meaningful — but for a script the tokenizer has seen little of, tokens land mid-character, on byte fragments that correspond to nothing at all.

This gives three tiers rather than two, and the tiers are the useful mental model: tokens aligned to words, tokens aligned to morphemes, and tokens aligned to nothing. The third tier is where quality falls off, and script family is a poor predictor of which tier you are in. Corpus volume is a good one.

No spaces, and what depends on them

Chinese and Japanese are written without spaces between words, and so are Thai, Khmer, Lao and Burmese — which are alphabetic or abugida-based, so this is not a logographic property either. What matters is that any code path which assumes whitespace-delimited words silently produces one enormous token for a whole sentence, or nothing.

• Word counts report 1 for a full paragraph, or report the character count, depending on the library.
• Line breaking requires the Unicode line-breaking algorithm rather than breaking at spaces; browsers implement it, PDF generators and canvas-based renderers frequently do not.
• Lexical search needs a segmenter before indexing. Without one, BM25 over Chinese either indexes whole sentences or single characters, and neither retrieves well.
• Chunking for retrieval must split on punctuation and token counts rather than on spaces — the subject of chunking Chinese text for RAG.

What follows in a real pipeline

The short version: stop reasoning in characters. Character counts mean different amounts of information per script, character-based limits truncate mid-meaning, and character-based chunking cuts inside a word in one script and inside a morpheme in another. Token counts are at least consistent with what the model charges you and what it can hold.

And treat script family as a description, not a diagnosis. When something works poorly in Chinese and well in English, the cause is almost never “because it is logographic”. It is a missing segmenter, a tokenizer with thin coverage, a character-based length limit, or a font stack — four concrete things, each of which you can check. The character-inventory side of the story, including how many characters a reader of each script actually needs, is on how many characters an AI model needs to read a language.