How Many Characters an AI Model Needs to Read a Given Language

“How many characters do you need to read Chinese?” has a published answer, and so does the same question for Korean, Japanese, Cherokee and Amharic. The numbers differ by three orders of magnitude — and they turn out to predict almost nothing about how well a model handles the language.

Two different questions

Keep these apart or every figure below becomes confusing. The first question is how many distinct symbols does the writing system use, which is a fact about the script. The second is how many code points does Unicode assign to it, which is a fact about the encoding and is usually a larger number, because Unicode encodes historical, regional and rarely used characters that no contemporary reader needs.

A third question — how many characters a model needs — turns out to be neither, and the last section explains why.

The inventories, with their sources

• Latin alphabet, English — 26 letters, two cases. Most European languages add accented letters: Spanish adds ñ, Polish adds nine accented letters, Icelandic keeps þ and ð.
• Greek — 24 letters. Cyrillic — 33 letters for Russian, with other languages adding or dropping letters from a wider repertoire.
• Hebrew — 22 consonant letters, five of which have distinct final forms, plus optional niqqud. Arabic — 28 letters, each with up to four contextual shapes, which is a rendering fact rather than an encoding one: the shapes are one code point apiece.
• Devanagari — roughly 48 base characters, 33 consonants and about 14 vowels, plus vowel signs and a large set of conjunct ligatures formed at rendering time.
• Thai — 44 consonant letters, about 15 vowel symbols and four tone marks.
• Cherokee — 85 syllabary characters, per the Unicode Cherokee chart; lowercase forms were added to Unicode in version 8.0 in 2015.
• Ethiopic — several hundred, built as base consonants times seven vowel orders. See how models handle syllabary scripts for the arithmetic.
• Japanese kana — about 46 base characters in each of hiragana and katakana, plus voicing marks and small forms.

Hangul: 24 letters or 11,172 blocks

Korean is the clearest case of the two-questions problem, and the arithmetic is exact rather than approximate. Hangul has 24 basic letters — 14 consonants and 10 vowels — which a learner can memorise in an afternoon. Those letters combine into syllable blocks, and Unicode encodes every possible block as its own precomposed code point:

19 possible initial consonants (choseong)
x 21 possible medial vowels     (jungseong)
x 28 possible final consonants  (jongseong, including "none")
= 11,172 syllable blocks

Unicode Hangul Syllables block: U+AC00 through U+D7A3 = 11,172 code points.

Both numbers are true and they answer different questions. A reader learns 24 shapes and a composition rule. A text-processing system that compares strings has to cope with 11,172 precomposed code points and with the decomposed jamo sequences that produce the same visible text — which is why Korean is the language where the difference between NFC and NFD bites hardest, and why two visually identical Korean strings so often fail an equality test. That specific failure is the subject of Korean NFC normalisation and string comparison.

Chinese: the count depends on what you are doing

There is no single number, but there are several published ones, each answering a defined question.

• 8,105 — the Table of General Standard Chinese Characters, published by China’s State Council in 2013. Its first level of 3,500 characters covers everyday use; the second and third levels add publishing and specialist characters.
• 2,136 — the jōyō kanji list for Japanese, in its 2010 revision by Japan’s Ministry of Education. This is the set taught through compulsory schooling and assumed by general-audience publishing.
• 20,992 — the code points in the base Unicode CJK Unified Ideographs block, U+4E00 to U+9FFF.
• Tens of thousands more — across the CJK extension blocks, which recent Unicode versions have pushed well past ninety thousand ideographs in total.

The practical answer for a reader is between the first two numbers: roughly three thousand characters carries a newspaper. The practical answer for a system is the last: you must be able to store and render anything, because a name, a place or a classical quotation will use a character outside every list.

That last point is not hypothetical, and it has a specific technical consequence worth checking in your own stack. The CJK extension blocks live outside the Basic Multilingual Plane, so those characters need surrogate pairs in UTF-16 and four bytes in UTF-8. Anything that assumes one code unit per character — an old database column type, a fixed-width field, a naive character-offset calculation — truncates them mid-character and produces a broken byte sequence rather than a shortened string. Japanese and Chinese personal and place names are the usual place this surfaces, because rare ideographs survive in surnames long after they leave general use, and a customer whose name your system cannot store will notice before you do.

Why inventory size barely matters to a model

Here is the part that overturns the intuition. A modern model does not have a character inventory. Byte-level tokenizers operate on UTF-8 bytes, of which there are 256 possible values, so every script is representable and none is impossible. Learning to read a script is not a matter of having enough character slots.

What varies is how efficiently a script is represented, and that is set by which byte sequences the tokenizer learned to merge, which is set by corpus frequency. Chinese, with thousands of characters, is handled well because there is a great deal of Chinese text. Cherokee, with 85, is handled poorly because there is very little Cherokee text. The inventory sizes are inverted relative to the outcomes, which is the cleanest available demonstration that inventory size is not the variable.

So the number to ask for is not the character count. It is the tokens per word your tokenizer produces for real text in that language, measured rather than assumed — and the reason Unicode’s coverage runs so far ahead of model coverage is exactly this: encoding a script is a committee decision, and learning one is a corpus problem. That divergence is the subject of why Unicode covers more scripts than AI models do.