Why Unicode Covers More Languages Than Any AI Model Does

Any modern system can store and display text in essentially every writing system humans use. Almost none can understand more than a few dozen languages. That gap is not an oversight; it follows from the fact that encoding a script is a fixed, finite job and learning a language is not.

The two numbers

On one side: Unicode 16.0, published by the Unicode Consortium in September 2024, encodes 168 scripts, covering — by the Consortium’s stated goal — the writing systems of the world, modern and historic. Between them those scripts can write the great majority of the roughly 7,100 living languages that have a written form at all, per Ethnologue’s 27th edition (2024). Any conforming system can store, transmit and render all of it.

On the other: the officially supported language lists on current model cards run to the tens. Meta’s Llama 3.1 card names eight officially supported languages while acknowledging broader training exposure; other vendors publish lists in the tens rather than the hundreds. The broadest published multilingual evaluation set, FLORES-200, covers 200 languages — and that is a translation benchmark, which is the most permissive measure of support there is.

So the honest framing of the gap: text that can be typed and stored spans thousands of languages; text that a model was evaluated on spans tens; and between them sits a band of a few hundred languages where models will produce fluent-looking output that nobody has checked. That middle band is the dangerous one, and it is sized in the count of languages with no AI support.

Why encoding is cheap and understanding is not

The asymmetry is structural, and stating it precisely explains why the gap will not close by effort alone.

Encoding a script is a bounded task with a finite deliverable. You determine the character inventory, assign code points, specify the properties — directionality, combining behaviour, case mapping, line breaking — and write the shaping rules. The work is substantial, it is done once, and it is done by a committee working with scholars and user communities. When Unicode adds a script, the script is added permanently. Nothing about the number of people who write it affects whether it can be encoded, which is why scripts with a few thousand living users are encoded alongside Latin.

Learning a language is unbounded and data-driven. A model does not acquire a language from a specification; it acquires it from text, in quantity, and the quantity required is enormous. There is no committee that can decide a language is supported. If the text does not exist, the capability does not exist, and the amount of text available is a fact about the world’s publishing and internet history rather than about anyone’s intentions.

This is why the two curves have different shapes. Unicode coverage grows steadily toward completeness because each addition is a finite project. Model coverage plateaus, because each additional language is harder than the last — the languages remaining are exactly those with the least digital text, which is why they were not already covered. The consequences of that plateau are traced in languages missing from training corpora.

The tokenizer sits between them

There is an intermediate layer that neither number describes, and it is where the gap first becomes visible in a bill.

Modern tokenizers are byte-level or have byte fallback, so they can encode any Unicode text without failing. A script the tokenizer has never seen does not produce an error; it produces a lot of tokens — one per UTF-8 byte, and characters outside the Basic Multilingual Plane take four bytes each. Text in an unsupported script therefore costs several times more per character than English while conveying less to the model, because the byte tokens carry no learned structure.

Petrov et al. document token-count differences of up to roughly 15× across languages for equivalent content in “Language Model Tokenizers Introduce Unfairness Between Languages” (NeurIPS 2023), arXiv 2305.15425. The practical effect is a three-layer gradient rather than a binary: fully supported languages are cheap and accurate, partially covered languages are expensive and unreliable, and uncovered scripts are very expensive and produce nothing useful. The middle layer is where most surprise costs come from.

What the gap does to a product

• Your input validation is far more permissive than your model. A form accepting UTF-8 accepts every script Unicode encodes. Nothing downstream checks whether the model can read it, so the failure surfaces as a bad answer rather than as a rejected input.
• Storage and display succeed while comprehension fails. The user sees their text rendered correctly, which is a strong signal that the system understands it. It does not.
• Cost scales inversely with support. The users you serve worst are the users you pay most to serve, because the token-inflation gradient runs the same direction as the quality gradient.
• Silence is the default failure. There is no “unsupported language” error in any major API. The model answers, in the language, plausibly. See how a model behaves on a language it does not know.

Designing for the gap

1. Detect script at the boundary and record it. The Unicode Script property is deterministic and requires no model. Storing it per record costs nothing and is the only reliable signal you will have about what you are holding.
2. Keep an explicit list of languages you claim to support and treat it as a product decision rather than a model property. Your list should be shorter than the model card’s, not longer.
3. Degrade visibly outside the list. Tell the user the language is not fully supported and offer an alternative. A stated limitation is survivable; a confident wrong answer in someone’s language is not.
4. Never let unsupported input fail closed at storage. Store the original bytes intact regardless. When support later arrives, historical data that was mangled on write cannot be recovered, and this is the one decision here that is irreversible.
5. Measure token inflation per script and put it next to your cost dashboard. It is the earliest quantitative warning that a language is outside the model’s competence, and it is available before any quality complaint arrives.