How AI Models Handle Syllabary Scripts Like Cherokee and Ge'ez

A syllabary should be efficient. One symbol per syllable means fewer characters than an alphabet for the same speech, without the thousands of characters a logography demands. In practice Cherokee and Ge’ez cost more tokens per word than English does, and the reason has nothing to do with the script’s design.

What a syllabary is, and what an abugida is

A true syllabary gives each syllable its own symbol, with no systematic relationship between the symbols for syllables sharing a consonant. Sequoyah’s Cherokee syllabary, devised in the 1820s, is the textbook case: 85 characters, each a whole syllable, and the shapes for ga, ge and gi are unrelated to one another. Japanese kana is the other well-known example, with two parallel sets of roughly 46 base characters each.

Ge’ez script — used for Amharic, Tigrinya, Tigre and liturgical Ge’ez — is technically an abugida, and the difference matters. Each symbol is still a syllable, but the symbols are systematic: a base consonant shape is modified in a regular way to indicate each of seven vowel orders. A reader who knows the base shape and the seven modifications can read a syllable they have never seen. Devanagari, Tamil, Thai and Khmer are also abugidas, built on the same consonant-plus-vowel-mark principle with different mechanics.

The relevant distinction for a model is not the linguistic label. It is whether the writing system’s internal regularity is visible in the encoding — and for Ge’ez it is not, because Unicode encodes each consonant-vowel combination as its own atomic code point rather than as a base plus a modifier. The systematic structure a human reader exploits is invisible to a byte-level tokenizer.

The inventory arithmetic

The size of a syllabary is a product, and the product is worth showing because it explains where each script lands:

Cherokee    85 symbols          (fixed inventory, Sequoyah's design)
Ge'ez core  26 consonants x 7 vowel orders = 182 base syllables
Amharic     33+ consonants x 7 orders      = 230+, plus labiovelars
Kana        ~46 base symbols x 2 sets, plus voicing marks and small forms

A few hundred, in every case. That sits neatly between an alphabet’s few dozen and the thousands of characters in everyday Chinese, which is exactly what the design intends. The Unicode charts are the place to confirm the encoded totals: the Cherokee block and the Ethiopic block published by the Unicode Consortium, with supplementary and extended blocks added in later versions for additional letters.

The token-efficiency profile

Now the mechanism. Written text length in characters follows the symbol-to-sound ratio: a syllabary needs one character where an alphabet needs two or three, so an Amharic word is shorter in characters than its English translation. That is the theoretical advantage.

What a model is charged for is tokens, and tokens come from byte-pair merges over UTF-8. Cherokee and Ethiopic characters both sit well above the Latin range, so each one costs three bytes in UTF-8. Three tiers of outcome are possible, and which one you get depends entirely on the tokenizer’s training corpus:

• Merges cover common syllables. One character becomes one token. The script’s density advantage survives and text is genuinely cheap per unit of meaning — this is roughly where Japanese kana sits.
• Merges cover some characters. Common characters are one token; rarer ones split into two or three byte fragments. Cost lands somewhere above Latin.
• No merges at all. Every character is three separate byte tokens. A five-syllable Amharic word costs fifteen tokens where its English translation costs two or three. This is the common case for Cherokee.

Why the theoretical advantage collapses

Because merges are learned by frequency, and these scripts are rare in the corpora tokenizers are trained on. Cherokee has a few thousand fluent speakers and a correspondingly tiny volume of digital text; Amharic and Tigrinya have tens of millions of speakers but far less web-published text than that population suggests. The tokenizer sees too little of either script to justify spending vocabulary slots on it, so it does not, and the script pays the full byte cost forever afterwards.

Note what this means: the penalty is a property of the tokenizer, decided before training, and it is not fixable by prompting, by a better model of the same family, or by anything you do at call time. It also compounds — more tokens per word means less content fits in the context window, higher cost per request, and, because the model sees fragmented rather than meaningful units, weaker representations of the words themselves. The same three-way squeeze is worked through for Amharic on the Amharic token cost page and named as a general mechanism on the tokenizer vocabulary bottleneck.

What this means in practice

Three things follow that are worth acting on.

Budget by measurement, not by character count. Run a representative page of your actual text through the tokenizer you will be billed by and divide by the word count. That single number tells you which of the three tiers you are in, and it is the only number that matters for cost and context planning. It will differ between providers, sometimes substantially.

Expect chunking to misbehave. If characters are splitting into byte fragments, a character-based chunk size and a token-based one diverge by a factor of three or more, and a fixed-token chunker will produce chunks holding a third of the text you expected. Size chunks by tokens and verify the resulting chunks contain whole sentences.

Watch what happens at the edges of a fragmented character. When a character is three byte tokens rather than one, several operations that feel atomic stop being atomic. Streaming output can deliver a partial character in one chunk, so a naive client that decodes each chunk independently prints a replacement character and then corrects itself — visible flicker in a chat interface, and corrupt data if you are writing chunks to a file. Truncating a response at a token limit can cut mid-character and leave an invalid trailing byte sequence. Both are worth testing explicitly in these scripts, because neither ever occurs in ASCII and so neither is exercised by an English test suite.

Do not read script family as a support signal. Kana and Ethiopic are both syllabic and their model support is nothing alike, because Japanese has vastly more training text than Amharic. Corpus volume predicts quality; script structure predicts character count. Confusing the two is the same error the reading direction is not a difficulty predictor page argues against for direction.