Why Amharic Text Costs More Tokens Than English

Amharic is written in a syllabary, and that changes the arithmetic in a way none of the alphabetic or abugida cases do. Each character is a whole syllable, so Amharic needs about half as many characters as a phonemic alphabet would — and each of those characters costs three UTF-8 bytes. The two effects partly cancel, and what is left is a coverage problem rather than an encoding one.

A grid of 275 syllables, three bytes each

The Ge’ez script, called fidel in Amharic, occupies U+1200 to U+137F with further characters in the Ethiopic Supplement and Extended blocks. The core of it is a grid: 33 base consonants multiplied by 7 vowel orders gives 231 distinct syllable characters, and the labialised series and a handful of additional consonants push the working set past 275. Every one of those code points is in the three-byte region of UTF-8.

This is a genuinely different structure from an abugida. In Bengali or Devanagari, a consonant-plus-vowel syllable is written as a consonant code point followed by a vowel-sign code point — two code points, six bytes. In Ge’ez the syllable ሰ, ሱ, ሲ, ሳ, ሴ, ስ or ሶ is one code point, three bytes. The syllabary is doing compression work that the abugida does not.

Fewer code points, wider code points

Work the trade through on the sentence used below. የኢትዮጵያ ተማሪዎች በትምህርት ቤት ውስጥ ናቸው። — “The students of Ethiopia are in school” — is twenty-six non-space characters. Romanised phonemically it would run to roughly fifty letters, because most of those twenty-six characters encode two phonemes. So the script buys back roughly a factor of two in character count and pays a factor of three in bytes per character. Net, on the encoding alone, Amharic sits at something like 1.5 times the byte cost of a Latin transcription of the same speech, not three times.

That is a meaningfully better starting position than Khmer or Burmese, which pay three bytes per code point and spend several code points per syllable. It is why the Amharic multiplier, derived below, lands lower than those two despite all three being three-byte scripts with small corpora. Byte width alone is not the predictor.

Two other parts of the block are worth knowing about because they are rarer than the fidel and therefore more expensive per character. The Ethiopic numerals ፩ to ፼, U+1369 to U+137C in the Unicode Ethiopic code chart, are their own code points with no positional value — they are letter-like numerals in the Greek and Roman tradition, so there is no digit-by-digit structure for a tokenizer to exploit and no arithmetic library that will parse them. And the traditional word separator ፡ (U+1361) is used in older and formal typography in place of the space character; text carrying it has no ASCII spaces at all, which removes the word-boundary anchor that byte-pair merges rely on most.

Deriving the multiplier

Amharic
  code points ............ 31  (26 Ethiopic incl. the '።' full stop
                                U+1362, plus 5 spaces)
  UTF-8 bytes ............ 83   (26 x 3, plus 5 ASCII spaces)

English "The students of Ethiopia are in school."
  characters ............. 39
  tokens (assumption: ~4 chars/token for English) ..... ~10

Ceiling (1 token per UTF-8 byte)
  83 / 10 = 8.3x English

Band    (assume 3 bytes/token, i.e. one token per fidel character --
         the realistic outcome for a vocabulary with a Ge'ez block
         but no Amharic word merges)
  83 / 3 = 28 tokens  ->  28 / 10 = 2.8x English

Band    (assume 6 bytes/token, i.e. two-syllable merges exist)
  83 / 6 = 14 tokens  ->  14 / 10 = 1.4x English

Derived, not measured, and the spread between those two bands is the whole story for Amharic. Whether your vocabulary has learned two-syllable merges decides whether Amharic costs you 1.4x or 2.8x. The measurement that tells you is per-syllable rather than per-sentence, because the question is about the grid:

import tiktoken
enc = tiktoken.get_encoding("o200k_base")

# Walk one consonant series across all seven vowel orders and ask
# whether each syllable is a single token or a pile of bytes.
series = "ሰሱሲሳሴስሶ"          # the 's' series
for ch in series:
    ids = enc.encode(ch)
    print(ch, hex(ord(ch)), len(ch.encode("utf-8")), "bytes",
          len(ids), "tokens")

# Then the same question for a whole word.
w = "ተማሪዎች"                 # "students"
print(w, len(w), "cp", len(w.encode("utf-8")), "bytes",
      len(enc.encode(w)), "tokens")

If each single syllable comes back as three tokens, the vocabulary has no Ethiopic entries at all and is at pure byte fallback: your real multiplier is near the 8.3x ceiling. If each comes back as one token, the block is covered and the word-level result tells you whether merges above the syllable exist.

Why a large symbol set defeats a merge table

A BPE vocabulary has a fixed size — roughly 100,000 entries in OpenAI’s cl100k_base and about double that in o200k_base, per the encodings published in OpenAI’s tiktoken repository. Those entries are allocated by frequency during training. A script with 26 base symbols spends 26 entries to reach full single-character coverage and can then spend everything else on merges. A script with 275 spends 275 before it has bought a single word-piece.

That is the structural reason a syllabary is hard for BPE specifically, and it applies regardless of corpus size. Combine it with Amharic’s actual corpus size — small, and much of it in image-only PDFs rather than machine-readable text — and the entries above the single-character level mostly do not get bought. Tigrinya shares the same script and the same grid with a smaller corpus still; see Tigrinya rather than assuming this page’s numbers carry over.

Consequences for Amharic products

• Amharic is cheaper than its script suggests. The 2.8x middle-of-band figure is well under Khmer’s or Burmese’s despite identical byte width. If you are triaging which languages to support, do not rank by Unicode block.
• The full stop is not a full stop. Amharic ends sentences with ። (U+1362), not with U+002E. A sentence splitter that looks for a period will return one enormous chunk for an entire Amharic document, which is a far more expensive mistake than the tokenizer penalty.
• Much Amharic source material is scanned. If your corpus comes from OCR, the token cost is the second problem; recognition of the fidel grid is the first. See OCR of Ge’ez-script documents.
• Test generation, not just comprehension. Models frequently read Amharic better than they write it, and the failure mode is producing a syllable from the right consonant series with the wrong vowel order — a one-code-point error that changes the word. See what LLM support for Amharic actually covers.