Why Tigrinya Text Costs More Tokens Than English

Tigrinya is written in Ge’ez script, which sits in the Unicode range U+1200–U+137F. Every character in that range takes three bytes in UTF-8, and a byte-level tokenizer that has no merges for a sequence falls back to emitting those bytes individually. That is the whole of the cost mechanism. What is specific to Tigrinya is how much of the script the tokenizer has actually seen.

What one Tigrinya glyph costs in bytes

UTF-8 encodes code points below U+0080 in one byte, below U+0800 in two, and below U+10000 in three. The Ethiopic block starts at U+1200, so every Tigrinya letter is three bytes without exception — including the word-space ፡ (U+1361) and the full stop ። (U+1362), which are Ethiopic punctuation and not the ASCII characters they resemble. The encoding rule is in RFC 3629, and the block itself is charted by the Unicode Consortium at U+1200 Ethiopic.

A byte-level BPE vocabulary always contains all 256 single bytes, because that is what makes it able to encode anything at all. So the worst case for any string is exactly its UTF-8 byte count: one token per byte. The best case is one token for the whole string. Real behaviour sits between the two, and where it sits depends entirely on how many Ge’ez byte sequences earned a merge during vocabulary training.

Deriving the multiplier

Take a plain Tigrinya sentence and its English gloss.

Tigrinya : ትግርኛ ቋንቋ ናይ ኤርትራን ትግራይን እዩ።
English  : Tigrinya is the language of Eritrea and Tigray.

Tigrinya : 22 Ethiopic letters + 1 Ethiopic full stop = 23 × 3 bytes = 69
           5 ASCII spaces                             =  5 × 1 byte  =  5
                                                        total        = 74 bytes

English  : 46 ASCII characters                        = 46 bytes

Now the two bounds. For English, OpenAI publishes a rule of thumb of roughly four characters per token on ordinary English prose — the figure behind its own tokenizer page. Forty-six characters is therefore about twelve tokens. For Tigrinya, the upper bound is 74 tokens, one per byte, which would be the count if the vocabulary contained no Ge’ez merges whatsoever. The lower bound, if every one of the 22 letters had earned its own merge, is around 24.

So the derived range is 24 / 12 = 2.0x at best and 74 / 12 = 6.2x at worst, for this sentence, against this English baseline. That is arithmetic, not a measurement. Nobody ran a tokenizer to produce those two numbers, and the reason they are given as a range rather than a value is that the position within the range is exactly the thing that differs between vocabularies.

Why Amharic’s number is not Tigrinya’s

Tigrinya and Amharic share a script and roughly nothing else about their position in a training corpus. Amharic has several times as many speakers, is a working language of the Ethiopian federal government, and carries the great majority of the Ge’ez-script text that exists on the public web. A tokenizer trained on web text learns Ge’ez merges from Amharic, and Tigrinya inherits whatever those merges happen to cover.

Overlap is high but it is not total, and the gap is visible in the Unicode chart. Tigrinya uses the ቐ series (U+1250 and the six vowel forms that follow it) for the ejective q̣ sound. Standard Amharic orthography does not use that series. Those code points exist in the block, and they appear in Tigrinya running text, and an Amharic-derived merge table has never had a reason to learn a single sequence containing them.

The consequence is not uniform degradation across a Tigrinya document. It is patchy: words built from letters Amharic also uses tokenize reasonably, and words containing a Tigrinya-specific series drop to near byte-per-token. A per-document average hides that. If you are budgeting context for Tigrinya, the words that blow the budget are predictable from their spelling.

The syllabary gives some of it back

Three bytes per character sounds like a straight tripling, and it is not, because a Ge’ez character is not a letter in the Latin sense. It is a syllable: a consonant and its vowel in one glyph. ት is not t, it is tə. So a Tigrinya word carries more phonemes per character than an English word does, and the character count of a Tigrinya sentence is well below the character count of its English translation — 22 against 46 in the example above.

That is why the derived worst case is around 6x rather than the 10x or more you get from an Indic abugida, where vowel signs and viramas are separate code points that inflate the count. The Ethiopic script is dense in information per code point. It is the byte width and the missing merges that undo the advantage, not the script’s design. The same argument applies to Amharic, which is why Ge’ez-script languages land in a different band from the Indic scripts.

Measuring it on your own text

The derivation above bounds the answer. Only your corpus gives the real one, and it takes a few lines. This prints the byte count, the token count and the implied bytes-per-token for each line of a file, which is the number that tells you whether the vocabulary is helping at all — a value near 1.0 means byte fallback.

import tiktoken

enc = tiktoken.get_encoding("o200k_base")

for line in open("tigrinya.txt", encoding="utf-8"):
    line = line.strip()
    if not line:
        continue
    b = len(line.encode("utf-8"))
    t = len(enc.encode(line))
    print(f"{b:5d} bytes  {t:5d} tokens  {b / t:4.2f} bytes/token  {line[:40]}")

Sort the output by bytes-per-token and read the worst lines. If the bottom of that list is dominated by words containing the ቐ series, the argument on this page is doing what it claims. tiktoken is OpenAI’s own library, so it gives you that vendor’s answer; other providers ship their own tokenizers and the numbers will differ.

For a budgeting rule, the practical consequence is this: a 4,000-token context window that holds roughly 3,000 English words holds somewhere between 700 and 1,500 words of Tigrinya on the derivation above. If you are chunking Tigrinya documents for retrieval, size the chunks in tokens rather than characters, or a chunk that is comfortable in English will silently overflow.