Why Armenian Text Costs More Tokens Than English

If script rarity predicted token cost, Armenian and Georgian would land in the same place. Both are ancient alphabets used by one language each, both have small web corpora, neither has a high-resource neighbour sharing its code points. They do not land in the same place, and the reason is a decision made in the Unicode code space long before any tokenizer existed.

Two bytes per letter, and why that is lucky

The Armenian block is U+0530 to U+058F. That is below U+0800, which is the boundary at which UTF-8 switches from two-byte to three-byte sequences. Every Armenian letter therefore costs two bytes. Georgian starts at U+10A0, comfortably above that boundary, so every Georgian letter costs three.

Nothing about the languages caused this. The early Unicode allocation put Latin, Greek, Cyrillic, Hebrew, Arabic, Syriac, Thaana and Armenian into the space below U+0800, and everything else went above it. Armenian made the cut; Georgian did not. It is an accident of 1990s standards work and it is worth a permanent one-third discount on every Armenian byte that will ever be transmitted.

Armenian is bicameral, with 38 letters in each case, and mostly writes with precomposed characters and no combining marks per the Unicode Armenian code chart. There are a few ligature code points in the U+FB13 to U+FB17 range, which well-formed modern text does not use but which can appear in material extracted from older documents.

Armenian carries one complication that has no equivalent in Georgian and that works directly against it in a merge table. There are two living orthographies: the classical Mesropian spelling used by the Western Armenian diaspora, and the reformed spelling adopted in Soviet Armenia in the 1920s and used in Eastern Armenian today. They differ in the spelling of very common words, not in rare ones. The practical effect on a tokenizer is that Armenian’s already small corpus is split between two spellings of the same lexicon, so each variant occurs at roughly half the frequency it otherwise would, and merges are bought by frequency. Georgian, with one orthography, concentrates its whole corpus on one set of byte sequences.

The comparison that breaks the rarity rule

Put the two head to head on the sentences used in this cluster. Armenian: Հայերեն գրականությունը հարուստ պատմություն ունի։ — “Armenian literature has a rich history.” Georgian: ქართული ენა უნიკალური ანბანით იწერება.

                        Armenian    Georgian
  code points ......... 48          38
  UTF-8 bytes ......... 92          104
  bytes per code point  1.92        2.74

Armenian's script is 30% cheaper per character. But the Armenian
sentence uses 26% more characters to say a comparable amount, so
the sentence-level byte counts come out within 12% of each other.

Conclusion: the per-character advantage is real and the
sentence-level advantage is much smaller, because average word
length in characters moves in the opposite direction.

That is the answer to the question this page is built on. Script rarity does not predict the multiplier on its own. Three things do, and they are independent of one another: bytes per code point, code points per word, and how many merges the vocabulary bought for the block. Armenian wins the first, loses the second, and ties the third.

Deriving the multiplier

Armenian
  code points ............ 48   (43 letters, 4 spaces, 1 verjaket '։')
  UTF-8 bytes ............ 92   (43 x 2, plus the 2-byte U+0589,
                                 plus 4 ASCII spaces)

English "Armenian literature has a rich history."
  characters ............. 39
  tokens (assumption: ~4 chars/token for English) ..... ~10

Ceiling (1 token per UTF-8 byte)
  92 / 10 = 9.2x English

Band    (assume 2 bytes/token, i.e. one token per Armenian letter --
         the vocabulary covers the block and nothing above it)
  92 / 2 = 46 tokens  ->  46 / 10 = 4.6x English

Band    (assume 4 bytes/token, i.e. two-letter merges exist)
  92 / 4 = 23 tokens  ->  23 / 10 = 2.3x English

Derived, not measured. Note that the Armenian ceiling of 9.2x is higher than Georgian’s 7.4x despite Armenian having the cheaper script, purely because Armenian orthography spends more letters per word. That inversion is the strongest evidence available that byte width taken alone is a poor predictor. Run the two together:

import tiktoken
enc = tiktoken.get_encoding("o200k_base")

pairs = {
    "hy": "Հայերեն գրականությունը հարուստ պատմություն ունի։",
    "ka": "ქართული ენა უნიკალური ანბანით იწერება.",
    "el": "Η αρμενική λογοτεχνία έχει πλούσια ιστορία.",
}
for lang, s in pairs.items():
    ids = enc.encode(s)
    b = len(s.encode("utf-8"))
    print(lang,
          len(s), "cp", b, "bytes", len(ids), "tokens",
          "| bytes/cp", round(b / len(s), 2),
          "| bytes/token", round(b / len(ids), 2))

Greek is included as the control: also a two-byte block below U+0800, also an isolated alphabet, but with a far larger corpus because of classical and scientific text. If Greek returns a much higher bytes-per-token figure than Armenian at the same byte width, the difference you are seeing is corpus size with the encoding variable held constant. See Greek.

Armenian punctuation is its own problem

Armenian does not use the shared ASCII punctuation set. The sentence ends with the verjaket ։ (U+0589), not with a full stop. The question mark is the harts’akan nshan ՞ (U+055E) and it is placed over the stressed vowel of the questioned word rather than at the end of the sentence; the exclamation ՜ (U+055C) behaves the same way. There is also a separate Armenian apostrophe, comma and hyphen.

• Sentence splitters break. A regular expression looking for [.!?] finds nothing in an Armenian paragraph and returns the whole thing as one chunk.
• The question mark is mid-word. Any tokenizer or normaliser that assumes punctuation is word-final will split Armenian interrogatives in the middle, producing two fragments per question.
• These characters are individually rare. Every one of them is more likely to be missing from the vocabulary than the letters are, so each costs its full two bytes as separate tokens.
• Numerals are letters. Traditional Armenian numerals are written with the alphabet itself, each letter carrying a numeric value, so a number in that notation is indistinguishable from a word at the byte level and no numeric parser will read it.

Cheap to fix, and worth more than the token saving: extend your sentence-boundary rules to include U+0589 and treat U+055E and U+055C as intra-word.

What actually predicts the multiplier

Taking Armenian and Georgian together, the honest predictor is a product of three factors rather than any single property of the script:

• Bytes per code point — fixed by the Unicode block, checkable in one line, and never improves.
• Code points per word — a property of the orthography and the morphology, and the factor most often forgotten. Armenian’s longer written words eat most of its byte-width advantage.
• Merges per block — the only factor a vendor can change, and the one that has been improving. This is what the vocabulary bottleneck is about.