Why Georgian Text Costs More Tokens Than English

Most expensive scripts have a cheaper relative somewhere. Ukrainian borrows a little from Russian, Urdu a little from Arabic, Tigrinya from Amharic. Georgian has nobody. Mkhedruli is used to write Georgian and essentially nothing else, so every merge in the vocabulary that helps Georgian had to be bought with Georgian text.

Thirty-three letters nobody else uses

Mkhedruli occupies U+10D0 to U+10FF within the Georgian block that starts at U+10A0, which puts every letter in the three-byte region of UTF-8. It is a true alphabet, not an abugida: thirty-three letters, one sound each, one code point each, no vowel signs, no viramas, no combining marks. There are no ligatures and no contextual forms. In structural terms it is the simplest script in this entire cluster.

It is also historically unicameral. Ordinary Georgian text has no case distinction at all, which removes one source of vocabulary waste that Latin and Cyrillic suffer — a merge table for English effectively pays twice for many words, once capitalised and once not.

The exception to that is worth knowing because it is a genuine trap. Uppercase Mtavruli forms were added in Unicode 11 for all-capitals styling, and they were placed not in the Georgian block but in Georgian Extended, U+1C90 to U+1CBF, per the Unicode Georgian code charts. So a Georgian heading set in capitals is a completely different byte sequence from the same words in lower case, in a block added to the standard so recently that it is far less likely to appear in any training corpus at all. Headings, buttons and titles — exactly the short strings where token cost is most visible per unit of meaning — are therefore the most expensive Georgian text you will handle, and they will not match their lower-case equivalents in an index. Case-fold Georgian before you compare or embed it.

So Georgian is simple, regular, and expensive anyway. The reason is not in the script’s design. It is in who else uses it.

No sibling language to borrow merges from

Byte-pair merges are learned from byte sequences, and byte sequences are shared between languages only when those languages share both a script and vocabulary. Cyrillic has Russian carrying the block; Devanagari has Hindi; Arabic script has Arabic. Every smaller language in those blocks gets at least the sub-word merges that its shared letter sequences happen to trigger.

The Georgian block has Georgian. Mingrelian, Svan and Laz use it, and their combined web presence is negligible. There is no high-resource donor. If a vocabulary was trained on a corpus in which Georgian is a rounding error, the practical result is that the block gets its 33 single-character entries at best, and possibly not even those — in which case each letter is spelled out as three separate raw bytes and Georgian sits at its absolute ceiling.

That is what makes Georgian the clean test of the general claim in the tokenizer vocabulary bottleneck for low-resource languages. There is no confounding variable.

Deriving the multiplier

The sentence is ქართული ენა უნიკალური ანბანით იწერება. — “The Georgian language is written with a unique alphabet.”

Georgian
  code points ............ 38  (33 Mkhedruli letters, 4 spaces, 1 '.')
  UTF-8 bytes ............ 104  (33 x 3, plus 5 ASCII)

English "The Georgian language is written with a unique alphabet."
  characters ............. 56
  tokens (assumption: ~4 chars/token for English) ..... ~14

Ceiling (1 token per UTF-8 byte -- and for Georgian this is a
         realistic outcome, not just a bound)
  104 / 14 = 7.4x English

Band    (assume 3 bytes/token: the block's single characters are in
         the vocabulary, nothing above them is)
  104 / 3 = 35 tokens  ->  35 / 14 = 2.5x English

Band    (assume 5 bytes/token: some Georgian word-pieces exist)
  104 / 5 = 21 tokens  ->  21 / 14 = 1.5x English

Derived, not measured. Georgian is the language in this cluster where the ceiling is most likely to be the answer, so the first thing to establish is which of those three rows you are actually in. This script answers that directly by asking whether any returned token decodes to valid text on its own:

import tiktoken
enc = tiktoken.get_encoding("o200k_base")

s = "ქართული ენა უნიკალური ანბანით იწერება."
ids = enc.encode(s)

fallback = 0
for i in ids:
    raw = enc.decode_single_token_bytes(i)
    try:
        raw.decode("utf-8")          # a complete character or word-piece
    except UnicodeDecodeError:
        fallback += 1                # a partial byte of a Georgian letter

print(len(s.encode("utf-8")), "bytes", len(ids), "tokens")
print("byte-fallback tokens:", fallback, "of", len(ids))
print("bytes per token:", round(len(s.encode("utf-8")) / len(ids), 2))

A high fallback count means the vocabulary has no Georgian entries and you are paying the 7.4x ceiling. A fallback count of zero with bytes-per-token near 3.0 means single characters are covered and nothing more. Above 3.0 means real Georgian merges exist, which is the outcome that has been improving with each vocabulary generation.

Consonant clusters and case endings

Georgian phonotactics permit consonant clusters that few languages allow — sequences of four, five and occasionally more consonants with no intervening vowel. Combined with seven grammatical cases and a verb system that marks both subject and object with affixes, the result is a large space of surface forms built from letter sequences that are unusual even by the standards of other languages using the same block.

This matters because BPE’s early merges are the frequent ones. A language whose common letter pairs are rare across the training corpus as a whole gets fewer of those cheap early merges, so even modest Georgian representation buys less than the same volume of a language with conventional syllable structure would.

Working with a near-worst-case script

• Assume 3x and verify, do not assume 7x. The ceiling makes an alarming headline and is increasingly not what modern vocabularies do. Run the fallback check above against the specific model you are billing on before provisioning around it.
• A 4,000-token window holds very little. At the middle band, 4,000 tokens is roughly 12,000 bytes, which is about 4,000 Georgian letters, which is on the order of 500 words. A single-page document can exceed a small context window.
• Chunk on characters, never on bytes. Splitting a Georgian string at an arbitrary byte offset lands mid-character two times in three and produces a replacement character that poisons the embedding for the whole chunk.
• Do not transliterate to Latin to save money. It roughly thirds the byte count and it makes your text unmatchable against every Georgian-script document you will ever want to retrieve.
• Armenian is the natural comparison and it behaves differently. Similar corpus size, similar isolation, but two bytes per letter instead of three. See Armenian for what that difference does and does not buy.