Why Burmese Text Costs More Tokens Than English

Burmese has two problems and only one of them is Unicode. The script is three bytes per code point and spends several code points on a single round cluster, which is arithmetic. The second problem is that a large share of Burmese text on the web is not in Unicode at all, which makes the effective training corpus smaller than the raw document count suggests.

Medials, vowels and the asat

The Myanmar block runs from U+1000 to U+109F, three bytes per code point in UTF-8, with further characters in the Myanmar Extended blocks. Burmese is an abugida and it builds a syllable by attaching several classes of mark to a base consonant, each of which is its own code point:

• Medial consonants U+103B to U+103E — the ya, ra, wa and ha medials, written as hooks and loops around the base. A syllable may carry more than one.
• Dependent vowel signs such as U+102C, U+102D and U+102E, placed before, after, above or below the base.
• The asat U+103A, which kills the inherent vowel and marks a syllable-final consonant. It is a small mark and it is everywhere.
• The virama U+1039, which stacks one consonant beneath another and, like Khmer’s COENG, renders as nothing at all.
• Myanmar digits U+1040 to U+1049, three bytes each, listed alongside the rest in the Unicode Myanmar code chart. They are also visually confusable with letters, which is a recurring source of OCR and input errors.

The word အက္ခရာ (“letter, character”) contains a U+1039 stacking ခ beneath က. That is a code point costing three bytes whose entire visual effect is to move a letter downward. Khmer does the same thing with a different code point; the argument is developed at length under Khmer and is not repeated here.

Code points per syllable, not characters per word

The right unit for Burmese is the syllable, and the right question is how many code points one costs. A minimal syllable is one consonant: one code point, three bytes. A typical syllable in running text carries a vowel sign and an asat, or a medial and a vowel: three to four code points, nine to twelve bytes. English syllables average around three bytes.

So the honest headline figure for Burmese is not “three times the bytes”. It is closer to three times the bytes per code point multiplied by two to three code points per syllable, against an English baseline that spends roughly one byte per letter. That is where the large multipliers come from, and it is why counting characters underestimates Burmese cost badly.

Burmese does use spaces, unlike Khmer and Thai, but they mark phrase boundaries rather than words, so they are a weak segmentation signal rather than an absent one. The sentence below has two spaces across thirty-six script characters.

Deriving the multiplier

The sentence is မြန်မာစာသည် အလွန်လှပသော အက္ခရာဖြစ်သည်။ — “Burmese is a very beautiful script.”

Burmese
  code points ............ 38   (36 Myanmar incl. the '။' terminator
                                 U+104B, plus 2 spaces)
  UTF-8 bytes ............ 110  (36 x 3, plus 2 ASCII spaces)
  space-delimited runs ... 3
  syllables .............. ~13

  code points per syllable: 36 / 13 = 2.8

English "Burmese is a very beautiful script."
  characters ............. 35
  tokens (assumption: ~4 chars/token for English) ..... ~9

Ceiling (1 token per UTF-8 byte)
  110 / 9 = 12.2x English

Band    (assume 3 bytes/token, one token per code point)
  110 / 3 = 37 tokens  ->  37 / 9 = 4.1x English

Band    (assume 6 bytes/token, i.e. two-code-point merges exist)
  110 / 6 = 18 tokens  ->  18 / 9 = 2.0x English

Derived, not measured. To find out which band you are in, strip the marks and re-measure: the difference is the price of the diacritic stacking specifically, separated from the base consonants.

import tiktoken
enc = tiktoken.get_encoding("o200k_base")

s = "မြန်မာစာသည် အလွန်လှပသော အက္ခရာဖြစ်သည်။"

# Medials U+103B-U+103E, asat U+103A, virama U+1039, and the
# dependent vowel signs U+102B-U+1035. Removing them destroys the
# text; this measures their cost, it is not a transformation to ship.
MARKS = set(range(0x102B, 0x1036)) | {0x1039, 0x103A} | set(range(0x103B, 0x103F))
bare = "".join(c for c in s if ord(c) not in MARKS)

for label, t in (("as written", s), ("bases only", bare)):
    ids = enc.encode(t)
    print(label, len(t), "cp", len(t.encode("utf-8")), "bytes",
          len(ids), "tokens")

print("share of code points that are marks:",
      round(1 - len(bare) / len(s), 2))

A mark share around forty percent is normal for running Burmese. If removing them cuts the token count by roughly the same proportion, the vocabulary is treating each code point independently and you are near the 4.1x band; a smaller reduction means merges are spanning base-plus-mark sequences and you are doing better than that.

The Zawgyi problem, and why the corpus is smaller than it looks

For most of the period during which the crawled web was being assembled, a large majority of Burmese content online was encoded not in Unicode but in Zawgyi, a font hack that reuses Myanmar block code points at different positions to avoid implementing proper text shaping. Zawgyi text and Unicode text are byte-level incompatible: the same visible sentence is a completely different byte sequence in each.

The consequence for a tokenizer is direct. Zawgyi-encoded Burmese in a training corpus contributes merges that do not help Unicode Burmese at all, and may actively consume vocabulary slots. Myanmar’s national migration to Unicode began in earnest around 2019 and a great deal of older archived material was never converted, so the Burmese text a vocabulary was trained on is some unknown mixture of two mutually useless encodings.

Thai, next door, has none of this: one encoding, a far larger and cleaner web corpus, and a longer history of NLP tooling. That is why Burmese sits worse than Thai despite comparable byte widths, and it is a corpus fact rather than a script fact. See Thai.

Working with Burmese text

• Do not trust space-splitting. Burmese spaces are phrase boundaries. Splitting on them gives you chunks of wildly uneven size, and splitting on the ။ terminator is more reliable.
• Segment on grapheme clusters. A boundary between a base consonant and its asat or medial produces two fragments that render as broken glyphs. Unicode’s UAX #29 grapheme cluster rules are the correct definition of an indivisible unit here.
• Expect quality to lag cost. Burmese is one of the languages where a model will answer fluently and be wrong, because the corpus is thin and partly mis-encoded. Budget for verification, not just for tokens; see why low-resource languages hallucinate more.
• A 4,000-token window holds about a page. At the middle band, 4,000 tokens is around 12,000 bytes, roughly 4,000 code points, roughly 1,400 Burmese syllables.