Why Lao Text Costs More Tokens Than English

Lao and Thai look alike, are structurally alike, and are related. If visual similarity between scripts predicted token cost, they would land in the same place. They do not, and the size of the gap is a good measure of how much of token cost is structure and how much is simply how much text exists.

No spaces, and why that matters to BPE

Lao is written without spaces between words. Spaces appear, but they mark phrase and clause boundaries rather than word boundaries, so a run of Lao text between two spaces is typically several words long.

This removes the single most useful signal a BPE tokenizer has. The merges that dominate any production vocabulary are space-prefixed word forms — which is why the token for a word with a leading space differs from the token for the same word without one, and why that distinction is worth a vocabulary slot at all. The leading space is what tells the merge process where a reusable unit begins.

Without it, BPE has to discover Lao word boundaries statistically from co-occurrence alone, competing for merge slots against languages that hand it the boundaries for free. It does discover some — frequent particles and function words do earn merges — but the process is far less efficient, and the merges it finds do not reliably align with words.

This handicap is shared exactly with Thai, and rather than restate it here, the practical consequences for retrieval are worked through in chunking Thai text for RAG, which applies to Lao without modification. Unicode’s own approach to the problem is worth reading: UAX #29 specifies text segmentation and is explicit that for scripts without word spaces, correct word boundaries require a dictionary, not a rule.

Vowels written around the consonant

Now the part that is Lao’s own arithmetic. The Lao block occupies U+0E80 to U+0EFF, three UTF-8 bytes per codepoint. The word ລາວ is three codepoints, nine bytes.

But three codepoints is a short syllable. Lao vowels are written before, after, above and below the base consonant, and several are circumfixes spelled as two separate codepoints that surround the consonant in the encoded order — a leading vowel codepoint, then the consonant, then a trailing vowel codepoint. Add a tone mark above and a final consonant and a single spoken syllable becomes four or five codepoints, which is twelve to fifteen bytes.

Two things follow that are worth stating precisely. First, the byte cost of a Lao syllable is roughly that of a whole short English word, and a Lao word of two syllables can be thirty bytes. Second, the leading vowel codepoint sits before the consonant it modifies in the byte stream, which means the byte sequence for a syllable is not the phonetic order and the merges BPE learns cut across the syllable in ways that do not correspond to any linguistic unit at all. A merge that starts at a leading vowel and runs into the following consonant is a real, frequent, useful merge that is not a morpheme, a syllable or a phoneme.

That is fine for compression and bad for everything downstream that assumes tokens carry meaning, which is most of what an embedding model assumes.

The Unicode Lao code chart documents the block range, the vowel signs and their placement classes.

Same structure as Thai, different corpus

Lao and Thai share the no-spaces problem, the three-byte floor, the circumfix vowels and the tone marks. Structurally the argument above applies to both, character for character. What they do not share is corpus size.

Thai has a large national web presence — a big commercial internet, decades of digitised news, a substantial Wikipedia. Lao has a fraction of it, from a much smaller population with lower connectivity, and the two scripts are disjoint Unicode blocks so none of Thai’s merges transfer. Whatever the merge auction awarded to Thai, Lao did not get a share of it.

There is one further wrinkle specific to Lao. The orthographic reforms of the mid-twentieth century made Lao spelling substantially phonemic, dropping the etymological Pali and Sanskrit consonants that Thai retains. So Lao has fewer distinct codepoints in ordinary use than Thai does. That sounds like it should help — a smaller effective alphabet means byte pairs repeat more often — and at the margin it does. It is nowhere near enough to offset a corpus two or three orders of magnitude smaller. Structure is a weak lever; data is a strong one, and this pair is the demonstration.

Deriving the multiplier

Assumptions, labelled: English at four characters per token per OpenAI’s published rule of thumb, so 0.25 tokens per character. Three bytes per Lao codepoint, from the block range. Lao merge efficiency of 1.2 to 1.6 bytes per token, on the combined no-spaces and small-corpus argument — the lowest range in this file, and close to the byte floor. And a length assumption: Lao writes the same content in about 0.85 codepoints per English character, higher than the Indic scripts because circumfix vowels and tone marks spend codepoints that Devanagari-family scripts do not.

A 200-character English passage at 50 tokens is about 170 Lao codepoints, so 510 bytes. At m of 1.4 that is about 364 tokens, a multiplier near 7.3x. At m of 1.2 it is 8.5x, and at the absolute byte floor, with no merges firing at all, 10.2x.

Chunking and budgeting for Lao

At 7.3x the practical consequences are severe enough to change architecture, not just configuration.

• Context. A 4,000-token budget holds roughly 550 English tokens’ worth of Lao — about 400 English words. A system prompt plus three retrieved chunks may not fit.
• Chunking. A 512-token chunk holds around 70 English tokens of content. Because Lao has no word spaces, you cannot fall back on splitting at word boundaries either. Split on the phrase-level spaces and on punctuation, and accept variable chunk sizes.
• Output limits. A max-tokens value tuned on English truncates Lao answers to a fraction of their intended length, and because tokens do not align with syllables the truncation lands mid-syllable and produces text that is not merely short but unreadable.
• Streaming. Partial tokens can leave a leading vowel without its consonant, which renders as a floating diacritic. Buffer to a codepoint boundary before displaying.

import tiktoken
enc = tiktoken.get_encoding("o200k_base")

lo = open("corpus.lo.txt", encoding="utf-8").read()
b, t = len(lo.encode("utf-8")), len(enc.encode(lo))
print(f"{len(lo)} cp  {b} B  {t} tok  m={b/t:.2f}")

# how big is a 512-token chunk in Lao characters?
print("chars per 512-token chunk:", round(512 * len(lo) / t))

That last line is the number to design around. It converts a token budget into a unit you can actually split text on, and it is the only figure on this page that comes from your corpus rather than from an assumption. For a neighbouring script with the same no-spaces property and a different set of problems above it, see Burmese.