Why Khmer Text Costs More Tokens Than English

Count the characters in a line of Khmer and you will get a number well below what the tokenizer charges you for, and the gap is not rounding. A meaningful fraction of the code points in any Khmer document are invisible: they exist to tell the renderer to stack one consonant beneath another, they cost three bytes each, and they display as nothing.

The character that costs three bytes and renders nothing

Khmer occupies U+1780 to U+17FF, in the three-byte region of UTF-8. Within it, U+17D2 is named KHMER SIGN COENG, and it is a stacker: it takes the following consonant and renders it as a subscript beneath the preceding one. It has no visual form of its own. In the word កម្មវិធី (“program”), the doubled ម is written as ម + U+17D2 + ម, three code points and nine bytes, and a reader sees one stacked cluster.

Vowels compound this further. Khmer dependent vowel signs are separate code points that may appear before, after, above or below the consonant they modify, and a syllable can carry more than one. A visually compact Khmer syllable is routinely three or four code points and therefore nine to twelve UTF-8 bytes. English writes a syllable in three or four bytes. The block also carries its own digits ០ to ៩ at U+17E0 to U+17E9 per the Unicode Khmer code chart, three bytes each and far rarer in training data than ASCII digits, which makes dates, prices and quantities in Khmer documents disproportionately expensive for the information they carry.

Burmese uses exactly the same trick with its own virama and its own stacked forms, so the mechanism there is the same one described here; see Burmese rather than reading this section twice.

No spaces between words

Khmer writes without spaces between words. Spaces appear at phrase and clause boundaries, roughly where English uses a comma, and the sentence terminator is ។ (U+17D4). This removes the single most valuable signal a byte-pair tokenizer has.

The reason it matters so much is specific to how BPE trains. In a space-delimited language the space byte acts as a natural anchor: the most frequent merges are word-initial sequences preceded by a space, which is why an English vocabulary is full of entries beginning with a leading space. A language with no spaces offers no such anchor. Merges have to be learned across a continuous byte stream in which the same three-byte sequence appears in every possible alignment, so the frequency of any given aligned sequence is diluted.

Thai and Lao share this property, and this is the reason all three behave worse than their byte counts alone predict. See Thai and Lao.

Deriving the multiplier

The sentence is កម្មវិធីនេះមិនដំណើរការទេ។ — “This program does not work.” It is a single unbroken string with no spaces anywhere in it.

Khmer
  code points ............ 25   (every one in the Khmer block, 3 bytes)
  UTF-8 bytes ............ 75   (25 x 3 exactly -- no ASCII at all)
  spaces ................. 0
  visible clusters ....... ~11  (what a reader perceives)
  invisible code points .. the COENG in ក-ម-U+17D2-ម, plus the
                           dependent vowel signs, none of which
                           stand alone

English "This program does not work."
  characters ............. 27
  tokens (assumption: ~4 chars/token for English) ..... ~7

Ceiling (1 token per UTF-8 byte)
  75 / 7 = 10.7x English

Band    (assume 3 bytes/token, one token per code point)
  75 / 3 = 25 tokens  ->  25 / 7 = 3.6x English

Band    (assume 4.5 bytes/token, some cluster-level merges exist)
  75 / 4.5 = 17 tokens  ->  17 / 7 = 2.4x English

Derived, not measured. The number worth extracting is not the multiplier but the share of it you are paying for characters nobody can see. This script prices them:

import tiktoken
enc = tiktoken.get_encoding("o200k_base")

s = "កម្មវិធីនេះមិនដំណើរការទេ។"
COENG = "\u17d2"

full = len(enc.encode(s))
# Same string with the stackers removed. It renders wrongly and means
# nothing -- it exists only to price the invisible characters.
stripped = len(enc.encode(s.replace(COENG, "")))

print("code points     ", len(s))
print("coeng occurrences", s.count(COENG))
print("tokens, as written", full)
print("tokens, no coeng  ", stripped)
print("tokens spent on invisible stackers:", full - stripped)

On a longer Khmer document that difference is not marginal. Khmer orthography uses subscript consonants heavily in loanwords and in Pali- and Sanskrit-derived vocabulary, which is much of the formal register, so technical and legal Khmer carries proportionally more COENG than conversational Khmer does.

Two spellings that look identical

Khmer has a further problem that costs tokens indirectly. The same visible cluster can often be encoded as more than one sequence of code points, because the vowel signs and the subscript consonant can legally appear in different orders. The Unicode Standard specifies a preferred order for Khmer, but real-world text — particularly text typed on older input methods or extracted from PDFs — frequently does not follow it.

• The same word tokenizes two different ways. Different byte sequences hit different merges, so one variant may cost noticeably more than the other for identical rendered text.
• Exact-match search silently fails. Two strings that look the same on screen do not compare equal, and no amount of case folding fixes it.
• Embeddings diverge. Two encodings of one document produce two different vectors, so a query written one way will not retrieve a document stored the other way.

Normalise to NFC on ingest, and if you control the corpus, run a Khmer-aware reordering pass as well — NFC alone does not fix Khmer sign ordering, because the sequences involved are all canonically distinct.

Chunking and budgeting Khmer

• You cannot chunk on whitespace. A splitter looking for spaces or newlines returns the entire paragraph. Split on ។ (U+17D4) for sentences, and use a dictionary-based word segmenter if you need finer granularity. The same approach applies to Thai; see chunking Thai text for RAG.
• Never split between a consonant and a following COENG. The result is a dangling stacker at the end of one chunk and an orphaned consonant at the start of the next, both of which render as broken text and embed as noise. Segment on grapheme clusters.
• A 4,000-token budget is roughly 12,000 bytes. At three bytes per code point that is about 4,000 Khmer code points, which at three or four code points per syllable is only around 1,100 syllables. A short article does not fit.