Why Thai Text Costs More Tokens Than English

The usual explanation for Thai’s token cost is that the characters are three bytes wide. That is true and it is the smaller half. The larger half is that Thai does not put spaces between words, and a byte-level BPE tokenizer is built around the assumption that it can see them.

The missing whitespace anchor

Before BPE merges anything, the tokenizer runs a pre-tokenization step: a regular expression that cuts the input into candidate pieces, typically on whitespace and punctuation boundaries, with a leading space attached to the following word. Merges are then learned and applied strictly inside those pieces. They never cross a pre-token boundary. This design comes straight from the byte-level BPE described in Radford et al., “Language Models are Unsupervised Multitask Learners” (OpenAI, 2019), and it is why English words so often tokenise as a single token that begins with a space.

Thai breaks both halves of that. There is no leading space to anchor a word, so the merge table cannot learn a distinctive “space-then-word” sequence for Thai the way it does for English and French. And there is no boundary to cut on, so a whole Thai clause arrives at the BPE stage as one enormous pre-token — a run of letters dozens of characters long that the merge table has to chew through from the bytes upward.

The consequence is that Thai merges have to be learned as free-floating character sequences that can appear anywhere inside a run, competing for slots against every anchored word in every space-separated language. They lose that competition, and the fallback is short pieces: two or three Thai characters, sometimes single bytes.

The arithmetic on a labelled sentence

ฟังก์ชันนี้ใช้หน่วยความจำมาก
"This function uses a lot of memory."

That Thai line is 28 code points and not one of them is a space. Thai occupies U+0E00–U+0E7F, which UTF-8 encodes in three bytes, so the line is exactly 84 bytes. The English is 35 ASCII bytes, near 9 tokens. The ceiling for the Thai, from the byte floor of byte-level BPE, is 84 tokens.

Derive the middle with the assumption named. Assume that frequent Thai sequences of two or three characters have earned merges — the common function words นี้, ความ, การ and the like — while the rest falls back to roughly one token per character. That places the sentence in the region of 15 to 30 tokens against the English 9, so a derived multiplier of about 1.7× to 3.3×. As always, the byte count is exact and the token range is derived from a stated assumption about merge coverage rather than measured.

Note something in Thai’s favour, because it is easy to overstate the penalty: 28 code points is a compact way to say what English says in 35 characters. Thai orthography is dense. The penalty is real but it is smaller than the three-bytes-per-character figure alone suggests.

Tone marks, and vowels stored out of order

Thai stacks. A single syllable can carry a consonant, a vowel sign above or below it, a tone mark above that, and the thanthakhat ์ that silences a letter — each one a separate code point of three bytes. ก์ is two code points; หน่ is three. The stacking is why the code point count runs ahead of what a reader would call letters, in the same way it does in Devanagari, though for different orthographic reasons.

There is also an ordering quirk with real consequences. The pre-posed vowels เ, แ, โ, ใ and ไ are written to the left of the consonant they follow phonetically, and Unicode stores them in that visual order — the vowel code point comes first in the string. So the logical and visual orders agree for these, unlike in most other scripts, which means naive sorting and prefix matching behave differently from what a reading-order-based intuition predicts. For tokenization it means the leading code point of a syllable can be a vowel, which is a poor anchor for a merge.

Segmentation is a separate job

Thai word boundaries are not derivable from the characters; they require a dictionary. The Unicode line-breaking and word-segmentation algorithms explicitly delegate Thai, Lao, Khmer and Burmese to dictionary-based segmentation, which is why ICU ships a Thai dictionary and why every general-purpose splitter that claims to be Unicode-aware still gets Thai wrong unless it has that dictionary loaded.

This matters for token cost indirectly but importantly: you cannot budget a chunk in words, cannot compute a sensible overlap, and cannot align a highlight to a token boundary without segmenting first. The tokenizer will happily hand you pieces, but its pieces are byte-frequency artefacts, not words, and treating them as words produces retrieval chunks that begin and end mid-word. The retrieval-side treatment is in chunking Thai text for RAG, and the same absence of spaces is what makes Thai OCR output hard to validate.

Measuring it, and what it does to chunking

import tiktoken
from pythainlp.tokenize import word_tokenize

enc = tiktoken.get_encoding("o200k_base")

th = "ฟังก์ชันนี้ใช้หน่วยความจำมาก"

print("codepoints", len(th))
print("bytes", len(th.encode("utf-8")))
print("tokens", len(enc.encode(th)))

words = word_tokenize(th, engine="newmm")
print("segmented words", len(words))
for w in words:
    print(w, len(w), len(enc.encode(w)))

Two ratios to read. Tokens divided by segmented words tells you how many model tokens an average Thai word costs — that is the number to use when sizing chunks, and it will typically be well above one. Tokens divided by code points tells you how close you are to the byte floor; anything near 1.0 means the merge table is contributing almost nothing and you are paying per character.

• Size chunks in tokens and cut on segmented words. These are two separate constraints and Thai is one of the few languages where satisfying one does not give you the other for free.
• Overlap in words, not characters. A character-based overlap will start the next chunk in the middle of a word roughly every time.
• Thai numerals ๐-๙ are distinct code points from ASCII digits, three bytes each, and rare enough in corpora to tokenise poorly. Normalise them if you are going to do arithmetic on them.
• Spaces in Thai are phrase separators. Where a space does appear, it marks something closer to a clause break than a word break, so splitting on it produces very long pieces rather than words.