Chunking Thai Text for RAG Without Spaces

Thai is written without spaces between words, but it does use spaces — between phrases and sentences. That combination is worse than having no spaces at all, because a splitter finds separators, produces plausible-looking chunks, and puts boundaries in the wrong places while appearing to work.

Thai spaces are not word breaks

In Thai orthography a space marks the end of a clause or a sentence, roughly where English would use a comma or a full stop. Words run together: ฉันกินข้าว is three words — ฉัน (I), กิน (eat), ข้าว (rice) — and one unbroken run of characters. A splitter that treats the space as a word separator will therefore produce units that are whole clauses, which is fine for sentence segmentation and useless for finding a safe cut inside a long clause.

Two more properties matter before you write any code. First, Thai writes vowels and tone marks as combining characters that attach above, below, before and after the base consonant, so a single visual character is often several code points. Second, some Thai text carries U+200B ZERO WIDTH SPACE as an explicit word hint, inserted by a CMS or an editor. Where present it is a gift; it is not present often enough to rely on.

Where a character-count cut lands

Take ฉันกินข้าว as a labelled sample. In code points it is:

ฉ  U+0E09  THAI CHARACTER CHO CHANG      (consonant)
ั  U+0E31  THAI CHARACTER MAI HAN AKAT    (combining vowel, above)
น  U+0E19  THAI CHARACTER NO NU           (consonant)
ก  U+0E01  THAI CHARACTER KO KAI          (consonant)
ิ  U+0E34  THAI CHARACTER SARA I          (combining vowel, above)
น  U+0E19  THAI CHARACTER NO NU           (consonant)
ข  U+0E02  THAI CHARACTER KHO KHAI        (consonant)
้  U+0E49  THAI CHARACTER MAI THO         (combining tone mark)
า  U+0E32  THAI CHARACTER SARA AA         (vowel, after)
ว  U+0E27  THAI CHARACTER WO WAEN         (consonant)

10 code points, 3 words, 7 visual clusters.

Now cut it at four code points, which is what a fixed-size splitter does when the budget runs out at that offset:

before: "ฉันกินข้าว"          (I eat rice)
cut at index 4
  chunk A: "ฉันก"        = ฉัน + the first consonant of กิน
  chunk B: "ินข้าว"      = a combining vowel with no base, + น + ข้าว

after (word-aware):
  chunk A: "ฉันกิน"      = I eat
  chunk B: "ข้าว"        = rice

Two distinct kinds of damage in one cut. Chunk A ends with a bare consonant torn off the front of กิน, so the word is gone from the index in either half. Chunk B begins with U+0E34, a combining vowel whose base consonant is in the previous chunk — an orphaned combining mark, which renderers display attached to whatever character happens to precede it, and which tokenizers encode as something no real Thai text contains. The second problem is a strict superset of the first: even if you did not care about word integrity, you cannot cut inside a grapheme cluster.

Dictionary-based word breaking

Thai word segmentation is a dictionary problem. There is no orthographic signal for where a word ends, so a segmenter matches the character stream against a lexicon and picks a decomposition. Two approaches are standard:

• Longest matching (maximal matching). Scan forward, take the longest dictionary entry that matches at the current position, advance. Fast, no model, and wrong on the specific class of inputs where a long word is really two short words. Good enough when its only job is to find a legal cut point.
• Dictionary-based dynamic programming. Enumerate all decompositions the lexicon permits and choose the one that minimises a cost — usually word count plus a penalty for unknown runs. This is what ICU’s Thai break iterator and the widely used Thai NLP libraries do, and it recovers most of what longest matching gets wrong.

For a retrieval pipeline you do not need perfect segmentation. You need candidate boundaries: positions where a cut is linguistically legal. A segmenter that occasionally merges two words still produces boundaries that are all valid, and the packing step only ever chooses among them.

The class of input where dictionary segmenters go wrong is worth knowing, because it is exactly what a technical corpus is made of. Proper nouns, transliterated English loanwords and product names are absent from the lexicon, so the segmenter falls back to an unknown-word heuristic and produces a run it cannot decompose. That run is often long, and a long unsegmentable run is the one thing that forces the packer into a blind cut. If your corpus is full of brand names, add them to the segmenter’s user dictionary before concluding the approach does not work.

If you cannot take a dependency, the fallback is grapheme clusters. They will not keep words intact, but they guarantee you never orphan a combining mark, which is the difference between a degraded chunk and a corrupt one. The same distinction appears in splitting Devanagari without breaking conjuncts.

The chunking pipeline

1. Normalise to NFC and strip any U+200B, recording their positions first if you want to use them as boundary hints.
2. Split on the space character and on newlines. These are your sentence and clause units — the strongest boundaries the document has.
3. Word-break each unit with a dictionary segmenter, producing a list of words that concatenates back to the original with no separator added.
4. Pack words into chunks against a token budget, breaking only between words.
5. Assert that joining all chunks reproduces the input exactly. This is the test that catches a segmenter that inserts spaces.

from pythainlp.tokenize import word_tokenize
import tiktoken, unicodedata

enc = tiktoken.get_encoding("o200k_base")
ntok = lambda s: len(enc.encode(s))

def chunk_thai(text, budget=300):
    text = unicodedata.normalize("NFC", text).replace("\u200b", "")
    chunks, cur, n = [], [], 0
    for clause in text.split(" "):
        for w in word_tokenize(clause, engine="newmm"):
            k = ntok(w)
            if cur and n + k > budget:
                chunks.append("".join(cur))
                cur, n = [], 0
            cur.append(w)
            n += k
        if cur:
            cur.append(" ")
            n += ntok(" ")
    if cur:
        chunks.append("".join(cur))
    assert "".join(chunks) == text.replace("\u200b", "")
    return chunks

The newmm engine is the dictionary-based maximal-matching segmenter; swapping the engine changes segmentation quality without changing the shape of the loop. The final assertion is not decoration. A segmenter configured with the wrong options will happily return tokens with whitespace normalised away, and reassembling those into chunks silently changes the document you are indexing.

Notice that the loop appends the space back after each clause. That is deliberate and not cosmetic: in Thai the space carries the clause boundary, so dropping it removes the only sentence-level punctuation the document has. A chunk reassembled without its spaces reads as one continuous run to a human and to a tokenizer alike, and the difference between spaced and unspaced Thai chunks is the difference between a paragraph and a sentence as the unit of retrieval.

Set the budget in tokens rather than characters, for a reason specific to Thai. Combining marks are separate code points, so a character count runs ahead of what a reader perceives as length; at the same time Thai tokenises inefficiently in most vocabularies, so the token count runs ahead of the character count. The two do not cancel and neither is predictable from the other. Measure against the tokenizer your embedding model actually uses.

Checks that catch a broken chunk

• No chunk starts with a combining mark. Check the Unicode general category of the first code point of every chunk: if it is Mn (non-spacing mark), the split was made inside a grapheme cluster. This one check catches the entire class of orphan errors and costs nothing.
• Round-trip equality. Concatenating chunks in order must reproduce the source text byte for byte, modulo whatever you deliberately stripped.
• Chunk length distribution. If the token counts cluster tightly at exactly the budget, the packer is filling to the limit and cutting wherever it lands — a sign the word breaker returned one giant token because it did not recognise the script.
• Spot-check a rendered chunk. Print three chunks and look at them. Thai text with a broken cluster renders visibly wrong — a mark sitting on the wrong consonant — and a person spots it faster than any assertion.