OCR for Thai Script Without Spaces Between Words

A Thai OCR result that looks like one enormous unbroken string is not broken. Thai does not put spaces between words, so a correct transcription of a Thai page is an unbroken string, and everything that goes wrong afterwards is a different problem with a different solution.

Two problems, one output

Conflating these two is the most expensive mistake in a Thai document pipeline, because it sends people to tune the wrong stage for weeks.

• Recognition turns ink into characters. In Thai this is a normal, tractable OCR problem with normal, tractable difficulties.
• Segmentation decides where words begin and end. In Thai this information is not in the ink at all. It cannot be recovered by a better scan, a better recogniser or more training data, because it was never written down.

Thai does use spaces: between clauses, around some phrases, and at sentence boundaries. So a page contains spaces, which is exactly enough to make a whitespace tokeniser look like it is working while producing tokens that are whole clauses.

The recognition problem

Thai script is an abugida with 44 consonant letters, and vowels written as marks placed before, after, above or below the consonant. Tone marks go above, potentially stacked on top of an upper vowel. The result is a line of text with up to four vertical levels: below-base vowels, the consonant line, above-base vowels, and tone marks above those.

That vertical stacking is the main recognition difficulty and it has a specific failure mode. A line-segmentation stage tuned for Latin type expects text to sit between two horizontal bounds and will clip the top or bottom level, silently discarding tone marks — which change the word. Adjacent lines set tightly can also have marks that overlap the neighbouring line’s bounding box, so a marker gets attached to the wrong line.

The second difficulty is that several Thai letters differ only in small details: the presence or absence of a loop, whether a loop opens left or right, or the number of humps. Traditional Thai typefaces mark most letters with a distinctive head loop, and that loop is the primary disambiguating feature. Modern loopless display typefaces, widely used in signage, advertising and contemporary UI design, remove it — and a recogniser trained on looped type degrades noticeably on loopless type for exactly that reason. If your corpus is modern printed material, check which typeface class it is in before blaming the engine.

Both problems are ordinary. With a reasonable scan and a Thai-trained line recogniser, character-level output is good. Nothing about the missing word spaces makes recognition harder.

The segmentation problem

Thai word segmentation is genuinely ambiguous, not merely unsolved. The standard illustration in Thai language processing is the string ตากลม, which can be read as ตา + กลม (round eyes) or as ตาก + ลม (to air something in the wind). Both segmentations use real words, both are grammatical, and a human reader picks between them using the surrounding sentence. There is no information in the characters themselves that resolves it.

This is why the Unicode Consortium’s text segmentation annex treats these scripts as a special case: its default word-boundary rules are defined for scripts that mark boundaries, and it states that languages including Thai, Lao, Khmer and Burmese require dictionary lookup rather than a rule-based algorithm. The annex is Unicode Standard Annex #29, Unicode Text Segmentation, and ICU implements the dictionary-based break iterator it describes.

A dictionary-based segmenter carries its own failure modes that matter for OCR output specifically. Out-of-vocabulary words — names, brands, transliterations, technical terms — have no dictionary entry, so the segmenter chops them into the longest known fragments it can find, producing several plausible-looking words that mean something else entirely. And because OCR output is imperfect, a single wrong character can change the segmentation of the surrounding twenty, since the segmenter is optimising over the whole string rather than locally. One recognition error therefore causes a burst of word errors, not one.

Which metric measures which

Because the two stages are distinct, the metrics have to be too, and the default is misleading.

Word error rate on Thai does not measure your OCR. It measures your segmenter, plus your OCR, with no way to separate them — and because segmentation disagreements are frequent even between two correct systems, a word error rate on Thai can look catastrophic while the transcription is nearly perfect. Report character error rate for the recognition stage. It is the only metric that is about the ink.

Then measure segmentation separately, against ground truth that a Thai speaker segmented, and expect a ceiling: two annotators do not fully agree on Thai word boundaries, because the definition of a word is partly a convention. Published Thai segmentation corpora each fix a tokenisation standard, and scores are only comparable within one standard. If you evaluate against a corpus that segments compounds differently from your production segmenter, you will measure the disagreement rather than the quality.

Signals the page gives you

One thing the printed page does supply, and most pipelines throw away: Thai text is broken across lines at word boundaries, without hyphenation. There is no rule permitting a Thai word to be split at the end of a line and continued on the next.

So every line break in a correctly typeset Thai document is a genuine word boundary, confirmed by the typesetter. That is free, high-quality segmentation ground truth on every page — and it is destroyed by the usual de-hyphenation and line-joining step that document pipelines apply to reflow text. If you join Thai lines before segmentation, you throw away a signal you cannot recover; if you keep the break positions as boundary hints and pass them to the segmenter, you get them for nothing.

Two other useful signals: existing spaces are always boundaries even though most boundaries have no space, so they can be treated as hard constraints; and punctuation, digits and any embedded Latin text mark boundaries too, which matters because mixed Thai-English text is common in technical material.

Everything downstream depends on getting this stage right, because a retrieval system over Thai inherits the segmentation of whatever indexed it. That is worked through in chunking Thai text for retrieval. The neighbouring scripts have related but distinct problems — Khmer adds sub-consonants stacked below the line, covered in OCR for Khmer stacked consonants.