Why Tone Marks Matter for AI Text in Tonal Languages

Stripping tone marks from Vietnamese does not degrade the text a little. It merges distinct words into one string, and the model then has to guess which was meant — from context, which is exactly the information that was adequate before you deleted the marks and is not adequate afterwards.

Six words that differ only by a tone mark

Vietnamese has six tones. Five are written with a diacritic above or below the vowel and the sixth (ngang, the level tone) is written with no mark at all. The canonical illustration is the syllable ma:

ma   ngang (level)     ghost
mà   huyền (falling)   but / which
má   sắc  (rising)     mother / cheek
mả   hỏi  (dipping)    grave, tomb
mã   ngã  (creaky)     horse; code
mạ   nặng (heavy)      rice seedling

Delete the diacritics and all six collapse to ma. This is not an edge case chosen for effect; the same collapse happens across most of the syllable inventory, because Vietnamese is largely monosyllabic and tone is one of the few features distinguishing one syllable from another. Undiacritised Vietnamese is genuinely ambiguous to native readers too — they disambiguate from context and get it right most of the time, which is precisely the standard the model is now being asked to meet.

The consequence for a model is specific rather than diffuse. Given ma in a sentence about a stable, it will read “horse”; given the same string in a sentence about a graveyard it will read “tomb”. Where context is thin — a product name, a form field, a short search query, a list item — there is nothing to disambiguate from and the model picks the training-frequency favourite. That is why short queries suffer worst, and why a search index built from stripped text ranks by a distribution rather than by meaning.

The same problem in Yoruba

Yoruba marks three tones: high with an acute accent, low with a grave accent, and mid with no mark. It also uses a subscript dot (or vertical line) to distinguish vowel quality, which is a separate system layered on the same letters. The standard illustration is igba:

igba    mid-mid       two hundred
igbá    mid-high      calabash
ìgbà    low-low       time, period
ìgbá    low-high      garden egg
igbà    mid-low       rope

Five words, one undiacritised string. A great deal of Yoruba on the open web is written without tone marks, because typing them requires keyboard support that many users do not have, and because informal writing habitually omits them. So a model trained on web Yoruba is trained substantially on the ambiguous form — which means the problem is not only that your input lost its marks, but that a large part of the training distribution lost them too, and the model has learned that ambiguity as normal.

Mandarin sits differently and is worth mentioning to head off the wrong generalisation. Mandarin is tonal, but Chinese characters encode the syllable including its tone, so writing in characters never loses the distinction. Tone marks only appear in Pinyin, which is a romanisation used for teaching, input and sorting rather than for publishing — and Pinyin written without its tone marks is drastically ambiguous, which is why Pinyin transliteration is a lossy round trip and Chinese text itself is not. The rule generalises as: a script that writes tone into the character is safe, and a script that writes tone as a removable mark is at risk from anything in your pipeline that removes marks.

Tone marks are not the same as accent marks

This distinction causes real bugs and is worth being pedantic about. Vietnamese writes two independent things with diacritics. The letters ă â ê ô ơ ư đ carry marks that are part of the letter’s identity — they encode a different vowel or consonant, not a tone. Separately, the tone diacritics sit on top of those letters. So ứ is the letter ư carrying the rising tone: two orthogonal pieces of information stacked on one glyph.

A naive “strip accents” routine — the widely copied idiom of decomposing to NFD and deleting every combining mark — destroys both at once. It turns ứ into u, discarding a letter distinction and a tone distinction in one step, and it turns đ into nothing useful at all because the stroke is not a combining mark. The result is not “Vietnamese without tones”; it is a lossier string than that. The same routine applied to Yoruba deletes the tone accents and the vowel-quality dots, collapsing an even larger set.

Where in a pipeline the marks get lost

Almost never on purpose, and almost never in the model call. The usual culprits, in rough order of frequency:

• Slug and identifier generation. A URL-safe-ify helper that transliterates to ASCII, applied to a title that is later fed back into an index or a prompt.
• Search normalisation. Accent-insensitive matching applied at index time as well as query time, which quietly makes the stored text ambiguous rather than merely the query.
• Legacy encodings and CSV round-trips. Text passed through a Latin-1 or Windows-1252 stage arrives with marks replaced or mangled.
• OCR and PDF extraction. A diacritic that renders as a separate glyph can be dropped or reordered by an extractor, which is the same class of bug as vowel signs going missing in Tamil and Malayalam OCR.
• Upstream user input. Users typing on a keyboard without the layout simply omit the marks, and no code in your system is at fault.

What to do about it

The rule that solves most of this: normalise for matching, never for storage. Keep the fully marked text as the record of truth, and derive a folded form as a separate field used only for comparison and recall. Then a search for ma can still find mã, while every prompt, every embedding and every displayed string is built from the marked original.

For input you cannot control, restoring the marks is a real task with a real name — diacritic restoration, or tone-mark prediction — and it is a task a language model does reasonably well for Vietnamese precisely because there is enough Vietnamese in training data for context to carry it. It does it much less well for Yoruba, for the data reason above. Where you do restore marks, keep the restored text flagged as derived, and never write it back over the user’s original: a wrong tone restoration in a stored name is a worse outcome than the missing mark you started with.