Why Vietnamese Diacritics Break Search Matching

A user searches for Tiếng Việt, the exact phrase is visibly present in the database, and the query returns nothing. Copy both strings into a comparison and they look identical in every font. They are not equal, and Vietnamese hits this harder than any other Latin script because it puts two marks on one vowel.

The symptom

The reports all rhyme: exact-match search fails on Vietnamese names, GROUP BY produces two rows for what is obviously one value, a uniqueness constraint lets a duplicate through, deduplication misses, or a WHERE name = ? with a pasted value returns zero rows while LIKE '%…%' on a fragment works. Every one of them is the same bug: two encodings of one word.

The reason Vietnamese is the worst case is structural. Vietnamese orthography combines a vowel-quality mark (the circumflex in ê, the horn in ơ and ư, the breve in ă) with a tone mark (acute, grave, hook, tilde, dot below). That is two diacritics on one base letter, and Unicode offers several legitimate ways to write the result.

The same word, three byte sequences

Take Việt. The interesting character is ệ — the letter e with a circumflex and a dot below.

Fully precomposed (NFC):
  V  U+0056
  i  U+0069
  ệ  U+1EC7   LATIN SMALL LETTER E WITH CIRCUMFLEX AND DOT BELOW
  t  U+0074
  -> 4 code points, UTF-8: 56 69 E1 BB 87 74        (6 bytes)

Fully decomposed (NFD):
  V  U+0056
  i  U+0069
  e  U+0065
     U+0323   COMBINING DOT BELOW          (combining class 220)
     U+0302   COMBINING CIRCUMFLEX ACCENT  (combining class 230)
  t  U+0074
  -> 6 code points, UTF-8: 56 69 65 CC A3 CC 82 74  (8 bytes)

Partially composed (the one that breaks everything):
  V  U+0056
  i  U+0069
  ê  U+00EA   LATIN SMALL LETTER E WITH CIRCUMFLEX
     U+0323   COMBINING DOT BELOW
  t  U+0074
  -> 5 code points, UTF-8: 56 69 C3 AA CC A3 74     (7 bytes)

Three representations of one word: six, eight and seven bytes. len() returns 4, 6 and 5. Any byte or code-point equality check says all three are different. Rendered in any Vietnamese-capable font, all three are indistinguishable.

Note the combining class numbers in the decomposed form, because they are what makes canonical ordering possible. The dot below has class 220 and the circumflex has class 230, so canonical ordering always places the dot before the circumflex regardless of which the user typed first. Without that rule there would be two decomposed forms as well, and no normalization could unify them.

Where the different forms come from

• The input method. Vietnamese IMEs such as UniKey offer an explicit choice between precomposed output (“Unicode dựng sẵn”) and combining output (“Unicode tổ hợp”). Two users on the same site can be typing genuinely different byte sequences for the same word, and neither has done anything wrong.
• The operating system. macOS file names are stored in a decomposed form, so anything read from a file listing arrives decomposed while anything typed into a web form on the same machine usually arrives composed.
• Legacy encodings converted late. Vietnamese text spent years in TCVN3, VNI-Windows and VISCII. Converters differ in whether they emit precomposed or partially composed Unicode, so an import from an old system frequently disagrees with the live application.
• PDF and OCR extraction. Text extracted from a PDF often comes out with marks separated, because that is how the glyphs were positioned on the page.

The fix

1. Normalize at every boundary where text enters. API handlers, form posts, file imports, message consumers. NFC, not NFKC — you want the canonical form that preserves the text, and the reasons are in the difference between NFC and NFKC.
2. Normalize the query with the same function. Normalizing storage and not queries fixes nothing; the mismatch just moves.
3. Backfill what is already stored. In PostgreSQL 13 or later, UPDATE t SET name = normalize(name, NFC) WHERE NOT name IS NORMALIZED does it in one statement and the predicate keeps the update from touching rows that are already fine.
4. Add the check at the edge of the type system. A constraint of CHECK (name IS NORMALIZED) turns the next un-normalized write into an error at insert time instead of a support ticket six months later.
5. Keep a folded column for recall. Users routinely type Vietnamese without diacritics on a non-Vietnamese keyboard, so viet should find Việt. Store an accent-stripped derived column alongside the NFC original, search both, and rank exact-diacritic matches above folded ones — the general pattern is in implementing diacritic-insensitive search.

import unicodedata

def nfc(s: str) -> str:
    return unicodedata.normalize("NFC", s)

def fold(s: str) -> str:
    """Accent-stripped key for recall. Keep alongside the original, never instead."""
    d = unicodedata.normalize("NFD", s)
    stripped = "".join(c for c in d if not unicodedata.combining(c))
    # đ / Đ has no combining decomposition and must be handled explicitly.
    return stripped.replace("đ", "d").replace("Đ", "D").casefold()

assert nfc("Vie\u0323\u0302t") == nfc("Vi\u1ec7t")
assert fold("Tiếng Việt") == "tieng viet"

The đ line in that function is not an afterthought. The Vietnamese letter đ (U+0111) is a base letter with a stroke through it, not a base plus a combining mark, so it has no canonical decomposition and the strip loop leaves it untouched. Every accent-folding implementation written by someone who does not read Vietnamese omits that line, and the result is a folded key containing a character no keyboard-less user will ever type.

The problem normalization does not fix

Normalization unifies different encodings of the same spelling. It cannot unify two different spellings, and Vietnamese has a live one.

In words where a tone mark falls on a vowel cluster, two conventions coexist: the older style places the mark on the first vowel and the newer style places it according to phonetic rules. So hòa and hoà are the same word, both written by real people, and their NFC forms are genuinely different because the mark is attached to a different letter. The same goes for thủy and thuỷ, quý and qúy.

No normalization form will merge these, because Unicode is correct that they are different sequences of characters. Handle it at a different layer:

• The accent-folded recall column collapses them automatically — hoa either way — which is the single strongest argument for having one.
• For exact matching on a small vocabulary such as names or place names, add an explicit alias table for the handful of affected vowel clusters. It is a bounded list, not an open-ended problem.
• Do not attempt to “correct” one convention to the other in stored data. Both are in current use, and rewriting a person’s name to the convention you prefer is the same class of error as stripping the diacritics entirely.

Sorting is a separate problem again

Normalizing to NFC makes equality work. It does not make ordering work, and Vietnamese ordering is unusual enough that a code-point sort is not merely slightly wrong.

The Vietnamese alphabet treats ă, â, đ, ê, ô, ơ and ư as letters in their own right, positioned next to the letter they derive from — a, ă, â, b, c, d, đ, e, ê and so on. Tone marks are then a lower-level distinction applied on top, so words are grouped first by the base letter and only then ordered by tone. A code-point sort produces neither: the precomposed characters live in scattered blocks from U+1EA0 to U+1EF9 plus the Latin-1 range, so the order it yields reflects Unicode’s allocation history and nothing about Vietnamese.

Correct Vietnamese order (base letter first, then tone):
  an, ăn, ân, ba, ca, da, đa, em, êm, on, ôn, ơn

Code-point order:
  an, ba, ca, da, em, on, ân, ôn, ăn, đa, êm, ơn

>>> sorted(["an", "ăn", "ân", "ba", "đa", "da"])
['an', 'ba', 'da', 'ân', 'ăn', 'đa']        # wrong in three places

Use an ICU collator with the vi locale — PyICU, Intl.Collator("vi"), or COLLATE "vi-VN-x-icu" in PostgreSQL — and pass the locale explicitly rather than relying on the server default. The accent-folded column from the previous section is for recall only; it is the wrong thing to sort on, because folding discards exactly the distinctions the Vietnamese collation is built to order by.