Combining Characters and Precomposed Characters: What Actually Differs

Two strings display as café. One has a length of 4, the other a length of 5. They compare unequal, hash differently, and produce different rows in a database with a unique index. Nothing is corrupt; they are two legal encodings of the same text, and Unicode considers both correct.

One glyph, two encodings

Take the character e-acute. Unicode gives it two representations.

• Precomposed: a single code point, U+00E9 LATIN SMALL LETTER E WITH ACUTE. One code point, one glyph.
• Decomposed: two code points, U+0065 LATIN SMALL LETTER E followed by U+0301 COMBINING ACUTE ACCENT. The second has no width of its own; it is drawn over whatever precedes it.

Unicode calls these canonically equivalent. That is a formal statement with a specific meaning: conforming software must not treat them as different text, and any process that distinguishes them is doing something Unicode does not guarantee. It is not a statement that they are the same string, and this is the gap every bug in this area falls into. Your language’s === compares code units. It has no idea what canonical equivalence is.

At the byte level

Here is the same glyph twice, in UTF-8, with the encoding shown explicitly.

Precomposed  "é"
  code point  U+00E9
  UTF-8       C3 A9              (2 bytes)
  UTF-16      00E9               (1 code unit)
  JS .length  1

Decomposed   "é"
  code points U+0065  U+0301
  UTF-8       65  CC 81          (3 bytes)
  UTF-16      0065  0301         (2 code units)
  JS .length  2

"\u00E9" === "\u0065\u0301"                          // false
"\u00E9".normalize("NFD") === "\u0065\u0301"          // true
"\u00E9" === "\u0065\u0301".normalize("NFC")          // true

The bytes are the whole story. A hash function sees C3 A9 in one case and 65 CC 81 in the other and produces two different hashes. A B-tree index sorts them into different places. A byte-length limit counts one as two and the other as three, which is how a name that fits in a 255-byte column in one form overflows it in the other.

Note also what .length is measuring. In JavaScript, Java and C# it is UTF-16 code units, so it is 1 and 2 respectively. In Python 3 it is code points, so it is 1 and 2 as well. In Go, len() on a string is bytes, so it is 2 and 3. None of those is the number a person would give, which is 1. The user-perceived unit is the grapheme cluster, and the only portable way to count those is a segmenter — Intl.Segmenter in JavaScript, ICU BreakIterator elsewhere.

Canonical ordering and combining classes

Once marks are separate code points, a second question appears: what if there are two of them? Vietnamese ệ is a base plus a circumflex plus a dot below. Which order?

Unicode answers with the canonical combining class, a number attached to every combining character. Marks that attach below have class 220; marks that attach above have class 230. Normalisation sorts adjacent marks by that number, stably, so a below-mark always precedes an above-mark regardless of how the text was typed. This is why e + U+0323 + U+0302 and e + U+0302 + U+0323 normalise to the same sequence, and it is the reason normalisation is a genuine canonical form rather than just a decomposition.

// Same two marks, typed in opposite orders.
const a = "e\u0323\u0302";   // dot below (ccc 220), circumflex (ccc 230)
const b = "e\u0302\u0323";   // circumflex first

a === b                                   // false
a.normalize("NFD") === b.normalize("NFD") // true  — reordered by class
a.normalize("NFC") === b.normalize("NFC") // true  — both become U+1EC7 "ệ"

Marks with class 0 are never reordered, and marks that attach to the same position keep their relative order, because reordering those would change what is drawn. The class number is data in the Unicode Character Database, published by the Unicode Consortium alongside each release; see the Unicode Standard Annex #15, Unicode Normalization Forms for the algorithm itself.

Why Unicode has both

The duplication is not an oversight. Unicode was designed to round-trip existing character sets: any text encoded in Latin-1, Shift-JIS or KOI8-R had to convert to Unicode and back without loss. Latin-1 contains é as a single byte, so Unicode needed a single code point for it, and that is where the precomposed characters come from. The combining marks exist because no fixed set of precomposed characters can cover every language, and new ones cannot be added freely once the standard is stable.

The consequence is that composition is incomplete and asymmetric. Decomposition always works: every precomposed character has a canonical decomposition. Composition does not, because there are combinations with no precomposed form at all. There is no single code point for z with a diaeresis, so z + U+0308 stays two code points through NFC. Any code that assumes NFC produces one code point per visible character is wrong on those, on all of Devanagari and Thai, and on every emoji sequence.

Composition is also subject to the composition exclusion list, a set of characters that decompose but deliberately do not recompose. So NFC(NFD(x)) === x holds for most text and not for all of it, which makes normalising once at the boundary far safer than normalising repeatedly and assuming idempotence across forms.

What this breaks

• Equality and uniqueness. A UNIQUE constraint on a byte-compared column accepts both forms as separate rows, and you get two accounts for one person. PostgreSQL 12 and later can avoid this with a non-deterministic ICU collation, at the cost of some index-based optimisations.
• Search. A tokeniser that saw NFC text at index time and NFD text at query time produces no matches at all, with no error. This is the failure that makes people suspect the search engine when the fault is in the ingest path.
• Filenames. The classic cross-platform case, because macOS and Linux disagree about which form ends up on disk. That is a bug worth understanding on its own.
• Truncation. Cutting a decomposed string at a fixed length can leave a combining mark with nothing to combine with. It then attaches to whatever is next, or renders on a dotted circle. Truncate on grapheme boundaries, not on code units.
• Regular expressions. /^.$/ matches the precomposed form and not the decomposed one. So does a character class written with precomposed characters in it.

The rule that follows is short: normalise at the edge. Pick NFC, apply it in the request handler and in the ingest job, and let everything behind that boundary assume one form. Normalising deep in the stack means every comparison has to remember to do it, and one that forgets is invisible until a user with an accent in their name signs up twice.