Why Korean Text Comparison Fails Without NFC Normalization

Two Korean strings render as 한글. One has a length of 2 and the other of 6. They fail an equality check, produce two rows under a unique index, and return no results when one is used to search text stored in the other form. Neither is wrong; Hangul has an algorithmic decomposition and both forms are valid Unicode.

The symptom

const a = "한글";                       // typed on Windows, NFC
const b = "한글";                       // read from a macOS filename, NFD

a === b            // false
a.length           // 2
b.length           // 6
[...a].length      // 2
[...b].length      // 6

Buffer.from(a, "utf8")   // ed 95 9c  ea b8 80              (6 bytes)
Buffer.from(b, "utf8")   // e1 84 92 e1 85 a1 e1 86 ab
                         // e1 84 80 e1 85 b3 e1 86 af      (18 bytes)

a === b.normalize("NFC")   // true

Six bytes against eighteen: the decomposed form is three times the size, which on its own is enough to overflow a VARCHAR sized in bytes. But the visible symptom is usually neither the length nor the bytes. It is that a user searches for their own name and finds nothing, or signs in and is told the account does not exist, while the record is plainly there in the table.

The Hangul composition algorithm

Hangul is unusual among scripts in that its composition is arithmetic rather than a lookup table. A syllable block is built from an initial consonant (choseong), a medial vowel (jungseong) and an optional final consonant (jongseong), and Unicode allocated all 11,172 possible combinations in one contiguous run, U+AC00 to U+D7A3, in a predictable order.

S = 0xAC00 + (L × 21 + V) × 28 + T

  L  choseong index    0–18   (19 initial consonants)
  V  jungseong index   0–20   (21 vowels)
  T  jongseong index   0–27   (0 = no final consonant)

Worked for 한 (han):
  L = 18  ㅎ  hieuh
  V = 0   ㅏ  a
  T = 4   ㄴ  nieun

  S = 0xAC00 + (18 × 21 + 0) × 28 + 4
    = 44032  + 378 × 28 + 4
    = 44032  + 10584 + 4
    = 54620
    = 0xD55C  → 한   ✓

The jamo the decomposed form uses:
  L → 0x1100 + L = 0x1112   ᄒ   choseong hieuh
  V → 0x1161 + V = 0x1161   ᅡ   jungseong a
  T → 0x11A7 + T = 0x11AB   ᆫ   jongseong nieun

And 글 (geul): L = 0 ㄱ, V = 18 ㅡ, T = 8 ㄹ
  S = 0xAC00 + (0 × 21 + 18) × 28 + 8 = 44544 = 0xAE00  →  글  ✓

NFD applies that arithmetic backwards, producing two or three conjoining jamo from U+1100–U+11FF. NFC applies it forwards. Because the mapping is a formula, every Hangul syllable decomposes and every valid jamo sequence recomposes: unlike Latin, there are no gaps and no composition exclusions, so NFC(NFD(x)) is reliably x for Korean text.

One consequence worth holding on to: the decomposed form uses the conjoining jamo block, and the choseong and jongseong ranges are different code points even for the same-looking consonant. Nieun as an initial is U+1102; as a final it is U+11AB. Code that tries to find a consonant by scanning for one code point will miss half its occurrences.

A second consequence is that Korean is unusually exposed to length-based bugs, because the ratio between the forms is larger than for any Latin script. A Latin accented character grows by one byte under decomposition; a Hangul syllable grows from three bytes to six or nine, so a column, a header value or a message field sized against composed Korean can overflow on the same text decomposed. The error arrives as a truncation or a database write failure with no mention of encoding, and it appears only for the subset of users whose input travelled through a decomposing path.

Truncation itself has a further trap. Cutting a decomposed string at a code-point boundary can leave a choseong and a jungseong without their jongseong, which is a valid and different syllable, or a bare final consonant with nothing to attach to. The visible result is a name that is not merely shortened but misspelled, which is worse than an ellipsis and much harder to trace back to a substring call.

Where decomposed Korean comes from

• macOS filenames. The historical HFS+ behaviour applied to Hangul as much as to Latin, so Korean filenames written on older Macs are decomposed on disk. Reading a directory and using the name as a database key transports the form into your data, and the same filesystem mismatch applies.
• Zip archives and uploads from macOS. Same mechanism, one step removed, and the reason a Korean-language document set uploaded from one laptop indexes differently from the same set uploaded from another.
• PDF and OCR extraction. Text layers frequently carry jamo rather than syllables, depending on the font encoding the producer used.
• Some IME and clipboard paths. An input method composes a syllable from keystrokes, and a partially composed syllable is jamo by definition. If a field captures input on keystroke rather than on composition end, it can store the intermediate form.

Model output is another source, and an underrated one: a language model emits whatever its tokeniser decodes to, and Korean training data contains both forms. There is no guarantee generated Korean is composed, so normalising model output before storing it is as necessary as normalising user input.

Compatibility jamo, a third form

There is a third block that catches people who thought they had the problem solved: Hangul Compatibility Jamo, U+3130–U+318F. These are the letters as they appear on a keyboard cap and in a list of the alphabet — ㄱ is U+3131, ㅏ is U+314F, ㅎ is U+314E. They are not the conjoining jamo, and they never combine into a syllable.

They carry compatibility decompositions to the conjoining block, which means NFKC handles them and NFC does not. This is the one place in Korean text handling where the choice between NFC and NFKC has a real consequence, and it is worth confirming in your own runtime rather than taking on trust:

const compat = "\u314E\u314F\u3134";   // ㅎ ㅏ ㄴ  compatibility jamo

compat.normalize("NFC").length    // unchanged — NFC does not touch them
compat.normalize("NFKC")          // check this in your runtime:
                                  // compatibility decomposition maps each
                                  // to a conjoining jamo, and canonical
                                  // composition then builds the syllable

// Run it. Print the code points either side. The answer is one line
// of evidence and it decides whether NFC is enough for your input.

If your input can contain compatibility jamo — which it can, if anybody pastes from a keyboard-layout page, a language-learning site or a spec document — NFC alone leaves those strings unmatched against the same text stored as syllables. NFKC is the stronger choice, at the cost of also folding full-width Latin, circled numbers and other compatibility variants elsewhere in the same string.

The fix

1. Normalise at every entry point, not in the comparison. HTTP handler, file ingest, message consumer, model-output parser. One helper, called at each boundary.
2. Choose NFC by default and NFKC if your corpus contains compatibility jamo or full-width characters. Write the choice down with the reason; a future reader will otherwise flip it.
3. Backfill. Existing rows are the actual bug for existing users. UPDATE ... SET name = normalize(name, NFC) in PostgreSQL, which has had a normalize() function since version 13, or a batched job in application code elsewhere.
4. Add a database-level check so it cannot recur: CHECK (name IS NFC NORMALIZED) in PostgreSQL 13 and later. This is a genuinely useful constraint because it fails the write rather than the read.
5. Consider a non-deterministic ICU collation if you cannot control every writer. PostgreSQL 12 introduced these, and a collation created with deterministic = false makes canonically equivalent strings compare equal. Know the cost first: LIKE and pattern matching are not supported on a non-deterministic collation and error out rather than degrading, so it is not a free retrofit.
6. Rebuild the search index after the backfill. The tokens stored before the fix are decomposed and no query will reach them.
7. Test with a fixture that is genuinely both forms. A test written by typing the same characters twice into an editor produces two identical strings and passes trivially; build one side with explicit escapes so the test can actually fail.