Normalizing German Umlauts and Eszett for Search

A German customer database contains Müller, Mueller and Muller, all of them the same family, none of them a typo. Making a search box find all three is not a matter of stripping accents, and the rule that handles ß introduces a problem that accent stripping does not have.

One name, three spellings in the wild

German umlauts have an official ASCII fallback, which is what makes this different from French or Spanish. When ä ö ü cannot be typed or displayed, the correct substitution is ae oe ue — not a o u. That convention is centuries old, it is what German speakers type on a non-German keyboard, and it is what passports, older systems, email addresses and domain names contain.

Müller   umlaut form,   as printed                    6 chars
Mueller  expanded form, passport / ASCII systems      7 chars
Muller   stripped form, foreign systems and imports   6 chars

Straße   as printed                                   6 chars
Strasse  expanded, and also the standard Swiss spelling 7 chars
Strase   what naive accent-stripping produces          6 chars  <- wrong

Three variants that a search box must unify, and note the last line: the generic accent-strip that works for French produces a string that is not a German word at all, because ß is not a letter with a diacritic on it. It is its own letter, and its ASCII form is two characters.

Eszett is not a decoration

ß (U+00DF) has no canonical decomposition and no compatibility decomposition, so no normalization form touches it. NFC leaves it, NFD leaves it, NFKC leaves it. It is unified only by case operations:

"ß".upper()      -> "SS"      one character becomes two
"ß".casefold()   -> "ss"      one character becomes two
"ẞ".lower()      -> "ß"       U+1E9E, CAPITAL SHARP S
"ẞ".casefold()   -> "ss"

unicodedata.normalize("NFKC", "ß") == "ß"     -> True   (unchanged)
unicodedata.decomposition("ß")                -> ""     (none)

The capital form ẞ at U+1E9E is worth knowing about because it is now a live spelling. The Council for German Orthography admitted the capital eszett as a permitted alternative to SS in all-caps spelling in its 2017 revision, so STRAẞE and STRASSE are both correct today and both occur in real data. Any fold that handles ß and forgets ẞ will miss the all-caps rows.

Case folding to ss gives you the expansion for free, which is why casefold() and not lower() is the right function for a German search key — and why German cannot simply reuse the strip-the-combining-marks recipe that general diacritic-insensitive search is built on. But it also means the expansion happens inside an operation nobody thinks of as changing text, which is where the next problem comes from.

The expansion changes the length

Every accent fold discussed elsewhere in this cluster is one-character-to-one-character: é becomes e, ñ becomes n, offsets are preserved, and a match found in the folded string can be highlighted in the original by using the same index. The German folds break that, in two places: ß to ss, and ü to ue.

original : "Die Straße ist gesperrt"
                 0123456789...
           "Straße" begins at index 4, ends at index 10

folded   : "die strasse ist gesperrt"
           "strasse" begins at index 4, ends at index 11    <- +1

original : "Herr Müller wohnt hier"
           "Müller" at 5..11
folded   : "herr mueller wohnt hier"
           "mueller" at 5..12                                <- +1

Every offset after the first expansion is shifted, and the shift
accumulates: two umlauts and one eszett in a sentence is +3.

The consequences are concrete and they are all silent — nothing throws, the highlight is simply one character off, or the snippet is truncated mid-word, or a stored annotation span drifts:

• Search highlighting that finds a match in the analysed text and applies the offsets to the raw text underlines the wrong characters.
• Fixed-width fields. A VARCHAR(20) that holds a folded key overflows for a name that fits comfortably unfolded. Size derived columns generously; the worst case is roughly twice the original for a string of nothing but umlauts and eszett.
• Regular expressions with fixed-length lookbehind. Some engines require a fixed width, and the assumption that a folded token has the width of the original is no longer true.
• Truncation for display. Cutting the folded string at N characters and mapping back to the original cuts in the wrong place.

The fix is to stop using folded offsets against unfolded text. Either keep an offset map produced by the fold — a list of original-index to folded-index pairs, built as you emit — or re-run the match against the original string with an accent-insensitive matcher once you know which token matched. The second is simpler and is usually fast enough, because it runs on a single result rather than on the corpus.

Two German folds, and when each is right

German has two standard sorting and folding conventions, both defined in DIN 5007, and they disagree on exactly the point at issue.

• DIN 5007-1, the dictionary rule. ä is treated as a, ö as o, ü as u. This is the rule for ordinary word lists, and it is what the CLDR German collation implements by default — which is why Ähre sorts next to Ahre and not after Z, in sharp contrast to the Nordic languages, where the same code points sort after z.
• DIN 5007-2, the phone-book rule. ä is treated as ae, ö as oe, ü as ue, ß as ss. This is the rule for lists of names, and it is the one that makes Müller and Mueller land together.

For a search index you generally want both, because users type all three variants. Generate two folded forms per token — the expanded one and the stripped one — and index both alongside the original. Then Müller, Mueller and Muller all reach the same document through at least one path. ICU ships a German-specific ASCII transform for exactly the expanded direction, so you rarely need to write the table yourself; the ICU transforms user guide documents how to invoke it.

Building the analyser

1. Normalize to NFC on write, so that ü is one code point rather than u plus a combining diaeresis. Every rule below matches on the composed character.
2. Emit the original token into the index, unfolded and case-preserved. This is what exact-match ranking scores against, and deleting it is how you end up unable to distinguish an exact hit from a fold hit.
3. Emit the expanded fold — ä→ae, ö→oe, ü→ue, ß→ss, ẞ→ss — then lowercase. This is the DIN 5007-2 form and it is the one that matters for names.
4. Emit the stripped fold — ä→a, ö→o, ü→u, ß→ss — for users who typed no umlaut at all. Note that ß still expands here: there is no correct single-character fold for it.
5. Apply the same three transforms to the query and search all fields, boosting the original field highest, the expanded fold next, the stripped fold lowest. A query for Müller then ranks the exact spelling above Mueller above Muller, which is the order a German user expects.
6. Do not use folded offsets for highlighting. Re-match against the original, or carry an offset map. This is the step this page exists for.