Text Normalisation for Search and Retrieval

Every normalisation step throws information away in exchange for making two things that were different compare equal. That is the whole trade, and the bugs come from making the trade accidentally — usually by calling .lower() and moving on.

Normalisation is a lossy decision

The question is never “should I normalise”. It is “which distinctions am I choosing to erase, and does anything downstream need them?” Erasing case means Apple and apple match, and also that you can no longer tell a company from a fruit. Erasing diacritics means café matches cafe, and also that Spanish año and ano collapse into one term — which is a difference native speakers notice.

The one structural rule that prevents most disasters: whatever you decide, the identical function must run at index time and at query time. A mismatch produces zero results for queries that are obviously correct, and it is invisible in unit tests because both halves work fine alone. Store the raw text unchanged, always, so any decision can be revisited without re-fetching the corpus.

The four Unicode forms

The character é has two legitimate encodings: the single codepoint U+00E9, or the letter e followed by U+0301, a combining acute accent. They render identically in every font. In Python, "café" == "café" is False, and their lengths differ by one. Text copied from a Mac filesystem tends to arrive decomposed; text from a web form tends to arrive composed. Both end up in the same index.

Unicode Standard Annex #15 defines four normalisation forms, and the distinction between them is two independent choices:

The practical policy that survives contact with real data: store NFC, and derive an NFKD-based folded key for matching, keeping both. Using the compatibility forms for storage is how you silently destroy mathematical notation and chemical formulae, where the superscript genuinely means something.

The two words doing the work in that table are canonical and compatibility, and the distinction is worth keeping straight because it is the difference between a reversible fix and a deliberate loss. Canonical equivalence (NFC and NFD) relates sequences that Unicode considers the same character differently encoded — the two spellings of é — and converting between them changes nothing a reader can see. Compatibility equivalence (NFKC and NFKD) relates characters that are merely similar in use: a superscript two and a digit two, a full-width and a half-width letter, a ligature and its letters. That conversion throws information away on purpose, which is exactly why it belongs in a matching key and never in what you store.

Case folding is not lower()

Lowercasing is a locale-sensitive, non-invertible operation with documented exceptions, and treating it as a simple character map produces real defects.

German sharp s. The uppercase of ß is historically SS, so "Straße".upper() gives "STRASSE" and lowercasing that gives "strasse" — which no longer equals "straße". Python provides str.casefold() for exactly this class of problem: it maps ß to ss, so both spellings fold to the same key. For matching, use casefold(), not lower().

Turkish dotted and dotless i. Turkish has four i letters: i/İ with dots and ı/I without. In Turkish locale rules, the lowercase of I is ı, not i. Language-neutral lowercasing therefore mangles Turkish words, and locale-aware lowercasing mangles English ones. Worse, language-neutral lowercasing of İ (U+0130) yields two codepoints — i plus a combining dot above — so the string gets longer, which breaks any code assuming lowercasing preserves length or offsets.

Offsets in general. Any normalisation can change string length. If you are storing character offsets for highlighting or entity spans, compute them against the stored raw text, never against a normalised copy, or your highlights will drift on exactly the documents that needed normalising.

The bugs, named

• Invisible characters. Zero-width space, zero-width joiner, byte-order mark, soft hyphen, and non-breaking space. They arrive from PDFs, word processors and copy-paste, they are invisible in every log and terminal, and they make two apparently identical strings unequal. Strip or normalise them explicitly; do not wait to discover them.
• Smart punctuation. A word processor turns ' into ’ and -- into an em dash. A user searching for don't with a straight apostrophe finds nothing. Fold quotes and dashes to ASCII in the matching key.
• Confusables. Cyrillic а, Greek ο and Latin a/o render identically. Unicode Technical Standard #39 defines a confusables mapping precisely for detecting this, and it matters for anything security-adjacent — domains, usernames, payee names.
• Length is not what you think. A single emoji with skin-tone and joiner sequences can be many codepoints. Any limit expressed in characters, any truncation, and any “first 200 characters” snippet can split a grapheme cluster and produce mojibake. Truncate on grapheme boundaries.

Writing it down as a policy

import unicodedata, re

INVISIBLE = re.compile(r"[\u00ad\u200b-\u200f\u2060\ufeff]")
QUOTES = str.maketrans({"\u2018": "'", "\u2019": "'",
                        "\u201c": '"', "\u201d": '"',
                        "\u2013": "-", "\u2014": "-"})

def storage_form(text: str) -> str:
    """What you keep. Reversible-ish, safe to display."""
    text = unicodedata.normalize("NFC", text)
    return INVISIBLE.sub("", text).replace("\u00a0", " ")

def match_key(text: str) -> str:
    """What you compare and index. Deliberately lossy."""
    text = storage_form(text).translate(QUOTES)
    text = unicodedata.normalize("NFKD", text)
    text = "".join(c for c in text if not unicodedata.combining(c))
    return " ".join(text.casefold().split())

Two functions, two purposes, and the split is the whole point. The decision to strip diacritics lives in match_key, where it is visible and arguable, rather than being smeared across a pipeline. When a Spanish speaker complains that año and ano now match, there is one line to discuss — and the honest resolution is usually to index both keys and boost exact matches, rather than to pick a side.