Normalizing Curly and Straight Quotation Marks Across Languages

The obvious normalisation is to replace U+201C and U+201D with an ASCII double quote and move on. It survives English. It mangles German (whose closing quote is the character you just mapped to an opening one), it destroys French spacing, and it silently changes the spelling of words in any language where the right single quotation mark is a letter rather than punctuation.

The characters involved

There are more of these than people expect, and several are visually near-identical at body-text size.

U+0022  "   QUOTATION MARK                  (ASCII, straight)
U+0027  '   APOSTROPHE                      (ASCII, straight)
U+2018  ‘   LEFT SINGLE QUOTATION MARK
U+2019  ’   RIGHT SINGLE QUOTATION MARK     — also THE apostrophe
U+201A  ‚   SINGLE LOW-9 QUOTATION MARK
U+201B  ‛   SINGLE HIGH-REVERSED-9
U+201C  “   LEFT DOUBLE QUOTATION MARK
U+201D  ”   RIGHT DOUBLE QUOTATION MARK
U+201E  „   DOUBLE LOW-9 QUOTATION MARK
U+201F  ‟   DOUBLE HIGH-REVERSED-9
U+00AB  «   LEFT-POINTING DOUBLE ANGLE      (guillemet)
U+00BB  »   RIGHT-POINTING DOUBLE ANGLE
U+2039  ‹   SINGLE LEFT-POINTING ANGLE
U+203A  ›   SINGLE RIGHT-POINTING ANGLE
U+300C  「  LEFT CORNER BRACKET              (Japanese)
U+300D  」  RIGHT CORNER BRACKET
U+02BC  ʼ   MODIFIER LETTER APOSTROPHE      — category Lm: a LETTER
U+02BB  ʻ   MODIFIER LETTER TURNED COMMA    — Hawaiian ʻokina, a letter
U+05F3  ׳   HEBREW PUNCTUATION GERESH
U+2032  ′   PRIME                            — minutes, feet, not a quote

None of these has a compatibility decomposition to ASCII. NFKC leaves every one of them exactly as it is, so normalisation forms are no help at all here and you are writing an explicit mapping table whether you like it or not.

What each language actually uses

English (US/UK)    “Wort”        U+201C … U+201D
German             „Wort“        U+201E … U+201C   ← closing is the
                                                     English OPENING mark
Swiss German       «Wort»        U+00AB … U+00BB
Polish             „Wort”        U+201E … U+201D   ← same opener as German,
                                                     different closer
French             « Wort »      U+00AB … U+00BB, with U+202F NARROW NO-BREAK
                                 SPACE inside both marks
Swedish, Finnish   ”Wort”        U+201D … U+201D   ← the same mark twice
Danish             »Wort«        U+00BB … U+00AB   ← guillemets pointing IN
Japanese           「Wort」       U+300C … U+300D
Hebrew             "Wort"        commonly ASCII; gershayim U+05F4 for acronyms

Read that table as a list of the assumptions a find-and-replace makes. A rule that maps U+201C to an opening ASCII quote is producing an opening mark where German has a closing one. A rule that maps guillemets to ASCII quotes in French leaves the narrow no-break spaces behind, so the text reads " Wort " with visible gaps inside the quotes. A rule that pairs marks by alternation breaks immediately on Swedish, where the opening and closing marks are the same character and there is nothing to alternate.

There is also no standard-library API for this. ECMA-402 does not expose quotation marks, so Intl cannot tell you what German uses. The data does exist: CLDR publishes quotationStart, quotationEnd, alternateQuotationStart and alternateQuotationEnd per locale in its delimiters data, available from the Unicode CLDR project. If you need locale-correct quotes, that file is the source; a table you typed from memory is not.

The apostrophe problem

U+2019 is doing two unrelated jobs, and this is the part that turns a cosmetic normalisation into a data-corruption one.

As punctuation, it closes a single quotation. As orthography, it is the apostrophe: Unicode’s own recommendation is that U+2019 is the preferred character for the apostrophe, over U+0027. So don’t, l’école and O’Brien contain a right single quotation mark that is part of the word, and a normaliser that strips or rewrites quotation marks is editing vocabulary.

In several orthographies the mark is unambiguously a letter, and Unicode gives those a separate code point in general category Lm precisely so they are not caught by punctuation rules: U+02BB is the Hawaiian ʻokina, and U+02BC is used as a letter in a number of transcription systems and orthographies. In practice, text arrives with all three characters used interchangeably for the same sound, because keyboards make U+2019 easy and the others hard.

The consequences are concrete. A search fold that deletes U+2019 turns Hawai’i into Hawaii, which is fine, and turns a Cyrillic or Caucasian transliteration into a different word, which is not. A “smart quotes” converter that decides direction by alternation gets rock ’n’ roll and the ’90s backwards every time, because both need a closing mark in a position where the alternation state says opening. And a CSV or JSON writer that runs a smart-quote pass over its own output produces a file that no parser will read, because the structural quotes have become typographic ones.

Two normalisers, not one

The confusion in most codebases is that one function is being asked to do two opposite jobs. Separate them and both become easy.

• The matching fold maps every quote-like character to a single canonical form, is applied symmetrically to the index and the query, and never touches stored content. It is allowed to be aggressive and lossy because its output is a key nobody reads.
• The display normaliser converts straight marks into the correct typographic marks for one specific locale, runs once at authoring or render time, and must be locale-aware. It is allowed to be conservative, and it must never run on content you will later compare or parse.

// The matching fold. Symmetric, lossy, key-only.
const QUOTE_FOLD = new Map([
  ["\u2018", "'"], ["\u2019", "'"], ["\u201A", "'"], ["\u201B", "'"],
  ["\u02BC", "'"], ["\u02BB", "'"], ["\u2039", "'"], ["\u203A", "'"],
  ["\u201C", '"'], ["\u201D", '"'], ["\u201E", '"'], ["\u201F", '"'],
  ["\u00AB", '"'], ["\u00BB", '"'], ["\u300C", '"'], ["\u300D", '"'],
]);

export function foldQuotes(input) {
  let out = "";
  for (const ch of input) out += QUOTE_FOLD.get(ch) ?? ch;
  // French guillemets carry narrow no-break spaces inside them;
  // collapse those too or the folded string keeps the gap.
  return out.replace(/[\u00A0\u202F\u2009]/gu, " ");
}

foldQuotes("« Bonjour »")   //  " Bonjour "
foldQuotes("„Guten Tag“")   //  "Guten Tag"

Building them

1. Decide which of the two you need. If the requirement is “a search for don’t should find don’t”, you need the fold and must not touch stored text. If it is “our published German copy should use the right marks”, you need the display normaliser and must not run it on user input.
2. For the fold: build the map above, apply it to both the indexed field and the query in the same function, and collapse the non-breaking spaces so French text folds to the same key as English.
3. For display: read the delimiters from CLDR for the target locale rather than hard-coding a pair. Handle the second level too — quotations inside quotations use the alternate pair, and in German that is the single low and single left marks, not the double ones.
4. Insert French spacing explicitly. U+202F NARROW NO-BREAK SPACE after the opening guillemet and before the closing one; an ordinary space allows a line break in the wrong place and an ASCII space is typographically wrong.
5. Never convert straight to curly automatically on text that is code, a filename, a CSV field, a URL or a template. Restrict the display normaliser to fields explicitly marked as prose.
6. Treat U+2019 inside a word as a letter. If the character has a letter on both sides of it, leave it alone regardless of what the rule says about quotation marks; that single condition removes the majority of apostrophe corruption.
7. Test with the table in this page as fixtures: a German sentence, a French one with guillemets, a Swedish one where both marks are identical, and an English one containing the ’90s. Any implementation that passes all four is unlikely to be caught out by the rest.