Prompt Whitespace Normaliser

Where these characters come from, and what they cost

Almost nothing on the list above is typed. A no-break space arrives from a word processor, a zero-width space from HTML, a byte-order mark from a file read with the wrong encoding, smart quotes from an editor’s autocorrect, CRLF from a Windows checkout of a prompt file. Each is invisible in the place you would look for it, and each is a different byte sequence from the thing it resembles.

The concrete costs are three. First, tokens: a no-break space is not the space token, so it does not merge with the following word, and a prompt full of them tokenizes measurably worse. Second, caching: an exact-prefix match is unforgiving, and a single stray character anywhere in the supposedly stable prefix moves the whole boundary. Third, comparison — the string equality in your test, your dedupe and your cache key all fail on characters no reviewer can see, which is why a prompt that “did not change” behaves differently after a round trip through a document.

One caution: the quote-straightening toggle is off by default because it is the only option here that changes meaning rather than encoding. In a system prompt that is usually fine. In user content, in a legal quotation or in a language where the apostrophe is a letter, it is not — normalise the frame you control and leave what the user wrote alone.