Text Preprocessing: What You Still Need to Do

The preprocessing pipeline in every NLP tutorial — lowercase, strip punctuation, remove stop words, stem — was designed to make a sparse bag-of-words model work. Feed the output of it to a language model and you have deleted signal for nothing.

The pipeline you were taught

The canonical sequence is: decode to text, lowercase, strip punctuation, tokenize on whitespace, drop stop words, stem, then count. Every step exists to shrink a vocabulary. In 2005 that mattered enormously: a bag-of-words model with a 200,000-term vocabulary over a million documents was a real memory problem, and collapsing Running, running and runs into one feature made the matrix smaller and the counts denser. The steps were compression, and they were correct for their target.

Nothing about that reasoning transfers to a model with a learned subword vocabulary. The vocabulary is fixed at 30,000 to 200,000 pieces regardless of what you send, casing is a piece of information the model was trained on, and punctuation carries syntax. You are not compressing anything; you are deleting evidence.

What subword tokenizers made pointless

• Lowercasing. It destroys the single strongest signal for entity detection in English — capitalisation. apple and Apple are different things and a modern tokenizer represents them differently on purpose.
• Punctuation stripping. Sentence boundaries, quoted speech, negation scope and code all live in punctuation. Removing it turns two sentences into one ungrammatical one.
• Stemming, for model input. A byte-pair encoder already splits running into pieces that share a prefix with run. Stemming first hands the tokenizer a word that does not exist and usually produces more tokens, not fewer.
• Stop word removal, for model input. Function words are how the model resolves who did what to whom. See the longer argument; the short version is that removing not is the most expensive two-line change in the field.

The four steps that matter more now

The pipeline did not get shorter, it got different. These four are worth real engineering attention, and three of them are usually missing from the codebases that still lowercase everything.

Encoding and Unicode normalization. Mojibake from a mis-decoded byte stream, and the fact that the same visible character has several byte representations, will silently halve retrieval recall. This is the one step to do first and do properly — the forms and the bugs they cause are a page of their own.

Boilerplate removal. If the text came from HTML, the navigation, cookie banner and footer are typically a large fraction of the characters and none of the meaning. They also repeat across every page, which poisons both TF-IDF statistics and embedding clusters. Libraries such as trafilatura and Readability exist for exactly this; the mistake is running a naive tag-stripper and calling it done.

Deduplication. Near-duplicates inflate term statistics, dominate clusters, and — if the text is training or evaluation data — leak. Lee et al., in Deduplicating Training Data Makes Language Models Better (ACL 2022), reported that removing duplicated sequences from training corpora improved models and cut memorised output substantially; the same reasoning applies at the far smaller scale of a retrieval corpus. MinHash over character shingles is the standard cheap tool.

Chunking. For anything that will be retrieved, how you split documents is a bigger lever on quality than any tokenization choice, and it has its own failure modes — splitting strategies compared covers them.

Notice what the four have in common. None of them is about vocabulary size, which is what the classical pipeline was optimising, and all four are about the input being wrong rather than large: wrong bytes, wrong content, duplicated content, or a unit of text that does not correspond to an answer. They are also all cheap — every one runs on a CPU at microseconds to low milliseconds per document, with no per-document fee — which is why it is worth doing them thoroughly before anything expensive touches the text. A document that arrives at an embedding model carrying a cookie banner has cost you tokens for the banner, diluted the vector with it, and will keep doing both on every re-index.

Preprocess for the destination

The rule that resolves every argument about this: preprocessing is not a property of your corpus, it is a property of the consumer. One document can go three places and should be treated differently in each.

A pipeline worth copying

The shared front half — the part every destination needs — is short enough to own rather than import:

import re, unicodedata

ZERO_WIDTH = re.compile(r"[\u200b-\u200f\ufeff]")
WS = re.compile(r"[^\S\n]+")          # runs of spaces, not newlines
BLANKS = re.compile(r"\n{3,}")

def clean(text: str) -> str:
    # 1. one canonical byte representation per visible character
    text = unicodedata.normalize("NFC", text)
    # 2. invisible characters that break matching and tokenizing
    text = ZERO_WIDTH.sub("", text)
    text = text.replace("\u00a0", " ")   # nbsp -> space
    # 3. collapse whitespace but keep paragraph structure
    text = WS.sub(" ", text)
    text = BLANKS.sub("\n\n", text)
    return text.strip()

Three things it deliberately does not do: it does not lowercase, it does not touch punctuation, and it does not collapse newlines entirely — paragraph boundaries are the cheapest chunking signal you have. Everything past this point belongs to one destination, and should live next to the code for that destination rather than in a shared preprocess() that four callers quietly disagree about.