Stop Words: A Practice Worth Questioning

Removing stop words is the most-copied step in NLP tutorials and one of the least examined. It was a storage optimisation for a constraint that disappeared decades ago, and the default English list contains the word not.

Where the practice came from

In an inverted index, the posting list for a term holds every document containing it. The posting list for the holds essentially every document in the collection. In the 1970s, when index size was measured against tape and disk that cost real money per megabyte, dropping the two hundred commonest words removed a large share of the index and a large share of the query processing cost, in exchange for queries that nobody was typing anyway — search interfaces of the era were keyword boxes, not sentences.

Both halves of that trade have inverted. Storage is effectively free at the scale most indexes operate, modern posting lists are compressed and skip-listed so a frequent term costs far less than its length suggests, and users type whole questions. The optimisation is still in every tutorial because it was in the tutorial before it.

Term weighting already does this

The deeper point is that stop word removal is a crude version of something the ranking function already does properly. Inverse document frequency is, by construction, a measure of how uninformative a term is: a term appearing in every document gets an IDF at or near zero, so its contribution to the score is at or near nothing. That is a continuous, corpus-specific version of the same judgement, and it is computed for free as part of TF-IDF and BM25.

The difference that matters is what happens to a term that is common in general English but rare in your corpus, or the reverse. A fixed list applies English-wide judgements to a corpus of legal filings, where shall and party are near-universal and carry nothing, while can may be genuinely discriminating. IDF learns that from the corpus. The list cannot.

The magnitudes are worth seeing rather than taking on trust. Under the smoothed formula in the TF-IDF page, a term appearing in every one of a million documents scores an IDF of about 1.0, while a term in a thousand of them scores about 7.9 — so before any tuning, the rare term already contributes roughly eight times as much per occurrence. BM25 compounds this: its IDF component is smaller still for near-universal terms, and its length normalisation removes the advantage a long document would otherwise get from repeating them. Deleting those terms saves you a factor you were already getting for free, and costs you the ability to answer a query where they were the only evidence.

Three things it breaks

• Negation, and therefore sentiment. NLTK’s English stop word list includes not, no, nor and the contracted forms. Run the standard tutorial pipeline over the room was not clean and you have indexed room clean. Every negated statement in the corpus now says the opposite of what it said. This is not a subtle failure and it ships regularly.
• Phrase queries made entirely of stop words. The standard examples are real queries: to be or not to be, The Who, Take That, let it be, The The. After stop word removal these are empty strings. The system cannot return a wrong answer; it has nothing left to search for.
• Short, function-word-carrying queries. flights to paris and flights from paris collapse to the same three terms. So do vitamin a and vitamin, since single letters are usually on the list too. Precisely the queries where every term counts are the ones that lose a term.

It is worth noticing that current search stacks already agree with this. Elasticsearch’s default standard analyzer ships with no stop word list at all — you have to opt in by choosing a language-specific analyzer or configuring a filter. The default changed because the default was wrong.

If you keep a list, own it

There are still defensible uses, and it is worth being precise about why each one is different from retrieval. Topic models produce unreadable output when function words dominate — a topic whose top terms are the, of and and is not a topic — because the model has no ranking function applying IDF on its behalf; it is fitting raw counts, so the removal has to happen upstream. RAKE goes further and uses the stop words structurally: it splits text at them to find candidate phrase boundaries, so the list is not a filter there but the algorithm itself. And a bag-of-words feature matrix for a small linear classifier benefits from dropping terms with no class signal, if only to keep the feature count down — though a regularised model will drive their weights towards zero anyway, so the gain is memory rather than accuracy.

What all three share is that nothing downstream is going to weight the terms for them. That is the actual rule: remove stop words when the consumer counts terms, keep them when the consumer weights terms or reads the sentence.

When you do keep one, three rules make it honest. Derive it from your own corpus by document frequency rather than importing a list built for general English — a term in ninety-five per cent of your documents is a stop word for you whatever a library thinks. Remove the negations from whatever list you start with, always. And apply it in exactly one place, to exactly one destination: never to text going to a model, never to the raw stored document, and never asymmetrically between index time and query time, which produces queries that match nothing for reasons that take an afternoon to find.