TF-IDF Explained and Implemented

TF-IDF is two ideas multiplied together, and both of them are things you already believe about language. It is worth implementing once from scratch, because the version in your library has four configuration choices that change the numbers and are usually left at defaults nobody read.

Two intuitions

The first: a term that appears often in a document is probably what the document is about. That is term frequency, and on its own it ranks every document by how many times it says the.

The second fixes that: a term that appears in nearly every document tells you nothing about which document you want. Karen Spärck Jones set this out in A Statistical Interpretation of Term Specificity and Its Application in Retrieval (Journal of Documentation, 1972), and the weighting she proposed — scale a term’s weight by the inverse of how many documents contain it — is still what everyone uses. It is a fifty-year-old idea that has survived every representation change since, which is unusual enough to be worth noticing.

Multiply them and each term in each document gets a weight that is high only when the term is frequent here and rare elsewhere. A document becomes a sparse vector over the vocabulary; similarity between two documents is the cosine between their vectors.

The formula, and the variant trap

The version most people write down is tfidf(t, d) = tf(t, d) × log(N / df(t)), where N is the number of documents and df(t) is how many contain the term. That formula has a hole: a term appearing in every document gets log(1) = 0, and a term appearing in no document divides by zero.

Which is why every library smooths it, and why two libraries disagree about the numbers. scikit-learn’s TfidfVectorizer documents its default as idf(t) = ln((1 + N) / (1 + df(t))) + 1 with smooth_idf=True, then L2-normalises each row. The trailing + 1 matters: it means a term present in every document still contributes rather than vanishing, which is a defensible choice and not the one the textbook formula makes. If you compare weights across two implementations without checking this, the discrepancy will look like a bug in your code.

Three more choices ride along with it: whether term frequency is raw, logarithmic (1 + log tf) or length-normalised; whether the document vector is L2-normalised, which is what makes cosine similarity equal a dot product; and what the analyser counts as a term. The last one is the biggest and it is not part of the formula at all — see what the analysis chain does.

Thirty lines that implement it

No dependencies, so nothing is hidden. This is the smoothed variant above, with L2 normalisation, and it is enough to rank a corpus of a few hundred thousand short documents:

import math, re
from collections import Counter

TOKEN = re.compile(r"\w+", re.UNICODE)

def tokenize(text):
    return TOKEN.findall(text.lower())

def fit(docs):
    """Return (idf, doc_vectors). docs: list[str]."""
    tokenized = [tokenize(d) for d in docs]
    n = len(tokenized)
    df = Counter()
    for toks in tokenized:
        df.update(set(toks))
    idf = {t: math.log((1 + n) / (1 + c)) + 1 for t, c in df.items()}

    vectors = []
    for toks in tokenized:
        tf = Counter(toks)
        vec = {t: c * idf[t] for t, c in tf.items()}
        norm = math.sqrt(sum(w * w for w in vec.values())) or 1.0
        vectors.append({t: w / norm for t, w in vec.items()})
    return idf, vectors

def cosine(a, b):
    # both already L2-normalised, so the dot product is the cosine
    small, large = (a, b) if len(a) < len(b) else (b, a)
    return sum(w * large.get(t, 0.0) for t, w in small.items())

def top_terms(vec, k=5):
    return sorted(vec.items(), key=lambda kv: -kv[1])[:k]

top_terms is the part worth keeping. Print it for any document and you get the model’s reason for every score it produces, in terms a non-engineer can read. That property — a decision you can explain by pointing at five words and their weights — is the one dense representations do not have, and it is why TF-IDF still appears in systems that have to answer to somebody.

What it cannot express

• Synonymy. car and automobile are orthogonal dimensions. A query and a document that say the same thing in different words score zero against each other.
• Word order. dog bites man and man bites dog are the identical vector. Bigrams patch the most common cases and multiply the vocabulary doing it.
• Negation and scope. not covered by warranty shares almost all of its weight with covered by warranty.
• Anything across languages. Two translations of one sentence have no terms in common.

Dense embeddings fix all four, because they place text in a space where meaning rather than spelling decides position. What they lose is the other side of the same coin: an exact rare term — a part number, an error code, a surname — may be diluted into a vector that is close to plausible neighbours and not close to the one document that actually contains it.

Choosing between the two representations

There is a second axis that decides it just as often, and it is not about quality at all. A sparse index is built by counting: one pass over the corpus, no model to download, no GPU, nothing to re-run when a vendor deprecates a model version. A dense index is built by running every document through a model, and when you change that model — a better one ships, or the one you used is retired — every vector in the store is invalid and the whole corpus must be re-embedded. That is a recurring operational cost with no equivalent on the sparse side, and planning the migration is a real project rather than a switch.

Not by accuracy, then, but by failure mode. If the query is a sentence expressing an intent, dense retrieval is the right default. If the query contains an identifier, a rare proper noun or a term of art that must match exactly, sparse weighting is, and no amount of embedding quality fixes the case where the correct document is the only one containing ERR_4127. Which is why production retrieval usually runs both and fuses the rankings — hybrid search exists because the two failure modes are close to disjoint.