Deduplicating and Decontaminating a Dataset

Deduplication and decontamination are the same computation pointed at two different questions: is this row a copy of another row in the same set, and is this training row a copy of something in the set I am going to be judged on?

Two jobs that share one technique

Deduplication removes near-identical rows within a corpus. The costs of not doing it are concrete: duplicated examples get extra weight in training for no reason anyone chose, they inflate every count you report, they waste tokens, and they raise memorisation of the duplicated text.

Decontamination removes training rows that overlap your evaluation data. Skipping it produces a score that is partly a measurement of memorisation, and the direction of the error is always flattering — which is why it is so rarely noticed by the person who benefits.

Both reduce to the same underlying question — is document A substantially contained in document B — so one implementation serves both, with different reference sets on the other side of the comparison.

The reason they are worth treating as one job is that the failure to do either is silent. Nothing errors. No metric turns red. A corpus that is 30% duplicates trains without complaint and reports a row count that looks like progress; a training set overlapping the eval set produces a score that goes up, which is the direction nobody investigates. Both are found only by running the check, which means the check has to be part of the pipeline rather than something somebody remembers to do before an important run.

What the deduplication studies reported

Lee and colleagues, in Deduplicating Training Data Makes Language Models Better (arXiv 2107.06499), examined standard pretraining corpora and built two tools that are still the reference implementations of the idea: ExactSubstr, which uses a suffix array to find every substring of at least a set length repeated anywhere in the corpus, and NearDup, a MinHash-based near-duplicate detector. Two findings are worth carrying:

• Deduplication sharply reduces memorised output. Models trained on deduplicated data emitted memorised training text far less often — they report roughly an order-of-magnitude reduction — with no degradation in perplexity. Less data, same quality, less regurgitation.
• Standard corpora contain train–validation overlap. A non-trivial share of validation examples in widely used datasets had near-duplicates in the training split, which means published validation numbers on those datasets are partly memorisation.

Kandpal, Wallace and Raffel (Deduplicating Training Data Mitigates Privacy Risks in Language Models, 2022) added the privacy dimension: the likelihood of a model regenerating a training sequence rises steeply with the number of times that sequence appeared in training. Deduplication is therefore a privacy control as well as a quality one, which is a useful thing to be able to say in a review.

On the contamination side, the practice of n-gram-based filtering against benchmarks goes back to the large-model reports themselves — the GPT-3 paper documents removing training documents overlapping its evaluation sets by n-gram match, and also documents a bug in that procedure, which is the most honest thing in the literature and a warning about how easy this is to get subtly wrong. Benchmark contamination covers the consequences for published scores; this page is about your own data.

Near-duplicate detection with MinHash

Comparing every pair of documents is quadratic and therefore impossible past a few hundred thousand rows. MinHash with locality-sensitive hashing turns it into an approximate lookup: represent each document by a small signature such that the probability two signatures agree equals their Jaccard similarity, then band the signatures so that similar ones land in the same bucket.

import re
from datasketch import MinHash, MinHashLSH

def shingles(text, k=5):
    """Word 5-grams. Normalise first — casing and punctuation are noise here."""
    words = re.sub(r"[^a-z0-9 ]", " ", text.lower()).split()
    return {" ".join(words[i:i+k]) for i in range(max(len(words) - k + 1, 1))}

def signature(text, num_perm=128):
    m = MinHash(num_perm=num_perm)
    for s in shingles(text):
        m.update(s.encode())
    return m

def dedupe(docs, threshold=0.8, num_perm=128):
    """Keep the first document of each near-duplicate group.
    'First' is a real decision: sort docs by whatever you value before
    calling this — length, source quality, recency — because the survivor
    of each cluster is whichever one arrived first."""
    lsh = MinHashLSH(threshold=threshold, num_perm=num_perm)
    keep, dropped = [], []
    for i, d in enumerate(docs):
        sig = signature(d["text"], num_perm)
        if lsh.query(sig):
            dropped.append((i, lsh.query(sig)[0]))    # keep the pairing: audit it
            continue
        lsh.insert(str(i), sig)
        keep.append(d)
    return keep, dropped

Two details that decide whether this works. The shingle size trades recall against precision — 5-grams over words is a reasonable default for prose, shorter for short documents, and character n-grams if you are dealing with code or heavy formatting. And the survivor of each cluster is whichever document was seen first, so sort the input by whatever quality signal you have before calling it, or you will keep the worst member of every group at random.

Keep the dropped pairs. A dedup pass that removes 40% of a corpus is either a triumph or a bug, and the only way to tell is to read twenty of the pairs it matched.

Decontamination against your eval set

Different question, different tool. Here you are not asking whether two documents are similar overall — you are asking whether a specific eval example appears inside a training document, which n-gram containment answers directly and similarity does not.

import re
from collections import defaultdict

def ngrams(text, n=13):
    words = re.sub(r"[^a-z0-9 ]", " ", text.lower()).split()
    return {" ".join(words[i:i+n]) for i in range(max(len(words) - n + 1, 0))}

def build_eval_index(eval_examples, n=13):
    """One inverted index over the eval set; training rows are streamed past it."""
    index = defaultdict(set)
    for ex in eval_examples:
        for g in ngrams(ex["input"] + " " + ex.get("expected", ""), n):
            index[g].add(ex["id"])
    return index

def contaminated(train_row, index, n=13, min_hits=1):
    """Any shared 13-gram is strong evidence — 13 words rarely coincide."""
    hits = set()
    for g in ngrams(train_row["text"], n):
        hits |= index.get(g, set())
    return hits if len(hits) >= min_hits else set()

def decontaminate(train, eval_examples, n=13):
    index = build_eval_index(eval_examples, n)
    clean, removed = [], []
    for row in train:
        hits = contaminated(row, index, n)
        (removed if hits else clean).append((row, hits) if hits else row)
    return clean, removed

The 13-gram convention comes from the large-model reports and it is a reasonable default because a thirteen-word sequence almost never coincides by chance in natural prose. Shorten it for short eval inputs — a 13-gram check cannot detect contamination of a six-word question — and be aware that exact n-gram matching misses paraphrased contamination entirely. For that you need embedding similarity, with all the threshold-tuning that implies.

Run this every time either side changes. Contamination is not a one-off cleanup: it is reintroduced every time somebody adds training data, and it is reintroduced silently.

Where to draw the threshold

There is no universal Jaccard threshold, and the reason is that the right answer depends on what a duplicate costs you.

Whatever threshold you pick, record it, record the shingle size and record how many rows it removed in the datasheet. Those three numbers are the difference between a dataset somebody can reason about a year later and one they have to redo.