Skip to content
Multigrid

Text Deduplicator

Find exact and near-duplicate records in pasted text, with the similarity measure and the threshold both in your hands.

Unique records
4

3 of 7 folded away — 42.9% of the input.

Records read
7
Exact duplicates after normalisation
2
Near duplicates above the threshold
1
Groups with more than one member
1
Chance LSH proposes a pair at similarity 0.80
100.0%
Estimated tokens before
≈ 103
Estimated tokens after
≈ 65

Closest pairs, with the similarity that put them together

  • 1.000 · lines 1 and 2
    Reset your password from the account page.
    Reset your password from the Account page.
  • 1.000 · lines 1 and 7
    Reset your password from the account page.
    Reset your password from the account page.
  • 1.000 · lines 2 and 7
    Reset your password from the Account page.
    Reset your password from the account page.
  • 0.822 · lines 1 and 3
    Reset your password from the account page.
    You can reset your password from the account page!
  • 0.822 · lines 2 and 3
    Reset your password from the Account page.
    You can reset your password from the account page!
  • 0.822 · lines 3 and 7
    You can reset your password from the account page!
    Reset your password from the account page.
Deduplicated, first occurrence kept
What this assumes: similarity is the Jaccard coefficient over character 5-grams: the shared k-grams divided by the total distinct k-grams. That is a lexical measure. Two sentences that mean the same thing in different words score near zero, and finding those needs embeddings, not this. Candidate pairs come from MinHash with 64 hashes banded 16×4, so the shortlist is approximate — the row above gives the probability a pair at your threshold is even proposed — but every similarity printed is the exact Jaccard, recomputed on the full shingle sets. Buckets with more than 200 members are skipped as degenerate. Input is capped at 2,000 records so the O(n²) worst case cannot lock up the tab. Token figures are estimates. Estimated, not tokenized. A real count needs the model's vocabulary; the authoritative number is usage.prompt_tokens on the response.

Why the threshold is a decision and not a setting

Every deduplicator has a knob, and the knob is where all the judgement lives. At 0.9 you remove copies — reposts, re-scrapes, the same support macro pasted twice. At 0.7 you start removing things that share a template: two invoices for different customers, two error messages with different IDs, two rows of a form letter. Those are near-duplicates by any lexical measure and they are also, quite often, exactly the records you needed to keep. Move the threshold and watch the closest-pairs list rather than the headline number; the pairs tell you what class of thing you are about to lose.

Shingle size trades the same way. Small k-grams make everything look similar because short strings recur everywhere; large ones only fire on long verbatim runs. Five characters is a common default for English prose and is wrong for code, where indentation and brackets dominate, and for CJK, where five characters is most of a sentence.

What this cannot see: paraphrase, translation, and the same fact stated from a different angle. If you are deduplicating a training set or a retrieval corpus, lexical dedupe is the cheap first pass that removes the bulk, and an embedding pass is the expensive second one that removes the rest. Running the cheap pass first is not laziness — it is how you avoid embedding a hundred thousand copies of the same paragraph.

The reasoning behind this

Related tools

Text Deduplicator · Multigrid