Duplicate Detection Threshold Finder
Paste your own items and sweep the similarity cutoff until the pairs it merges stop looking like duplicates.
6 pairs are above the threshold, collapsing 16 items into 11 groups — 5 removed as duplicates.
| threshold | pairs above | groups left | removed |
|---|---|---|---|
| 0.95 | 0 | 16 | 0 |
| 0.90 | 1 | 15 | 1 |
| 0.85 | 1 | 15 | 1 |
| 0.80 | 3 | 13 | 3 |
| 0.75 | 5 | 12 | 4 |
| 0.70 | 6 | 11 | 5 |
| 0.65 | 8 | 9 | 7 |
| 0.60 | 10 | 8 | 8 |
| 0.55 | 12 | 7 | 9 |
| 0.50 | 12 | 7 | 9 |
| 0.45 | 14 | 6 | 10 |
| 0.40 | 18 | 5 | 11 |
Look for the place the 'removed' column jumps. A cliff means a natural break in your data; a smooth ramp means there isn't one, and you are choosing a trade-off rather than finding a truth.
| pair | similarity | verdict |
|---|---|---|
| line 3: How do I reset my password line 4: how do i reset the password? | 0.778 | merged |
| line 10: Cancel my subscription line 11: How do I cancel my subscription? | 0.778 | merged |
| line 15: Is there an API rate limit? line 16: What is the API rate limit | 0.717 | merged |
| line 8: Where can I download my invoice line 9: How do I download an invoice | 0.655 | kept apart |
| line 10: Cancel my subscription line 12: I want to cancel the subscription | 0.655 | kept apart |
| line 13: Do you support single sign-on? line 14: Do you support SSO? | 0.625 | kept apart |
The three pairs just above your threshold and the three just below. This is the boundary you are actually setting — if a 'merged' row is not a duplicate, or a 'kept apart' row is, move the slider.
Everything on this page runs in your browser. Nothing you paste is uploaded, logged, or put in the URL — only the settings above the input are, so a configured tool can be linked to.
There is no default threshold, and anyone who gives you one is guessing
The right cutoff depends on how long your items are, how formulaic the language is, and what a mistake costs in each direction. Short, templated strings — support subject lines, product titles — cluster tightly and need a high cutoff, often above 0.85, because unrelated items already share a lot of surface. Long free text needs a lower one. The sweep table above is the honest way to pick: find where the number removed jumps, then read the boundary pairs and decide with your eyes.
What to do before you dedupe at all
Exact matching is free and catches more than people expect. Normalise case, collapse runs of whitespace, strip punctuation and trailing markers, then hash — that removes the easy half of most corpora in linear time, and it is deterministic, so it never needs a threshold. Only what survives that should reach a similarity comparison.
Then be aware of the asymmetry in the cost of an error. In training data, a missed duplicate inflates a benchmark and a wrongly merged pair quietly deletes a distinct example — the second is worse and much harder to notice later. In a support inbox, merging two different customers' questions is a visible failure and missing a duplicate costs a few seconds. Set the threshold according to which of those you are doing, not according to a number from a blog post.