Text Similarity: Every Method and What It Actually Detects
5 min read · updated August 3, 2026
“How similar are these two texts?” is four different questions wearing one sentence. Character-level similarity, set-overlap, lexical weighting and semantic similarity are answered by different algorithms with cost differences of five orders of magnitude, and picking the wrong one is the usual reason a deduplication or matching system disappoints.
Say which similarity you mean
Four distinct notions, and it is worth naming yours before opening a library:
- Surface similarity. Are these nearly the same characters? Jonathan Smith against Jonathon Smtih. Typos, OCR errors, transliteration.
- Set similarity. Do these share most of their content? Two copies of an article with a changed header. This is near-duplicate detection.
- Topical similarity. Are these about the same thing? Two documents on interest rate policy that share vocabulary without being copies.
- Semantic equivalence. Do these mean the same thing? how do I cancel against ending your subscription, which share no content words at all.
No single metric covers all four. A method that is excellent at the first is blind to the last by construction, and vice versa.
The methods, by what they detect
| Method | Description |
|---|---|
| exact hash | SHA-256 of the normalised text. Detects: byte-identical documents only. Constant time per document, no comparison needed — you look the hash up. Always the first stage of any deduplication pipeline, because it removes the easy majority for free. |
| edit distance | Levenshtein and relatives. Detects: character-level differences, and tells you how many. The only method that answers 'how far apart are these two strings'. O(mn) in the naive dynamic program — see the fuzzy matching page for the faster variants. |
| Jaro-Winkler | A similarity in [0,1] weighted towards agreement in the first characters. Detects: name-shaped variation specifically. Designed for record linkage on personal names and consistently better there than raw edit distance; poor on long text. |
| Jaccard on shingles | Intersection over union of overlapping character or word n-grams. Detects: near-duplicates and partial copies, robust to reordering of whole blocks. The definition on which MinHash is built. |
| MinHash + LSH | A fixed-length signature whose collision probability equals Jaccard similarity (Broder, 1997). Detects: the same thing as Jaccard, at a scale where computing Jaccard directly is impossible. This is how web-scale deduplication is actually done. |
| SimHash | A locality-sensitive hash where similar documents get signatures within a small Hamming distance (Charikar, 2002). Detects: near-duplicates, with a 64-bit signature per document, which is why it is preferred when memory is the constraint. |
| TF-IDF cosine | Cosine between weighted sparse vectors. Detects: topical similarity through shared distinctive terms. Blind to synonyms and word order. Cheap, exact and explainable — you can print which terms produced the score. |
| BM25 | Asymmetric: scores a document against a query rather than two documents against each other. Detects: relevance with saturation and length correction. The right tool when one side is short and one is long. |
| embedding cosine | Cosine between dense vectors from a trained model. Detects: semantic equivalence, paraphrase, and cross-language similarity. Blind in the opposite direction: two texts differing only in a single identifier or a negation can be near-identical vectors. |
| cross-encoder | A model that reads both texts together and scores the pair. Detects: everything above, most accurately. Cannot be indexed — the score exists only for a pair you actually run — so it is a re-ranking stage over candidates, never a search. |
Complexity, and why all-pairs is out
The arithmetic that shapes every real system: comparing every pair in a collection of n documents is n(n−1)/2 comparisons. For n = 1,000 that is about 500,000 — trivial. For n = 100,000 it is 5 billion. For n = 1,000,000 it is about 500 billion, and at an optimistic 100 nanoseconds per comparison that is roughly fourteen hours of CPU for the cheapest possible metric. With edit distance at, say, 10 microseconds per pair it is over 150 years.
This is why every method in the table above splits into two families, and it is the most useful distinction on the page. Indexable methods — hashes, MinHash, SimHash, sparse vectors, dense vectors — let you find candidates in sub-linear time through an inverted index or an approximate nearest-neighbour structure, so you never enumerate pairs. Pairwise-only methods — edit distance, Jaro-Winkler, cross-encoders — produce a score for a pair you already have. The architecture is always: index to get candidates, then score the candidates precisely. Trying to use a pairwise metric as a search is the mistake that turns a two-hour job into an impossible one.
The cost column, derived
Assumptions, all replaceable: 1 million documents, 300 tokens each, 300 million tokens total. One 4-vCPU box at an assumed $0.05 per hour.
- MinHash deduplication. Shingling and hashing is linear in characters. A million documents is minutes to low hours on one box; call it under $1 of compute, plus the signature storage — 128 32-bit hashes per document is 512 bytes, so about 512 MB.
- TF-IDF index and cosine search. One pass to build, then queries served from an inverted index. Compute cost rounds to the same box. Memory is the sparse matrix, which for 300M tokens with a pruned vocabulary is comfortably within a normal server.
- Dense embeddings. 300 million tokens, at an assumed embedding price of $0.02 per million tokens, is $6 once — plus re-embedding whenever the corpus changes or the model does, which is the line people forget. Storage at 768 float32 dimensions is 3 KB per document, about 3 GB, and the ANN index on top of that. See what vector storage actually costs and why the dimension count is a budget decision.
- Cross-encoder re-ranking. Priced per pair, not per document, so the cost is set by your candidate count. Re-ranking the top 50 for a million queries is 50 million model calls, which is a different order of magnitude from everything above and is why it only ever runs on a short list.
The pattern worth extracting: the cheap methods are cheap by factors of hundreds, not percentages, and the expensive ones are only affordable because something cheap ran first.
Choose by the failure you cannot afford
Accuracy is the wrong axis because each method fails in a characteristic direction. Ask which mistake would be worst.
If merging two different customers is unacceptable, you need a method that sees characters — an embedding will happily place Invoice 4471 and Invoice 4417 at cosine 0.99. If missing a paraphrase is unacceptable, you need embeddings, because no amount of lexical weighting relates cancel to terminate. If the decision has to be explainable to a regulator or a customer, sparse weighting is the only option here whose score you can decompose into terms. And if the corpus is large enough that all-pairs is out — which by the arithmetic above is any corpus over about a hundred thousand documents — then indexability is not a preference, it is the constraint that eliminates half the table.