Address Matching and Normalization Across Data Sources

Two records for the same building, from two systems, with nothing in common but the address string. Fuzzy string similarity on the whole string is the obvious approach and it is the one that fails, because the fields that must match exactly and the fields that may match loosely are different fields.

Parse before you compare

Here are the two records.

A: "Flat 2, 221B Baker Street, London NW1 6XE"
B: "221b baker st., apt 2, london, nw1 6xe, united kingdom"

Levenshtein distance between those strings is large, and Jaro-Winkler — which weights a common prefix — is misleadingly low because the strings start differently. Any whole-string comparator gets this wrong, and it gets it wrong in both directions: it will also score “221 Baker Street” and “221B Baker Street” as near-identical when they are different addresses.

So parse first. An address parser assigns a label to each token — house number, road, unit, city, postcode, country. libpostal, the widely used open-source parser, does this with a statistical sequence model trained on OpenStreetMap and OpenAddresses data rather than with regular expressions, which is what lets it handle the two orderings above without a rule for each. Commercial parsers work the same way.

A -> unit=2  house_number=221B  road=Baker Street
     city=London  postcode=NW1 6XE  country=(none)

B -> unit=2  house_number=221b  road=baker st.
     city=london  postcode=nw1 6xe  country=united kingdom

The comparison is now six small comparisons instead of one large one, and each can have its own rule. That is the entire trick.

Normalising the components

Normalisation is per-field and mostly boring, which is why it gets skipped and why it is where most of the recall comes from.

• Case and punctuation go first, along with whitespace collapsing and Unicode normalisation to NFC so that a precomposed é and a decomposed one compare equal.
• Street-type abbreviations expand or contract against a fixed list. In the United States that list is USPS Publication 28, which defines the standard suffix abbreviations and secondary-unit designators; other countries have their own postal standards. Pick one direction — always expand, or always contract — and apply it to both sides. Mixing directions is how st ends up meaning both “street” and “saint” in the same pipeline.
• Unit designators collapse to a canonical token: flat, apt, apartment, suite, ste, unit, no. all mean the same thing for matching purposes, while the unit value itself stays as written.
• Postcodes get their internal whitespace and case regularised to the national format, which for UK postcodes meansNW16XE and NW1 6XE become one token.
• House numbers keep their suffix as a separate field. 221B parses to number 221 and suffix B. This matters in the scoring step, because the number must match exactly and the suffix is sometimes legitimately absent from one source.

Blocking, and why it is not optional

Matching one file of 100,000 addresses against itself means comparing every pair:

pairs = n(n-1)/2 = 100,000 x 99,999 / 2 = 4,999,950,000

Five billion comparisons, and a comparison that involves a parse and six string similarities is not free. Two files of a million records each would be 10^12. This is why every record-linkage system blocks: only pairs that agree on some cheap key are compared at all.

Block on postcode. Assume the 100,000 records spread over 30,000 distinct postcodes, so a mean of 3.33 records per block:

pairs per block = 3.33 x 2.33 / 2 = 3.88
total           = 3.88 x 30,000 = 116,370

reduction = 4,999,950,000 / 116,370 = 42,966x

Four orders of magnitude, from one line of SQL. The assumption to notice is the mean: blocks are not uniform, and the cost is driven by the largest ones, since pairs grow with the square of block size. One postcode holding 2,000 records contributes 2 million pairs on its own — more than the entire estimate above. Check the block-size distribution, not the mean, and split or drop the pathological blocks.

The other thing blocking does is lose matches. Any pair whose postcodes differ — because one is missing, mistyped, or genuinely changed after a boundary revision — is never compared, and no amount of scoring quality recovers it. The standard answer is several blocking passes whose candidate sets are unioned: postcode; a phonetic key on the street name plus the house number; and the H3 cell of a geocoded point at resolution 9. Each pass has different blind spots, and a pair has to be missed by all of them to be lost.

Scoring a candidate pair

Field-by-field comparators with weights, summed. Weights below are a starting point to be fitted on labelled pairs, not constants.

field          comparator            A vs B            score  weight
house_number   exact                 221 vs 221         1.00   0.30
number_suffix  exact, null-tolerant  B vs b             1.00   0.10
road           Jaro-Winkler          "baker street"     1.00   0.25
                                     vs "baker street"
unit           exact, null-tolerant  2 vs 2             1.00   0.15
postcode       exact                 NW16XE             1.00   0.20

weighted total = 1.00  ->  match

Now a pair that shows why the comparators differ. Compare “221 Baker Street” against “221B Baker Street”: road scores 1.00, postcode 1.00, house number 1.00, and the suffix disagrees — one is B, the other is absent. Treating that as a null and scoring it 1.00 merges two different addresses. Treating it as a mismatch and scoring it 0 gives 0.30 + 0 + 0.25 + 0.15 + 0.20 = 0.90, which still clears most thresholds.

The honest handling is a three-way outcome per field — agree, disagree, missing — with different weights for each, which is what the Fellegi-Sunter model of record linkage formalises: each field contributes the log ratio of the probability of that agreement pattern among true matches to its probability among non-matches. Rare agreements count for more. Two records agreeing on the street name “High Street” is weak evidence in Britain and two agreeing on a rare street name is strong evidence, and a flat weight cannot express that.

The geocode as corroboration

String scoring alone cannot separate a genuine match from a near-miss on a long street. Geocode both sides and use the distance as an independent signal:

both rooftop, 4 m apart      -> corroborates, accept
both rooftop, 380 m apart    -> contradicts, reject or review
one rooftop, one ZIP centroid, 900 m apart
                             -> uninformative; the centroid is
                                kilometres-scale by construction

That third line is the point. The distance is only evidence if both coordinates are precise enough for it to mean something, so the match level has to be read alongside the distance — see what rooftop, interpolated and ZIP-level actually mean. Using a distance threshold without checking the tier is how a pipeline rejects correct matches in rural areas, where interpolated points are routinely hundreds of metres out, while accepting wrong ones in dense city blocks.

What stays hard

• Ranges and fractions. “100-104 Main St” covers three addresses; “12 1/2” is a real house number in parts of the United States; “12A” and “12 A” tokenise differently.
• Corner properties with two addresses. The same building is legitimately 1 High Street and 42 Church Lane, and no string method will ever connect them. Only the geocode does.
• Addressing systems that are not street-and-number. Japanese addresses are a nested area hierarchy ending in a block and building number, with no street name in the usual sense. Parsers trained mostly on Western data degrade badly here, and the fix is per-country handling rather than a better comparator.
• PO boxes and care-of lines, which are not locations at all and should be routed out of the pipeline rather than matched.
• Vanity addresses. A named building with an officially registered address that shares no tokens with it. These need an alias table; there is no algorithm.

The same machinery, with different fields, is what deduplicates places rather than addresses — see duplicate POI detection, where the name is fuzzy and the coordinate is the strongest signal you have.