Three names, one problem
The vocabulary is fragmented because the problem was solved independently in several fields, and the words survive from each.
- Deduplication — within one dataset. Output: a shorter dataset. This is what most people actually need.
- Record linkage — across two datasets with no shared key, a term from statistics and epidemiology. Output: a mapping between them.
- Entity resolution — the general case, including building a single canonical record from many sources. Output: an entity, and the records that belong to it.
If you are cleaning a mailing list, you want deduplication and the rest is vocabulary. The distinction matters when you must keep both source systems intact and link them, rather than collapse them.
Deterministic versus probabilistic
Deterministic matching applies rules: same email means same person; same surname plus date of birth plus postcode means same person. It is explainable, auditable, and fails predictably — which matters enormously in regulated work, where "the model said so" is not an acceptable answer.
Probabilistic matching scores agreement across several fields and links above a threshold. It catches what rules miss, at the cost of a dial you have to set, and errors in both directions: missed matches, and — worse — different people merged into one.
Those two failures are not equal. A missed duplicate is untidy; a wrong merge destroys data and can be genuinely harmful when the records are people. Set the threshold accordingly, and review before committing.
Blocking, and why it exists
Comparing every record with every other is quadratic: 10,000 records is roughly 50 million comparisons, and 100,000 is five billion. That is why naive fuzzy matching freezes.
Blocking is the fix — only compare records that share something cheap, such as the same postcode, the same first letter of surname, or the same email domain. You trade a small number of missed matches for an enormous reduction in work. Every practical system does this, and it is the main reason a fuzzy pass has row limits: on this site the API caps requests deliberately rather than letting a browser tab lock up.
What to try before anything clever
Most real datasets resolve without a model.
- Normalize hard. Lowercase, trim, strip punctuation, standardize phone numbers and country names. A large share of "fuzzy" duplicates are exact duplicates wearing different formatting.
- Match on the strongest field you have. Email, phone, tax ID, account number. One reliable identifier beats five weak ones.
- Only then go fuzzy, on what is left, with blocking, and look at the results.
Skipping step one and reaching for probabilistic matching is the most common mistake — it is slower, less accurate, and harder to explain than the trimming that would have solved it.
Deciding what survives
Resolution tells you which records belong together. It does not tell you what the surviving record should say, and that is a separate decision with real consequences: keep the newest, keep the most complete, or merge field by field taking the best value for each. Merging is riskier than removing, because a wrong merge cannot be undone from the result alone. Keep the source records until you are confident. See deduping CRM contacts for how this plays out on real contact data.
FAQ
Frequently asked questions
What is entity resolution?
Working out which records refer to the same real-world thing — the same person, company, or product — when there is no shared identifier. "J. Smith, 12 Oak St" and "Jane Smith, 12 Oak Street" are two records and one person. Deciding that is entity resolution.
Is entity resolution the same as deduplication?
Deduplication is entity resolution applied within one dataset. The broader term also covers linking records across separate systems, where the output is a mapping rather than a shortened list. Same underlying problem, different deliverable.
What is record linkage?
The same idea, from statistics and public health rather than software — joining records across datasets that share no key, such as a hospital register and a death register. Deterministic linkage matches on agreed fields; probabilistic linkage scores agreement across several and links above a threshold.
What is a good fuzzy matching threshold?
Start at 0.88 for names and tune from there. Below about 0.85 you begin merging different people; at 1.00 you are doing exact matching after normalization. The right number depends on how costly each kind of error is, which is a business question rather than a technical one.
Do I need a machine learning model for this?
Usually not. Normalizing thoroughly and then matching on a strong field — email, phone, tax ID — resolves most real datasets. Learned models earn their place at large scale with no reliable identifier, and they cost training data, evaluation, and explaining a decision nobody can read.
Last updated