Step 1: normalise, then compare
Almost every "the dedupe missed one" complaint is a normalisation problem. Before matching, strip these:
- Case. The domain half of an address is case-insensitive by specification. The local part technically is not, but Gmail, Outlook, and Yahoo all treat it as insensitive in practice, so lowercase the whole address.
- Whitespace. A trailing space from a copy and paste makes two identical addresses unequal to any exact matcher.
- Wrappers. Exports often carry
Jane Smith <jane@example.com>or amailto:prefix. Pull out the bare address first.
In a spreadsheet that is one helper column: =LOWER(TRIM(A2)), filled down, and you dedupe on the helper rather than the original.
Step 2: decide what counts as the same mailbox
Two variants are the same person even after normalising:
Plus-addressing. jane+newsletter@gmail.com and jane@gmail.com deliver to the same inbox at Gmail, and at most providers that support the convention. It is a convention though, not a rule every mail server follows, so stripping the tag across all domains can merge two people who genuinely have different addresses. If your list is mostly consumer webmail, stripping is usually right; if it is business domains, be conservative.
Gmail dots. j.a.n.e@gmail.com and jane@gmail.com are the same Gmail account. This is a Gmail behaviour specifically, so apply it only to gmail.com and googlemail.com addresses. Removing dots from other domains will merge different people.
Step 3: clean the list
For a pasted column of addresses, the text list tool trims, lowercases, drops blanks, and removes repeats in one pass, keeping the first occurrence and the original order. For a full export with names, tags, and signup dates, use the CSV tool and dedupe on the email column so the rest of each row rides along. Both run in your browser, which matters when the file is a customer list.
Why do this before the upload
Your sending platform will reject exact repeats at import, so this is not about duplicates it cannot see. It is about the ones it can: variants that pass the exact check and become two subscriber records for one human. That person then gets every campaign twice, which is the fastest way to earn an unsubscribe or a spam complaint, and complaint rate is the number mailbox providers watch when deciding whether you land in the inbox. There is a plain cost angle too, since most senders bill on contacts stored, so a duplicate record is a contact you pay for and gain nothing from.
One honest limit: deduping is not validation. It says nothing about whether an address still exists. If you are importing an old list, run it through a verification service as a separate step. And if the list came out of a CRM rather than a signup form, the CRM dedupe workflow covers matching on name as well as email.
FAQ
Frequently asked questions
Are email addresses case sensitive?
The domain half never is. The local part technically can be, but in practice the major providers including Gmail, Outlook, and Yahoo treat it as case-insensitive, so Bob@Example.com and bob@example.com reach the same person. Lowercase everything before you dedupe or you will keep both.
Should I treat user+tag@gmail.com as a duplicate of user@gmail.com?
For Gmail, yes: everything after the plus is ignored on delivery, and so are dots in the local part. On other domains plus-addressing is a convention rather than a guarantee, so stripping it can merge two genuinely different mailboxes. Decide per domain, not globally.
My email platform already removes duplicates on import. Why clean first?
It removes exact matches only. Case variants, plus-addressed forms, and the same person appearing in two source files with different column layouts all survive that check. Cleaning first also means the import report tells you about real problems instead of noise.
Does deduplicating improve deliverability?
It removes one specific risk: sending the same person the same message twice, which invites unsubscribes and spam complaints, and complaint rate is what mailbox providers weigh when deciding inbox placement. It does nothing about invalid or stale addresses, which is a separate validation job.
Last updated