Remove duplicates in Python with pandas

Look first with duplicated()

duplicated() returns a boolean Series. The argument that matters is keep: by default it marks second and later occurrences, which is what you delete, but keep=False marks every row in a duplicate group, which is what you review.

import pandas as pd

df = pd.read_csv("contacts.csv")

groups = df[df.duplicated(subset=["email"], keep=False)]
print(len(groups), "rows across", groups["email"].nunique(), "groups")
print(groups.sort_values("email").head(20))

drop_duplicates: subset and keep

df.drop_duplicates()                                  # identical rows
df.drop_duplicates(subset=["email"])                  # first per email
df.drop_duplicates(subset=["email"], keep="last")     # last per email
df.drop_duplicates(subset=["first", "last", "zip"])   # composite key
df.drop_duplicates(subset=["email"], keep=False)      # drop the whole group

Original row order is preserved and the original index comes along with it, so add ignore_index=True if you want a clean 0..n index in the result. If you want the newest row per key rather than the first in file order, sort before dropping: df.sort_values("updated_at").drop_duplicates("email", keep="last").

Two habits worth building. First, report what you removed rather than trusting the call: len(df) - len(clean) is one line and it is the number you will be asked for. Second, before deciding on a key, run df["email"].value_counts().head() to see the worst offenders; a key whose top value repeats hundreds of times usually means a placeholder such as an empty string or noreply@ rather than a genuine duplicate, and collapsing those to one row silently loses real records.

Also watch the difference between duplicate rows and a duplicate index. After a concat or a merge you can end up with repeated index labels while every row is distinct; df.index.duplicated() is the check for that, and reset_index(drop=True) the fix. drop_duplicates will not help, because it never looks at the index.

Normalise before you compare

pandas compares values exactly, so "Ann " and "ann" are two different people to it. Clean the key first:

key = (df["email"]
       .astype("string")
       .str.strip()
       .str.lower()
       .str.replace(r"\s+", " ", regex=True))

clean = df.loc[~key.duplicated(keep="first")]

Building the key separately and filtering with ~key.duplicated() keeps the original values in the output, which matters when you are handing the file back to someone who expects their own formatting. Watch out for missing values too: NaN compares equal to NaN in drop_duplicates, so several blank keys collapse to one row, which is usually not what you want.

Fuzzy matching in Python

For names and addresses, exact equality is not enough. rapidfuzz is the usual choice:

from rapidfuzz import fuzz, process

names = df["full_name"].tolist()
process.extract("Jane Smith", names, scorer=fuzz.token_sort_ratio, limit=5)

token_sort_ratio sorts the words before comparing, so "Smith, Jane" scores against "Jane Smith" as a near-perfect match. Scores run 0 to 100. The thing to plan for is cost: comparing every pair is quadratic, so on anything past a few thousand rows you need blocking, meaning you only compare records that share a cheap signal such as the first letter of the surname or the postcode. The recordlinkage library does that for you.

You probably do not need our tool

Stated plainly: if you already have pandas open, the five lines above are the answer and no web tool improves on them. Where this site helps is the case where Python is not the right amount of effort, or the person doing the cleaning does not write Python. The CSV tool and the Excel tool do column-selected exact dedupe in the browser with nothing uploaded, and fuzzy matching in the API covers the name-variant case. If your data lives in a database rather than a DataFrame, the SQL guide has the equivalent queries.

Last updated July 27, 2026