Find Fuzzy Duplicates in CSV Files

Regular duplicate detection only catches exact matches. But what about typos? "Jon Smith" and "John Smith" are clearly the same person, but they won't show up as duplicates.

Fuzzy duplicate detection finds rows that are similar but not identical.

What is Fuzzy Matching?

Fuzzy matching uses algorithms to calculate how similar two strings are. The most common is Levenshtein distance-the number of edits needed to transform one string into another.

String A	String B	Distance	Similarity
John	Jon	1	75%
Smith	Smyth	1	80%
Microsoft	Microsft	1	89%

Find Fuzzy Duplicates Online

-> Fuzzy Duplicate Finder

1. Upload your CSV
2. Select the column to compare
3. Set similarity threshold (80% recommended)
4. Download grouped results

Understanding the Output

The tool adds two columns:

• _DUPLICATE_GROUP - Number identifying which rows are similar
• _SIMILARITY - How closely each row matches the group

Example output:

Name,Email,_DUPLICATE_GROUP,_SIMILARITY
John Smith,john@email.com,1,1
Jon Smith,jon@email.com,1,0.89
Jonathan Smith,jonathan@email.com,1,0.72

Choosing the Right Threshold

Threshold	Catches	Risk
90%+	Minor typos only	Few false positives
80%	Common variations	Good balance
70%	Significant differences	More false positives
60%	Very loose matching	Many false positives

Recommended: Start with 80% and adjust based on results.

Common Use Cases

Contact Deduplication

• "Robert Johnson" vs "Rob Johnson"
• "Mary O'Brien" vs "Mary OBrien"

Product Matching

• "iPhone 15 Pro" vs "iPhone15 Pro"
• "Samsung Galaxy S24" vs "Samsung Galaxy S 24"

Address Cleanup

• "123 Main St" vs "123 Main Street"
• "New York, NY" vs "New York NY"

Company Name Normalization

• "Microsoft Corp" vs "Microsoft Corporation"
• "Apple Inc." vs "Apple"

Python Alternative

from rapidfuzz import fuzz, process
import pandas as pd

df = pd.read_csv("customers.csv", dtype=str, keep_default_na=False)
names = df["name"].str.strip().str.casefold().tolist()

for index, name in enumerate(names):
    for match, score, match_index in process.extract(
        name, names[index + 1:], scorer=fuzz.WRatio, score_cutoff=88
    ):
        print(index, index + 1 + match_index, score, name, match)

This example still compares many pairs. On a large file, create blocks first, such as the same email domain, postcode, phone suffix or first letter. Compare records only inside plausible blocks.

Similar does not mean duplicate

Two people can share a name. Two companies can have similar trading names. Never delete the lower-scoring row automatically unless a reliable identifier also matches. A safer output contains both row numbers, the compared fields and the score, followed by a human decision or a documented business rule.

Choose thresholds from labelled examples rather than intuition. Review known matches and known non-matches, then measure how the threshold behaves. Names, addresses and product descriptions need different normalisation and scoring rules.

Before comparison, normalize case and surrounding whitespace, but preserve the source values in separate columns. Removing punctuation, titles or legal suffixes can help matching, yet those details may also distinguish genuine records.

Keep an audit column for the final decision: confirmed duplicate, separate record, or unresolved. That small step prevents the same pair from being reviewed repeatedly and makes later merges defensible.

Merge records only after matching them

Finding a likely pair and deciding what survives are separate operations. Before merging, define a field-level rule: keep the newest phone number, retain the verified email, combine non-conflicting tags, and preserve the original system identifiers. Never assume the row with more populated fields is automatically correct.

Create a crosswalk from every retired identifier to the surviving record. That crosswalk is essential when orders, cases or other tables still reference the old IDs. Test the process on a labelled sample, then review borderline scores rather than lowering the threshold until everything appears matched. A conservative unresolved queue is safer than a confident false merge.

---

Catch the typos. HappyCSV finds fuzzy duplicates that exact matching misses.