Duplicate Line Remover
Remove duplicate lines from a list, with case and whitespace options.
What is Dedupe Lines?
The Duplicate Line Remover cleans pasted lists (emails, tags, URLs, log lines) by removing duplicate lines in a single pass, with configurable comparison (case sensitivity, whitespace trimming) and an explicit removed-lines report for trust.
How it works
A Set-based single pass keeps the first occurrence of each line. When case-insensitive or trim modes are on, lines are compared by their normalized form. The removed lines are collected and shown alongside the cleaned result.
- Paste your list. One item per line — emails, names, URLs, log output, anything list-shaped.
- Configure comparison. Toggle case sensitivity, whitespace trimming, blank-line filtering and sorting to match how strict your dedupe needs to be.
- Read the report. The cleaned result and the removed lines (with count) are shown side by side, so you can verify before trusting the output.
Examples
Clean a mailing list
A pasted list with repeated email addresses is deduplicated keeping the first occurrence of each.
Case-insensitive dedupe
With case-insensitivity on, 'alice@x.com' and 'Alice@x.com' count as duplicates; with it off they are both kept.
Common mistakes
Whitespace makes lines look different
'foo' and 'foo ' are distinct lines. Enable 'Trim whitespace' to treat them as duplicates.
Case sensitivity default
Comparison is case-sensitive by default — flip the toggle if your data varies in capitalization.
Alternatives
CSV Deduplicator
For deduplicating rows in structured CSV data by key columns.
Text Sorter
Often used after dedupe — sort the cleaned list alphabetically.
The Ultimate Text Guide
Line endings, case conventions, and other text fundamentals behind clean lists.
Frequently asked questions
Why does removing duplicates change my file's line order?
It doesn't by default — first occurrences keep their original position. Only enabling 'Sort result' reorders the output.
Does it handle case-insensitive duplicates?
Yes. Toggle 'Case-sensitive' off to compare lines ignoring capitalization — the first occurrence is still the one kept.