Remove Duplicate Lines
The list apple, Apple, banana, apple, banana has 5 lines. With case ignored, 3 duplicate lines are removed and 2 remain: apple, banana.
How to use
- Paste your text, one item per line.
- Choose the options you need: ignore case, trim spaces, remove blank lines, and a sort order.
- Read the cleaned list and the number of duplicate lines removed, then copy it.
How it works
Each line is compared with the lines before it using a set, so the work grows in step with the number of lines. Tens of thousands of lines are handled instantly.
The first time a line appears it is kept. Later copies are removed, so the original order is preserved unless you choose a sort.
Trimming removes spaces and tabs at both ends of each line before comparing, so "apple" and "apple " count as the same line. With Ignore case on, "Apple" and "apple" also match and the first spelling is kept.
Windows (CRLF), old Mac (CR) and Unix (LF) line endings are all accepted. The result uses plain line feeds.
Sorting uses locale-aware comparison with English rules and keeps equal lines in their original order.
FAQ
Which copy of a duplicate is kept?
The first one. Every later line that matches it is removed.
Does it treat "Apple" and "apple" as duplicates?
Only when Ignore case is on. By default the comparison is case sensitive.
Why is the count lower than the lines I deleted?
The count only includes duplicates. Blank lines removed by the blank line option are shown separately.
Can it handle a very large list?
Yes. Tens of thousands of lines work fine. Very large pastes are limited only by your browser's memory.
Is my text uploaded?
No. The work happens in your browser and nothing is sent to a server.