Home/Text tools/Remove duplicate lines
Text toolRemove Duplicate Lines
Paste a list, pick which copy of each repeated line survives, and see exactly how many were removed. Useful for merged spreadsheets, mailing lists and log files.
Result
0 lines removed
4 unique lines kept out of 7.
What keep-first and keep-last actually do to order
Both settings keep the same set of unique lines — the difference is entirely about which position in the list survives, and that matters whenever line order carries information, such as "most recent entry wins".
| Line # | Input | Keep first | Keep last |
|---|---|---|---|
| 1 | apple | apple ✓ | — |
| 2 | banana | banana ✓ | — |
| 3 | apple | — | — |
| 4 | cherry | cherry ✓ | — |
| 5 | banana | — | banana ✓ |
| 6 | apple | — | apple ✓ |
| 7 | cherry | — | cherry ✓ |
Keep-first preserves the order the unique lines first appeared in: apple, banana, cherry. Keep-last preserves whichever position each line occupied last, output in that same last-seen order: banana, apple, cherry. If your list is a log where later entries have corrected data, keep-last is usually right; if it is a form where the first submission should count, keep-first is.
Why trimming changes the answer on real pasted lists
Copy a column out of a spreadsheet and you will often get invisible trailing spaces on some rows and not others — a cell that was typed carefully next to one where someone hit space before pressing enter. Without whitespace-insensitive comparison, "apple" and "apple " count as two different lines and neither gets removed.
The same applies to case. A mailing list built from three different sign-up forms will have Apple, apple and APPLE as three "different" entries unless case-insensitive matching is switched on. Turn both options on for messy pasted data; turn them off when case and spacing are meaningful, such as comparing exact variable names in code.
Questions people ask
Does the comparison affect what gets displayed, or only what counts as a match?
Only the match. The line that survives is shown exactly as it was typed, including its original case and spacing — the ignore-case and ignore-whitespace toggles only decide which lines are treated as the same, not how the surviving copy is formatted.
What happens to blank lines?
By default they are left alone, since a blank line is often meaningful spacing rather than a duplicate. Switch "treat blank lines as duplicates" on to collapse runs of blank lines down to whichever the keep setting picks, same as any other repeated line.
Is there a line limit?
No fixed one. Comparison is done with a lookup table rather than checking every line against every other line, so it stays fast into the tens of thousands of lines. Extremely large pastes (millions of lines) may make the browser tab briefly unresponsive.
Does this sort my list?
No. Order is untouched except for which occurrence of a duplicate survives. Use the sort lines tool afterwards if you also want alphabetical or numeric order.