How to Remove Duplicate Lines From a List

First decide: keep the order, or sort?

Some methods sort the list as a side effect of removing duplicates. That's fine for a keyword list, and wrong for a ranked list, a playlist or a log where position carries meaning. Every method below says which it does. When order is kept, the first occurrence of each line survives.

In the browser

Paste the list into Remove Duplicate Lines. It keeps the original order by default, and has switches for the cases that trip other tools: ignore case, trim spaces, drop empty lines. It also lists which lines were repeated and how many times — useful when the duplicates themselves are the finding. Sorting (A–Z, natural, by length, shuffle) is optional. The text stays on your device.

In a spreadsheet

• Excel: select the column, then Data → Remove Duplicates. Keeps order, edits in place, and is case-insensitive — "Apple" and "apple" count as the same. To keep the original, use =UNIQUE(A2:A500) in a new column instead (Excel 365 and 2021+).
• Google Sheets: Data → Data cleanup → Remove duplicates, or =UNIQUE(A2:A). The UNIQUE function is case-sensitive.

Note the difference: the same list can give different counts in Excel's button and in a UNIQUE formula elsewhere, purely because of case handling.

In a text editor

Notepad++ has Edit → Line Operations → Remove Duplicate Lines (keeps order) and Remove Consecutive Duplicate Lines (only touches repeats that sit next to each other). Sublime Text has Edit → Permute Lines → Unique. In editors without a built-in command, paste into a dedupe tool and back.

On the command line

# Sorted output, duplicates removed
sort -u list.txt > unique.txt

# Keep original order, first occurrence wins
awk '!seen[$0]++' list.txt > unique.txt

# Case-insensitive, order kept
awk '!seen[tolower($0)]++' list.txt > unique.txt

# Show only the lines that were duplicated, with counts
sort list.txt | uniq -c | sort -rn | awk '$1 > 1'

# PowerShell: sorted, case-insensitive
Get-Content list.txt | Sort-Object -Unique | Set-Content unique.txt

The classic trap: uniq on its own only removes adjacent duplicates. uniq list.txt on an unsorted file leaves most repeats in place. Sort first, or use the awk line.

In Python, list(dict.fromkeys(lines)) removes duplicates and keeps order, since dictionaries preserve insertion order.

Why some duplicates survive

If two lines look identical but are not removed, they are not identical. The usual suspects:

To see which lines differ and how, paste two suspects into a diff checker — it will highlight the stray character.

Email lists: one extra rule

The part of an address after the @ is case-insensitive, and in practice almost every provider treats the part before it the same way. Lower-case the whole list (or use "ignore case") before deduplicating, or the same person will receive two copies. Do not strip dots or "+tags" globally, though — that is Gmail-specific behaviour and would merge genuinely different addresses elsewhere.