Free Duplicate Line Remover Online

Remove duplicate lines as you type: ignore case and extra spaces, keep first or last, drop blank lines, sort, or list only repeated lines with counts.

No login. Files processed for your request and discarded. Free to use. Files processed & discarded →

Compress PDF — it's free or choose from 130+ tools

130+ Free Tools
— Files Processed
— Happy Visitors
— Pages Explored
0 Files Stored

Remove Duplicate Lines Online

Part of Text tools: See all Text tools.

What is Duplicate Line Remover?

A duplicate line remover that works as you type. Paste a list and each repeated line is removed while the first copy stays in place. You can ignore upper and lower case, spaces at the ends of lines or repeated spaces, keep the last copy instead of the first, remove or keep blank lines, sort the result, list only the lines that appear once or only the repeated ones, and count how often each line appears. It runs in your browser.

How to use Duplicate Line Remover

  1. Paste your text in the top box, one item per line. The result updates immediately.
  2. Choose what to show: every line once, only lines that appear once, only repeated lines, or each line with its count.
  3. Tick Ignore upper/lower case or Treat repeated spaces as one if near-identical lines should match. Ignoring spaces at the start and end is on by default.
  4. Pick how to handle blank lines and the order of the result: original, A to Z, Z to A, natural or most repeated first.
  5. Copy the result or download it as a .txt file. The repeated lines report shows each duplicate, how many times it appeared and where it first appeared.

Why use this tool?

Lists copied from spreadsheets, emails and exports are full of near-duplicates that differ only in capitals or stray spaces. This tool lets you choose exactly what counts as the same line, always gives back your original text, shows which lines were repeated and how often, and never uploads your list.

FAQ

How do I remove duplicate lines from a list?
Paste the list into the box. Duplicate lines are removed as you type and the first copy of each line stays in its original position. Copy the result or download it as a text file.
Can it ignore upper and lower case?
Yes. Tick Ignore upper/lower case and lines such as Apple, APPLE and apple are treated as one. The result keeps the first version exactly as you typed it.
Why are two lines that look the same not removed?
Usually one has an extra space or tab. Ignore spaces at start and end is on by default; also tick Treat repeated spaces as one for gaps inside the line. Lines with different spelling are never merged.
Can I see which lines were duplicated?
Yes. The repeated lines report lists every line that appeared more than once, the number of times and the line number where it first appeared. Choose Only lines that were repeated to get them as a list.
How do I find lines that are in one list but not the other?
Paste both lists into the box, one after the other, and set Show to Only lines that appear once. Lines that are in both lists are left out, so what remains is in only one of them. This assumes neither list has duplicates of its own.
Does it keep the original order?
Yes, unless you choose a sort order. With Keep which copy set to Last copy, each line is placed where it last appeared instead of where it first appeared.
Is my text uploaded anywhere?
No. All processing runs in your browser. Nothing you paste is sent to our server or stored.

What this duplicate line remover does

Paste any list with one item per line and every repeated line disappears straight away. There is no button to press: the result box updates as you type or paste, and a short summary under it tells you how many lines went in, how many came out and how many duplicates were taken away. Lines stay in the order you gave them unless you ask for a different order, so the first time each item appears is the copy you keep.

Real lists are rarely tidy, so the tool lets you decide what counts as the same line. Ben@Example.com and ben@example.com can be treated as one address. A keyword copied with a trailing space can match the same keyword without it. Two spaces between words can count as one. Blank lines can be removed, kept in place, or treated like any other line. Underneath the result, a repeated lines report lists every line that appeared more than once, how many times it appeared, and the line number where it first showed up.

How to remove duplicate lines

  1. Paste or type your text in the top box. Each line is one item, whether it is an email address, a URL, a product code or a whole sentence.
  2. Leave Show on Every line once to remove duplicates. The cleaned list appears in the Result box immediately.
  3. Tick Ignore upper/lower case if Apple and apple should count as the same item. Ignore spaces at start and end is on by default because stray spaces from copy and paste are the most common reason two lines that look identical do not match.
  4. Choose what happens to blank lines and whether you want the original order, A to Z, Z to A, natural order or the most repeated items first.
  5. Press Copy, or Download .txt to save the list as a file. Use result as input lets you run a second pass with different settings.

What counts as a duplicate

By default two lines are duplicates when they are exactly the same after spaces at the start and end are ignored. That is a strict rule on purpose: it never merges lines that really are different. Each option loosens it in one clear way. Ignore upper/lower case compares lines in lowercase, so CAT, Cat and cat are one item. Treat repeated spaces as one turns any run of spaces or tabs inside a line into a single space before comparing, which helps with text pasted from PDFs and spreadsheets. If you switch off Ignore spaces at start and end, a line with a trailing space is treated as a different line, which is useful when spacing matters, for example in code or fixed-width data.

Accented letters are compared in a normalised form, so an é typed as one character and an é built from e plus an accent mark are treated as the same letter. The tool does not do fuzzy matching: colour and color, or Jon and John, stay as separate lines because no simple rule can decide they are the same thing without sometimes getting it wrong.

Whatever the comparison rule, the line you get back is your original text. If three variants of an email address match, the result shows the first one exactly as you typed it, unless you choose Keep which copy: Last copy, or tick Trim spaces in the result to strip the spaces from the start and end of every output line.

Four ways to look at a list

  • Every line once is the usual dedupe. Each distinct line appears one time, in its original place.
  • Only lines that appear once leaves out every line that has a twin. This is how you find the items that are in one list but not the other: paste both lists together and anything shared between them disappears.
  • Only lines that were repeated shows one copy of each line that appeared two or more times, which is the quick way to answer which entries are duplicated in this export.
  • Each line with its count writes the number of times each line appears, a tab, and then the line. Choose Most repeated first to turn a log file or a survey answer column into a frequency list. Because the count and the line are separated by a tab, you can paste the result straight into Excel or Google Sheets and it fills two columns.

Blank lines and line breaks

Remove all blank lines is the default, because empty lines are usually leftovers from copy and paste. Treat like any line keeps a single blank line, which matches what most older duplicate removers do. Keep them where they are leaves every blank line in its original position, so paragraphs or groups you separated with empty lines keep their shape while only the repeated text lines are removed. A line holding only spaces or tabs counts as blank. A final line break at the very end of your text is not counted as an extra empty line.

Windows, Mac and Linux line endings are all understood, and the result always uses plain line breaks, which every editor, spreadsheet and code tool accepts.

Sorting the result

Original order keeps the list as you wrote it. A to Z and Z to A sort alphabetically using your browser language rules, so accented letters sit next to their plain versions instead of after Z. Natural order reads numbers inside the text as numbers, so file2 comes before file10 and version 1.9 before version 1.10 in the usual cases. Most repeated first is mainly useful with the count view. Sorting happens after duplicates are removed, so this is not a plain sorter: every option here gives you each line once (or only the lines you chose to show).

Common uses

  • Cleaning an email or contact list before importing it into a mailing tool, with case ignored so the same address in different capitals is not imported twice.
  • Merging keyword lists from several SEO tools into one list without repeats, then sorting it A to Z.
  • Finding which IDs, SKUs or order numbers appear more than once in an export by switching to Only lines that were repeated.
  • Counting how often each error message shows up in a log file with the count view and Most repeated first.
  • Comparing two lists: paste both, choose Only lines that appear once, and you see the items that are in only one of them.
  • Tidying a list of URLs before building a sitemap or a redirect map. To pull the links out of a page or document first, use the URL extractor.

Limits and what this tool does not do

The tool works line by line. It does not look for duplicate words inside a sentence, and it does not compare columns inside CSV rows: a whole row is one line, so two rows count as duplicates only when the entire row matches. To dedupe on one column of a spreadsheet, use the remove duplicates feature of your spreadsheet program, or convert the data with the CSV to JSON converter and work on the field you need.

Input is capped at five million characters, which is roughly a hundred thousand lines of normal text, so your browser stays responsive. In testing, three hundred thousand short lines were processed in well under a second on an ordinary laptop, but very old phones will be slower. The repeated lines report shows the two hundred most repeated lines; the counts in the summary always cover the whole list.

Privacy

Everything happens in your browser. The text you paste is not uploaded to our server, not stored and not logged, which matters when the list is customer emails or internal IDs. Downloading the result creates the file on your device. Closing the tab clears the text.

Related tools

Check how long the cleaned list is with the word counter, change its capitalisation with the case converter, compare the list before and after with the text diff tool, or search and replace patterns across it with find and replace.

Also try

Related tools that work well with this one: