Remove duplicate lines
Remove duplicate lines. Runs entirely on your device, nothing is uploaded.
Runs on your device. The file is never uploaded.
Remove duplicate lines keeps the first copy of each line and drops every later repeat, leaving the survivors in the order they arrived. caseSensitive and trim both default to true, and the line written out is yours, whitespace and all. The unit is always a whole line, so a word repeated within one survives.
Questions
Which copy of a duplicate line is kept?
The first one. The tool walks your file top to bottom, remembers each line it has already seen, and skips any later line that matches. Original order is preserved for everything that survives, so this is not a sort followed by a uniq pass. The line that gets written out is your original line, complete with any whitespace, even when the comparison ignored that whitespace.
How does it decide two lines are the same?
By comparing whole lines, after two optional adjustments. trim, on by default, ignores leading and trailing whitespace during the comparison. caseSensitive, on by default, means Apple and apple count as different lines; turn it off and the comparison is done in lower case. Nothing else is normalised, so a line with a trailing tab and one without are equal with trim on, and different with it off.
Can it remove duplicate words or repeated phrases inside a line?
No. The unit is always a whole line, split on CRLF or LF. A line that repeats a word twice is one line and is kept as it is. For repeated values inside a row of CSV, use the CSV dedupe tool, which compares columns. For repeated substrings anywhere in the text, Find and replace (regex) is the tool with the right shape.
Why did my Windows file come back with different line endings?
Because the tool splits on CRLF or LF and joins the surviving lines back together with LF. A CRLF file therefore comes out with Unix line endings, even if no line was removed. The content of each line is unchanged; only the separator between them is. Convert the result back if the destination needs CRLF.
Does it need to load my whole file into memory?
Yes. Every unique line is held in a set while the file is scanned, so memory grows with the number of distinct lines, not the file size on disk. A list of a few million short lines is fine on a laptop and will be slower on a phone. Nothing is streamed to a server, because nothing is sent anywhere at all.