Remove duplicate CSV rows
Strip repeated rows from a CSV. Compares whole records, not physical lines, so a quoted field containing a newline is one record and stays one record. Runs entirely on your device.
Runs on your device. The file is never uploaded.
Remove duplicate CSV rows keeps the first copy of each record and drops every later one, writing the survivors as -deduped.csv. A record is compared as its whole list of cells, so a cell holding the separator cannot collide with another row. Matching on a single column is not offered, and caseSensitive is on by default.
Questions
Which duplicate does it keep?
The first one. Rows are walked in file order and each new row is remembered; a row identical to one already seen is skipped. So the earliest copy survives and the later ones are dropped, and the surviving rows stay in their original order.
How is a duplicate decided?
By the whole record, all cells compared together, not by a key column. The cells are compared as a list rather than as a joined string, so two different records cannot collide when a cell contains the separator. There is no option to dedupe on one column here.
Can I ignore differences in capitalisation?
Yes. Turn caseSensitive off, which is on by default, and every cell is lowercased before comparison. The row that is kept is still written out exactly as it appeared in the file, so only the matching is case insensitive, not the output.
Does it treat my header as data?
Not when hasHeader is on, which is the default. The first row is set aside, never compared, and written back at the top of the result. Turn hasHeader off and the first row is treated as an ordinary row and can itself be removed as a duplicate.
Will it delete real rows that contain newlines?
No. Records are parsed with quote state tracked across the whole file, so a quoted cell holding a newline stays one record. An earlier version that split on line breaks dropped genuine rows that happened to match a fragment of a multi-line record. That is exactly the bug the shared parser exists to prevent.
What do I get back?
One CSV named after your file with -deduped.csv on the end. Everything runs in your browser with no upload. The whole file plus a set of seen rows is held in memory, so memory use grows with the number of distinct rows, and a large file is limited by your device rather than by any rule here.