Check that a line is really an item
This operation suits lists, not general prose or structured records with embedded newlines. A CSV field can contain a line break inside quotes, so a CSV record is not always a physical line. If identity depends on a database column or a person’s details, prepare the correct record comparison outside this simple line tool.
Choose case sensitivity deliberately
With case matching enabled, “Rome” and “rome” remain different. Without it, the first is kept when their normalized comparison keys are equal. The preserved spelling stays as originally entered. Codes, paths and identifiers can be case-sensitive, so a shorter result is not evidence that the chosen rule was appropriate.
Worked example
Rome rome Rome Milan
| Case-sensitive: removed | 1 |
|---|---|
| Case-insensitive: removed | 2 |
| Order kept | First occurrence |
Ignoring edge spaces changes the comparison
The edge-space option trims only the comparison key. If the first retained line has leading or trailing spaces, those original spaces remain in the output. Internal spaces are not collapsed. This keeps deduplication separate from cleaning the first occurrence, and allows you to review those two changes independently.
Decide what to do with empty lines
A final line break creates a final logical empty line. With empty-line removal off, one empty occurrence may be retained; repeated equal empty lines can still be duplicates. With removal on, empty or whitespace-only lines are removed. A paragraph’s blank line can carry structure, which is why you should not process a manuscript as though it were a list.
Reconcile the result before replacing the source
Compare the original, removed and kept counts. Check a sample containing an exact duplicate, a case difference, an edge space and an empty final line. The original line count must equal removed plus kept. Use undo if the rule was too broad, and export the verified result only after checking that the first-occurrence order is still useful.