Make sure a line is one item
One city per line suits deduplication. Narrative, verse and irregular tables may not: repetitions can be refrains, necessary headers or parts of larger records.
Spreadsheet rows with tabs are compared as entire rows. Shared surnames do not make people with different addresses duplicates. To compare one column, extract it with original-row references. Full-record deletion needs a tool understanding relevant columns and identity keys, not guesses from a line cleaner.
Exact matching is conservative
Case-sensitive matching distinguishes Roma and roma; Roma also differs unless edges are ignored. Set stores unique values; the tool applies this principle to keys while preserving first appearance.
The declared “exact” scope still follows CRLF and NFC normalization, not original-file bytes. That suits many editorial lists but not necessarily identifiers or forensic checks. Preserve originals and use appropriate verification. See NFC/NFD.
Ignoring edges does not remove them
Trimming forms a comparison key, while retained text comes from normalized input. If Roma is first, its spaces can remain. Equality testing is separate from rewriting output.
Internal San Marco and San Marco can remain distinct. Keep equality, occurrence choice and rewriting separate. If cleanup is needed, perform and verify it explicitly; see extra spaces.
Case-insensitive matching
ROMA, Roma and roma may represent one shared category label. Keys are lowercased using the selected interface language; displayed text retains the first spelling rather than selecting preferred editorial style. The literal Italian examples keep their original spelling.
Case-sensitive passwords and codes may require these values distinct. Do not enable simply to minimize rows. Ignoring case does not ignore accents or group similar names, abbreviations and spelling variants; those require further review.
The six-line test
Original lines are Roma, roma, Roma , Roma, Milano and a final empty line. Default matching keeps five; ignoring edges keeps four; also ignoring case keeps three including empty.
Removing empty lines leaves Roma and Milano. This was checked on the current function, not inferred from words. Reproduce in the tool with its final newline; omitting it changes the sample.
Empty and whitespace-only lines
Empty-line deletion is separate from deduplication. The first empty row can survive without explicit removal. Spaces-only rows can have different keys unless edges are ignored; removal discards rows empty after whitespace trimming too.
Blank lines may divide meaningful groups. Process groups separately or preserve a category field. Global deduplication can remove an entry needed in another category because equal text can have different roles.
First occurrence and ordering
Results preserve priorities, itineraries and document order rather than sorting alphabetically. Apply sorting separately afterwards if required to keep deletion review clear.
An initial lowercase variant survives a later polished spelling. Correct deliberately and consider this before reordering input. Keep line numbers or references for important data; deleted-row totals alone do not document variants or equivalence decisions.
Reconcile the result
All example options give six input = four removed + two retained. Also test one pair for every intended equivalence and one that must remain distinct; arithmetic alone is insufficient.
Disable one option at a time on a small sample for surprises. Invisibles or internal spaces can prevent matches; broad rules can erase legitimate entries. Inspect actual pairs rather than adjusting totals manually.
Four criteria, same six original lines
| Options | Retained | Removed | Result |
|---|---|---|---|
| Exact | 5 | 1 | Roma; roma; ␠Roma␠; Milano; empty |
| Ignore edges | 4 | 2 | Roma; roma; Milano; empty |
| Ignore edges and case | 3 | 3 | Roma; Milano; empty |
| Also remove empty | 2 | 4 | Roma; Milano |
Sources and checks
- MDN: Set · Unique values and insertion order
- MDN: String.trim() · Whitespace removal at string edges