Recognize clues, not diagnoses
Apparently identical entries may not deduplicate; visible words may not search; lengths may surprise. Accent forms, apostrophes and different comparison rules can cause similar symptoms.
Isolate a short line before changing archives. Use an invented equivalent for confidential content. Specific codes reveal which transformation repairs the issue; broad deletion lists obscure both causes and unwanted changes.
Use code points
General Punctuation includes U+200B ZERO WIDTH SPACE, U+200C ZERO WIDTH NON-JOINER, U+200D ZERO WIDTH JOINER and U+2060 WORD JOINER. They have different contextual functions.
Record position and code. A marker like [U+200B] displays a location but is not the original character; counting it adds notation. Keep working copies separate from diagnostic representations.
Appearance can conceal different sequences
Locally AB has two code points, while A + U+200B + B has three code points and three graphemes with almost identical appearance. Exact comparison differs.
Do not conclude every invisible adds one grapheme. A + U+200D + B yielded two graphemes and three code points because grouping differs. Counts do not classify characters: inspect codes for diagnosis and declared methods for measurement.
Some invisible elements are essential
Emoji joiners combine figures; deletion can split them. Other scripts use elements shaping letter joins. Cleanup intended for Italian lists should not run blindly over multilingual writing.
See emoji counts. Inspect cleanup previews or choose a tool with targeted categories. Easier matching may not suit reader-facing output. Preserve originals rather than judging reduced length alone.
Nonbreaking spaces differ
Nonbreaking spaces have width and prevent pair separation. Replacing may help matching; deleting can fuse words.
Distinguish substitution and removal. See nonbreaking spaces for U+0020, U+00A0 and U+202F. Their suitability is a workflow choice, not a universal correctness rule.
Clean in controlled steps
Duplicate, identify one category, test a few lines and record changed occurrences. Consider separate matching keys for search-only issues; test downstream imports for exports.
Apply one change at a time. Simultaneous invisible removal, space compression and case conversion obscure causes. Inspect collisions before deleting rows. See deduplication rules for explicit equality.
Respect exact-identity data
Passwords, signatures, codes and technical values can require exact sequences. Follow their system format or data owner rather than editorial cleanup. Arbitrary deletion does not verify other content.
Direction controls in mixed scripts need contextual inspection. This guide is not a security validator or exhaustive sequence detector. It supports identifying and understanding changes proportionate to the task with a retained trace.
Verify visually and textually
Check affected appearance and codes, repeat failed search or matching, then inspect final length in the counter. Record whether changes remove, replace or regroup elements.
If unresolved, inspect normalization and source encoding. Do not delete progressively broader categories merely to force equality. Minimal examples plus originals and logs allow rule repair without reconstructing damaged text.
Describe locations clearly
Determine whether inspector columns count UTF-16 units or perceived symbols, particularly after composed emoji. Add neighboring text.
“U+200B between initial A and final B” avoids ambiguous indices. Preserve newlines in multiline tests and record removed blanks; changed structure shifts original positions. Keep original sample, visible-code view and transformed result together.
Identify before changing
| Element | Question | Possible choice |
|---|---|---|
| U+200B in a list | Is separation needed? | Test retention or removal |
| U+200D in an emoji | Part of the visual sequence? | Retain complete sequence |
| U+00A0 between words | Must the pair stay together? | Retain or replace with space |
| Direction control | Affects reading order? | Inspect context; avoid global deletion |
Sources and checks
- Unicode: General Punctuation · Identification of zero-width spaces, joiners and narrow no-break space
- Unicode UTS #51: Unicode Emoji · Emoji sequences, modifiers and zero-width joiners
- MDN: Intl.Segmenter · Locale-aware segmentation including graphemes