Two representations of Caffè
Literal Italian “Caffè” can end in precomposed è or e plus combining grave. Both have five graphemes locally, but five and six code points respectively. Exact comparison may therefore differ.
This can explain search and deduplication failures without mistyping. Sources can preserve different representations. Inspect codes before manual correction. Wrong decoding producing wrong letters instead needs the original file’s encoding checked; normalization cannot replace decoding.
Meaning of NFC and NFD
Unicode distinguishes canonical composition and decomposition. NFC composes where possible; NFD canonically decomposes. The normalization annex gives full rules.
A shared form supports consistent comparison of equivalent strings, not arbitrary similar words. E and È remain different; literal casa and Casa retain case differences. Keep normalization, case conversion and whitespace removal separate rather than hiding them in one cleanup command.
When normalization is useful
Lists from multiple sources may need alternate accent compositions treated alike. Preserve a reader-facing form and a separate equality key.
Normalization before byte measurement must be agreed. Verifying original files requires preserving what is checked. Passwords, signatures, identifiers and exact-byte data follow their system protocol, not generic editorial advice. Catalog practices do not automatically suit technical values.
Retain meaningful accents
Decomposing then removing combining marks can create nonequivalent words. A deliberately permissive search key may need that, but delivered text is changed. Italian accents can distinguish words and grammar.
Keep correct displayed accents. Declare ignored differences as comparison choices without replacing originals with simplified keys. Ignoring apostrophes or hyphens in search likewise does not authorize deleting them in publication. See case and accents.
Compatibility forms need another decision
Compatibility normalization can change special symbols into ordinary sequences, as String.normalize() documents. Easier matching alone does not justify applying it automatically.
Archives may intentionally distinguish symbols, numbering and special forms. Choose compatibility forms only for a defined reason, document it and keep originals. Deduplication can be more permissive than display or print when these uses remain separate.
Reproduce the experiment
Compare è with e plus combining grave before and after NFC. Locally they initially differ and then match. Count code points and bytes to observe changes alongside stable perceived text.
testo.normalize('NFC') returns a new JavaScript string without manual deletion. Share invented examples. If real strings still differ, inspect nonbreaking spaces, invisibles and apostrophes that accent normalization does not equate.
Our counting method
The counter normalizes to NFC before analysis, making engine measures consistent without promising visibly rewritten editor content. Typed and analyzed representations can serve different purposes.
Both Caffè forms yield five primary characters while original-byte figures differ. Read methods before comparing. Measure the actual technical delivery in its required encoding too; see characters and bytes.
Record rules before deduplication
Retain initial and final row counts and inspect deletions. Equality after transformation is a workflow decision rather than a universal archive property.
Editorial lists might merge canonical forms while keeping the first spelling; technical inventories may retain all sequences. See duplicate lines for comparison keys and retained rows, allowing rule corrections without rebuilding deleted data.
Include a negative test
NFC leaves e and è distinct. This negative test complements two equivalent forms of è, showing that accents are not indiscriminately discarded.
Include both a pair to unify and a pair to preserve. If the latter merges, the rule is too broad. Understand the transformation rather than manually patching exceptions. Retest expected cases after tool or source changes.
Two representations of one word
| Representation | Graphemes | Code points | UTF-8 bytes |
|---|---|---|---|
| Caffè with precomposed è | 5 | 5 | 6 |
| Caffe + combining grave | 5 | 6 | 7 |
| Both normalized to NFC | 5 | 5 | 6 |
Sources and checks
- Unicode UAX #15: Normalization Forms · Canonical equivalence and NFC/NFD forms
- MDN: String.normalize() · Normalization API and canonical versus compatibility forms