Unicode and UTF-8: why the letter is not necessarily a single byte

Useful to know

A text may appear simple on the screen and be more complex in the computer's memory. One sees letters while the file contains a sequence of bytes. In order to recover the correct text, one must know how this sequence is encoded. Unicode and UTF-8 are involved in this task but do not mean the same thing. Unicode provides a common pattern for characters and their code points, and UTF-8 defines how corresponding values are represented by bytes. [1]

UTF-8 uses variable length: one Unicode scalar value is encoded with between one and four bytes. The ASCII characters retain their single-byte performance. Many other characters, including the usual Bulgarian letters, require more. Hence a practical result: the number of bytes in a file is not equal to the number of letters in it. A limit of one hundred bytes and a limit of one hundred visible characters can allow different texts, although the number in the condition is the same. [1]

And the code point does not always correspond to a stand-alone visible sign. Unicode supports combining characters that are applied to a primary character. For example, an accented letter can be represented as a base letter and a separate combining accent. When displayed, these elements form one visual unit. This explains why a program that cuts text after a certain number of code points can separate something that the reader perceives as a single character. [2]

Some visually identical texts may have different sequences of code points. Normalization provides standardized forms for processing such cases. NFC uses canonical composition, and NFD - canonical decomposition. There are also forms with compatibility conversions that are not interchangeable with the first in each task. The choice of form must be consistent with the purpose: comparing, indexing or storing the exact typed text. Normalization is not a translation and is not a spell check. [3]

Let's imagine a catalog where editors enter foreign first names from different keyboards. Two equally-looking entries may not match when comparing their raw bytes. The well-defined processing of Unicode allows the search to account for canonically equivalent representations. This does not mean automatically removing all diacritics or merging different letters. Such changes are a separate product solution and can change the meaning of the name. [2, 3]

For the developer, the useful question is which unit measures the particular operation. Transfer size is measured in bytes; processing Unicode can work with code points; editing visible text often requires attention to component characters. Different measurements do not contradict each other, but describe different properties. When the system consistently uses the correct encoding and appropriate text operations, Cyrillic and international names cease to be an exception and are processed as a natural part of the content.