# UTF-8: The Encoding That Won by Being Clever

> UTF-8 was sketched on a placemat in 1992 and now carries ~98% of the web. It won because of five design properties — self-synchronization, ASCII compatibility, sort-order preservation — that no committee would have dared require.

URL: https://harshith.in/blog/utf8-by-design/
Author: Harshith Sunku (Staff Engineer) — https://harshith.in/about/
Tags: fundamentals, systems

"Strings are arrays of characters" stopped being true decades ago, but an enormous
amount of code still acts like it is — and then someone types an emoji into the name
field. Understanding UTF-8 takes about ten minutes, fixes a whole category of bugs,
and as a bonus is one of the most elegant pieces of bit-level design you'll ever see.
Ken Thompson and Rob Pike sketched it on a diner placemat in 1992.

## Code Points Are Not Bytes

Unicode assigns every character a number — a **code point** — from U+0000 to
U+10FFFF: `A` is U+0041, `é` is U+00E9, `中` is U+4E2D, `🚀` is U+1F680. That's the
abstract layer. An **encoding** decides how those numbers become bytes. UTF-8 is a
variable-length encoding: 1 to 4 bytes per code point, and the bit patterns are the
whole story:

```text
  U+0000..U+007F     0xxxxxxx                              (ASCII, 1 byte)
  U+0080..U+07FF     110xxxxx 10xxxxxx                     (2 bytes)
  U+0800..U+FFFF     1110xxxx 10xxxxxx 10xxxxxx            (3 bytes)
  U+10000..U+10FFFF  11110xxx 10xxxxxx 10xxxxxx 10xxxxxx   (4 bytes)
```

The code point's bits are packed into the `x` positions. `é` (U+00E9, binary
`11101001`) becomes `110_00011 10_101001` = `0xC3 0xA9` — which is why broken
pipelines show `é`: someone decoded those two bytes as Latin-1.

## The Design Properties That Won

Look at the leading bits and notice what falls out:

1. **ASCII is valid UTF-8, unchanged.** Every 7-bit ASCII file ever written was
   already UTF-8. Adoption required converting nothing.
2. **Self-synchronizing.** Continuation bytes always start `10`; lead bytes never
   do. Drop into the middle of a stream and you can find the next character
   boundary within 3 bytes. A corrupted byte damages one character, not the rest
   of the file.
3. **No false ASCII.** Bytes 0x00–0x7F appear *only* as themselves — never inside
   a multi-byte sequence. So `strchr(s, '/')`, null-terminated strings, and every
   ASCII-era parser work on UTF-8 without modification. (UTF-16 fails this: it's
   full of zero bytes.)
4. **Memcmp sorts correctly.** Byte-wise comparison of UTF-8 strings equals
   code-point order. Databases and filesystems can sort without decoding.
5. **No byte-order problem.** It's a byte stream; there's no 16-bit unit to swap,
   no BOM required (Windows tools add one anyway — `EF BB BF` — and Unix tools
   choke on it; that's a compatibility scar, not part of the design).

UTF-16, which Windows, Java, and JavaScript adopted early, has none of properties
2–5 and *still* turned out variable-length once Unicode outgrew 16 bits — surrogate
pairs are the permanent tax on that bet.

## Where Code Goes Wrong

Almost every Unicode bug is one confusion: treating **byte count**, **code point
count**, and **what a user sees** as the same number. They're three numbers:

```text
  "🇮🇳"  flag of India
    bytes (UTF-8):   8
    code points:     2   (regional indicators I + N)
    what users see:  1
```

`len()` in Python counts code points; `strlen` in C counts bytes; a text cursor
should move by **grapheme clusters** (what users see), which requires Unicode's
segmentation rules — use a library. Truncating a string at byte 100 can cut a
character in half and produce invalid UTF-8; truncate at a boundary (back up while
`(byte & 0xC0) == 0x80`). And validate at the edges: overlong encodings (encoding
`/` as two bytes) are illegal precisely because they were used to sneak path
traversal past security checks — a validator must reject them, and every competent
decoder since has.

There's one more layer: the same visible text can be different code points —
`é` as U+00E9 or as `e` + combining accent U+0301. Compare user-visible strings
without normalizing (NFC/NFD) and "identical" strings won't match. Filesystems
disagree here too: macOS normalizes filenames, Linux stores bytes.

## Takeaways

- Code points are numbers; UTF-8 is how they become bytes — 1 to 4 of them.
- The bit layout gives ASCII compatibility, self-synchronization, safe ASCII
  scanning, and memcmp ordering; that's why it carries ~98% of the web.
- Bytes, code points, and grapheme clusters are three different lengths; know
  which one each API returns.
- Truncate on boundaries, validate (reject overlong forms) at input edges,
  normalize before comparing.
- Declare/assume UTF-8 everywhere; encoding guessing is how `é` happens.