What Is Unicode and UTF-8? A Plain-English Guide
If you've ever seen garbled text like "é", wondered why emoji have codes like U+1F600, or received a "UnicodeDecodeError" in Python — this guide is for you. Understanding Unicode and UTF-8 is foundational for any developer working with text.
The Problem Unicode Solves
Before Unicode, every language had its own encoding standard. ASCII handled English (128 characters). Latin-1 covered Western European. Shift-JIS handled Japanese. These standards were incompatible — a file written in one encoding often looked like garbage in another. This garbled text is called mojibake (文字化け, Japanese for "character transformation").
Unicode's goal: a single, universal standard that assigns a unique number to every character in every writing system on Earth — plus emoji, mathematical symbols, and historical scripts.
What Is Unicode?
Unicode is a character set — a numbered list of over 149,000 characters. Each character gets a code point, written as U+XXXX. Examples:
| Character | Code Point | Description |
|---|---|---|
| A | U+0041 | Latin capital letter A |
| é | U+00E9 | Latin small letter e with acute |
| 日 | U+65E5 | CJK Unified Ideograph (Sun/Day in Japanese) |
| 😀 | U+1F600 | Grinning face emoji |
| → | U+2192 | Rightwards arrow |
| € | U+20AC | Euro sign |
What Is UTF-8?
Unicode defines the code points — but not how to store them as bytes. That's where encodings come in. UTF-8 is the most popular Unicode encoding. It uses 1–4 bytes per character:
ASCII characters (A–Z, 0–9, punctuation)
Latin Extended, Arabic, Hebrew, and more
CJK characters, most symbols
Emoji, historical scripts, rare symbols
UTF-8's key advantage: it's backward compatible with ASCII. Any ASCII file is automatically valid UTF-8. This is why UTF-8 became the dominant encoding on the web (over 98% of websites).
UTF-8 vs UTF-16 vs UTF-32
| Encoding | Bytes per char | Best for | Used by |
|---|---|---|---|
| UTF-8 | 1–4 (variable) | Web, files, APIs | HTML, JSON, Python, Linux |
| UTF-16 | 2–4 (variable) | Mixed multilingual text | Windows, Java, JavaScript strings |
| UTF-32 | 4 (fixed) | Fast random access | Some databases, C++ internals |
Common Encoding Pitfalls
Mojibake (é instead of é)
Usually caused by reading a UTF-8 file as Latin-1 (ISO-8859-1). Fix: ensure all file readers specify UTF-8 encoding explicitly.
Python UnicodeDecodeError
Occurs when reading bytes that aren't valid UTF-8 with default settings. Fix: use open(f, encoding="utf-8", errors="replace") or detect encoding with chardet.
MySQL "?" characters
Often from storing UTF-8 data in a latin1 column. Fix: use utf8mb4 charset for MySQL tables (utf8mb4 supports emoji; MySQL's "utf8" does not).
Email "=?UTF-8?" encoding
Email headers use "quoted-printable" or "base64" encoding for non-ASCII subjects. This is normal MIME encoding — email clients decode it automatically.
Convert Unicode code points instantly