Text and Unicode

Text is not bytes

A str holds characters. A bytes holds numbers between 0 and 255. Files and networks carry bytes, so text has to be encoded on the way out and decoded on the way back:

text = "Müller"
data = text.encode("utf-8")

print(type(text).__name__, type(data).__name__)
print(len(text), len(data))
print(data)
print(data.decode("utf-8"))

str bytes
6 7
b'M\xc3\xbcller'
Müller

Six characters, seven bytes. The ü takes two. That gap is the whole subject.

UTF-8 is the answer

It can represent every character, and plain ASCII text encodes to exactly the same bytes it always did:

print("abc".encode("utf-8"))
print("abc".encode("ascii"))
print("€".encode("utf-8"))

b'abc'
b'abc'
b'\xe2\x82\xac'

Name it explicitly whenever you open a file. The default depends on the platform, which is how the same code produces different results on two machines.

Decoding with the wrong encoding

The bytes are fine; the interpretation is not. This is where mojibake comes from:

data = "Müller".encode("utf-8")

print(data.decode("utf-8"))
print(data.decode("latin-1"))

try:
    data.decode("ascii")
except UnicodeDecodeError as problem:
    print("UnicodeDecodeError at byte", problem.start)

Müller
Müller
UnicodeDecodeError at byte 1

ü on a web page always means this: UTF-8 bytes read as Latin-1.

Code points

Each character has a number. ord() gives it, chr() goes back:

print(ord("A"), ord("ü"), ord("€"))
print(chr(65), chr(252))
print("ü", "\N{EURO SIGN}")

65 252 8364
A ü
ü €

Length is characters, not width

print(len("ü"))
print(len("👍"))
print(len("👨‍👩‍👦"))

1
1
5

That family emoji is five code points joined by invisible characters. Nothing is broken — "how many characters" is simply a harder question than it looks, and len() answers a precise version of it.

The same letter, written two ways

é can be one code point or two: e plus a combining accent. They look identical and are not equal:

import unicodedata

one = "é"        # e-acute as a single code point
two = "é"       # a plain e, then a combining accent

print([hex(ord(c)) for c in one])
print([hex(ord(c)) for c in two])
print(one == two)
print(len(one), len(two))
print(unicodedata.normalize("NFC", two) == one)

['0xe9']
['0x65', '0x301']
False
1 2
True

Both render as e-acute and neither is wrong. Printing the code points is the only way to see the difference. Normalise with unicodedata.normalize before comparing text that came from different sources. This is why a search box sometimes fails to find a name that is visibly right there.

Handling bad bytes

When you cannot fix the source, choose how to fail:

data = b"caf\xff"

print(data.decode("utf-8", errors="replace"))
print(data.decode("utf-8", errors="ignore"))

caf�
caf

replace keeps a marker so you can see something was lost. ignore throws it away silently, which is usually the wrong trade.