Encoding and Decoding in Python 2

This summary applies only to Python 2.x.

Default encoding and the source declaration

Declare the source encoding near the beginning:

# coding: utf8

It must appear in the first two lines. Without it, either of these lines containing Chinese text raises a SyntaxError before execution because Python 2 defaults to ASCII:

a = '美丽'
b = u'美丽'

The declaration makes the file encoding UTF-8. Inspect the current default encoding with:

sys.getdefaultencoding()

Unicode and UTF-8

Python uses unicode as the intermediate type for encoding conversions: str → unicode → str. The intermediate value is a Unicode object.

Unicode is a character representation independent of storage. UTF-8 is a binary encoding of Unicode. GBK, GB2312, GB18030, and UTF-8 are different encodings; transcoding converts between them.

Conversion

Calling encode directly on a str implicitly decodes it with the default encoding first:

# coding: utf8
s = '美丽'
s.encode('gbk')

The source bytes are UTF-8, but the default ASCII decoder cannot decode them. One workaround is changing the default:

# coding: utf8
import sys

reload(sys)
sys.setdefaultencoding('utf-8')

s = '美丽'
s.encode('gbk')

A clearer approach is to specify the source encoding:

# coding: utf8
s = '美丽'
s.decode('utf8').encode('gbk')

For a unicode value with the u prefix, encode it directly:

# coding: utf8
s = u'美丽'
s.encode('gbk')

Terminal encoding

Terminal encoding is another source of garbled output and errors. Personal computers commonly use UTF-8, while operations servers may use ASCII. Windows terminals often use a different code page. If otherwise correct encoding logic still fails, check the terminal encoding.

Originally published on SegmentFault.