1.
The str/bytes
nightmare
before python2 EOL
Kir Chou
A9 (Amazon Product Search)
1

2.
Kir Chou
2
note35 kir.chou

3.
3

4.
Google Trend
4
REF
encoding > recursion

5.
Outline
1. Objective
2. Background
3. Python String 101 – What is string?
4. Python String 101 – Why is this design?
5. 5 + 1 Treatments
5

6.
🚨 Confusing Terms 🚨
• Str: str() of python
• Bytes: bytes() of python
• Text: unicode() in Python2 or str() in Python3
• String: Text ∪ Bytes
6
You just need to know that those
terms are different in this talk.

7.
Objective
7

8.
Dunning–Kruger effect curve
8
😃
🤨
🥺 🧐
😯
✋😌👌
🤯

9.
9
What is it?
Why is the design?
Treatment?
Dunning–Kruger effect curve

10.
Background
10

11.
Python2 End-Of-Life
https://pythonclock.org/
11

12.
Optimistic Numbers
12
By Jetbrains 2018 survey
Victor Stinner (PyCon 2018)
Python 3: ten years later

13.
13
Python2.7
Legacy
Dependencies

14.
14

15.
https://www.youtube.com/watch?v=BS-HyV3V7GI
15

16.
16
Migration – Pain Points
painful
easy

17.
Python String 101
text vs bytes
17
(unicode)

18.
18
Text: How to present info in memory?
ℙ ƴ ℌ ø ἤ
e2
84
99
c6
b4
e2
98
82
e2
84
8c
c3
b8
e1
bc
a4

19.
19
ℙ ƴ ℌ ø ἤ
e2
84
99
c6
b4
e2
98
82
e2
84
8c
c3
b8
e1
bc
a4
Bytes: How to store info into memory?

20.
Python3
>>> 'python'
'python'
>>> 'ℙƴ ℌøἤ'
'ℙƴ ℌøἤ'
20
Default: text

21.
Python3
>>> b'python'
b'python'
>>> 'python'.encode(encoding='ascii')
b'python'
>>> b'ℙƴ ℌøἤ'
SyntaxError: bytes can only contain ASCII literal
characters.
>>> 'ℙƴ ℌøἤ'.encode(encoding='utf-8')
b'xe2x84x99xc6xb4xe2x98x82xe2x84x8cxc3
xb8xe1xbcxa4'
21
b' ' : ascii code (0~127)
text.encode(encoding)

22.
Python2
>>> 'python'
'python'
>>> 'ℙƴ ℌøἤ'
'xe2x84x99xc6xb4xe2x98x82xe2x84x8cx
c3xb8xe1xbcxa4'
22
Default: auto-encoded bytes

23.
Python2
>>> u'ℙƴ ℌøἤ'
u'u2119u01b4u2602u210cxf8u1f24'
23
u' ': text

24.
Python String 101
encode vs decode
24
Text -> Bytes
encode(encoding)
Bytes -> Text
decode(encoding)

25.
Python3
>>> dir(str())
[…, 'encode', …]
>>> dir(bytes())
[…, 'decode', …]
25
Text -> Bytes
encode(encoding)
Bytes -> Text
decode(encoding)

26.
Python2
>>> dir(str())
[…, 'decode', 'encode', …]
>>> dir(bytes())
[…, 'decode', 'encode', …]
>>> dir(unicode())
[…, 'decode', 'encode', …]
26

27.
Python2
>>> dir(str())
[…, 'decode', 'encode', …]
>>> dir(bytes())
[…, 'decode', 'encode', …]
>>> dir(unicode())
[…, 'decode', 'encode', …]
27

28.
Python String 101
Python2 Design Philosophy
28

29.
29

30.
Python2 is well-designed for ascii code
BUT…
• Most of encoding only support one way: utf-8, latin-1
>>> 'ℙƴ ℌøἤ'.encode()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte
0xe2 in position 0: ordinal not in range(128)
30

31.
>>> dir(str())
[…, 'decode', 'encode', …]
>>> dir(bytes())
[…, 'decode', 'encode', …]
>>> dir(unicode())
[…, 'decode', 'encode', …]
31
But why?

32.
Exceptions
• base64, rot13
>>> 'python'.encode('base64')
'cHl0aG9un'
>>> 'cHl0aG9un'.decode('base64')
'python'
>>> import base64
>>> base64.b64encode('python'.encode('utf-8'))
b'cHl0aG9u'
32
2
3

33.
Python String 101
Fundamental Difference - Encoding
33

34.
You must be told, or you have to guess
By Ned Batchelder
34

35.
ASCII encoding behave similar,
BUT…
>>> b'python' == u'python'
False
35
>>> b'python' == u'python'
True
3
2

36.
Treatment 1
Knowing the Fact of String
36

37.
The Fact of String
By Ned Batchelder
• Python3: string = unicode
• Encoding needs to be handled manually
• One-way encode/decode behind str/bytes (Good)
• Python2: string = auto encoded bytes
• Ascii codes are perfectly handled
• Two-way encode/decode behind str/bytes (Broken after
python is broadly used in many different human languages.)
37

38.
Treatment 2
Explicit Declaration
(From community approach to supporting python3)
38

39.
Consistent IO
• Standard I/O (bytes)
• Python2: sys.stdin / sys.stdout
• Python3: sys.buffer.stdin / sys.buffer.stdout
• File IO
import io # consistent api in both versions
io.open('path/to/file', 'wt') # text, bytes
39

40.
Bytes or Text (with Encoding)
• u'' or b''?
• No more raw string ''
• Which encoding is used for the text?
• No more guess, always provide encoding: latin-1, utf-8…
40
def my_encrypt(text, encoding='utf-8'):
… 32

41.
41
>>> # -*- coding: utf-8 -*-
sys.getdefaultencoding()
Encoding of your Operating System
32

42.
Treatments 3
Unicode Sandwich
(From community approach to supporting python3)
42

43.
Unicode Sandwich
Decode ASAP, Encode ALAP
43
Don’t care about
encoding!

44.
Treatments 4 (minor)
Python2/3 compatibility
(From community approach to supporting python3)
44

45.
45
>>> from __future__ import unicode_literals
Add unicode_literals
after porting to python3
2

46.
Treatments 5
Typing
(Learn from dropbox migration notes)
46

47.
Typing + [mypy] [pyre]
def encrypt(data, key):
return cipher
47
def encrypt(data: bytes, key: bytes) -> bytes:
return cipher
def encrypt(data, key):
# type: (bytes, bytes) -> bytes
return cipher
3
3
2
2

48.
Treatments SP
C-API
48

49.
C-API
49
3
2
#if PY_MAJOR_VERSION >= 3
#define Py2str_FromStringAndSize(x, y) PyBytes_FromStringAndSize(x, y)
#else
#define Py2str_FromStringAndSize(x, y) PyString_FromStringAndSize(x, y)
#endif
Example @ pyconjp 2019 sprint
https://github.com/note35/SupportingPython3-notes

50.
Conclusion
50

51.
51
Q: Supporting python3?
A: ✋😌👌

52.
Take-Home Messages
Write text/bytes explicitly with typing
Always provide encoding for bytes
Apply Unicode Sandwich if possible
Copy encode/decode from StackOverflow
52
Python String is no longer a nightmare! 🎉
(Even you only write Python3)

53.
Verify Your Understanding!
https://stackoverflow.com/search?q=UnicodeEncodeError
53

54.
54

55.
Reference: Docs
• Python2 unicode
• Python3 unicode
• pyporting
• python-future.org/compatible_idioms.pdf
• 2018 Jetbrains Survey
• Dropbox Migration Notes
55

56.
Reference: Talks
• Ned Batchelder: Pragmatic Unicode, or, How do I
stop the pain?
• Guido van Rossum: BDFL Python 3 retrospective
• Brett Cannon - How to make your code Python
2/3 compatible - PyCon 2015
• Edward Schofield - Writing Python 2/3 compatible
code
• Victor Stinner - Python 3: ten years later - PyCon
2018
56