Python2 unicode-pt1

Python2’s unicode problem
A primer for the Python3ista

Intro
●
This is an entry level talk on a complex problem

Intro
●
●
It’s aimed at giving you a peek at the problem...

Intro
●
●
It’s aimed at giving you a peek at the problem
●
...so you’ll start to have a conceptual
understanding

Intro
●
●
●
understanding
– But solving the problem is another talk

Intro
●
●
●
understanding
– But solving the problem is another talk
– In fact, having a complete understanding of the
problem is another talk

New job, old tech
●
So you learned on Python3….

New job, old tech
●
●
But your new job requires maintaining
Python2...

New job, old tech
●
●
But your new job requires maintaining
Python2...
●
What’s this unicode problem everyone’s talking
about?

The unicode problem
🎵Oh you wanna know what the big deal is?

The unicode problem
I can tell you that

The unicode problem
I can tell you that
Oh, I can tell you that 🎶

But first, let's learn about PHP

Because every good explanation of a problem
begins with an example in PHP.

In PHP, you can define a variable to hold a
string

That looks like this:
$a = "1";

and then you can define a second variable to
hold an int
$a = "1";

Which looks like this:
$a = "1";
$b = 1;

and then you can compare those...
$a = "1";
$b = 1;
if ($a == $b) {
echo "n";
}

which outputs this:
$ php -r '$a = "1"; $b = 1; if ($a == $b)
{ echo "Equaln"; }'
Equal

Huh

In PHP, you can define a variable to hold a
string:
$a = "1";

and then you can define a second variable to
hold another string:
$a = "1";
$b = "1.0";

And then you can compare those:
$a = "1";
$b = "1.0";
if ($a == $b) {
echo "Equaln";
}

which does just what you expect...:
$ php -r '$a = "1"; $b = "1.0"; if ($a ==
$b) { echo "Also equaln"; }'
Also equal

Okay, let’s learn about Python3

In Python3, you can define a variable to hold a
string (Not idiomatic... bear with me):
a = str("1");

And you can define another variable to hold an
int:
a = str("1");
b = int(1);

and then you can compare them:
a = str("1");
b = int(1);
print("Equal") if a == b else print("Unequal")

which shows that Python has chosen a
distinctly different path than PHP:
python3 -c 'a = str("1"); b = int(1) ; print("Equal") if
a == b else print("Unequal")'
Unequal

Choices
Different languages make different choices

Choices: PHP
●
PHP imagines a world where all input is text

Choices: PHP
●
●
The language should turn text into types the
programmer expects

Choices: PHP
●
●
programmer expects
●
PHP coerces strings to types before comparing
them

Choices: PHP
if (float(“1”) == float(“1.0”)) {
echo(“Equal”);
}
Just imagine that PHP is doing this:

Choices: PHP
●
●
programmer expects
●
PHP coerces strings to types before comparing
them
●
But what if you want to compare strings?

Choices: PHP
if “1” == “1.0” {
echo(“Equal”);
}
Possible...

Choices: PHP
if “1” === “1.0” {
echo(“Equal”);
}
…with a different operator

Choices: PHP
if “1” === “1.0” {
echo(“Equal”);
}
Not always convenient or intuitive to seasoned
programmers...

Choices: Python
●
Variables are strongly typed

Choices: Python
●
●
The language forces the programmer to match
up the types

Choices: PHP
if int(“1”) == 1:
print(“Equal”)
if “1” == str(1):
print(“Equal”)
You could convert:

Choices: PHP
a = “1”
b = [(“1”, “digit”), (“one”, “word”)]
for entry in b:
if a == entry[1]:
print(entry[0])
You could define a different way to compare:

Choices: Python
●
●
The language forces the programmer to match
up the types
●
The power is in your hands

Let’s talk about Python3 bytes

In Python3, you can define a text string
(an immutable sequence of human readable
characters):
a = str(“ñ”)

And you can define a byte string
(an immutable sequence of bytes):
a = str(“ñ”)
b = bytes(b"xc3xb1")

And when you attempt to compare those...
a = str(“ñ”)
b = bytes(b"xc3xb1")
print("Equal") if a == b else print("Unequal")

...they continue to do what you expect:
python3 -c 'a = str("ñ"); b = bytes(b"xc3
xb1") ; print("Equal") if a == b else
print("Unequal")'
Unequal

If you, the programmer, decide that you want to
convert and compare, you can do that:
a = str("ñ")
b = bytes(b"xc3xb1").decode("latin1")
if a == b:
print(f"{a} == {b}: Equal")
else:
print(f"{a} == {b}: Unequal")'
# OUTPUT: ñ == Ã±: Unequal

With way that you choose to convert having a
hand in the results:
a = str("ñ")
b = bytes(b"xc3xb1").decode("utf-8")
if a == b:
else:
print(f"{a} == {b}: Unequal")'
# OUTPUT: ñ == ñ: Equal

Let’s talk about idiomatic Python3

●
I’ve been using constructors

●
– Make clear we’re dealing with different types

●
●
Python3 has string literals...

Let's talk about idiomatic Python3
...for text strings:
a = “z”

...and for bytes strings:
a = “z”
b = b"xc3xb1"

These are the same as using the constructor:
a = “ñ”
b = b"xc3xb1"
if a == b.decode("utf-8"):
else:
print(f"{a} == {b}: Unequal")
# ñ == ñ: Equal

●
●
Python3 has string literals…
●
Python3 has syntactic sugar for byte strings if
they only contain characters present in ASCii.

This is the sugar:
a = “z”
b = b”z”

But this is just sugar. They are still different types
which compare unequal:
a = “z”
b = b”z”
if a == b:
print(“Equal”)
else:
print("Unequal")
# Unequal

Unless you explicitly convert them:
a = “z”
b = b”z”
if a == b.decode(“utf-8”):
print(“Equal”)
else:
print("Unequal")
# Equal

Warning: they may still compare unequal when
you decode...
a = “z”
b = b”z”
if a == b.decode(“ebcdic-cp-ch”):
print(“Equal”)
else:
print("Unequal")
# Unequal

●
...because b”z” is actually 0x7a
●
and the encoding determines which human
character that maps to
a = “z”
b = b”z”
if a == b.decode(“ebcdic-cp-ch”):
print(“Equal”)
else:
print("Unequal")
# Unequal

Let’s talk about Python2!
(finally)

●
PHPPython2 will coerce bytes type to text type

●
(Ouch)

●
– There are many circumstances when it will coerce

Let’s talk about Python2
●
String formatting with %
a = b“test %s” % u”string” # u”test string”
b = u”test %s” % b“string” # u”test string”

●
Adding strings
a = b“this” + u”too” # u”this too”
b = u”this” + b“too” # u”this too”

●
Joining strings
a = u”“.join(b“yep”, b”pers”) # u”yeppers”
b = b””.join(u””, b“empty too”) # u”empty too”

●
Comparisons
if u”test” == b”test”:
print(‘Equal’)
# Equal

●
So what’s the problem?

●
●
Python2 can only coerce ASCii characters

●
●
●
Attempting to coerce other characters will fail

Most coercions that fail, fail with a traceback...
a = u“test %s” % b”coffee” # u”test coffee”
b = u”test %s” % b”café”
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode
byte 0xc3 in position 3: ordinal not in
range(128)

… but with comparisons, the implicit coercion
results in a warning
if b"coffee" == u"coffee":
print("Equal")
# Equal
if b"café" == u"café":
print("Equal")
__main__:1: UnicodeWarning: Unicode equal
comparison failed to convert both arguments to
Unicode - interpreting them as being unequal

Depending on your locale settings….
echo -n 'hello' > ‘み.txt’
LC_ALL=pt_BR.utf-8 python2 -c 'print(open(u"
u307f.txt").read())'
hello

...encoding can also traceback
echo -n 'hello' > ‘み.txt’
LC_ALL=pt_BR.iso88591 python2 -c
'print(open(u"u307f.txt").read())'
File "<string>", line 1, in <module>
UnicodeEncodeError: 'latin-1' codec can't
encode characters in position 0-3: ordinal not
in range(256

●
●
●
Attempting to coerce other characters will fail
(Ouch) (Ouch)

What is the underlying difference?

●
Python3 is modeled around them being friendly
but non-substitutable types

●
– Explicit conversions exist but not implicit

●
– Explicit conversions exist but not implicit
– Most APIs take one or the other

●
– Explicit conversion exist but not implicit
●
Python2 is modeled around text and bytes
being largely substitutable

●
●
– Plethora of implicit conversions

●
●
– Plethora of implicit conversions
– Most APIs accept either one….

Let’s talk about Liskov
●
The Liskov Substitution Principle formulates a
property of good object design

●
The Liskov Substitution Principle formulates an
essential property of good object design

●
●
Behaviors of child objects cannot modify the
behaviors of parent objects

●
●
Behaviors of child objects cannot modify the
behaviors of parent objects
●
Allows safely substituting the child for the
parent

●
Text and byte strings do not satisfy Liskov

●
(The Python authors knew this. There’s no
parent-child relationship between them)

●
– translate()
– decode()
– encode()
–

●
– translate()
– decode()
– encode()
●
What does that mean for us?

Changing our expectations
●
Would you expect this to work?
– assert [u“one”] + u“two” == [u”one”, u”two”]

Change our expectations
●
●
How about this?
– assert add(1, u”two”) == 3

●
●
How about this?
●
So why do we expect this to work?
– assert concat( b“one”, u”two) == b”onetwo”
●
Two different types; not substitutable: it is up to
the caller to decide what to do

●
●
How about this?
●
So why do we expect this to work?
– assert concat( b“one”, u”two) == b”onetwo”

The Unicode Sandwich
●
Python2’s text type is called “unicode()”

●
●
Make all human-readable strings unicode type

●
●
●
Make all binary data str type

●
●
●
– Use a naming convention to identify variables that
hold binary data

●
●
●
hold binary data
●
Transform to the appropriate type immediately
after data enters your application

●
●
●
hold binary data
●
Transform to the appropriate type immediately
after data enters your application
●
Transform to the type an external API expects
just before calling the API

Writing APIs: General
●
Create APIs that accept text if they need
human-readable data

●
human-readable data
●
Create APIs that accept bytes if they deal with
binary data

●
human-readable data
●
binary data
●
Use a naming convention to identify functions
which return bytes

●
human-readable data
●
binary data
●
Use a naming convention to identify functions
which return bytes
●
Avoid making functions which mix text and
bytes

Writing APIs: Mixing
So you want to disregard my advice and write
functions which allow mixing….
assert concat(b”xe4xb8x80”, u” ”二 ) == u“一 ”二

Questions:
●
Should this return text or bytes?
assert concat(b”xe4xb8x80”, u”二”) == u“一二”

Questions:
●
●
What encoding should it use to convert?

Questions:
●
●
What encoding should it use?
●
What should happen when it can’t convert?
assert concat(b”xe4xb8xff”, u”二”) == u“一二”
File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'utf-8' codec can't decode
bytes in position 0-1: invalid continuation
byte

Answer:
●
Return bytes or text: Naming conventions!
def b_concat(str1, str2) -> bytes:
def concat(str1, str2) -> unicode:

Answer:
●
Return bytes or text: Naming conventions
●
Encoding: Caller is in control
def b_concat(str1, str2, encoding=”utf-8”) ->
bytes:
def concat(str1, str2, encoding=”utf-8”) ->
unicode:

Answer:
●
Return bytes or text: Naming conventions!
●
Encoding: Caller is in control
●
Handling errors: Caller is in control
def b_concat(str1, str2, encoding=”utf-8”,
errors=”strict”) -> bytes:
def concat(str1, str2, encoding=”utf-8”,
errors=”strict”) -> unicode:

The danger of mixed APIs
●
Now that you know how to write mixed APIs, a
reminder not to do it.
●
Mixed APIs encourage sloppy programming
●
Instead of understanding the types you are
using and controlling them you get used to
throwing any type at it and getting useful output.
●
Don’t do that.

Exceptions
●
Sometimes you’ll have an API that is type-less.
●
Like repr()… give it any type of data and get
something sensible.
●
What else could be like that?
●
Debug logging:
– Logging.debug(“Not a message, an object”)
– Logging.debug(Configparser(filename))
– Logging.debug(b”Above would sensibly log the
particulars about the ConfigParser object. This logs
the particulars about a bytes object”)

Exceptions
●
Debug logging

Exceptions
●
Debug logging
●
10:00:UTC|u”Not a message, an object”|

Exceptions
●
Debug logging
●
10:00:UTC|u”Not a message, an object”
– Logging.debug(pathlib.Path(“/etc/passwd”))

Exceptions
●
Debug logging
●
●
10:00:UTC|PosixPath('/')

Exceptions
●
Debug logging
●
●
– Logging.debug(b”Remember: logging objects”)

Exceptions
●
Debug logging
●
●
– Logging.debug(b”Remember: logging objects”)
●
10:00:UTC|b”Remember: logging objects”

Finis
●
Thanks to Gary Bernhardt of Destroy All
Software which inspired the format of this talk
●
https://www.destroyallsoftware.com/talks/wat
●
Kumar McMillan’s Pycon talk on unicode in
Python2; old but good introduction to the solution
●
http://farmdev.com/talks/unicode/
●
I’m Toshio Kuratomi (@abadger1999, @abadger,
and <abadger gmail>
●
Hope you had fun!

Python2 unicode-pt1

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Python2 unicode-pt1

Similar to Python2 unicode-pt1 (20)

Recently uploaded

Recently uploaded (20)

Python2 unicode-pt1