Goes through the strategies of porting to Python 3 and how to solve the most common problems.

1. Porting to Python 3
Lennart Regebro
EuroPython 2010, Birmingham

2. Python 3 is here

3. Loads of goodies!
● print is a function
● All strings are unicode
● Binary data handled by new bytes type
● dict.keys/items/values() are generators
● Integers are all long
● 3/2 == 1.5
● The standard library is reorganized
● New easter egg

4. Choosing a strategy

5. Only support Python 3
● For end-user software and systems
● Run 2to3 once
● Then fix the code until it works.
● Done!

6. Separate branches
● For stable python modules
● If you only do bugfixes
● Make a branch for the old version
● Run 2to3 once
● Fix until it works
● Bug fix both branches

7. How to distribute
● Separate package names on pypi
● Add a warning and error in the setup.py:
“For Python 2 support, please use version
3.x of MyCoolModule”
● Only release binary eggs (bdist_egg)
● Include both src2/ and src3/ and have
different setup() parameters for different
Python versions.

8. Continuous conversion with 2to3
● If you need to support Python 2 and Python
3 and the code is changing a lot
● Keep developing in Python 2
● Convert to Python 3 for testing and install
● Only one source distribution
● Compatibility hacks may be needed

9. Solution: Distribute!
● Very little work to set up 2to3 support
● Runs 2to3 on testing and installing:
– Only one source tree
– Only one distribution

10. Using Distribute
setup(name='py3examples',
...
test_suite='py3example.tests.test_suite',
use_2to3=True,
convert_2to3_doctests=['py3example/README.txt'],
)

11. Single code base; no conversion
● If 2to3 for some reason is infeasible
● Support Python 2 and Python 3 from the
same code without 2to3
● No distribution worries!
● Loads of compatibility hacks
● Fun, but ugly!

12. When?
● For end-user systems: When you feel like it
● For modules: As soon as you can
● I.e. when your dependencies do

13. Preparing for the port

14. Get rid of warnings
● Run under Python 2.6 with -3
● Fix all deprecation warnings

15. Use // instead of /
In Python 2:
>>> 3/2
1
In Python 3:
>>> 3/2
1.5
In Python 2.2 and later:
>>> 3//2
1

16. Prepare for bytes
● Use separate variables for binary data
● Add 'b' and 't' to file flags

17. Write tests
Increasing your test coverage will
simplify porting and increase
confidence in the port

18. Porting

19. Porting with 2to3
2to3 -w .
2to3 -w -d .
2to3 -w -d README.txt

20. Common Problems

21. Comparisons
● cmp() is gone
● cmp = lambda a, b: (a > b) - (a < b)

22. Bytes vs Strings vs Unicode
The unicode type is now called string
The string type is gone
The bytes type handles binary data

23. Bytes:
● Used for binary data
>>> file = open('maybe_a.gif', 'rb')
>>> file.read(6)
b'GIF89'
>>> b'GIF89'[2]
70

24. Solutions
>>> import sys
>>> if sys.version < '3':
... def b(x):
... return x
... else:
... import codecs
... def b(x):
... return codecs.latin_1_encode(x)[0]
>>> b('binaryx67data')

25. Still behaves differently
>>> data = open('a.gif', 'rb').read()
>>> data[2]
Python 2: 'F'
Python 3: 70

26. Wrapper class for Python 2
class bites(str):
def __new__(cls, value):
if isinstance(value[0], int):
# It's a list of integers
value = ''.join([chr(x) for x in value])
return super(bites, cls).__new__(cls, value)
def itemint(self, index):
return ord(self[index])
def iterint(self):
for x in self:
yield ord(x)

27. Wrapper class for Python 3
class bites(bytes):
def __new__(cls, value):
if isinstance(value, str):
# It's a unicode string:
value = value.encode('ISO-8859-1')
return super(bites, cls).__new__(cls, value)
def itemint(self, x):
return self[x]
def iterint(self):
for x in self:
yield x

28. Usage
>>> from bites import bites
>>> bites('GIF89a').itemint(2)
70
>>> for x in bites('GIF89a').iterint():
... print(x),
71 73 70 56 57 97

29. Mixed/unknown files
● Open in binary mode, and decode
>>> data = open('maybe_a.gif', 'rb').read()
>>> if data[:6] != bites('GIF89a'):
... str = data.decode('Latin-1')

30. Unorderable types
● Many built in types that were comparable
now are not
● __cmp__ is no longer supported

31. Use “Rich comparison methods”
● __lt__
● __le__
● __eq__
● __ge__
● __gt__
● __ne__

32. Doctests
● Binary data now is a b'' literal
● Unicode data no longer has the u''
● Classes and types have often changed
names or at least __repr__
● Exceptions are formated differently as a
result, and exception matching doesn't
work under both versions.

33. Test with ==
Go from:
>>> myfunction()
u'Foobar'
To:
>>> myfunction() == u'Foobar'
True

34. Capture exceptions
>>> try:
... dosomethingthatfails()
... print “ExpectedException not raised”
... except ExpectedException:
... print “Success”
Success

35. Without 2to3

36. Workaround for stdlib reorg
try:
from StringIO import StringIO
except ImportError:
from io import StringIO

37. Workaround for except:
● Python 2:
except Exception, e:
● Python 3:
except Exception as e:
● Both:
except Exception:
e = sys.exc_info[1]

38. Workaround for print()
● For simple print cases:
print(“Something”) works everywhere.
print(“Something”, “else”) doesn't.
● Make your own print function!

39. Modernize your code

40. Don't float ints
Python 2:
>>> float(3)/2
1.5
Python 2.2 and later:
>>> from __future__ import division
>>> 3/2
1.5

41. Sorting
● Old way:
– Needs lists
– Slow
– Comparison methods
● New way:
– Sort any iterable
– Fast
– Key methods = trivial (often lambdas)

42. Old way
>>> def lastnamefirst(a, b):
... s = "%s, %s" % (a.lastname, a.firstname)
... o = "%s, %s" % (b.lastname, b.firstname)
... return cmp(s, o)
>>> thelist = [Person('Donald', 'Duck'), Person('Paul', 'Anka')]
>>> thelist.sort(cmp=lastnamefirst)
>>> thelist
[Paul Anka, Donald Duck]

43. Exponential slowness
● The cmp function compares pairs
● Averages (from Jarret Hardie):
– 4 items: 6 calls (1.5 per item)
– 10 items: 22 calls (2.2 per item)
– 100 items: 528 calls (5.28 per item)
– 40,000 items: 342,541 calls (8.56 per item)

44. New way
>>> def lastnamefirst(a):
... return "%s, %s" % (a.lastname, a.firstname)
>>> thetuple = (Person('Donald', 'Duck'), Person('Paul', 'Anka'),)
>>> sorted(thetuple, key=lastnamefirst)
[Paul Anka, Donald Duck]

45. Fast!
The method is called exactly one time per item

46. Use iterator methods
Instead of
{}.keys(), {}.values(), {}.items()
Use
{}.iterkeys(), {}.itervalues(), {}.iteritems()

47. Use iterator methods
● In Python 3, {}.keys(), {}.values(),
{}.items() are iterators
● 2to3 wraps them in list(), so d.values()
becomes list(d.values())
● By explicitly using iterators in Python 2
code, you avoid the list()

48. Put an in in it
● Old: adict.has_key(foo)
● New: foo in adict

49. C-code
● There are API changes
● But you can handle them with ifdefs.

50. 2 * 3 = Six
● A module to help with compatibility
● Several compatibility methods:
– b()
– print_()
– etc...
● Tons of helpful constants:
– integer_types
– string_types
– etc...

51. That's it!

52. Comparison Example
class ComparableMixin(object):
def __lt__(self, other): return self._compare(other, (-1,))
def __le__(self, other): return self._compare(other, (-1, 0,))
def __eq__(self, other): return self._compare(other, (0,))
def __ge__(self, other): return self._compare(other, (0, 1,))
def __gt__(self, other): return self._compare(other, (1,))
def __ne__(self, other): return self._compare(other, (-1, 1,))

53. Comparison Example
class Orderable(ComparableMixin):
def __init__(self, firstname, lastname):
self.firstname = firstname
self.lastname = lastname
def _compare(self, other, truthvalues):
try:
s = "%s, %s" % (self.lastname, self.firstname)
o = "%s, %s" % (other.lastname, other.firstname)
value = (s > o) - (s < o)
return value in truthvalues
except AttributeError:
return NotImplemented