• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Porting to Python 3

Porting to Python 3



Lennart Regebro's talk about how to port to Python 3 from PyCon 2011 in Atlanta.

Lennart Regebro's talk about how to port to Python 3 from PyCon 2011 in Atlanta.



Total Views
Views on SlideShare
Embed Views



0 Embeds 0

No embeds



Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

    Porting to Python 3 Porting to Python 3 Presentation Transcript

    • Porting to Python 3 Lennart Regebro @ PyCon Atlanta 2011Sebastian, an albino Burmese python. Photo by Emmanuel “Tambako” Keller
    • Python 3: Pretty scary
    • Not actually dangerous
    • Not going awayStackoverflow questions about Python 3
    • Third-party projectsThe number of projects on the CheeseShop
    • Check the statushttp://py3ksupport.appspot.com/Currently 34% out of the top 50 projects has Python 3 support
    • Choosing a strategy
    • Only support Python 3● For end-user software and systems● Run 2to3 once● Then fix the code until it works.
    • Separate trees● For stable Python libraries● Make a branch or copy● Run 2to3 once● Fix until it works● Bug fix both trees
    • How to distributeInclude both src2/ and src3/ and have differentparameters for different Python versions:import sysfrom distutils.core import setupif sys.version < 3: package_dir = {: src2}else: package_dir = {: src3}setup(name=foo, version=1.0, package_dir = package_dir)
    • Continuous conversion with 2to3● If you need to support Python 2 and Python 3 and the code is changing a lot● Keep developing in Python 2● Convert to Python 3 for testing and install● Compatibility hacks may be needed● Distribute helps
    • Single code base; no conversion● If 2to3 for some reason is infeasible● Support Python 2 and Python 3 from the same code without 2to3● Loads of compatibility hacks● Fun, but ugly!● Adding Python 2 support to Python 3 is often less work than the other way around● Check out six
    • Try with 2to3 first!● Run 2to3 to see if it changes much● If in doubt use Distribute!
    • When?● For end-user systems: When you feel like it ● Not a good thing to do the week before deployment ● Your customers want Python 3!● For libraries: As soon as you can ● I.e. when your dependencies runs on Python 3
    • Preparing for the port
    • Get rid of warnings● Run under Python 2.7 with -3● Fix all deprecation warnings● [Use iterator methods on dicts]
    • Prepare for bytes● Use separate variables for binary data● Add b and t to file flags
    • Use key instead of cmp>>> def compare(a, b):... return cmp(a[1:], b[1:])>>> names = [Adam, Donald, John]>>> names.sort(cmp=compare)>>> names[Adam, John, Donald]
    • Slow!● The cmp function compares pairs● Averages (from Jarret Hardie): ● 4 items: 6 calls (1.5 per item) ● 10 items: 22 calls (2.2 per item) ● 100 items: 528 calls (5.28 per item) ● 40,000 items: 342,541 calls (8.56 per item)
    • New way>>> def keyfunction(item):... return item[1:]>>> names = [Adam, Donald, John]>>> names.sort(key=keyfunction)>>> names[Adam, John, Donald]
    • Fast and simple!● The key method is called exactly one time per item● The key method is simpler to implement than a cmp method
    • Way simpler>>> names = [Adam, Donald, John]>>> names.sort(key=len)>>> names[Adam, John, Donald]>>> names = [Adam, Benno, april]>>> names.sort(key=str.lower)>>> names[Adam, april, Benno]
    • Use rich comparison methods● No support for __cmp__ in Python 3● Use rich comparison operators instead: __lt__, __le__, __eq__, __ge__, __gt__, __ne__● Not entirely trivial to implement ● The @totalordering recipe is WRONG!
    • My mixinclass ComparableMixin(object): def _compare(self, other, method): try: return method(self._cmpkey(), other._cmpkey()) except (AttributeError, TypeError): # _cmpkey not implemented, or return different type, # so I cant compare with "other". return NotImplemented def __lt__(self, other): return self._compare(other, lambda s, o: s < o) def __le__(self, other): return self._compare(other, lambda s, o: s <= o) def __eq__(self, other): return self._compare(other, lambda s, o: s == o) def __ge__(self, other): return self._compare(other, lambda s, o: s >= o) def __gt__(self, other): return self._compare(other, lambda s, o: s > o) def __ne__(self, other): return self._compare(other, lambda s, o: s != o)
    • http://bit.ly/richcomp
    • Write tests Increasing your test coverage will simplify porting andincrease confidence in the port
    • Porting with 2to3
    • Running 2to3 manually2to3 -w .2to3 -w -d .2to3 -w -d README.txt
    • Using 2to3 with Distributesetup(name=mystuff, ... test_suite=mystuff.tests.test_suite, use_2to3=True,)
    • Common Problems
    • Non-ported imports>>> from urllib2 import urlparse
    • Replacing UserDict● UserDict.IterableUserDict → collections.UserDict● UserDict.DictMixin → collections.MutableMapping● UserDict.UserDict → collections.UserDict
    • Bytes vsStrings vsUnicode
    • BytesUsed for binary data:>>> file = open(maybe_a.gif, rb)>>> file.read(6) == GIF89aFalse>>> file.read(6)bGIF89a
    • Byte literals>>> type(bGIF89a)<class bytes>>>> bGIF89a[2]70An alias for str in Python 2.6 and 2.7:>>> type(bGIF89a)<class str>>>> bGIF89a[2]F
    • Python < 2.6>>> import sys>>> if sys.version < 3:... def b(x):... return x... else:... def b(x):... return x.encode(latin-1)>>> b(binaryx67data)
    • Still behaves differently>>> data = open(a.gif, rb).read()>>> data[2]Python 2: FPython 3: 70
    • Manipulating binary data● >= 2.6: Use bytearray● <= 2.5: Helper methods to iterate and index
    • Reading from files● Python 2: Always 8-bit strings● Python 3: 8-bit bytes or Unicode strings
    • The io module>>> import io>>> infile = io.open(UTF-8.txt, rt, encoding=UTF-8)
    • The codecs module>>> import codecs>>> infile = codecs.open(UTF-8.txt, rt, encoding=UTF-8)
    • Without 2to3
    • stdlib reorganisationtry: from StringIO import StringIOexcept ImportError: from io import StringIO
    • Workaround for except:● Python 2: except Exception, e:● Python 3: except Exception as e:● Python 2.6 and 2.7 handles both● Python 3 and Python <= 2.5: except Exception: e = sys.exc_info()[1]
    • Workaround for print()● For simple print cases: print(“Something”) works everywhere. print(“Something”, “else”) doesnt.● Use print_() from the six module● Or make your own
    • The return of Bytes vs Strings vs Unicode
    • Python < 2.6>>> import sys>>> if sys.version < 3:... def u(x):... return x.decode(... unicode_escape)... else:... def u(x):... return x>>> u(Unicxf6de)Unicöde
    • Intcompability 1L no longer works>>> import sys>>> if sys.version > 3:… long = int>>> long(1)1
    • C Extensions● No 2to3● Most differences can be handled with #ifndefs and macros● There is talk about making a compatibility header file.
    • Python 2 module initPyMODINIT_FUNCinitspam(void){ (void) Py_InitModule("spam", SpamMethods);}
    • Python 3 module initstatic struct PyModuleDef spammodule = { PyModuleDef_HEAD_INIT, "spam", /* name of module */ spam_doc, /* module documentation */ -1, /* size of per-interpreter state*/ SpamMethods};PyMODINIT_FUNCPyInit_spam(void){ return PyModule_Create(&spammodule);}
    • Writing your own fixers● If you must modify the API● Can be very complex due to edge cases● Requires a whole talk by itself
    • 2 * 3 = Six● A module to help with compatibility● Several compatibility methods: ● b() and u() ● print_() ● etc...● Tons of helpful constants: ● integer_types ● string_types ● etc...
    • Additional Resources● http://docs.python.org/py3k/howto/pyporting.html● http://wiki.python.org/moin/PortingToPy3k/● python-porting@python.org● “Porting to Python 3”● Ill be sprinting, come and ask!
    • KEALAVV7http://python3porting.com/