Porting to Python 3

  • 2,648 views
Uploaded on

Lennart Regebro's talk about how to port to Python 3 from PyCon 2011 in Atlanta.

Lennart Regebro's talk about how to port to Python 3 from PyCon 2011 in Atlanta.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
2,648
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
22
Comments
0
Likes
2

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Porting to Python 3 Lennart Regebro @ PyCon Atlanta 2011Sebastian, an albino Burmese python. Photo by Emmanuel “Tambako” Keller
  • 2. Python 3: Pretty scary
  • 3. Not actually dangerous
  • 4. Not going awayStackoverflow questions about Python 3
  • 5. Third-party projectsThe number of projects on the CheeseShop
  • 6. Check the statushttp://py3ksupport.appspot.com/Currently 34% out of the top 50 projects has Python 3 support
  • 7. Choosing a strategy
  • 8. Only support Python 3● For end-user software and systems● Run 2to3 once● Then fix the code until it works.
  • 9. Separate trees● For stable Python libraries● Make a branch or copy● Run 2to3 once● Fix until it works● Bug fix both trees
  • 10. How to distributeInclude both src2/ and src3/ and have differentparameters for different Python versions:import sysfrom distutils.core import setupif sys.version < 3: package_dir = {: src2}else: package_dir = {: src3}setup(name=foo, version=1.0, package_dir = package_dir)
  • 11. Continuous conversion with 2to3● If you need to support Python 2 and Python 3 and the code is changing a lot● Keep developing in Python 2● Convert to Python 3 for testing and install● Compatibility hacks may be needed● Distribute helps
  • 12. Single code base; no conversion● If 2to3 for some reason is infeasible● Support Python 2 and Python 3 from the same code without 2to3● Loads of compatibility hacks● Fun, but ugly!● Adding Python 2 support to Python 3 is often less work than the other way around● Check out six
  • 13. Try with 2to3 first!● Run 2to3 to see if it changes much● If in doubt use Distribute!
  • 14. When?● For end-user systems: When you feel like it ● Not a good thing to do the week before deployment ● Your customers want Python 3!● For libraries: As soon as you can ● I.e. when your dependencies runs on Python 3
  • 15. Preparing for the port
  • 16. Get rid of warnings● Run under Python 2.7 with -3● Fix all deprecation warnings● [Use iterator methods on dicts]
  • 17. Prepare for bytes● Use separate variables for binary data● Add b and t to file flags
  • 18. Use key instead of cmp>>> def compare(a, b):... return cmp(a[1:], b[1:])>>> names = [Adam, Donald, John]>>> names.sort(cmp=compare)>>> names[Adam, John, Donald]
  • 19. Slow!● The cmp function compares pairs● Averages (from Jarret Hardie): ● 4 items: 6 calls (1.5 per item) ● 10 items: 22 calls (2.2 per item) ● 100 items: 528 calls (5.28 per item) ● 40,000 items: 342,541 calls (8.56 per item)
  • 20. New way>>> def keyfunction(item):... return item[1:]>>> names = [Adam, Donald, John]>>> names.sort(key=keyfunction)>>> names[Adam, John, Donald]
  • 21. Fast and simple!● The key method is called exactly one time per item● The key method is simpler to implement than a cmp method
  • 22. Way simpler>>> names = [Adam, Donald, John]>>> names.sort(key=len)>>> names[Adam, John, Donald]>>> names = [Adam, Benno, april]>>> names.sort(key=str.lower)>>> names[Adam, april, Benno]
  • 23. Use rich comparison methods● No support for __cmp__ in Python 3● Use rich comparison operators instead: __lt__, __le__, __eq__, __ge__, __gt__, __ne__● Not entirely trivial to implement ● The @totalordering recipe is WRONG!
  • 24. My mixinclass ComparableMixin(object): def _compare(self, other, method): try: return method(self._cmpkey(), other._cmpkey()) except (AttributeError, TypeError): # _cmpkey not implemented, or return different type, # so I cant compare with "other". return NotImplemented def __lt__(self, other): return self._compare(other, lambda s, o: s < o) def __le__(self, other): return self._compare(other, lambda s, o: s <= o) def __eq__(self, other): return self._compare(other, lambda s, o: s == o) def __ge__(self, other): return self._compare(other, lambda s, o: s >= o) def __gt__(self, other): return self._compare(other, lambda s, o: s > o) def __ne__(self, other): return self._compare(other, lambda s, o: s != o)
  • 25. http://bit.ly/richcomp
  • 26. Write tests Increasing your test coverage will simplify porting andincrease confidence in the port
  • 27. Porting with 2to3
  • 28. Running 2to3 manually2to3 -w .2to3 -w -d .2to3 -w -d README.txt
  • 29. Using 2to3 with Distributesetup(name=mystuff, ... test_suite=mystuff.tests.test_suite, use_2to3=True,)
  • 30. Common Problems
  • 31. Non-ported imports>>> from urllib2 import urlparse
  • 32. Replacing UserDict● UserDict.IterableUserDict → collections.UserDict● UserDict.DictMixin → collections.MutableMapping● UserDict.UserDict → collections.UserDict
  • 33. Bytes vsStrings vsUnicode
  • 34. BytesUsed for binary data:>>> file = open(maybe_a.gif, rb)>>> file.read(6) == GIF89aFalse>>> file.read(6)bGIF89a
  • 35. Byte literals>>> type(bGIF89a)<class bytes>>>> bGIF89a[2]70An alias for str in Python 2.6 and 2.7:>>> type(bGIF89a)<class str>>>> bGIF89a[2]F
  • 36. Python < 2.6>>> import sys>>> if sys.version < 3:... def b(x):... return x... else:... def b(x):... return x.encode(latin-1)>>> b(binaryx67data)
  • 37. Still behaves differently>>> data = open(a.gif, rb).read()>>> data[2]Python 2: FPython 3: 70
  • 38. Manipulating binary data● >= 2.6: Use bytearray● <= 2.5: Helper methods to iterate and index
  • 39. Reading from files● Python 2: Always 8-bit strings● Python 3: 8-bit bytes or Unicode strings
  • 40. The io module>>> import io>>> infile = io.open(UTF-8.txt, rt, encoding=UTF-8)
  • 41. The codecs module>>> import codecs>>> infile = codecs.open(UTF-8.txt, rt, encoding=UTF-8)
  • 42. Without 2to3
  • 43. stdlib reorganisationtry: from StringIO import StringIOexcept ImportError: from io import StringIO
  • 44. Workaround for except:● Python 2: except Exception, e:● Python 3: except Exception as e:● Python 2.6 and 2.7 handles both● Python 3 and Python <= 2.5: except Exception: e = sys.exc_info()[1]
  • 45. Workaround for print()● For simple print cases: print(“Something”) works everywhere. print(“Something”, “else”) doesnt.● Use print_() from the six module● Or make your own
  • 46. The return of Bytes vs Strings vs Unicode
  • 47. Python < 2.6>>> import sys>>> if sys.version < 3:... def u(x):... return x.decode(... unicode_escape)... else:... def u(x):... return x>>> u(Unicxf6de)Unicöde
  • 48. Intcompability 1L no longer works>>> import sys>>> if sys.version > 3:… long = int>>> long(1)1
  • 49. C Extensions● No 2to3● Most differences can be handled with #ifndefs and macros● There is talk about making a compatibility header file.
  • 50. Python 2 module initPyMODINIT_FUNCinitspam(void){ (void) Py_InitModule("spam", SpamMethods);}
  • 51. Python 3 module initstatic struct PyModuleDef spammodule = { PyModuleDef_HEAD_INIT, "spam", /* name of module */ spam_doc, /* module documentation */ -1, /* size of per-interpreter state*/ SpamMethods};PyMODINIT_FUNCPyInit_spam(void){ return PyModule_Create(&spammodule);}
  • 52. Writing your own fixers● If you must modify the API● Can be very complex due to edge cases● Requires a whole talk by itself
  • 53. 2 * 3 = Six● A module to help with compatibility● Several compatibility methods: ● b() and u() ● print_() ● etc...● Tons of helpful constants: ● integer_types ● string_types ● etc...
  • 54. Additional Resources● http://docs.python.org/py3k/howto/pyporting.html● http://wiki.python.org/moin/PortingToPy3k/● python-porting@python.org● “Porting to Python 3”● Ill be sprinting, come and ask!
  • 55. KEALAVV7http://python3porting.com/