This document discusses porting the Twisted Python library to Python 3. It describes Twisted as a networking library that uses asynchronous programming. The author ported over 325 pull requests to help move Twisted to Python 3. Some challenges included changes to Python 3 like print becoming a function and str handling, as well as porting old C extensions. Running extensive unit tests was important. While difficult, porting benefits code quality and keeps Twisted compatible with Python's direction.

1. The Onward Journey:
Porting Twisted to Python 3
Craig Rodrigues
<rodrigc@crodrigues.org>

2. Twisted

3. What is Twisted?
Python library, written by Glyph Lefkowitz and many others
provides basic building blocks for writing networking clients and
servers:
protocol implementations: SSH, IRC, SMTP, IMAP
event “reactors”: select, kqueue, epoll, IOCP, asyncio…...

4. More resources
Web site with docs, examples, tutorials: http://www.twistedmatrix.com
Main code repository: https://github.com/twisted/twisted
O’Reilly book:

5. Twisted sub-projects
Klein (similar to Flask, Bottle) https://github.com/twisted/klein
Treq (similar to Requests) https://github.com/twisted/treq

6. Which projects Twisted?
buildbot
scrapy
Lots of other projects:
https://twistedmatrix.com/trac/wiki/ProjectsUsingTwisted

7. Who uses Twisted?
Hipchat
Apple Calendar Server
Lots of companies: https://twistedmatrix.com/trac/wiki/SuccessStories

8. Twisted and asynchronous programming
Twisted has promoted the technique of using callbacks and asynchronous
programming for writing network servers for a long time
Twisted’s Deferred class is central to this, and has been used extensively
Guido van Rossum consulted with Glyph in the design of Python’s new
asyncio framework ( Python 3.4 and higher )

9. Interesting things
First commit to Twisted was in 2001
Twisted includes a tool trial which runs unittest-style tests. (similar to pytest
or nose)
Unittests and code coverage are very important to Twisted development
process

10. Why bother?

11. Motivation
I like Twisted and the community behind it
I had some time in between jobs and wanted to improve my Python skills and
learn more about Python 3
I wanted to help out projects which depended on Twisted, but couldn’t move
to Python 3, such as buildbot
Core Python devs are dropping Python 2 support in 2020:
https://pythonclock.org/

12. Major motivation: Twisted moved to GitHub
in 2016!!
Following process to submit patches via Subversion was cumbersome
Moving to GitHub and pull requests made things easier for submitters and
reviewers
Integration with Continuous Integration (CI) was improved: codecov, travis,
appveyor, buildbot
Submitting patches is easier!!

13. Moving Twisted to GitHub

14. Moving to Python 3

15. Twisted started moving to Python 3
Original Python 3 porting plan developed in 2012
Worked on by various developers: Jean-Paul Calderone, Itamar Turner-
Trauring, Amber Brown, Glyph Lefkowitz, Ralph Meijer, and others
Canonical funded some Python 3 porting work
Some parts ported, many parts still unported

16. Moving Twisted to Python 3 is a tough job!
Old codebase (since 2001)
Advanced framework which uses many, many features of Python
Twisted development process requires unit tests and coverage
Submitting hundreds of patches in Subversion workflow was slow moving

17. Porting to Python 3

18. What has changed in Python 3?
Lots of little changes to make the language cleaner
Deprecated code has been removed
Some changes are backwards incompatible. Previously working code is now
broken on Python 3
http://python3porting.com has an extensive list of changes in Python 3

19. print is now a function
print “hello world”
now must be:
print(“hello world”)

20. dict.has_key() is gone
some_dict = { “one” : 1, “two”: 2}
some_dict.has_key(“one”)
Now should be:
“one” in some_dict

21. obj.__cmp__() and cmp() is gone
Developers are supposed to implement __lt__(), __gt__(), __ge__(), __le__(),
__ne__(), __eq__() functions on an object instead of __cmp__()
Developers need to use <, >, >=, <=, !=, == operators instead of cmp() which is
gone

22. Less things allocate lists
These functions no longer allocate lists in Python 3:
range(), dict.items(), dict.values(), map(), filter()
Users should iterate over these functions, which only allocate items as they
are needed:
for n in range(99):
...

23. C API for Python C extensions changed
C API changed in a backwards incompatible way
Very challenging when porting the Twisted IOCP reactor (Windows only)
which has parts written in C

24. Python str type has changed
Python 2:
u”Some unicode string 銩” is of type unicode
“Some string” is of type str and also of type bytes
b”Some string” is of type str and also of type bytes
type(unicode) != type(str), type(str) == type(bytes)
Python 3:
u”Some string 銩” is of type str

25. Python str type has changed
Twisted protocols must send out bytes over the wire on sockets
Lots of code written assuming type(str) == type(bytes), did not account for
unicode
This type of porting needs extensive analysis and testing, cannot be
automated

26. Python str type has changed
Ned Batchelder unicode presentation very good:
https://nedbatchelder.com/text/unipain/unipain.html
“Unicode sandwich” technique (write bytes to sockets and files, keep data as
unicode internally in application) cannot be used 100% when dealing with
network protocols

27. Technique for porting to Python 3

28. Porting technique: use virtualenvs
Checkout the code from git
Create a python2 virtualenv in one window:
virtualenv myenv_2
source myenv_2/bin/activate
python setup.py develop
Create a python3 virtualenv in another window:
python3 -m venv myenv_3

29. Porting technique: run unit tests
After modifying code, run unittests using tox and trial
See what breaks, make sure it works on Python 2.7 and Python 3
Write new unit tests if necessary
Always try to improve code coverage:
https://codecov.io/gh/twisted/twisted/

30. Python 3 status for Twisted

31. June 3, 2016
Python 2.7: 8425 tests PASS
Python 3.5: 4834 tests PASS ( approx. 57% of Python 2.7 tests)

32. March 21, 2017
Python 2.7: 9692 tests PASS
Python 3.5: 9025 tests PASS ( approx. 93% of Python 2.7 tests)
My contributions: 325 pull requests!

33. What’s left?
A few modules need to be ported such as:
twisted.mail
twisted.news
twisted.web

34. Lessons learned

35. What I learned
Extensive unittests and code coverage are very important for this kind of
effort
Porting an old and large codebase to Python 3 can be a lot of work
Benefits of porting: code cleanliness and keeping up with Python
direction...the benefits vs. the effort required sometimes doesn’t feel worth
it

36. Hope for the future and performance
CPython 3.6 and 3.7 performance seems to be improving and is comparable
to Python 2.7:
http://speed.python.org
Pypy 3.5 just came out….hopefully better performance with that
Now that Python has asyncio built in, the “Twisted way” of doing things has
some validation, and is pervading more libraries and projects in Python

37. Thanks
All who started before me on the Python 3 effort: Jean-Paul, Itamar, Amber,
Glyph, Ralph, many others
All who helped code review my patches: Adi Roiban, Alex Gaynor, Glyph,
many others
Special thanks to Abhishek Choudhary for help on code reviews