0
Asychronous I/O in
Python 3
Feihong Hsu
Chicago Python User Group
July 11, 2013
What is PEP 3156?
• "Asynchronous IO Support Rebooted"
• Describes a new event loop API
• Aims to be compatible with curre...
The big take-away
We're going to be able to write
asynchronous code without threads or
callbacks!
Threads vs event loop
• I'm not going to talk about this
• Twisted being around for more than a
decade kinda validates the...
Twisted vs Tulip
Ugh, how to explain this?
I could just give you a boring list of
differences...
Or I could use an analogy...
Programming with
Twisted is like...
Being a mystical ninja warrior!
Mystical Ninja Warrior
• Can defeat demons and demigods by
deploying an army of shadow clones
• Mental gymnastics required...
Programming with Tulip
is like...
Playing PlantsVs Zombies
Playing PlantsVs
Zombies
• Can defeat a zombie horde by deploying
and managing a large number of
horticultural specimens
•...
Alrighty
With the analogies out of the way, let's look
at some examples.
Fetching a web page in
Twisted
https://github.com/feihong/tulip-talk/blob/
master/examples/1-twisted-download.py
Control flow diagram
I was going to draw one but then I got
sleepy.
Problems with callback-
based asynch code
• Can be hard to understand control flow
• Error messages are not friendly
• Debu...
Fetching a web page in
Tulip
https://github.com/feihong/tulip-talk/blob/
master/examples/2-tulip-download.py
Fetching a web page in
Tulip w/ callbacks
https://github.com/feihong/tulip-talk/blob/
master/examples/3-tulip-download-
ca...
Fetching a web page in
Twisted w/ inline
callbacks
https://github.com/feihong/tulip-talk/blob/
master/examples/4-twisted-d...
The Tulip API
• Basics are documented in PEP 3156
• Not available in Python Package Index
• Includes sockets, file I/O, etc...
Coroutines vs tasks
When I first started, the most confusing
thing about Tulip.
Coroutines are executed by "yield from".
Ta...
Coroutine
• Basically a function that contains at least
one "yield from" statement.
• Not the same thing as a generator
fu...
Coroutine
@tulip.coroutine
def download(url):
response = yield from tulip.http.request('GET', url)
return yield from respo...
Task
• An object that manages a coroutine
• Roughly equivalent to the Deferred object
in Twisted
Task
@tulip.task
def download(url):
response = yield from tulip.http.request('GET', url)
return yield from response.read()...
Task
In Tulip, there are two ways of creating tasks:
tulip.task is a decorator that produces
a task-wrapping function
tuli...
Why bother to use
tasks?
• To interoperate with callback-based
frameworks like Twisted
• To cancel an already-running coro...
Wait, so what's a
Future?
PEP 3156 makes frequent mention of
Futures.
But talking about Futures is a little confusing,
sin...
Future is the superclass
of Task
• Future don't necessarily manage a
coroutine
• In practice you never create Future
objec...
Methods on Future
• cancel(), cancelled()
• running(), done()
• result(), set_result()
• add_done_callback(),
remove_done_...
Web development with
Tulip
• All the classes you need are in the tulip.http
module
• Make subclass of
tulip.http.ServerHtt...
Hello World!
https://github.com/feihong/tulip-talk/blob/
master/examples/5-hello-world-http.py
Some observations
The HTTP API is fairly simple, but a bit low
level for everyday web programming.
Expect a thousand micro...
Introducing viol
• Tiny web framework based on Tulip
• After initial page load, messages between
client and server are exc...
Tulip API demos
Now for a bunch of demos...
Tulip API demos
Serial
http://www.youtube.com/v/jMgRUI7V_mk?
version=3&autoplay=1&&start=1546
Simultaneous
http://www.youtube.com/v/jMgRUI7V_mk?
version=3&autoplay=1&&start=1666
Wait for both
coroutines to finish
http://www.youtube.com/v/jMgRUI7V_mk?
version=3&autoplay=1&&start=1713
Loop through
coroutine results in
order of finishing
http://www.youtube.com/v/jMgRUI7V_mk?
version=3&autoplay=1&&start=1869
Execute synchronous
code in another thread
http://www.youtube.com/v/jMgRUI7V_mk?
version=3&autoplay=1&&start=1930
Talk to event loop from
another thread
http://www.youtube.com/v/jMgRUI7V_mk?
version=3&autoplay=1&&start=2060
A regex debugging tool
http://www.youtube.com/v/jMgRUI7V_mk?
version=3&autoplay=1&&start=2158
Chat application
http://www.youtube.com/v/jMgRUI7V_mk?
version=3&autoplay=1&&start=2236
Tulip Links
• PEP 3156
• Guido's PyCon 2013 keynote
• Tulip project page on GoogleCode
Other links
• An Introduction to Asynchronous
Programming and Twisted
• Code for this talk on Github
• Viol project page o...
Questions?
Upcoming SlideShare
Loading in...5
×

Asynchronous I/O in Python 3

11,780

Published on

Feihong talks about PEP 3156 and basic usage of Tulip, the reference implementation.

Video: http://pyvideo.org/video/2194/asynchronous-io-in-python-3

Source code: https://github.com/feihong/tulip-talk/

Published in: Technology, Spiritual
0 Comments
15 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
11,780
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
72
Comments
0
Likes
15
Embeds 0
No embeds

No notes for slide

Transcript of "Asynchronous I/O in Python 3"

  1. 1. Asychronous I/O in Python 3 Feihong Hsu Chicago Python User Group July 11, 2013
  2. 2. What is PEP 3156? • "Asynchronous IO Support Rebooted" • Describes a new event loop API • Aims to be compatible with current popular event loops (Twisted,Tornado, etc.) • Should land in the standard library in 3.4 (early 2014)
  3. 3. The big take-away We're going to be able to write asynchronous code without threads or callbacks!
  4. 4. Threads vs event loop • I'm not going to talk about this • Twisted being around for more than a decade kinda validates the event loop model • For a detailed discussion see An Intro to Asynchronous Programming and Twisted
  5. 5. Twisted vs Tulip Ugh, how to explain this? I could just give you a boring list of differences... Or I could use an analogy...
  6. 6. Programming with Twisted is like... Being a mystical ninja warrior!
  7. 7. Mystical Ninja Warrior • Can defeat demons and demigods by deploying an army of shadow clones • Mental gymnastics required to coordinate shadow clones requires years of brutal training • Ninja magic comes with a terrible price: the more clones you make, the more insane you become
  8. 8. Programming with Tulip is like... Playing PlantsVs Zombies
  9. 9. Playing PlantsVs Zombies • Can defeat a zombie horde by deploying and managing a large number of horticultural specimens • Not particularly mentally taxing • Somehow end up playing for hours without eating or sleeping
  10. 10. Alrighty With the analogies out of the way, let's look at some examples.
  11. 11. Fetching a web page in Twisted https://github.com/feihong/tulip-talk/blob/ master/examples/1-twisted-download.py
  12. 12. Control flow diagram I was going to draw one but then I got sleepy.
  13. 13. Problems with callback- based asynch code • Can be hard to understand control flow • Error messages are not friendly • Debugging is harder
  14. 14. Fetching a web page in Tulip https://github.com/feihong/tulip-talk/blob/ master/examples/2-tulip-download.py
  15. 15. Fetching a web page in Tulip w/ callbacks https://github.com/feihong/tulip-talk/blob/ master/examples/3-tulip-download- callbacks.py
  16. 16. Fetching a web page in Twisted w/ inline callbacks https://github.com/feihong/tulip-talk/blob/ master/examples/4-twisted-download-inline- callbacks.py
  17. 17. The Tulip API • Basics are documented in PEP 3156 • Not available in Python Package Index • Includes sockets, file I/O, etc. • Includes concurrency data structures like locks, queues, semaphores • Includes SSL and HTTP (with support for websockets!)
  18. 18. Coroutines vs tasks When I first started, the most confusing thing about Tulip. Coroutines are executed by "yield from". Tasks are not executed by "yield from".
  19. 19. Coroutine • Basically a function that contains at least one "yield from" statement. • Not the same thing as a generator function, which is a function that contains a yield expression. • Tulip will barf if you try to make it execute a generator.
  20. 20. Coroutine @tulip.coroutine def download(url): response = yield from tulip.http.request('GET', url) return yield from response.read() Calling "download()" returns a generator object, but otherwise does nothing!You need to do "yield from download()" to run the body of a coroutine.
  21. 21. Task • An object that manages a coroutine • Roughly equivalent to the Deferred object in Twisted
  22. 22. Task @tulip.task def download(url): response = yield from tulip.http.request('GET', url) return yield from response.read() Calling "download()" actually does run the body of the function.The "yield from" part is done implicitly for you.
  23. 23. Task In Tulip, there are two ways of creating tasks: tulip.task is a decorator that produces a task-wrapping function tulip.Task is a constructor that accepts a coroutine
  24. 24. Why bother to use tasks? • To interoperate with callback-based frameworks like Twisted • To cancel an already-running coroutine • To start a new coroutine from within another coroutine
  25. 25. Wait, so what's a Future? PEP 3156 makes frequent mention of Futures. But talking about Futures is a little confusing, since there are two Future classes: tulip.Future and concurrent.futures.Future.
  26. 26. Future is the superclass of Task • Future don't necessarily manage a coroutine • In practice you never create Future objects, only Task objects • Futures are acceptable to yield from expressions
  27. 27. Methods on Future • cancel(), cancelled() • running(), done() • result(), set_result() • add_done_callback(), remove_done_callback() • ...
  28. 28. Web development with Tulip • All the classes you need are in the tulip.http module • Make subclass of tulip.http.ServerHttpProtocol • Override the handle_request() method
  29. 29. Hello World! https://github.com/feihong/tulip-talk/blob/ master/examples/5-hello-world-http.py
  30. 30. Some observations The HTTP API is fairly simple, but a bit low level for everyday web programming. Expect a thousand microframeworks to bloom in the near future. Speaking of microframeworks...
  31. 31. Introducing viol • Tiny web framework based on Tulip • After initial page load, messages between client and server are exchanged via websockets • Makes code demos a bit more visual
  32. 32. Tulip API demos Now for a bunch of demos...
  33. 33. Tulip API demos
  34. 34. Serial http://www.youtube.com/v/jMgRUI7V_mk? version=3&autoplay=1&&start=1546
  35. 35. Simultaneous http://www.youtube.com/v/jMgRUI7V_mk? version=3&autoplay=1&&start=1666
  36. 36. Wait for both coroutines to finish http://www.youtube.com/v/jMgRUI7V_mk? version=3&autoplay=1&&start=1713
  37. 37. Loop through coroutine results in order of finishing http://www.youtube.com/v/jMgRUI7V_mk? version=3&autoplay=1&&start=1869
  38. 38. Execute synchronous code in another thread http://www.youtube.com/v/jMgRUI7V_mk? version=3&autoplay=1&&start=1930
  39. 39. Talk to event loop from another thread http://www.youtube.com/v/jMgRUI7V_mk? version=3&autoplay=1&&start=2060
  40. 40. A regex debugging tool http://www.youtube.com/v/jMgRUI7V_mk? version=3&autoplay=1&&start=2158
  41. 41. Chat application http://www.youtube.com/v/jMgRUI7V_mk? version=3&autoplay=1&&start=2236
  42. 42. Tulip Links • PEP 3156 • Guido's PyCon 2013 keynote • Tulip project page on GoogleCode
  43. 43. Other links • An Introduction to Asynchronous Programming and Twisted • Code for this talk on Github • Viol project page on Github
  44. 44. Questions?
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×