Python Coroutines, Present and Future

22,602 views

Published on

A talk on Python coroutines for asynchronous programming, both *now* in Tornado and Toro, and in the future in Tulip and Python 3.4.

Published in: Technology, Design
  • Be the first to comment

Python Coroutines, Present and Future

  1. 1. Python CoroutinesPresent and Future A. Jesse Jiryu Davis http://emptysquare.net 10gen
  2. 2. Python Coroutines Present and Future Agenda:• Coroutines are wonderful for async• …but weird.• Understanding the weirdness.• Coroutine Kung Fu.• The Future!
  3. 3. Coroutines arewonderful for async
  4. 4. Async with callbackfrom tornado.web import RequestHandlerclass AsyncHandler(RequestHandler): @asynchronous def get(self): http_client = AsyncHTTPClient() http_client.fetch( "http://example.com", callback=self.on_fetch) def on_fetch(self, response): do_something_with_response(response) self.render("template.html")
  5. 5. Async with coroutinefrom tornado.web import RequestHandlerclass GenAsyncHandler(RequestHandler): @asynchronous @gen.coroutine def get(self): http_client = AsyncHTTPClient() response = yield http_client.fetch( "http://example.com") do_something_with_response(response) self.render("template.html")
  6. 6. Async with coroutinefrom tornado.web import RequestHandlerclass GenAsyncHandler(RequestHandler): @asynchronous @gen.coroutine def get(self): http_client = AsyncHTTPClient() future = http_client.fetch( "http://example.com") response = yield future do_something_with_response(response) self.render("template.html")
  7. 7. Agenda:• Coroutines are wonderful for Async I/O• …but weird. • yield • future • coroutine
  8. 8. yielddef make_generator(): yield 0 yield 1g = make_generator()print(g)# <generator object make_generatorat 0x1006b4b40>
  9. 9. def make_generator(): yield 0 yield 1g = make_generator()for i in g: print(i)
  10. 10. def make_generator(): yield 0 yield 1g = make_generator()while True: try: i = g.__next__() print(i) except StopIteration: break
  11. 11. def make_generator(): yield 0 yield 1g = make_generator()while True: try: i = g.send(None) print(i) except StopIteration: break
  12. 12. Prints:def make_generator(): return_value = yield 0 0 print got, return_value got 10 return_value = yield 1 1 print got, return_value got 11g = make_generator()i = g.send(None) # Start gwhile True: try: print(i) i = g.send(i + 10) except StopIteration: break
  13. 13. Agenda:• Coroutines are wonderful for Async I/O• …but weird. • yield • future • coroutine
  14. 14. Futureclass GenAsyncHandler(RequestHandler): @asynchronous @gen.coroutine def get(self): http_client = AsyncHTTPClient() future = http_client.fetch( "http://example.com") response = yield future do_something_with_response(response) self.render("template.html")
  15. 15. future = Future()future.done() # Falsefuture.set_result(foo)future.set_exception( SomeException(error message))future.add_done_callback(callback)# Return result or raise errorfuture.result()
  16. 16. class GenAsyncHandler(RequestHandler): @asynchronous @gen.coroutine def get(self): http_client = AsyncHTTPClient() future = http_client.fetch( "http://example.com") response = yield future do_something_with_response(response) self.render("template.html")
  17. 17. Agenda:• Coroutines are wonderful for Async I/O• …but weird. • yield • future • coroutine
  18. 18. # part of the Tornado frameworkclass Runner(object): def __init__(self, make_generator): self.gen = make_generator() # Starts done, with result None self.future = NullFuture()
  19. 19. class Runner(object): # ... "recurse" def run(self): while True: if not self.future.done(): self.future.add_done_callback(self.run) return value = self.future.result() try: self.future = self.gen.send(value) except (StopIteration, Return) as e: return
  20. 20. class GenAsyncHandler(RequestHandler): @asynchronous @gen.coroutine def get(self): http_client = AsyncHTTPClient() future = http_client.fetch( "http://example.com") response = yield future do_something_with_response(response) self.render("template.html")
  21. 21. Agenda:• Coroutines are wonderful for Async I/O• …but weird.• Handling the weirdness.
  22. 22. @gen.coroutinedef print_code(): response = yield get(http://example.com) print response.code@gen.coroutine weirddef get(url): client = AsyncHTTPClient() response = yield client.fetch(url) raise gen.Return(response)print_code()
  23. 23. @gen.coroutinedef print_code(): future = get(http://example.com) response = yield future print response.code@gen.coroutinedef get(url): client = AsyncHTTPClient() response = yield client.fetch(url) raise gen.Return(response)print_code() weird
  24. 24. Python 3.3@gen.coroutinedef print_code(): future = get(http://example.com) response = yield future print response.code@gen.coroutinedef get(url): client = AsyncHTTPClient() response = yield client.fetch(url) return responseprint_code() normal
  25. 25. @gen.coroutinedef print_code(): try: code = yield get(http://example.com) print code except HTTPError, e: normal print e@gen.coroutinedef get(url): client = AsyncHTTPClient() response = yield client.fetch(url) raise gen.Return(response.code)print_code()
  26. 26. Agenda:• Coroutines are wonderful for Async I/O• …but weird.• Handling the weirdness.• Coroutine Kung Fu.
  27. 27. Fan-out@gen.coroutinedef f(): client = AsyncHTTPClient() responses = yield [ client.fetch(http://mongodb.org), client.fetch(http://10gen.com)] print responsesf()
  28. 28. Fan-out@gen.coroutinedef f(): client = AsyncHTTPClient() future0 = client.fetch(http://mongodb.org) future1 = client.fetch(http://10gen.com) responses = yield [future0, future1] print responsesf()
  29. 29. ToroSynchronization Primitives for Tornado Coroutines
  30. 30. event = toro.Event()@gen.coroutinedef waiter(): print "Ill wait right here" yield event.wait() # Yield a Future print "Im done waiting"@gen.coroutinedef setter(): print "About to set" event.set() print "Done setting"waiter()setter()
  31. 31. q = toro.Queue(maxsize=3)@gen.coroutinedef producer(): for item in range(5): print Sending, item yield q.put(item)@gen.coroutinedef consumer(): while True: item = yield q.get() print tt, Got, itemconsumer()producer()
  32. 32. $ python producer_consumer.pySending 0 ! ! Got 0Sending 1 ! ! Got 1Sending 2 ! ! Got 2Sending 3 ! ! Got 3Sending 4 ! ! Got 4
  33. 33. Agenda:• Coroutines are wonderful for Async I/O• …but weird.• Handling the weirdness.• Kung Fu.• The Future!
  34. 34. Tulip• A standard event loop• A standard coroutine library• For inclusion in Python 3.4
  35. 35. yield from
  36. 36. @gen.coroutinedef print_code(): Tornado response = yield get(http://example.com) print response.code@gen.coroutinedef get(url): client = AsyncHTTPClient() response = yield client.fetch(url) raise gen.Return(response)@tulip.coroutinedef print_code(): Tulip response = yield from get(http://example.com) print(response.status)@tulip.coroutine "yield from"def get(url): request = tulip.http.request(GET, url) response = yield from request return response normal return
  37. 37. Guido on"The Difference Between yield and yield-from":http://bit.ly/yieldfrom
  38. 38. q = tulip.queues.Queue(maxsize=3)@tulip.coroutinedef producer(): for item in range(5): print(Sending, item) yield from q.put(item)@tulip.coroutinedef consumer(): while True: item = yield from q.get() print(tt, Got, item)Task(consumer())Task(producer())
  39. 39. StopIteration A. Jesse Jiryu Davis http://emptysquare.net 10gen

×