Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Python Coroutines, Present and Future

23,092 views

Published on

A talk on Python coroutines for asynchronous programming, both *now* in Tornado and Toro, and in the future in Tulip and Python 3.4.

Published in: Technology, Design

Python Coroutines, Present and Future

  1. 1. Python CoroutinesPresent and Future A. Jesse Jiryu Davis http://emptysquare.net 10gen
  2. 2. Python Coroutines Present and Future Agenda:• Coroutines are wonderful for async• …but weird.• Understanding the weirdness.• Coroutine Kung Fu.• The Future!
  3. 3. Coroutines arewonderful for async
  4. 4. Async with callbackfrom tornado.web import RequestHandlerclass AsyncHandler(RequestHandler): @asynchronous def get(self): http_client = AsyncHTTPClient() http_client.fetch( "http://example.com", callback=self.on_fetch) def on_fetch(self, response): do_something_with_response(response) self.render("template.html")
  5. 5. Async with coroutinefrom tornado.web import RequestHandlerclass GenAsyncHandler(RequestHandler): @asynchronous @gen.coroutine def get(self): http_client = AsyncHTTPClient() response = yield http_client.fetch( "http://example.com") do_something_with_response(response) self.render("template.html")
  6. 6. Async with coroutinefrom tornado.web import RequestHandlerclass GenAsyncHandler(RequestHandler): @asynchronous @gen.coroutine def get(self): http_client = AsyncHTTPClient() future = http_client.fetch( "http://example.com") response = yield future do_something_with_response(response) self.render("template.html")
  7. 7. Agenda:• Coroutines are wonderful for Async I/O• …but weird. • yield • future • coroutine
  8. 8. yielddef make_generator(): yield 0 yield 1g = make_generator()print(g)# <generator object make_generatorat 0x1006b4b40>
  9. 9. def make_generator(): yield 0 yield 1g = make_generator()for i in g: print(i)
  10. 10. def make_generator(): yield 0 yield 1g = make_generator()while True: try: i = g.__next__() print(i) except StopIteration: break
  11. 11. def make_generator(): yield 0 yield 1g = make_generator()while True: try: i = g.send(None) print(i) except StopIteration: break
  12. 12. Prints:def make_generator(): return_value = yield 0 0 print got, return_value got 10 return_value = yield 1 1 print got, return_value got 11g = make_generator()i = g.send(None) # Start gwhile True: try: print(i) i = g.send(i + 10) except StopIteration: break
  13. 13. Agenda:• Coroutines are wonderful for Async I/O• …but weird. • yield • future • coroutine
  14. 14. Futureclass GenAsyncHandler(RequestHandler): @asynchronous @gen.coroutine def get(self): http_client = AsyncHTTPClient() future = http_client.fetch( "http://example.com") response = yield future do_something_with_response(response) self.render("template.html")
  15. 15. future = Future()future.done() # Falsefuture.set_result(foo)future.set_exception( SomeException(error message))future.add_done_callback(callback)# Return result or raise errorfuture.result()
  16. 16. class GenAsyncHandler(RequestHandler): @asynchronous @gen.coroutine def get(self): http_client = AsyncHTTPClient() future = http_client.fetch( "http://example.com") response = yield future do_something_with_response(response) self.render("template.html")
  17. 17. Agenda:• Coroutines are wonderful for Async I/O• …but weird. • yield • future • coroutine
  18. 18. # part of the Tornado frameworkclass Runner(object): def __init__(self, make_generator): self.gen = make_generator() # Starts done, with result None self.future = NullFuture()
  19. 19. class Runner(object): # ... "recurse" def run(self): while True: if not self.future.done(): self.future.add_done_callback(self.run) return value = self.future.result() try: self.future = self.gen.send(value) except (StopIteration, Return) as e: return
  20. 20. class GenAsyncHandler(RequestHandler): @asynchronous @gen.coroutine def get(self): http_client = AsyncHTTPClient() future = http_client.fetch( "http://example.com") response = yield future do_something_with_response(response) self.render("template.html")
  21. 21. Agenda:• Coroutines are wonderful for Async I/O• …but weird.• Handling the weirdness.
  22. 22. @gen.coroutinedef print_code(): response = yield get(http://example.com) print response.code@gen.coroutine weirddef get(url): client = AsyncHTTPClient() response = yield client.fetch(url) raise gen.Return(response)print_code()
  23. 23. @gen.coroutinedef print_code(): future = get(http://example.com) response = yield future print response.code@gen.coroutinedef get(url): client = AsyncHTTPClient() response = yield client.fetch(url) raise gen.Return(response)print_code() weird
  24. 24. Python 3.3@gen.coroutinedef print_code(): future = get(http://example.com) response = yield future print response.code@gen.coroutinedef get(url): client = AsyncHTTPClient() response = yield client.fetch(url) return responseprint_code() normal
  25. 25. @gen.coroutinedef print_code(): try: code = yield get(http://example.com) print code except HTTPError, e: normal print e@gen.coroutinedef get(url): client = AsyncHTTPClient() response = yield client.fetch(url) raise gen.Return(response.code)print_code()
  26. 26. Agenda:• Coroutines are wonderful for Async I/O• …but weird.• Handling the weirdness.• Coroutine Kung Fu.
  27. 27. Fan-out@gen.coroutinedef f(): client = AsyncHTTPClient() responses = yield [ client.fetch(http://mongodb.org), client.fetch(http://10gen.com)] print responsesf()
  28. 28. Fan-out@gen.coroutinedef f(): client = AsyncHTTPClient() future0 = client.fetch(http://mongodb.org) future1 = client.fetch(http://10gen.com) responses = yield [future0, future1] print responsesf()
  29. 29. ToroSynchronization Primitives for Tornado Coroutines
  30. 30. event = toro.Event()@gen.coroutinedef waiter(): print "Ill wait right here" yield event.wait() # Yield a Future print "Im done waiting"@gen.coroutinedef setter(): print "About to set" event.set() print "Done setting"waiter()setter()
  31. 31. q = toro.Queue(maxsize=3)@gen.coroutinedef producer(): for item in range(5): print Sending, item yield q.put(item)@gen.coroutinedef consumer(): while True: item = yield q.get() print tt, Got, itemconsumer()producer()
  32. 32. $ python producer_consumer.pySending 0 ! ! Got 0Sending 1 ! ! Got 1Sending 2 ! ! Got 2Sending 3 ! ! Got 3Sending 4 ! ! Got 4
  33. 33. Agenda:• Coroutines are wonderful for Async I/O• …but weird.• Handling the weirdness.• Kung Fu.• The Future!
  34. 34. Tulip• A standard event loop• A standard coroutine library• For inclusion in Python 3.4
  35. 35. yield from
  36. 36. @gen.coroutinedef print_code(): Tornado response = yield get(http://example.com) print response.code@gen.coroutinedef get(url): client = AsyncHTTPClient() response = yield client.fetch(url) raise gen.Return(response)@tulip.coroutinedef print_code(): Tulip response = yield from get(http://example.com) print(response.status)@tulip.coroutine "yield from"def get(url): request = tulip.http.request(GET, url) response = yield from request return response normal return
  37. 37. Guido on"The Difference Between yield and yield-from":http://bit.ly/yieldfrom
  38. 38. q = tulip.queues.Queue(maxsize=3)@tulip.coroutinedef producer(): for item in range(5): print(Sending, item) yield from q.put(item)@tulip.coroutinedef consumer(): while True: item = yield from q.get() print(tt, Got, item)Task(consumer())Task(producer())
  39. 39. StopIteration A. Jesse Jiryu Davis http://emptysquare.net 10gen

×