Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Python meetup: coroutines, event loops, and non-blocking I/O

2,167 views

Published on

An introduction to the notions that made node.js famous: asynchronous I/O in the Python world.

Published in: Engineering, Technology
  • Be the first to comment

Python meetup: coroutines, event loops, and non-blocking I/O

  1. 1. Tikitu de Jager • @tTikitu • tikitu@buzzcapture.com going async coroutines event loops non-blocking I/O PUN • Utrecht • 20-6-2014
  2. 2. magic import asyncio import asyncio_redis ! @asyncio.coroutine def my_subscriber(channel): # Create connection connection = yield from asyncio_redis.Connection.create(host='localhost', port=6379) # Create subscriber. subscriber = yield from connection.start_subscribe() # Subscribe to channel. yield from subscriber.subscribe([channel]) # Wait for incoming events. while True: reply = yield from subscriber.next_published() print('Received: ', repr(reply.value), 'on channel', reply.channel) ! loop = asyncio.get_event_loop() asyncio.async(my_subscriber('channel-1')) asyncio.async(my_subscriber('channel-2')) loop.run_forever() source: asyncio-redis
  3. 3. non-blocking I/O queueing for coffee starbucks just waiting around
  4. 4. coffee as metaphor for I/O ❖ blocking I/O is queueing for coffee ❖ guy in front wants 17 litres of kopi luwak ❖ all I want is an espresso ❖ non-blocking I/O is the starbucks model ❖ give your order, then go wait somewhere ❖ they call you back when it’s ready
  5. 5. “non-blocking”? starbucks queueing is not useful if your application is ❖ CPU-bound ❖ I/O-bound by pushing bits it is useful if you spend most of your time waiting doing stuff still takes time
  6. 6. waiting most I/O is not pushing bits: ❖ server waits for connections ❖ call a service: wait for response ❖ wait for socket buffer to fill if you’re just waiting: yield the CPU … but then who will give it back to you? …
  7. 7. event loop did anything happen? how about now? callback hell
  8. 8. you know this ❖ GUI programming: “when this button is clicked…” ❖ (old-fashioned) javascript onclick &c ❖ event loop checks for events and runs callbacks ❖ (select module makes polling for events easy)
  9. 9. callbacks for non-blocking I/O? a_socket.recv(bufsize=16) event_loop.when_socket_has_ready_buffer(a_s, 16, callback_f) “callback hell”
  10. 10. coroutines stop/go generators yield
  11. 11. coroutines and generators a coroutine is a routine (function) that can pause and resume its execution def a_coroutine(): do_some_stuff() yield do_some_more_stuff() def a_coroutine(): do_some_stuff() yield to some_other_coroutine # invented syntax do_some_more_stuff() A generator is a coroutine that can only yield to its caller
  12. 12. yield to the event loop import asyncio import asyncio_redis ! @asyncio.coroutine def my_subscriber(channels): # [snip] # Wait for incoming events. while True: reply = yield from subscriber.next_published() print('Received: ', repr(reply.value), 'on channel', reply.channel) ! loop = asyncio.get_event_loop() asyncio.async(my_subscriber('channel-1')) asyncio.async(my_subscriber('channel-2')) loop.run_forever()
  13. 13. roll-your-own event loop def loop(): while True: for coroutine in waiting_list: if io_is_ready_for(coroutine): running_list.push(coroutine) coroutine = running_list.pop() coroutine.next() (p.s. don’t do this)
  14. 14. what’ll it be? twisted gevent asyncio node.js
  15. 15. twisted ❖ networking protocols ❖ callbacks (methods on a “protocol” class) ❖ e.g. connectionMade(self), dataReceived(self, data) ❖ “you don't port an application to Twisted: You write a Twisted application in most cases.” —Jesse Noller ❖ “deferred”: abstraction now usually called Future or Promise (proxy for a value that will be computed later)
  16. 16. asyncio ❖ similar high-level protocols but also ❖ intends to provide a base layer for other libraries ❖ yield from ❖ python3 stdlib ❖ (how I got interested in this whole business)
  17. 17. yield from event_loop.please_run_for_me(a_generator()) ! def a_generator(): for val in nested_generator(): yield val ! def nested_generator(): for val in deeper_nested_generator(): yield val ! def deeper_nested_generator(): event_loop.register(self, for_io_op='recv', on_socket=a_socket) yield return a_socket.recv() … but what if we have to support generator send()? def a_function(): nested_function() def nested_function(): deeper_nested_function() def deeper_nested_function(): return a_socket.recv()
  18. 18. send() the wrong way gen = a_generator() gen.next() gen.send(1) gen.send(2) gen.next() ! def a_generator(): gen = nested_generator() to_yield = gen.next() while True: to_send = yield to_yield if to_send is None: to_yield = gen.next() else: to_yield = gen.send(to_send) D anger! Untested probably incorrect code!
  19. 19. next() send() throw() close() _i = iter(EXPR) try: _y = next(_i) except StopIteration as _e: _r = _e.value else: while 1: try: _s = yield _y except GeneratorExit as _e: try: _m = _i.close except AttributeError: pass else: _m() raise _e except BaseException as _e: _x = sys.exc_info() try: _m = _i.throw except AttributeError: raise _e else: try: _y = _m(*_x) except StopIteration as _e: _r = _e.value break else: try: if _s is None: _y = next(_i) else: _y = _i.send(_s) except StopIteration as _e: _r = _e.value break RESULT = _r RESULT = yield from EXPR def a_generator(): yield from nested_generator() def nested_generator(): yield from deeper_nested_generator() def deeper_nested_generator(): event_loop.register(self, for_io_op='recv', on_socket=a_socket) yield return a_socket.recv()
  20. 20. asyncio ❖ yield from ❖ its own low-level I/O operations (socket recv &c.) ❖ futures, promises, timers, … ❖ networking protocols
  21. 21. gevent def a_generator(): yield from nested_generator() def nested_generator(): yield from deeper_nested_generator() def deeper_nested_generator(): event_loop.register(self, for_io_op='recv', on_socket=a_socket) yield return a_socket.recv() from gevent import monkey; monkey.patch_all() ! def a_function(): nested_function() def nested_function(): deeper_nested_function() def deeper_nested_function(): return a_socket.recv() # monkey-patched! ! import gevent jobs = [gevent.spawn(a_function) for _ in range(5)] gevent.wait(jobs) how?!
  22. 22. greenlets ❖ full coroutines ❖ monkey-patched primitives can “yield to” the event loop directly ❖ C extension for CPython
  23. 23. node.js really?
  24. 24. the summaries modules tech buzzwords
  25. 25. module summary twisted ❖ venerable ❖ focus on networking protocols ❖ perceived as large and complex; porting not easy gevent ❖ near “drop-in” in synchronous code ❖ python 3 support is … coming? asyncio ❖ python 3 stdlib ❖ (“Tulip” is a python 2 port: “yield From(a_generator)”) ❖ aims to be both low-level and protocol-level library
  26. 26. tech summary greenlets ❖ full coroutines ❖ CPython hack; some pypy support yield from ❖ python 3 ❖ nested generators ❖ goldilocks solution? callbacks ❖ low-tech, no special support needed ❖ promises, futures, etc: there is space for useful abstraction ❖ node.js
  27. 27. buzzwords ❖ non-blocking I/O: yield the CPU instead of waiting ❖ an event loop gives it back to you when what you’re waiting for happens ❖ coroutines and generators let you write synchronous- style functions and still yield to the event loop mid-way
  28. 28. that’s all folks and yes we’re hiring
  29. 29. thank you’s and references ❖ y’all for your attention ❖ @saghul for getting me started on this whole business ❖ Peter Portante for a 2011 PyCon talk on coroutines ❖ Reinout & Maurits for PyGrunn summaries ❖ PEPs: ❖ 3156 (asyncio) ❖ 380 (yield from)

×