Python meetup: coroutines, event loops, and non-blocking I/O
Upcoming SlideShare
Loading in...5
×
 

Python meetup: coroutines, event loops, and non-blocking I/O

on

  • 463 views

An introduction to the notions that made node.js famous: asynchronous I/O in the Python world.

An introduction to the notions that made node.js famous: asynchronous I/O in the Python world.

Statistics

Views

Total Views
463
Views on SlideShare
426
Embed Views
37

Actions

Likes
0
Downloads
3
Comments
0

3 Embeds 37

https://www.buzzcapture.com 17
http://www.buzzcapture.com 13
http://www.slideee.com 7

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Python meetup: coroutines, event loops, and non-blocking I/O Python meetup: coroutines, event loops, and non-blocking I/O Presentation Transcript

  • Tikitu de Jager • @tTikitu • tikitu@buzzcapture.com going async coroutines event loops non-blocking I/O PUN • Utrecht • 20-6-2014
  • magic import asyncio import asyncio_redis ! @asyncio.coroutine def my_subscriber(channel): # Create connection connection = yield from asyncio_redis.Connection.create(host='localhost', port=6379) # Create subscriber. subscriber = yield from connection.start_subscribe() # Subscribe to channel. yield from subscriber.subscribe([channel]) # Wait for incoming events. while True: reply = yield from subscriber.next_published() print('Received: ', repr(reply.value), 'on channel', reply.channel) ! loop = asyncio.get_event_loop() asyncio.async(my_subscriber('channel-1')) asyncio.async(my_subscriber('channel-2')) loop.run_forever() source: asyncio-redis
  • non-blocking I/O queueing for coffee starbucks just waiting around
  • coffee as metaphor for I/O ❖ blocking I/O is queueing for coffee ❖ guy in front wants 17 litres of kopi luwak ❖ all I want is an espresso ❖ non-blocking I/O is the starbucks model ❖ give your order, then go wait somewhere ❖ they call you back when it’s ready
  • “non-blocking”? starbucks queueing is not useful if your application is ❖ CPU-bound ❖ I/O-bound by pushing bits it is useful if you spend most of your time waiting doing stuff still takes time
  • waiting most I/O is not pushing bits: ❖ server waits for connections ❖ call a service: wait for response ❖ wait for socket buffer to fill if you’re just waiting: yield the CPU … but then who will give it back to you? …
  • event loop did anything happen? how about now? callback hell
  • you know this ❖ GUI programming: “when this button is clicked…” ❖ (old-fashioned) javascript onclick &c ❖ event loop checks for events and runs callbacks ❖ (select module makes polling for events easy)
  • callbacks for non-blocking I/O? a_socket.recv(bufsize=16) event_loop.when_socket_has_ready_buffer(a_s, 16, callback_f) “callback hell”
  • coroutines stop/go generators yield
  • coroutines and generators a coroutine is a routine (function) that can pause and resume its execution def a_coroutine(): do_some_stuff() yield do_some_more_stuff() def a_coroutine(): do_some_stuff() yield to some_other_coroutine # invented syntax do_some_more_stuff() A generator is a coroutine that can only yield to its caller
  • yield to the event loop import asyncio import asyncio_redis ! @asyncio.coroutine def my_subscriber(channels): # [snip] # Wait for incoming events. while True: reply = yield from subscriber.next_published() print('Received: ', repr(reply.value), 'on channel', reply.channel) ! loop = asyncio.get_event_loop() asyncio.async(my_subscriber('channel-1')) asyncio.async(my_subscriber('channel-2')) loop.run_forever()
  • roll-your-own event loop def loop(): while True: for coroutine in waiting_list: if io_is_ready_for(coroutine): running_list.push(coroutine) coroutine = running_list.pop() coroutine.next() (p.s. don’t do this)
  • what’ll it be? twisted gevent asyncio node.js
  • twisted ❖ networking protocols ❖ callbacks (methods on a “protocol” class) ❖ e.g. connectionMade(self), dataReceived(self, data) ❖ “you don't port an application to Twisted: You write a Twisted application in most cases.” —Jesse Noller ❖ “deferred”: abstraction now usually called Future or Promise (proxy for a value that will be computed later)
  • asyncio ❖ similar high-level protocols but also ❖ intends to provide a base layer for other libraries ❖ yield from ❖ python3 stdlib ❖ (how I got interested in this whole business)
  • yield from event_loop.please_run_for_me(a_generator()) ! def a_generator(): for val in nested_generator(): yield val ! def nested_generator(): for val in deeper_nested_generator(): yield val ! def deeper_nested_generator(): event_loop.register(self, for_io_op='recv', on_socket=a_socket) yield return a_socket.recv() … but what if we have to support generator send()? def a_function(): nested_function() def nested_function(): deeper_nested_function() def deeper_nested_function(): return a_socket.recv()
  • send() the wrong way gen = a_generator() gen.next() gen.send(1) gen.send(2) gen.next() ! def a_generator(): gen = nested_generator() to_yield = gen.next() while True: to_send = yield to_yield if to_send is None: to_yield = gen.next() else: to_yield = gen.send(to_send) D anger! Untested probably incorrect code!
  • next() send() throw() close() _i = iter(EXPR) try: _y = next(_i) except StopIteration as _e: _r = _e.value else: while 1: try: _s = yield _y except GeneratorExit as _e: try: _m = _i.close except AttributeError: pass else: _m() raise _e except BaseException as _e: _x = sys.exc_info() try: _m = _i.throw except AttributeError: raise _e else: try: _y = _m(*_x) except StopIteration as _e: _r = _e.value break else: try: if _s is None: _y = next(_i) else: _y = _i.send(_s) except StopIteration as _e: _r = _e.value break RESULT = _r RESULT = yield from EXPR def a_generator(): yield from nested_generator() def nested_generator(): yield from deeper_nested_generator() def deeper_nested_generator(): event_loop.register(self, for_io_op='recv', on_socket=a_socket) yield return a_socket.recv()
  • asyncio ❖ yield from ❖ its own low-level I/O operations (socket recv &c.) ❖ futures, promises, timers, … ❖ networking protocols
  • gevent def a_generator(): yield from nested_generator() def nested_generator(): yield from deeper_nested_generator() def deeper_nested_generator(): event_loop.register(self, for_io_op='recv', on_socket=a_socket) yield return a_socket.recv() from gevent import monkey; monkey.patch_all() ! def a_function(): nested_function() def nested_function(): deeper_nested_function() def deeper_nested_function(): return a_socket.recv() # monkey-patched! ! import gevent jobs = [gevent.spawn(a_function) for _ in range(5)] gevent.wait(jobs) how?!
  • greenlets ❖ full coroutines ❖ monkey-patched primitives can “yield to” the event loop directly ❖ C extension for CPython
  • node.js really?
  • the summaries modules tech buzzwords
  • module summary twisted ❖ venerable ❖ focus on networking protocols ❖ perceived as large and complex; porting not easy gevent ❖ near “drop-in” in synchronous code ❖ python 3 support is … coming? asyncio ❖ python 3 stdlib ❖ (“Tulip” is a python 2 port: “yield From(a_generator)”) ❖ aims to be both low-level and protocol-level library
  • tech summary greenlets ❖ full coroutines ❖ CPython hack; some pypy support yield from ❖ python 3 ❖ nested generators ❖ goldilocks solution? callbacks ❖ low-tech, no special support needed ❖ promises, futures, etc: there is space for useful abstraction ❖ node.js
  • buzzwords ❖ non-blocking I/O: yield the CPU instead of waiting ❖ an event loop gives it back to you when what you’re waiting for happens ❖ coroutines and generators let you write synchronous- style functions and still yield to the event loop mid-way
  • that’s all folks and yes we’re hiring
  • thank you’s and references ❖ y’all for your attention ❖ @saghul for getting me started on this whole business ❖ Peter Portante for a 2011 PyCon talk on coroutines ❖ Reinout & Maurits for PyGrunn summaries ❖ PEPs: ❖ 3156 (asyncio) ❖ 380 (yield from)