More than just a pretty web framework:        the Tornado IOLoop             Gavin M. Roy
About myYearbookAccording to comScore,myYearbook is one of the 25 most-trafficked sites in the United Statesas measured by...
Tornado @                    myYearbook       • At myYearbook, we used Tornado in high-             request-velocity appli...
Async Network                   Programming       • Core revolves around an I/O loop        • Let the Operating System ind...
Async Network              ProgrammingPyCon 2012               Follow me on Twitter: @Crad
Callback Passing Styledef start(foo):      # Do stuff with foo and when done call the next function      stuff(callback=ne...
tornado.IOLoop       • Core of Tornado’s network stack       • Fast and easy to use       • Single instance per process   ...
tornado.IOStream       • Convenient utility class for dealing with the             IOLoop       • Does most of the work fo...
IOStream: How to read       • read_until_regex             (regex, callback)       • read_until             (delimiter, ca...
IOStream: What’s Going On?      reading(), writing(), closed()PyCon 2012                   Follow me on Twitter: @Crad
Serious  Business Uses       SSL             And that’s why SSLIOStream Exists.PyCon 2012                                 ...
tornado.netutil.TCPServerPyCon 2012              Follow me on Twitter: @Crad
Hello Async Worldfrom tornado import ioloopfrom tornado import netutilclass EchoServer(netutil.TCPServer):      def handle...
What just happened?              tornado.stack_context.StackContext                        “Slight Magic”PyCon 2012       ...
Diving DeeperPyCon 2012               Follow me on Twitter: @Crad
tornado.IOLoop       • IOLoop.instance()       • ioloop.add_handler             (fd, handler, events)       • ioloop.updat...
Events?             READ, WRITE, ERRORPyCon 2012                  Follow me on Twitter: @Crad
“Example”import socketfrom tornado import ioloopdef on_events(fd, events, error=None):      if events & ioloop.IOLoop.READ...
Timers     add_timeout returns a                                    reference                                    deadline ...
add_callback(callback)PyCon 2012            Follow me on Twitter: @Crad
“Pythonic”             SimplificationPyCon 2012               Follow me on Twitter: @Crad
tornado.gen             for coding not in Callback Passing Style:@gen.enginedef foo():    http_client = AsyncHTTPClient() ...
tornado.platform.twistedPyCon 2012             Follow me on Twitter: @Crad
Last Thoughts       • Single threaded, use processes       • Watch out for resource contention       • Benchmark your appl...
multiprocessing.reduction:# Main processfrom multiprocessing.reduction import reduce_handleh = reduce_handle(client_socket...
Upcoming SlideShare
Loading in...5
×

More than just a pretty web framework: the Tornado IOLoop

7,827

Published on

Tornado, often thought of as a web development framework and toolset is built on top of a protocol-agnostic IOLoop, presenting an alternative to Twisted as a foundation for asynchronous application development in Python. This talk covers the Tornado IOLoop, its features and the process of writing drivers and applications using it.

Video available at https://www.youtube.com/watch?v=3BYN3ouwkRA

Published in: Technology, Business
0 Comments
19 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
7,827
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
112
Comments
0
Likes
19
Embeds 0
No embeds

No notes for slide

More than just a pretty web framework: the Tornado IOLoop

  1. 1. More than just a pretty web framework: the Tornado IOLoop Gavin M. Roy
  2. 2. About myYearbookAccording to comScore,myYearbook is one of the 25 most-trafficked sites in the United Statesas measured by page views, byminutes, and by minutes per visitorper month. http://www.myyearbook.com/careersPyCon 2012 Follow me on Twitter: @Crad
  3. 3. Tornado @ myYearbook • At myYearbook, we used Tornado in high- request-velocity applications. • Web apps • Finite State Machine Game Server • Take what you need? We do.PyCon 2012 Follow me on Twitter: @Crad
  4. 4. Async Network Programming • Core revolves around an I/O loop • Let the Operating System indicate when to read, when to write and when there are errors on sockets. • select, poll, epoll (Linux), kqueue (BSD) • Callback-Passing StylePyCon 2012 Follow me on Twitter: @Crad
  5. 5. Async Network ProgrammingPyCon 2012 Follow me on Twitter: @Crad
  6. 6. Callback Passing Styledef start(foo): # Do stuff with foo and when done call the next function stuff(callback=next_step, data=foo);def next_step(bar): # Call more_stuff to parse bar more_stuff(callback=last_step, data=bar);def last_step(baz): # Send the response send_response(baz) # Let our handler know we are done finish()PyCon 2012 Follow me on Twitter: @Crad
  7. 7. tornado.IOLoop • Core of Tornado’s network stack • Fast and easy to use • Single instance per process Works on windows but not supported. C-Based epoll, not required • Cross-platform • Easy to read source: http://goo.gl/VFeAF • Used for client libraries & server applicationsPyCon 2012 Follow me on Twitter: @Crad
  8. 8. tornado.IOStream • Convenient utility class for dealing with the IOLoop • Does most of the work for you • This too is for use by clients and server applicationsPyCon 2012 Follow me on Twitter: @Crad
  9. 9. IOStream: How to read • read_until_regex (regex, callback) • read_until (delimiter, callback) • read_bytes (num_bytes, callback, streaming_callback=None) • read_until_close (callback, streaming_callback=None)PyCon 2012 Follow me on Twitter: @Crad
  10. 10. IOStream: What’s Going On? reading(), writing(), closed()PyCon 2012 Follow me on Twitter: @Crad
  11. 11. Serious Business Uses SSL And that’s why SSLIOStream Exists.PyCon 2012 Follow me on Twitter: @Crad
  12. 12. tornado.netutil.TCPServerPyCon 2012 Follow me on Twitter: @Crad
  13. 13. Hello Async Worldfrom tornado import ioloopfrom tornado import netutilclass EchoServer(netutil.TCPServer): def handle_stream(self, stream, address): self._stream = stream self._read_line() def _read_line(self): self._stream.read_until(n, self._handle_read) def _handle_read(self, data_in): self._stream.write(You sent: %s % data_in) self._read_line()if __name__ == __main__: server = EchoServer() server.listen(2007) ioloop.IOLoop.instance().start() Source: http://goo.gl/eB7z4PyCon 2012 Follow me on Twitter: @Crad
  14. 14. What just happened? tornado.stack_context.StackContext “Slight Magic”PyCon 2012 Follow me on Twitter: @Crad
  15. 15. Diving DeeperPyCon 2012 Follow me on Twitter: @Crad
  16. 16. tornado.IOLoop • IOLoop.instance() • ioloop.add_handler (fd, handler, events) • ioloop.update_handler (fd, handler, events) • ioloop.remove_handler(fd)PyCon 2012 Follow me on Twitter: @Crad
  17. 17. Events? READ, WRITE, ERRORPyCon 2012 Follow me on Twitter: @Crad
  18. 18. “Example”import socketfrom tornado import ioloopdef on_events(fd, events, error=None): if events & ioloop.IOLoop.READ: print Socket read: %r % fd.recv(1024) if events & ioloop.IOLoop.ERROR: print Error received: %r % error if events & ioloop.IOLoop.WRITE: pass_ioloop = ioloop.IOLoop.instance()fd = socket.socket() # Other stuff needs to be done hereevents_desired = ioloop.IOLoop.READ | ioloop.IOLoop.ERROR_ioloop.add_handler(fd, on_events, events_desired)_ioloop.start()PyCon 2012 Follow me on Twitter: @Crad
  19. 19. Timers add_timeout returns a reference deadline can be unix timestamp or a datetime.timedelta • add_timeout (deadline, callback) • remove_timeout (timeout)PyCon 2012 Follow me on Twitter: @Crad
  20. 20. add_callback(callback)PyCon 2012 Follow me on Twitter: @Crad
  21. 21. “Pythonic” SimplificationPyCon 2012 Follow me on Twitter: @Crad
  22. 22. tornado.gen for coding not in Callback Passing Style:@gen.enginedef foo(): http_client = AsyncHTTPClient() response1, response2 = yield [gen.Task(http_client.fetch, url1), gen.Task(http_client.fetch, url2)]PyCon 2012 Follow me on Twitter: @Crad
  23. 23. tornado.platform.twistedPyCon 2012 Follow me on Twitter: @Crad
  24. 24. Last Thoughts • Single threaded, use processes • Watch out for resource contention • Benchmark your application • Fork work to remote workers • multiprocessing.reductionPyCon 2012 Follow me on Twitter: @Crad
  25. 25. multiprocessing.reduction:# Main processfrom multiprocessing.reduction import reduce_handleh = reduce_handle(client_socket.fileno())pipe_to_worker.send(h) #instance of multiprocessing.Pipe# Worker processfrom multiprocessing.reduction import rebuild_handleh = pipe.recv()fd = rebuild_handle(h)client_socket = socket.fromfd(fd, socket.AF_INET, socket.SOCK_STREAM)client_socket.send("hello from the worker processrn") Questions?PyCon 2012 Follow me on Twitter: @Crad
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×