Real-Time Python Web: Gevent and Socket.io
Upcoming SlideShare
Loading in...5
×
 

Real-Time Python Web: Gevent and Socket.io

on

  • 18,878 views

 

Statistics

Views

Total Views
18,878
Views on SlideShare
18,549
Embed Views
329

Actions

Likes
31
Downloads
271
Comments
0

13 Embeds 329

http://lanyrd.com 261
https://twitter.com 24
http://harajuku-tech.posterous.com 13
http://note.harajuku-tech.org 10
http://aws.w3db.us 5
http://selebtweet.com 4
http://www.selebtweet.com 3
http://a0.twimg.com 2
http://twitter.com 2
https://www.linkedin.com 2
http://local.remote.me 1
https://si0.twimg.com 1
http://www.techgig.com 1
More...

Accessibility

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

CC Attribution License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Real-Time Python Web: Gevent and Socket.io Real-Time Python Web: Gevent and Socket.io Presentation Transcript

  • Real-Time Web: Gevent and Socket.io Rick Copeland @rick446 [email_address]
    • Getting started with Gevent
    • AJAX, push, WebSockets, wha?
    • ZeroMQ for fun and multiprocessing
    • Putting it all together
  • A (very) Brief Survey of Python Asynchronous Programming
    • AsynCore
      • In stdlib, used for stdlib SMTP server
      • Nobody cares about it anymore 
    • Twisted
      • Large community, vast amounts of code
      • Callbacks hurt my brain
    • Stackless
      • Cool, cooperative multithreading
      • Needs a custom Python
    • Event-based green threads
      • Like stackless, but in regular Python
      • Know when you yield
  • Let’s Go Green: Async that Doesn’t Hurt Your Brain
    • Greenlets: Cooperative, lightweight threads
    • Very forgiving – mutexes rarely needed
    • Use it for IO!
  • Gevent: Greenlets
    • Spawn helpers
      • spawn(my_python_function, *args, **kwargs)
      • Also spawn_later(), spawn_link(), etc.
    • Greenlet class
      • Like threads but cooperative
      • Useful properties: .get(), .join(), .kill(), .link()
    • Timeouts
      • Timeout(seconds, exception).start()
    • Pools: for limiting concurrency, use Pool.spawn
  • Gevent: Communication
    • Event
      • set()
      • clear()
      • wait()
    • Queue
      • Modeled after Queue.Queue
      • .get()
      • .put()
      • __iter__()
      • PriorityQueue, LifoQueue, JoinableQueue
  • Gevent: Networking
    • “ Green” versions of sockets, select(), ssl, and dns
    • Quick and dirty:
    import gevent.monkey gevent .monkey.patch_all()
  • Gevent: Servers
    • Simple callback interface
    • Creates one greenlet per connection (but remember, that’s OK!)
    def handle(socket, address): print 'new connection!’ server = StreamServer( ( '127.0.0.1', 1234), handle) # creates a new server server .start() # start accepting new connections
  • Gevent: WSGI
    • gevent.wsgi
      • Fast (~4k requests/s)
      • No streaming, pipelining, or ssl 
    • gevent.pywsgi:
      • Full featured
      • Slower (“only” 3k requests/s)
    from gevent import pywsgi def hello_world(env, start_response): start_response( '200 OK', [('Content-Type', 'text/html')]) yield '<b>Hello world</b>’ server = pywsgi.WSGIServer( ( '0.0.0.0', 8080), hello_world) server .serve_forever()
    • Getting started with Gevent
    • AJAX, push, WebSockets, wha?
    • ZeroMQ for fun and multiprocessing
    • Putting it all together
  • What is the real-time web?
    • No page refreshes
    • Server push
    • Examples: chat, realtime analytics, …
    • Implementation
      • Flash (eww…)
      • Polling (2x eww…)
      • Long polling (wow – that’s clever :-/ )
      • HTML5 WebSockets ( Super-easy! Awesome! Security vulnerabilities! Immature spec! )
  • SocketIO to the Rescue “ Socket.IO aims to make realtime apps possible in every browser and mobile device, blurring the differences between the different transport mechanisms.”
  • Socket.io Example <script src=&quot;/socket.io/socket.io.js&quot; ></script> <script> var socket = io.connect( 'http://localhost'); socket.on('news', function (data) { console.log(data); socket.emit( 'my other event’, { my : 'data' }); }); </script>
  • gevent_socketio def hello_world(environ, start_response): if not environ[ 'PATH_INFO'] .startswith( '/socket.io'): return serve_file(environ, start_response) socketio = environ[ 'socketio'] while True: socketio .send( 'Hello, world') gevent .sleep( 2)
    • Getting started with Gevent
    • AJAX, push, WebSockets, wha?
    • ZeroMQ for fun and multiprocessing
    • Putting it all together
  • ZeroMQ Overview
    • C library with Python bindings
    • ZMQ “sockets” are message based, delivery is via a dedicated communication thread
    • ZMQ transports (tcp, inproc, unix, multicast)
    • ZMQ socket types
      • REQ/RES
      • PUSH/PULL
      • PUB/SUB
  • pyzmq and gevent_zmq
    • pyzmq works great for threading
    • gevent_zmq is necessary for gevent (single-threaded)
    • Be careful when forking or otherwise using multiprocessing!
      • Gevent has a global “hub” of greenlets
      • ZeroMQ has a global thread & “context” for communication
      • Best to wait till done forking before initializing ZeroMQ
  • ZeroMQ: bind/connect and pub/sub
    • from gevent_zeromq import zmq
    • context = zmq.Context()
    • sock_queue = context.socket(zmq.PUB)
    • sock_queue.bind( 'inproc://chat')
    zmq_sock = context.socket(zmq.SUB) zmq_sock.setsockopt(zmq.SUBSCRIBE, &quot;&quot;) zmq_sock .connect( 'inproc://chat')
    • Getting started with Gevent
    • AJAX, push, WebSockets, wha?
    • ZeroMQ for fun and multiprocessing
    • Putting it all together
  • WebChat: Design Incoming Greenlet ZMQ send Outgoing Greenlet Socket.io ZMQ recv JSON Messages JSON Messages Socket.io
  • WebChat: HTML <h1> Socket.io Chatterbox </h1> <div id=&quot;status&quot; style=&quot;border:1px solid black;&quot; > Disconnected </div> <form> <input id=&quot;input&quot; style=&quot;width: 35em;&quot; > </form> <div id=&quot;data&quot; style=&quot;border:1px solid black;&quot; > </div> <script src=&quot;/js/jquery.min.js&quot; ></script> <script src=&quot;/js/socket.io.js&quot; ></script> <script src=&quot;/js/test.js&quot; ></script>
  • WebChat: Javascript Setup ( function () { // Create and connect socket var socket = new io.Socket( 'localhost'); socket.connect(); // Socket status var $status = $( '#status'); socket.on('connect', function () { $status.html( '<b>Connected: ' + socket.transport.type + '</b>'); }); socket.on('error', function () { $status.html( '<b>Error</b>'); }); socket.on('disconnect', function () { $status.html( '<b>Closed</b>'); });
  • WebChat: Javascript Communication // Send data to the server var $form = $( 'form'); var $input = $( '#input'); $form.bind('submit', function () { socket.send($input.val()); $input.val( ''); return false ; }); // Get data back from the server var $data = $( '#data'); socket.on('message', function (msg) { msg = $.parseJSON(msg) ; var u = msg.u || 'SYSTEM’; $data.prepend($( '<em>' + u + '</em>: ' + msg.m + '<br/>')); }); })();
  • WebChat: Server def chat(environ, start_response): if not environ[ 'PATH_INFO'] .startswith( '/socket.io): return serve_file(environ, start_response) socketio = environ[ 'socketio'] #... handle auth ... zmq_sock = context.socket(zmq.SUB) zmq_sock.setsockopt(zmq.SUBSCRIBE, &quot;&quot;) zmq_sock .connect( 'inproc://chat') greenlets = [ gevent.spawn(incoming, uname, socketio), gevent.spawn(outgoing, zmq_sock, socketio) ] gevent .joinall(greenlets)
  • WebChat: Greenlets def incoming(uname, socketio): while True: for part in socketio .recv(): sock_queue.send(json.dumps( dict( u =uname, m=part))) def outgoing(zmq_sock, socketio): while True: socketio.send(zmq_sock.recv())
  • Get the Code! Socket.io http://socket.io MIT License Chatterbox http://sf.net/u/rick446/pygotham Apache License ZeroMQ http://www.zeromq.org LGPL License Gevent http://gevent.org MIT License
  • Rick Copeland @rick446 [email_address]