Python concurrency: libraries overview
Upcoming SlideShare
Loading in...5
×
 

Python concurrency: libraries overview

on

  • 2,791 views

 

Statistics

Views

Total Views
2,791
Views on SlideShare
2,783
Embed Views
8

Actions

Likes
4
Downloads
85
Comments
0

1 Embed 8

http://www.slideshare.net 8

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

CC Attribution-NonCommercial-ShareAlike LicenseCC Attribution-NonCommercial-ShareAlike LicenseCC Attribution-NonCommercial-ShareAlike License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Python concurrency: libraries overview Python concurrency: libraries overview Presentation Transcript

  • Built-in solutions 3-rd Party Solutions Summary Concurrency in Python Overview of current solutions Andrii V. Mishkovskyi December 28, 2009 Andrii V. Mishkovskyi Concurrency in Python
  • Built-in solutions Based on system threads 3-rd Party Solutions Process-based Summary Coroutines threading module Kernel threads Similar to JVM’s threading module Lots of synchronization primitives Andrii V. Mishkovskyi Concurrency in Python
  • Built-in solutions Based on system threads 3-rd Party Solutions Process-based Summary Coroutines Global Interpreter Lock Simple locking mechanism Doesn’t allow execution of more than 1 thread at a time Unless it’s IO-bound thread CPU-bound threads don’t benefit from parallel execution Andrii V. Mishkovskyi Concurrency in Python
  • Built-in solutions Based on system threads 3-rd Party Solutions Process-based Summary Coroutines multiprocessing module API similar to multithreading Operates on process level Nice way of avoiding GIL issues Andrii V. Mishkovskyi Concurrency in Python
  • Built-in solutions Based on system threads 3-rd Party Solutions Process-based Summary Coroutines PEP 342 Simple support through “enhanced generators” yield expression No built-in library or framework Andrii V. Mishkovskyi Concurrency in Python
  • Twisted Built-in solutions Stackless Python 3-rd Party Solutions Kamaelia Summary cogen Others Twisted Emacs of Python concurrency Tons of unit tests Years of active development Supports almost every protocol there is Andrii V. Mishkovskyi Concurrency in Python
  • Twisted example from twisted.internet import reactor, protocol class Echo(protocol.Protocol): def dataReceived(self, data): self.transport.write(data) factory = protocol.ServerFactory() factory.protocol = Echo reactor.listenTCP(8000, factory) reactor.run()
  • Twisted example from twisted.internet import reactor, protocol class Echo(protocol.Protocol): def dataReceived(self, data): self.transport.write(data) factory = protocol.ServerFactory() factory.protocol = Echo reactor.listenTCP(8000, factory) reactor.run()
  • Twisted example from twisted.internet import reactor, protocol class Echo(protocol.Protocol): def dataReceived(self, data): self.transport.write(data) factory = protocol.ServerFactory() factory.protocol = Echo reactor.listenTCP(8000, factory) reactor.run()
  • Twisted example from twisted.internet import reactor, protocol class Echo(protocol.Protocol): def dataReceived(self, data): self.transport.write(data) factory = protocol.ServerFactory() factory.protocol = Echo reactor.listenTCP(8000, factory) reactor.run()
  • Twisted example from twisted.internet import reactor, protocol class Echo(protocol.Protocol): def dataReceived(self, data): self.transport.write(data) factory = protocol.ServerFactory() factory.protocol = Echo reactor.listenTCP(8000, factory) reactor.run()
  • Twisted Built-in solutions Stackless Python 3-rd Party Solutions Kamaelia Summary cogen Others Stackless Python Patch to CPython to not use C stack Ability to monkeypatch existing codebase with “stackless sockets” 100% compatible with CPython Andrii V. Mishkovskyi Concurrency in Python
  • Stackless Example, part I import stackless import stacklesssocket stacklesssocket.install() import socket class Server(object): def __init__(self, conn): self.serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) self.serversocket.setsockopt( socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) self.serversocket.bind(conn) self.serversocket.listen(5) stackless.tasklet(self.accept)()
  • Stackless Example, part I import stackless import stacklesssocket stacklesssocket.install() import socket class Server(object): def __init__(self, conn): self.serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) self.serversocket.setsockopt( socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) self.serversocket.bind(conn) self.serversocket.listen(5) stackless.tasklet(self.accept)()
  • Stackless Example, part I import stackless import stacklesssocket stacklesssocket.install() import socket class Server(object): def __init__(self, conn): self.serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) self.serversocket.setsockopt( socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) self.serversocket.bind(conn) self.serversocket.listen(5) stackless.tasklet(self.accept)()
  • Stackless Example, part II def accept(self): while self.serversocket.accept: (clientsocket, address) = self.serversocket.accept() stackless.tasklet( self.manage)(clientsocket, address) stackless.schedule() def manage(self, clientsocket, address): clientsocket.send( cliensocket.recv(4096)) s = Server((’0.0.0.0’, 8000)) stackless.run()
  • Stackless Example, part II def accept(self): while self.serversocket.accept: (clientsocket, address) = self.serversocket.accept() stackless.tasklet( self.manage)(clientsocket, address) stackless.schedule() def manage(self, clientsocket, address): clientsocket.send( cliensocket.recv(4096)) s = Server((’0.0.0.0’, 8000)) stackless.run()
  • Stackless Example, part II def accept(self): while self.serversocket.accept: (clientsocket, address) = self.serversocket.accept() stackless.tasklet( self.manage)(clientsocket, address) stackless.schedule() def manage(self, clientsocket, address): clientsocket.send( cliensocket.recv(4096)) s = Server((’0.0.0.0’, 8000)) stackless.run()
  • Twisted Built-in solutions Stackless Python 3-rd Party Solutions Kamaelia Summary cogen Others Kamaelia Developed by BBC Research Follows UNIX principles Components all the way round Andrii V. Mishkovskyi Concurrency in Python
  • Kamaelia Example import Axon from Kamaelia.Chassis.ConnectedServer import SimpleServer class Echo(Axon.Component.component): def main(self): while 1: while self.dataReady("inbox"): self.send(self.recv("inbox"), "outbox") yield 1 SimpleServer(protocol=Echo, port=1500).run()
  • Kamaelia Example import Axon from Kamaelia.Chassis.ConnectedServer import SimpleServer class Echo(Axon.Component.component): def main(self): while 1: while self.dataReady("inbox"): self.send(self.recv("inbox"), "outbox") yield 1 SimpleServer(protocol=Echo, port=1500).run()
  • Kamaelia Example import Axon from Kamaelia.Chassis.ConnectedServer import SimpleServer class Echo(Axon.Component.component): def main(self): while 1: while self.dataReady("inbox"): self.send(self.recv("inbox"), "outbox") yield 1 SimpleServer(protocol=Echo, port=1500).run()
  • Kamaelia Example import Axon from Kamaelia.Chassis.ConnectedServer import SimpleServer class Echo(Axon.Component.component): def main(self): while 1: while self.dataReady("inbox"): self.send(self.recv("inbox"), "outbox") yield 1 SimpleServer(protocol=Echo, port=1500).run()
  • Twisted Built-in solutions Stackless Python 3-rd Party Solutions Kamaelia Summary cogen Others cogen Coroutine-based Extensible scheduler Quite low-level (if compared to Kamaelia) Andrii V. Mishkovskyi Concurrency in Python
  • cogen example, part I from cogen.core import sockets from cogen.core import schedulers from cogen.core.coroutines import coroutine @coroutine def server(): srv = sockets.Socket() srv.bind((0.0.0.0’, 8000)) srv.listen(10) while True: conn, addr = yield srv.accept() m.add(handler, args=(conn, addr))
  • cogen example, part I from cogen.core import sockets from cogen.core import schedulers from cogen.core.coroutines import coroutine @coroutine def server(): srv = sockets.Socket() srv.bind((0.0.0.0’, 8000)) srv.listen(10) while True: conn, addr = yield srv.accept() m.add(handler, args=(conn, addr))
  • cogen example, part I from cogen.core import sockets from cogen.core import schedulers from cogen.core.coroutines import coroutine @coroutine def server(): srv = sockets.Socket() srv.bind((0.0.0.0’, 8000)) srv.listen(10) while True: conn, addr = yield srv.accept() m.add(handler, args=(conn, addr))
  • cogen example, part I from cogen.core import sockets from cogen.core import schedulers from cogen.core.coroutines import coroutine @coroutine def server(): srv = sockets.Socket() srv.bind((0.0.0.0’, 8000)) srv.listen(10) while True: conn, addr = yield srv.accept() m.add(handler, args=(conn, addr))
  • cogen example, part II @coroutine def handler(sock, addr): data = yield sock.read() yield sock.write(data) sock.close() return m = schedulers.Scheduler() m.add(server) m.run()
  • cogen example, part II @coroutine def handler(sock, addr): data = yield sock.read() yield sock.write(data) sock.close() return m = schedulers.Scheduler() m.add(server) m.run()
  • cogen example, part II @coroutine def handler(sock, addr): data = yield sock.read() yield sock.write(data) sock.close() return m = schedulers.Scheduler() m.add(server) m.run()
  • cogen example, part II @coroutine def handler(sock, addr): data = yield sock.read() yield sock.write(data) sock.close() return m = schedulers.Scheduler() m.add(server) m.run()
  • Twisted Built-in solutions Stackless Python 3-rd Party Solutions Kamaelia Summary cogen Others Others Concurrence – nice asynchrounous framework with async MySQL driver eventlet – based on coroutines, developed and used by “Second Life” py.execnet – distribute tasks over network Andrii V. Mishkovskyi Concurrency in Python
  • Built-in solutions 3-rd Party Solutions Summary GIL doesn’t harm 90% of the existing code But Twisted, Kamaelia and others are often cleaner Python is not Erlang, it’s better ;) Andrii V. Mishkovskyi Concurrency in Python