Python concurrency: libraries overview

  • 1,899 views
Uploaded on

 

More in: Technology , Education
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,899
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
86
Comments
0
Likes
4

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Built-in solutions 3-rd Party Solutions Summary Concurrency in Python Overview of current solutions Andrii V. Mishkovskyi December 28, 2009 Andrii V. Mishkovskyi Concurrency in Python
  • 2. Built-in solutions Based on system threads 3-rd Party Solutions Process-based Summary Coroutines threading module Kernel threads Similar to JVM’s threading module Lots of synchronization primitives Andrii V. Mishkovskyi Concurrency in Python
  • 3. Built-in solutions Based on system threads 3-rd Party Solutions Process-based Summary Coroutines Global Interpreter Lock Simple locking mechanism Doesn’t allow execution of more than 1 thread at a time Unless it’s IO-bound thread CPU-bound threads don’t benefit from parallel execution Andrii V. Mishkovskyi Concurrency in Python
  • 4. Built-in solutions Based on system threads 3-rd Party Solutions Process-based Summary Coroutines multiprocessing module API similar to multithreading Operates on process level Nice way of avoiding GIL issues Andrii V. Mishkovskyi Concurrency in Python
  • 5. Built-in solutions Based on system threads 3-rd Party Solutions Process-based Summary Coroutines PEP 342 Simple support through “enhanced generators” yield expression No built-in library or framework Andrii V. Mishkovskyi Concurrency in Python
  • 6. Twisted Built-in solutions Stackless Python 3-rd Party Solutions Kamaelia Summary cogen Others Twisted Emacs of Python concurrency Tons of unit tests Years of active development Supports almost every protocol there is Andrii V. Mishkovskyi Concurrency in Python
  • 7. Twisted example from twisted.internet import reactor, protocol class Echo(protocol.Protocol): def dataReceived(self, data): self.transport.write(data) factory = protocol.ServerFactory() factory.protocol = Echo reactor.listenTCP(8000, factory) reactor.run()
  • 8. Twisted example from twisted.internet import reactor, protocol class Echo(protocol.Protocol): def dataReceived(self, data): self.transport.write(data) factory = protocol.ServerFactory() factory.protocol = Echo reactor.listenTCP(8000, factory) reactor.run()
  • 9. Twisted example from twisted.internet import reactor, protocol class Echo(protocol.Protocol): def dataReceived(self, data): self.transport.write(data) factory = protocol.ServerFactory() factory.protocol = Echo reactor.listenTCP(8000, factory) reactor.run()
  • 10. Twisted example from twisted.internet import reactor, protocol class Echo(protocol.Protocol): def dataReceived(self, data): self.transport.write(data) factory = protocol.ServerFactory() factory.protocol = Echo reactor.listenTCP(8000, factory) reactor.run()
  • 11. Twisted example from twisted.internet import reactor, protocol class Echo(protocol.Protocol): def dataReceived(self, data): self.transport.write(data) factory = protocol.ServerFactory() factory.protocol = Echo reactor.listenTCP(8000, factory) reactor.run()
  • 12. Twisted Built-in solutions Stackless Python 3-rd Party Solutions Kamaelia Summary cogen Others Stackless Python Patch to CPython to not use C stack Ability to monkeypatch existing codebase with “stackless sockets” 100% compatible with CPython Andrii V. Mishkovskyi Concurrency in Python
  • 13. Stackless Example, part I import stackless import stacklesssocket stacklesssocket.install() import socket class Server(object): def __init__(self, conn): self.serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) self.serversocket.setsockopt( socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) self.serversocket.bind(conn) self.serversocket.listen(5) stackless.tasklet(self.accept)()
  • 14. Stackless Example, part I import stackless import stacklesssocket stacklesssocket.install() import socket class Server(object): def __init__(self, conn): self.serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) self.serversocket.setsockopt( socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) self.serversocket.bind(conn) self.serversocket.listen(5) stackless.tasklet(self.accept)()
  • 15. Stackless Example, part I import stackless import stacklesssocket stacklesssocket.install() import socket class Server(object): def __init__(self, conn): self.serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) self.serversocket.setsockopt( socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) self.serversocket.bind(conn) self.serversocket.listen(5) stackless.tasklet(self.accept)()
  • 16. Stackless Example, part II def accept(self): while self.serversocket.accept: (clientsocket, address) = self.serversocket.accept() stackless.tasklet( self.manage)(clientsocket, address) stackless.schedule() def manage(self, clientsocket, address): clientsocket.send( cliensocket.recv(4096)) s = Server((’0.0.0.0’, 8000)) stackless.run()
  • 17. Stackless Example, part II def accept(self): while self.serversocket.accept: (clientsocket, address) = self.serversocket.accept() stackless.tasklet( self.manage)(clientsocket, address) stackless.schedule() def manage(self, clientsocket, address): clientsocket.send( cliensocket.recv(4096)) s = Server((’0.0.0.0’, 8000)) stackless.run()
  • 18. Stackless Example, part II def accept(self): while self.serversocket.accept: (clientsocket, address) = self.serversocket.accept() stackless.tasklet( self.manage)(clientsocket, address) stackless.schedule() def manage(self, clientsocket, address): clientsocket.send( cliensocket.recv(4096)) s = Server((’0.0.0.0’, 8000)) stackless.run()
  • 19. Twisted Built-in solutions Stackless Python 3-rd Party Solutions Kamaelia Summary cogen Others Kamaelia Developed by BBC Research Follows UNIX principles Components all the way round Andrii V. Mishkovskyi Concurrency in Python
  • 20. Kamaelia Example import Axon from Kamaelia.Chassis.ConnectedServer import SimpleServer class Echo(Axon.Component.component): def main(self): while 1: while self.dataReady("inbox"): self.send(self.recv("inbox"), "outbox") yield 1 SimpleServer(protocol=Echo, port=1500).run()
  • 21. Kamaelia Example import Axon from Kamaelia.Chassis.ConnectedServer import SimpleServer class Echo(Axon.Component.component): def main(self): while 1: while self.dataReady("inbox"): self.send(self.recv("inbox"), "outbox") yield 1 SimpleServer(protocol=Echo, port=1500).run()
  • 22. Kamaelia Example import Axon from Kamaelia.Chassis.ConnectedServer import SimpleServer class Echo(Axon.Component.component): def main(self): while 1: while self.dataReady("inbox"): self.send(self.recv("inbox"), "outbox") yield 1 SimpleServer(protocol=Echo, port=1500).run()
  • 23. Kamaelia Example import Axon from Kamaelia.Chassis.ConnectedServer import SimpleServer class Echo(Axon.Component.component): def main(self): while 1: while self.dataReady("inbox"): self.send(self.recv("inbox"), "outbox") yield 1 SimpleServer(protocol=Echo, port=1500).run()
  • 24. Twisted Built-in solutions Stackless Python 3-rd Party Solutions Kamaelia Summary cogen Others cogen Coroutine-based Extensible scheduler Quite low-level (if compared to Kamaelia) Andrii V. Mishkovskyi Concurrency in Python
  • 25. cogen example, part I from cogen.core import sockets from cogen.core import schedulers from cogen.core.coroutines import coroutine @coroutine def server(): srv = sockets.Socket() srv.bind((0.0.0.0’, 8000)) srv.listen(10) while True: conn, addr = yield srv.accept() m.add(handler, args=(conn, addr))
  • 26. cogen example, part I from cogen.core import sockets from cogen.core import schedulers from cogen.core.coroutines import coroutine @coroutine def server(): srv = sockets.Socket() srv.bind((0.0.0.0’, 8000)) srv.listen(10) while True: conn, addr = yield srv.accept() m.add(handler, args=(conn, addr))
  • 27. cogen example, part I from cogen.core import sockets from cogen.core import schedulers from cogen.core.coroutines import coroutine @coroutine def server(): srv = sockets.Socket() srv.bind((0.0.0.0’, 8000)) srv.listen(10) while True: conn, addr = yield srv.accept() m.add(handler, args=(conn, addr))
  • 28. cogen example, part I from cogen.core import sockets from cogen.core import schedulers from cogen.core.coroutines import coroutine @coroutine def server(): srv = sockets.Socket() srv.bind((0.0.0.0’, 8000)) srv.listen(10) while True: conn, addr = yield srv.accept() m.add(handler, args=(conn, addr))
  • 29. cogen example, part II @coroutine def handler(sock, addr): data = yield sock.read() yield sock.write(data) sock.close() return m = schedulers.Scheduler() m.add(server) m.run()
  • 30. cogen example, part II @coroutine def handler(sock, addr): data = yield sock.read() yield sock.write(data) sock.close() return m = schedulers.Scheduler() m.add(server) m.run()
  • 31. cogen example, part II @coroutine def handler(sock, addr): data = yield sock.read() yield sock.write(data) sock.close() return m = schedulers.Scheduler() m.add(server) m.run()
  • 32. cogen example, part II @coroutine def handler(sock, addr): data = yield sock.read() yield sock.write(data) sock.close() return m = schedulers.Scheduler() m.add(server) m.run()
  • 33. Twisted Built-in solutions Stackless Python 3-rd Party Solutions Kamaelia Summary cogen Others Others Concurrence – nice asynchrounous framework with async MySQL driver eventlet – based on coroutines, developed and used by “Second Life” py.execnet – distribute tasks over network Andrii V. Mishkovskyi Concurrency in Python
  • 34. Built-in solutions 3-rd Party Solutions Summary GIL doesn’t harm 90% of the existing code But Twisted, Kamaelia and others are often cleaner Python is not Erlang, it’s better ;) Andrii V. Mishkovskyi Concurrency in Python