Python concurrency: libraries overview

2,200 views
2,142 views

Published on

Published in: Technology, Education
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,200
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
94
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

Python concurrency: libraries overview

  1. 1. Built-in solutions 3-rd Party Solutions Summary Concurrency in Python Overview of current solutions Andrii V. Mishkovskyi December 28, 2009 Andrii V. Mishkovskyi Concurrency in Python
  2. 2. Built-in solutions Based on system threads 3-rd Party Solutions Process-based Summary Coroutines threading module Kernel threads Similar to JVM’s threading module Lots of synchronization primitives Andrii V. Mishkovskyi Concurrency in Python
  3. 3. Built-in solutions Based on system threads 3-rd Party Solutions Process-based Summary Coroutines Global Interpreter Lock Simple locking mechanism Doesn’t allow execution of more than 1 thread at a time Unless it’s IO-bound thread CPU-bound threads don’t benefit from parallel execution Andrii V. Mishkovskyi Concurrency in Python
  4. 4. Built-in solutions Based on system threads 3-rd Party Solutions Process-based Summary Coroutines multiprocessing module API similar to multithreading Operates on process level Nice way of avoiding GIL issues Andrii V. Mishkovskyi Concurrency in Python
  5. 5. Built-in solutions Based on system threads 3-rd Party Solutions Process-based Summary Coroutines PEP 342 Simple support through “enhanced generators” yield expression No built-in library or framework Andrii V. Mishkovskyi Concurrency in Python
  6. 6. Twisted Built-in solutions Stackless Python 3-rd Party Solutions Kamaelia Summary cogen Others Twisted Emacs of Python concurrency Tons of unit tests Years of active development Supports almost every protocol there is Andrii V. Mishkovskyi Concurrency in Python
  7. 7. Twisted example from twisted.internet import reactor, protocol class Echo(protocol.Protocol): def dataReceived(self, data): self.transport.write(data) factory = protocol.ServerFactory() factory.protocol = Echo reactor.listenTCP(8000, factory) reactor.run()
  8. 8. Twisted example from twisted.internet import reactor, protocol class Echo(protocol.Protocol): def dataReceived(self, data): self.transport.write(data) factory = protocol.ServerFactory() factory.protocol = Echo reactor.listenTCP(8000, factory) reactor.run()
  9. 9. Twisted example from twisted.internet import reactor, protocol class Echo(protocol.Protocol): def dataReceived(self, data): self.transport.write(data) factory = protocol.ServerFactory() factory.protocol = Echo reactor.listenTCP(8000, factory) reactor.run()
  10. 10. Twisted example from twisted.internet import reactor, protocol class Echo(protocol.Protocol): def dataReceived(self, data): self.transport.write(data) factory = protocol.ServerFactory() factory.protocol = Echo reactor.listenTCP(8000, factory) reactor.run()
  11. 11. Twisted example from twisted.internet import reactor, protocol class Echo(protocol.Protocol): def dataReceived(self, data): self.transport.write(data) factory = protocol.ServerFactory() factory.protocol = Echo reactor.listenTCP(8000, factory) reactor.run()
  12. 12. Twisted Built-in solutions Stackless Python 3-rd Party Solutions Kamaelia Summary cogen Others Stackless Python Patch to CPython to not use C stack Ability to monkeypatch existing codebase with “stackless sockets” 100% compatible with CPython Andrii V. Mishkovskyi Concurrency in Python
  13. 13. Stackless Example, part I import stackless import stacklesssocket stacklesssocket.install() import socket class Server(object): def __init__(self, conn): self.serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) self.serversocket.setsockopt( socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) self.serversocket.bind(conn) self.serversocket.listen(5) stackless.tasklet(self.accept)()
  14. 14. Stackless Example, part I import stackless import stacklesssocket stacklesssocket.install() import socket class Server(object): def __init__(self, conn): self.serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) self.serversocket.setsockopt( socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) self.serversocket.bind(conn) self.serversocket.listen(5) stackless.tasklet(self.accept)()
  15. 15. Stackless Example, part I import stackless import stacklesssocket stacklesssocket.install() import socket class Server(object): def __init__(self, conn): self.serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM) self.serversocket.setsockopt( socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) self.serversocket.bind(conn) self.serversocket.listen(5) stackless.tasklet(self.accept)()
  16. 16. Stackless Example, part II def accept(self): while self.serversocket.accept: (clientsocket, address) = self.serversocket.accept() stackless.tasklet( self.manage)(clientsocket, address) stackless.schedule() def manage(self, clientsocket, address): clientsocket.send( cliensocket.recv(4096)) s = Server((’0.0.0.0’, 8000)) stackless.run()
  17. 17. Stackless Example, part II def accept(self): while self.serversocket.accept: (clientsocket, address) = self.serversocket.accept() stackless.tasklet( self.manage)(clientsocket, address) stackless.schedule() def manage(self, clientsocket, address): clientsocket.send( cliensocket.recv(4096)) s = Server((’0.0.0.0’, 8000)) stackless.run()
  18. 18. Stackless Example, part II def accept(self): while self.serversocket.accept: (clientsocket, address) = self.serversocket.accept() stackless.tasklet( self.manage)(clientsocket, address) stackless.schedule() def manage(self, clientsocket, address): clientsocket.send( cliensocket.recv(4096)) s = Server((’0.0.0.0’, 8000)) stackless.run()
  19. 19. Twisted Built-in solutions Stackless Python 3-rd Party Solutions Kamaelia Summary cogen Others Kamaelia Developed by BBC Research Follows UNIX principles Components all the way round Andrii V. Mishkovskyi Concurrency in Python
  20. 20. Kamaelia Example import Axon from Kamaelia.Chassis.ConnectedServer import SimpleServer class Echo(Axon.Component.component): def main(self): while 1: while self.dataReady("inbox"): self.send(self.recv("inbox"), "outbox") yield 1 SimpleServer(protocol=Echo, port=1500).run()
  21. 21. Kamaelia Example import Axon from Kamaelia.Chassis.ConnectedServer import SimpleServer class Echo(Axon.Component.component): def main(self): while 1: while self.dataReady("inbox"): self.send(self.recv("inbox"), "outbox") yield 1 SimpleServer(protocol=Echo, port=1500).run()
  22. 22. Kamaelia Example import Axon from Kamaelia.Chassis.ConnectedServer import SimpleServer class Echo(Axon.Component.component): def main(self): while 1: while self.dataReady("inbox"): self.send(self.recv("inbox"), "outbox") yield 1 SimpleServer(protocol=Echo, port=1500).run()
  23. 23. Kamaelia Example import Axon from Kamaelia.Chassis.ConnectedServer import SimpleServer class Echo(Axon.Component.component): def main(self): while 1: while self.dataReady("inbox"): self.send(self.recv("inbox"), "outbox") yield 1 SimpleServer(protocol=Echo, port=1500).run()
  24. 24. Twisted Built-in solutions Stackless Python 3-rd Party Solutions Kamaelia Summary cogen Others cogen Coroutine-based Extensible scheduler Quite low-level (if compared to Kamaelia) Andrii V. Mishkovskyi Concurrency in Python
  25. 25. cogen example, part I from cogen.core import sockets from cogen.core import schedulers from cogen.core.coroutines import coroutine @coroutine def server(): srv = sockets.Socket() srv.bind((0.0.0.0’, 8000)) srv.listen(10) while True: conn, addr = yield srv.accept() m.add(handler, args=(conn, addr))
  26. 26. cogen example, part I from cogen.core import sockets from cogen.core import schedulers from cogen.core.coroutines import coroutine @coroutine def server(): srv = sockets.Socket() srv.bind((0.0.0.0’, 8000)) srv.listen(10) while True: conn, addr = yield srv.accept() m.add(handler, args=(conn, addr))
  27. 27. cogen example, part I from cogen.core import sockets from cogen.core import schedulers from cogen.core.coroutines import coroutine @coroutine def server(): srv = sockets.Socket() srv.bind((0.0.0.0’, 8000)) srv.listen(10) while True: conn, addr = yield srv.accept() m.add(handler, args=(conn, addr))
  28. 28. cogen example, part I from cogen.core import sockets from cogen.core import schedulers from cogen.core.coroutines import coroutine @coroutine def server(): srv = sockets.Socket() srv.bind((0.0.0.0’, 8000)) srv.listen(10) while True: conn, addr = yield srv.accept() m.add(handler, args=(conn, addr))
  29. 29. cogen example, part II @coroutine def handler(sock, addr): data = yield sock.read() yield sock.write(data) sock.close() return m = schedulers.Scheduler() m.add(server) m.run()
  30. 30. cogen example, part II @coroutine def handler(sock, addr): data = yield sock.read() yield sock.write(data) sock.close() return m = schedulers.Scheduler() m.add(server) m.run()
  31. 31. cogen example, part II @coroutine def handler(sock, addr): data = yield sock.read() yield sock.write(data) sock.close() return m = schedulers.Scheduler() m.add(server) m.run()
  32. 32. cogen example, part II @coroutine def handler(sock, addr): data = yield sock.read() yield sock.write(data) sock.close() return m = schedulers.Scheduler() m.add(server) m.run()
  33. 33. Twisted Built-in solutions Stackless Python 3-rd Party Solutions Kamaelia Summary cogen Others Others Concurrence – nice asynchrounous framework with async MySQL driver eventlet – based on coroutines, developed and used by “Second Life” py.execnet – distribute tasks over network Andrii V. Mishkovskyi Concurrency in Python
  34. 34. Built-in solutions 3-rd Party Solutions Summary GIL doesn’t harm 90% of the existing code But Twisted, Kamaelia and others are often cleaner Python is not Erlang, it’s better ;) Andrii V. Mishkovskyi Concurrency in Python

×