This document summarizes Gavin M. Roy's presentation on concurrency with multiprocessing in Python. It discusses using threads via the threading module, issues with the Global Interpreter Lock (GIL) in Python, and how to use the multiprocessing module to achieve true parallelism across multiple processes. It provides examples of creating threads and processes that run concurrently and examples of how to share objects between processes using connections, queues, pipes, managers and reduction tools.

1. Concurrency with Multiprocessing in Python
PhillyPug & Philly.rb RedSnake Meeting
February 8th, 2011
Gavin M. Roy
myYearbook.com

2. green threads in threading

3. from threading import Thread
import time
class MyThread(Thread):
def run(self):
print "%s running" % self.name
time.sleep(5)
print "%s done" % self.name
for x in xrange(0,10):
thread = MyThread()
thread.start()
thread.join(0)

4. gmr-0x04:pika gmr$ python threads.py
MyThread-1 running
MyThread-2 running
MyThread-3 running
MyThread-4 running
MyThread-5 running
MyThread-6 running
MyThread-7 running
MyThread-8 running
MyThread-9 running
MyThread-10 running
MyThread-1 done
MyThread-2 done
MyThread-3 done
MyThread-4 done
MyThread-5 done
MyThread-6 done
MyThread-7 done
MyThread-8 done
MyThread-9 done
MyThread-10 done

5. threading
• Locks
• Reentrant Locks
Reentrant knows who
own the locks and the
recursion level
• Conditions
Additional classes like
Queue.Queue
• Semaphores
• Events
• Timers

7. This GIL From David Beeazley’s GIL Visualization
http://www.dabeaz.com/GIL/gilvis/fourthread.html

8. enter multiprocessing

9. multiprocessing module
• All the things threading has • Connections
• Exchanging Objects • Managers TCP Server coordinating
shared objects
• Queues and Pipes • SyncManager Process Sync
• Shared State • Logging
• Pipes and Queues
• Pools

10. from multiprocessing import Process
import time
class MyThread(Process):
def run(self):
print "%s running" % self.name
time.sleep(5)
print "%s done" % self.name
for x in xrange(0,10):
thread = MyThread()
thread.start()
thread.join(0)

11. gmr-0x04:pika gmr$ python processes.py
Thread-1 running
Thread-2 running
Thread-3 running
Thread-4 running
Thread-5 running
Thread-6 running
Thread-7 running
Thread-8 running
Thread-9 running
Thread-10 running
Thread-1 done
Thread-2 done
Thread-3 done
Thread-4 done
Thread-5 done
Thread-6 done
Thread-7 done
Thread-8 done
Thread-9 done
Thread-10 done

12. multiprocessing.reduction
Photo By Susan NYC: http://www.flickr.com/photos/en321/33868864/

13. # Process #1
from multiprocessing.reduction import reduce_handle
import socket
# Create a socket
sock = socket.socket(socket.AF_INET,
socket.SOCK_STREAM, 0)
# Do stuff here, client or server wise
# Create the pickled socket handle
handle = reduce_handle(sock.fileno)
# Process #2
from multiprocessing.reduction import rebuild_handle
import socket
# In other process
fd = rebuild_handle(handle)
sock = socket.fromfd(fd, socket.AF_INET,
socket.SOCK_STREAM)
# Now I can read and write from the socket too