Python Multiprocessing Spoon-fed - Blue Raster Esri Developer Summit 2013 Lightning Talk

1,485 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,485
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
13
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Python Multiprocessing Spoon-fed - Blue Raster Esri Developer Summit 2013 Lightning Talk

  1. 1. Flickr - jystewarts photostream Python Multiprocessing Spoon-fedBrendan Collins Blue Raster @blueraster
  2. 2. Flickr - jystewarts photostream Hello Multiprocessing Module!• Python standard library module• Allows for parallel code execution• Easy to use• Nice compliment to arcpy module
  3. 3. Flickr - jystewarts photostream The Multiprocessing Mission:• Fully leverage all CPUs• Increase Scalability• Improve Performance• Alternative to Threading
  4. 4. Flickr - jegcarpios photostreamThreads: Fluffy the three-headed dog
  5. 5. Flickr - jegcarpios photostream Threads: Fluffy the three-headed dog• Each thread has same parent process• Threads share memory• Good for I/O Bound tasks (Scraping websites)• Poor for CPU Bound Tasks (Geoprocessing)• At the mercy of Global Interpreter Lock (GIL)• Problems can affect other threads
  6. 6. Flickr - Blacksmiths photostreamProcesses: Hounds of Resurrection
  7. 7. Flickr - Blacksmiths photostreamProcesses: Hounds of Resurrection• Each process is a separate python instance• Memory space is NOT shared• Easily distributed across all cores• If one process dies, others are not affected• Communication via IPC
  8. 8. Step 1: Basic Setup
  9. 9. Multiprocessing Pitfalls• All worker arguments must be picklable• Schema locks (File Geodatabase…)• Script tools must be run “out-of-process”• Worker startup cost
  10. 10. Flickr - jystewarts photostream In Closing:• Jason Pardy’s Multiprocessing post – blogs.esri.com/esri/arcgis/2011/08/29/multiproce ssing (link)• Follow Me: – @brendancol – bcdcspatial.blogspot.com (link)• Follow Blue Raster: – @blueraster – blueraster.com/blog (link)

×