Building a cutting-edge data processing environment on a budget

11,067 views

Published on

As a penniless academic I wanted to do "big data" for science. Open source, Python, and simple patterns were the way forward. Staying on top of todays growing datasets is an arm race. Data analytics machinery —clusters, NOSQL, visualization, Hadoop, machine learning, ...— can spread a team's resources thin. Focusing on simple patterns, lightweight technologies, and a good understanding of the applications gets us most of the way for a fraction of the cost.

I will present a personal perspective on ten years of scientific data processing with Python. What are the emerging patterns in data processing? How can modern data-mining ideas be used without a big engineering team? What constraints and design trade-offs govern software projects like scikit-learn, Mayavi, or joblib? How can we make the most out of distributed hardware with simple framework-less code?

Published in: Technology

Building a cutting-edge data processing environment on a budget

  1. 1. Building a cutting-edge data processing environment on a budget Ga¨l Varoquaux e This talk is not about rocket science!
  2. 2. Building a cutting-edge data processing environment on a budget Ga¨l Varoquaux e Disclaimer: this talk is as much about people and projects as it is about code and algorithms.
  3. 3. Growing up as a penniless academic I did a PhD in quantum physics
  4. 4. Growing up as a penniless academic I did a PhD in quantum physics Vacuum (leaks) Electronics (shorts) Lasers (mis-alignment) Best training ever for agile project management
  5. 5. Growing up as a penniless academic I did a PhD in quantum physics Vacuum (leaks) Electronics (shorts) Lasers (mis-alignment) Computers were only one of the many moving parts Matlab Instrument control
  6. 6. Growing up as a penniless academic I did a PhD in quantum physics Vacuum (leaks) Electronics (shorts) Lasers (mis-alignment) Shaped my vision of computing as a means to an end Computers were only one of the many moving parts Matlab Instrument control
  7. 7. Growing up as a penniless academic 2011 Tenured researcher in computer science
  8. 8. Growing up as a penniless academic 2011 Today Tenured researcher in computer science Growing team with data science rock stars
  9. 9. 1 Using machine learning to understand brain function Link neural activity to thoughts and cognition G Varoquaux 6
  10. 10. 1 Functional MRI t Recordings of brain activity G Varoquaux 7
  11. 11. 1 Cognitive NeuroImaging Learn a bilateral link between brain activity and cognitive function G Varoquaux 8
  12. 12. 1 Encoding models of stimuli Predicting neural response ñ a window into brain representations of stimuli “feature engineering” a description of the world G Varoquaux 9
  13. 13. 1 Decoding brain activity “brain reading” G Varoquaux 10
  14. 14. 1 Data processing feats Visual image reconstruction from human brain activity [Miyawaki, et al. (2008)] “brain reading” G Varoquaux 11
  15. 15. 1 Data processing feats Visual image reconstruction from human brain activity [Miyawaki, et al. (2008)] “if it’s not open and verifiable by others, it’s not science, or engineering...” Stodden, 2010 G Varoquaux 11
  16. 16. 1 Data processing feats Visual image reconstruction from human brain activity [Miyawaki, et al. (2008)] Make it work, make it right, make it boring
  17. 17. 1 Data processing feats Visual image reconstruction from human brain activity [Miyawaki, et al. (2008)] Make it work, make it right, make it boring http://nilearn.github.io/auto examples/ plot miyawaki reconstruction.html Code, data, ... just worksTM G Varoquaux http://nilearn.github.io ni 11
  18. 18. 1 Data processing feats Visual image reconstruction from human brain activity [Miyawaki, et al. (2008)] Make it work, make it right, make it boring ge len al ch nt me p elo ev ed arhttp://nilearn.github.io/auto examples/ ftw plot miyawaki reconstruction.html So Code, data, ... just worksTM G Varoquaux http://nilearn.github.io ni 11
  19. 19. 1 Data accumulation When data processing is routine... “big data” for rich models of brain function Accumulation of scientific knowledge and learning formal representations G Varoquaux 12
  20. 20. 1 Data accumulation When data processing is routine... “big data” for rich models of brain function “A theory is a good theory if it satisfies two requirements: It must accurately describe a large class of observations on the basis of a model that contains only a few arbitrary elements, and it must make definite predictions about the results of future observations.” Stephen Hawking, A Brief History of Time. Accumulation of scientific knowledge and learning formal representations G Varoquaux 12
  21. 21. 1 Petty day-to-day technicalities Buggy code Slow code Lead data scientist leaves New intern to train I don’t understand the code I have written a year ago G Varoquaux 13
  22. 22. 1 Petty day-to-day technicalities Buggy code A lab is no different from a startup Slow code Difficulties Risks LeadRecruitment leaves data scientist Bus factor Technical dept New Limited resources intern to train (people & hardware) I don’t understand the code I have written a year ago G Varoquaux 13
  23. 23. 1 Petty day-to-day technicalities Buggy code A lab is no different from a startup Slow code Difficulties Risks LeadRecruitment leaves data scientist Bus factor Technical dept New Limited resources intern to train (people & hardware) I don’t understand the code I have written a year ago Our mission is to revolutionize brain data processing on a tight budget G Varoquaux 13
  24. 24. 2 Patterns in data processing G Varoquaux 14
  25. 25. 2 The data processing workflow agile Interaction... Ñ script... Ñ module... ý interaction again... Consolidation, progressively Low tech and short turn-around times G Varoquaux 15
  26. 26. Paradigm shift as the dimensionality of data grows y 2 From statistics to statistical learning # features, not only # samples From parameter inference to prediction x Statistical learning is spreading everywhere G Varoquaux 16
  27. 27. 3 Let’s just make software to solve all these problems. G Varoquaux c Theodore W. Gray 17
  28. 28. 3 Design philosophy 1. Don’t solve hard problems The original problem can be bent. 2. Easy setup, works out of the box Installing software sucks. Convention over configuration. 3. Fail gracefully Robust to errors. Easy to debug. 4. Quality, quality, quality What’s not excellent won’t be used. G Varoquaux 18
  29. 29. 3 Design philosophy 1. Don’t solve hard problems The original problem can be bent. 2. Easy setup, works out of the box Installing software sucks. Not “one software to rule them all” Convention over configuration. Break down projects by expertise 3. Fail gracefully Robust to errors. Easy to debug. 4. Quality, quality, quality What’s not excellent won’t be used. G Varoquaux 18
  30. 30. G Varoquaux 19
  31. 31. Vision Machine learning without learning the machinery Black box that can be opened Right trade-off between ”just works” and versatility (think Apple vs Linux) G Varoquaux 19
  32. 32. Vision Machine learning without learning the machinery Black box that can be opened Right trade-off between ”just works” and versatility (think Apple vs Linux) We’re not going to solve all the problems for you I don’t solve hard problems Feature-engineering, domain-specific cases... Python is a programming language. Use it. Cover all the 80% usecases in one package G Varoquaux 19
  33. 33. 3 Performance in high-level programming High-level programming is what keeps us alive and kicking G Varoquaux 20
  34. 34. 3 Performance in high-level programming The secret sauce Optimize algorithmes, not for loops Know perfectly Numpy and Scipy - Significant data should be arrays/memoryviews - Avoid memory copies, rely on blas/lapack line-profiler/memory-profiler scipy-lectures.github.io Cython G Varoquaux not C/C++ 20
  35. 35. 3 Performance in high-level programming Hierarchical clustering PR #2199 The secret sauce 1.Optimize algorithmes,clustersloops Take the 2 closest not for 2. Merge them 3. Update the distance matrix Know perfectly Numpy and Scipy ... - Significant data should be arrays/memoryviews Faster with constraints: sparse distance matrix - Avoid memory copies, rely on blas/lapack - Keep a heap queue of distances: cheap minimum line-profiler/memory-profiler - Need sparse growable structure for neighborhoods scipy-lectures.github.io skip-list in Cython! Oplog nq insert, remove, access Cython not C/C++ bind C++ map[int, float] with Cython Fast traversal, possibly in Cython, for step 3. G Varoquaux 20
  36. 36. 3 Performance in high-level programming Hierarchical clustering PR #2199 The secret sauce 1.Optimize algorithmes,clustersloops Take the 2 closest not for 2. Merge them 3. Update the distance matrix Know perfectly Numpy and Scipy ... - Significant data should be arrays/memoryviews Faster with constraints: sparse distance matrix - Avoid memory copies, rely on blas/lapack - Keep a heap queue of distances: cheap minimum line-profiler/memory-profiler - Need sparse growable structure for neighborhoods scipy-lectures.github.io skip-list in Cython! Oplog nq insert, remove, access Cython not C/C++ bind C++ map[int, float] with Cython Fast traversal, possibly in Cython, for step 3. G Varoquaux 20
  37. 37. 0 3 0 0 38 01 7 87 9 1 78 4 5 9 40 7990 8779 4 1 5 49 0771 0775 9447 13 6 97 17 52 79 7 3 70 74 27 97 4 4 7 47 6553 0771 4661 7001 7992 48 7 75 34 18 12 15 27 9 8 54 49 87 24 57 7 0 7 9 03 7221 4226 9004 7117 4779 788 78 9 34 15 65 49 78 97 8 5 45 54 53 95 88 7 9 7 56 46 35 51 87 5 1 67 63 58 19 7 7 8 73 34 80 90 1 7 32 49 09 0 8 7 24 90 98 7 4 45 08 8 7 5 56 84 4 6 61 4 5 2 14 60 4 0 2 3 0 0 0 38 01 7 87 9 1 78 94 0 79 87 5 4 90 79 4 1 5 49 0771 0775 9447 3 7 1 36 9770 1774 5227 7997 4 65 07 46 70 79 4 7 47 53 71 61 01 92 8 9 4 87 7554 3449 1887 1224 1557 277 90 3 72 42 90 71 47 78 7 0 21 26 04 17 79 8 78 9 34 15 65 49 78 97 8 5 45 54 53 95 88 7 9 7 56 46 35 51 87 5 1 67 63 58 19 7 7 8 73 34 80 90 0 1 7 32 49 09 0 3 8 7 24 90 98 0 0 38 7 4 45 08 8 01 7 87 6 84 75 5 9 1 78 4 6 61 4 4 5 2 14 5 9 40 7990 8779 60 4 4 1 5 49 0771 0775 9447 2 0 13 6 97 17 52 79 7 3 70 74 27 97 4 4 7 47 6553 0771 4661 7001 7992 48 7 75 34 18 12 15 27 9 8 54 49 87 24 57 7 0 7 9 03 7221 4226 9004 7117 4779 788 78 9 34 15 65 49 78 97 8 5 45 54 53 95 88 7 9 7 56 46 35 51 87 5 1 67 63 58 19 7 7 8 73 34 80 90 1 7 32 49 09 0 8 7 24 90 98 7 4 45 08 8 7 5 56 84 4 6 61 4 5 2 14 60 4 2 0 3 Architecture of a data-manipulation toolkit Separate data from operations, but keep an imperative-like language bokeh, chaco, hadoop, Mayavi, CPUs G Varoquaux 21
  38. 38. 3 Architecture of a data-manipulation toolkit Separate data from operations, but keep an imperative-like language Object API exposes a data-processing language fit, predict, transform, score, partial fit Instantiated without data but with all the parameters Objects pipeline, merging, etc... G Varoquaux 21
  39. 39. 3 Architecture of a data-manipulation toolkit Separate data from operations, but keep an imperative-like language Object API exposes a data-processing language fit, predict, transform, score, partial fit Instantiated without data but with all the parameters Objects pipeline, merging, etc... configuration/run pattern curry in functional programming Ideas from MVC pattern G Varoquaux traits, pyre functools.partial 21
  40. 40. 4 Big data on small hardware G Varoquaux 22
  41. 41. h h isdata on smallishardware ll g 4 Big a Big sm “Big data”: Petabytes... Distributed storage Computing cluster G Varoquaux Mere mortals: Gigabytes... Python programming Off-the-self computers 22
  42. 42. 4 On-line algorithms Process the data one sample at a time Compute the mean of a gazillion numbers Hard? G Varoquaux 23
  43. 43. 4 On-line algorithms Process the data one sample at a time Compute the mean of a gazillion numbers Hard? No: just do a running mean G Varoquaux 23
  44. 44. 4 On-line algorithms Converges to expectations Mini-batch = bunch observations for vectorization Example: K-Means clustering X = np.random.normal(size=(10 000, 200)) scipy.cluster.vq. sklearn.cluster. MiniBatchKMeans(n clusters=10, kmeans(X, 10, n init=2).fit(X) iter=2) 11.33 s 0.62 s G Varoquaux 23
  45. 45. 4 On-the-fly data reduction Big data is often I/O bound Layer memory access CPU caches RAM Local disks Distant storage Less data also means less work G Varoquaux 24
  46. 46. 4 On-the-fly data reduction Dropping data 1 loop: take a random fraction of the data 2 run algorithm on that fraction 3 aggregate results across sub-samplings Looks like bagging: bootstrap aggregation Exploits redundancy across observations Run the loop in parallel G Varoquaux 24
  47. 47. 4 On-the-fly data reduction Random projections (will average features) sklearn.random projection random linear combinations of the features Fast clustering of features sklearn.cluster.WardAgglomeration on images: super-pixel strategy Hashing when observations have varying size (e.g. words) sklearn.feature extraction.text. HashingVectorizer stateless: can be used in parallel G Varoquaux 24
  48. 48. 4 On-the-fly data reduction Example: randomized SVD Random projection sklearn.utils.extmath.randomized svd X = np.random.normal(size=(50000, 200)) %timeit lapack = linalg.svd(X, full matrices=False) 1 loops, best of 3: 6.09 s per loop %timeit arpack=splinalg.svds(X, 10) 1 loops, best of 3: 2.49 s per loop %timeit randomized = randomized svd(X, 10) 1 loops, best of 3: 303 ms per loop linalg.norm(lapack[0][:, :10] - arpack[0]) / 2000 0.0022360679774997738 linalg.norm(lapack[0][:, :10] - randomized[0]) / 2000 0.0022121161221386925 G Varoquaux 24
  49. 49. 4 Biggish iron Our new box: 48 cores 384G RAM 70T storage 15 ke (SSD cache on RAID controller) Gets our work done faster than our 800 CPU cluster It’s the access patterns! “Nobody ever got fired for using Hadoop on a cluster” A. Rowstron et al., HotCDP ’12 G Varoquaux 25
  50. 50. 5 Avoiding the framework joblib G Varoquaux 26
  51. 51. 5 Parallel processing big picture Focus on embarassingly parallel for loops Life is too short to worry about deadlocks Workers compete for data access Memory bus is a bottleneck The right grain of parallelism Too fine ñ overhead Too coarse ñ memory shortage Scale by the relevant cache pool G Varoquaux 27
  52. 52. 5 Parallel processing joblib Focus on embarassingly parallel for loops Life is too short to worry about deadlocks >>> from joblib import Parallel, delayed >>> Parallel(n jobs=2)(delayed(sqrt)(i**2) ... for i in range(8)) [0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0] G Varoquaux 27
  53. 53. 5 Parallel processing joblib IPython, multiprocessing, celery, MPI? joblib is higher-level No dependencies, works everywhere Better traceback reporting Memmaping arrays to share memory (O. Grisel) On-the-fly dispatch of jobs – memory-friendly Threads or processes backend G Varoquaux 27
  54. 54. 5 Parallel processing joblib IPython, multiprocessing, celery, MPI? joblib is higher-level No dependencies, works everywhere Better traceback reporting Memmaping arrays to share memory (O. Grisel) On-the-fly dispatch of jobs – memory-friendly Threads or processes backend G Varoquaux 27
  55. 55. 5 Parallel processing Queues Queues: high-performance, concurrent-friendly Difficulty: callback on result arrival ñ multiple threads in caller ` risk of deadlocks Dispatch queue should fill up “slowly” ñ pre dispatch in joblib ñ Back and forth communication Door open to race conditions G Varoquaux 28
  56. 56. 5 Parallel processing: what happens where joblib design: Caller, dispatch queue, and collect queue in same process Benefit: robustness Grand-central dispatch design: dispatch queue has a process of its own Benefit: resource managment in nested for loops G Varoquaux 29
  57. 57. 5 Caching For reproducibility: avoid manually chained scripts (make-like usage) For performance: avoiding re-computing is the crux of optimization G Varoquaux 30
  58. 58. 5 Caching The joblib approach For reproducibility: avoid manually chained scripts (make-like usage) For performance: avoiding re-computing is the crux of optimization Memoize pattern mem = joblib.Memory(cachedir=’.’) g = mem.cache(f) b = g(a) # computes a using f c = g(a) # retrieves results from store G Varoquaux 30
  59. 59. 5 Caching The joblib approach Challenges in the context of big data For reproducibility: avoid b are big chained scripts (make-like usage) a & manually For performance: Design goals avoiding re-computing is the crux of optimization a & b arbitrary Python objects No dependencies Drop-in, framework-less code Memoize pattern mem = joblib.Memory(cachedir=’.’) g = mem.cache(f) b = g(a) # computes a using f c = g(a) # retrieves results from store G Varoquaux 30
  60. 60. 5 Caching The joblib approach For bricks for out-of-core algorithms coming soon Lego reproducibility: avoid manually chained scripts ąąą result = g.call and shelve(a)(make-like usage) For performance: ąąą result avoiding re-computing is the crux argument hash=”...”) MemorizedResult(cachedir=”...”, func=”g...”,of optimization ąąą c = result.get() Memoize pattern mem = joblib.Memory(cachedir=’.’) g = mem.cache(f) b = g(a) # computes a using f c = g(a) # retrieves results from store G Varoquaux 30
  61. 61. 5 Efficient input argument hashing – joblib.hash Compute md5‹ of input arguments Trade-off between features and cost Black boxy Robust and completely generic G Varoquaux 31
  62. 62. 5 Efficient input argument hashing – joblib.hash Compute md5‹ of input arguments Implementation 1. Create an md5 hash object 2. Subclass the standard-library pickler = state machine that walks the object graph 3. Walk the object graph: - ndarrays: pass data pointer to md5 algorithm (“update” method) - the rest: pickle 4. Update the md5 with the pickle ‹ md5 is in the Python standard library G Varoquaux 31
  63. 63. 5 Fast, disk-based, concurrent, store – joblib.dump Persisting arbritrary objects Once again sub-class the pickler Use .npy for large numpy arrays (np.save), pickle for the rest ñ Multiple files Store concurrency issues Strategy: atomic operations ` try/except Renaming a directory is atomic Directory layout consistent with remove operations Good performance, usable on shared disks (cluster) G Varoquaux 32
  64. 64. 5 Making I/O fast Fast compression CPU may be faster than disk access in particular in parallel Standard library: zlib.compress with buffers (bypass gzip module to work online + in-memory) G Varoquaux 33
  65. 65. 5 Making I/O fast Fast compression CPU may be faster than disk access in particular in parallel Standard library: zlib.compress with buffers (bypass gzip module to work online + in-memory) Avoiding copies zlib.compress: C-contiguous buffers Copyless storage of raw buffer + meta-information (strides, class...) G Varoquaux 33
  66. 66. 5 Making I/O fast Fast compression CPU may be faster than disk access in particular in parallel Standard library: zlib.compress with buffers (bypass gzip module to work online + in-memory) Avoiding copies zlib.compress: C-contiguous buffers Copyless storage of raw buffer + meta-information (strides, class...) Single file dump coming soon File opening is slow on cluster Challenge: streaming the above for memory usage G Varoquaux 33
  67. 67. 5 Making I/O fast Fast compression CPU may be faster than disk access in particular in parallel StandardWhat matters on large with buffers library: zlib.compress systems (bypass gzip module to stored Numbers of bytes work online + in-memory) brings network/SATA bus down Avoiding copies zlib.compress: C-contiguous buffers Memory usage Copyless storage brings buffer of raw compute nodes down + meta-information (strides, class...) Number of atomic file access Single file dump brings shared storage down soon coming File opening is slow on cluster Challenge: streaming the above for memory usage G Varoquaux 33
  68. 68. y axis scale: 1 is np.save 5 Benchmarking to np.save and pytables G Varoquaux NeuroImaging data (MNI atlas) 34
  69. 69. 6 The bigger picture: building an ecosystem Helping your future self G Varoquaux 35
  70. 70. 6 Community-based development in scikit-learn Huge feature set: benefits of a large team Project growth: More than 200 contributors „ 12 core contributors 1 full-time INRIA programmer from the start Estimated cost of development: $ 6 millions COCOMO model, http://www.ohloh.net/p/scikit-learn G Varoquaux 36
  71. 71. 6 The economics of open source Code maintenance too expensive to be alone scikit-learn „ 300 email/month nipy „ 45 email/month joblib „ 45 email/month mayavi „ 30 email/month “Hey Gael, I take it you’re too busy. That’s okay, I spent a day trying to install XXX and I think I’ll succeed myself. Next time though please don’t ignore my emails, I really don’t like it. You can say, ‘sorry, I have no time to help you.’ Just don’t ignore.” G Varoquaux 37
  72. 72. 6 The economics of open source Code maintenance too expensive to be alone scikit-learn „ 300 email/month nipy „ 45 email/month joblib „ 45 email/month mayavi „ 30 email/month Your “benefits” come from a fraction of the code Data loading? Maybe? Standard algorithms? Nah Share the common code... ...to avoid dying under code Code becomes less precious with time And somebody might contribute features G Varoquaux 37
  73. 73. 6 Many eyes makes code fast Bench WiseRF anybody? L. Buitinck, O. Grisel, A. Joly, G. Louppe, J. Nothman, P. Prettenhofer G Varoquaux 38
  74. 74. 6 6 steps to a community-driven project 1 Focus on quality 2 Build great docs and examples 3 Use github 4 Limit the technicality of your codebase 5 Releasing and packaging matter 6 Focus on your contributors, give them credit, decision power http://www.slideshare.net/GaelVaroquaux/ scikit-learn-dveloppement-communautaire G Varoquaux 39
  75. 75. 6 Core project contributors Number of commits Normalized number of commits since 2009-06 Individual committer G Varoquaux Credit: Fernando Perez, Gist 5843625 40
  76. 76. 6 The tragedy of the commons Individuals, acting independently and rationally according to each one’s self-interest, behave contrary to the whole group’s long-term best interests by depleting some common resource. Wikipedia Make it work, make it right, make it boring Core projects (boring) taken for granted ñ Hard to fund, less excitement They need citation, in papers & on corporate web pages G Varoquaux 41
  77. 77. Solving problems that matter The 80/20 rule 80% of the usecases can be solved with 20% of the lines of code scikit-learn, joblib, nilearn, ... @GaelVaroquaux I hope
  78. 78. Cutting-edge ... environment ... on a budget 1 Set the goals right Don’t solve hard problems What’s your original problem? @GaelVaroquaux
  79. 79. Cutting-edge ... environment ... on a budget 1 Set the goals right 2 Use the simplest technological solutions possible Be very technically sophisticated Don’t use that sophistication @GaelVaroquaux
  80. 80. Cutting-edge ... environment ... on a budget 1 Set the goals right 2 Use the simplest technological solutions possible 3 Don’t forget the human factors With your users (documentation) With your contributors @GaelVaroquaux
  81. 81. Cutting-edge ... environment ... on a budget 1 Set the goals right 2 Use the simplest technological solutions possible 3 Don’t forget the human factors A perfect design? @GaelVaroquaux

×