Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Open-Source Frameworks for Deep Learning: an Overview

347 views

Published on

The rise of deep learning over the last decade has led to profound changes in the landscape of the machine learning software stack both for research and production. In this talk we will provide a comprehensive overview of the open-source deep learning frameworks landscape with both a theoretical and hands-on approach. After a brief introduction and historical contextualization, we will highlight common features and distinctions of their recent developments. Finally, we will take at deeper look into three of the most used deep learning frameworks today: Caffe, Tensorflow, PyTorch; with practical examples and considerations worth reckoning in the choice of such libraries.

Published in: Data & Analytics
  • Be the first to comment

Open-Source Frameworks for Deep Learning: an Overview

  1. 1. Open-Source Frameworks for Deep Learning: an Overview Vincenzo Lomonaco University of Bologna ML Course - 2018 PhD Student @ University of Bologna Founder of ContinualAI.org vincenzolomonaco.com
  2. 2. About Me • PhD Student @ University of Bologna • Visiting Scholar @ ENSTA ParisTech and @ Purdue University • Phd Students’ Representative of the Department of Computer Science and Engineering • Teaching Assistant of the courses Machine Learning and Computer Architectures • Author and Technical reviewer of the online course Deep Learning with R and book R Deep Learning Essentials • Founder of Continual AI
  3. 3. Continual AI Community http://continualai.org https://continualai.herokuapp.com
  4. 4. Outline 1. Introduction a. From Machine Learning to Deep Learning b. History of Deep Learning Frameworks 2. Open-Source Deep Learning Frameworks a. Main Features and Distinctions b. Caffe, Tensorflow and Pytorch 3. Practical Advices 4. What next?
  5. 5. The Machine Learning Software Stack “Hidden Technical Debt in Machine Learning Systems” by D. Sculley et al.
  6. 6. The Machine Learning Software Stack For Prototyping and R&D ● Scikit-learn (Python), Caret (R), Weka (Java) For Production ● Kubernetes, KubeFlow, Hadoop ● H2O, MLlib, SparkML, FlinkML Setting up a Machine Learning Framework for Production: http://code.hootsuite.com/setting-up-a-machine-learning-framework-for-production/
  7. 7. From Machine Learning to Deep Learning For Deep Learning the line between Prototyping and Production is more blurred ● Caffe / Caffe 2 ● Torch / PyTorch ● Tensorflow ● CNTK, MxNet, Gluon, Chainer, etc.
  8. 8. From Machine Learning to Deep Learning Why ● Prototyping with neural nets on massive datasets needs efficiency ● Limited focus on: 1. (Deep) Neural Networks. 2. Gradient-Based Optimization. 3. Transparent (multi) CPU/GPU acceleration
  9. 9. History of Deep Learning Frameworks 2010 2013 2014 2015 2016 20182017 ...
  10. 10. History of Deep Learning Frameworks
  11. 11. History of Deep Learning Frameworks Andrej Karpathy, 10 mar 2018 : https://twitter.com/karpathy/status/972295865187512320/
  12. 12. Deep Learning Frameworks: 2018 Figures Deep Learning Framework Power Scores 2018 : https://towardsdatascience.com/deep-learning-framework-power-scores-2018-23607ddf297a/
  13. 13. Deep Learning Frameworks: 2018 Figures Deep Learning Framework Power Scores 2018 : https://towardsdatascience.com/deep-learning-framework-power-scores-2018-23607ddf297a/
  14. 14. Outline 1. Introduction a. From Machine Learning to Deep Learning b. History of Deep Learning Frameworks 2. Open-Source Deep Learning Frameworks a. Main Features and Distinctions b. Caffe, Tensorflow and Pytorch 3. Practical Advices 4. What next?
  15. 15. Main Features and Distinctions 1. Static vs Dynamic Graph 2. Differentiation Support 3. Hardware / Platform Support 4. Model / Utils Support and Performance 5. Multi-CPU/GPU support 6. API level 7. License and Community Support Comparison of deep learning software: https://en.wikipedia.org/wiki/Comparison_of_deep_learning_software
  16. 16. Static vs Dynamic Graph Static (define-AND-run) Dynamic (define-BY-run)
  17. 17. Differentiation Support Types of Differentiation 1. Numerical Differentiation 2. Symbolic Differentiation 3. Automatic Differentiation 4. Hard-coded Differentiation “Automatic Differentiation in Machine Learning: a Survey”: http://www.jmlr.org/papers/volume18/17-468/17-468.pdf
  18. 18. Hardware / Platform Support Platform Support ● Windows, Unix (MAC OS, Linux) ● Android OS, iOS, Embedded Systems Hardware Acceleration ● Cuda / OpenCL Support ● OpenMP, MPI Support ● AI Chips
  19. 19. Model / Utils Support and Performance Types of models supported ● FC-NNs, CNNs, RNNs, etc. ● General purpose algebraic functions Utils support ● Pre-trained models Zoo ● Data format and loading ● Monitoring / Visualization tools Performance ● Very different from task to task and model to model, etc.
  20. 20. Multi-CPU / GPU support Types of parallelization ● Data Parallelism ● Model Parallelism Hardware for parallelization ● CPUs ● GPUs ● Distributed clusters “Large Scale Distributed Deep Networks”: http://papers.nips.cc/paper/4687-large-scale-distributed-deep-networks.pdf
  21. 21. API Level Important features ● Usability ● Flexibility ● Expandability ● Easy to Debug
  22. 22. License and Community Support Licence ● Apache 2.0, MIT, BSD, GNU GPL, Freemium, etc. ● Most of them allow also commercial use (but look at patent licence too) Community Support ● Contribution Diversity ● Active Development ● Supportive Q&A ● Number of users
  23. 23. Caffe, Tensorflow and Pytorch ● Static Graph ● Hard-coded Differentiation ● Windows, Unix, Embedded ● Mainly Feedforward models ● Multi-CPU/GPUs ● Prototxt, C++, Python API ● BSD Licence ● Low community support ● Static Graph (with lazy eval) ● Automatic Differentiation ● Windows, Unix, Embedded ● General purpose graph ● Multi-CPU/GPUs ● C++ and (many) Python API ● Apache 2.0 ● High community support ● Dynamic Graph ● Automatic Differentiation ● Windows, Unix, Embedded ● General purpose graph ● Multi-CPU/GPUs ● C++ and Python API ● BSD Licence ● High community support
  24. 24. Caffe: Deep learning framework by BAIR “Caffe: Convolutional architecture for fast feature embedding”: https://arxiv.org/pdf/1408.5093.pdf
  25. 25. Caffe: Deep learning framework by BAIR Key features ● Shallow framework architecture ● High-level performance for Feed-forward architectures ● Efficient support for multi-thread data loading with LMDB ● 3 different API levels (Prototxt, Python, C++) Data Loaders Net Modules Solver C++ Prototxt Python GPU CPU
  26. 26. Caffe: Deep learning framework by BAIR
  27. 27. [Hands-on (15 minutes)] Training a ConvNet with Caffe
  28. 28. Tensorflow: Deep learning framework by Google “Tensorflow: a system for large-scale machine learning.”: https://www.usenix.org/system/files/conference/osdi16/osdi16-abadi.pdf
  29. 29. Tensorflow: Deep learning framework by Google
  30. 30. Tensorflow: Deep learning framework by Google Key features ● Great models / utils support ● Large API variants at different levels ● Solid codebase ● Amazing community support and permissive Apache 2.0 licence Tensorflow course: https://ekababisong.org/gcp-ml-seminar/tensorflow/
  31. 31. [Hands-on (15 minutes)] Training a ConvNet with Tensorflow
  32. 32. Pytorch: Deep learning framework by Facebook “Automatic differentiation in PyTorch”: https://openreview.net/pdf?id=BJJsrmfCZ
  33. 33. Pytorch: Deep learning framework by Facebook
  34. 34. Pytorch: Deep learning framework by Facebook Key features ● Great models / utils support ● Nice integration with ONNX and Caffe2 ● Great for highly-dynamic graphs ● Simple, neat API ● Easy to learn Python API C++ API
  35. 35. [Hands-on (15 minutes)] Training a ConvNet with Pytorch
  36. 36. Outline 1. Introduction a. From Machine Learning to Deep Learning b. History of Deep Learning Frameworks 2. Open-Source Deep Learning Frameworks a. Main Features and Distinctions b. Caffe, Tensorflow and Pytorch 3. Practical Advices 4. What next?
  37. 37. Practical Advices
  38. 38. Practical Advices ● Your don’t need to choose one framework for life ● Be flexible, understand the core and common ideas ● On-the-run conversion may be painful! ● Choose the best depending on the task at hand and the already existing resources
  39. 39. What Next?
  40. 40. Questions? Vincenzo Lomonaco University of Bologna ML Course - 2018 PhD Student @ University of Bologna Founder of ContinualAI.org vincenzolomonaco.com

×