This document provides an overview of open-source deep learning frameworks. It discusses the history and evolution of frameworks from 2010 to present. The main frameworks covered are Caffe, TensorFlow, and PyTorch. Each framework is characterized by features like differentiation support, hardware acceleration, model types, and licensing. Practical advice is given to choose a framework based on the task and existing resources rather than sticking to one for all projects. The document aims to help researchers and practitioners understand and select among leading deep learning software options.

1. Open-Source Frameworks for
Deep Learning: an Overview
Vincenzo Lomonaco
University of Bologna ML Course - 2018
PhD Student @ University of Bologna
Founder of ContinualAI.org
vincenzolomonaco.com

2. About Me
• PhD Student @ University of Bologna
• Visiting Scholar @ ENSTA ParisTech and @ Purdue
University
• Phd Students’ Representative of the Department of
Computer Science and Engineering
• Teaching Assistant of the courses Machine Learning
and Computer Architectures
• Author and Technical reviewer of the online course
Deep Learning with R and book R Deep Learning
Essentials
• Founder of Continual AI

3. Continual AI Community
http://continualai.org https://continualai.herokuapp.com

4. Outline
1. Introduction
a. From Machine Learning to Deep Learning
b. History of Deep Learning Frameworks
2. Open-Source Deep Learning Frameworks
a. Main Features and Distinctions
b. Caffe, Tensorflow and Pytorch
3. Practical Advices
4. What next?

5. The Machine Learning Software Stack
“Hidden Technical Debt in Machine Learning Systems” by D. Sculley et al.

6. The Machine Learning Software Stack
For Prototyping and R&D
● Scikit-learn (Python), Caret
(R), Weka (Java)
For Production
● Kubernetes, KubeFlow,
Hadoop
● H2O, MLlib, SparkML,
FlinkML
Setting up a Machine Learning Framework for Production: http://code.hootsuite.com/setting-up-a-machine-learning-framework-for-production/

7. From Machine Learning to Deep Learning
For Deep Learning the line between
Prototyping and Production is
more blurred
● Caffe / Caffe 2
● Torch / PyTorch
● Tensorflow
● CNTK, MxNet, Gluon,
Chainer, etc.

8. From Machine Learning to Deep Learning
Why
● Prototyping with neural nets on
massive datasets needs efficiency
● Limited focus on:
1. (Deep) Neural Networks.
2. Gradient-Based Optimization.
3. Transparent (multi) CPU/GPU
acceleration

9. History of Deep Learning Frameworks
2010 2013 2014 2015 2016 20182017
...

10. History of Deep Learning Frameworks

11. History of Deep Learning Frameworks
Andrej Karpathy, 10 mar 2018 : https://twitter.com/karpathy/status/972295865187512320/

12. Deep Learning Frameworks: 2018 Figures
Deep Learning Framework Power Scores 2018 : https://towardsdatascience.com/deep-learning-framework-power-scores-2018-23607ddf297a/

13. Deep Learning Frameworks: 2018 Figures
Deep Learning Framework Power Scores 2018 : https://towardsdatascience.com/deep-learning-framework-power-scores-2018-23607ddf297a/

14. Outline
1. Introduction
a. From Machine Learning to Deep Learning
b. History of Deep Learning Frameworks
2. Open-Source Deep Learning Frameworks
a. Main Features and Distinctions
b. Caffe, Tensorflow and Pytorch
3. Practical Advices
4. What next?

15. Main Features and Distinctions
1. Static vs Dynamic Graph
2. Differentiation Support
3. Hardware / Platform Support
4. Model / Utils Support and Performance
5. Multi-CPU/GPU support
6. API level
7. License and Community Support
Comparison of deep learning software: https://en.wikipedia.org/wiki/Comparison_of_deep_learning_software

16. Static vs Dynamic Graph
Static (define-AND-run) Dynamic (define-BY-run)

17. Differentiation Support
Types of Differentiation
1. Numerical Differentiation
2. Symbolic Differentiation
3. Automatic Differentiation
4. Hard-coded Differentiation
“Automatic Differentiation in Machine Learning: a Survey”: http://www.jmlr.org/papers/volume18/17-468/17-468.pdf

18. Hardware / Platform Support
Platform Support
● Windows, Unix (MAC OS, Linux)
● Android OS, iOS, Embedded
Systems
Hardware Acceleration
● Cuda / OpenCL Support
● OpenMP, MPI Support
● AI Chips

19. Model / Utils Support and Performance
Types of models supported
● FC-NNs, CNNs, RNNs, etc.
● General purpose algebraic
functions
Utils support
● Pre-trained models Zoo
● Data format and loading
● Monitoring / Visualization tools
Performance
● Very different from task to
task and model to model, etc.

20. Multi-CPU / GPU support
Types of parallelization
● Data Parallelism
● Model Parallelism
Hardware for parallelization
● CPUs
● GPUs
● Distributed clusters
“Large Scale Distributed Deep Networks”: http://papers.nips.cc/paper/4687-large-scale-distributed-deep-networks.pdf

21. API Level
Important features
● Usability
● Flexibility
● Expandability
● Easy to Debug

22. License and Community Support
Licence
● Apache 2.0, MIT, BSD,
GNU GPL, Freemium,
etc.
● Most of them allow also
commercial use (but look
at patent licence too)
Community Support
● Contribution Diversity
● Active Development
● Supportive Q&A
● Number of users

23. Caffe, Tensorflow and Pytorch
● Static Graph
● Hard-coded Differentiation
● Windows, Unix, Embedded
● Mainly Feedforward models
● Multi-CPU/GPUs
● Prototxt, C++, Python API
● BSD Licence
● Low community support
● Static Graph (with lazy eval)
● Automatic Differentiation
● Windows, Unix, Embedded
● General purpose graph
● Multi-CPU/GPUs
● C++ and (many) Python API
● Apache 2.0
● High community support
● Dynamic Graph
● Automatic Differentiation
● Windows, Unix, Embedded
● General purpose graph
● Multi-CPU/GPUs
● C++ and Python API
● BSD Licence
● High community support

24. Caffe: Deep learning framework by BAIR
“Caffe: Convolutional architecture for fast feature embedding”: https://arxiv.org/pdf/1408.5093.pdf

25. Caffe: Deep learning framework by BAIR
Key features
● Shallow framework architecture
● High-level performance for
Feed-forward architectures
● Efficient support for multi-thread
data loading with LMDB
● 3 different API levels (Prototxt,
Python, C++)
Data Loaders
Net Modules
Solver
C++ Prototxt Python
GPU CPU

26. Caffe: Deep learning framework by BAIR

27. [Hands-on (15 minutes)] Training a
ConvNet with Caffe

28. Tensorflow: Deep learning framework by
Google
“Tensorflow: a system for large-scale machine learning.”: https://www.usenix.org/system/files/conference/osdi16/osdi16-abadi.pdf

29. Tensorflow: Deep learning framework by
Google

30. Tensorflow: Deep learning framework by
Google
Key features
● Great models / utils
support
● Large API variants at
different levels
● Solid codebase
● Amazing community
support and permissive
Apache 2.0 licence
Tensorflow course: https://ekababisong.org/gcp-ml-seminar/tensorflow/

31. [Hands-on (15 minutes)] Training a
ConvNet with Tensorflow

32. Pytorch: Deep learning framework by
Facebook
“Automatic differentiation in PyTorch”: https://openreview.net/pdf?id=BJJsrmfCZ

33. Pytorch: Deep learning framework by
Facebook

34. Pytorch: Deep learning framework by
Facebook
Key features
● Great models / utils support
● Nice integration with ONNX
and Caffe2
● Great for highly-dynamic
graphs
● Simple, neat API
● Easy to learn
Python API C++ API

35. [Hands-on (15 minutes)] Training a
ConvNet with Pytorch

36. Outline
1. Introduction
a. From Machine Learning to Deep Learning
b. History of Deep Learning Frameworks
2. Open-Source Deep Learning Frameworks
a. Main Features and Distinctions
b. Caffe, Tensorflow and Pytorch
3. Practical Advices
4. What next?

37. Practical Advices

38. Practical Advices
● Your don’t need to choose one framework for life
● Be flexible, understand the core and common
ideas
● On-the-run conversion may be painful!
● Choose the best depending on the task at hand
and the already existing resources

39. What Next?

40. Questions?
Vincenzo Lomonaco
University of Bologna ML Course - 2018
PhD Student @ University of Bologna
Founder of ContinualAI.org
vincenzolomonaco.com