Numerous frameworks are aiming to be the king of AI: among them we find Pytorch, Tensorflow, CNTK and MXNet as full frameworks and Keras and fastai as high level structures. To ensure interoperability, it is very important to find a common ground between different implementations, and tools to convert models from one framework to another. It is also very important to provide tools for “agnostic deployment”, freeing the developers on choosing the best framework to deploy, regardless of the deployment environment.

1. Constraints of model deployment and production
environments
Davide Boschetto
Deep Learning Specialist - Software R&D at Microtec GmbH
PhD in Image Analysis at IMT Lucca
davide.boschetto@microtec.eu - davide.boschetto@alumni.imtlucca.it
November 15, 2019
Davide Boschetto SFSCon 2019 November 15, 2019 1 / 17

2. Overview
1 Deep Learning: What and Why
2 Frameworks
State of the Art - 2019
Production
Interoperability
3 Conclusions
Suggestions for R&D Developers
Suggestions for Entrepreneurs
Davide Boschetto SFSCon 2019 November 15, 2019 2 / 17

3. ML and Deep Learning
Traditional Programming: from data and an algorithm, obtain an
output
Machine Learning perspective: from data and output, obtain an
algorithm / the rules
Davide Boschetto SFSCon 2019 November 15, 2019 3 / 17

4. AI Impact
Deep learning already revolutionized most computer vision tasks
(from image segmentation to object classification to DLSS)
NLP will be (is) next (BERT, ULMFit, ...)
Neural Architecture Search is growing
Artificial General Intelligence is still (relatively) far ahead of us
Davide Boschetto SFSCon 2019 November 15, 2019 4 / 17

5. AI key inputs
Without a sufficient amount of data, rules are very hard to find and
learn
Generalization is hard with small or unbalanced datasets
Plenty of different network architectures for each specific problem
Davide Boschetto SFSCon 2019 November 15, 2019 5 / 17

6. Neural Architectures
Image classification:
Residual Networks
(ResNet/ResNeXt), Squeeze
and Excitation (SEnet),
Inception/Xception networks
Semantic Segmentation
(U-Nets, V-Nets, ...)
Object / Instance Detection:
Mask R-CNN, Faster
R-CNN, ...
Davide Boschetto SFSCon 2019 November 15, 2019 6 / 17

7. Frameworks
People mostly use Python for data science / deep learning / AI
The most known frameworks today are:
Tensorflow (+ Keras), originating from Google
Pytorch (with Caffe), originating from Facebook
GPUs are practically mandatory
CUDA+cuDNN: monopoly
ROCm support growing
Davide Boschetto SFSCon 2019 November 15, 2019 7 / 17

8. Frameworks
Pytorch dominates Research environments (2:1 / 3:1 versus
Tensorflow at conferences)
Simpler, great APIs and great performance
Tensorflow still dominates Production (”easier” deployment)
Davide Boschetto SFSCon 2019 November 15, 2019 8 / 17

9. Production environments and challenges
Big focus on deployment and production environments in 2019:
Pytorch 1.0, Tensorflow Serving / TF Lite, TensorRT, ...
Not everything is compatible, though: i.e., different default padding
convention
Hard to retrofit major releases (TF2 / Keras / Caffe)
Davide Boschetto SFSCon 2019 November 15, 2019 9 / 17

10. Frameworks and Licensing
The most famous frameworks are open source. Closed source
alternatives exist (Halcon?), but offer a completely different
experience (obviously)
FAANG open-source lots of their code and environments: why?
By open sourcing their products, they stay relevant and innovative.
Given that their business depends on users, this is how they stay at
the front line of innovation
Moreover, openness implies that a community can contribute to
improve a tool, accelerating internal product development at the
same time
Openness, scrutiny, accountability are key factor to build trust around
the whole AI field
Davide Boschetto SFSCon 2019 November 15, 2019 10 / 17

11. Interoperability
Each framework wants to be the king..
Davide Boschetto SFSCon 2019 November 15, 2019 11 / 17

12. Interoperability: ONNX
Luckily, middle-ground open framework exists: ONNX - UFF (but...)
Davide Boschetto SFSCon 2019 November 15, 2019 12 / 17

13. Deployment: Zoo
Still, it’s a mess: Pytorch supports ONNX, Tensorflow does not,
TensorRT wants UFF files, ...
Being able to fully iterate is hard: choosing a framework is inevitable,
if you don’t want to import all the things into your production
environment (another kind of Dependency Hell)
Davide Boschetto SFSCon 2019 November 15, 2019 13 / 17

14. Suggestions for Developers
Choose a stable release of one framework that can serve your needs of
both research and production environments
Stick with it for a while: all frameworks offer the tools to improve
from traditional computer vision techniques.
All innovations will be available on all frameworks sooner or later (if
not, implement them yourself!): you can’t go wrong with either
Tensorflow or Pytorch
Keep your GitHub and Kaggle profiles updated, stay relevant if you
can
Davide Boschetto SFSCon 2019 November 15, 2019 14 / 17

15. Suggestion on AI Teams
An AI team in the R&D group is ALWAYS a good idea
A PhD is not necessary, but it’s a nice touch (can be paired by
Seniority)
AI teams require different skills and management techniques than
traditional software engineering teams
Most problems are found in the data collection and model deployment
phases
Davide Boschetto SFSCon 2019 November 15, 2019 15 / 17

16. Conclusions
Establish an AI focused team / a transversal team of AI specialists,
do it now
Lots of problems you might be struggling with could already be solved
by AI
AI is not the holy grail: make sure that the stakeholders really know
what can be done by AI and, more importantly, what can’t be done
with it
Davide Boschetto SFSCon 2019 November 15, 2019 16 / 17

17. Thank you!
Any questions?
davide.boschetto@microtec.eu
davide.boschetto@alumni.imtlucca.it
Davide Boschetto SFSCon 2019 November 15, 2019 17 / 17