What is Deep Learning Rise of Deep Learning Phases of Deep Learning - Training and Inference AI & Limitations of Deep Learning Apache MXNet History, Apache MXNet concepts How to use Apache MXNet and Spark together for Distributed Inference.

1. Naveen Swamy
Distributed Deep Learning Inference
using Apache MXNet* and Apache Spark
Senior Software Engineer
Amazon AI
*

2. Outline
• Review of Deep Learning
• Apache MXNet Framework
• Distributed Inference using MXNet and Spark

3. Input layer
(Raw pixels)
Output
(object identity)
3rd hidden layer
(object parts)
2nd hidden layer
(corners & contours)
1st hidden layer
(edges)
• Originally inspired by our biological
neural systems.
• A System that learns important
features from experience.
• Layers of Neurons learning concepts.
• Deep learning != deep understanding
Deep Learning
Credit: Ian Goodfellow etal., Deep Learning Book
CAR PERSON DOG

4. Algorithmic Advances
(Faster Learning)
Abundance of Data
(Deeper Networks)
High Performance Compute
GPUs
(Faster Experiments)
Bigger and Better Models = Better AI Products

5. Why does Deep Learning matter?
Autonomous
Vehicles
Personal Assistants
Solve Intelligence ???
Health care

6. Deep Learning & AI, Limitations
Artificial Intelligence
DL Limitations:
• Requires lots of data and
compute power.
• Cannot detect Inherent bias in
data - Transparency.
• Uninterpretable Results.
Machine Learning
Deep
Learning

7. Deep Learning Training
forward
dog
dog
?
error
labels
data
backward
• Pass data through the network – forward pass
• Define an objective – Loss function
• Send the error back – backward pass
Model: Output of Training a neural network
X2 h2 w6 = 0.5
y = 1.0
y` = 0.9
loss = y
l = 0.1
y
X1 h1
w2 = 0.5
w1 = 0.5
w3 = 0.5
w5 = 0.4
w
4
=
0.5
0.1
0.1
backward pass
forward pass

8. Deep Learning Inference
• Real time Inference: Tasks that require immediate result.
• Batch Inference: Tasks where you need to run on a large data sets.
o Pre-computations are necessary - Recommender Systems.
o Backfilling with state-of-the art models.
o Testing new models on historic data.
model
forward
dog

9. Types of Learning
• Supervised Learning – Uses labeled training data learning to
associate input data to output.
Example: Image classification, Speech Recognition, Machine translation
• Unsupervised Learning - Learns patterns from Unlabeled data.
Example: Clustering, Association discovery.
• Active Learning – Semi-supervised, human in the middle..
• Reinforcement Learning – learn from environment, using rewards and
feedback.

10. Outline
• Apache MXNet Framework
• Distributed Inference using MXNet and Spark

11. Apache MXNet - Background
• Apache (incubating) open source project
• Framework for building and training
DNNs
• Created by academia (CMU and UW)
• Adopted by AWS as DNN framework of
choice, Nov 2016
http://mxnet.apache.org

12. Why MXNet

13. chik-fil-A waffle-fry freshness with Apache MXNet Computer Vision

14. Apache MXNet Customer Momentum

15. MXNet – NDArray & Symbol
• NDArray– Imperative Tensor Operations that work on both CPU and
GPUs.
• Symbol APIs – similar to NDArray but adopts declarative programming
for optimization.
Computation GraphSymbolic Program

16. MXNet - Module
High level APIs to work with Symbol
1) Create Graph
2) Bind
3) Pass data

17. Demo

18. Simple, Easy-to-
Understand Code
Flexible, Imperative
Structure
Dynamic Graphs
High Performance
§ Neural networks can be defined using simple, clear, concise code
§ Plug-and-play neural network building blocks – including predefined
layers, optimizers, and initializers
§ Eliminates rigidity of neural network model definition and brings
together the model with the training algorithm
§ Intuitive, easy-to-debug, familiar code
§ Neural networks can change in shape or size during the training process
to address advanced use cases where the size of data fed is variable
§ Important area of innovation in Natural Language Processing (NLP)
§ There is no sacrifice with respect to training speed
§ When it is time to move from prototyping to production, easily cache
neural networks for high performance and a reduced memory footprint
MXNet Gluon – Imperative & Fast
Credits: Cyrus Vahid

19. Outline
• Distributed Inference using MXNet and Spark

20. Distributed Inference
Challenges
• Similar to large scale data
processing systems
High Performance DL framework
Distributed Cluster
Resource Management
Job Management
Efficient Partition of Data
Deep Learning Setup
Apache Spark:
• Multiple Cluster Managers
• Works well with MXNet.
• Integrates with Hadoop & big data tools.

21. MXNet + Spark for Inference.
• ImageNet trained ResNet-18 classifier.
• For demo, CIFAR-10 test dataset with 10K Images.
• PySpark on Amazon EMR, MXNet is also available in Scala.
• Inference on CPUs, can be extended to use GPUs.

22. Distributed Inference Pipeline
download
S3 keys
on driver
create RDD
and
partition
fetch batch
of images
on executor
decode to
numpy array
run
prediction
collect
predictions
initialize model only once
mapPartitions

23. MXNet + Spark for Inference.
Onthedriver

24. Ontheexecutor

26. Summary
• Overview of Deep Learning
o How Deep Learning works and Why Deep Learning is a big deal.
o Phases of Deep Learning
o Types of Learning
• Apache MXNet – Efficient deep learning library
o NDArray/Symbol/Module
• Apache MXNet and Spark for distributed Inference.

27. What’s Next ?
• Released v1.3.0 with many new features and improvements –
Addition of Clojure support, User friendly APIs in Scala, Improved RNN support in Gluon, …
• Working on Java APIs for Inference.
• MXNet community is fast evolving, join hands to democratize
AI.

28. Resources/References
• https://github.com/apache/incubator-mxnet
• Blog- Distributed Inference using MXNet and Spark
• Distributed Inference code sample on GitHub
• Apache MXNet Gluon Tutorials
• Apache MXNet – Flexible and efficient deep learning.
• The Deep Learning Book
• MXNet – Using pre-trained models
• Amazon Elastic MapReduce

30. Thank You
wamy@amazon.com