An overview of artificial intelligence, machine learning, and how to tackle some of these problems on AWS. Some specifics given on Apache MXNet

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Angel Pizarro | TBDM | Research and Technical Computing
July 19, 2017
Machine Learning on the Cloud with
Apache MXNet

2. Today’s Agenda
What is artificial intelligence?1
What are the challenges with AI?2
Apache MXNet overview3
Demo (time permitting)4

3. A system or service which can perform tasks
that usually require human intelligence
Artificial Intelligence

4. Supervised Learning Unsupervised Learning
Machine Learning
Supervised Learning:
• Learning from “labelled data”
• Classification, Regression, Prediction, Function
Approx
Unsupervised Learning:
• Method to find similar groups in the data clusters
• Groups that are similar to near clusters
• Groups different far away from each other

5. Euclidean Distance Score Pearson Correlation Score
Machine Learning (Classification, Regression
and Ranking)

6. Hierarchical Clustering K-Mean Clustering
Machine Learning (Unsupervised learning -
Discovering Clusters or Groups)
Builds hierarchy of groups by merging
two most similar groups. Computationally Intensive
(k randomly placed centriods)

7. Decision trees are one of the simplest machine-learning methods
Based on classifying observations, which after training looks like series of if-then statements
Machine Learning (Decision Trees)

8. 0.2
-0.1
...
0.7
Input Output
1 1 1
1 0 1
0 0 0
3
mx.sym.Pooling(data, pool_type="max", kernel=(2,2), stride=(2,2)
lstm.lstm_unroll(num_lstm_layer, seq_len, len, num_hidden, num_embed)
4 2
2 0
4=Max
1
3
...
4
0.2
-0.1
...
0.7
mx.sym.FullyConnected(data, num_hidden=128)
2
mx.symbol.Embedding(data, input_dim, output_dim = k)
Queen
4 2
2 0
2=Avg
Input Weights
cos(w, queen) = cos(w, king) - cos(w, man) + cos(w, woman)
mx.sym.Activation(data, act_type="xxxx")
"relu"
"tanh"
"sigmoid"
"softrelu"
Neural Art
Face Search
Image Segmentation
Image Caption
“People Riding Bikes”
Bicycle, People,
Road, Sport
Image Labels
Image
Video
Speech
Text
“People Riding Bikes”
Machine Translation
“Οι άνθρωποι
ιππασίας ποδήλατα”
Events
mx.model.FeedForward model.fit
mx.sym.SoftmaxOutput
Anatomy of a Deep Learning Model
mx.sym.Convolution(data, kernel=(5,5), num_filter=20)
Deep Learning Models

10. Deep Learning | Applications

11. Automatic Grading of Diabetic Retinopathy through
Deep Learning using AWS

12. Dermatologist-level classification of skin
cancer
http://cs.stanford.edu/people/esteva/nature/

13. FDA-Approved Medical Imaging

14. Autonomous Driving Systems

15. Real Time, Per Pixel
Object Segmentation

16. Centimeter-accurate
positioning

17. TX1 with customized board
Drone
Realtime detection and tracking on TX1
~10 frame/sec with 640x480 resolution
TX1 on Flying Drone

18. https://aws.amazon.com/lex/
Natural Language Understanding (NLU) & Automatic Speech Recognition (ASR)
as in Amazon ALEXA - Powered by Deep Learning

19. Challenges with ML/AI

20. Data
The Challenge For Artificial Intelligence: SCALE

21. Data Training
The Challenge For Artificial Intelligence: SCALE

22. Data Training Prediction
The Challenge For Artificial Intelligence: SCALE

23. Aggressive migration
New data created on AWS
Data
Training Prediction
PBs of existing data
The Challenge For Artificial Intelligence: SCALE

24. Tons of GPUs
Elastic capacity
Training
Prediction
Pre-built images
Aggressive migration
New data created on AWS
Data
PBs of existing data
The Challenge For Artificial Intelligence: SCALE

25. Tons of GPUs and CPUs
Serverless
At the Edge, On IoT Devices
Prediction
Tons of GPUs
Elastic capacity
Training
Pre-built images
Aggressive migration
New data created on AWS
Data
PBs of existing data
The Challenge For Artificial Intelligence: SCALE

26. AI Services
Amazon
Rekognition
Amazon
Polly
Amazon
Lex
More to come
in 2017
Amazon AI: Democratized Artificial Intelligence

27. AI Services
Amazon
Rekognition
Amazon
Polly
Amazon
Lex
More to come
in 2017
Amazon AI: Democratized Artificial Intelligence
AI Platform
More to come
in 2017
Amazon
Machine Learning
Amazon Elastic
MapReduce
Spark &
SparkML

28. AI Services
Amazon
Rekognition
Amazon
Polly
Amazon
Lex
More to come
in 2017
AI Engines
Apache
MXNet
TensorFlow Caffe Theano KerasTorch CNTK
Amazon AI: Democratized Artificial Intelligence
AI Platform
More to come
in 2017
Amazon
Machine Learning
Amazon Elastic
MapReduce
Spark &
SparkML

29. P2 Batch/ECS Lambda
AWS
Greengrass
FPGAEMR
More to
come
in 2017
Infrastructure
AI Services
Amazon
Rekognition
Amazon
Polly
Amazon
Lex
More to come
in 2017
AI Engines
Apache
MXNet
TensorFlow Caffe Theano KerasTorch CNTK
AI Platform
More to come
in 2017
Amazon
Machine Learning
Amazon Elastic
MapReduce
Spark &
SparkML
Amazon AI: Democratized Artificial Intelligence

30. EC2 P2 Instance | Up to 16 GPUs
This instance type incorporates up to 8 NVIDIA Tesla K80 Accelerators,
each running a pair of NVIDIA GK210 GPUs.
Each GPU provides 12 GiB of memory (accessible via 240 GB/second of
memory bandwidth), and 2,496 parallel processing cores.
Available in PDX, IAD and DUB Regions
Instance Name GPU Count vCPU Count Memory
Parallel
Processing
Cores
GPU Memory
Network
Performance
p2.xlarge 1 4 61 GiB 2,496 12 GiB High
p2.8xlarge 8 32 488 GiB 19,968 96 GiB 10 Gigabit
p2.16xlarge 16 64 732 GiB 39,936 192 GiB 20 Gigabit

31. Core of the next AWS GPU instance family
Custom built for artificial intelligence
Train in hours, not days
NVIDIA Volta
Training Artificial Intelligence With GPUs

32. Core of the next AWS GPU instance family
Custom built for artificial intelligence
Train in hours, not days
NVIDIA Volta
MXNet is already optimized for Volta
Training Artificial Intelligence With GPUs

33. Up To 40,000
CUDA Cores
Apache
MXNet
Python 3 Notebooks
& Examples
(and others)
https://aws.amazon.com/amazon-ai/amis/
AWS Deep Learning AMI: One-Click Deep
Learning

34. Apache MXNet | Overview

35. Apache MXNet
Programmable Portable High Performance
Near linear scaling
across hundreds of GPUs
Highly efficient
models for mobile
and IoT
Simple syntax,
multiple languages
Most Open Best On AWS
Optimized for
deep learning on
AWS
Accepted into the
Apache Incubator

36. Deep Learning using MXNet @Amazon
• Applied Research
• Core Research
• Alexa
• Demand Forecasting
• Risk Analytics
• Search
• Recommendations
• AI Services | Rek, Lex, Polly
• Q&A Systems
• Supply Chain Optimization
• Advertising
• Machine Translation
• Video Content Analysis
• Robotics
• Lots of Computer Vision..
• Lots of NLP/U..
*Teams are either actively evaluating, in development, or transitioning to scale production

37. Multi-GPU Scaling With MXNet

38. Multi-Machine Scaling With MXNet

39. Deep Learning Framework Comparison
Apache MXNet TensorFlow Cognitive Toolkit
Industry Owner
N/A – Apache
Community
Google Microsoft
Programmability
Imperative and
Declarative
Declarative only Declarative only
Language
Support
R, Python, Scala, Julia,
Cpp. Javascript, Go,
Matlab and more..
Python, Cpp.
Experimental Go and
Java
Python, Cpp,
Brainscript.
Code Length|
AlexNet (Python)
44 sloc 107 sloc using TF.Slim 214 sloc
Memory Footprint
(LSTM)
2.6GB 7.2GB N/A
*sloc – source lines of code

40. Apache MXNet | The Basics

41. Apache MXNet | The Basics
• NDArray: Manipulate multi-dimensional arrays in a command line
paradigm (imperative).
• Symbol: Symbolic expression for neural networks (declarative).
• Module: Intermediate-level and high-level interface for neural
network training and inference.
• Loading Data: Feeding data into training/inference programs.
• Mixed Programming: Training algorithms developed using
NDArrays in concert with Symbols.

42. import numpy as np
a = np.ones(10)
b = np.ones(10) * 2
c = b * a
d = c + 1
• Straightforward and flexible.
• Take advantage of language
native features (loop,
condition, debugger).
• E.g. Numpy, Matlab, Torch, …
•Hard to optimize
PROS
CONS
Easy to tweak
in Python
Imperative Programming

43. • More chances for
optimization
• Cross different languages
• E.g. TensorFlow, Theano,
Caffe
•Less flexible
PROS
CONSC can share memory with
D because C is deleted
later
A = Variable('A')
B = Variable('B')
C = B * A
D = C + 1
f = compile(D)
d = f(A=np.ones(10),
B=np.ones(10)*2)
A B
1
+
X
Declarative Programming

44. IMPERATIVE
NDARRAY API
DECLARATIVE
SYMBOLIC
EXECUTOR
>>> import mxnet as mx
>>> a = mx.nd.zeros((100, 50))
>>> b = mx.nd.ones((100, 50))
>>> c = a + b
>>> c += 1
>>> print(c)
>>> import mxnet as mx
>>> net = mx.symbol.Variable('data')
>>> net = mx.symbol.FullyConnected(data=net, num_hidden=128)
>>> net = mx.symbol.SoftmaxOutput(data=net)
>>> texec = mx.module.Module(net)
>>> texec.forward(data=c)
>>> texec.backward()
NDArray can be set
as input to the graph
Mixed Programming Paradigm

45. Embed symbolic expressions into imperative
programming
texec = mx.module.Module(net)
for batch in train_data:
texec.forward(batch)
texec.backward()
for param, grad in zip(texec.get_params(), texec.get_grads()):
param -= 0.2 * grad
Mixed Programming Paradigm

46. • Fit the core library with all dependencies into a
single C++ source file
• Easy to compile on any platform
Amalgamation
BlindTool by Joseph Paul Cohen, demo on Nexus 4
RUNS IN BROWSER
WITH JAVASCRIPT

47. Roadmap / Areas of Investment
• Usability
• Keras Integration WIP
• MinPy being merged (Dynamic Computation graphs, Std Numpy
interface)
• Documentation (installation, native documents, etc.)
• Tutorials, examples | Jupyter Notebooks
• Platform support
(Linux, Windows, OS X, mobile …)
• Language bindings
(Python, C++, R, Scala, Julia, JavaScript …)
• Sparse datatypes and LSTM performance improvements
• Deploy your model your way: Lambda, Amazon EC2/Docker, Raspberry Pi

48. Apache MXNet | Developer Tools and
Resources

49. One-Click GPU or CPU
Deep Learning
AWS Deep Learning AMI
Up to~40k CUDA cores
Apache MXNet
TensorFlow
Theano
Caffe
Torch
Keras
Pre-configured CUDA drivers, MKL
Anaconda, Python3
Ubuntu and Amazon Linux
+ AWS CloudFormation template
+ Container image

50. Application Examples | Jupyter Notebooks
• https://github.com/dmlc/mxnet-notebooks
• Basic concepts
• NDArray - multi-dimensional array computation
• Symbol - symbolic expression for neural networks
• Module - neural network training and inference
• Applications
• MNIST: recognize handwritten digits
• Check out the distributed training results
• Predict with pre-trained models
• LSTMs for sequence learning
• Recommender systems
• Train a state of the art Computer Vision model (CNN)
• Lots more..

51. Developer Resources
MXNet Resources:
• MXNet Blog Post | AWS Endorsement
• Read up on MXNet and Learn More: mxnet.io
• MXNet Github Repo
• MXNet Recommender Systems Talk | Leo Dirac
Developer Resources:
• Deep Learning AMI |Amazon Linux
• Deep Learning AMI | Ubuntu
• CloudFormation Template Instructions
• Deep Learning Benchmark
• MXNet on Lambda
• MXNet on ECS/Docker
• MXNet on Raspberry Pi | Image Detector using Inception Network

52. Apache MXNet | Jupyter Notebook Demo

53. Thank You!
pizarroa@amazon.com