Developers spend much time and effort delivering machine learning models that can make fast and accurate predictions in real time. These models become even more critical for edge devices where memory and processing power are constrained. Amazon SageMaker Neo lets developers run and develop models in the most optimized way: train the models once and run them anywhere in the cloud and at the edge. In this chalk talk, we dive deep into Neo and show you how this capability of Amazon SageMaker automatically optimizes models built on TensorFlow, Apache MXNet, PyTorch, and ONNX.

1. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Train once, deploy anywhere on the cloud
and at the edge with Amazon SageMaker
Neo
Vebhhav Singh
Sr. Solutions Architect
AWS
A I M 3 0 3

2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon AI/ML Stack
M L S E R V I C E S
M L F R A M E W O R K S
A I S E R V I C E S
(ML researchers
and academics)
(ML developers
and data scientists)
A m a z o n
S a g e M a k e r
G R O U N D T R U T H A L G O R I T H M S
N O T E B O O K S
M A R K E T P L A C E
U N S U P E R V I S E D
L E A R N I N G
S U P E R V I S E D
L E A R N I N G
R E I N F O R C E M E N T
L E A R N I N G
O P T I M I Z A T I O N
( N E O )
T R A I N I N G
H O S T I N G
D E P L O Y M E N T
Vision Speech Language Chatbots &
Contact Centers
Verticals
A m a z o n
R e k o g n i t i o n
I m a g e
A m a z o n
R e k o g n i t i o n
V i d e o
A m a z o n
P o l l y
A m a z o n
T r a n s c r i b e
A m a z o n
T r a n s l a t e
A m a z o n
C o m p r e h e n d A m a z o n
L e x
A m a z o n
F o r e c a s t
A m a z o n
T e x t r a c t
A m a z o n
P e r s o n a l i
(Applicaion
Developers)

3. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Deployment of ML models is complex
Repeat for every model and every change in the model
BYO
AWS
Build with your
own algorithms
Build with built-in algorithms
from AWS
Train with
TensorFlow,
MXNet ,
or PyTorch
Optimize
your models
Deploy to
the cloud
Deploy to
the edge
A B
A/B test

4. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
The majority of the cost and
complexity of ML in
production is due to Inference
Inference (Prediction)
90%
Training
10%

5. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Problem: Not all models are skinny
Models that are accurate tend to
be big and slow
Models are chained to the
framework in which they were
trainedS
L
M
Accuracy
Performance

6. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Problem: Not all targets are easy
Need enormous expertise …
Application Development: Cloud-Native or Embedded System
Machine Learning: Model Training and Parameter Tuning
Performance Tuning: Troubleshooting and Optimization
Frameworks: TensorFlow or MXNet or PyTorch or Chainer
Hardware: Cloud Server or Edge Device
Computer Architecture: x86 or RISC or GPU or FPGA or ASIC
…and endless time

7. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Problem: Not every path is a catwalk

8. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
What if you could train your model once and run it
anywhere, in the cloud or at the edge, with twice the
speed and no loss in accuracy?

9. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon SageMaker Neo

10. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Increase inference performance by 2x
Reduce runtime footprint by 100x

11. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon SageMaker Neo
K e y f e a t u r e s
Machine Learning Compiler for CPU or GPU
Compact Open Source Runtime
https://aws.amazon.com/sagemaker/neo

12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Neo bridges Amazon SageMaker & AWS IoT Greengrass
Data labeling &
pre-built
notebooks
for common
problems
Model and
algorithm
marketplace &
built-in, high-
performance
algorithms
One-click
training on the
highest
performing
infrastructure
One-click model
optimization
One-click
deployment
Improves performance on
selected hardware
Extends AWS IoT
onto your devices
NeoAmazon SageMaker AWS IoT Greengrass
ML models train once run anywhere

13. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Neo delivers compilation as a service
Parses
model
Optimizes
tensors
Generates
code
Optimizes
graph
Convert a TensorFlow,
MXNet, PyTorch, or
XGBoost model into a
common format
Detect patterns in the
ML model structure to
reduce the execution
time
Detect patterns in the shape
of input data to allocate
memory efficiently
Use a low-level compiler to
generate machine code for
each target
No additional cost for Amazon SageMaker users

14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Neo delivers compilation as a service
Parses
model
Optimizes
tensors
Generates
code
Optimizes
graph
Graph pruning
Layer fusion
Constant folding
Layout transforms
Nested parallelism
Tiling
Tensorization
Auto-tuning
Uses Treelite and Apache TVM for model optimization

15. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Neo delivers a compact runtime
Dispatches
model
Partitions
graph
Matches model with
execution backend
Sends subgraph to
suitable accelerator
Framework Size
MXNet 450 MB
TensorFlow 660 MB
PyTorch 1000 MB
Neo 1 MB
Open-source software enables device-specific customization

16. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

17. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Steps for sample deployment
- Get an existing model
- Compile model for new hardware
- Copy the artifacts to new hardware
- Deploy the Neo runtime
- Run the inference

18. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Neo: Open-source project

19. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Q&A

20. Thank you!
S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Vebhhav Singh
vebhhavs@amazon.com