Build-Train-Deploy-Machine-Learning-Models-at-Any-Scale

© 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Build, Train, and Deploy Machine
Learning Models at Scale
A I M X X X
Ian Perez Ponce
Global Business Development
IoT & Edge Compute @AWS

What you will learn today
• Fundamental building blocks for a coherent AI/ML strategy
• Amazon SageMaker overview
• How to simplify building, training, and deployment of ML
• Algorithms, frameworks, bring your own, and automatic model tuning
• Amazon SageMaker Neo and AWS IoT Greengrass better together
• Deployment walkthrough

Our mission at AWS
Put machine learning in the
hands of every developer

Why AWS for AI/ML workloads?
Broadest and deepest set
of AI and ML services for
your business
Solving the toughest
problems holding back ML
- cost, ease of use, and
data
75% lower inference cost
70% cost reduction in data-
labeling
“No ML experience required”
Unmatched support for
the most popular
frameworks
Built on the most
comprehensive cloud
platform optimized for
Machine Learning
85% of TensorFlow
projects in the cloud
happen on AWS
200+ new features and
services since Jan 2018
AWS holds the top spots on
Stanford’s benchmark, for
fastest training time, lowest
cost, lowest inference latency
Deepest set of security and
encryption features, with robust
analytics capabilities

S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Bringing AI into your digital transformation requires a
new “stack” that makes it easier to put ML to work

FRAMEWORKS INTERFACES INFRASTRUCTURE
AI Services
VISION SPEECH LANGUAGE CHATBOTS FORECASTING RECOMMENDATIONS
ML Services
ML Frameworks + Infrastructure
P O L L Y T R A N S C R I B E T R A N S L A T E C O M P R E H E N D L E X F O R E C A S TR E K O G N I T I O N
I M A G E
R E K O G N I T I O N
V I D E O
T E X T R A C T P E R S O N A L I Z E
Ground Truth Notebooks Algorithms + Marketplace Reinforcement Learning Training Optimization Deployment HostingAmazon SageMaker
F P G A SE C 2 P 3
& P 3 D N
E C 2 G 4 E C 2 C 5 I N F E R E N T I AG R E E N G R A S S E L A S T I C
I N F E R E N C E
The AWS ML Stack
Broadest and deepest set of capabilities

1
2
3
ML is still too complicated for everyday developers
Build Train Deploy

1
2
3
Amazon SageMaker
Build, Train, and Deploy ML Models at Scale
• Notebook
instances
• Call APIs from
your device
• Apache Spark,
SageMaker Spark

Successful models require high-quality data

Easily integrate
human labelers
Get accurate
results
K E Y F E A T U R E S
Automatic labeling
via machine
learning
Ready-made and
custom workflows
Label
management
Quickly label
training data
Private and public
human workforce
Amazon SageMaker Ground Truth
Label machine learning training data easily and accurately

Amazon SageMaker Ground Truth
How it works

1
2
3
Amazon SageMaker
General Purpose
• Factorization Machines
• K-Means
• K-NN
• Linear Learner
• Object2Vec
• PCA
• XGBoost
NLP
• LDA
• Neural Topic Model
• Seq2Seq
• BlazingText
Application Specific:
• DeepAR
• IP Insights
• Random Cut Forest
Computer Vision
• Image Classification
• Object Detection
• Semantic
Segmentation
• Designed for
speed and scale
• Supervised,
unsupervised,
computer vision,
and NLP

1
2
3
Amazon SageMaker
• 20 lines of
Python
• Open sourced
• Local mode for
testing
• BYO (Docker)

1
2
3
Amazon SageMaker
• Efficient meta-model
hyperparameter
tuning
• Works with
algorithms,
frameworks, and BYO

1
2
3
G R E E N G R A S SE C 2 C 5
E C 2 P 3
E C 2 F 1
I n s t a n c e s
Amazon SageMaker

1
2
3
Amazon SageMaker
• Realtime
endpoints
• Batch transform
• AWS Greengrass
• AWS DeepLens

1
2
3
Amazon SageMaker
Build Train Deploy

So what’s next for machine
learning? How do you teach
machine learning models to make
decisions when there is no training
data?

Machine Learning Types
Reinforcement learning
(RL)
Supervised learning
(ASR, computer vision)
Unsupervised learning
(anomaly detection,
identifying text topics)
Amount of labeled training data required
Complexityofdecisions

What is a RL environment?
Representation
of the real world
Programmed
to represent real-
world conditions
Enables interaction
with user or a
computer program
Dynamic and updates
itself based on the
interactions and
programmed behavior

Amazon SageMaker RL
Reinforcement learning for every developer and data scientist
2D & 3D physics
environments and
OpenGym support
Support Amazon Sumerian, AWS
RoboMaker, and the open source
Robotics Operating System
(ROS) project
Fully
managed
Example notebooks
and tutorials

Reinforcement Learning use cases
AUTONOMOUS CARS FINANCIAL TRADING DATACENTER COOLINGFLEET LOGISTICS
Achieve outcomes, not decisions

What if you could train your model
once and run it anywhere, in the
cloud or at the edge, with twice the
speed and no loss in accuracy?

Repeat for every model and every change in the
model
BYO
AWS
Build with your
own algorithms
Build with built-in
algorithms from AWS
Train with
TensorFlow,
MXNet,
or PyTorch
Optimize
your models
Deploy to
the cloud
Deploy to
the edge
A B
A/B test
Multi-site deployment of ML models is complex

Models that are accurate tend
to be big and slow
Models are chained to the
framework in which they were
trainedS
L
M
Accuracy
Performance
Problem: Not all models are skinny

Problem: Not all targets are easy
Need enormous expertise …
Application development: Cloud-native or embedded system
Machine learning: Model training and parameter tuning
Performance tuning: Troubleshooting and optimization
Frameworks: TensorFlow or MXNet or PyTorch or Chainer
Hardware: Cloud server or edge device
Computer architecture: x86 or RISC or GPU or FPGA or ASIC
…and endless time

Problem: Not every path is a catwalk

Open-source device runtime and compiler,
1/10th the size of original frameworks
Amazon SageMaker Neo
Train once, run anywhere with 2x the performance

Neo
Amazon SageMaker Neo
Train once, run anywhere

Parses
model
Optimizes
tensors
Generates
code
Optimizes
graph
Convert a TensorFlow,
MXNet, PyTorch, or
XGBoost model into a
common format
Detect patterns in
the ML model
structure to reduce
the execution time
Shape of input data to
allocate memory
efficiently
Use a low-level compiler to
generate machine code for
each target
Neo Delivers Compilation as a Service
Uses Treelite and Apache TVM for model optimization
No additional cost for Amazon SageMaker users

Dispatches
model
Partitions
graph
Matches model with
execution backend
Sends subgraph to
suitable accelerator
Neo Delivers a Compact Runtime
Open-source software enables device-specific customization
Framework Size
MXNet 450 MB
TensorFlow 660 MB
PyTorch 1000 MB
Neo 1 MB

Data labeling &
pre-built
notebooks
for common
problems
Model and
algorithm
marketplace &
Built-in, high-
performance
algorithms
One-click
training on the
highest
performing
infrastructure
One-click model
optimization
One-click
deployment
Improves
performance on
selected hardware
Extends AWS IoT
services onto
your devices
NeoAmazon SageMaker AWS IoT Greengrass
Amazon SageMaker Neo and AWS IoT Greengrass

Certified HW
Linux / Other OS
GGC
L1 L2 L3
runC
Containers
Device
Mapper
Nodes in
Containers
Code/functions built in AWS Lambda
Lambdas can access Local Resources via
device mapper nodes
ML Inference,
Amazon Sagemaker Neo
AWS IoT Greengrass
A containerized IoT edge runtime

AWS IoT Core
Data Transport & Routing
Amazon SageMaker
 Process Optimization
 Product Quality
 Equipment Health
 Output Forecast
 Equipment Efficiency
 Leak Detection
 Asset Planning
 Ops Optimization
 Tool Productivity
AWS IoT Analytics
Data Aggregation, Enrichment,
Cleansing, Time Series
Processing, Model Config
Model creation
Optimization
RetrainingData Collection, Model Inference
AWS IoT
Greengrass
a:FreeRTOS
Intelligence
and outcomes
ML: Train & optimize in the Cloud; Infer at the Edge

Creating endpoints
Easy deployment to
production REST API
Scalable, high
throughput, and high
reliability

Creating endpoints
Model aws sagemaker create-model
--model-name model1
--primary-container ‘{“Image”: “123.dkr.ecr.amazonaws.com/algo”,
“ModelDataUrl”: “s3://bkt/model1.tar.gz”}’
--execution-role-arn arn:aws:iam::123:role/me

Creating endpoints
Model
Endpoint
configuration
aws sagemaker create-model
--model-name model1
aws sagemaker create-endpoint-config
--endpoint-config-name model1-config
--production-variants ‘{“InitialInstanceCount”: 2,
“InstanceType”: “ml.m4.xlarge”,
”InitialVariantWeight”: 1,
”ModelName”: “model1”,
”VariantName”: “AllTraffic”}’

Creating endpoints
Model
Endpoint
configuration
Endpoint
--model-name model1
aws sagemaker create-endpoint
--endpoint-name my-endpoint

Updating endpoints
Blue-green
deployments means
no scheduled
downtime
Deploy one or more
models behind the
same endpoint

Updating endpoints
New model aws sagemaker create-model
--model-name model2

Updating endpoints
New model
New endpoint
configuration
--model-name model2

Updating endpoints
New model
New endpoint
configuration
Same
endpoint
--model-name model2
“ModelDataUrl”: “s3://bkt/model2.tar.gz”}
aws sagemaker update-endpoint

Reduced risk deployments
Incrementally retrain
models with new data
Try new models and
improved algorithms

Two model
endpoint
configuration
--endpoint-config-name both-models-config
--production-variants ‘[{“InitialInstanceCount”: 2,
”VariantName”: “model1-traffic”},
{“InitialInstanceCount”: 2,
”VariantName”: “model2-traffic”}]’

Two model
endpoint
configuration
Same
endpoint

Two model
endpoint
configuration
Same
endpoint
Swap
aws sagemaker update-endpoint-weights-and-capacities
--desired-weights-and-capacities ‘{“VariantName”: ”model1”,
“DesiredWeight”: 5}’

Automatically scaling endpoints
SageMaker console settings:
• Min and max instances
• Target invocations per
instance
• Scaling cool downs

Why automatic scaling?

Automatic scaling in action

Scaling criteria
Algorithms have
different memory,
CPU, or GPU
requirements
Autoscale based on
endpoint instance’s
CloudWatch Metrics

Creating an automatic scaling policy
Variant aws application-autoscaling register-scalable-target
--service-namespace sagemaker
--resource-id endpoint/my-endpoint/variant/model2
--scalable-dimension sagemaker:variant:DesiredInstanceCount
--min-capacity 2
--max-capacity 5

Creating an automatic scaling policy
Variant
Policy
aws application-autoscaling register-scalable-target
--min-capacity 2
--max-capacity 5
aws application-autoscaling put-scaling-policy
--policy-name model2-scaling
--policy-type TargetTrackingScaling
--target-tracking-scaling-policy-configuration
‘{"TargetValue": 50,
"CustomizedMetricSpecification":
{"MetricName": "CPUUtilization",
"Namespace": "/aws/sagemaker/Endpoints",
"Dimensions":
[{"Name": "EndpointName", "Value": "my-endpoint"},
{"Name": "VariantName","Value": ”model2"}],
"Statistic": "Average",
"Unit": "Percent”}}’

Scale by utilization

Why AWS for AI/ML workloads?

One-click
model training
and deployment
Train once
run anywhere
10x
better algorithm
performance
2x
performance increases from
model optimization with Neo
70%
cost reduction for data labeling
using Ground Truth
75%
cost reduction for inference with
Elastic Inference
R E DUCE CO STS IN CREASE PE RFORMANCE E A SE-OF-USE
AMAZON SAGEMAKER
Machine Learning made simple

AWS is framework agnostic
Choose from popular frameworks
Run them fully managed Or run them yourself

Subscribe in a
single click
Available in
Amazon SageMaker
KEY FEATURES
Automatic labeling via machine learning
IP protection
Automated billing and metering
Browse or search
AWS Marketplace
SELLERS
Broad selection of paid, free, and
open-source algorithms and models
Data protection
Discoverable on your AWS bill
BUYERS
AWS Marketplace for Machine Learning
ML algorithms and models available instantly

Over 200 algorithms and models
Natural Language
Processing
Grammar & Parsing Text OCR Computer Vision
Named Entity
Recognition
Video Classification
Speech Recognition Text-to-Speech Speaker Identification Text Classification 3D Images Anomaly Detection
Text Generation Object Detection Regression Text Clustering
Handwriting
Recognition
Ranking
A V A I L A B L E A L G O R I T H M S & M O D E L S
S E L E C T E D V E N D O R S

Resources
SageMaker Product Page
https://aws.amazon.com/sagemaker
SageMaker Console
https://console.aws.amazon.com/sagemaker/
Ground Truth Product Page
https://aws.amazon.com/sagemaker/groundtruth
Neo Product Page
https://aws.amazon.com/sagemaker/neo
SageMaker 10-Minute Tutorial
https://aws.amazon.com/getting-started/tutorials/build-train-deploy-machine-learning-model-sagemaker/
SageMaker Related Blogs
https://aws.amazon.com/blogs/machine-learning/category/artificial-intelligence/sagemaker/
Greengrass ML Inference Product Page
https://aws.amazon.com/greengrass/ml/

Thank you!
Ian Perez Ponce | iperezpo@
Global Business Development, IoT & Edge Compute @AWS

Build-Train-Deploy-Machine-Learning-Models-at-Any-Scale

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Build-Train-Deploy-Machine-Learning-Models-at-Any-Scale

Similar to Build-Train-Deploy-Machine-Learning-Models-at-Any-Scale (20)

More from Amazon Web Services

More from Amazon Web Services (20)

Build-Train-Deploy-Machine-Learning-Models-at-Any-Scale