Building Deep Learning Applications with TensorFlow and SageMaker on AWS - Tel Aviv Summit 2018

© 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Guy Ernest
Principal Solution Arrchitect, AWS Deep Learning
Building Deep Learning Applications
with TensorFlow and SageMaker on
AWS

Amazon business is built on deep
learning

Amazon is Hiring!
Roles: Machine Learning Scientists, Research Scientists,
Machine Learning Managers, Research Managers, Software
Engineers, Software Development Managers, Product
Managers, Technical Program Managers etc!
Locations: Sunnyvale, Palo Alto, Pasadena, Seattle,
Pittsburgh, Boston, New York City, Aachen Germany,
Cambridge UK, Turin Italy, Hyderabad, India

How many times did I say “data” so far?

Data Lakes from AWS
Analytics
• Unmatched durability, and availability at EB scale
• Best security, compliance, and audit capabilities
• Object-level controls for fine-grain access
• Fastest performance by retrieving subsets of data
• The most ways to bring data in
• 2x as many integrations with partners
• Analyze with broadest set of analytics & ML services
Machine
Learning
Real-time DataOn-premises
Data Lake
on AWS
MovementData Movement

Data Lakes, Analytics, and IoT Portfolio from
AWS
Broadest, deepest set of analytic services
Amazon SageMaker
AWS Deep Learning AMIs
Amazon Rekognition
Amazon Lex
AWS DeepLens
Amazon Comprehend
Amazon Translate
Amazon Transcribe
Amazon Polly
Amazon Athena
Amazon EMR
Amazon Redshift
Full-text search
Amazon Elasticsearch service
Amazon QuickSight
AWS Direct Connect
AWS Snowball
AWS Snowmobile
AWS Database Migration Service
AWS IoT Core
Amazon Kinesis Data Firehose
Amazon Kinesis Data Streams
Amazon Kinesis Video Streams
Data Lake
on AWS
Storage | Archival Storage | Data Catalog
AnalyticsMachine Learning
Real-time DataOn-premises MovementData Movement

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Agenda
• Advanced Topics in Amazon SageMaker
• Integration between Spark and Amazon SageMaker
• Amazon SageMaker Built-in Algorithm
– Time series forecasting using DeepAR Forecasting
– Image Classification (Transfer learning with ResNet)
• ML training and deployment using any ML framework (including TensorFlow)
• Hyper-parameters Optimization
• Q&A

What is Amazon SageMaker?
One-click training
for ML, DL, and
custom algorithms
Easier training with
hyperparameter
optimization
Highly-optimized
machine learning
algorithms
Deployment
without
engineering effort
Fully-managed
hosting at scale
Build
Pre-built
notebook
instances
Deploy
Train

Amazon SageMaker
1 2 3 4
I I I I
Notebook Instances Algorithms ML Training Service ML Hosting Service

Amazon SageMaker integration with
Apache Spark (on Amazon EMR)

Calling Spark from Amazon SageMaker
https://aws.amazon.com/blogs/machine-learning/build-amazon-sagemaker-notebooks-backed-by-spark-in-amazon-emr/

Calling Amazon SageMaker from Spark I
https://github.com/aws/sagemaker-spark

Calling Amazon SageMaker from Spark II

Integrating Amazon SageMaker in Spark
Pipeline

Using Spark for Batch Prediction
https://aws.amazon.com/blogs/machine-learning/distributed-inference-using-apache-mxnet-and-apache-spark-on-amazon-emr/

How to build ML algorithms to scale

Scalable Architecture
GPU
GPU
GPU Local
State
Shared
State
Local
State
Local
State
File/Pipe Modes
Cloud Optimized

Linear Learner
Regression:
Estimate a real valued function
Binary Classification:
Predict a 0/1 class
˜y = hw, xi + t ˜y =
(
1 if hw, xi > t
0 else

Linear Learner
Regression (mean squared error)
SageMaker Other
1.02 1.06
1.09 1.02
0.332 0.183
0.086 0.129
83.3 84.5
Classification (F1 Score)
SageMaker Other
0.980 0.981
0.870 0.930
0.997 0.997
0.978 0.964
0.914 0.859
0.470 0.472
0.903 0.908
0.508 0.508
30 GB datasets for web-spam and web-url classification
0
0.2
0.4
0.6
0.8
1
1.2
0 5 10 15 20 25 30
CostinDollars
Billable time in Minutes
sagemaker-url sagemaker-spam other-url other-spam

K-Means Clustering
Method Accurate? Passes Efficient
Tuning
Comments
Lloyds [1] Yes* 5-10 No
K-Means ++ [2] Yes k+5 to k+10 No scikit-learn
K-Means|| [3] Yes 7-12 No spark.ml
Online [4] No 1 No
Streaming [5,6] No 1 No Impractical
Webscale [7] No 1 No spark streaming
Coresets [8] No 1 Yes Impractical
SageMaker Yes 1 Yes
[1] Lloyd, IEEE TIT, 1982
[2] Arthur et. al. ACM-SIAM, 2007
[3] Bahmani et. al., VLDB, 2012
[4] Liberty et. al., 2015
[5] Shindler et. al, NIPS, 2011
[6] Guha et. al, IEEE Trans. Knowl. Data Eng. 2003
[7] Sculley, WWW, 2010
[8] Feldman et. al.

0
1
2
3
4
5
6
7
8
10 100 500
BillableTimeinMinutes Number of Clusters
sagemaker other
K-Means Clustering
k SageMaker Other
Text
1.2GB
10 1.18E3 1.18E3
100 1.00E3 9.77E2
500 9.18.E2 9.03E2
Images
9GB
10 3.29E2 3.28E2
100 2.72E2 2.71E2
500 2.17E2 Failed
Videos
27GB
10 2.19E2 2.18E2
100 2.03E2 2.02E2
500 1.86E2 1.85E2
Advertising
127GB
10 1.72E7 Failed
100 1.30E7 Failed
500 1.03E7 Failed
Synthetic
1100GB
10 3.81E7 Failed
100 3.51E7 Failed
500 2.81E7 Failed
Running Time vs. Number of Clusters
~10x Faster!

Built-In ML Algorithm Example
amazon-sagemaker-examples/introduction_to_amazon_algorithms/deepar_synthetic
Time Series Forecasting using DeepAR Forecasting algorithm
Amazon uses forecasting to drive business decisions
• Forecasting the product and labor in fulfillment centers
• Elastically scaling AWS compute and storage capacity for AWS customers
Benefits
• Cold start forecasting – forecasting on a new time series using multiple related
time series in training data
• Probabilistic forecasting – forecasting “the amount of sneakers sold in a week is
between X and Y with Z% probability”

Built-In ML Algorithms Example
amazon-sagemaker-examples/introduction_to_amazon_algorithms/deepar_synthetic
Time Series Forecasting using DeepAR Forecasting algorithm
Training Data
Test Data
Training at
SageMaker
Model Artifact
Deployment
at SageMaker
Amazon SageMakerAmazon ECR

Built-In ML Algorithms
amazon-sagemaker-workshop/Image-classification-transfer-learning.ipynb
Classifying images using Image Classification algorithm
Training Mode
• Full training using your own data only
• Transfer learning using your own data with ImageNet pre-trained model
Demo Scenario
• CALTECH 256 Training dataset (30607 images with 256 categories)
(http://www.vision.caltech.edu/Image_Datasets/Caltech256/)
• Use 15420 images for training

Built-In ML Algorithms – Create Job

Built-In ML Algorithms – Hyper-Parameters

Built-In ML Algorithms – Input Data

Built-In ML Algorithms – Input Data II

Built-In ML Algorithms (2/2)
label - mars
probability – 0. 1373346447944641
Inference result of a model trained from scratch

label - mars
probability – 0. 1373346447944641
Inference result of a model trained from scratch
15,420 training images are NOT sufficient.
Gather more training
images
Conduct Transfer
Learning

Conduct Transfer
Learning

label – bathtub
probability - 0.998812198638916
Inference result of a model from Transfer Learning

Bring Your Own Script
(TensorFlow/Apache MXNet)

Bring Your Own Script (TensorFlow)
amazon-sagemaker-workshop/TensorFlow_Distributed_MNIST.ipynb
Deep Learning Framework: TensorFlow
Neural Network Model: Bring your own network
Dataset: MNIST
Training Mode: Distributed Training with 2 or more nodes
model_fn
MNIST
Training Data
mnist.py
Defines network architecture
Main script used for training and hosting
• model_fn : a function that contains all the logic to support training, evaluation, and
prediction
• train_input_fn : preprocess and load training data
• eval_input_fn : preprocess and load evaluation data
• serving_input_fn : defines the features to be passed to the model during prediction

Bring Your Own Script (TensorFlow)
amazon-sagemaker-examples/sagemaker-python-sdk/tensorflow_resnet_cifar10_with_tensorboard
Deep Learning Framework: TensorFlow
Neural Network Model: Bring your own model
Dataset: MNIST
Training Mode: Distributed Training with 2 or more nodes
MNIST
Training Data
Training at
SageMaker
Model Artifact
Deployment
at SageMaker
Amazon SageMaker
Mnist.py
Amazon ECR

Bring Your Own Container (Any ML/DL
Framework)

Bring Your Own Container (R)
sample-notebooks/advanced_functionality/r_bring_your_own
Any Machine Learning or Deep Learning framework
Any Programming Language
mars.R
plumber.R
Dockerfile
Functions for training (fit) or serving the model
Using plumber package to create a lightweight HTTP server for serving
Defining a Docker container image for Training and Hosting

Bring Your Own Container (R)
sample-notebooks/advanced_functionality/r_bring_your_own
Use the same method for training and hosting model
mars.R
plumber.R
Dockerfile
Build&PushDocker
Image
Amazon ECR
Training Data
Training at
SageMaker
Model Artifact
Deployment
at SageMaker
Amazon SageMaker

Hyper-Parameter Optimization (HPO)

Tuning Job Parameters
{
"Strategy": "Line"|"Random"|"Fill”|"Bayesian",
"TuningJobObjective": {
"Type": "Maximize"|"Minimize",
"MetricName": "string"
},
"ResourceLimits": {
"MaxNumberOfTrainingJobs": integer,
"MaxParallelTrainingJobs": integer,
"MaxClockTimeInHours": integer },
"ParameterRanges": {...}
}

Grid Search vs. Random Search
Core illustration from Random Search for Hyper-Parameter Optimization by Bergstra and Bengio.

Random Search vs. Fill Search

Bayesian Search

Building Deep Learning Applications with TensorFlow and SageMaker on AWS - Tel Aviv Summit 2018

Building Deep Learning Applications with TensorFlow and SageMaker on AWS - Tel Aviv Summit 2018

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Building Deep Learning Applications with TensorFlow and SageMaker on AWS - Tel Aviv Summit 2018

Similar to Building Deep Learning Applications with TensorFlow and SageMaker on AWS - Tel Aviv Summit 2018 (20)

More from Amazon Web Services

More from Amazon Web Services (20)

Building Deep Learning Applications with TensorFlow and SageMaker on AWS - Tel Aviv Summit 2018