This document discusses building machine learning models for prediction on edge devices like mobile phones and IoT devices. It outlines 5 approaches: 1) Calling application services, 2) Building/training models in the cloud and deploying them to devices, 3) Hosting models behind APIs in the cloud, 4) Utilizing platform APIs on devices, and 5) Passing input data directly to AWS Lambda. It also discusses challenges like limited resources on edge devices and the benefits of performing ML on the edge like latency, bandwidth, and privacy. Finally, it introduces Amazon SageMaker as a fully managed service for training and hosting custom models.

1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Top 5 Ways to Build Machine
Learning Prediction on the Edge
for Mobile and IoT
Dennis Hills
Developer Advocate, AWS Mobile
Pop-up Loft

2. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Smart Applications
Sense
Generate and receive rich data
about the environment
Infer
Extract relevance from huge
amounts of data in real time
Action
Take smart actions

3. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Benefits of ML on the Edge
Latency Bandwidth Availability Privacy

4. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Deep Learning challenges at the Edge
• Resource-constrained devices
• CPU, memory, storage, power consumption.
• Network connectivity
• Availability, cost, bandwidth, latency.
• On-device prediction may be the only option.
• Deployment
• Updating code and models in mobile apps or a
fleet of devices is not easy.

5. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Prediction/Inference on the Edge

6. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Call Application Services 1
PLATFORM SERVICES
APPLICATION SERVICES
FRAMEWORKS & INTERFACES
Caffe2 CNTK
Apache
MXNet
PyTorch
TensorFlo
w
Torch Keras Gluon
AWS Deep Learning AMIs
Amazon SageMaker AWS DeepLens
Transcribe Translate Polly Comprehend LexRekognition

7. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Build/Train in Cloud – Deploy Model to Device 2
For iOS => Use Core ML to access a local on-device
trained model.
For Android => Use TensorFlow Lit to interact with the ML
Model on the device

8. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Build/Train in Cloud – Host Model Behind API 3
• Train a model in SageMaker (or bring your own).
• Deploy it to a prediction endpoint.
• Invoke the HTTP endpoint from your devices.
Best when
Devices are not powerful enough for local inference.
Models can’t be easily deployed to devices.
Additional cloud-based data is required for prediction.
Prediction activity must be centralized.
Requirements
Network is available and reliable.
Devices support HTTP.

9. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Utilize Platform APIs 4
The Vision framework from Apple performs face and face landmark detection,
text detection, barcode recognition, image registration, and general feature
tracking. Vision also allows the use of custom Core ML models for tasks like
classification or object detection.
Use Apple’s Natural Language framework to perform tasks like language and
script identification, tokenization, lemmatization, parts-of-speech tagging, and
named entity recognition.

10. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Pass Input Data Directly to AWS Lambda 5
• Train a model in SageMaker (or bring your own).
• Host it in S3 (or embed it in a Lambda function).
• Write a Lambda function performing prediction.
• Invoke it through AWS IoT/ API Gateway.
Best when
Devices can support neither HTTP nor local
inference (e.g. Arduino).
Costs must be kept as low as possible.
Requirements
Network is available and reliable
(MQTT is less demanding than HTTP).
Devices are provisioned in AWS IoT (certificate, keys).

11. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Recap
Call Application Services
Build/Train in Cloud – Deploy Model to Device
Build/Train in Cloud – Host Model Behind API
Utilize Platform APIs
Pass Input Data Directly to AWS Lambda
1
2
3
4
5

12. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Build/Train Custom Models

13. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
End-to-end
Machine Learning
Platform
Zero setup Flexible model
training
Pay by the second
Introducing Amazon SageMaker

14. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon-optimized
algorithms using
the AWS SDK…
… or Apache Spark
IM Estimators
Bring your own
deep learning
script…
… or your custom
algorithm Docker
image
Distributed training that works with you

15. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon SageMaker
1 2 3 4
I I I I
Notebook Instances Algorithms ML Training Service ML Hosting Service

16. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Start with notebook samples

17. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Modify to access your data sources

18. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Train your model

19. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Deploy your model

20. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Perform inferences

21. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
From SageMaker Notebooks
Parameters
Hardware
Start Training
Host model

22. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank you
Get Started:
aws.amazon.com/mobile
AWS Mobile Twitter:@AWSforMobile
Dennis Hills: @dmennis