AWS Greengrass allows running machine learning inference models locally on edge devices by (1) compiling common ML frameworks to run on devices, (2) deploying pretrained models from AWS services like SageMaker, and (3) invoking the models from Lambda functions on the edge device without connecting to the cloud. This enables tasks like image classification using cameras without transmitting data to the cloud.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Machine Learning Inference at the
Edge
Shiwei Ni
Senior Software Development Engineer
AWS Greengrass
I O T 3 2 2

3. Use case – home automation
Camera device
Edge device
Lightbulb
Device
software
Light switch
Bucket with
SageMaker
model
S3 training
data bucket
AWS Cloud

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What is AWS Greengrass?
• Event-driven programming on edge device
• Secure way of ingest data from other local devices
• Invoke Lambda with no cold start
• Use local device event to trigger Lambda
• Secure way of making call to AWS without hard-coding your credentials
• Secure way of ingesting data from other local devices
• Run lambda securely in containers
• Edge application updateable from cloud

5. Ingest data from local device into cloud
Bucket with
SageMaker
model
S3 training
data bucket
Greengrass
Deployments
AWS Cloud
Camera device
Edge device
Lightbulb
Light switch
Containerized
Lambda
Local event
Subscriptions
Greengrass Core
Securely upload filtered enriched data

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Pre-compiled ML framework

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Steps towards ML at edge with AWS Greengrass
1. Install AWS Greengrass
2. Compile ML runtime for targeted device
3. Install pre-compiled ML runtime provided by AWS Greengrass
4. Configure Greengrass group to include a machine learning resource
(Amazon Simple Storage Service (Amazon S3) or Amazon SageMaker)
5. Migrate/build inference code
6. Deploy the group

8. Deploy an inference lambda using Greengrass
Deployed Model
Bucket with
SageMaker
model
S3 training
data bucket
Greengrass
Deployments
AWS Cloud
Camera device
Edge device
Lightbulb
Predicted
Light switch
Lambda Local event
Subscriptions
Greengrass Core
Configure lambda to use model

9. new AWS Greengrass Inference Connector
• First available with image classification
• integrated with AWS SageMaker Training
• x86, ARMv7, GPU-accelerated on NVidia TX2
• A deployable edge inference service with no coding required
new AWS Greengrass Machine Learning Inference SDK
• One API call get inference result locally on the edge
• First available in Python

10. Greengrass Inference Connector – hands-off inference
Image Classification Connector
Model
Bucket with
SageMaker
model
S3 training
data bucket
Greengrass
Deployments
AWS Cloud
Camera device
Edge device
Lightbulb
Light switch
Lambda Local event
Subscriptions
Greengrass Core
Upload filtered enriched data
Serving
lambda
perform inference Configure Greengrass
Connector to use model

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Example – Using ML SDK
import greengrass_machine_learning_python_sdk as gg_ml
def infer():
logging.info('invoking DeepGreen’)
try:
resp = client.invoke_inference_service(
AlgoType=‘image-classification’,
ServiceName='imageClassification’,
ContentType='image/jpeg’,
Body=image_content
)
except gg_ml.GreengrassInferenceException as e:
logging.error(‘inference error’)
logging.info('resp: {}'.format(resp))
predictions = resp['Body'].read()
return predictions

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Sample inference output

13. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.