In this chalk talk, we show you how to build a connected, frictionless retail checkout experience, inspired by Amazon Go. We briefly explain the user experience, technical specifications, and we then delve into the implementation details of the project. We show you how to create SSD Object Detection and Visual Search models in Amazon SageMaker with the details of the end-to-end lifecycle of the dataset, including creation, training, tuning and deployment. We also, show you how to write AWS Lambda functions for IoT edge devices with AWS Greengrass, and how AWS Greengrass can interact simultaneously with IoT sensors, such as a door or weight. Finally, we explain the details of a sensor fusion algorithm that combines the data to the customer purchase transaction.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Create a Frictionless, Connected Retail Checkout
Experience Inspired by Amazon Go
Renato Barbosa
R&D Engineer
AWS
I O T 3 4 9 - R
Renato Pedigoni
R&D Engineer
AWS

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
R&D and Innovation
• Prototyping team
• Fail fast to innovate faster

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What are we building?
• Demonstrate a new customer experience for buying beverages
• Tenets
• Reduce payment friction
• Hands-on experience
• Customer can change their mind
• Simple selection

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What are we building?
(1) (3)(2)

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Landing our project idea
QR code to
unlock to the
door
(1)
Customer
opens the
door and take
a beverage
(2)
Customer
closes the
door
(3) (4)
Take a picture
from the top,
process the
image find
removed
products
(5)
Receipt is
displayed

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Next steps
IoT mini fridge User interfaceComputer Vision model

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Computer Vision model
Camera

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Computer Vision model
Fridge top view
(bottles)
Object detection
Identify bottles
Image classification
Identify brands for each bottle cap
Stella Stella
Stella Stella
Heineken
Heineken Water
Water

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Object detection – Synthetical dataset
Background image Object image Generated image
Image generation

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Object detection – Synthetical dataset
Annotation fileGenerated image
Pascal VoC Standard
Train/Val files

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Object detection – Hyperparameter tuning job

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Object detection – Model deploy

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Object detection – Non max suppression
Before NMS After NMS

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Image classification – Synthetical dataset
Background image Generated image
Image generation

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Image classification – Model training/deploy

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Image classification – Retrain use real images
Generated image Real images
Image labeling
Real blurry images

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
IoT hardware
• Mini fridge
• Raspberry PI
• Webcam
• Load sensor
• Temperature sensor
• Magnetic sensor
• Magnetic lock
• 4-channel relay
• RGB LED lightstrip

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
IoT software: AWS Greengrass
AWS Greengrass

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
IoT software: Lambda Functions
Lambda
Authenticate
Receives MQTT
message from UI and
unlocks the door
(1)
Lambda
DoorSensor
Reads door sensor
and sends MQTT
message when door is
closed
(2)
Lambda
PlaceOrder
Lock the door, take
picture, call ML
endpoint and presents
receipt
(3)
Lambda
SensorRecoding
Reads temperature
and weight metrics
and writes to local
InfluxDB
(4)

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
PlaceOrder Lambda definition
Subscriptions
Resources
• RPi GPIO (Lights, Lock)
• Camera
• DataFolder (Local SQLite3)
DoorSensor PlaceOrderrndexpress/{}/door/close
PlaceOrder rndexpress/{}/receipt/pending Interface
PlaceOrder rndexpress/{}/receipt/fill Interface

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
PlaceOrder Lambda execution
Triggered by topic
door/close
Publish to topic
receipt/pending
Publish to topic
receipt/fill

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Greengrass automation
• Greengo - http://greengo.io
• greengo.yaml
• Lambdas
• Subscriptions
• Resources
• Automate tasks
• greengo create
• greengo update
• greengo deploy

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
User interface
• ReactJS
• Websocket to AWS IoT using 'aws-amplify’ and Amazon Cognito
• MQTT

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

32. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Renato Barbosa
rbbarbos@amazon.com
Renato Pedigoni
renatogp@amazon.com

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.