Business are continuously looking for ways to leverage artifical intelligence to help scale their customer service and support departments. In this session we will step through the process of building a Virtual Concierge experience, powered by Amazon Sumerian, that is able to recognise a visitor at the edge with the AWS DeepLens. You will gain an understanding of the machine learning algorithms that underpin this solution.

1. S U M M I T
SYDNEY

2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Build an AI virtual concierge
Julian Bright
Solutions Architect, ISV
Amazon Web Services

3. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
What you’ll get out of this session
• An overview of the AI and machine learning services
• Why an AI virtual concierge?
• An understanding of how facial recognition works
• Learn how to build an AI virtual concierge
• See live demo

4. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I TS U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Put machine learning in the
hands of every developer
Our mission at AWS

5. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Some of our machine learning customers…

6. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
The Amazon machine learning stack: broadest and
deepest set of capabilities
AI SERVICES
Easily add intelligence to applications without machine learning skills
Vision | Documents | Speech | Language | Chatbots | Forecasting | Recommendations
ML SERVICES
Build, train, and deploy machine learning models fast and at scale
Data labelling | Pre-built algorithms and notebooks | One-click training and deployment
ML FRAMEWORKS AND
INFRASTRUCTURE
Flexibility and choice, highest-performing infrastructure
Support for ML frameworks | Compute options purpose-built for machine learning

7. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
The Amazon machine learning stack: broadest and
deepest set of capabilities
AI SERVICES
ML SERVICES
Build, train, and deploy machine learning models fast and at scale
Data labeling | Pre-built algorithms and notebooks | One-click training and deployment
ML FRAMEWORKS AND
INFRASTRUCTURE
Flexibility and choice, highest-performing infrastructure
Support for machine learning frameworks | Compute options purpose-built for ML
A M A Z O N
R E K O G N I T I O N
I M A G E
A M A Z O N
P O L L Y
A M A Z O N
T R A N S C R I B E
A M A Z O N
T R A N S L A T E
A M A Z O N
C O M P R E H E N D
& C O M P R E H E N D
M E D I C A L
A M A Z O N
L E X
A M A Z O N
R E K O G N I T I O N
V I D E O
Vision Speech Chatbots
A M A Z O N
F O R E C A S T
A M A Z O N
T E X T R A C T
A M A Z O N
P E R S O N A L I S E
Language Forecasting Recommendations

8. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
The Amazon machine learning stack: broadest and
deepest set of capabilities
AI SERVICES
ML SERVICES
ML FRAMEWORKS AND
INFRASTRUCTURE
Flexibility and choice, highest-performing infrastructure
Support for machine learning frameworks | Compute options purpose-built for ML
A M A Z O N
R E K O G N I T I O N
I M A G E
A M A Z O N
P O L L Y
A M A Z O N
T R A N S C R I B E
A M A Z O N
T R A N S L A T E
A M A Z O N
C O M P R E H E N D
& C O M P R E H E N D
M E D I C A L
A M A Z O N
L E X
A M A Z O N
R E K O G N I T I O N
V I D E O
Vision Speech Chatbots
A M A Z O N
F O R E C A S T
A M A Z O N
T E X T R A C T
A M A Z O N
P E R S O N A L I S E
Language Forecasting Recommendations
A M A Z O N
S A G E M A K E R
B U I L D T R A I N D E P L O Y
Pre-built algorithms and notebooks
Data labeling (A M A Z O N G R O U N D T R U T H )
Algorithms and models ( A W S M A R K E T P L A C E
F O R M A C H I N E L E A R N I N G )
One-click model training and tuning
Optimisation (A M A Z O N N E O )
Reinforcement learning
One-click deployment & hosting

9. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
The Amazon machine learning stack: broadest and
deepest set of capabilities
AI SERVICES
ML SERVICES
ML FRAMEWORKS AND
INFRASTRUCTURE
A M A Z O N
R E K O G N I T I O N
I M A G E
A M A Z O N
P O L L Y
A M A Z O N
T R A N S C R I B E
A M A Z O N
T R A N S L A T E
A M A Z O N
C O M P R E H E N D
& C O M P R E H E N D
M E D I C A L
A M A Z O N
L E X
A M A Z O N
R E K O G N I T I O N
V I D E O
Vision Speech Chatbots
A M A Z O N
F O R E C A S T
A M A Z O N
T E X T R A C T
A M A Z O N
P E R S O N A L I S E
Language Forecasting Recommendations
A M A Z O N
S A G E M A K E R
B U I L D T R A I N D E P L O Y
Pre-built algorithms and notebooks
Data labeling (A M A Z O N G R O U N D T R U T H )
Algorithms and models ( A W S M A R K E T P L A C E
F O R M A C H I N E L E A R N I N G )
One-click model training and tuning
Optimisation (A M A Z O N N E O )
Reinforcement learning
One-click deployment & hosting
F r a m e w o r k s I n t e r f a c e s I n f r a s t r u c t u r e
A M A Z O N
E C 2 P 3
& P 3 d n
A M A Z O N
E C 2 C 5
F P G A s A W S
G R E E N G R A S S
A M A Z O N
E L A S T I C
I N F E R E N C E

10. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
The Amazon machine learning stack: broadest and
deepest set of capabilities
AI SERVICES
ML SERVICES
ML FRAMEWORKS AND
INFRASTRUCTURE
A M A Z O N
R E K O G N I T I O N
I M A G E
A M A Z O N
P O L L Y
A M A Z O N
T R A N S C R I B E
A M A Z O N
T R A N S L A T E
A M A Z O N
C O M P R E H E N D
& C O M P R E H E N D
M E D I C A L
A M A Z O N
L E X
A M A Z O N
R E K O G N I T I O N
V I D E O
Vision Speech Chatbots
A M A Z O N
F O R E C A S T
A M A Z O N
T E X T R A C T
A M A Z O N
P E R S O N A L I S E
Language Forecasting Recommendations
A M A Z O N
S A G E M A K E R
B U I L D T R A I N D E P L O Y
Pre-built algorithms and notebooks
Data labeling (A M A Z O N G R O U N D T R U T H )
Algorithms and models ( A W S M A R K E T P L A C E
F O R M A C H I N E L E A R N I N G )
One-click model training and tuning
Optimisation (A M A Z O N N E O )
Reinforcement learning
One-click deployment & hosting
F r a m e w o r k s I n t e r f a c e s I n f r a s t r u c t u r e
A M A Z O N
E C 2 P 3
& P 3 d n
A M A Z O N
E C 2 C 5
F P G A s A W S
G R E E N G R A S S
A M A Z O N
E L A S T I C
I N F E R E N C E

11. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Customer reception areas

13. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Visitor management technology

14. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

15. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Virtual concierge experience
1.
Recognise
visitors
2. Welcome
visitor
3. Lookup
calendar
4. Contact
host
5. Notify
visitor

16. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Virtual concierge experience
1.
Recognise
visitors
2. Welcome
visitor
3. Lookup
calendar
4. Contact
host
5. Notify
visitor

17. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
How does face recognition work?
Object detection Image classification

18. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Face recognition with machine learning
1 Detect > 2 Align > 3 Represent > 4 Classify
https://research.fb.com/publications/deepface-closing-the-gap-to-human-level-performance-in-face-verification/

19. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Face recognition probability
99.99% 98%

20. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon SageMaker Neo: Train once, run anywhere
Neo

21. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Sagemaker model compilation
from sagemaker import get_execution_role, Session
from sagemaker.mxnet.model import MXNetModel
# Upload pre-trained model
role = get_execution_role()
model_data = Session().upload_data(path='model.tar.gz', key_prefix='model’)
model = MXNetModel(model_data=model_data, role=role, entry_point='predict.py’)
Pre-trained model
Custom inference code

22. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Sagemaker model compilation
from sagemaker import get_execution_role, Session
from sagemaker.mxnet.model import MXNetModel
# Upload pre-trained model
role = get_execution_role()
model_data = Session().upload_data(path='model.tar.gz', key_prefix='model’)
model = MXNetModel(model_data=model_data, role=role, entry_point='predict.py’)
# Compile for runtime
compiled_model = model.compile(job_name='neo', role=role, target_instance_family='ml_m4’,
input_shape={'data':[1,3,112,112]}, output_path='compiled’)
Choose compile target

23. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Sagemaker model compilation
from sagemaker import get_execution_role, Session
from sagemaker.mxnet.model import MXNetModel
# Upload pre-trained model
role = get_execution_role()
model_data = Session().upload_data(path='model.tar.gz', key_prefix='model’)
model = MXNetModel(model_data=model_data, role=role, entry_point='predict.py’)
# Compile for runtime
compiled_model = model.compile(job_name='neo', role=role, target_instance_family='ml_m4’,
input_shape={'data':[1,3,112,112]}, output_path='compiled’)
# Deploy the compiled model
predictor = compiled_model.deploy(initial_instance_count=1, instance_type='ml.m4.xlarge')
Choose instance type

24. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Model deployment for
Amazon DeepLens
1. Load pre-trained model in
Amazon SageMaker Notebook
2. Compile Amazon SageMaker Neo
model to target Amazon
DeepLens
3. AWS IOT Greengrass to deploy
model and AWS Lambda
4. Face detection at the Edge

25. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Greengrass and AWS Lambda performance
# Greengrass infinite loop
while True:
frame = get_frame()
bbox, threshold = detect_face(frame)
if threshold > detect_threshold:
face = crop(frame, bbox)
if classify(face) > sim_threshold:
upload_face()
draw_frame(bbox, face_name)
publish_metrics()
1000
28
22
6
3
1 10 100 1000
Upload Face
Detect Face
Classify Face
Read Frame
Publish Metrics
Time in milliseconds

26. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Greengrass and AWS Lambda performance
# Greengrass infinite loop
while True:
frame = get_frame()
bbox, threshold = detect_face(frame)
if threshold > detect_threshold:
face = crop(frame, bbox)
if classify(face) > sim_threshold:
upload_face()
draw_frame(bbox, face_name)
publish_metrics()
1000
28
22
6
3
1 10 100 1000
Upload Face
Detect Face
Classify Face
Read Frame
Publish Metrics
Time in milliseconds
Detect face

27. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Greengrass and AWS Lambda performance
# Greengrass infinite loop
while True:
frame = get_frame()
bbox, threshold = detect_face(frame)
if threshold > detect_threshold:
face = crop(frame, bbox)
if classify(face) > sim_threshold:
upload_face()
draw_frame(bbox, face_name)
publish_metrics()
Crop and
classify face
1000
28
22
6
3
1 10 100 1000
1
2
3
4
5
Time in milliseconds

28. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Greengrass and AWS Lambda performance
# Greengrass infinite loop
while True:
frame = get_frame()
bbox, threshold = detect_face(frame)
if threshold > detect_threshold:
face = crop(frame, bbox)
if classify(face) > sim_threshold:
upload_face()
draw_frame(bbox, face_name)
publish_metrics()
1000
28
22
6
3
1 10 100 1000
Upload Face
Detect Face
Classify Face
Read Frame
Publish Metrics
Time in milliseconds
Upload face to
Amazon S3

29. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Face registration and detection
Amazon
DynamoDB
Visitors
SNS
Amazon S3
Visitor Face
Amazon API
Gateway
Register
Amazon
SageMaker
Endpoint
2. Index face
Amazon
Rekognition
Collection
4. Save visitor 7.Face detected
SNS
Recognise
Face
Amazon SNS
Face Detected
AWS
DeepLens
Amazon S3
Visitor Model
Returning
Visitor
New
Visitor
3. Classify face

30. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

31. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Virtual concierge experience
1.
Recognise
visitors
2. Welcome
visitor
3. Lookup
calendar
4. Contact
host
5. Notify
visitor

32. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
AWS Step Functions
“Serverless” visual workflow management
• Coordinate multiple AWS Lambda functions
• Handle errors with automatic retry support
• Run parallel tasks or activities for callbacks
• Logs the state of each step, to help debug
Task
Choice
Handle errors
Parallel tasks
Start
End

33. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon Sumerian as a frontend
Instructional design
Brand engagement
Data visualisation
Simulation
Virtual showcases
Virtual tours
Digital signage Customer service
agents and chatbots

34. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Receptionist workflow
Amazon
DynamoDB
Appointment
Amazon SNS
Face Detected
AWS API
Gateway
Notification
5. Get appointment
Amazon SNS
Email Host
4. Welcome visitor
Host
Amazon
Sumerian
Scene
Visitor
Amazon
Rekognition
Collection
6. Notify host
7. Host reply
AWS Step
Function
Workflow
Amazon
DynamoDB
Appointment
2. Create session 3. Start workflow

35. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

36. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
“Shootsta is all about using technology
to enhance our client experience.”
Tim Moylan
CTO, Shootsta

37. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Summary
Registration
• Amazon S3
• Amazon
Rekognition
• Amazon
Sagemaker Neo
Recognise
• AWS IoT
Greengrass
• AWS DeepLens
• Amazon SNS
Workflow
• AWS Step
Functions
• AWS Lambda
• Amazon
DynamoDB
Frontend
• Amazon
Sumerian
• Amazon
Cognito
• Amazon SQS

38. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Call to action
Dev Labs: Build an AI Virtual Concierge
https://github.com/stephensalim/devlabs-vc
Customer and Partner Sumerian exhibits
Visy and Olikka
Face Recognition with MXNet and Amazon Sagemaker Neo
https://github.com/isvlabs/aiml-lab/

39. Thank you!
S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Julian Bright