Join us for a deep dive on how to use Amazon Rekognition for real world image analysis. Learn how to integrate Amazon Rekognition with other AWS services to make your image libraries searchable. Also learn how to verify user identities by comparing their live image with a reference image, and estimate the satisfaction and sentiment of your customers. We also share best practices around fine-tuning and optimizing your Amazon Rekognition usage and refer to AWS CloudFormation templates.

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS re:INVENT
Deep dive on Amazon Rekognition
architectures for image analysis
M a y a n k T h a k k a r
A W S S o l u t i o n s A r c h i t e c t
M C L 3 1 8
N o v e m b e r 2 9 , 2 0 1 7

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What to expect from the session
• Why Amazon Rekognition?
• What is Amazon Rekognition?
• Real-world use cases
• Demo
• Amazon Rekognition Ecosystem

3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Rekognition
Benefits

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Why should I use Amazon Rekognition?
Power with ease
Artificial intelligence
at the core
Scalable image
analysis
Integrated with
popular AWS
services
Low cost
Decrease time-to-
market

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Rekognition
E x t r a c t r i c h m e t a d a t a f r o m v i s u a l c o n t e n t

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The Amazon ML / AI Stack
Services
Platforms
Frameworks
Infrastructure
Apache
MXNet
Torch
Cognitive
Toolkit
KerasTheano
Caffe2
& Caffe
TensorFlow
AWS Deep Learning AMI
GPU MobileCPU IoT
Amazon ML Amazon ECSSpark & EMR
Amazon
Kinesis
AWS Batch
Vision Speech LanguageAmazon Rekognition

7. Amazon Rekognition: What does it offer?
Object and scene detection
Facial analysis
Image moderation
Face comparison
Facial recognition
Celebrity recognition

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Labels-based
operations
• DetectLabels
• DetectModerationLabels
Face-based
operations
• DetectFaces
• CompareFaces
• IndexFaces
• DeleteFaces
• SearchFacesByImage
• SearchFaces
• RecognizeCelebrities
• GetCelebrityInfo
Collection
management
operations
• CreateCollection
• DeleteCollection
• ListCollection
APIs
- Non Storage Based
- Not counted towards images processed

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Real-world use cases
O p e n i n g n e w f r o n t i e r s

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
1. Policing user-generated content

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Age range – 26–43 years
Wearing glasses – 99.9%
Eyes closed – 94%
Mouth open – 96%
Eyes closed – 94%
1. Policing user-generated content
Barrack Obama –
100%
Not smiling – 60.3%
Female – 100%

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Current challenges
• Manual process for checking images – Labor intensive
• Non-uniformity – Results vary from resource to resource
• Scalability – Difficult to keep up with the rate of image generation

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS solution
2. Submit
picture
4. DetectFaces
8. SearchFaces
- Blacklist
- Whitelist
- Duplicate check
- Persons of
interest
1. Live pic
3. Store live
pic
Amazon
Rekognition
Amazon
Rekognition
Lambda Step functions
5.RecognizeCele
brities
Amazon
Rekognition
7.DetectModeration
Labels
Amazon
Rekognition
9. Store metadata
and analysis Amazon
DynamoDB
Elasticsearch
Blacklist images

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Demo

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Challenges solved
• Scalable solution – Pay as you go
• Achieve greater levels of uniform policing
• Improve security – Search against known blacklists
• Add new features and improve customer experience
• Finding best user picture from library

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How much will it cost to process 1K images?
Service Cost Comments
Amazon S3 $0.08 3 MB per pic X 1K images X $0.023 / GB - month
AWS Step Functions $0.23 11 state transitions per image X 1K images X $0.025 /1000 transitions
Amazon Rekognition $6.01 5 APIs (with images) processed per iteration X 1.2K iterations = $6
1K Face metadata stored X $0.01 per 1000 face metadata stored = $0.01
Amazon Elasticsearch $2.20 24 hours X [t2.medium ($0.073 per hour) with 100 GB Amazon EBS ($13.5
per month)]
Amazon DynamoDB $0 10-GB data size with 100-KB item size, 25 read/sec, and 25 writes / sec –
free tier eligible
AWS Lambda $0 20 invocations per image X 1K images = (~25,200 GB-seconds) – free tier
eligible
Total cost to store, analyze, and index 1K images: ~ $8.52 #
* - Including 20% failure rates
# – Including free tier

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
2. Optimize check-in / check-out process

18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
2. Optimize check-in / check-out process
Face matched -
97%
New customerFace matched - 94%
Face matched -
99%
VIP customer

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Current challenges
• Slow and lengthy
• Error prone – Depends on traditional inspection methods
• Security risk – Relatively easy to circumvent
• Manual labor intensive – Staffing challenges

20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS solution
1a. Scan ID
2. Submit
details
4. Verify ID
5. Return
reference picture
6a. IndexFaces
ExternalImageID
7. CompareFaces
8. Store Metadata
and final decision
1b. Live pic
3. Store live
pic
API Gateway
Amazon
Rekognition
Amazon
Rekognition
Amazon
DynamoDB
Secure verification source
Lambda
6b. FaceId

21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Challenges solved
• Faster streamlined process – Decrease wait times
• Don’t compromise security
• Ability to check against a known blacklist using the SearchFaces API
• Automate check-in / check-out process – curb proxies
• Add more value – Re-use your resources effectively
• Custom service – Provide VIP service to repeat customers

22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
3. Live demographic analysis

23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
3. Live demographic analysis
Female: 100%
Happy: 99.9%
Age Range: 11–18
Female: 100%
Happy: 99.6%
Age Range: 27–44
Male: 99.7%
Happy: 97.9%
Age Range: 26–43

24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Current challenges
• Paper / online forms -> resource intensive
• Low fulfillment rate -> imperfect data
• Lengthy execution-to-results workflow

25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS solution
2. Submit
picture
4. DetectFaces
5. SearchFaces -
Blacklist
6. Store metadata
and analysis
1. Live pic
3. Store live
pic
API Gateway
Amazon
Rekognition
Amazon
Rekognition
Amazon
DynamoDB
Lambda
Amazon
Redshift
Amazon
QuickSight

26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Challenges solved
• Achieve near 100% coverage – Use strategic camera
placement
• Improve Security – Search against known blacklists
• Unlock new behavioral patterns – Co-relate
demographic data with other data
• Near real-time analysis – Respond immediately, based
on current demographics

27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Real-time store heat map
Section A
Section D
Section B
Section C
Section F
Section E
Section G
8–19 years
20–35 years
Male
Female
Time

28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
4. Person(s) of interest

29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
4. Person(s) of interest
CelebrityDetected – 95%
Hypothetical person
of interest – 86%

30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Solution
aws rekognition recognize-celebrities –image “S3Object={Bucket=mybucket,Name=cam.jpg}”
aws rekognition search-faces-by-image –image “S3Object={Bucket=mybucket,Name=cam.jpg}”
--collection-id “persons-of-interest"
aws rekognition create-collection --collection-id “persons-of-interest”
aws rekognition index-faces --image “S3Object={Bucket=mybucket,Name=subject.jpg}”
--collection-id “persons-of-interest”
{
"FaceMatches": [
{"Face": {"BoundingB
"Height":
0.2683333456516266,
"Left":
0.5099999904632568,
"Top":
0.1783333271741867,
"Width":
0.17888888716697693},
"
1
CreateCollection
DeleteCollection
ListCollections
{
"FaceMatches": [
{"Face": {"BoundingB
"Height":
0.2683333456516266,
"Left":
0.5099999904632568,
"Top
CompareFaces
DetectFaces
DetectLabels
DetectModerationLabels
GetCelebrityInfo
RecognizeCelebrities
2
IndexFaces
SearchFaces
SearchFacesByImage
ListFaces
DeleteFaces
3

31. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Other use cases
People counting: Eye-level cameras capture people entering and exiting a
place. Amazon Rekognition can then be used to keep a live count of people
inside the place at any given time.
Suspicious object left behind: Amazon Rekognition receives an image of
each person, which is used to associate individuals with suspicious objects
(e.g., backpacks). Each time a new face is matched, the application
compares the metadata for that individual and issues an alert when the
suspicious objects are no longer detected.

32. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Rekognition
Best practices

33. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Best practices: Interfacing with Amazon Rekognition
• Max image size
• Amazon S3 : 15 MB
• API calls: 5 MB (base64 encoded)
• Image format
• PNG or JPEG
• Image resolution
• Min 80 px, 1024 (x or y) px recommended
• Size of face should occupy ~5%+ of image for detection
• Collections are for faces! (not cats, cartoons, …)
• For AWS CLI, upload images to Amazon S3
• AWS CLI cannot pass image bytes
…

34. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Best practices: Interfacing with Amazon Rekognition
• Max number of faces in a single face collection is 1 million
• Latency still under a second!
• Max matching faces the search API returns = 4096
• Use Delete* with caution
• Use IAM to manage permissions
• Use Amazon CloudWatch to observe/alert on Amazon
Rekognition Metrics
• Know the Amazon Rekognition API and save some $$$
• RecognizeCelebrities returns face information from image (max 15),
no need to call DetectFaces if you are just looking for presence of
faces
…

35. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Rekognition
Ecosystem

36. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Integration with other services
Decoupling
Amazon
SQS
Media processing
Amazon API
Gateway
AWS Batch
Amazon
EC2
Amazon
ECS
ComputeMicroservices
Storage
Data stores
Amazon
S3
AWS
Lambda
Amazon
SNS
Amazon
DynamoDB
AWS
Elemental
Amazon Elastic
Transcoder
Amazon
EFS
Amazon
Kinesis
Amazon
Elasticsearch
Amazon RDS

37. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Summary
• Amazon Rekognition
• Benefits
• Feature set
• Practical use cases
• Policing user content
• Optimize check-in / check-out process
• Live sentiment analysis
• Persons of interest
• Best practices

38. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
References
• Amazon Rekognition
• https://aws.amazon.com/rekognition/
• Amazon Rekognition on AWS AI blog
• https://aws.amazon.com/blogs/ai/tag/amazon-rekognition/
• Serverless Image Recognition processing backend
• https://github.com/awslabs/lambda-refarch-imagerecognition
• Policing user content demo code
• https://github.com/aws-samples/amazon-rekognition-policing-user-
content

39. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
More on Amazon Rekognition at re:Invent
• MCL330 - Humans, Plants, and Chairs: Insights from Analyzing over 30 Million
Instagram Posts with Amazon Rekognition
• Chalk Talk: November 30 , 11.30 am, Aria, Level 3, Starvine 9
• ARC326 - Create a Serverless Image Processing Platform – Workshop
• Workshop: November 30 , 2.30 pm, Venetian, Level 1, Marco Polo 805
• MCL403 - Building an Intelligent Multi-Modal User Agent with Voice, Natural
Language Understanding, and Facial Animation
• Chalk Talk: November 30 , 5.30 pm, Aria, Level 3, Starvine 9

40. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
THANK YOU!