Presentation given at Raleigh DevDay on August 1, 2017 during the AI Session

2. WIFI: awsDevDay | PASS: CodeHappy
U P N E X T :
Amazon Rekognition
Best Practices

3. T H A N K S T O O U R F R I E N D S A T :

4. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Rekognition Best Practices
Integrating Deep Learning-based Image Analysis
into your own Applications

5. Images – Universal, Ubiquitous, & Essential
There are 3,700,000,000 internet users in 2017
1,200,000,000 photos will be taken in 2017 (9% YoY Growth)
Source: InfoTrends Worldwide

6. Amazon Rekognition
Deep learning-based image recognition service
Search, verify, and organize millions of images
Object and Scene
Detection
Facial
Analysis
Face
Comparison
Facial
Recognition
Celebrity
Recognition
Image
Moderation

7. Why use Rekognition?
• Object & Scene Detection
Photo-sharing apps can power smart searches
and quickly find cherished memories, such as
weddings, hiking, or sunsets
• Facial Analysis
Retail businesses can understand the
demographics and sentiment of in-store
customers
• Face Comparison
Hotels & hospitality businesses can provide
seamless access for guests and VIPs
• Facial Recognition
Public safety teams can leverage face collections
to automate tracking of persons of interest

8. Object & Scene Detection
Maple
Villa
Plant
Garden
Water
Swimming Pool
Tree
Potted Plant
Backyard
Flower
Chair
Coffee Table
Living Room
Indoors
Object and scene detection makes it easy for you to add features that search,
filter, and curate large image libraries.
Identify objects and scenes and provide confidence scores
DetectLabels

9. Demographic DataDemographic Data
Facial LandmarksFacial Landmarks
Sentiment ExpressedSentiment Expressed
Image QualityImage Quality
Brightness: 25.84
Sharpness: 160
General Attributes
Facial Analysis
Analyze facial characteristics in multiple dimensions
DetectFaces

10. Face Comparison
Measure the likelihood that faces are of the same person
Similarity 93%Similarity 93% Similarity 0%Similarity 0%
CompareFaces

11. Facial Recognition
Find similar faces in a large collection of images
SearchFacesByImage
Search
Index
Collection

12. Celebrity Recognition & Image Moderation
Newly released Rekognition features
Detect explicit and suggestive contentRecognize thousands of famous individuals
DetectModerationLabelsRekognizeCelebrities

13. • Digital Asset Management
• Media and Entertainment
• Travel and Hospitality
• Influencer Marketing
• Systems Integration
• Digital Advertising
• Consumer Storage
• Law Enforcement
• Public Safety
• eCommerce
• Education
Use Cases & Customers

14. Interfacing with Rekognition
Build, test and deploy for Rekognition using SDKs & API calls
aws rekognition detect-labels –image
“S3Object={Bucket=mybucket,Name=image.jpg}” |
grep -E ‘(Vehicle|Automobile|Car)’ | mail -s “Alert! Car on Property!” me@site.com
RubyiOS PythonAndroid Node.js.NET PHP AWS CLIJavaScriptJava Xamarin
Or use the AWS CLI

15. Integrating & Extending Rekognition
3rd Party Software
• AWS AI AMI
• OpenCV
• ImageMagick
• FFMPEG
• YOLO, CCV, …
AWS Services
• Amazon S3
• AWS Lambda
• AWS SQS, SNS
• AWS Batch
• Amazon EC2
AWS Partners
• Media Intelligence
• Asset Management
• Image Hosting
• Image Processing
• Brand Management

16. {
"FaceMatches": [
{"Face": {"BoundingB
"Height":
0.2683333456516266,
"Left":
0.5099999904632568,
"Top":
0.1783333271741867,
"Width":
0.17888888716697693},
"
{
"FaceMatches": [
{"Face": {"BoundingB
"Height":
0.2683333456516266,
"Left":
0.5099999904632568,
"Top":
0.1783333271741867,
"Width":
0.17888888716697693},
"
Rekognition APIs – Overview
Rekognition’s computer vision API operations can be grouped into
Non-storage API operations, and Storage-based API operations
CompareFaces
DetectFaces
DetectLabels
DetectModerationLabels
GetCelebrityInfo
RecognizeCelebrities
Non-storage API Operations
CreateCollection
DeleteCollection
DeleteFaces
IndexFaces
ListCollections
SearchFaces
SearchFacesByImage
Storage-based API Operations
ListFaces

17. Collections and Access Patterns
Logging - public events; visitor logs; digital libraries
• One large collection per event/time period
• Enables wide searches
Social Tagging - photo storage and sharing
• One collection per application user
• Enables automated friend tagging
Person Verification - employee gate check
• One collection for each person to be verified
• Enables detection of stolen/shared IDs

18. {
"FaceMatches": [
{"Face": {"BoundingB
"Height":
0.2683333456516266,
"Left":
0.5099999904632568,
"Top":
0.1783333271741867,
"Width":
0.17888888716697693},
"
CreateCollection
DeleteCollection
DeleteFaces
IndexFaces
ListCollections
SearchFaces
SearchFacesByImage
ListFaces
{
"FaceMatches": [
{"Face": {"BoundingB
"Height":
0.2683333456516266,
"Left":
0.5099999904632568,
"Top":
0.1783333271741867,
"Width":
0.17888888716697693},
"
CompareFaces
DetectFaces
DetectLabels
DetectModerationLabels
GetCelebrityInfo
RecognizeCelebrities
Rekognition APIs – Advanced Usage
Decision trees and processing pipelines
Why?
• Many use cases require more than a single
operation to arrive at actionable data
How?
• S3 event notifications, Lambda, Step Functions
• Dynamo for persistent pipeline storage
• Augmenting results with 3rd Party AI/ML
• OpenCV, MXNet, etc. on EC2 Spot, ECS, AI/ML AMI
Sample Use Cases
• Person of interest near a celebrity
• Multi-pass motion detection enhancement
• Subjects leaving a location without possessions
https://aws.amazon.com/amazon-ai/amis

19. Rekognition APIs – Advanced Usage
Person of Interest Near a Celebrity

20. aws rekognition recognize-celebrities –image ”S3Object={Bucket=mybucket,Name=cam.jpg}”
aws rekognition search-faces-by-image –image ”S3Object={Bucket=mybucket,Name=cam.jpg}”
--collection-id ”persons-of-interest"
aws rekognition create-collection --collection-id ”persons-of-interest”
aws rekognition index-faces --image ”S3Object={Bucket=mybucket,Name=subject.jpg}”
--collection-id ”persons-of-interest”
Rekognition APIs – Advanced Usage
Person of Interest Near a Celebrity
{
"FaceMatches": [
{"Face": {"BoundingB
"Height":
0.2683333456516266,
"Left":
0.5099999904632568,
"Top":
0.1783333271741867,
"Width":
0.17888888716697693},
"
CompareFaces
DetectFaces
DetectLabels
DetectModerationLabels
GetCelebrityInfo
RecognizeCelebrities
2
{
"FaceMatches": [
{"Face": {"BoundingB
"Height":
0.2683333456516266,
"Left":
0.5099999904632568,
"Top":
0.1783333271741867,
"Width":
0.17888888716697693},
"
CreateCollection
DeleteCollection
DeleteFaces
IndexFaces
ListCollections
SearchFaces
SearchFacesByImage
ListFaces
3
1

21. Encryption & Security
• APIs - Non-storage vs. storage-based
API operations
• Encryption - S3 in-transit and at-rest
w/ HTTPS, KMS
• Tampering - Lock down IAM roles and
policies Content - purge or lifecycle to
Glacier w/vault lock
• Use least common privilege -Lambda,
EC2 and other infrastructure
• Hydration - EBS encryption for boot,
data and snapshotted volumes
Best Practices

22. Encryption & Security
Best Practices
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"rekognition:CreateCollections",
"rekognition:IndexFaces
],
"Resource": "*"
}
]
}
Lambda : Collection Index
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"rekognition:ListCollections",
"rekognition:ListFaces",
"rekognition:SearchFaces",
"rekognition:SearchFacesByImage"
],
"Resource": "*"
}
]
}
Lambda : Collection Search

23. Interfacing with Rekognition
• S3 input for API calls - max image size of 15MB
• 5MB limit for non-s3 (Base64 encoded) API calls
• Minimum image resolution (x or y) of 80 pixels
• Image data must be in PNG or JPG format
• Max number of faces in a single face collection is 1 million
• The max matching faces the search API returns is 4096
• Use at least 1024 (x or y) px as input – or extract regions
Size of face should occupy ~5%+ of image for detection
• Collections are for faces!
Optimizing your input & requests for best performance
…
Use Amazon CloudWatch to observe & alert off Rekognition Metrics

24. Optimizing your Input Source
Images
• Image enhancement, extraction & stabilization
• Unsharp mask, deconvolution – CPU impact
• e.g. ImageMagick, OpenCV, scikit-image
Video
• Video stabilization - motion / optical flow analysis
• Scene change detection vs. frame extraction
• Offset - PTS vs. seconds and why it matters
• e.g. FFMPEG w/deshake, vidstab, OpenCV
Optimizing your input & requests for best performance

25. Searchable Image Library
Photo Upload Amazon S3 AWS Lambda
Property Search Amazon Elasticsearch
Detect Objects & Scenes
User captures an image
for their property listing
Mobile app uploads
the image to S3
A Lambda function is triggered
and calls Rekognition
Rekognition retrieves the image from
S3 and returns labels for the property
and amenities
Lambda pushes the labels and
confidence scores to Elasticsearch
Other users can search properties
by landmarks, category, etc.
Real Estate Property Search
“ …Amazon Rekognition generates a rich set of tags directly from images of
properties. This makes it relatively simple to build a smart search feature
that helps customers discover houses based on their specific needs… “

26. Searchable Image Library
Real Estate Property Search
Photo Upload Amazon S3 AWS Lambda
Property Search Amazon Elasticsearch
Detect Objects & Scenes
User captures an image
for their property listing
Mobile app uploads
the image to S3
A Lambda function is triggered
and calls Rekognition
Rekognition retrieves the image
from S3 and returns labels for the
property and amenities
Lambda pushes the labels and
confidence scores to
Elasticsearch
Other users can search
properties by landmarks,
category, etc.
1
2
3
• Optimize the client
• Event based, decoupled infra
• Buffering - SQS, SNS, Kinesis
• Rate Control - high volume S3
image ingest
• Dynamo – scale label storage
• Elasticsearch - operational &
performance statistics
• CloudFront - search cache
AWS
Lambda
Amazon
S3
Amazon
SQS
AWS
CloudFormation
Amazon
CloudWatch
Amazon
Kinesis
Amazon
CloudFront
Amazon
DynamoDB
Amazon
ElasticSearch

27. Sentiment Analysis
Amazon RedshiftAmazon Quicksight
Live Subject In-Store Camera Application
Amazon S3
Analyze Faces
User captures an image
for their property listing
In-store cameras capture
live images of shoppers
A Lambda function is
triggered and calls
Rekognition
Rekognition analyzes the image and
returns facial attributes detected, which
include emotion and demographic detail
Return data is normalized
and staged in S3 en route
to Redshift
Marketing Reports
Periodic ingest of data
into Redshift
Regular analysis to identify
trends in demographic activity
and in-store sentiment over time
“ …The benefits for our clients are huge. We can find a
brand’s ‘content soulmate’ and better match the right content
creators to campaigns. .… “
Trend reporting for store locations

28. Sentiment Analysis
Trend reporting for store locations
• Reduce API volume - perform
motion detection and frame pre-
processing on the capture device
• Note - customer photos not
stored on AWS
• S3 - A parquet file content lake to
feed other services – EMR,
Lambda
• Resell as a service – API
gateway + Lambda + S3
Live Subject In-Store Camera Application
Amazon S3
Analyze Faces
User captures an image
for their property listing
In-store cameras
capture live images
of shoppers
A Lambda function is
triggered and calls
Rekognition
Rekognition analyzes the image
and returns facial attributes
detected, which include emotion
and demographic detail
Return data is
normalized and
staged in S3 en route
to Redshift
Marketing
Reports Periodic ingest of
data into Redshift
Regular analysis to identify
trends in demographic
activity and in-store
sentiment over time
Amazon Quicksight Amazon Redshift
1
2
3
AWS
Lambda
Amazon
S3
Amazon
SQS
AWS
CloudFormation
Amazon
CloudWatch
Amazon
ElasticSearch
Amazon
Redshift
Amazon
QuickSight
Amazon API
Gateway*

29. “ …Rekognition allow us to index twice as much content as we do currently - from 3500 hours a
year to 7500 hours a year which would allow us to index 100% of our first run content and it was
shockingly easy to set up, even with 97,000 entities from our database.… “
Face-Based User Verification
Authenticated User
Image Capture Application
Amazon S3
Compare Faces
If the similarity score is over 80%, the
application returns a green status. If not, an
alert is issued to security staff.
The application captures a
live image of each employee
as they scan their access
card
Rekognition compares the live image
and the badge image – and returns
a similarity score
The application retrieves the
user’s badge from S3
Confirm user identities by comparing their live image with a reference image

30. Face-Based User Verification
Confirm user identities by comparing their live image with a reference image
• S3 Encryption of badge images –
SSE-S3, SSE-KMS, SSE-C
• Prevent tampering with bucket
policies & IAM RO permissions
• Extend by using Rek. collections
• Cloudtrail - Logging & Auditing
w/ tamper-proof log signatures
• Tie notification into SNS/SES,
Custom CloudWatch Logs
metrics, or ElasticSearch
w/alerts
Authenticated
User
Image Capture Application
Amazon S3
Compare Faces
If the similarity score is over 80%, the
application returns a green status. If not, an
alert is issued to security staff.
The application captures
a live image of each
employee as they scan
their access card
The application retrieves the
user’s badge from S3
Rekognition compares the
live Image and the badge
Image – and returns a
similarity score
1
2
3
AWS KMS AWS
CloudTrail
AWS
Lambda
Amazon
S3
Amazon
SNS
AWS
CloudFormation
Amazon
CloudWatch
Amazon
SES

31. • Built in 3 weeks
• Indexed against 99,000 people
• Index created in one day
• Saved ~9,000 hours a year in
manual curation costs
• Live video with frame sampling
Automating Footage Tagging with
Amazon Rekognition
Previously, only about half of all footage was indexed due to the immense
time requirements required by manual processes

32. Automating Footage Tagging with
Amazon Rekognition
Solution Architecture
EncodersStills
Extraction &
FeedsResults
Cache
Bucket
R3
Amazon
Rekognition
users
Stills
Frames
SQS
Trigger
1
2
3
4

33. Quickly Identifying Persons of Interest
with Amazon Rekognition
• More than 300,000 photo leads indexed within 1-2 days
• Identification process went from 2-3 days, to minutes – greatly
increasing the ability for law enforcement to act quickly
• Within 1 week of going live, the application identified a suspect in a
case that had no leads, leading to an arrest
There was no software on the market that allows users to quickly search
hundreds of thousands of images using a face in another image

34. Quickly Identifying Persons of Interest
with Amazon Rekognition
Solution Architecture
MS SQL
Database
ColdFusion
Web Service
Mugshot
Bucket
T2.Micro
MySQL DB
instance
Amazon
Rekognition
Amazon
Cognito
iOSMobile
Client
client PHP
1
2
3
4

35. Rekognition - Summary
• Leverage Amazon internal experiences with
AI, ML and Computer Vision
• Managed API services with embedded AI for
maximum accessibility and simplicity
• Full stack of deep learning image processing
algorithms for specialized applications
• Integrates natively with other AWS Services
• Extensible by Design

36. Thank You!

37. Don’t Forget Evaluations!