Best practices for integrating Amazon Rekognition into your own application

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Best practices for integrating Amazon
Rekognition into your own application
AWS DevDay BENELUX 2018
Julien Lépine, Sr. Manager, Solutions Architecture

Images – Universal, Ubiquitous, & Essential
There are 3,700,000,000 internet users in 2017
1,200,000,000,000 photos will be taken in 2017 (9% YoY Growth)
Source: InfoTrends Worldwide

The AWS Machine Learning Stack
FRAMEWORKS
KERAS

FRAMEWORKS
KERAS
PLATFORMS Amazon SageMaker Amazon Mechanical Turk

Intelligent Services Powered By Deep Learning
FRAMEWORKS
KERAS
PLATFORMS
APPLICATION SERVICES
R E K O G N I T I O N R E K O G N I T I O N
V I D E O
P O L L Y T R A N S C R I B E T R A N S L A T E C O M P R E H E N D L E X
Amazon SageMaker Amazon Mechanical Turk

Amazon Rekognition
Deep learning-based image recognition service
Search, verify, and organize millions of images
Object and Scene
Detection
Facial
Analysis
Face
Comparison
Facial
Recognition
Celebrity
Recognition
Image
Moderation

Why use Rekognition?
johnf
• Object & Scene Detection
Photo-sharing apps can power smart searches
and quickly find cherished memories, such as
weddings, hiking, or sunsets
• Facial Analysis
Retail businesses can understand the
demographics and sentiment of in-store
customers
• Face Comparison
Hotels & hospitality businesses can provide
seamless access for guests and VIPs
• Facial Recognition
Public safety teams can leverage face collections
to automate tracking of persons of interest

Object & Scene Detection
Maple
Villa
Plant
Garden
Water
Swimming Pool
Tree
Potted Plant
Backyard
Flower
Chair
Coffee Table
Living Room
Indoors
Object and scene detection makes it easy for you to add features that search,
filter, and curate large image libraries.
Identify objects and scenes and provide confidence scores
DetectLabels

Facial Analysis
Analyze facial characteristics in multiple dimensions
Demographic Data
Facial Landmarks
Sentiment Expressed
Image Quality
Brightness: 23.6
Sharpness: 99.9
General Attributes
DetectFaces

Face Comparison
Measure the likelihood that faces are of the same person
Similarity 93% Similarity 0%
CompareFaces

Facial Recognition
Find similar faces in a large collection of images
SearchFacesByImage

Celebrity Recognition & Image Moderation
Recognize thousands of famous individuals
RekognizeCelebrities
Detect explicit and suggestive content
DetectModerationLabels

Text detection
DetectText

Use Cases & Customers
• Digital Asset Management
• Media and Entertainment
• Travel and Hospitality
• Influencer Marketing
• Systems Integration
• Digital Advertising
• Consumer Storage
• Law Enforcement
• Public Safety
• eCommerce
• Education

Playout &
Distribution
Filtering & Quality Control
Visual Effects & Editing
Application & Filesystem
Texture & Asset Search
Analytics
Sentiment Analysis
Other Amazon AI Services
(Lex, Polly)
DAM & Archive
Auto-categorization
Metadata Augmentation
Digital Supply Chain
Tag on Ingest
Live and VOD Feature Extraction
Celebrity Detection
Publishing
Value Add
API-based services
OTT
Filtering &
Quality Control
Acquisition
Pre-processing &
optimization
Use Cases Across Media Segments

Interfacing with Rekognition
Build, test and deploy for Rekognition using SDKs & API calls
aws rekognition detect-labels –image
“S3Object={Bucket=mybucket,Name=‘image.jpg’}” |
grep -E ‘(Vehicle|Automobile|Car)’ | mail -s “Alert! Car on Property!” me@site.com
RubyiOS PythonAndroid Node.js.NET PHP AWS CLIJavaScriptJava Xamarin
Or use the AWS CLI

Integrating & Extending Rekognition
3rd Party Software
• AWS AI AMI
• OpenCV
• ImageMagick
• FFMPEG
• YOLO, CCV, …
AWS Services
• Amazon S3
• AWS Lambda
• AWS SQS, SNS
• AWS Batch
• Amazon EC2
AWS Partners
• Media Intelligence
• Asset Management
• Image Hosting
• Image Processing
• Brand Management

Demo – Notebook

{
"FaceMatches": [
{"Face": {"BoundingB
"Height":
0.2683333456516266,
"Left":
0.5099999904632568,
"Top":
0.1783333271741867,
"Width":
0.17888888716697693},
"
{
"FaceMatches": [
"Height":
0.2683333456516266,
"Left":
0.5099999904632568,
"Top":
0.1783333271741867,
"Width":
0.17888888716697693},
"
Rekognition APIs – Overview
Rekognition’s computer vision API operations can be grouped into
Non-storage API operations, and Storage-based API operations
CompareFaces
DetectFaces
DetectLabels
RecognizeCelebrities
CreateCollection
DeleteCollection
DeleteFaces
IndexFaces
ListCollections
SearchFaces
SearchFacesByImage
ListFaces
DetectText

Collections and Access Patterns
Logging - public events and visitor logs
• One large collection per event/time period
• Enables wide searches
Social Tagging - photo storage and sharing
• One collection per application user
• Enables automated friend tagging
Person Verification - employee gate check
• One collection for each person to be verified
• Enables detection of stolen/shared IDs

{
"FaceMatches": [
"Height":
0.2683333456516266,
"Left":
0.5099999904632568,
"Top":
0.1783333271741867,
"Width":
0.17888888716697693},
"
CreateCollection
DeleteCollection
DeleteFaces
IndexFaces
ListCollections
SearchFaces
SearchFacesByImage
ListFaces
{
"FaceMatches": [
"Height":
0.2683333456516266,
"Left":
0.5099999904632568,
"Top":
0.1783333271741867,
"Width":
0.17888888716697693},
"
CompareFaces
DetectFaces
DetectLabels
GetCelebrityInfo
Rekognition APIs – Advanced Usage
Decision trees and processing pipelines
Why?
• Many use cases require more than a single
operation to arrive at actionable data
How?
• S3 event notifications, Lambda, Step Functions
• Dynamo for persistent pipeline storage
• Augmenting results with 3rd Party AI/ML
• OpenCV, MXNet... on EC2 Spot, ECS, AI/ML AMI
Sample Use Cases
• Person of interest near a celebrity
• Multi-pass motion detection enhancement
• Subjects leaving a location without possessions

Person of Interest Near a Celebrity

aws rekognition recognize-celebrities –image ”S3Object={Bucket=mybucket,Name=‘cam.jpg’}”
aws rekognition search-faces-by-image –image ”S3Object={Bucket=mybucket,Name=‘cam.jpg’}”
--collection-id ”persons-of-interest"
aws rekognition create-collection --collection-id ”persons-of-interest”
aws rekognition index-faces --image ”S3Object={Bucket=‘mybucket’,Name=‘subject.jpg’}”
--collection-id ”persons-of-interest” [ --externalimageid “person-UUID” ]
Person of Interest Near a Celebrity
{
"FaceMatches": [
"Height":
0.2683333456516266,
"Left":
0.5099999904632568,
"Top":
0.1783333271741867,
"Width":
0.17888888716697693},
"
CompareFaces
DetectFaces
DetectLabels
GetCelebrityInfo
2
{
"FaceMatches": [
"Height":
0.2683333456516266,
"Left":
0.5099999904632568,
"Top":
0.1783333271741867,
"Width":
0.17888888716697693},
"
CreateCollection
DeleteCollection
DeleteFaces
IndexFaces
ListCollections
SearchFaces
SearchFacesByImage
ListFaces
3
1
4

Encryption & Security
• APIs - Non-storage vs. storage-based API
operations
• Encryption - S3 in-transit and at-rest w/
HTTPS, KMS
• Tampering - Lock down IAM roles and
policies Content - purge or lifecycle to
Glacier w/vault lock
• Use least common privilege -Lambda,
EC2 and other infrastructure
• Hydration - EBS encryption for boot, data
and snapshotted volumes
Best Practices

Encryption & Security
Best Practices
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"rekognition:CreateCollections",
"rekognition:IndexFaces
],
"Resource": "*"
}
]
}
Lambda : Collection Index
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"rekognition:ListCollections",
"rekognition:ListFaces",
"rekognition:SearchFaces",
"rekognition:SearchFacesByImage"
],
"Resource": "*"
}
]
}
Lambda : Collection Search

Interfacing with Rekognition
• S3 input for API calls - max image size of 15MB
• 5MB limit for non-s3 (Base64 encoded) API calls
• Use at least 1024 (x or y) px as image input (80px min)
• Size of face should occupy ~5%+ of image for detection
• Image data must be in PNG or JPG format
• Max number of faces in a single face collection is 1 million
• Max matching faces the search API returns = 4096, paginated
• CompareFaces & RecognizeCelebrities = 15 faces max / image
• Collections are for faces! (not cats, cartoons, …)
Optimizing your input & requests for best performance
…
Use Amazon CloudWatch to observe/alert on Rekognition Metrics

Optimizing your Input Source
Images
• Image enhancement, extraction & stabilization
• Unsharp mask, deconvolution – CPU impact
• e.g. ImageMagick, OpenCV, scikit-image
Video
• Video stabilization - motion / optical flow analysis
• Scene change detection vs. frame extraction
• Offset - PTS vs. seconds and why it matters
• e.g. FFMPEG w/deshake, vidstab, OpenCV
Optimizing your input & requests for best performance
Rekognition provides ImageQuality for Sharpness + Brightness

Searchable Image Library
Photo Upload Amazon S3 AWS Lambda
Property Search Amazon Elasticsearch
Detect Objects & Scenes
User captures an image
for their property listing
Mobile app uploads
the image to S3
A Lambda function is triggered
and calls Rekognition
Rekognition retrieves the image from S3 and
returns labels for the property and amenities
Lambda pushes the labels and confidence
scores to Elasticsearch
Other users can search properties by
landmarks, category, etc.
Real Estate Property Search

Searchable Image Library
• Optimize the client
• Event based, decoupled infra
• Buffering - SQS, SNS, Kinesis
• Rate Control - high volume S3 image ingest
• DynamoDB – scale label storage
• Elasticsearch - operational & performance statistics
• CloudFront - search cache
Real Estate Property Search
AWS
Lambda
Amazon
S3
Amazon
SQS
AWS
CloudFormation
Amazon
CloudWatch
Amazon
Kinesis
Amazon
CloudFront
Amazon
DynamoDB
Amazon
ElasticSearch

Sentiment Analysis
Amazon RedshiftAmazon Quicksight
Live Subject In-Store Camera Application
Amazon S3
Analyze Faces
Shoppers enter and
browse in retail store
In-store cameras capture
live images of shoppers
A Lambda function is triggered
and calls Rekognition Rekognition analyzes the image and returns
facial attributes detected, which include
emotion and demographic detail
Return data is normalized and
staged in S3 en route to Redshift
Marketing Reports
Periodic ingest of data into
Redshift
Regular analysis to identify trends in
demographic activity and in-store
sentiment over time
Trend reporting for retail store locations

Sentiment Analysis
• Reduce API volume - perform motion detection and frame pre-processing on the
capture device
• Customer photos not stored on AWS
• S3 - A content lake to feed other services – EMR, Lambda
• Resell as a service – API gateway + Lambda + S3
AWS
Lambda
Amazon
S3
Amazon
SQS
AWS
CloudFormation
Amazon
CloudWatch
Amazon
ElasticSearch
Amazon
Redshift
Amazon
QuickSight
Amazon API
Gateway*
Trend reporting for retail store locations

Face-Based User Verification
Authenticated User
Image Capture Application
Amazon S3
Compare Faces
If the similarity score is over 92%, the
application returns a green status. If not,
an alert is issued to security staff.
The application captures a live
image of each employee as
they scan their access card
Rekognition compares the live image
and the badge image – and returns
a similarity score
The application retrieves the
user’s badge from S3
Confirm user identities by comparing their live image with a reference image

Face-Based User Verification
• S3 Encryption of badge images – SSE-S3, SSE-KMS, SSE-C
• Prevent tampering using bucket policies & IAM RO permissions
• Extend by using Collections
• Cloudtrail - Logging & Auditing w/ tamper-proof log signatures
• Tie notification into SNS/SES, Custom CloudWatch Logs metrics, or ElasticSearch
w/alerts
AWS KMS AWS
CloudTrail
AWS
Lambda
Amazon
S3
Amazon
SNS
AWS
CloudFormation
Amazon
CloudWatch
Amazon
SES
Confirm user identities by comparing their live image with a reference image

““Using AWS, we can test more than twice as many
algorithms at a time as we could previously. It’s plug
and play.
Matchmaking Site Shaadi.com Doubles
Algorithm Testing Using AWS
After years of steady growth,
Shaadi.com was faced with
an aging IT infrastructure that
limited its ability to scale and
innovate.
Shaadi.com migrated from a
private hosted cloud to AWS. The
company used the AWS Database
Migration Service to keep source
databases running during the
migration, and uses Amazon
Rekognition to automate the
screening of profile pictures.
• Migrated from private cloud
to AWS in three months
• Doubled testing of matchmaking
algorithms
• Reduced time for users to get
photos on profiles by 95%
SolutionChallenge Benefits
Ajay Poddar, Vice President of Engineering, Shaadi.com
Company: Shaadi.com
Industry: Media & Entertainment
Country: India
Employees: 500+
Website: www.shaadi.com
About Shaadi.com
Shaadi.com, one of India’s best-
known brands and one of the world’s
largest matchmaking services, has
helped people around the world find
their soulmates since 1996 and has
touched more than 35 million lives.

““I got a prototype of our service up and running within
four hours and into production within a week.
Artfinder Powers Art-Matching Services
Using AWS
Artfinder.com needs to match art
lovers with pieces they’ll like.
Shaadi.com runs its website and
recommendation tools using AWS
services including Amazon EC2,
Amazon Machine Learning,
Amazon Rekognition and Amazon
Kinesis Firehose.
• Increased revenue 75% year-over-
year
• Innovates with art-curating Twitter
bot
• Launches AI services in four hours
instead of weeks
David Tilleyshort, Chief Technology Officer, Artfinder
Company: Artfinder.com
Industry: Art
Country: UK / US
Employees: 50+
Website: www.artfinder.com
About Artfinder.com
Artfinder is an online art
marketplace, allowing thousands of
artists to sell directly to buyers

““
Aella Credit Uses AWS to Improve
Identity Verification, Grow Business
Aella Credit wanted to
innovate and grow faster, but
it was limited by its
technology environment. The
company needed a better
way to validate employee IDs
and government-issued IDs
in real time.
Aella Credit uses AWS to
support its online loan-
processing software. The
company also takes
advantage of Amazon
Rekognition to improve its
identity verification
capabilities.
• Improves facial recognition
accuracy by 40%
• Increases availability of loan
processing software
• Grows from 5,000 to 200,000
customers in several months
Identity verification is a major problem for financial
services companies in Nigeria, and we can overcome
that challenge by using Amazon Rekognition. That
gives us a competitive edge as a startup.
Wale Akanbi, Chief Technology Officer, Aella Credit
Company: Aella Credit
Industry: Financial Services
Country: Nigeria
Employees: 50
Website: www.aellacredit.com
About Aella Credit
Aella Credit is a financial services
technology startup that provides
easy access to credit to the world’s
underbanked. The company
provides machine learning–driven
risk assessment in both a B2B
integration with
employers/cooperatives and a B2C
model to determine applicant
eligibility for loans.

““We weren't expecting the high degree of facial-
recognition accuracy we’re getting. It’s very exciting—
and setting up Amazon Rekognition was shockingly
easy.
C-SPAN Uses Amazon Rekognition
to Cut Video-Indexing Time in Half
Had developed an automated
facial recognition solution to help
human indexers, but it was slow. It
could only index half of the
incoming content by speaker,
limiting the ability of users to
search archived content
C-SPAN implemented Amazon
Rekognition to automatically match
uploaded screen shots to a
collection of 97,000 known faces.
• Uploaded 97,000 images in less
than two hours
• Enables C-SPAN to more than
double video indexed—from 3,500
to 7,500 hours per year
• Reduced labor required to index an
hour of video from 60 to 20 minutes
• Deployed in less than three weeks
Alan Cloutier, Technical Manager, C-SPAN
Company: C-SPAN
Industry: Media & Entertainment
Country: US
Employees: 500+
Website: www.c-span.org
About C-SPAN
C-SPAN is a not-for-profit
organization funded by the United
States cable industry to increase
transparency by broadcasting and
archiving government proceedings.

Solution Architecture
EncodersStills
Extraction &
Feeds
Results Cache
Bucket
R3
Amazon
Rekognition
users
Stills
Frames
SQS
Trigger
1
2
3
4
Best Practices: Media Conformance, Infrastructure Decoupling, API Call Optimization
Automating Footage Tagging

Rekognition Video

Demo – Digital Asset Management

Rekognition - Summary
• Leverages Amazon internal experiences with AI, ML and Computer Vision
• Provides a full stack of deep learning image processing algorithms for a
wide variety of applications
• Eliminates training & infrastructure heavy lifting
• Native integration with other AWS Services
• Consume as a Building Block – Extensible by Design

Go Build!
Thank you

Best practices for integrating Amazon Rekognition into your own application

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Best practices for integrating Amazon Rekognition into your own application

Similar to Best practices for integrating Amazon Rekognition into your own application (20)

More from Amazon Web Services

More from Amazon Web Services (20)

Best practices for integrating Amazon Rekognition into your own application