Adding image and video analysis to your app

1. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Prakash Palanisamy
Solutions Architect, Amazon Web Services
Adding image and video analysis to
your applications

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Images – Universal, Ubiquitous, & Essential
There are 3,700,000,000 internet users in 2017
1,200,000,000,000 photos will be taken in 2017 (9% YoY Growth)
Source: InfoTrends Worldwide

3. Amazon Rekognition
Easy-to-use deep learning-based computer vision analysis
Fully
managed
service
Easy-to-use
API
Low costImage
analysis

4. Amazon Rekognition Image
Detect objects, scenes, and faces, extract text, recognize celebrities, and identify
unsafe content in images
Object and Scene
Detection
Facial
Analysis
Face
Recognition
Text in ImageUnsafe Image
Detection
Celebrity
Recognition
Face
Comparison

5. Why use Rekognition?
johnf
• Object & Scene Detection
Photo-sharing apps can power smart searches
and quickly find cherished memories, such as
weddings, hiking, or sunsets
• Facial Analysis
Retail businesses can understand the
demographics and sentiment of in-store
customers
• Face Comparison
Hotels & hospitality businesses can provide
seamless access for guests and VIPs
• Facial Recognition
Provide secondary authentication for existing
applications

6. Object & Scene Detection
Object and scene detection makes it easy for you to add features that search,
filter, and curate large image libraries.
Identify objects and scenes and provide confidence scores
DetectLabels
Flower
Arrangement
Chair
Coffee Table
Living Room Indoors
Furniture
Cushion
Vase
Maple
Villa
Plan
t
Garden
Water
Swimming Pool
Tree
Potted Plant
Backyard
Pati
o

7. Facial Analysis
Analyze facial characteristics in multiple dimensions
DetectFaces
Image Quality
Facial Landmarks
Demographic Data Emotion Expressed
General Attributes
Facial Pose
Brightness 23.6%
Sharpness 99.9%
EyeLeft,EyeRight,Nose
RightPupil,LeftPupil
MouthRight,LeftEyeBrowUp
Bounding Box...
Age Range 29-45
Gender:Male 96.5%
Happy 83.8%
Surprised 0.65%
Smile:True 23.6%
EyesOpen:True 99.8%
Beard:True 99.5%
Mustache:True 99.9%...
Pitch 1.446
Roll 5.725
Yaw 4.383

8. Face Comparison
Measure the likelihood that faces are of the same person
Similarity 93% Similarity 0%
CompareFaces

9. Facial Recognition
Find similar faces in a large collection of images
SearchFacesByImage
Search
Index
Collection

10. Celebrity Recognition & Image Moderation
Newly released Rekognition features
Detect explicit and suggestive contentRecognize thousands of famous individuals
DetectModerationLabelsRecognizeCelebrities

11. {
"FaceMatches": [
{"Face": {"BoundingB
"Height":
0.2683333456516266,
"Left":
0.5099999904632568,
"Top":
0.1783333271741867,
"Width":
0.17888888716697693},
"
{
"FaceMatches": [
{"Face": {"BoundingB
"Height":
0.2683333456516266,
"Left":
0.5099999904632568,
"Top":
0.1783333271741867,
"Width":
0.17888888716697693},
"
Rekognition APIs – Overview
Rekognition’s computer vision API operations can be grouped into
Non-storage API operations, and Storage-based API operations
CompareFaces
DetectFaces
DetectLabels
DetectModerationLabels
GetCelebrityInfo
RecognizeCelebrities
Non-storage API Operations
CreateCollection
DeleteCollection
DeleteFaces
IndexFaces
ListCollections
SearchFaces
SearchFacesByImage
Storage-based API Operations
ListFaces

12. Amazon
API Gateway
Amazon
API Gateway
AWS
Lambda
Amazon
DynamoDB
Amazon S3
Amazon
CloudFront
Amazon
Rekognition
Client
Camera
Sentiment Analytics
Dashboard
Sentiment Analytics Demo

13. Video Analysis
Limitations of traditional solutions
Temporal information lost Motion context lost

14. Amazon Rekognition Video
Analyze activity, recognize, and track people in stored and live video
Facial
Analysis

15. Amazon Kinesis Video Streams

16. Amazon Kinesis Video Streaming
Capture, process, and store video streams for analytics/ML
Amazon AI
Services
Apache
MXNet
TensorFlow
OpenCV
Custom
Video
Processing
Kinesis
Video
Streams
Stream video from millions of devices
Easily build vision-enabled apps
Secure
Durable, searchable storage
Serverless

17. Amazon
Kinesis Video
Streams
Amazon
API Gateway
AWS
Lambda
Amazon
DynamoDB
Amazon S3
Amazon
Rekognition
Video
Client
Camera
Dashboard
Rekognition Video Demo
Amazon
Kinesis
Analytics
Stream
Destination

18. Usecases

19. Searchable Image Library
Photo Upload Amazon S3 AWS Lambda
Property Search
Detect Objects & Scenes
User captures an image
for their property listing
Mobile app uploads
the image to S3
A Lambda function is triggered
and calls Rekognition
Rekognition retrieves the image from S3 and
returns labels for the property and amenities
Lambda pushes the labels and confidence
scores to Elasticsearch
Other users can search properties by
landmarks, category, etc.
Amazon Elasticsearch
Real Estate Property Search

20. Face-Based User Verification
Authenticated User
Image Capture Application
Amazon S3
Compare Faces
If the similarity score is over 92%, the
application returns a green status. If not,
an alert is issued to security staff.
The application captures a live
image of each employee as
they scan their access card
Rekognition compares the live image
and the badge image – and returns
a similarity score
The application retrieves the
user’s badge from S3
Confirm user identities by comparing their live image with a reference image

21. Facial Recognition
Identify individuals by matching a live image to a collection of images of known persons
Images SearchFacesByImage Face Collection
Person Details Table
Photo AppEnd User Amazon S3
Rekognition searches the face collection for matches to
the reference image and returns an array of face metadata
for potential face matches, ordered by similarity
If source images are required,
they are retrieved from S3
The photo app displays search
results to the end user

22. Collections and Access Patterns
Logging - visitor logs, digital libraries
• Easily find specific images from a digital library
• Find certain images by using a reference image
Social Tagging - photo storage and sharing
• One collection per application user
• Automated friend tagging
Person Verification - employee gate check
• One collection for each person to be verified
• Detection of stolen/shared IDs

23. aws rekognition recognize-celebrities –image “S3Object={Bucket=mybucket,Name=cam.jpg}”
aws rekognition search-faces-by-image –image “S3Object={Bucket=mybucket,Name=cam.jpg}”
--collection-id “persons-of-interest"
aws rekognition create-collection --collection-id “persons-of-interest”
aws rekognition index-faces --image “S3Object={Bucket=mybucket,Name=subject.jpg}”
--collection-id “persons-of-interest”
Rekognition APIs – Advanced Usage
Person of Interest Near a Celebrity
{
"FaceMatches": [
{"Face": {"BoundingB
"Height":
0.2683333456516266,
"Left":
0.5099999904632568,
"Top":
0.1783333271741867,
"Width":
0.17888888716697693},
"
CompareFaces
DetectFaces
DetectLabels
DetectModerationLabels
GetCelebrityInfo
RecognizeCelebrities
2
{
"FaceMatches": [
{"Face": {"BoundingB
"Height":
0.2683333456516266,
"Left":
0.5099999904632568,
"Top":
0.1783333271741867,
"Width":
0.17888888716697693},
"
CreateCollection
DeleteCollection
DeleteFaces
IndexFaces
ListCollections
SearchFaces
SearchFacesByImage
ListFaces
3
1

24. Customer/Partner References

25. Face Search—Media and Entertainment
A u t o m a t i n g F o o t a g e
T a g g i n g w i t h A m a z o n
R e k o g n i t i o n
Indexed 99,000 people
Saves ~9,000 hours a year in labor

26. Automating Footage Tagging with
Amazon Rekognition
EncodersStills
Extraction &
FeedsResults
Cache
Bucket
R3
Amazon
Rekognition
users
Stills
Frames
SQS
Trigger
1
2
3
4

27. Amazon Rekognition Customers

28. Royal Wedding:

29. Amazon Rekognition Image
Video Analytics for media use cases
Object, Scene, Face, Sentiment Detection and
Analysis
Image Moderation
Celebrity Recognition
Text Image Recognition
ML Starter Kit Expansions
for additional use cases
(coming throughout 2018):Amazon Rekognition Video
Object & Activity Detection
Real-time Live Streaming
Metadata generation
Video search by content
Video search by image
+ other Services: Transcribe, Translate, Comprehend
Available today!*
* aws.amazon.com/answers/media-entertainment/video-frame-based-analysis/
Enhanced Metadata tagging
Audio to Text (Close Captioning)
Enhanced Compliance Marking
A Good Starting Point: ML Starter Kits

30. Demos

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Thank you!