This document discusses building a personal image search engine using Amazon Rekognition. It begins by noting the explosive growth of personal photos. It then introduces Amazon Rekognition as a deep learning image recognition service that can detect objects, scenes, faces, and compare faces. The document outlines the APIs for object detection, facial analysis, face comparison, and facial recognition. It provides examples of how these APIs could be used to power photo tagging, targeted advertising, and facial verification. It then discusses using Rekognition to build a personal photo search engine based on TF-IDF (term frequency-inverse document frequency) modeling of image labels. Finally, it covers Amazon Rekognition pricing and availability.

1. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Sunil Mallya, Startup Solutions Architect
January 19, 2017
Amazon Rekogition
Building a personal image search engine

2. Images – explosive growth trends
Source: InfoTrends Worldwide Consumer Photos Captured and Stored.
2013 -2017 prepared for Mylio.

3. Amazon Rekognition
Deep learning-based image recognition service
Search, verify, and organize millions of images
Object and Scene
Detection
Facial
Analysis
Face
Comparison
Facial
Recognition

4. Amazon Rekognition API
DetectLabels
Object and Scene Detection
Detect objects, scenes, and concepts in images

5. Amazon Rekognition API
DetectLabels
{
"Confidence": 94.62968444824219,
"Name": "adventure"
},
{
"Confidence": 94.62968444824219,
"Name": "boat"
},
{
"Confidence": 94.62968444824219,
"Name": "rafting"
},
. . .

6. Object and Scene Detection
Flower
ChairCoffee Table
Living Room
Indoors

7. Object and Scene Detection
Maple
Villa
Plant
Garden
Water
Swimming Pool
Tree
Potted Plant
Backyard

8. Using Rekognition Object and Scene Detection
 Photo-sharing apps can power smart searches and
quickly find cherished memories, such as weddings,
hiking, or sunsets.
 Vacation rental markets can automatically label host-
uploaded images with tags, such as fireplace, kitchen,
or swimming pool.
 Travel sites and forums can classify user generated
images with labels such as beach, camping, or
mountains.

9. Amazon Rekognition API
Facial Analysis
Detect face and key facial characteristics
DetectFaces

10. Amazon Rekognition API
[
{
"BoundingBox": {
"Height": 0.3449999988079071,
"Left": 0.09666666388511658,
"Top": 0.27166667580604553,
"Width": 0.23000000417232513
},
"Confidence": 100,
"Emotions": [
{"Confidence": 99.1335220336914,
"Type": "HAPPY" },
{"Confidence": 3.3275485038757324,
"Type": "CALM"},
{"Confidence": 0.31517744064331055,
"Type": "SAD"}
],
"Eyeglasses": {"Confidence": 99.8050537109375,
"Value": false},
"EyesOpen": {Confidence": 99.99979400634766,
"Value": true},
"Gender": {"Confidence": 100,
"Value": "Female”}
DetectFaces

11. Demographic Data
Facial Landmarks
Sentiment Expressed
Image Quality
Facial Analysis
General Attributes

12. Demographic Data
Facial Landmarks
Sentiment Expressed
Image Quality
Facial Analysis
Brightness: 25.84
Sharpness: 160
General Attributes

13. Using Rekognition Facial Analysis
 Photo printing service can recommend the best
photos to their users
 Online dating applications can improve their match
recommendations using face attributes
 Retail businesses can understand the demographics
and sentiment of in-store customers
 Ad-tech services can display dynamic and
personalized content to customers

14. Facial Analysis - Use Case
(Retail – In-store and Online)
Demographic and Sentiment Analysis
Female
Happy
Smiling
Male
No Facial Hair
Happy
Female
Sad
No Eyeglasses

15. Look Your Best All Day
Time for A New Look?
Facial Analysis - Use Case
(Targeted Marketing)
Demographic and Sentiment Analysis
PersonAPersonB
Sees
Sees

16. CompareFaces
Amazon Rekognition API
Face Comparison
Face-based user verification

17. Amazon Rekognition API
CompareFaces
{
"FaceMatches": [
{"Face": {"BoundingBox": {
"Height": 0.2683333456516266,
"Left": 0.5099999904632568,
"Top": 0.1783333271741867,
"Width": 0.17888888716697693},
"Confidence": 99.99845123291016},
"Similarity": 96
},
{"Face": {"BoundingBox": {
"Height": 0.2383333295583725,
"Left": 0.6233333349227905,
"Top": 0.3016666769981384,
"Width": 0.15888889133930206},
"Confidence": 99.71249389648438},
"Similarity": 0
}
],
"SourceImageFace": {"BoundingBox": {
"Height": 0.23983436822891235,
"Left": 0.28333333134651184,
"Top": 0.351423978805542,
"Width": 0.1599999964237213},
"Confidence": 99.99344635009766}
}

18. Face Comparison

19.  IoT and camera manufacturers can integrate face-
based verification directly into their products
 Application of face comparison in locating person of
interest for Public Safety
 Hotels & hospitality businesses can provide seamless
access for guests and VIPs
 Online exams or polls can verify presence of registered
person by comparing against image captured by
webcam.
Using Rekognition Face Comparison

20. Amazon Rekognition API
Facial Recognition
Index and Search faces in a collection
Index
Search
Collection
IndexFaces
SearchFacesByImage

21. Amazon Rekognition API
f7a3a278-2a59-5102-a549-a12ab1a8cae8
&
v1
02e56305-1579-5b39-ba57-9afb0fd8782d
&
v2
Face ID & vector<float>Face
4c55926e-69b3-5c80-8c9b-78ea01d30690
&
v3transformed
stored
{
f7a3a278-2a59-5102-a549-a12ab1a8cae8,
02e56305-1579-5b39-ba57-9afb0fd8782d,
4c55926e-69b3-5c80-8c9b-78ea01d30690
}
IndexFaces Collection

22. Amazon Rekognition API
Face
{
f7a3a278-2a59-5102-a549-a12ab1a8cae8,
02e56305-1579-5b39-ba57-9afb0fd8782d,
4c55926e-69b3-5c80-8c9b-78ea01d30690
}
SearchFacesByImage
Collection
Nearest neighbor
search
Face ID

23. Facial Recognition

24. Using Facial Recognition
 Family photo sharing apps can use face recognition to
group all faces of the same person in a family
 Entertainment and news organizations can index decades
of archived images to find celebrities
 Secure campuses / workplaces can use face search to
ensure all personnel in their facilities are authorized to be
there
 Public safety teams can leverage face collections to
automate tracking of persons of interest

25. Now lets build something

26. Personal Photo Search Engine

27. Traditional Search Engine Basics
• Vector space model
• Cosine Similarity
q
documents and queries are represented as vectors
src: https://en.wikipedia.org/wiki/Vector_space_model

28. Document Representation
• Convert a document into a list of words or terms.
• But Images don’t have terms!
• Rekognition gives us labels with confidence
• This can be used as a proxy for terms and term
frequency
quick oak is called
Freq 4 2 1 1
The Oak is called the king
of trees.
The Oak quivers in the
breeze,
{
"Confidence":
94.62968444824219,
"Name": "adventure"
},
{
"Confidence":
94.62968444824219,
"Name": "boat"
},
{
"Confidence":
94.62968444824219,
"Name": "rafting"
},
. . .

29. TF-IDF: Term frequency, Inverse document frequency


30. Scikit-learn makes it easy
• Generate TF-IDF Model for documents
• Build the query as a document
• Find similar documents
vec = sklearn.feature_extraction.DictVectorizer()
counts = vec.fit_transform(d_index.values()).toarray()
transformer = sklearn.feature_extraction.text.TfidfTransformer(smooth_idf=True)
tfidf = transformer.fit_transform(counts)
from sklearn.metrics.pairwise import cosine_similarity
cosine_similarities = cosine_similarity(test, tfidf).flatten()
related_docs_indices = cosine_similarities.argsort()[:-100:-1] # TOP 100 results
test = np.zeros((tfidf[0].shape))
for word in query.split(‘ ‘):
idx = features.index(word)
test[0][idx] = 1
{
img1 : [{cat, 90}, {animal, 95} ..],
img2 : [{sea, 70}, {water, 91}..] …
}

31. Finally, Its time to code #realdeal

32. Amazon Rekognition – Availability and Pricing
1. Released General Availability
2. Available in 3 regions,
1. US East (N. Virginia)
2. US West (Oregon)
3. EU (Ireland)
3. Pricing
• Pay as you go
• Free Tier – 5000 images per month for first 12 months
• Tiered Pricing designed

33. Amazon Rekognition – Pricing Details
*Images processed: For APIs with image as input, it’s the number of images analyzed. For APIs with no image input 1 API call = 1 image processed.
Image Analysis Tiers
Price per 1000
images processed
First 1 million images processed* per month $1.00
Next 9 million images processed* per month $0.80
Next 90 million images processed* per month $0.60
Over 100 million images processed* per month $0.40
*Images processed: For APIs with image as input, it’s the number of images analyzed. For APIs with no image input 1 API call = 1 image processed.

34. Summary
1. Fully managed and easy-to-use image recognition service
2. Four primary capabilities
1. Object and Scene Detection
2. Facial Analysis
3. Face Comparison
4. Facial Recognition
3. Integrated with AWS and AI Services
• Amazon S3
• Lex and Polly
4. Scalable and low cost

35. Thank You!
smallya@amazon.com