In this session, we’ll learn how to build and deploy end to end solutions for ingesting and processing computer vision solutions, using machine learning models connected to live video streams, and getting insights such as face detection and object analysis. At the end of the session developers of all skill levels will be able to build their own deep learning powered, computer-vision applications. Attendees will learn how to experiment with different projects for face detection, object recognition and other video-based AWS Machine Learning services.

1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
© 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Sara Mitchell
Solutions Architect, AWS, London
Building an end-to-end video
recognition service

2. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Streaming
Amazon Kinesis
Analytics
KCL
Apps
AWS Lambda
COLLECT STORE CONSUMEPROCESS/ANALYZE
Amazon ES
Apache Kafka
Amazon Kinesis
Streams
Amazon Kinesis
Firehose
Amazon
DynamoDB
Amazon ElastiCache
Amazon RDS
Amazon Aurora
HotHotWarm
Fast
Stream
SQLNoSQLCacheFileStream
Mobile apps
Web apps
Devices
Sensors
IoT platforms
AWS IoT
Data centers AWS Direct
Connect
Migration
Snowball
Logging
Amazon
CloudWatch
AWS
CloudTrail
RECORDS
FILES
STREAMS
Amazon
QuickSight
Analysis&visualizationDataSceince
DataTransport&LoggingIoTApplications
Amazon EMR
Amazon Redshift
& Spectrum
Presto
Amazon
EMR
FastSlow
Amazon Athena
BatchInteractivePredictive
AmazonAI
Amazon S3
Amazon DAX
Import/expo
rt
Lex PollyAML Rekognition
AWS DL AMI
AI Apps
Amazon ECS
Apps
Model
Train/
Eval
Models
Deploy
ETL
AWS Greengrass

3. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Rekognition
Extract rich metadata from visual content
Object and Scene
Detection
Facial
Analysis
Face
Comparison
Facial
Recognition
Celebrity
Recognition
Image
Moderation

4. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
https://console.aws.amazon.com/rekognition/home

5. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• Built in 3 weeks
• Indexed against 99,000 people
• Index created in one day
• Saved ~9,000 hours a year in
manual curation costs
• Live video with frame sampling
Automating Footage Tagging with
Amazon Rekognition
Previously, only about half of all footage was indexed due to the
immense time requirements required by manual processes

6. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Automating Footage Tagging with
Amazon Rekognition
Solution Architecture
EncodersStills
Extraction &
FeedsResults
Cache
Bucket
R3
Amazon
Rekognition
users
Stills
Frames
SQS
Trigger
1
2
3
4

7. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
© 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Rekognition from HTML Client Demo

8. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Data Ingestion
• Ingest from Mac Camera
• Take a snapshot
• Query Amazon Rekognition – detect labels
• Query Amazon Rekognition Collection – is this person known?
Amazon
Rekognition Image
Collection
Extract meta-data for the image
Validate image matches with
stored image
Amazon
Cognito
HTML
Unauthenticated
identity
<scripttype="text/javascript"src="https://s3.amazonaws.com/.../webcam.min.js"></script>

9. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Rekognition APIs – example
Detect labels using
ML model
Rekognition
stores meta-data
about image

10. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Rekognition APIs – example
Match against image
collection using ML
models

11. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Rekognition APIs – example
How close a match?

12. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Rekognition APIs – Advanced Usage
Decision trees and processing pipelines
Why?
• Many use cases require more than a single
operation to arrive at actionable data
How?
• S3 event notifications, Lambda, Step Functions
• DynamoDB for persistent pipeline storage
• Augmenting results with 3rd Party AI/ML
• OpenCV, MXNet, etc. on EC2 Spot, ECS, AI/ML AMI
Sample Use Cases
• Person of interest near a celebrity
• Multi-pass motion detection enhancement
• Subjects leaving a location without possessions
IndexFaces
DetectLabels
“person”

13. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
aws rekognition recognize-celebrities –image “S3Object={Bucket=mybucket,Name=cam.jpg}”
aws rekognition search-faces-by-image –image “S3Object={Bucket=mybucket,Name=cam.jpg}”
--collection-id “persons-of-interest"
aws rekognition create-collection --collection-id “persons-of-interest”
aws rekognition index-faces --image “S3Object={Bucket=mybucket,Name=subject.jpg}”
--collection-id “persons-of-interest”
Rekognition APIs – Advanced Usage
{
"FaceMatches": [
{"Face": {"BoundingB
"Height":
0.2683333456516266,
"Left":
0.5099999904632568,
"Top":
0.1783333271741867,
"Width":
0.17888888716697693},
"
CompareFaces
DetectFaces
DetectLabels
DetectModerationLabels
GetCelebrityInfo
RecognizeCelebrities
2
{
"FaceMatches": [
{"Face": {"BoundingB
"Height":
0.2683333456516266,
"Left":
0.5099999904632568,
"Top":
0.1783333271741867,
"Width":
0.17888888716697693},
"
CreateCollection
DeleteCollection
DeleteFaces
IndexFaces
ListCollections
SearchFaces
SearchFacesByImage
ListFaces
3
1

14. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

15. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Rekognition Video
Features
Detects objects,
activities, scenes.
Label name,
timestamp,
confidence
Detects and analyzes
faces. Timestamp,
face bounding box,
age range, emotion,
gender, pose …
Search video for
matches in a face
collection.
Timestamp, person &
face bounding boxes,
matched face IDs,
similarity % …
Detects and tracks
unique people,
including occlusions
and shot changes,
Persona and Face
bounding box,
person ID
Recognizes
celebrities
throughout video.
Celebrity name,
bounding box,
confidence %
Detects nudity and
explicit nudity.
Timestamp,
confidence, label
name.

16. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

17. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Rekognition Video Streams
• CCTV for building entry
• Ensure workers on a building
site are registered and known.
• Foreign object detection on a
runway
• Trespassing on railway track
• Customer personalization
• Looking for celebrities in your
street
• Working out who’s at the
front door

18. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Amazon Rekognition Video
Who’s at the front door?
Producer
Application

19. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
CreateKinesisVideoStream
aws kinesisvideo create-stream --stream-name "video-
stream-name" --data-retention-in-hours "24”

20. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Requirements for your application code
• Producer Application
• Any video data generating device
• Streaming media data in real time
• Streaming media data after buffering it for a few seconds
• Streaming after-the-fact media uploads
• Kinesis Video Streams Producer Libraries and SDK
• Java
• Android
• C++
• Consumer Application
• Lambda function - python
• Triggered by the data arriving on the Kinesis Data Stream
• Remember data flow is Camera > Kinesis Video Stream > Rekognition Data Processor > Kinesis Data Stream

21. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
© 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Rekognition with Kinesis Video Stream Demo

22. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS DeepLens
• A deep-learning enabled HD video camera
• Compute capable of running sophisticated deep
learning computer vision models in real time on
the device.
• Perform inference at the edge
• Trigger a lambda function when an action takes
place
• Apply more advanced analytics on the cloud
• Amazon Kinesis Video Streams
• Amazon Rekognition video
CPU
Intel Atom® Processor
MEMORY
8GB RAM
OS
Ubuntu OS-16.04 LTS
BUILT-IN STORAGE
16GB Memory (expandable)
GRAPHICS
Intel Gen9 Graphics Engine

23. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
When model detect a
face
Register and
Recognise face
Store in a database
http://coffee.us-east-
1.elasticbeanstalk.com/
Read values
Uploads trigger
Lambda
Detect Mug
Check face collection
Coffee Leaderboard Architecture

24. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What happens on the device?

25. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The Leaderboard

26. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What have we seen
Simple HTML
• Build a web page to process video content as snapshots
• Use Amazon Rekognition to analyse those images
Process live video
• Stream video through Amazon Kinesis Video Stream
• StreamProcessor interacts with Amazon Rekognition Video
Custom image processing at the edge
• Amazon DeepLens and your own machine learning model

27. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thanks to:

28. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Video Analysis
http://dub-
front.deepvideoanalysis.cloud/results.html#!?identifier=intro
ducingamazongo.mp4
Quick start Architecture:
https://aws.amazon.com/answers/media-
entertainment/video-frame-based-analysis/

29. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
© 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.© 2018, Amazon Web Services, Inc. or Its Affiliates. All rights reserved.
Thank You
aws.amazon.com/blogs/machine-learning/