Using AWS to Ingest, Store, Archive, Share and carry out Analysis of Video Content.pdf

© 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Paul Armstrong
Principal Solutions Architect, Amazon Web Services
Using AWS to Ingest, Store, Archive,
Share and carry out Analysis of Video
Content

© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Streaming
Amazon Kinesis
Analytics
KCL
Apps
AWS Lambda
COLLECT STORE CONSUMEPROCESS/ANALYZE
Amazon ES
Apache Kafka
Amazon Kinesis
Streams
Amazon Kinesis
Firehose
Amazon
DynamoDB
Amazon ElastiCache
Amazon RDS
Amazon Aurora
HotHotWarm
Fast
Stream
SQLNoSQLCacheFileStream
Mobile apps
Web apps
Devices
Sensors
IoT platforms
AWS IoT
Data centers AWS Direct
Connect
Migration
Snowball
Logging
Amazon
CloudWatch
AWS
CloudTrail
RECORDS
FILES
STREAMS
Amazon
QuickSight
Analysis&visualizationDataSceince
DataTransport&LoggingIoTApplications
Amazon EMR
Amazon Redshift
& Spectrum
Presto
Amazon
EMR
FastSlow
Amazon Athena
BatchInteractivePredictive
AmazonAI
Amazon S3
Amazon DAX
Import/export
Lex PollyAML Rekognition
AWS DL AMI
AI Apps
Amazon ECS
Apps
Model
Train/
Eval
Models
Deploy
ETL
AWS Greengrass

Building a video ingestion system is complex
• Cumbersome to rewrite code for various devices and development
environments
• Difficult to elastically scale to millions of devices
• Requires support for cadence, latency, and jitter on streams
• Requires secure, highly scalable, available storage
• Need easy-to-use APIs to retrieve, process, and replay video

Stream video from millions of devices
Easily build vision-enabled apps
Secure
Durable, searchable storage
Fully managed
Amazon Kinesis Video Streams
S t r e a m v i d e o a n d t i m e - e n c o d e d d a t a f o r a n a l y t i c s
Amazon AI
Services
Apache MxNet
TensorFlow
Custom Video
Processing
3rd Party
Partners
Kinesis
Video
Streams

Amazon Kinesis Video Streams Concept

• Flexible SDK for integration with on-
device hardware media pipelines
• Out-of-the-box integration with AWS
auth mechanism: SigV4
• Flexible Put API to stream Frames or
buffered Fragments or after-the-fact
uploads
• Build custom integrations with diverse
camera and device types
• Secure authN and authZ to connect
devices w/ image sensors to AWS
• Stream video data based on the
preferred transmission scenario
Video Streams SDK offers … … that enables developers to
Kinesis Video Streams Producer SDK
Easily connect and stream from camera sources

Ingestion Options
Camera with sdk directly to Stream e.g. Partner cameras
Camera to stream to proxy e.g. IP Camera
Camera at the edge e.g. Deep Lens

Kinesis Video Streams
• Kinesis Video Stream:
• Customer-owned entity that transports video-like data such as frames,
samples, and fragments
• Stores durably, and makes it available for both real-time and ad hoc
consumption.
• Kinesis Video Stream only has a single producer publishing data into it.
• Producer:
• Producers put data into Kinesis Video Streams, and include hardware devices
like security cameras, body-worn cams, smartphones, dashcams, etc.
• A single producer can generate one or more Kinesis Video Streams
• Consumer:
• Consumers are applications that get data from Streams to process.
• Consumers get data in real-time/continuous or in batch-oriented path
• Consumers run on the customers own Amazon EC2 instances.

View Video Archive
• Ingest from existing Camera using Producer to put on stream, for long term
storage and archive retrieval
• User request footage from HTML
• Search for start time in stream and duration
• Store extracted footage on S3 with a pre-signed URL to securely share with 3rd
parties
Producer SDK
Video Stream
userAmazon API
Gateway
GetFrame
Lambda
function
S3
Search for Stream
Footage
Share with 3rd Parties
Amazon
Cognito

View Video Archive
List the Streams to
Select Footage from

View Video Archive
Pass the Time and
Duration to the
Stream

View Video Archive
Store the fragments
in S3 and return pre-
signed URL

Amazon Rekognition
Extract rich metadata from visual content
Object and Scene
Detection
Facial
Analysis
Face
Comparison
Facial
Recognition
Celebrity
Recognition
Image
Moderation

https://console.aws.amazon.com/rekognition/hom
e

Automating Footage Tagging with Amazon Rekognition
• Built in 3 weeks
• Indexed against 99,000 people
• Index created in one day
• Saved ~9,000 hours a year in manual
curation costs
• Live video with frame sampling
Previously, only about half of all footage was indexed due to the
immense time requirements required by manual processes

Automating Footage Tagging with
Amazon Rekognition
Solution Architecture
EncodersStills
Extraction &
FeedsResults
Cache
Bucket
R3
Amazon
Rekognition
users
Stills
Frames
SQS
Trigger
1
2
3
4

Rekognition from HTML Client Demo

Data Ingestion
• Ingest from Mac Camera
• Take a snapshot
• Query Amazon Rekognition – detectfaces
• Query Amazon Rekognition Collection – searchfacesbyImage
Amazon
Rekognition
Image
Collection
Extract meta-data for the image
Validate image matches with
stored image
Amazon
Cognito
HTML
Unauthenticated
identity
<scripttype="text/javascript"src="https://s3.amazonaws.com/.../jpeg_camera_update.js"></script>

Rekognition APIs – example
Detect labels using
ML model
Rekognition
returns meta-
data about image

Rekognition APIs – example
Match against image
collection using ML
models
Confidence on
Match?

Rekognition APIs – Advanced Usage
Decision trees and processing pipelines
Why?
• Many use cases require more than a single operation to arrive
at actionable data
How?
• S3 event notifications, Lambda, Step Functions
• DynamoDB for persistent pipeline storage
• Augmenting results with 3rd Party AI/ML
• OpenCV, MXNet, etc. on EC2 Spot, ECS, AI/ML AMI
Sample Use Cases
• Person of interest near a celebrity
• Multi-pass motion detection enhancement
• Subjects leaving a location without possessions
IndexFaces
DetectLabels
“person”

aws rekognition recognize-celebrities –image “S3Object={Bucket=mybucket,Name=cam.jpg}”
aws rekognition search-faces-by-image –image “S3Object={Bucket=mybucket,Name=cam.jpg}”
--collection-id “persons-of-interest"
aws rekognition create-collection --collection-id “persons-of-interest”
aws rekognition index-faces --image “S3Object={Bucket=mybucket,Name=subject.jpg}”
--collection-id “persons-of-interest”
Rekognition APIs – Advanced Usage
{
"FaceMatches": [
{"Face": {"BoundingB
"Height":
0.2683333456516266,
"Left":
0.5099999904632568,
"Top":
0.1783333271741867,
"Width":
0.17888888716697693},
"
CompareFaces
DetectFaces
DetectLabels
DetectModerationLabels
GetCelebrityInfo
RecognizeCelebrities
2
{
"FaceMatches": [
{"Face": {"BoundingB
"Height":
0.2683333456516266,
"Left":
0.5099999904632568,
"Top":
0.1783333271741867,
"Width":
0.17888888716697693},
"
CreateCollection
DeleteCollection
DeleteFaces
IndexFaces
ListCollections
SearchFaces
SearchFacesByImage
ListFaces
3
1

Amazon Rekognition Video
Features
Detects objects,
activities, scenes.
Label name,
timestamp,
confidence
Detects and analyzes
faces. Timestamp,
face bounding box,
age range, emotion,
gender, pose …
Search video for
matches in a face
collection.
Timestamp, person &
face bounding boxes,
matched face IDs,
similarity % …
Detects and tracks
unique people,
including occlusions
and shot changes,
Persona and Face
bounding box,
person ID
Recognizes
celebrities
throughout video.
Celebrity name,
bounding box,
confidence %
Detects nudity and
explicit nudity.
Timestamp,
confidence, label
name.

Amazon Rekognition Video Streams
• CCTV for building entry
• Ensure workers on a building
site are registered and known.
• Foreign object detection on a
runway
• Trespassing on railway track
• Customer personalization
• Looking for celebrities in your
street
• Working out who’s at the
front door

Amazon Rekognition Video
Who’s at the front door?
Producer
Application

Requirements for your application code
• Producer Application
• Any video data generating device
• Streaming media data in real time
• Streaming media data after buffering it for a few seconds
• Streaming after-the-fact media uploads
• Kinesis Video Streams Producer Libraries and SDK
• Java
• Android
• C++
• Consumer Application
• Lambda function - python
• Triggered by the data arriving on the Kinesis Data Stream
• Remember data flow is Camera > Kinesis Video Stream > Rekognition Data Processor > Kinesis Data
Stream

Detecting Faces at the Edge Demo

AWS DeepLens
• A deep-learning enabled HD video camera
• Compute capable of running sophisticated deep
learning computer vision models in real time on the
device.
• Perform inference at the edge
• Trigger a lambda function when an action takes place
• Apply more advanced analytics on the cloud
• Amazon Kinesis Video Streams
• Amazon Rekognition video
CPU
Intel Atom® Processor
MEMORY
8GB RAM
OS
Ubuntu OS-16.04 LTS
BUILT-IN STORAGE
16GB Memory (expandable)
GRAPHICS
Intel Gen9 Graphics Engine

Face Detection
user
Amazon API
Gateway
S3
Kinesis Data
Stream
deeplens-
processor
deeplens_viewer
Search_faces
_by_image
detect_faces
Cropped
faces
DeepLens
Viewer
Matched Faces
In Collection
Amazon ES
Amazon Kinesis
Firehose
DeepLensDetected
Faces
Amazon
Rekognition
Get
Sentiment of
all detected
faces
DeepLensDetected
FacesData
Presented
Image Store
Deeplens_upload.py
Upload Images to
Rekognition Collection, meta
store and image store using
json template
AWS
Greengrass
Face
detection
DeepLens
SSD
+ResNet
Model
Display
Detected
Images and
Meta data
Display Time
Series of
Emotions

What happens on the device?
Load Model to GPU

Resize captured
frame, doinference

Resize captured
frame with a border
Crop just the face,
send to Kinesis data
stream

Add the framing box
to the detected face

Storing meta data and the Image Collection
json file, with meta
data and location of
image to display
Process json, resize
images
Add image to
rekognition
collection
Deeplens_upload.py

Storing meta data and the Image Collection
Store meta data in
DynamoDB table
Deeplens_upload.py

Processing Detected Faces
Process Images
from Kinesis stream
and match with
collection

Get Sentiment from
detected faces and
send by Firehose to
ElasticSearch

Stored detected
image ids in the
collection in
DynamoDB

Detected Faces Viewer
Query DynamoDB table
for detected faces in the
last 1 minute

For unique records
determine the
associated meta data
and image
Determine unique
images detected

Get smile url and return
json to html page

Sentiment Analysis

What have we seen
Simple HTML
• Build a web page to process video content as snapshots
• Use Amazon Rekognition to analyze those images
Process live video
• Stream video through Amazon Kinesis Video Stream
• StreamProcessor interacts with Amazon Rekognition Video
• Support for Archive and Sharing of Video
Custom image processing at the edge
• Amazon DeepLens and your own machine learning model

Further Links
Single Shot Multi Object Detection Inference Service
https://github.com/zhreshold/mxnet-ssd
Sagemaker
https://aws.amazon.com/sagemaker/
Image Classification Algorithm
https://docs.aws.amazon.com/sagemaker/latest/dg/image-classification.html
Kinesis Video Streams
https://aws.amazon.com/kinesis/video-streams/
DeepLens
https://aws.amazon.com/deeplens/
Amazon Elasticsearch service
https://aws.amazon.com/elasticsearch-service/

Please complete the session survey in the
summit mobile app.

Thank You
armpaul@amazon.co.uk
aws.amazon.com/blogs/machine-learning/

Using AWS to Ingest, Store, Archive, Share and carry out Analysis of Video Content.pdf

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Using AWS to Ingest, Store, Archive, Share and carry out Analysis of Video Content.pdf

Similar to Using AWS to Ingest, Store, Archive, Share and carry out Analysis of Video Content.pdf (20)

More from Amazon Web Services

More from Amazon Web Services (20)

Using AWS to Ingest, Store, Archive, Share and carry out Analysis of Video Content.pdf