Learn how to generate metadata from your media and make videos searchable by objects, people, activities, dialog, and more by using Amazon Machine Learning tools. Learn how to make videos more valuable and enable a wide range of use cases, including searching and indexing your video library. Learn how to use AWS Elemental MediaConvert to create video highlight clips from keywords, automatically or on demand, for content like sports videos and from sources like nature cameras or security feeds. Finally, learn how to use Amazon Transcribe and Amazon Translate with AWS Media Services to produce captions and translations to expand audience reach for corporate videos, marketing and sales material, and training videos.

1. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Bryan Samis
Specialized Solutions Architect, AWS
SRV322
Increase the Value of Video with ML &
Media Services

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why Machine Learning for Media?
Increase Engagement
and Content
Discovery
Accelerate existing
production processes
Launch new content
or services

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
• Quick Service Overview
• Content Indexing & Metadata
Generation for image and video
• Clipped Content Retrieval
• Media Analysis Solution

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Machine Learning Stack
Platforms
Application services
A m a z o n
R e k o g n i t i o n
A m a z o n
R e k o g n i t i o n
V i d e o
P o l l y T r a n s c r i b e T r a n s l a t e C o m p r e h e n dL e x
Amazon SageMaker Amazon Mechanical Turk
Frameworks KERAS
NVIDIA
Tesla V100 GPUs
(14x faster than P2)
P3
Machine Learning
AMIs
5,120 Tensor cores
128 GB of memory
1 Petaflop of compute
NVLink 2.0
Infrastructure
&

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Automating Footage Tagging with Amazon Rekognition
• Built in 3 weeks
• Indexed against 99,000 people
• Index created in one day
• Saved ~9,000 hours a year in manual
curation costs
• Live video with frame sampling
Previously, only about half of all footage was indexed due to the immense time
requirements required by manual processes

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Key Services For Today
Processing ML Orchestration
AWS Step Functions
AWS LambdaAmazon
Transcribe
Amazon
Rekognition
AWS Elemental
MediaConvert
AWS Elemental
MediaLive

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Elemental MediaConvert
• Professional grade video features and quality
• No software or hardware infrastructure to manage
• Automatically scales in response to variations in incoming
video volume
• Ability to manage capacity and control order in which jobs
are processed
• Pay for what you use, billed by the second of content
produced
AWS Elemental
MediaConvert

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Rekognition
Object & scene
detection
Facial
analysis
Face
comparison
Face
search
Celebrity
detection
Image
moderation
Text
detection
"Amazon Rekognition allows us to scalably
identify and track actors across millions of frames
of content with much higher reliability than any
other solution we've used.”
- Jared Browarnik, Co-Founder & CTO, TheTake
“Amazon Rekognition enables us to quickly and
efficiently add value through various automated
metadata tagging processes, and images and
video segments are much easier to find for our
enterprise and our customers.”
- Shane Murphy, Solutions Engineer, Scrippsnetworks

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Rekognition Video
Object, scene &
Activity detection
Face
search
Facial analysis Activity pathing
Unsafe content
detection
Celebrity
detection
Text in images

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Transcribe
A fully managed and continuously trained automatic speech recognition (ASR)
service that takes in audio and automatically generates accurate transcripts
Support for audio in
many formats and
low fidelity
§
Amazon S3
integration
Hello/
Hola
Time stamps and
confidence scores
English and SpanishPunctuation

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Transcribe – Use Cases
Call Center
Subtitles
MeetingsContent indexingCompliance

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Step Functions
VisualizeDefine Monitor
Task A single unit of work
Choice Adds branching logic
Parallel Fork and join the data
Wait Delay for a specified time
Fail Stops an execution
Succeed Stops an execution successfully
Pass Passes its input to its output

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Using Amazon ML Services for Media

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Using Amazon ML Services for Media
• Use services such as Amazon Rekognition &
Amazon Transcribe to generate metadata about
your content
• Store that metadata and make it searchable
• Retrieve only the portion of the content
you want
• Prepare it for timely use
Live and file
Sources
Amazon ML
Services
ML
Amazon
DynamoDB
Database
Live and file
Content
Content Indexing / Metadata Generation Content Retrieval / Action Metadata
AWS Elemental
Media Services
Media processing
AWS Elemental
Media Services
Media processing

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Content Indexing / Metadata Generation

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Content Indexing / Metadata Generation
File-based
content
Live
content
MediaLive
Kinesis
Video
Streams
MediaConvert
Amazon Rekognition
(Image)
• JPEG/PNG
• Up to 15 MB
Amazon Rekognition
(Video)
• H.264 video
• MP4/MOV file
• Up to 8 GB
Transcribe
• FLAC/MP3/WAV/MP
4
• Up to 2 hours
• Up to 1 GB

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Content Indexing / Metadata Generation − ImageTheChallenge
• A broadcaster
wants to add
metadata to
existing archive of
video content
• Index metadata
and video to make
it searchable
• Keep costs low
TheSolution
• Use AWS
Elemental
MediaConvert to
extract frames
from video content
• Use Amazon
Rekognition to
analyze and create
metadata for video
content
TheBenefit
• Video tagged with
objects, scene
and celebrities
detection
• Five-second frame
extraction keeps
cost low while
providing
searching index

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Content Indexing / Metadata Generation − Image
AWS Elemental
MediaConvert job
transcodes file and
extracts JPEG frames to S3
bucket.
AWS Lambda function
triggered by Amazon S3
object-created event tells
Amazon Rekognition to
analyze the JPEG file.
Amazon Rekognition
performs requested
operation on image (i.e.,
object detection, celebrity
recognition, etc.).
Amazon Rekognition returns result
to AWS Lambda, which stores tags
and confidence scores in Amazon
DynamoDB, Amazon Redshift,
Amazon Elasticsearch Service,
Amazon RDS, or whichever service
best suits the use case.
AWS Elemental
MediaConvert
File-based
processing
Amazon S3
Storage
AWS Lambda
Serverless
Amazon
Rekognition
ML / AI
Amazon
DynamoDB
Database
File
Source

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MediaConvert Configuration
Framerate
determines the
number of images
that will be extracted
from the video per
second. 1/5 indicates
to create one JPEG
every 5 seconds.

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Lambda Function Code Example – Amazon Rekognition Image

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Rekognition Image Results

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Demo

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Content Indexing / Metadata Generation – VideoTheChallenge
• A content producer
wants to log who is
in each scene of
new episode of a
show
• Raw video files are
~200 GB for 60 min
TheSolution
• Use AWS Elemental
MediaConvert to
compress video
content (but retain
quality
• Use Amazon
Rekognition Video to
analyze and create
metadata for video
content
TheBenefit
• Video tagged
celebrities detection
and timing and
position of celebrity
• Video files reduces
to <8 GB for 60 mins
to reduce costs

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Content Indexing / Metadata Generation – Video
AWS Elemental MediaConvert
job transcodes source file to
H.264/MP4 at a bit rate such that
the file size is <8 GB.
AWS Lambda function triggered by
Amazon S3 object-created event
tells Amazon Rekognition to analyze
the video file.
Amazon Rekognition Video
performs requested operation on
video (i.e., person tracking,
celebrity recognition, etc.).
Amazon Rekognition returns result to
AWS Lambda, which stores tags and
confidence scores in Amazon
DynamoDB, Amazon Redshift, Amazon
Elasticsearch Service, Amazon RDS, or
whichever service best suits the use
case.
AWS Elemental
MediaConvert
File-based
processing
Amazon S3
Storage
AWS Lambda
Serverless
Amazon
Rekognition
ML / AI
Amazon
DynamoDB
Database
File
Source

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MediaConvert Configuration
Use Container MPEG-
4 Container (MP4)
and a file extension of
mp4.
Set Video Codec to
MPEG-4 AVC (H.264).
Select bit rate accordingly
so output file is smaller
than 8 GB. For example, a
60-minute movie at 7 Mbps
will be approximately
3.2 GB.

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Lambda Function Code Example – Amazon Rekognition Video

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Lambda Function Code Example – Amazon Rekognition Video

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Rekognition Video Results

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Demo

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Content Indexing / Metadata Generation – TranscriptionTheChallenge
• An online training
provider has 1,000s of
hours of video that
need captions
• Video is in a variety of
formats
TheSolution
• Use AWS Elemental
MediaConvert create
audio only version of
content
• Use Amazon
Transcribe to generate
timestamped
transcription
• Convert Amazon
Transcribe output to
captions file
TheBenefit
• All formats of video
content get captions
added to make them
more accessible
• Option to run Amazon
Transcribe output
through Amazon
Translate to get multi-
language captions

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Content Indexing / Metadata Generation – Transcription
AWS Elemental MediaConvert
job transcodes source file,
creating audio-only rendition for
Amazon Transcribe
AWS Elemental
MediaConvert also
creates normal
audio/video output
AWS Lambda function triggered
by Amazon S3 object-created
event creates a new Transcribe
job
Amazon Transcribe outputs
JSON file of detected words
and timing
Lambda function converts Amazon
Transcribe JSON into subtitle format
(such as WebVTT, SRT, or TTML) and
delivers to
Amazon S3 bucket with content
AWS Elemental
MediaConvert
File-based
processing
AWS Lambda
Serverless
Amazon
Transcribe
ML / AI
File
Source
Amazon S3
Storage
Amazon S3
STORAGE

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MediaConvert Configuration

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Lambda Function Code Example − Amazon Transcribe

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Polling Transcribe with AWS Step Functions

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Transcribe Results

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Transcribe to Captions

37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Transcribe to Captions Approaches
Cadence Heuristic – Use larger than
average gaps between words as delimiter
Fixed Duration – Aggregate all words
within a given duration
What about scene changes? Non-dialog sound
elements?

38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Content Retrieval

39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Content Retrieval with MediaConvert ClippingTheChallenge
• The content
producer would like
to create a promo
clip of all of the
scenes from their
episode that contain
a particular actor.
• Remember, the
source file is 60
minutes long and
200 GB.
TheSolution
• Amazon Rekognition
video facial
recognition identifies
when the star
appears in the source
video.
• AWS Elemental
MediaConvert uses
time references to
selectively transcode
source video.
TheBenefit
• Faster and more
cost-effective clip
generation as only
the video contents
that has been
identified as
featuring the
celebrity is
transcoded.

40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Content Retrieval with MediaConvert Clipping
AWS Elemental MediaConvert
transcodes clips from the source file,
using only the time range(s) specified
AWS Elemental
MediaConvert
File-based
processing
Amazon S3
Storage
Clipped file
Output
Amazon
DynamoDB
Database
AWS Lambda
Serverless
Lambda function
queries database for
metadata being
searched
Lambda function creates
MediaConvert transcode job
specifying time(s) from
source to clip

41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
MediaConvert Input Clipping Configuration

42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Media Analysis Solution
https://aws.amazon.com/answers/media-entertainment/media-analysis-solution/
• Generate searchable metadata from
your media assets using Amazon
Rekognition, Amazon Transcribe,
Amazon Comprehend, and Amazon
Elasticsearch Service
• Deploy in minutes with a single click
using AWS CloudFormation
• Interact via API or demo web UI
• Orchestrated with Step Functions,
extensible and easily customizable

43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Thanks!
Please complete the session survey
in the mobile app.
samisb@amazon.com

44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Submit session feedback
1. Tap the Schedule icon.
2. Select the session you
attended.
3. Tap Session Evaluation to
submit your feedback.