Learn how to generate metadata from your media and make videos searchable by objects, people, activities, dialog, and more by using Amazon Machine Learning tools. Make videos more valuable and enable a wide range of use cases, including searching and indexing your video library. Use AWS Elemental MediaConvert to create video highlight clips from keywords, automatically or on demand, for content like sports videos and from sources like nature cameras or security feeds. Use Amazon Transcribe and Amazon Translate with AWS Media Services to produce captions and translations to expand audience reach for corporate videos, marketing and sales material, and training videos.

1. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Bryan Samis
Solutions Architect
SRV322
Create Amazing Video
Experiences and Increase the
Value of Content Using Artificial
Intelligence and AWS Media
Services

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
• Brief introduction to services
• Using ML, AI, and deep learning with media
• Content indexing and metadata generation
• Add searchable metadata to a video archive
• Log when celebrities appear in new episodes
• Generate captions for a collection of video assets
• Retrieving content
• Transcode just the video clip that contains a specific person
• Everything on the internet is better with cats

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Services Used
• Amazon Rekognition
• AWS Lambda
• AWS Elemental MediaConvert
• AWS Elemental MediaLive
• Amazon Transcribe
• Amazon Translate

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Rekognition
• Image and video recognition powered by Amazon AI/ML technology
• Image recognition
• Object & scene detection
• Facial detection & recognition
• Facial analysis & sentiment
detection
• Image moderation
• Celebrity recognition
• Text in images
• Video recognition
• Activity detection
• Person tracking

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Rekognition
File Requirements
• Image recognition
• JPEG or PNG image
• Up to 15 MB
• Video recognition
• MOV or MP4 file with H.264 video
• Up to 8 GB

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Elemental MediaConvert
AWS Elemental MediaConvert is a file-based video processing service that enables anyone,
with any size content library, to easily and reliably transcode on-demand content for broadcast
and multiscreen delivery
• Access to professional grade video features and quality
• No software or hardware infrastructure to manage
• Automatically scales in response to variations in incoming video volume
• Ability to manage capacity and control order in which jobs are processed

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Transcribe
• Speech transcription service powered by Amazon AI/ML technology
• Launched in preview at re:Invent 2017
• Input file types accepted are FLAC, MP3, MP4, and WAV
• Files can be up to 2 hours
• Produces JSON output with full transcription and word timing

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Transcribe
• Common Media & Entertainment use cases for Amazon Transcribe include:
• Generating automated closed captions and subtitles
• Indexing audio content
• Detecting compliance and objectionable content

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Using Amazon AI/ML Services for
Media

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Using Amazon AI/ML Services for Media
Content Indexing & Metadata Generation
• Use services such as Amazon Rekognition
& Amazon Transcribe to generate metadata
about your content
• Store that metadata and make it searchable
Content Retrieval & Action Metadata
• Database tells you the scene exists in
a given file at a given time
• Retrieve it for timely use
Live and File
SOURCES
AWS Elemental
Media Services
MEDIA
PROCESSING
Amazon ML/AI
Services
ML / AI
Amazon
DynamoDB
DATABASE
Live and File
CONTENT
AWS Elemental
Media Services
MEDIA
PROCESSING

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Content Indexing & Metadata
Generation

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Content Indexing & Metadata Generation
To send content to AWS AI/ML services, we need to
compress video, extract frames, and extract audio
File-based content
• AWS Elemental MediaConvert
• Amazon Elastic Transcoder
Live content
• AWS Elemental MediaLive
• Amazon Kinesis Video Data Streams

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Content Indexing & Metadata Generation –
AWS Elemental MediaConvert and Amazon Rekognition
Thechallenge
• A broadcaster
wants to add
metadata to
existing archive of
video content
• Index metadata
and video to make
it searchable
• Keep costs low
Thesolution
• Use AWS
Elemental
MediaConvert to
extract frames
from video content
• Use Amazon
Rekognition to
analyze and create
metadata for video
content
Thebenefit
• Video tagged with
objects, scene
and celebrities
detection
• Five-second frame
extraction keeps
costs low while
providing
searching index

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Content Indexing & Metadata Generation –
AWS Elemental MediaConvert and Amazon Rekognition
AWS Elemental
MediaConvert job
transcodes file and
extracts JPEG frames
to S3 bucket
AWS Lambda function
triggered by S3 object-
created event tells Amazon
Rekognition to analyze the
JPEG file
Amazon Rekognition
performs requested
operation on image (i.e.,
object detection, celebrity
recognition, etc.)
Amazon Rekognition returns
result to AWS Lambda, which
stores tags and confidence
scores in Amazon DynamoDB,
Amazon Redshift, Amazon
Elasticsearch Service, Amazon
RDS, or whichever service best
suits the use case
• Use AWS Elemental MediaConvert to extract still frames from a video
AWS Elemental
MediaConvert
FILE-BASED
PROCESSING
Amazon S3
STORAGE
AWS Lambda
SERVERLESS
Amazon
Rekognition
ML / AI
Amazon
DynamoDB
DATABASE
File
SOURCE

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Content Indexing & Metadata Generation –
AWS Elemental MediaConvert and Amazon Rekognition
Add new file output group to an AWS Elemental MediaConvert job

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Content Indexing & Metadata Generation –
AWS Elemental MediaConvert and Amazon
RekognitionAdd frame capture (JPEG) output the job
Framerate
determines the
number of images
that will be
extracted from the
video per second.
1/5 indicates to
create one JPEG
every 5 seconds.

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Rekognition
AWS Lambda function to invoke Amazon Rekognition on our extracted JPEG to
detect celebrities

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Rekognition
Result from our image

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Live Demo

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Content Indexing & Metadata Generation –
AWS Elemental MediaConvert and Amazon Rekognition
Video
Thechallenge
• A content producer
wants to log who is
in each scene of
new episode of a
show
• Raw video files are
~200 GB for 60 min.
Thesolution
• Use AWS Elemental
MediaConvert to
compress video
content (but retain
quality
• Use Amazon
Rekognition Video to
analyze and create
metadata for video
content
Thebenefit
• Video tagged
celebrities detection
and timing and
position of celebrity
• Video files reduces
to <8 GB for 60 min.
to reduce costs

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Content Indexing & Metadata Generation –
AWS Elemental MediaConvert and Amazon Rekognition
Video
AWS Elemental MediaConvert
job transcodes source file to
H.264/MP4 at a bit rate such
that the file size is <8 GB.
AWS Lambda function triggered
by S3 object-created event tells
Amazon Rekognition to analyze
the video file.
Amazon Rekognition Video
performs requested operation on
video (i.e., person tracking,
celebrity recognition, etc.).
Amazon Rekognition returns result
to AWS Lambda, which stores tags
and confidence scores in Amazon
DynamoDB, Amazon Redshift,
Amazon Elasticsearch Service,
Amazon RDS, or whichever service
best suits the use case.
Use AWS Elemental MediaConvert to compress files >8 GB and feed it
to Amazon Rekognition
AWS Elemental
MediaConvert
FILE-BASED
PROCESSING
Amazon S3
STORAGE
AWS Lambda
SERVERLESS
Amazon
Rekognition
ML / AI
Amazon
DynamoDB
DATABASE
File
SOURCE

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Elemental MediaConvert
Add H.264/MP4 output to MediaConvert job

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Elemental MediaConvert
Add H.264/MP4 output to AWS
Elemental MediaConvert job
Use container
MPEG-4 Container
(MP4) and a file
extension of mp4.
Set Video codec to
MPEG-4 AVC
(H.264).
Select bit rate
accordingly so output file
is smaller than 8 GB. For
example, a 60-min.
movie at 7 Mbps is
approximately 3.2. GB

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Rekognition Video
AWS Lambda function to invoke Amazon Rekognition on our transcoded
video to detect celebrities

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Rekognition Video
Result from our video

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Live Demo

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Content Indexing / Metadata Generation –
AWS Elemental MediaConvert and Amazon Transcribe
Thechallenge
• An online training
provider has 1,000s of
hours of video that
need captions
• Video is in a variety of
formats
Thesolution
• Use AWS Elemental
MediaConvert create
audio only version of
content
• Use Amazon
Transcribe to generate
timestamped
transcription
• Convert Amazon
Transcribe output to
captions file
Thebenefit
• All formats of video
content get captions
added to make them
more accessible
• Option to run Amazon
Transcribe output
through Amazon
Translate to get multi-
language captions

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Transcribe
AWS Elemental MediaConvert
job transcodes source file,
creating audio-only rendition
for Amazon Transcribe
AWS Elemental
MediaConvert also
creates normal audio
& video output
AWS Lambda function
triggered by S3 object-
created event creates a new
Transcribe job
Amazon Transcribe
outputs JSON file of
detected words and timing
AWS Lambda function converts
Amazon Transcribe JSON into
subtitle format (such as
WebVTT, SRT, or TTML) and
delivers to S3 bucket with
content
AWS Elemental
MediaConvert
FILE-BASED
PROCESSING
AWS Lambda
SERVERLESS
Amazon
Transcribe
ML / AI
File
SOURCE
Amazon S3
STORAGE
Amazon S3
STORAGE

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Elemental MediaConvert
Add audio-only WAV output to the job. Start by adding an additional file output group.

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Elemental MediaConvert
Configure audio-only uncompressed WAV output

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Transcribe
AWS Lambda function to create Amazon Transcribe job from WAV file
created by AWS Elemental MediaConvert

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Transcribe
Transcribe creates JSON file with complete
transcription, and word-by-word timing

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Transcribe
Must convert Amazon Transcribe JSON into usable closed caption or subtitle
format, such as SRT
• Not a trivial problem. Need to determine sentence boundaries and which words
to combine into the same captions.
Example:

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Transcribe
• Some ideas for ways to tackle this problem:
• Calculate the cadence of the wording, and look for larger-than-average gaps between
words. Use these points as our breaks.
• Use a fixed caption duration of 1–2 seconds, and “aggregate” all words that fall within
that duration.
• None of these methods are perfect. Analyzing audio alone won’t necessarily account for
scene changes, gaps in dialog, nondialog sound elements, etc.
• But they can get us close.

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Content Retrieval

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Content Retrieval
AWS Elemental MediaConvert
Thechallenge
• The content
producer would like
to create a promo
clip of all of the
scenes from their
episode that contain
a particular actor
• Remember, the
source file is 60
minutes long and
200 GB
Thesolution
• Rekognition Video’s
facial recognition,
identifies when the
star appears in the
source video
• AWS Elemental
MediaConvert uses
time references to
selectively transcode
source video
TheBenefit
• Faster and more
cost-effective clip-
generation as only
the video contents
that has been
identified as
featuring the
celebrity is
transcoded

37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Content Retrieval AWS Elemental
MediaConvert
transcodes clips from
the source file, using
only the time range(s)
specified
AWS Elemental
MediaConvert
FILE-BASED
PROCESSING
Amazon S3
STORAGE
Clipped File
OUTPUT
Amazon
DynamoDB
DATABASE
AWS Lambda
SERVERLESS
Lambda function
queries database
for metadata being
searched
Lambda function creates
MediaConvert transcode
job specifying time(s) from
source to clip

38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Content Retrieval
Use AWS Elemental MediaConvert “Input Clipping” feature to clip a file to specific times

39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Live Demo

40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Bringing It All Together – The Cat-Finder 5001!

41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cat-Finder 5001
INTERFACE
LIVEINDEXER
Website
Amazon
Rekognition
ML/AI
Amazon
DynamoDB
DATABASE
AWS Lambda
PREKOG
AWS Lambda
VOD
AWS Lambda
AWS Lambda
PARSE
Amazon S3
STORAGE
jpg json mp4
Amazon
CloudWatch
m3u8
ffprobe / ffmpeg
LIVE VIDEO DELIVERY
AWS Elemental
MediaConvert
FILE-BASED
PROCESSING
AWS Elemental
MediaTailor
PERSONALIZATION AND
MONETIZATION
AWS Elemental
MediaPackage
ORIGINATION AND
JIT PACKAGING
Amazon
CloudFront
CDN
AWS Elemental
Live
LIVE VIDEO
PROCESSING
AWS Elemental
MediaLive
LIVE VIDEO
PROCESSING
AWS Elemental
MediaStore
MEDIA-OPTIMIZED
STORAGE
Live TV
SOURCE
DISPLAY DEVICES

42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cat-Finder 5001 Demo

43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Video Analytics Starter Kit – Available Now!
https://aws.amazon.com/answers/media-entertainment/video-frame-based-analysis/
A m a z o n R e k o g n i t i o n I m a g e
V i d e o A n a l y t i c s S t a r t e r
K i t
f o r s p e c i f i c u s e c a s e s :
O b j e c t , s c e n e , & f a c e
s e n t i m e n t d e t e c t i o n a n d
a n a l y s i s
• I m a g e m o d e r a t i o n
• C e l e b r i t y r e c o g n i t i o n
• T e x t I m a g e r e c o g n i t i o n
A m a z o n R e k o g n i t i o n V i d e o
O b j e c t & a c t i v i t y
d e t e c t i o n
P e r s o n t r a c k i n g
R e a l - t i m e l i v e s t r e a m i n g
V i d e o A n a l y t i c s S t a r t e r K i t E x p a n s i o n s w i l l e v e n t u a l l y a l s o i n c l u d e
f e a t u r e s f r o m o t h e r A I s e r v i c e s , s u c h a s A m a z o n T r a n s c r i b e a n d
A m a z o n T r a n s l a t e .
• M e t a d a t a g e n e r a t i o n
• V i d e o s e a r c h b y c o n t e n t
• V i d e o s e a r c h b y i m a g e

44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Custom Media Analysis Solutions
AWS Media Services can output raw video, audio, or frame captures to Amazon S3,
and then any AI/ML solution can process and take action
Managed Services
Amazon Rekognition
Amazon Transcribe
Amazon Translate
Amazon Comprehend
Third-party services
… and others
Custom Solutions
Amazon SageMaker
MXNet on EC2
TensorFlow on EC2
OpenCV on EC2
… and others
AWS Elemental
MediaConvert
FILE-BASED
PROCESSING
Amazon S3
STORAGE
File
SOURCE

45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Please complete the session survey in
the summit mobile app.

46. Submit Session Feedback
1. Tap the Schedule icon. 2. Select the session
you attended.
3. Tap Session
Evaluation to submit your
feedback.

47. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Thank you!