The document discusses using machine learning and AWS media services for video analysis and enrichment. It provides examples of ML use cases for media like automated metadata tagging, closed captioning, and content recommendations. It then describes the AWS media analysis solution and live captioning solution, which apply services like Amazon Rekognition, Transcribe, and Translate to generate metadata and subtitles from video content.

1. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Increase the value of video using ML and AWS
media services
Scott Woods
Solutions architect
AWS
S V C 3 0 1

2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Agenda
Why machine learning (ML) for media?
Use cases and relevant services
AWS solutions
• Media analysis solution
• Live captioning and translation solution

3. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

4. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Major shift in the
Media and Entertainment
industry
Content is no longer king
The viewer is king
Desired shift to a direct-to-consumer
business model
Success relies on a rich viewer experience
Rich viewer experiences rely on data and
analytics

5. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Modern data architecture for media enrichment
Insightstoenhanceviewerengagement,personalization,monetization
Improve UX/QoS
Recommend
Monetize
Infer
Store
and process
Content lake on AWS
Storage | Archival storage | Data catalog
Analyze
Viewer data
Internet of Things
(IoT) sensors
Social media
Video
AMAZON
QUICKSIGHT
AMAZON
SAGEMAKER
Engage and personalize

6. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

7. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Sample ML use cases for media
Automate the creation of rich metadata index (object, scene, activity, faces, etc.), extracted from audio-visual
content and integrated into digital and media asset management systems
AWS services to build use case: Amazon Rekognition, Amazon SageMaker, Amazon Transcribe, Amazon Comprehend
M e d i a m e t a d a t a
t a g g i n g
Automated captions, transcription, and translation of audio content
AWS services to build use case: Amazon Transcribe, Amazon Translate, Amazon Comprehend
C l o s e d c a p t i o n i n g
Detect potentially inappropriate content to avoid compliance issues in global markets and to increase brand
safety for advertisers
AWS services to build use case: Amazon Rekognition, Amazon Transcribe, Amazon SageMaker
A u t o m a t e d
c o m p l i a n c e m a r k i n g
Identify objects and emotion in content that enables users to integrate personalized ads into subscriber video
streams
AWS services to build use case: Amazon Spark/Amazon EMR, Amazon SageMaker, AWS Elemental MediaTailor
A d p e r s o n a l i z a t i o n

8. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Sample ML use cases for media
Identify and path player activity in the field or identify the precise timecodes when actors enter and leave a
scene, both visually and in spoken dialog (off-screen voice)
AWS services to build use case: Amazon Rekognition, Amazon Transcribe
P l a y e r / a c t o r
i d e n t i f i c a t i o n a n d
a c t i v i t y p a t h i n g
Translate transcripts and metadata. Improve localization workflows and search experience
AWS services to build use case: Amazon Translate
L a n g u a g e
t r a n s l a t i o n
Analysis disparate data and enable the consumer to make personalized content choices and the business to
predict behavior
AWS services to build use case: Amazon SageMaker, Amazon Personalize
C o n t e n t
r e c o m m e n d a t i o n s
Detect and pixelate faces captured incidentally to preserve the privacy of non-persons of interest in news feeds
and security footage
AWS services to build use case: Amazon Rekognition, Amazon Transcribe, Amazon SageMaker
A u t o m a t e d
r e d a c t i o n

9. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Use case: From manual tagging to
AI-based tagging
Background
• Over 200,000 hours of content
• Only half of content is manually tagged
• How can we enrich our metadata in AWS?
• How can we unleash the value of content we already own
once in AWS?
Challenge
• Large-scale video library
• High accuracy required
• Limited budget
• Ability to extract from video
• Keep up with daily increase in content
Results
• Solution developed within 3 weeks
• Live video-frame-based analysis
• Established, searchable baseline archive
• All content is now tagged and indexed
• Over 99,000 faces indexed and searchable
• Saved ~9,000 hours a year in manual curation costs

10. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Use case: Event broadcasting enrichment

11. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon ML stack
M L f r a m e w o r k s a n d
i n f r a s t r u c t u r e
A I s e r v i c e s
Amazon Rekognition
Image
Amazon
Polly
Amazon
Transcribe
Amazon
Translate
Amazon Comprehend
and Amazon Comprehend
Medical
Amazon LexAmazon Rekognition
Video
Vision Speech Chatbots
A m a z o n
S a g e M a k e r
Build Train
Amazon
Forecast
Amazon
Textract
Amazon
Personalize
Deploy
Pre-built algorithms and notebooks
Data labeling (Amazon SageMaker Ground Truth)
One-click model training and tuning
Optimization (Amazon SageMaker Neo)
One-click deployment and hosting
M L s e r v i c e s
Frameworks Interfaces Infrastructure
Amazon Elastic Cloud
Compute (Amazon EC2)
P3 and P3dn
Amazon EC2
C5
FPGAs AW IoT Greengrass Amazon Elastic
Inference
Hyperparameter optimizationAlgorithms and models (AWS Marketplace
for Machine Learning)
Language Forecasting Recommendations
Notebook hosting
Reinforcement learning
Auto-scaling
Virtual private cloud
Private Link
Amazon Elastic Inference integration

13. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon Rekognition
Deep learning–based image-analysis service
Object and scene
detection
Facial
analysis
Face search
and match
Text in imageContent moderation Celebrity recognition

14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon Rekognition Video
Analysis for live or stored video content

15. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Amazon Transcribe
Automatic speech recognition for stored or real-time audio
Time
stamps
Support for
both regular and
telephony audio
Punctuation
and
formatting
English,
Spanish,
French, Italian,
Brazilian
Portuguese,
Korean,
German
Recognize
multiple
speakers
Custom
dictionaries
? !

16. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

17. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
AWS media solutions
Accelerate revenue
Get started with your project quickly and start
monetizing your content
One-click launch
Launch a fully tested solution starter kit in your
AWS account with one click
Well-architected
Rely on the architecture as it reflects AWS best
practices
Agility
Save the resources of developing a solution
from the beginning
Expand and add
Use the framework and available guides to expand the solution
starter kit based on your needs

18. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
AWS media solutions
Go to: aws.amazon.com/media/solutions

19. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

20. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Media analysis solution
Label and face detection
Celebrity detection
Face search
Activity tracking
Video-on-demand subtitle generation
Key entity and phrase detection
Automated
metadata
generationAmazon
Rekognition
Amazon
Rekognition Video
Amazon
Transcribe
Amazon
Translate
Amazon
Comprehend
Transcription
Context
Amazon SageMaker + additional solutions for media use cases

21. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Media analysis solution
Amazon Rekognition
Amazon Transcribe
Amazon Comprehend
Amazon Simple
Storage Service
(Amazon S3)
AWS Step
Functions
Lambda
Amazon
Elasticsearch
Service
Image Video
AudioText
Amazon SageMaker

22. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

23. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

24. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Live subtitling solution
• Purpose-built to help customers to create
live streaming subtitling workflows on the
AWS cloud
• Serverless architecture
• Uses new Amazon Transcribe Streaming to
extract speech for subtitles, then Amazon
Translate to convert to other subtitle
languages
• Deploy in minutes with a single click using a
AWS CloudFormation template
• Automatically spins up all resources needed
for the entire workflow
AWS Elemental
MediaLive
Lambda function
(caption creation)
Amazon
Transcribe
Amazon S3
bucket
Lambda function
(Amazon Transcribe Streaming)
AWS Elemental
MediaPackage
Amazon
Translate
Manifests
Live channel source
Amazon S3
bucket
Multi-language (optional)
PCM audio
Transcripts
Subtitles, manifests, and video files
PCM audio
Amazon
CloudFront

25. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T
Live subtitling solution
• Purpose-built to help customers to create
Live Streaming Subtitling workflows on the
AWS Cloud
• Serverless architecture
• Uses new Amazon Transcribe Streaming to
extract speech for subtitles. Then Amazon
Translate to convert into other subtitle
languages
• Deploy in minutes with a single click using a
CloudFormation template
• Automatically spins up all resources needed
for the entire workflow
• Optionally add text to speech with Amazon
Polly or use Amazon Comprehend to
understand text sentiment
AWS Elemental
MediaLive
Lambda function
(caption creation)
Amazon
Transcribe
Amazon S3
bucket
Lambda function
(Amazon Transcribe Streaming)
AWS Elemental
MediaPackage
Amazon
Translate
Manifests
Amazon S3
bucket
Multi-language (optional)
PCM audio
Transcripts
Subtitles, manifests, and video files
PCM audio
Amazon
CloudFront
Amazon
Polly
Amazon
Comprehend
Alternate audio (optional) Insights and sentiment (optional)

26. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

27. Thank you!
S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Scott Woods
mswoods@amazon.com