SlideShare a Scribd company logo
1 of 26
Download to read offline
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS re:INVENT
Building a Serverless Pipeline to
Transcode a Two-Hour Video in Minutes
L u k e Y o u n g b l o o d — P r i n c i p a l S o l u t i o n s A r c h i t e c t , A W S
G r e g F e m e c — S o f t w a r e D e v e l o p e r , R e v v e l
S R V 3 1 4
N o v e m b e r 2 7 , 2 0 1 7
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Agenda
Overview of serverless technology
Introduction and challenges
Building a serverless transcoding pipeline
Lessons learned
Best practices and recommendations
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
No servers to provision
or manage
Scales with usage
Never pay for idle Availability and fault
tolerance built in
Serverless means…
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Serverless application
SERVICES (ANYTHING)
Changes in
data state
Requests to
endpoints
Changes in
resource state
EVENT SOURCE FUNCTION
Node.js
Python
Java
C#
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Common AWS Lambda Use Cases
Web
applications
• Static
websites
• Complex web
apps
• Packages for
Flask and
Express
Data
processing
• Real time
• MapReduce
• Batch
Chatbots
• Powering
chatbot logic
Backends
• Apps and
services
• Mobile
• IoT
</></>
Amazon
Alexa
• Powering
voice-enabled
apps
• Alexa Skills
Kit
IT
automation
• Policy engines
• Extending
AWS services
• Infrastructure
management
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Introduction
Who am I?
• Greg Femec—Dev at Revvel
What does Revvel do?
• Revvel is a small team that has
experience innovating at places
like Hulu, Netflix, and Amazon
• We are focused on accelerating
Verizon’s efforts in building a
next-generation television service
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
The Challenge
Lots of
video to
transcode
Unpredictable load
Inputs from 30-second 480p clips to barely
compressed full-length films
Wide range of input formats
Lots of
output
formats to
support
Progressive
HLS (via ts chunks)
DASH (via fragmented mp4)
DRM
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Early Solutions
Started
with
SaaS
Solutions
Didn’t have enough control
Not cost effective
Built a
Transcode
Farm on
Amazon
EC2
Solved some initial problems
Created operations challenges
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Operations Challenges with Amazon EC2
Scaling Wanted to avoid queuing
Cold start problem
Excess,
unused
capacity
Long running jobs
1-hour minimum (now 1 sec.)
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Why Serverless?
0 to 100 real quick No wasted capacity Fewer operations
challenges
Large standby capacity Flexible resource allocation
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Challenges of Video Transcoding
Big
files
Don’t want to download entire file
Hard to stream video input and output
Slow
jobs
HD transcode of movies takes
many hours
Hard to pause and resume jobs
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Solution for Efficiently Transcoding Videos
Divide
and
conquer
Transcode movie in 5-second chunks
Use a MapReduce-type approach
Directly use for HLS
Merge together for Progressive, DASH
Avoid storing full files on disk
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Services
AWS
Lambda
Instant scale
Ability to parallelize workloads
Amazon
S3
AWS
Lambda
Amazon
S3
Multi-part uploads
Range requests
FFmpeg Swiss-army knife of transcoding
Python Supported by AWS Lambda
Python
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Architecture Diagram for Serverless
Transcoding
Cache
Chunk Lambda Function 1
Cache
Cache
S3 Source
Storage
S3 Output
Storage
(.ts)
Cache
Merge Function
Encrypt Function
Encrypt Function
OpenSSL
Encrypt Function
OpenSSL
Encrypt Function
OpenSSL
S3 Output
Storage
(.mp4)
S3 Output
Storage
(HLS)
S3 Output
Storage
(DASH)Chunk Lambda Function 2
Chunk Lambda Function N
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Chunk Function
Decode a small
portion of video
Transcode to
desired
resolution/bitrate
Upload .ts chunk
to S3
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Decoding Challenge and Solution
Large
input files
FFmpeg supports http input via range requests but...
Too many requests from seeking
We built a
cache to
mirror S3
HTTP daemon running in Lambda function
Proxy all read requests from FFmpeg to Amazon S3
Store 5 MB chunks of files in memory
Keep read requests open
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
FFmpeg Challenges
Static
binaries
FFmpeg was unreliable if dynamically
linked
Larger code size, but still fits within
Lambda limits
Forked
process
Ignore stdin
Be careful with container re-use
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Merge Function
Read in .ts
chunks
Combine
into MP4
Upload to
S3
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Uses a multi-part S3 upload
Delays uploading the first chunk for moov atom
Lies to FFmpeg when seeking or reading too far
Files too big to fit in /tmp
Built a custom FTP daemon
FFmpeg supports FTP output, but needs to seek
Output File Size Challenge
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Concat Function
Read in .ts
chunks
Combine /
Encrypt
Upload to
S3
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Execution Challenges
Time
challenge
Sample AES requires repackaging
Longer movies wouldn't finish in a single function
Solution Used a single multi-part upload across multiple
functions
Be careful about back pressure when parallelizing
downloads
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
GIT and
Docker
Hashed
Code
.zip
— Please have tests (even a simple health check)
Lambda
Versions/
Aliases
Deployment and CI/CD Pipeline
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
General Lambda Advice
Be aware of
container
reuse
Disk
Memory
Ports
Logging If running multiple processes, consider debugging with ps, top, etc.
Variable
run-time
Be prepared to retry functions that time-out, particularly with network-
bound functions
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Results
Statistics ~150K hours of transcoded video
~400 million Lambda function invocations
Cost
savings
Similar on-demand EC2 capacity would
cost ~$60K a month
Average Lambda bill was ~$6K per
month for the past 6 months
Speed Transcoding a 2 hour movie on EC2 takes ~4-6 hours
On Lambda, it takes less than 10 minutes to transcode
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Final Thoughts
Limitations 5 minutes to save the world
Must be able to efficiently seek in input video file
Higher price per CPU hour
Decoding overhead
Future
plans
More output formats
Better efficiency for small files
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank you!

More Related Content

What's hot

GPSTEC305-Machine Learning in Capital Markets
GPSTEC305-Machine Learning in Capital MarketsGPSTEC305-Machine Learning in Capital Markets
GPSTEC305-Machine Learning in Capital MarketsAmazon Web Services
 
CMP323_AWS Batch Easy & Efficient Batch Computing on Amazon Web Services
CMP323_AWS Batch Easy & Efficient Batch Computing on Amazon Web ServicesCMP323_AWS Batch Easy & Efficient Batch Computing on Amazon Web Services
CMP323_AWS Batch Easy & Efficient Batch Computing on Amazon Web ServicesAmazon Web Services
 
ALX202_Integrate Alexa voice technology into your product with the Alexa Voic...
ALX202_Integrate Alexa voice technology into your product with the Alexa Voic...ALX202_Integrate Alexa voice technology into your product with the Alexa Voic...
ALX202_Integrate Alexa voice technology into your product with the Alexa Voic...Amazon Web Services
 
MCL306_Making IoT Smarter with AWS Rekognition.pdf
MCL306_Making IoT Smarter with AWS Rekognition.pdfMCL306_Making IoT Smarter with AWS Rekognition.pdf
MCL306_Making IoT Smarter with AWS Rekognition.pdfAmazon Web Services
 
DAT321_How Careem Used Amazon ElastiCache for Redis to Accelerate Their Ride ...
DAT321_How Careem Used Amazon ElastiCache for Redis to Accelerate Their Ride ...DAT321_How Careem Used Amazon ElastiCache for Redis to Accelerate Their Ride ...
DAT321_How Careem Used Amazon ElastiCache for Redis to Accelerate Their Ride ...Amazon Web Services
 
CON213_Hands-on Kubernetes on AWS
CON213_Hands-on Kubernetes on AWSCON213_Hands-on Kubernetes on AWS
CON213_Hands-on Kubernetes on AWSAmazon Web Services
 
CMP314_Bringing Deep Learning to the Cloud with Amazon EC2
CMP314_Bringing Deep Learning to the Cloud with Amazon EC2CMP314_Bringing Deep Learning to the Cloud with Amazon EC2
CMP314_Bringing Deep Learning to the Cloud with Amazon EC2Amazon Web Services
 
GPSWKS408-GPS Migrate Your Databases with AWS Database Migration Service and ...
GPSWKS408-GPS Migrate Your Databases with AWS Database Migration Service and ...GPSWKS408-GPS Migrate Your Databases with AWS Database Migration Service and ...
GPSWKS408-GPS Migrate Your Databases with AWS Database Migration Service and ...Amazon Web Services
 
How to Assess Your Organization's Readiness to Migrate at Scale to AWS - ENT2...
How to Assess Your Organization's Readiness to Migrate at Scale to AWS - ENT2...How to Assess Your Organization's Readiness to Migrate at Scale to AWS - ENT2...
How to Assess Your Organization's Readiness to Migrate at Scale to AWS - ENT2...Amazon Web Services
 
GPSBUS220-Refactor and Replatform .NET Apps to Use the Latest Microsoft SQL S...
GPSBUS220-Refactor and Replatform .NET Apps to Use the Latest Microsoft SQL S...GPSBUS220-Refactor and Replatform .NET Apps to Use the Latest Microsoft SQL S...
GPSBUS220-Refactor and Replatform .NET Apps to Use the Latest Microsoft SQL S...Amazon Web Services
 
STG203_Get Rid of Tape and Modernize Backup with AWS
STG203_Get Rid of Tape and Modernize Backup with AWSSTG203_Get Rid of Tape and Modernize Backup with AWS
STG203_Get Rid of Tape and Modernize Backup with AWSAmazon Web Services
 
GPSTEC319-Build Once Deploy Many Architecting and Building Automated Reusable...
GPSTEC319-Build Once Deploy Many Architecting and Building Automated Reusable...GPSTEC319-Build Once Deploy Many Architecting and Building Automated Reusable...
GPSTEC319-Build Once Deploy Many Architecting and Building Automated Reusable...Amazon Web Services
 
GPSWKS301_Comprehensive Big Data Architecture Made Easy
GPSWKS301_Comprehensive Big Data Architecture Made EasyGPSWKS301_Comprehensive Big Data Architecture Made Easy
GPSWKS301_Comprehensive Big Data Architecture Made EasyAmazon Web Services
 
DEV305_Manage Your Applications with AWS Elastic Beanstalk.pdf
DEV305_Manage Your Applications with AWS Elastic Beanstalk.pdfDEV305_Manage Your Applications with AWS Elastic Beanstalk.pdf
DEV305_Manage Your Applications with AWS Elastic Beanstalk.pdfAmazon Web Services
 
ARC303_Running Lean Architectures How to Optimize for Cost Efficiency
ARC303_Running Lean Architectures How to Optimize for Cost EfficiencyARC303_Running Lean Architectures How to Optimize for Cost Efficiency
ARC303_Running Lean Architectures How to Optimize for Cost EfficiencyAmazon Web Services
 
Scaling Up to Your First 10 Million Users
Scaling Up to Your First 10 Million UsersScaling Up to Your First 10 Million Users
Scaling Up to Your First 10 Million UsersAmazon Web Services
 
DEV326_DevOps Essentials An Introductory Workshop on CICD Practices
DEV326_DevOps Essentials An Introductory Workshop on CICD PracticesDEV326_DevOps Essentials An Introductory Workshop on CICD Practices
DEV326_DevOps Essentials An Introductory Workshop on CICD PracticesAmazon Web Services
 

What's hot (20)

GPSTEC305-Machine Learning in Capital Markets
GPSTEC305-Machine Learning in Capital MarketsGPSTEC305-Machine Learning in Capital Markets
GPSTEC305-Machine Learning in Capital Markets
 
ARC205_Born in the Cloud
ARC205_Born in the CloudARC205_Born in the Cloud
ARC205_Born in the Cloud
 
CMP323_AWS Batch Easy & Efficient Batch Computing on Amazon Web Services
CMP323_AWS Batch Easy & Efficient Batch Computing on Amazon Web ServicesCMP323_AWS Batch Easy & Efficient Batch Computing on Amazon Web Services
CMP323_AWS Batch Easy & Efficient Batch Computing on Amazon Web Services
 
ALX202_Integrate Alexa voice technology into your product with the Alexa Voic...
ALX202_Integrate Alexa voice technology into your product with the Alexa Voic...ALX202_Integrate Alexa voice technology into your product with the Alexa Voic...
ALX202_Integrate Alexa voice technology into your product with the Alexa Voic...
 
MCL306_Making IoT Smarter with AWS Rekognition.pdf
MCL306_Making IoT Smarter with AWS Rekognition.pdfMCL306_Making IoT Smarter with AWS Rekognition.pdf
MCL306_Making IoT Smarter with AWS Rekognition.pdf
 
DAT321_How Careem Used Amazon ElastiCache for Redis to Accelerate Their Ride ...
DAT321_How Careem Used Amazon ElastiCache for Redis to Accelerate Their Ride ...DAT321_How Careem Used Amazon ElastiCache for Redis to Accelerate Their Ride ...
DAT321_How Careem Used Amazon ElastiCache for Redis to Accelerate Their Ride ...
 
CON213_Hands-on Kubernetes on AWS
CON213_Hands-on Kubernetes on AWSCON213_Hands-on Kubernetes on AWS
CON213_Hands-on Kubernetes on AWS
 
CMP314_Bringing Deep Learning to the Cloud with Amazon EC2
CMP314_Bringing Deep Learning to the Cloud with Amazon EC2CMP314_Bringing Deep Learning to the Cloud with Amazon EC2
CMP314_Bringing Deep Learning to the Cloud with Amazon EC2
 
GPSWKS408-GPS Migrate Your Databases with AWS Database Migration Service and ...
GPSWKS408-GPS Migrate Your Databases with AWS Database Migration Service and ...GPSWKS408-GPS Migrate Your Databases with AWS Database Migration Service and ...
GPSWKS408-GPS Migrate Your Databases with AWS Database Migration Service and ...
 
SID402_An AWS Security Odyssey
SID402_An AWS Security OdysseySID402_An AWS Security Odyssey
SID402_An AWS Security Odyssey
 
How to Assess Your Organization's Readiness to Migrate at Scale to AWS - ENT2...
How to Assess Your Organization's Readiness to Migrate at Scale to AWS - ENT2...How to Assess Your Organization's Readiness to Migrate at Scale to AWS - ENT2...
How to Assess Your Organization's Readiness to Migrate at Scale to AWS - ENT2...
 
GPSBUS220-Refactor and Replatform .NET Apps to Use the Latest Microsoft SQL S...
GPSBUS220-Refactor and Replatform .NET Apps to Use the Latest Microsoft SQL S...GPSBUS220-Refactor and Replatform .NET Apps to Use the Latest Microsoft SQL S...
GPSBUS220-Refactor and Replatform .NET Apps to Use the Latest Microsoft SQL S...
 
STG203_Get Rid of Tape and Modernize Backup with AWS
STG203_Get Rid of Tape and Modernize Backup with AWSSTG203_Get Rid of Tape and Modernize Backup with AWS
STG203_Get Rid of Tape and Modernize Backup with AWS
 
Introducing Amazon EKS
Introducing Amazon EKSIntroducing Amazon EKS
Introducing Amazon EKS
 
GPSTEC319-Build Once Deploy Many Architecting and Building Automated Reusable...
GPSTEC319-Build Once Deploy Many Architecting and Building Automated Reusable...GPSTEC319-Build Once Deploy Many Architecting and Building Automated Reusable...
GPSTEC319-Build Once Deploy Many Architecting and Building Automated Reusable...
 
GPSWKS301_Comprehensive Big Data Architecture Made Easy
GPSWKS301_Comprehensive Big Data Architecture Made EasyGPSWKS301_Comprehensive Big Data Architecture Made Easy
GPSWKS301_Comprehensive Big Data Architecture Made Easy
 
DEV305_Manage Your Applications with AWS Elastic Beanstalk.pdf
DEV305_Manage Your Applications with AWS Elastic Beanstalk.pdfDEV305_Manage Your Applications with AWS Elastic Beanstalk.pdf
DEV305_Manage Your Applications with AWS Elastic Beanstalk.pdf
 
ARC303_Running Lean Architectures How to Optimize for Cost Efficiency
ARC303_Running Lean Architectures How to Optimize for Cost EfficiencyARC303_Running Lean Architectures How to Optimize for Cost Efficiency
ARC303_Running Lean Architectures How to Optimize for Cost Efficiency
 
Scaling Up to Your First 10 Million Users
Scaling Up to Your First 10 Million UsersScaling Up to Your First 10 Million Users
Scaling Up to Your First 10 Million Users
 
DEV326_DevOps Essentials An Introductory Workshop on CICD Practices
DEV326_DevOps Essentials An Introductory Workshop on CICD PracticesDEV326_DevOps Essentials An Introductory Workshop on CICD Practices
DEV326_DevOps Essentials An Introductory Workshop on CICD Practices
 

Similar to SRV314_Building a Serverless Pipeline to Transcode a Two-Hour Video in Minutes

Driving Innovation with Containers - CON203 - re:Invent 2017
Driving Innovation with Containers - CON203 - re:Invent 2017Driving Innovation with Containers - CON203 - re:Invent 2017
Driving Innovation with Containers - CON203 - re:Invent 2017Amazon Web Services
 
High-Throughput Genomics on AWS - LFS309 - re:Invent 2017
High-Throughput Genomics on AWS - LFS309 - re:Invent 2017High-Throughput Genomics on AWS - LFS309 - re:Invent 2017
High-Throughput Genomics on AWS - LFS309 - re:Invent 2017Amazon Web Services
 
LFS309-High-Throughput Genomics on AWS.pdf
LFS309-High-Throughput Genomics on AWS.pdfLFS309-High-Throughput Genomics on AWS.pdf
LFS309-High-Throughput Genomics on AWS.pdfAmazon Web Services
 
CTD405_Building Serverless Video Workflows
CTD405_Building Serverless Video WorkflowsCTD405_Building Serverless Video Workflows
CTD405_Building Serverless Video WorkflowsAmazon Web Services
 
透過最新的 AWS 服務在 2019 年為您的業務轉型 (Level 200)
透過最新的 AWS 服務在 2019 年為您的業務轉型 (Level 200)透過最新的 AWS 服務在 2019 年為您的業務轉型 (Level 200)
透過最新的 AWS 服務在 2019 年為您的業務轉型 (Level 200)Amazon Web Services
 
How Nextdoor Built a Scalable, Serverless Data Pipeline for Billions of Event...
How Nextdoor Built a Scalable, Serverless Data Pipeline for Billions of Event...How Nextdoor Built a Scalable, Serverless Data Pipeline for Billions of Event...
How Nextdoor Built a Scalable, Serverless Data Pipeline for Billions of Event...Amazon Web Services
 
How Netflix Encodes at Scale - CMP309 - re:Invent 2017
How Netflix Encodes at Scale - CMP309 - re:Invent 2017How Netflix Encodes at Scale - CMP309 - re:Invent 2017
How Netflix Encodes at Scale - CMP309 - re:Invent 2017Amazon Web Services
 
Batch Processing with Containers on AWS - CON304 - re:Invent 2017
Batch Processing with Containers on AWS - CON304 - re:Invent 2017Batch Processing with Containers on AWS - CON304 - re:Invent 2017
Batch Processing with Containers on AWS - CON304 - re:Invent 2017Amazon Web Services
 
Genomics on aws-webinar-april2018
Genomics on aws-webinar-april2018Genomics on aws-webinar-april2018
Genomics on aws-webinar-april2018Brendan Bouffler
 
I Want to Analyze and Visualize Website Access Logs, but Why Do I Need Server...
I Want to Analyze and Visualize Website Access Logs, but Why Do I Need Server...I Want to Analyze and Visualize Website Access Logs, but Why Do I Need Server...
I Want to Analyze and Visualize Website Access Logs, but Why Do I Need Server...Amazon Web Services
 
Become a Serverless Black Belt: Optimizing Your Serverless Applications - SRV...
Become a Serverless Black Belt: Optimizing Your Serverless Applications - SRV...Become a Serverless Black Belt: Optimizing Your Serverless Applications - SRV...
Become a Serverless Black Belt: Optimizing Your Serverless Applications - SRV...Amazon Web Services
 
Moving to Amazon ECS – the Not-So-Obvious Benefits - CON356 - re:Invent 2017
Moving to Amazon ECS – the Not-So-Obvious Benefits - CON356 - re:Invent 2017Moving to Amazon ECS – the Not-So-Obvious Benefits - CON356 - re:Invent 2017
Moving to Amazon ECS – the Not-So-Obvious Benefits - CON356 - re:Invent 2017Amazon Web Services
 
DEV325_Application Deployment Techniques for Amazon EC2 Workloads with AWS Co...
DEV325_Application Deployment Techniques for Amazon EC2 Workloads with AWS Co...DEV325_Application Deployment Techniques for Amazon EC2 Workloads with AWS Co...
DEV325_Application Deployment Techniques for Amazon EC2 Workloads with AWS Co...Amazon Web Services
 
CON309_Containerized Machine Learning on AWS
CON309_Containerized Machine Learning on AWSCON309_Containerized Machine Learning on AWS
CON309_Containerized Machine Learning on AWSAmazon Web Services
 
ARC311_Serverless Encoding at Scale with Content Moderation via Deep Learning...
ARC311_Serverless Encoding at Scale with Content Moderation via Deep Learning...ARC311_Serverless Encoding at Scale with Content Moderation via Deep Learning...
ARC311_Serverless Encoding at Scale with Content Moderation via Deep Learning...Amazon Web Services
 
Stack Mastery: Create and Optimize Advanced AWS CloudFormation Templates - DE...
Stack Mastery: Create and Optimize Advanced AWS CloudFormation Templates - DE...Stack Mastery: Create and Optimize Advanced AWS CloudFormation Templates - DE...
Stack Mastery: Create and Optimize Advanced AWS CloudFormation Templates - DE...Amazon Web Services
 

Similar to SRV314_Building a Serverless Pipeline to Transcode a Two-Hour Video in Minutes (20)

Driving Innovation with Containers - CON203 - re:Invent 2017
Driving Innovation with Containers - CON203 - re:Invent 2017Driving Innovation with Containers - CON203 - re:Invent 2017
Driving Innovation with Containers - CON203 - re:Invent 2017
 
High-Throughput Genomics on AWS - LFS309 - re:Invent 2017
High-Throughput Genomics on AWS - LFS309 - re:Invent 2017High-Throughput Genomics on AWS - LFS309 - re:Invent 2017
High-Throughput Genomics on AWS - LFS309 - re:Invent 2017
 
LFS309-High-Throughput Genomics on AWS.pdf
LFS309-High-Throughput Genomics on AWS.pdfLFS309-High-Throughput Genomics on AWS.pdf
LFS309-High-Throughput Genomics on AWS.pdf
 
Serverless Architectures.pdf
Serverless Architectures.pdfServerless Architectures.pdf
Serverless Architectures.pdf
 
CTD405_Building Serverless Video Workflows
CTD405_Building Serverless Video WorkflowsCTD405_Building Serverless Video Workflows
CTD405_Building Serverless Video Workflows
 
透過最新的 AWS 服務在 2019 年為您的業務轉型 (Level 200)
透過最新的 AWS 服務在 2019 年為您的業務轉型 (Level 200)透過最新的 AWS 服務在 2019 年為您的業務轉型 (Level 200)
透過最新的 AWS 服務在 2019 年為您的業務轉型 (Level 200)
 
How Nextdoor Built a Scalable, Serverless Data Pipeline for Billions of Event...
How Nextdoor Built a Scalable, Serverless Data Pipeline for Billions of Event...How Nextdoor Built a Scalable, Serverless Data Pipeline for Billions of Event...
How Nextdoor Built a Scalable, Serverless Data Pipeline for Billions of Event...
 
How Netflix Encodes at Scale - CMP309 - re:Invent 2017
How Netflix Encodes at Scale - CMP309 - re:Invent 2017How Netflix Encodes at Scale - CMP309 - re:Invent 2017
How Netflix Encodes at Scale - CMP309 - re:Invent 2017
 
Batch Processing with Containers on AWS - CON304 - re:Invent 2017
Batch Processing with Containers on AWS - CON304 - re:Invent 2017Batch Processing with Containers on AWS - CON304 - re:Invent 2017
Batch Processing with Containers on AWS - CON304 - re:Invent 2017
 
Genomics on aws-webinar-april2018
Genomics on aws-webinar-april2018Genomics on aws-webinar-april2018
Genomics on aws-webinar-april2018
 
I Want to Analyze and Visualize Website Access Logs, but Why Do I Need Server...
I Want to Analyze and Visualize Website Access Logs, but Why Do I Need Server...I Want to Analyze and Visualize Website Access Logs, but Why Do I Need Server...
I Want to Analyze and Visualize Website Access Logs, but Why Do I Need Server...
 
Become a Serverless Black Belt: Optimizing Your Serverless Applications - SRV...
Become a Serverless Black Belt: Optimizing Your Serverless Applications - SRV...Become a Serverless Black Belt: Optimizing Your Serverless Applications - SRV...
Become a Serverless Black Belt: Optimizing Your Serverless Applications - SRV...
 
Moving to Amazon ECS – the Not-So-Obvious Benefits - CON356 - re:Invent 2017
Moving to Amazon ECS – the Not-So-Obvious Benefits - CON356 - re:Invent 2017Moving to Amazon ECS – the Not-So-Obvious Benefits - CON356 - re:Invent 2017
Moving to Amazon ECS – the Not-So-Obvious Benefits - CON356 - re:Invent 2017
 
STG401_This Is My Architecture
STG401_This Is My ArchitectureSTG401_This Is My Architecture
STG401_This Is My Architecture
 
DEV325_Application Deployment Techniques for Amazon EC2 Workloads with AWS Co...
DEV325_Application Deployment Techniques for Amazon EC2 Workloads with AWS Co...DEV325_Application Deployment Techniques for Amazon EC2 Workloads with AWS Co...
DEV325_Application Deployment Techniques for Amazon EC2 Workloads with AWS Co...
 
CON309_Containerized Machine Learning on AWS
CON309_Containerized Machine Learning on AWSCON309_Containerized Machine Learning on AWS
CON309_Containerized Machine Learning on AWS
 
ARC311_Serverless Encoding at Scale with Content Moderation via Deep Learning...
ARC311_Serverless Encoding at Scale with Content Moderation via Deep Learning...ARC311_Serverless Encoding at Scale with Content Moderation via Deep Learning...
ARC311_Serverless Encoding at Scale with Content Moderation via Deep Learning...
 
GPSTEC325-Enterprise Storage
GPSTEC325-Enterprise StorageGPSTEC325-Enterprise Storage
GPSTEC325-Enterprise Storage
 
Stack Mastery: Create and Optimize Advanced AWS CloudFormation Templates - DE...
Stack Mastery: Create and Optimize Advanced AWS CloudFormation Templates - DE...Stack Mastery: Create and Optimize Advanced AWS CloudFormation Templates - DE...
Stack Mastery: Create and Optimize Advanced AWS CloudFormation Templates - DE...
 
Deploying SAP Solutions on AWS
Deploying SAP Solutions on AWSDeploying SAP Solutions on AWS
Deploying SAP Solutions on AWS
 

More from Amazon Web Services

Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...Amazon Web Services
 
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...Amazon Web Services
 
Esegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS FargateEsegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS FargateAmazon Web Services
 
Costruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWSCostruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWSAmazon Web Services
 
Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot Amazon Web Services
 
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...Amazon Web Services
 
OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...Amazon Web Services
 
Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows WorkloadsMicrosoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows WorkloadsAmazon Web Services
 
Database Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatareDatabase Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatareAmazon Web Services
 
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJSCrea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJSAmazon Web Services
 
API moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e webAPI moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e webAmazon Web Services
 
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatareDatabase Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatareAmazon Web Services
 
Tools for building your MVP on AWS
Tools for building your MVP on AWSTools for building your MVP on AWS
Tools for building your MVP on AWSAmazon Web Services
 
How to Build a Winning Pitch Deck
How to Build a Winning Pitch DeckHow to Build a Winning Pitch Deck
How to Build a Winning Pitch DeckAmazon Web Services
 
Building a web application without servers
Building a web application without serversBuilding a web application without servers
Building a web application without serversAmazon Web Services
 
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...Amazon Web Services
 
Introduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container ServiceIntroduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container ServiceAmazon Web Services
 

More from Amazon Web Services (20)

Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
Come costruire servizi di Forecasting sfruttando algoritmi di ML e deep learn...
 
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
Big Data per le Startup: come creare applicazioni Big Data in modalità Server...
 
Esegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS FargateEsegui pod serverless con Amazon EKS e AWS Fargate
Esegui pod serverless con Amazon EKS e AWS Fargate
 
Costruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWSCostruire Applicazioni Moderne con AWS
Costruire Applicazioni Moderne con AWS
 
Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot Come spendere fino al 90% in meno con i container e le istanze spot
Come spendere fino al 90% in meno con i container e le istanze spot
 
Open banking as a service
Open banking as a serviceOpen banking as a service
Open banking as a service
 
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
Rendi unica l’offerta della tua startup sul mercato con i servizi Machine Lea...
 
OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...OpsWorks Configuration Management: automatizza la gestione e i deployment del...
OpsWorks Configuration Management: automatizza la gestione e i deployment del...
 
Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows WorkloadsMicrosoft Active Directory su AWS per supportare i tuoi Windows Workloads
Microsoft Active Directory su AWS per supportare i tuoi Windows Workloads
 
Computer Vision con AWS
Computer Vision con AWSComputer Vision con AWS
Computer Vision con AWS
 
Database Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatareDatabase Oracle e VMware Cloud on AWS i miti da sfatare
Database Oracle e VMware Cloud on AWS i miti da sfatare
 
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJSCrea la tua prima serverless ledger-based app con QLDB e NodeJS
Crea la tua prima serverless ledger-based app con QLDB e NodeJS
 
API moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e webAPI moderne real-time per applicazioni mobili e web
API moderne real-time per applicazioni mobili e web
 
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatareDatabase Oracle e VMware Cloud™ on AWS: i miti da sfatare
Database Oracle e VMware Cloud™ on AWS: i miti da sfatare
 
Tools for building your MVP on AWS
Tools for building your MVP on AWSTools for building your MVP on AWS
Tools for building your MVP on AWS
 
How to Build a Winning Pitch Deck
How to Build a Winning Pitch DeckHow to Build a Winning Pitch Deck
How to Build a Winning Pitch Deck
 
Building a web application without servers
Building a web application without serversBuilding a web application without servers
Building a web application without servers
 
Fundraising Essentials
Fundraising EssentialsFundraising Essentials
Fundraising Essentials
 
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
AWS_HK_StartupDay_Building Interactive websites while automating for efficien...
 
Introduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container ServiceIntroduzione a Amazon Elastic Container Service
Introduzione a Amazon Elastic Container Service
 

SRV314_Building a Serverless Pipeline to Transcode a Two-Hour Video in Minutes

  • 1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS re:INVENT Building a Serverless Pipeline to Transcode a Two-Hour Video in Minutes L u k e Y o u n g b l o o d — P r i n c i p a l S o l u t i o n s A r c h i t e c t , A W S G r e g F e m e c — S o f t w a r e D e v e l o p e r , R e v v e l S R V 3 1 4 N o v e m b e r 2 7 , 2 0 1 7
  • 2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Agenda Overview of serverless technology Introduction and challenges Building a serverless transcoding pipeline Lessons learned Best practices and recommendations
  • 3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. No servers to provision or manage Scales with usage Never pay for idle Availability and fault tolerance built in Serverless means…
  • 4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Serverless application SERVICES (ANYTHING) Changes in data state Requests to endpoints Changes in resource state EVENT SOURCE FUNCTION Node.js Python Java C#
  • 5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Common AWS Lambda Use Cases Web applications • Static websites • Complex web apps • Packages for Flask and Express Data processing • Real time • MapReduce • Batch Chatbots • Powering chatbot logic Backends • Apps and services • Mobile • IoT </></> Amazon Alexa • Powering voice-enabled apps • Alexa Skills Kit IT automation • Policy engines • Extending AWS services • Infrastructure management
  • 6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Introduction Who am I? • Greg Femec—Dev at Revvel What does Revvel do? • Revvel is a small team that has experience innovating at places like Hulu, Netflix, and Amazon • We are focused on accelerating Verizon’s efforts in building a next-generation television service
  • 7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. The Challenge Lots of video to transcode Unpredictable load Inputs from 30-second 480p clips to barely compressed full-length films Wide range of input formats Lots of output formats to support Progressive HLS (via ts chunks) DASH (via fragmented mp4) DRM
  • 8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Early Solutions Started with SaaS Solutions Didn’t have enough control Not cost effective Built a Transcode Farm on Amazon EC2 Solved some initial problems Created operations challenges
  • 9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Operations Challenges with Amazon EC2 Scaling Wanted to avoid queuing Cold start problem Excess, unused capacity Long running jobs 1-hour minimum (now 1 sec.)
  • 10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Why Serverless? 0 to 100 real quick No wasted capacity Fewer operations challenges Large standby capacity Flexible resource allocation
  • 11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Challenges of Video Transcoding Big files Don’t want to download entire file Hard to stream video input and output Slow jobs HD transcode of movies takes many hours Hard to pause and resume jobs
  • 12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Solution for Efficiently Transcoding Videos Divide and conquer Transcode movie in 5-second chunks Use a MapReduce-type approach Directly use for HLS Merge together for Progressive, DASH Avoid storing full files on disk
  • 13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Services AWS Lambda Instant scale Ability to parallelize workloads Amazon S3 AWS Lambda Amazon S3 Multi-part uploads Range requests FFmpeg Swiss-army knife of transcoding Python Supported by AWS Lambda Python
  • 14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Architecture Diagram for Serverless Transcoding Cache Chunk Lambda Function 1 Cache Cache S3 Source Storage S3 Output Storage (.ts) Cache Merge Function Encrypt Function Encrypt Function OpenSSL Encrypt Function OpenSSL Encrypt Function OpenSSL S3 Output Storage (.mp4) S3 Output Storage (HLS) S3 Output Storage (DASH)Chunk Lambda Function 2 Chunk Lambda Function N
  • 15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Chunk Function Decode a small portion of video Transcode to desired resolution/bitrate Upload .ts chunk to S3
  • 16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Decoding Challenge and Solution Large input files FFmpeg supports http input via range requests but... Too many requests from seeking We built a cache to mirror S3 HTTP daemon running in Lambda function Proxy all read requests from FFmpeg to Amazon S3 Store 5 MB chunks of files in memory Keep read requests open
  • 17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. FFmpeg Challenges Static binaries FFmpeg was unreliable if dynamically linked Larger code size, but still fits within Lambda limits Forked process Ignore stdin Be careful with container re-use
  • 18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Merge Function Read in .ts chunks Combine into MP4 Upload to S3
  • 19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Uses a multi-part S3 upload Delays uploading the first chunk for moov atom Lies to FFmpeg when seeking or reading too far Files too big to fit in /tmp Built a custom FTP daemon FFmpeg supports FTP output, but needs to seek Output File Size Challenge
  • 20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Concat Function Read in .ts chunks Combine / Encrypt Upload to S3
  • 21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Execution Challenges Time challenge Sample AES requires repackaging Longer movies wouldn't finish in a single function Solution Used a single multi-part upload across multiple functions Be careful about back pressure when parallelizing downloads
  • 22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. GIT and Docker Hashed Code .zip — Please have tests (even a simple health check) Lambda Versions/ Aliases Deployment and CI/CD Pipeline
  • 23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. General Lambda Advice Be aware of container reuse Disk Memory Ports Logging If running multiple processes, consider debugging with ps, top, etc. Variable run-time Be prepared to retry functions that time-out, particularly with network- bound functions
  • 24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Results Statistics ~150K hours of transcoded video ~400 million Lambda function invocations Cost savings Similar on-demand EC2 capacity would cost ~$60K a month Average Lambda bill was ~$6K per month for the past 6 months Speed Transcoding a 2 hour movie on EC2 takes ~4-6 hours On Lambda, it takes less than 10 minutes to transcode
  • 25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Final Thoughts Limitations 5 minutes to save the world Must be able to efficiently seek in input video file Higher price per CPU hour Decoding overhead Future plans More output formats Better efficiency for small files
  • 26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Thank you!