Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

SRV314_Building a Serverless Pipeline to Transcode a Two-Hour Video in Minutes

2,175 views

Published on

Learn how Verizon’s Revvel team built a world-class video transcoding pipeline on AWS. This session shows how Revvel migrated from transcoding video on EC2 instances to a serverless pipeline using AWS Lambda and Amazon S3. You will gain insights on how they were able to achieve massive parallelization that gives Revvel the ability to transcode an entire movie or TV series into multiple device formats and bitrates within minutes, while never paying for idle resources. In addition, the Revvel team will cover best practices and lessons learned in creating a custom CI/CD pipeline for Lambda functions that allows them to test code quality during an upgrade to a Lambda function. Revvel is a team within Verizon, chartered with building and innovating video solutions. In November 2016, Verizon acquired Vessel to accelerate their efforts in building a next-generation television service, and Revvel was born.

  • Be the first to comment

SRV314_Building a Serverless Pipeline to Transcode a Two-Hour Video in Minutes

  1. 1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS re:INVENT Building a Serverless Pipeline to Transcode a Two-Hour Video in Minutes L u k e Y o u n g b l o o d — P r i n c i p a l S o l u t i o n s A r c h i t e c t , A W S G r e g F e m e c — S o f t w a r e D e v e l o p e r , R e v v e l S R V 3 1 4 N o v e m b e r 2 7 , 2 0 1 7
  2. 2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Agenda Overview of serverless technology Introduction and challenges Building a serverless transcoding pipeline Lessons learned Best practices and recommendations
  3. 3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. No servers to provision or manage Scales with usage Never pay for idle Availability and fault tolerance built in Serverless means…
  4. 4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Serverless application SERVICES (ANYTHING) Changes in data state Requests to endpoints Changes in resource state EVENT SOURCE FUNCTION Node.js Python Java C#
  5. 5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Common AWS Lambda Use Cases Web applications • Static websites • Complex web apps • Packages for Flask and Express Data processing • Real time • MapReduce • Batch Chatbots • Powering chatbot logic Backends • Apps and services • Mobile • IoT </></> Amazon Alexa • Powering voice-enabled apps • Alexa Skills Kit IT automation • Policy engines • Extending AWS services • Infrastructure management
  6. 6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Introduction Who am I? • Greg Femec—Dev at Revvel What does Revvel do? • Revvel is a small team that has experience innovating at places like Hulu, Netflix, and Amazon • We are focused on accelerating Verizon’s efforts in building a next-generation television service
  7. 7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. The Challenge Lots of video to transcode Unpredictable load Inputs from 30-second 480p clips to barely compressed full-length films Wide range of input formats Lots of output formats to support Progressive HLS (via ts chunks) DASH (via fragmented mp4) DRM
  8. 8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Early Solutions Started with SaaS Solutions Didn’t have enough control Not cost effective Built a Transcode Farm on Amazon EC2 Solved some initial problems Created operations challenges
  9. 9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Operations Challenges with Amazon EC2 Scaling Wanted to avoid queuing Cold start problem Excess, unused capacity Long running jobs 1-hour minimum (now 1 sec.)
  10. 10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Why Serverless? 0 to 100 real quick No wasted capacity Fewer operations challenges Large standby capacity Flexible resource allocation
  11. 11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Challenges of Video Transcoding Big files Don’t want to download entire file Hard to stream video input and output Slow jobs HD transcode of movies takes many hours Hard to pause and resume jobs
  12. 12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Solution for Efficiently Transcoding Videos Divide and conquer Transcode movie in 5-second chunks Use a MapReduce-type approach Directly use for HLS Merge together for Progressive, DASH Avoid storing full files on disk
  13. 13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Services AWS Lambda Instant scale Ability to parallelize workloads Amazon S3 AWS Lambda Amazon S3 Multi-part uploads Range requests FFmpeg Swiss-army knife of transcoding Python Supported by AWS Lambda Python
  14. 14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Architecture Diagram for Serverless Transcoding Cache Chunk Lambda Function 1 Cache Cache S3 Source Storage S3 Output Storage (.ts) Cache Merge Function Encrypt Function Encrypt Function OpenSSL Encrypt Function OpenSSL Encrypt Function OpenSSL S3 Output Storage (.mp4) S3 Output Storage (HLS) S3 Output Storage (DASH)Chunk Lambda Function 2 Chunk Lambda Function N
  15. 15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Chunk Function Decode a small portion of video Transcode to desired resolution/bitrate Upload .ts chunk to S3
  16. 16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Decoding Challenge and Solution Large input files FFmpeg supports http input via range requests but... Too many requests from seeking We built a cache to mirror S3 HTTP daemon running in Lambda function Proxy all read requests from FFmpeg to Amazon S3 Store 5 MB chunks of files in memory Keep read requests open
  17. 17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. FFmpeg Challenges Static binaries FFmpeg was unreliable if dynamically linked Larger code size, but still fits within Lambda limits Forked process Ignore stdin Be careful with container re-use
  18. 18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Merge Function Read in .ts chunks Combine into MP4 Upload to S3
  19. 19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Uses a multi-part S3 upload Delays uploading the first chunk for moov atom Lies to FFmpeg when seeking or reading too far Files too big to fit in /tmp Built a custom FTP daemon FFmpeg supports FTP output, but needs to seek Output File Size Challenge
  20. 20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Concat Function Read in .ts chunks Combine / Encrypt Upload to S3
  21. 21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Execution Challenges Time challenge Sample AES requires repackaging Longer movies wouldn't finish in a single function Solution Used a single multi-part upload across multiple functions Be careful about back pressure when parallelizing downloads
  22. 22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. GIT and Docker Hashed Code .zip — Please have tests (even a simple health check) Lambda Versions/ Aliases Deployment and CI/CD Pipeline
  23. 23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. General Lambda Advice Be aware of container reuse Disk Memory Ports Logging If running multiple processes, consider debugging with ps, top, etc. Variable run-time Be prepared to retry functions that time-out, particularly with network- bound functions
  24. 24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Results Statistics ~150K hours of transcoded video ~400 million Lambda function invocations Cost savings Similar on-demand EC2 capacity would cost ~$60K a month Average Lambda bill was ~$6K per month for the past 6 months Speed Transcoding a 2 hour movie on EC2 takes ~4-6 hours On Lambda, it takes less than 10 minutes to transcode
  25. 25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Final Thoughts Limitations 5 minutes to save the world Must be able to efficiently seek in input video file Higher price per CPU hour Decoding overhead Future plans More output formats Better efficiency for small files
  26. 26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Thank you!

×