AWS re:Invent 2016: Turner's cloud native media supply chain for TNT, TBS, Adult Swim, Cartoon Network, CNN (MAE302)


Published on

As Turner continues to make the transition from a traditional broadcast organization to a consumer-centric, data-driven media company, we are being challenged to re-think our approach to content supply. There is a need to achieve new levels of agility, flexibility and scalability to meet the rapidly evolving demands of our top media brands - including TBS, TNT, Cartoon Network, Adult Swim and CNN. To that end, we are transitioning the infrastructure that acquires, processes and distributes media for consumer-facing systems to the cloud. At the core of this environment is our Supply Chain Management application. The SCM app provides business and technical process management via an HTML based UI framework, State Machine, Rules Engine, Cost Model, Forms Service. We took advantage of several AWS specific services, including Lambda, S3, Dynamo DB, SNS, Elastic, Cloud Formation and Code Commit. The entire system is instance-less with all application code running in either the browser or within Lambda's. To ease development and debugging we created a method to run all JS libraries in the browser, switching to Lambda when we deploy with Code Commit. Cloud media processing infrastructure is BEING created on demand via an integration with SDVI. The SDVI and SCM apps exchange events and data via SNS and S3.

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

AWS re:Invent 2016: Turner's cloud native media supply chain for TNT, TBS, Adult Swim, Cartoon Network, CNN (MAE302)

  1. 1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Michael Koetter | VP Digital Media System Turner Usman Shakeel | WW Tech Lead (M&E) Amazon Web Services 11/28/2016 MAE302 Turner’s Cloud Native Content Supply Chain for TNT, TBS, Adult Swim, Cartoon Network, CNN
  2. 2. Playout & Distribution Visual Effects & Editing Analytics DAM & Archive Digital Supply Chain Publishing OTT Acquisition AWS Media Segments
  3. 3. Key Evolutions in our Industry The Newcomers (Netflix, Amazon) effect Direct B2C (OTT, Digital) Know thy consumer and create the content they dig Cost Crunch on production/post-production Content has gravity and getting heavier Annual Expenditure on Programming ($B) “Millennials spend nearly 50% of their time watching movies and TV shows on devices other than a TV, primarily laptops/desktops, followed by smartphones and tablets” Source: Deloitte Digital Democracy Survey 2016 Millions Netflix Subscribers
  4. 4. The New Normal Media – ALL IN – to the Cloud
  5. 5. Where do I begin? Media – ALL IN – to the Cloud De-Coupling Micro-services Buzzword? Specific scope Interoperable services Allow rapid innovation
  6. 6. An inside-out view Playout & Distribution Production/Post Production B2B/Cinematic Delivery DAM & Archive Digital Supply Chain Publishing OTT Content Acquisition Ingestion Manual QC Auto-QC Metadata Extraction Transcoding Delivery Packaging Clips/Edits Complex workflow/State management Bursty sub-workflow tasks (transcoding/auto-QC, Packaging) Elastic Licensing Content movement between workflow tasks (a common repository) Different delivery formats for each affiliate and downstream Manual tasks – (content movement or interactivity) CMS/DAM Normalization
  7. 7. Normalization All-In Cloud-Enabled Domino Effect Playout & Distribution Live TV Production/Post Production Editorial Dailies Post Social Media Marketing/Promo B2B/Cinematic Delivery DPP, IMF, NABA DAM & Archive Active Archive Golden Copy Preservation Digital Supply Chain Publishing OTT Live-To-VOD AVOD, SVOD, TVOD Catchup TV Cloud PVR Content Acquisition Live File Ingestion Manual QC Auto-QC Metadata Extraction Transcoding Delivery Packaging Clips/Edits CMS/DAM
  8. 8. ETC DI Workflow for Major Studios An Industry View of Media Supply Chain  Complex workflow  State management  Bursty workflow tasks (transcoding/auto- QC, Packaging)  Elastic Licensing  Content movement  Delivery formats for each affiliate and downstream  Manual tasks (content movement or interactivity)
  9. 9. Media Supply Chain (simplified) Quality Check Create, edit, & collaborate in the cloud Distribute content to playout/OTT solution Import mastered content, along with metadata into S3
  10. 10. Macro Level Ideas/Highlights • Amazon EC2 New Instances • Cloud Storage • Amazon S3, Amazon Glacier (new retrieval tiers) • Amazon EFS • Amazon EBS new Volume types and performance • NLE in the Cloud (Remote Application Streaming) • Graphics Amazon WorkSpaces • Microservices/Serverless • AWS Lambda • Amazon API Gateway
  11. 11. Media Supply Chain at Turner
  12. 12. Michael Koetter VP Digital Media Systems @ Turner Stuff for which I’m responsible: • Content Creation Systems • Content Supply Chain Systems • Technology Infrastructure
  13. 13. Roll Video…
  14. 14. OVERALL OBJECTIVE. We are leading the media industry’s transformation through our approach to technology, data science and innovation. 14
  15. 15. SUPPLY CHAIN MISSION Connecting CONTENT PRODUCERS with content monetization PLATFORMS to drive Turner’s business across all brands, networks and products.
  16. 16. ETC DI Workflow for Major Studios An Industry View
  17. 17. D2CESTSVOD ` ` CONNECTED DEVICES PAYTVBROADBAND Linear Network Origination Downstream Processing Content Suppliers SchedulingRightsTitles Ordering Turner Business Systems Cloud Media Processing Turner’s Content Supply Chain
  18. 18. Content Receiving 26,000 Items Processed (Jan – Nov 2016)
  19. 19. Direct to Consumer ~ 1400 movie initial load
  20. 20. SVOD Initial Load (deal dependent)
  21. 21. Promo Production ~ 1000 items per program per agency
  22. 22. Flexibility Analytics Elasticity SUPPLY CHAIN REQUIREMENTS Emerging challenges are putting pressure on our traditional processing model.
  23. 23. Flexible Process Engine Media Micro-Services
  24. 24. Supply Chain Components Upload Auto QC Input Processing Master Version Output Processing Deliver Cloud Library Work Order Management (Process Engine) Content tracking and flow control. Integration API for ODT & Playout Automation. Final content delivery to distribution systems Output sequence conforming, trigger insertion and transcode. Creation of golden library master, preservation in library and creation of versions for specific platform needs Normalization of incoming distribution masters Automated quality check of incoming distribution masters Content submission to CSC Ingest of material into library’s normalized media data model Selection of material from master asset and creation of new and unique sequences.
  25. 25. 25 Order Core FlexibleProcessEngine • Fully abstracted • Instance-less Order Types • Rapid Development • Simplified Environment • Helper Classes
  26. 26. Order Types 26 Order Core State Machine Resource Allocation Rules Engine Order Scaffold Generator HTML Template Document DB Form Data UI Shared Widgets FlexibleProcessEngine
  28. 28. to implementSimple Agile Available Scalable Zero effort to administer on demand without ”clustering” coding practices Instance-less
  29. 29. FlexibleProcessEngine
  30. 30. FlexibleProcessEngine
  31. 31. FlexibleProcessEngine
  32. 32. AWS S3 Rules Engine incl.Lambda Cron State Management CSC Work Order Architecture Version 1.3 Chris Hinton, 8th Nov 2016 SMTP AWS DynamoDB Client / Chrome Quark / Topbar Work Orders App List / Search / Filter Work Order Item CRUD & Business Logic JS Lambda AWSAPIGateway S3HTTPS Conversations forms & utils JS Other AWS Document Management document templates Audit Log KPI & Cost Turner Quark & Content Apps Client JS Turner Network Public Internet SNS SDVI External Orchestrator PSP FlexibleProcessEngine
  33. 33. Media Micro-Services Upload Auto QC Input Processing Master Version Output Processing Deliver Cloud Library Work Order Management
  34. 34. Dynamically created infrastructure Usage based licensing Media-aware analytics and optimization Fine grained media processing as a micro-service Media Micro-Services
  35. 35. Media Micro-Services
  36. 36. Content Receiving Workflow Input files in S3 S3 Multi-Part Client S3 Transfer Acceleration EC2 instances S3 Notification Trigger a Lambda Function to update portal and job queue File Transmission complete SDVI Amazon SNS Integration Supplier 1 Supplier 3 S3 Life Cycle Policies for purging temp files S3 as input/output to different tasks Supplier 2 Order fulfillment State Web Portal Upload Content File to S3 Save Order Amazon Dynamo DB Order Portal REST Client API Process Rules, Update State
  37. 37. Turner’s Media Supply Chain in the AWS Cloud AWS cloud – US East 1 Turner Campus - Atlanta Workorders Content Amazon API Gateway AWS Lambda Turner VPC SDVI VPC Proxy Amazon Elasticsearch AWS SNS Content Distributors Turner Employee Legacy Infrastructure DynamoDB
  38. 38. Macro Implications
  39. 39. Cloud Content Security Strategy (Considerations) 39 • Traditional on prem content security has been perimeter based • Perimeters can be breached • Risk domains are very large • Logging is not always robust • Cloud security affordances enabled Turner’s Information Security Office to take a favorable position for change MacroImplications
  40. 40. Cloud Content Security Strategy 40 • Separate AWS account for Content storage with limited employee access (and no compute) • S3 buckets with Bucket Policies limiting Get and Puts based on Amazon VPC Endpoints (Turner and SDVI) • Time-limited tokens for SDVI & other vendors to access pre-air & master quality content • Bucket Access logging into AWS CloudTrail S3 Bucket • On-Prem Low-Res Proxy Streaming system limits access to Turner's network for better cost & performance. MacroImplications
  41. 41. Change Control Continuous Delivery 41 • Media industry is traditionally averse to change. • ”Set it and forget it” is held as an ideal • Changes occur infrequently & at the most low impact times… yet they are in reality always changing. • Systems have been architected around these principles • If the business wants agility and flexibility, we need to change our delivery model MacroImplications
  42. 42. Change Control Continuous Delivery 42 MacroImplications Automated Daily Release Process Faster delivery of value to the business Release in smaller increments, control risk Reduced “Bit Rot” & Improved Developer Productivity
  43. 43. Change Control Continuous Delivery 43 MacroImplications On Prem (UI) Stateful Infrastructure (EC2, S3, AWS IAM, Dynamo DB) Stateless Infrastructure (Lambda, API gateway) What? How? When?
  44. 44. Challenges Overcome
  45. 45. ChallengesOvercome Javascript development and realtime debugging Out of Order Messages Tagging Lambda for dynamic rule-sets 15 second message read-filter allows all arriving messages to buffer. Had to create an EC2 Instance to proxy requests between lambda & on-premises Automated run-time selection within code
  46. 46. • More Workflows • Cloud Master Library • Cloud Editing & Versioning • Cloud Based Channel Origination Playout & Distribution Visual Effects & Editing Analytics DAM & Archive Digital Supply Chain Publishing OTT Acquisition What’sNext
  48. 48. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Chris Hinton VP, Product Development Turner BEST Chris Farris Engineering Manager, Turner BEST Brigitte Mabry Mgr, Content Logistics Turner NATO Thanks to: Questions?
  49. 49. Remember to complete your evaluations!
  50. 50. Related Sessions Industry Pre-Day: Monday 11/28 • MAE303 - Discovery Channel's Broadcast Workflows and Channel Origination on AWS • MAE304 - High Performance Cinematic Production in the Cloud • MAE301 - Accelerating the Transition to Broadcast and OTT Infrastructure in the Cloud Regular Conference: Tuesday – Thursday 11/29 – 12/1 • ARC406 - Encoding Artifacts to Emmy Awards: Taking on Terabyte-Scale, 1-Gbps, 4K Video Processing in the Cloud • CMP312 - Powering the Next Generation of Virtual Reality with Verizon • CTD305 - Media Delivery from the Cloud: Integrated AWS Solutions for Premium Over the Top (OTT) Content • STG302 - Deep Dive on Amazon Glacier