Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Hollywood's Cloud-Based Content Lakes: Modernized Media Archives (MAE203) - AWS re:Invent 2018

4,421 views

Published on

Content lake architecture can evolve the media workflow by providing efficiency from content security all the way to value-added services, such as machine learning and content monetization. In this session, technical leaders from 21st Century Fox, Warner Bros., and Astro Malaysia discuss the migration of their petabyte-scale video libraries (production and distribution archives) to the cloud in order to increase the customer reach and value of their media archives. Discover some of the lessons learned, the TCO analysis around various different storage tiers, the challenges and best practices from 10s of petabytes ingest, storage, and value-added compute at scale.

  • Be the first to comment

  • Be the first to like this

Hollywood's Cloud-Based Content Lakes: Modernized Media Archives (MAE203) - AWS re:Invent 2018

  1. 1. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Hollywood’s Cloud-Based Content Lakes: Modernized Media Archives M A E 2 0 3 Mike Davis Storage Business Development AWS David Sugg VP Content Arch & Engineering Warner Bros. Entertainment Aik-Chong Phuah CTO Astro Malaysia
  2. 2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Focusing on the consumer requires agile infrastructure New business opportunities are rapidly emerging in the media market The ability to leverage your growing asset collection is handicapped by insufficient metadata and federated search Stagnating tape infrastructure is not keeping pace with business agility
  3. 3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. This creates a widening Value Gap, where you don’t have the agility to respond to new opportunities, and IT costs continue to rise unpredictably.
  4. 4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS allows studios, broadcasters, and sports leagues to close the Value Gap by building automated hybrid workflows that improve agility, reduce cycle times, and reduce costs.
  5. 5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Global media firms with Asset Management workloads on AWS Digital Asset Management & Supply Chain Content Creation & Post Production Distribution (OTT, Broadcast, Publish) Machine Learning & Analytics
  6. 6. Offline tape migration Egress optimization Proprietary conversion Metadata export Asset QC & registry Index/ Analyze Proxy generation Amazon Rekognition Amazon Transcribe Amazon Comprehend DAM/CMS modernization Metadata import Well architected Security, access, DR Lifecycle and cost optimization Export Normalization & Augmentation Import & Operationalize Records Access Curation Workflow Import/Quarantine Discovery Migration requires methodical planning
  7. 7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Aik-Chong Phuah CTO Astro Malaysia
  8. 8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Knowing & serving our customer better 920k 33k hours Connected boxes On Demand Content 2M Downloads Astro Go 20M Monthly UV Digital IPs 76% Household 5.6mil homes 3.5hrs viewing per day 16M Weekly Listeners Radio 24M H1 2018 On Demand Content downloads 1.5M Registered Customers Home shopping eCommerce
  9. 9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Production Storage Site 1 Tape Archive 2 Site 2 MAM Inter-site Fiber Mirroring (Secondary Site with similar setup as Site 1) Media Asset Management (MAM) Tape Archive 1 Production Storage Media Ingest & Quality Check Edit & Compliance High level architecture – CurrentIncomingMediaSource AWS Direct Connect, Singapore Back Up Link to Singapore (1Gbps) Amazon S3 Amazon EC2 AWS Elemental Amazon CloudFront AWS Services (Singapore region) Primary Link (1Gbps)
  10. 10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Current architecture • Costly to maintain and to continue to expand • Hard to keep up to date with new LTO formats • Infra was initially designed for SD before HD and UHD • Archives fully utilized (20PB unique content) • Inability to meet content growth rate of 60K hours (2PB) per year • Manual tape handling risk due to offline data tape storage UnscalableHigh InvestmentLegacy System • Aging LTO library (since 2006) • Frequent issues last 12 months: • 8x tape snaps • 12x drive failures • 6x robotic breakdowns • Inability to defragment tape due to high usage of drives • Unable to commit to content publishing time • Unpredictable turnaround time of content retrieval - 15mins to 24hrs retrieval SLA breach Dual Site Ops • Conventional broadcast workflow • Driven by MAM for media processing, playout, & streaming • 1Gbps inter-site fiber link Media Archive • Archived & mirrored between both sites managed by MAM • SL8500 LTO library each per site • Interdependent on-premise servers, online stores, & databases • 15% content not mirrored due to legacy processesChallenges
  11. 11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. New architecture with AWS • Both LTO archives will be migrated to centralized Amazon Glacier Vault and retired • Scalable and robust infrastructure, predictable cost model to support our growth • Reduced overall total cost of ownership • Moved from capex-intensive approach to “pay-per-use” Cost Optimization Scalability Efficiency Experience • Predictable retrieval: standard, bulk & expedite • Improved viewer experience, and time to market Centralized archive and database • Unified archive for all content enabling both linear broadcast & VOD (DTH and OTT) • Single database and MAM Fully redundant DX • 20Gbps fully redundant Direct Connect to AWS POP in Kuala Lumpur relieving bandwidth over inter-site fiber link • Redundant DC to AWS Singapore • Simplified architecture, improved robustness and SLA • Active productions can be buffered to S3 content lake for fast retrieval while pending for archiving in Glacier • Other processes taking place in parallel
  12. 12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. High level architecture – New AWS DX, Singapore Back Up Link to Singapore (1Gbps) Production Storage Site 1 AWS DX POP Kuala Lumpur Site 2 Direct Connect ~20Gbps MAM Inter-site Fiber Mirroring (Secondary Site with similar setup as Site 1) Media Asset Management (MAM) Production Storage Media Ingest & Quality Check Tape Archive 2Tape Archive 1 Edit & Compliance IncomingMediaSource 10Gbps (temp) 10Gbps (temp) S3EC2 ElementalCloudFront AWS Services (Singapore region) Amazon Glacier Back Up Link to Singapore (1Gbps) Direct Connect ~20Gbps Migration Storage Migration Storage
  13. 13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. TCO analysis Direct Connect Costs Migration Costs Parallel run Costs Workflow Development AWS Base Costs 20% savings over a 5-year period moving from on-premise tape archive to Amazon Glacier
  14. 14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. In-the-Works Cloud Automation Cloud Playout Content Censorship Machine Translation Cloud Orchestration Cloud Editing Speech Transcription Metadata Enrichment Next Up
  15. 15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. David Sugg VP Content Arch & Engineering Warner Bros. Entertainment
  16. 16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Warner Bros. is the industry leader across key businesses #1 OR #2 WW BOX OFFICE 9 OF LAST 10 YEARS $4B+ WW BOX OFFICE 7 OF THE LAST 8 YEARS THEATRICAL #1 TV STUDIO FOR LAST 9 SEASONS ONE OF THE LARGEST LICENSORS OF US PROGRAMMING TELEVISION ONLY STUDIO WITH SCALED GAMES PUBLISHING OVER $1 BILLION IN 2017 REVENUE GAMES
  17. 17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. With some of the world’s greatest franchises Classic Animation Horror
  18. 18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. The 100-year challenge Studios have a different perspective on the preservation of intellectual property. We strive to preserve our content for hundreds of years. • What does it take? • Where’s the “Superman Crystal”? © 2018, Warner Bros. Entertainment, Inc. or its affiliates. All rights reserved.
  19. 19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. The 100-year challenge: Threats The Digital Archive Threat Matrix
  20. 20. “Fixity, in the preservation sense, means the assurance that a digital file has remained unchanged, i.e. fixed.” Bailey, 2014 © 2018, Warner Bros. Entertainment, Inc. or its affiliates. All rights reserved.
  21. 21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. • Fixity is often managed via checksums • Can be applied at ingest if the fixity is known for the files, throughout the lifecycle via regular fixity checks and during any event such as a file move or copy. Bit rot, media failure, and the fixity problem © 2018, Warner Bros. Entertainment, Inc. or its affiliates. All rights reserved.
  22. 22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Bit rot, media failure, and the fixity problem • Every bit is important • How do we know bits are safe? • Current standards require regular checks • No user accessible processes to expose the fixitiy status of an object © 2018, Warner Bros. Entertainment, Inc. or its affiliates. All rights reserved.
  23. 23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. How do we manage this today? T10K and LTO8 tape libraries Redundant, geographically dispersed sites Replication managed at storage level
  24. 24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. How do we manage this today? Challenges with using the cloud:Challenges with current solution: • Elasticity • Of compute • Of storage • Capital costs • Time to value • Volume of data • Perceived costs • Perceived security issues
  25. 25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. How do we manage this today? Internet Colo Location 1 Vendor 1 System 2 Vendor/System n AccessControl File/MetadataTransfer High Speed Working Storage T10k/LTO8 Tape Libraries DAM/Orchestration Colo Location 2 AccessControl High Speed Working Storage T10k/LTO8 Tape Libraries DAM/Orchestration High Speed Private Interconnect File/MetadataTransfer
  26. 26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. What might the future look like? Internet Cloud Provider Vendor 1 System 2 Vendor/System N AccessControl File/MetadataTransfer DAM/Orchestration Colo Location 1 AccessControl High Speed Working Storage T10k/LTO8 Tape Libraries DAM/Orchestration File/MetadataTransfer Working Storage GlacierEBS EFS / EC2Lambda RDS EC2Lambda RDS Direct Connect WBContentServicesLayer IngestAPIOrderAPIOutputAPIDataAPI
  27. 27. Thank you! © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Mike Davis, AWS, mikdav@amazon.com Aik-Chong Phuah, Astro Malaysia, pacphuah@astro.com.my David Sugg, Warner Bros., David.sugg@warnerbros.com
  28. 28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

×