Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

AWS for Media: Content in the Cloud, Miles Ward (Amazon Web Services) and Bhavik Vyas (Aspera)


Published on

Published in: Technology
  • Be the first to comment

AWS for Media: Content in the Cloud, Miles Ward (Amazon Web Services) and Bhavik Vyas (Aspera)

  1. 1. AWS for Media: Content in the Cloud Miles Ward Solutions Architect Bhavik Vyas Director of Cloud Solutions
  2. 2. Today’s Agenda:• Examine common needs and challenges we’ve seen from Media and Entertainment customers.• Provide examples of how customers have met these needs using AWS.• Dig in on patterns for Storage, Processing, and Distribution.• Learn how to get started.
  3. 3. Media Industry ScenariosMedia Storage – Library storage for processing – ArchivingMedia Processing – Preparing source/mezzanine files for further processing – Encode/Transcode – Encryption, watermarking – Applying individualization and monetization mechanisms (advertising, PPV, subscription, purchase, etc.)Media Delivery – Audio and video streaming of Live and VOD content • Managed distribution to set top boxes • Over the top (OTT) distribution • Internet distribution – Static image distribution (web page chrome and images) – Rich media web applications • Flash/Silverlight/etc. applications can be cached at the edge for scalability and faster download/startup
  4. 4. Media Industry Trends and Needs• Media industry is at an inflection point• Infrastructure requirements growing exponentially• As the media industry has “gone digital”, content distribution requirements have also evolved – Scalable and secure media storage, processing and distribution – Anytime, anywhere, any device consumption – Personalized content and experiences – Expectation of low latency, global distribution – Support multiple business models (purchase, PPV, subscription)• Media has unusual performance, security and reliability characteristics compared to other industries
  5. 5. Today’s Ubiquitous Media is Complex and Challenging Technical Requirements High Performance Computing Server Farms Content Distribution Networks DAM, DRM, Encoding, Editing, and Video Player Software Large Number of File Formats Massive Storage Networks
  6. 6. On-Premise Infrastructure is Costly & Complex Large Capital Expenditures Underutilized IT Assets Patching Software Out of Datacenter SpaceScaling down as needed Slow IT Deployments Contract negotiation Scaling up quickly Prices too high for IT products Managing physical growth“ IT spends 80% of its time and resources keeping the lights on ”
  7. 7. The Cloud Provides a Better way Instant Elasticity Rapid Time to Value Focus on InnovationOn-Demand not Infrastructure Pricing No CapEx
  8. 8. How the Cloud Can Reduce Digital Lifecycle Complexity Massive Scale Rapid File Transfer Lower IT Costs High Reliability Accredited Security
  9. 9. Massive Scale Makes it Easier to Hit Deadlines Problem: Timeline overruns to support PS3 launch Solution: AWS’s Scale, 1200 virtual machines on-demand Benefits:17,000 titles (80TB of data) transcoded in days, able to support launch
  10. 10. Massive Scale Makes it Easier to Focus on YourCore Business Problem: Massive global growth in online gaming Solution: AWS’s Global Infrastructure Benefits: Scaled to handle 4 million players in 8 weeks
  11. 11. StorageS3 – Storage for Source / Mezzanine files Exceptional Durability: 99.999999999% Secure EconomicalEBS – Storage for Media during Processing POSIX Block Device Dynamic Persistent for non-contiguous processingDirect Connect – Leverage your existing storage systems viaa low latency Nx10Gbps link to AWS
  12. 12. S3 Use Cases and UsersMedia ServingMedia SharingFile Storage / BackupStatic ContentBig Data
  13. 13. Storage: Scale of Amazon S3 Total Number of Objects Stored in Amazon S3 Peak Requests: 762 Billion 500,000+ per second 262 Billion 102 Billion 14 Billion 40 Billion 2.9 Billion Q4 2006 Q4 2007 Q4 2008 Q4 2009 Q4 2010 Q4 2011
  14. 14. Media Ingest and Storage Architecture 14
  15. 15. Moving Files to and From Amazon S3 is Fast InternetMulti-part Upload Up to 700 Mb/s direct 1Gb and 10 Gb to Amazon S3 Connections
  16. 16. Moving Files Faster Also Makes it Easier toHit Deadlines 9 Minutes: Other 7 Seconds: Amazon than Amazon S3 S3
  17. 17. ProcessingEC2– Servers in the Cloud by the Hour High-Performance, Scalable, Secure Origin servers, custom application servers,monetization and DRM systemsEMR – A coordination framework for distributed computing Minimizes Administration Automates distribution Simplifies workflowsCG1 - CUDA– GPGPU-enabled instances for super high FPperformance, transcoding.
  18. 18. Batch Processing with EC2
  19. 19. Easily Scale Up and Down Internet Video App on Amazon EC2 From 50 to 5,000 servers in 3 days 5,000 Scaled to peak of 5,000 instances Number of EC2 Instances in 3 days Launch of Facebook application 0 Monday Tuesday Wednesday Thursday Friday Saturday SundayThe Animoto Blog
  20. 20. AWS Reduces IT Costs Saved 70% OPPORTUNITY COST WASTAGE Traditional Hardware Saved 50% Actual Demand Automated Virtualization Reduced costs by over 50%
  21. 21. Why Vimeo Chose AWS“Our users dont care whether itsa peak time or not, and weprefer not to pay for transcodingmachines the five days a weekwe arent using them.This workload is perfect for EC2
  22. 22. CG1 – GPGPU is here = 10x faster, 2x higher res, lower cost
  23. 23. DistributionCloudfront – Next-Gen Content Delivery Network Performance-oriented Multi-format Secure EconomicalRoute 53 – DNS API controlled, allows programmatic shifts Weighted Records Alias records for ELBEC2– Origin servers, custom application servers, monetizationand DRM systems
  24. 24. Introduction to CloudFrontCloudFront in a nutshell. Our Content Delivery Network Delivers: • Low latency. Improves media load times. • High bandwidth. Enables high bit rate HD video and other media applications. • Redundant. Eliminates single points of failure. • Scalable. Ensure great experience as number of end users grows. • Global. Worldwide network provides great experience regardless of geography. • Cost-effective. Pay as you go model provides flexibility for your business. 25
  25. 25. Introduction to CloudFrontKey features for CloudFront.• RTMP (Flash) and HTTP delivery• Live and VOD• Adaptive Bitrate Streaming• HTTP/HTTPS File Delivery• Private Content• Programmatic Invalidation• Industry-compliant, detailed Access Logs Amazon CloudFront• AWS Management Console• Full control via APIs 26
  26. 26. Introduction to CloudFront Delivery of Non-Cached Video Download Request S3 bucket or routed to most custom originwith your stored Object requested from origin CloudFront Edge optimal edge location 0 End-user requests video video content 2 Locations 1 Media stored in Data transfer of3 video to CloudFront cache 4 5 End-user enjoys video Data transfer of edge location to cached video cache download to end user 27
  27. 27. Introduction to CloudFront Delivery of Non-Cached Video Download Request S3 bucket or routed to most custom originwith your stored video content CloudFront Edge optimal edge location 0 End-user requests video Locations 1 Skipping these steps means lower latency for your users! Media read from cache 2 3 End-user enjoys video Data transfer of cached video download to end user 28
  28. 28. Introduction to CloudFrontAWS Content and Media Serving Architecture 29
  29. 29. AWS’ Reliable Infrastructure Makes it Easier to Focus on Other Tasks“As with all the AWS services we leverage, using AmazonCloudFront is so simple and reliable that the team doesn’t have tothink about it. It all just works, freeing us to focus on building coolapplications.” >1Pb/mo through Cloudfront
  30. 30. Introduction to CloudFrontAWS and CloudFront’s global presence.AWS Regions US East (Northern Virginia) US West (Northern California) US West (Northern Oregon) Europe (Dublin) Asia Pacific (Singapore) Asia Pacific (Toyko)AWS CloudFront LocationsUnited States: Europe: Asia: South America:Ashburn, VA New York, NY (2) Amsterdam Hong Kong Sao PauloDallas, TX Palo Alto, CA Dublin TokyoJacksonville, FL San Jose, CA Frankfurt SingaporeLos Angeles, CA (2) Seattle, WA London Osaka!Miami, FL South Bend, IN ParisNewark, NJ St. Louis, MO Stockholm Milan! 31
  31. 31. Introduction to CloudFront Rapid pace of innovation (and price cuts). » San Jose, CA Edge Location » South bend » Price Drop Indiana Edge » Free Inbound » Price Drop Location Data Transfer » HTTPS Support » Second NY Edge » Stockholm Edge » Lower 1-Hour TTLs Location » Access Logging Capability Location » Access Logs for Streaming » Large Object » CloudFront Management Console » Live Streaming » NYC Edge Location Support Support » CloudFront tops » Custom Origins 20K customers » Private Content » Streaming Video on Demand » Service Level Agreement» CloudFront Public Beta » Jacksonville Edge Location » New Lower Pricing Tiers » Singapore Edge Location » Paris Edge Location » Private Streaming » (IAM) Identity & Access Management » Sao Paulo Edge Location » Enhanced CloudFront Logs » Second LA Edge Location » Default Root Object » Invalidation 32
  32. 32. Needs and ExamplesWhat do M&E customers expect from a CDN:• Great performance to a global audience.• Reliable delivery to wide range of clients.• Scalability for unpredictable spikes in demand• Control for secure content delivery• Support for HD high-bandwidth streaming• Reporting and analytics to track & analyze viewing patterns• Easy to configure and manage solutions• Cost-effective solutions 33
  33. 33. Needs and ExamplesNeed: Great performance to a global audience.• Amazon is a metrics Gomez Large Object Test driven company. Japan 12.000• We focus on metrics 10.000 9.673 that capture the end Download Time (Seconds) user experience: the 8.000 “last mile,” not internet 6.000 “backbone” data 3.688 4.000 3.441 centers. 3.126 2.000• Your customers do not live in data centers. 0.000 Last Mile CloudFront Comparison A Comparison B Comparison C Last mile data based on 7,907 observations taken between 17-SEP-2011 and 1-Oct-2011 *The Gomez tests were designed and conducted by Amazon using the Compuware Corporation performance network. The test results have not been reviewed, approved or endorsed by Compuware Corporation 34
  34. 34. Needs and ExamplesNeed: Reliable delivery to wide range of clients.• Multiple delivery protocols for different platforms and devices • Adobe RTMP • HTTP Streaming for iOS • Microsoft Silverlight• Options for live and on-demand video • Full control over origin for live streaming ?• Reliability backed by CloudFront Service Level Agreement 36
  35. 35. Needs and ExamplesExample: Video banner ad customer. 80Volume of Data Delivered (Gbps) 70 Peak usage over 60 Gbps 60 50 40 30 20 10 0 8:00 AM 9:00 AM 10:00 AM 11:00 AM 12:00 PM 1:00 PM 2:00 PM 3:00 PM 4:00 PM 5:00 PM 6:00 PM 7:00 PM 8:00 PM 9:00 PM Confidential – Shared Under NDA 37
  36. 36. Needs and ExamplesNeed: Control for secure content delivery.• Private Content Feature authenticate users with signed URLS • Uses policy-driven access controls for control and flexibility • Restrict on resource or path, time, source IP • Signatures generated using asymmetric encryption• SSL delivery and RTMPE streaming encrypt bytes on the wire• Origin Access Identities secure your content in Amazon S3• Identify and Access Management (IAM) to control who can configure your CloudFront distributions
  37. 37. Need: Easy to configure and manage solutions.• Self service signup and configuration – anytime, from anywhere• AWS Management Console to create and manage AWS infrastructure like S3, EC2, Cloudfront• Programmatic APIs for integration into your own systems and workflow 39
  38. 38. Large Partner Ecosystem Makes Managing Easier
  40. 40. HOW BIG IS ―BIG DATA‖? (M&E EXAMPLE) STANDARD FORMATS CD DVD Blu-ray .75 GB 4.7 GB 25 GB COMPRESSED VIDEO FORMAT SD 720p HD 1080p HDStream rate (Mbps) 1.5 5 8 1 hour of content .7 GB 2.3 GB 3.6 GB UNCOMPRESSED VIDEO FORMAT SD 720p HD 1080p HDStream rate (Mbps) 125 553 1290 1 hour of content 56 GB 249 GB 581 GB UNCOMPRESSED VIDEO FORMAT Digital Cinema 2K Digital Cinema 4K 3D Cinema StreamStream rate (Mbps) 1900 3800 5730 1 hour of content 855 GB 1710 GB 2579 GB A single digital cinema production can be 800K–1M 2K/4K frames
  42. 42. SECOND MAJOR BOTTLENECKS: LOCAL HTTP I/O1st Bottleneck - WAN 2nd Bottleneck — Data Center
  43. 43. BIG-DATA and WAN TRANSFER WITH TCPTCP WAS DESIGNED IN THE EARLY 80’S • When data was small & bandwidth was limited • Fantastic for reliable data delivery • Not fast enough for big-dataTCP IS THE ENGINE THAT DRIVES • FTP, HTTP & HTTPS • RSYNC, SCP • CIFS & NFSTCP DOES NOT LIKE NETWORK LATENCY/ RTT • Geographic distance increases latency • Network congestion increases latencyTCP DOES NOT LIKE PACKET LOSS • Loss is caused by congestion • Different network capacity • Wireless and satellite communications
  44. 44. ASPERA FASP™: HIGH-PERFORMANCE BIG-DATA TRANSPORTMAXIMUM LINE-RATE WAN TRANSFER SPEED • Transfer performance scales with bandwidth independent of transfer distance and resilient to packet loss • Optimal end-to-end throughput efficiencyCONGESTION AVOIDANCE AND POLICY CONTROL • Automatic, full utilization of available bandwidth • On-the-fly prioritization and bandwidth allocationUNCOMPROMISING SECURITY AND RELIABILITY • Secure, user/endpoint authentication • AES-128 cryptography in transit & at-restSCALABLE MANAGEMENT, MONITORING AND CONTROL • Real-time progress, performance and bandwidth utilization • Detailed transfer history, logging, and manifestENTERPRISE-CLASS FILE DELIVERY • Transfers up to thousands of times faster than FTP/HTTP(S) • Precise and predictable transfer times • Extreme scalability (concurrency and throughput)
  45. 45. SOLUTION: ASPERA ON-DEMAND DIRECT-TO-S3 client cargo downloaderpoint-to-point mobile appsconnect plug-in
  46. 46. OVERCOMING BOTH BOTTLENECKS #1 — TRANSFER DATA TO EC2 OVER WAN EFFECTIVE THROUGHPUT• http transfer over WAN (single stream) <10 Mbps• Typical internet conditions • 50–250ms latency & 0.1–3% packet loss <10 to 100 Mbps• 15 parallel http streams• Aspera fasp transfer over WAN to EC2 up to 1Gbps (per EC2 Extra Large Instance) #2 — TRANSFER DATA FROM EC2 TO S3 EFFECTIVE THROUGHPUT• Standard single stream http 10 to 100 Mbps• Aspera S3 Proxy up to 1Gbps • With parallel I/O http streams (per EC2 Extra Large Instance) 10 TB transferred per 24 hours
  47. 47. ASPERA PRODUCT & TECHNOLOGY PORTFOLIO DISTRIBUTE COLLABORATE AUTOMATE Complete portfolio of Global person-to-person Web-based application servers and clients for and project-based and SDK for creating andhigh-speed data delivery exchange and managing automated file- and distribution. collaboration of files based workflows. and directories. TRANSPORT Our unique, patented faspTM transport technologies provide unparalleled speed, efficiency, and bandwidth control over any size, distance, and network.
  49. 49. Next StepsGetting Started• Simply sign up for AWS at• Store some content in S3, put a cloudfront distribution on it, and compare!• Try your media processing applications on EC2• Take advantage of the Free Tier to experiment with more advanced services . 51
  50. 50. Thank You!