Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Deep Dive on Data Archiving in Amazon S3 & Amazon Glacier, with Special Guest, 20th Century Fox - STG304 - re:Invent 2017

772 views

Published on

Learn about ways to archive data for compliance or cost savings, and balance retrieval speed and cost to fit your specific use case. We examine concepts such as active archiving (archive storage with fast retrieval times), compliance archiving, and many more.

  • Be the first to comment

  • Be the first to like this

Deep Dive on Data Archiving in Amazon S3 & Amazon Glacier, with Special Guest, 20th Century Fox - STG304 - re:Invent 2017

  1. 1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Deep Dive on Data Archiving in Amazon S3 & Amazon Glacier, With Special Guest, 20th Century Fox H e n r y Z h a n g , S e n i o r P r o d u c t M a n a g e r, A m a z o n G l a c i e r B i l l W a l k e r, V P C l o u d E n g i n e e r i n g , 2 0 t h C e n t u r y F o x F i l m W h i l R e l i f o r d , V P I n f o r m a t i o n Te c h n o l o g y, 2 0 t h C e n t u r y F o x F i l m S T G 3 0 4
  2. 2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Data movement OnlineOfflin e Data security and management Complete set of data building blocks Amazon EFS Amazon EBS AWS Snow family AWS Storage Gateway Family AWS Direct Connect Amazon EFS File Sync Amazon S3 Transfer Acceleration Storage Partners Amazon Kinesis Data Streams Amazon Kinesis Video Streams Amazon S3 Amazon Glacier AWS KMS AWS IAM AWS CloudWatch AWS CloudTrail AWS CloudFormation AWS Lambda Amazon Macie AWS QuickSight
  3. 3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Benefits of Amazon S3 & Amazon Glacier Durable, Available, & Scalable Security & Compliance Query In Place Flexible Management Ecosystem Low-Cost
  4. 4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Just Say No X No capital investment X No commitment X No capacity planning X No idle capacity X No onerous media handling X No complex technology refreshes X No undifferentiated heavy lifting
  5. 5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Global Availability
  6. 6. Consideration 1 – Archive with Data Lifecycle Management
  7. 7. Archiving in S3 and Glacier w/ Data Lifecycle S3 Standard Active data Archive dataInfrequently accessed data S3 - Infrequent Access Amazon Glacier Synchronous access Async accessSynchronous access $0.023/GB/mo. $0.004/GB/mo.$0.0125/GB/mo.
  8. 8. - Transition Standard to Standard-IA - Transition Standard-IA to Amazon Glacier - Transition based on object tags - Expiration and versioning Data lifecycle management T T+3 days T+5 days T+ 15 days T + 25 days T + 30 days T + 60 days T + 90 days T + 150 days T + 250 days T + 365 days Data access frequency over time Lifecycle Policy Storage Class Analysis
  9. 9. Save money on storage • 45% saving over S3 Standard • 68% saving over S3 Standard-IA * Assumes the highest public pricing tier
  10. 10. Transition older videos to Standard-IA
  11. 11. Glacier - Flexible Data Retrieval Options All of your Glacier data is accessible with any of three retrieval options Standard Retrieval • Standard access • 3-5 hours • $0.01/GB Bulk Retrieval • Batch/Bulk access • 5-12 hours • $0.0025/GB Expedited Retrieval • Rare urgent access • 1-5 minutes • $0.03/GB On-site tape replacement Off-site tape replacement
  12. 12. Consideration 2 – Durability
  13. 13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 99.999999999% Durability Durability for long-term preservation Built-in Fixity Checking Automatic recovery
  14. 14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Traditional Redundancy DRPrimary
  15. 15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. S3 and Glacier’s Built-in Durability AZ1 AZ2 AZ3  99.999999999% durability  Objects replicated across at least three availability zones  AZ’s separated in power, network, flood domains  Automated periodic data integrity checks  Tolerant of concurrent failures  Storage devices, nodes, racks  Switch and network  Datacenters  Optional cross-region, cross-account replication
  16. 16. Cross-Region Replication with Lifecycle Archiving S3 Bucket A Amazon Glacier S3 Bucket B
  17. 17. Consideration 3 – Accessibility
  18. 18. Multiple Ways to Access S3 and Glacier 1. Use S3 and Glacier via S3 Lifecycle Management 2. Direct Amazon Glacier API/SDK 3. AWS Storage Gateway 4. 3rd party tools and gateways FastGlacier
  19. 19. Amazon Glacier – Direct access/APIs Create Vault Configure Access Upload Archives Register Archive ID Data Upload Initiate Retrieval Async Retrieval Completion Completion Notification Download Data Data Retrieval
  20. 20. Amazon Glacier – Third-Party Tools and Gateways • Consumer grade: less than $50 • Example: Cloudberry, FastGlacier, Arq (Haystack Software) • Small / medium business: $500 - $1,000 • Example: Synology, Veeam, QNap • Enterprise gateway and data management software • Example: NetApp AltaVault, CommVault, StorNext, StoreReduce, Vidispine
  21. 21. Which Option Should I Choose? • Use S3 lifecycle managed Amazon Glacier if the S3 object keys are sufficient for index/search capability • Use Amazon Glacier directly if you already plan to store more metadata/indices in a database • Use 3rd party tools to minimize coding
  22. 22. Consideration 4 – Compliance
  23. 23. Amazon Glacier Vault Lock allows you to easily set compliance controls on individual vaults and enforce them via a lockable policy Time-based retention MFA authentication Controls govern all records in a vault Immutable policy Two-step locking Compliance Storage with Vault Lock
  24. 24. Vault Lock for Compliance Storage • Non-overwrite, non-erasable records • Time-based retention with “ArchiveAgeInDays” control • Policy lockdown (strong governance) • Legal hold with vault-level tags • Configure optional designated third-party access and grant temporary access
  25. 25. Amazon Glacier received a third-party assessment from Cohasset Associates on how Amazon Glacier with Vault Lock can be used to meet the requirements of SEC Rule 17a-4(f) and CFTC 1.31(b)-(c).
  26. 26. Example Control: 1-year Record Retention • Deny delete archive operation • From anybody (root, administrators, users, business partners) • When ArchiveAgeInDays is <= 365 days Archive age computed from the time an archive lands in a vault
  27. 27. Example Control: 1-Year Record Retention
  28. 28. Vault Lock in the Amazon Glacier Console
  29. 29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 20th Century Fox Film Bill Walker, VP Cloud Engineering Whil Reliford, VP Information Technology
  30. 30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
  31. 31. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. 20th Century Fox Film Bill Walker, VP Cloud Engineering Whil Reliford, VP Information Technology
  32. 32. Who is 20th Century Fox? 32 Theatrical Home Entertainment International Television Distribution • Highest Grossing Film • The Most Inspiring Film • The Most Kick A** Film • All Major OTT Platforms • VOD / SVOD / PVOD / EST • We Still Sell Discs! • 195 Countries • 15 Formats / 68 Languages • 2.5+ Million Files / Year
  33. 33. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Make Early, Make Once, Use Many Media Types
  34. 34. Previous On-Premise Distribution Model 23,000,000 Files 23,000+ B2B Partners
  35. 35. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. FOX Marketing Assets – “Make Once Use Many”
  36. 36. Starting Key Art
  37. 37. Global 1-Sheet Reuse
  38. 38. Theatrical Marketing Reuse to… Living 1-Sheets Outdoor Art Digital Billboard Marketing Presentation Trailer Countdown Graphics Online Banner
  39. 39. Home Entertainment DVD / Blue Ray Labels Publicity Digital Billboard Standee Title treatment Packaging Electronic Sell Through
  40. 40. TV Distribution Billboards Layered Art Outdoor Art Key Art Background
  41. 41. Global Trailer Reuse Video assets such as Trailers and TV Spots can be dubbed and subtitled for localized distribution.
  42. 42. 13,188 Assets 510 External Users Worldwide: $748M Production: $200M
  43. 43. SALES MGMT RIGHTS MGMT BUSINESS MGMT M E T A D A T A M G M T MEDIA PROCESSINGDIGITAL ARCHIVE • Sales pipeline • Deal Management • Forecasting • Windows • Available • Rights • Content catalog • Global identifier • Consumer/tech metadata • Contracts • Order fulfillment • Customer Mgmt • Mastering • Localization • Graphics & editorial SCREENERS PLATFORM MARKETING/BQ PLATFORM • Sports • News • Entertainment • Customers • VIP / Press • Publicity SALES PLATFORM • 23K Customers • SVOD • Pay TV • Basic TV • Basic TV on demand • Free TV • Free TV on demand PI CONTENT SECURITY Supported formats across 68 languages: • Theatrical • Non-theatrical • DVD/BD • EST • VOD • NVOD • PPV home • PPV local API • Watermarking • Forensic burn-in • Piracy search • 500K Orders • 2.4M Deliveries• Content library • 80K titles • Digital masters 20th Century Fox Cloud Transformation
  44. 44. Initial Storage Migration – Fox Distribution Cache 23,000 B2B Partners 23M Assets 8 PB Monthly Deliveries: 200,000+ Territories: 195+
  45. 45. Data Center – S3/Glacier Migration STATS: 8 PB Content 23M Assets > 60 Days 16 gbps Average 16 gbps / 15 TB 2 x 10 G DR Asset Repository 8 PB 23M+ ASSETS Offsite 2300 Tapes Las Vegas Tape Library Amazon Glacier Amazon S3 Los Angeles Production Asset Repository 8 PB 23M+ ASSETS ON DEMAND
  46. 46. End State REPLICATION INGEST 100 TB/DAY FUTURE DR Asset Repository U S E A S T U S W E S T Transcoding Amazon Glacier Amazon S3 Amazon Glacier Amazon S3
  47. 47. 20th & 21st Century Fox Global Supply Chain AWS & Storage Architecture MEDIA PROCESSING MEDIA DISTRIBUTIONCONTENT LIBRARY (ARCHIVE) Wyoming DR Digital Vault (Tape) Los Angeles Prod Digital Vault & WIPs (Tape) S3 GLACI ER GLACIER Fox AWS Media Operations Fox AWS Distribution Cache S3 GLACIER OTT Platform Playout Platforms FoxFast Platform Marketing Platforms News Platforms B2B Platforms Sports Platforms Other 21CF Platforms8 PB (now) 16 PB (6 mos) 23 PB Finished Assets  VS Tenant #1 VS Tenant #3 VS Tenant #2 Masters 
  48. 48. Benefits of Moving to S3 and Glacier • Hard cost savings due to move from physical storage and tape to AWS • Reallocation of highly skilled engineering resources to higher value functions Cost Savings • Likelihood of corruption is significantly reduced relative to physical storage • Diversifying risk by distributing content across multiple data centers Durability • In previous scenario, we needed 60 days preparation to provision storage for new use cases • No need for outages on the environment to add storage to it Scalability • Ability to start copying content in days versus months • Ability to include multiple regions, including international Speed and Flexibility
  49. 49. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. STG313 – Big Data Breakthroughs – Wed, 12:15 PM OR 7:00 PM STG303 – Deep Dive on Amazon Glacier – Thurs, 1:45 PM STG312 – Best Practices for Building a Data Lake in Amazon S3 & Amazon Glacier – Thurs, 3:15 PM Learn more…
  50. 50. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. For Enterprise Storage Engineers • Learn how to architect and manage highly available solutions on AWS storage services • Advance toward AWS certifications • Help your organization migrate to the cloud faster Online at www.aws.training • Access 100+ new digital training courses including advanced training on storage • Deep Dives on S3, EFS, and EBS • Migrating and Tiering Storage to AWS (Hybrid Solutions) At re:Invent • Visit Hands-on Labs at the Venetian • Attend a proctored “Introduction to EFS” Spotlight Lab on Thursday at 3pm at the Venetian • Meet Storage experts at the Ask the Experts in Hands-on Labs room at the Venetian New Storage Training
  51. 51. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Q&A Amazon S3 Amazon Glacier
  52. 52. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Thank you!

×