Deep Dive on Archiving and Compliance
•
1 like•568 views
by Henry Zhang, Sr. Product Manager, AWS Compared to storing long-term datasets on-premise, archiving in the cloud is a smart alternative whether you’re looking for an active archive solution, tape replacement, or to fulfill a compliance requirement. Learn how AWS customers are simplifying their archiving strategies and meeting compliance needs using Amazon Glacier.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Deep Dive on Archiving and Compliance
Similar to Deep Dive on Archiving and Compliance (20)
More from Amazon Web Services
More from Amazon Web Services (20)
Deep Dive on Archiving and Compliance
- 1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Deep Dive on Archiving and Compliance with Amazon Glacier Henry Zhang Senior Product Manager, Amazon Glacier
- 2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Cloud Data Migration Direct Connect Snow* data transport family 3rd Party Connectors Transfer Acceleration Storage Gateway Kinesis Firehose The AWS Storage Portfolio Object Amazon GlacierAmazon S3 Block Amazon EBS (persistent) Amazon EC2 Instance Store (ephemeral) File Amazon EFS
- 3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Satellite Image Archive • DigitalGlobe takes Satellite imagery of the Earth • 100PB image library = 6 billion square kilometers • 1PB new image every year • Images to be archived and retained for decades
- 4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Patient data–Philips Healthcare • HealthSuite digital platform powered by AWS • 15 petabytes of patient data • Archived for decades (beyond the lifetime of patients) • Uses AWS HIPAA-eligible services in the BAA
- 5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Public sector–King County • Most populous county in Washington state • Replaced tape solution for backup from 17 agencies • Meets compliance requirement • Saved $1MM in first year; no more tape refresh or management churn
- 6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Archive: Data retained for the long term, for compliance or potential future reference Data archiving needs are growing everywhere • Media assets, 4K, 8K • Health care/life sciences • Financial services • Regulated industries • Oil and gas/geospatial • Digital preservation • Long-term backups • Logs
- 7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Consideration 1 – Total Archive Cost
- 8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Traditional archiving approaches • Tape libraries, robots, drives, media • Onsite (online and offline) • Offsite tape out/vaulting • Specialized software and personnel • Tape refresh every 3-5 years
- 9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved How can AWS help with your archival? Metered usage: Pay as you go No capital investment No commitment No risky capacity planning Avoid risks of physical media handling Control your geographic locality for performance and compliance
- 10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved 1 PB raw storage 800 TB usable storage 600 TB allocated storage 400 TB application data Storage pricing - pay only for what you use AWS Cloud Storage Amazon Glacier starts at $0.004/GB/month Price dropped by 43% on 11/21/2016
- 11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Consideration 2 – Durability
- 12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved 99.999999999% Durability Durability for long-term preservation Built-in Fixity Checking Automatic recovery
- 13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Consideration 3 – Accessibility
- 14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Accessing Amazon Glacier 1. Direct Amazon Glacier API/SDK 2. Amazon S3 lifecycle integration 3. Third-party tools and gateways FastGlacier
- 15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Amazon Glacier – Direct access/APIs Create Vault Configure Access Upload Archives Register Archive ID Data Upload Initiate Retrieval Async Retrieval Completion Completion Notification Download Data Data Retrieval
- 16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Use Glacier via S3 Object Lifecycle S3 Standard Active data Archive dataInfrequently accessed data S3 - Infrequent Access Amazon Glacier Synchronous access Async accessSynchronous access $0.023/GB/mo. $0.004/GB/mo.$0.0125/GB/mo.
- 17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved - Transition Standard to Standard-IA - Transition Standard-IA to Amazon Glacier - Transition based on object tags - Expiration and versioning Data lifecycle management T T+3 days T+5 days T+ 15 days T + 25 days T + 30 days T + 60 days T + 90 days T + 150 days T + 250 days T + 365 days Data access frequency over time
- 18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Transition older videos to Standard-IA
- 19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Save money on storage 45% saving over S3 Standard-IA 68% saving over S3 Standard-IA * Assumes the highest public pricing tier
- 20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Amazon Glacier – Third-party tools and gateways • Consumer grade: less than $50 • Example: Cloudberry, FastGlacier, Arq (Haystack Software) • Small / medium business: $500 - $1,000 • Example: Synology, Veeam, QNap • Enterprise gateway and data management software • Example: NetApp AltaVault, CommVault, StorNext, StoreReduce, Vidispine
- 21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Which option should I choose? • Use S3 lifecycle managed Amazon Glacier if the S3 object keys are sufficient for index/search capability • Use Amazon Glacier directly if you already plan to store more metadata/indices in a database • Use 3rd party tools to minimize coding
- 22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Amazon Glacier – Data Retrieval Tiers Standard Retrieval • Current model • 3-5 hours • Disaster Recovery Bulk Retrieval • Batch/Bulk access • 5-12 hours • PB scale re-transcoding or video/image analysis Expedited Retrieval • Emergency access • 1-5 minutes • Last minute play-out schedule swap $0.03/GB $0.01/GB $0.0025/GB On-site tape replacement Off-site tape replacement
- 23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved • Media distribution backbone (Ve.nue platform) • Over-The-Top (OTT) broadcast service • 20PBs of media assets, 1MM+ hours of high-res content • Assets to be archived and retained for decades Video archives
- 24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Comprehensive media lifecycle @SonyDADCNMS
- 25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved “If physical deliveries can happen within one hour based on unpredictable requests, surely we are able to exceed such expectations digitally” @SonyDADCNMS
- 26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Our migration The Challenge • Seamlessly migrate a platform that enables content delivery across all devices and more than 1,200 distribution points worldwide • Store 20 petabytes of motion picture and television content • Equating to 1,000,000 M+ hours of content • At a growth curve of ~1 petabyte every quarter Desired Goals: • One-hour delivery turn around time • Agile, scalable, predictable cost model and infrastructure • Investing in innovation vs. hardware @SonyDADCNMS
- 27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved On-premises asset storage workflow @SonyDADCNMS
- 28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved AWS Cloud-based asset storage workflow @SonyDADCNMS
- 29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Amazon Glacier vs. on-premises cost comparison @SonyDADCNMS
- 30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Consideration 4 - Compliance
- 31. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Amazon Glacier Vault Lock allows you to easily set compliance controls on individual vaults and enforce them via a lockable policy Time-based retention MFA authentication Controls govern all records in a vault Immutable policy Two-step locking Compliance storage with Vault Lock
- 32. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Vault Lock for compliance storage • Non-overwrite, non-erasable records • Time-based retention with “ArchiveAgeInDays” control • Policy lockdown (strong governance) • Legal hold with vault-level tags • Configure optional designated third-party access and grant temporary access
- 33. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Amazon Glacier received a third-party assessment from Cohasset Associates on how Amazon Glacier with Vault Lock can be used to meet the requirements of SEC Rule 17a-4(f) and CFTC 1.31(b)-(c).
- 34. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Proofpoint • Cloud-based security and compliance for the enterprise: threat research, email, mobile, social, digital risk • Founded 2002, public in 2012 • $350M annual revenue, $3B market cap • Big AWS user
- 35. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Proofpoint SocialPatrol Policy controls and enforcement for social • Combats fraudulent brand impersonation • Moderates content at scale • Ensures compliance in publishing • Integrates with social APIs • 150+ classifiers using NLP and ML • Text, links, images, meta data • Ingesting >1M social posts per day • Built in AWS
- 36. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Proofpoint SocialPatrol Archive with Glacier SEC Rule 17a-4(f)-compliant archive, purpose-built for social, enabled by Amazon Glacier and Vault Lock PFPT in AWS Policy engine MySQL/C*/SolrSocial Amazon Glacier & Vault Lock
- 37. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Proofpoint SocialPatrol Archive The customer specifies the retention period in Proofpoint Social:
- 38. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Proofpoint SocialPatrol Archive Via AWS API we create a vault for that customer:
- 39. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Proofpoint SocialPatrol Archive Via AWS API, we lock the vault, and specify policy to observe a legal hold via a tag.
- 40. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved Proofpoint SocialPatrol Archive As social content flows in, we record its purge date and surface that to the user. Each piece of social content is an archive in the vault.
- 41. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved aws.amazon.com/activate Everything and Anything Startups Need to Get Started on AWS
Editor's Notes
- Amazon EBS provides highly available, consistent, low-latency block storage for Amazon EC2, to help tune applications with the right storage capacity, performance and cost. EBS is designed for workloads that require persistent storage accessible by single EC2 instances. Typical use cases include Big Data analytics engines (like the Hadoop/HDFS ecosystem and Amazon EMR), relational and NoSQL databases (like Microsoft SQL Server and MySQL or Cassandra and MongoDB), stream and log processing applications (like Kafka and Splunk), and data warehousing applications (like Vertica and Teradata). Amazon EFS provides simple, scalable, fully managed file system storage for sharing data between Amazon EC2 instances in the AWS Cloud. It delivers a file system interface with standard file system access semantics for Amazon EC2 instances. Amazon EFS grows and shrinks capacity automatically, and provides high throughput with consistently low latencies. Amazon EFS is designed for high availability and durability, and provides performance for a broad spectrum of workloads and applications, including Big Data and analytics, media processing workflows, content management, web serving, container storage, and home directories. Amazon S3 is object storage designed to store and access any type of data over the Internet. It is secure, 99.999999999% durable, and scales past tens of trillions of objects. Amazon S3 is used for backup and recovery, tiered archive, user-driven content (like photos, videos, music and files), data lakes for Big Data analytics and data warehouse platforms, or as a foundation for serverless computing design. Amazon Glacier is an extremely low-cost, highly durable storage for long-term backup and archive. Amazon Glacier is a solution for customers who want low-cost storage for infrequently accessed data. It can replace tape while assisting with compliance in highly regulated organizations like healthcare, life science, and financial services. Amazon Cloud Data Migration services help customers migrate data into and out of the AWS Cloud in offline, online, or streaming models.
- In addition to media files, we also see healthcare and life science customers archiving long term data on AWS. Philips Healthcare run their HealthSuite Digital Platform on AWS and stores PBs of patient data, which must be retained for the lifetime of the patient and beyond. Philips Healthcare supports over 1500 hospitals in the U.S. which produce patient records and medical images daily. Philips uses a number of HIPAA eligible services on AWS that helps them meet heathcare specific compliance requirements for storing Personal Health Information (PHI).
- Finally, we also see customers in the public sector storing long term archive data on AWS. King county is the largest county in Washington State and is also home to the AWS headquarter. They replaced tape-based backup solutions in 17 agencies with AWS storage services that not only allowed them to meet their existing compliance requirements, but also helped them reduce tape management overhead and increased agility – they saved $1MM in the first year after switching to AWS.
- Across many industry segments, we see more data produced every day and an increasing desire to store and retain more of it for longer (if not keep everything forever) as long as it is operationally and financially feasible. More data is created due to higher resolution cameras, 4k, 8k videos, advancement in medical imaging/genomics sequencing technology, and the growing breadth and depth of regulations that require more firms to retain more data for potential audit purposes (call logs, voice mails, now even social media). On the other hand, customers now have access to new tools to analyze massive amounts of historical data (example ad-tech industry). They want to retain user activity logs for longer periods so that they can go back and run a new algorithm to derive new insights in the future that may create new avenues for monetization. All of this business demand is putting pressure on reliable, scalable, and cost-effective data archiving solutions.
- Traditionally, customer have relied on on-premises storage arrays (NAS, SAN, and tapes) to archive data as well as expensive, purpose build compliance storage hardware to retain regulatory/compliance storage. These solutions typically require a hefty upfront capex investment and require on-going maintenance and capacity planning, and for those that use tapes, there is always the burden of tape refresh every few years to go from gen N-2 to N. Many customers have found the traditional purchasing and operational model burdensome and that it often hampers business growth/time to market.
- Amazon Web Services give you reliable, durable long term storage options without the up-front capital expenditures and complex capacity-planning burden of on-premises storage. Amazon storage services remove the need for complex and time-consuming capacity planning, ongoing negotiations with multiple hardware and software vendors, specialized training, and maintenance of offsite facilities or transportation of storage media to third party offsite locations. You only pay for what you use and you can choose one of the N AWS world wide regions based on your compliance and data sovereignty objectives. AWS does not move your data out of a region unless you specifically request to do so.
- The other thing to note is our strong histroy of price cuts. Normally when you buy capital equipment and price is reduced, no one calls you and offers you a refund for what you’ve already purchased. AWS frequently cut pricing as we contiue to gain scale and realize efficienes in our operational model. Just last year we cut S3 pricing by 65%. This year we introduced S3-IA which provides roughly 60% savings compared to S3 and cut Glacier pricing by 30%.
- S3 is highly durable. Your data is stored across three separate facilities giving you geo-redundancy and we can sustain data loss in two facilities simultaneously and your data is still safe, providing a statistical measure of 11 9’s of durability. Consider what it would take to architect for such a level of durability in your own data centres
- Across the board, we provide 3 storage options with 3 different performance characteristics and price points. On the left, we have S3 Standard which is our high performance object storage for the internet, designed for very active, hot workloads. Data in S3 Standard is available in milliseconds and costs $0.03/GB/month (starting at). On the right hand side, we have Glacier, our cold storage service designed for long term archival and infrequently accessed data. Data in Glacier has a 3-5 hour access latency and Glacier costs $0.007/GB/month (starting at). Between the hot and cold options, we have a “warm” option – S3 infrequent access designed for data you plan to access maybe a few times a year or what we think of as “active archive”. S3-IA costs $0.0125/GB/mo (starting at). From an archiving perspective, customers typically use S3IA and Glacier together. Just a quick note terminology – S3 stores data in buckets and each piece of data is an object; Glacier stores data in vaults (equivalent of S3 buckets) and each piece of data is called an archive (similar to object). You will hear me use bucket/vault/object/archive later on.
- Customers like the 3 storage options and find it very flexible and easy to pick one that suits their need. What’s more, they also like how we help them tie it together with the Data Lifecycle Management feature that allows you use the 3 storage options in tandem and tier data from hot to warm to cold as data ages. If you think about the typical lifecycle of data, newly created active data is access very frequently. Think about a new video clip you create and share with your friends and family on S3. People will be consuming this new data actively, this new video will be played back frequently, shared and commented on very frequently. As this video becomes older, a smaller number of people will engage, it will be LESS FREQUENTLY accessed and can be archived to S3IA. As time goes on, the video becomes colder and can be archived in Glacier for the lowest cost. Beyond tiering data, Data Lifecycle Management can also automate expiration/deletion of data and supports storing multiple versions of the same object.
- You can transition objects from S3 Standard to SIA after 30 day and then transition to Glacier after 365 days.
- One of the key advantages of using automated storage tiering is cost reduction. S3->SIA saves 58% and and SIA to Glacier saves 44%.
- Remove all logos so we are not recommending, list increasing benefits as you pay more.
- Remove all logos so we are not recommending, list increasing benefits as you pay more.
- Add “…” for more options
- Many media customers store video archives on AWS and keep them indefinitely. Video assets are typically the core creation/work of art of creative professionals which are sometimes referred to as their “crown jewels”. Sony DADC is Sony’s new media division and recently launched Ve.nue, a media processing and distribution service powered by AWS. Sony DADC stores PBs of video assets on AWS and intends to deep the data indefinitely. What we see across many media customers is that they store high res master/mezzanine files in the cloud that can be transcoded to new delivery formats for monetization when necessary, say for a new iPhone, or a new director’s cut/special edition release.
- Decision to 1 year
- Now, these customers can use Glacier as compliance storage with Glacier’s Vault Lock capability. We launched Vault Lock in summer 2015 which allows customers to set compliance controls on the Glacier storage containers (vault) via a lockable policy. For example, for customers who used to buy WORM storage/drives for records retention, they can now easily set up a Vault Lock with say 7 year retention and Glacier will enforce the retention control such that any archives stored in the Vault cannot be deleted until it has been stored for 7 years. We recognize that data retention is one of the most common archive use cases and we launched Vault Lock to make life simpler for these customers. However, Vault Lock does more than data retention (WORM). It can be used to enforce a number of compliance objectives, such as protection on data access. For example, a pharmaceutical company can lock their top secret drug formula in a Vault that requires a 3 way multi-factor authentication for access.
- Let’s take a look at how Vault Lock can help you achieve compliance archiving objectives in more detail. To start, it lets you quickly configure non-overwrite/non-erasable records so you can use it as WORM storage. You can then specify time based retention, which is defined by the “ArchiveAgeInDays” control code. The Vault Lock policy is immutable after you test and lock it down, providing a strong form of governance, and finally, Vault Lock supports legal hold for cases when a firm gets a subpoena and they must retain all the related records for as long as the legal investigation continues. In that case the legal hold will override the underlying time-based retention – the record cannot be deleted even if it has exited the planned retention window. Those in the financial industry such as broker dealers also need to designate a 3rd party (D3P) to have read access to the regulatory data for continuity reasons. Glacier makes it easy for you to set up your D3P and our financial services page include a few partners that can provide such service.
- We understand that choosing a compliance storage offering or switching from one to another requires internal alignment such as convincing your compliance offer. For financial customers, we made this easier by obtaining a 3rd party compliance assessment from a reputable independent audit firm - Cohasset Associates, who has been in the compliance industry for over 40 years and has produced similar compliance evaluations for many compliance storage products on the market. Cohasset Associates found that Amazon Glacier with Vault Lock can be used to meet the requirements of financial services records retention rules, specifically, SEC Rule 17a-4(f) and CFTC 1.31(b)-(c). You can download a copy of this report from the Glacier website and share that with your compliance officer/relevant decision maker.
- Talk about our data hierarchy – customer maps to a Vault, social post is in an Archive. Retention and legal hold are set at Vault level. Walk through the policy. Note that we set it with less than 20 lines of json.