Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

How to backup, restore and archive your data on AWS

129 views

Published on

AWS offers storage, networking, and data transfer services so you can build and deploy solutions to extend backup and archive targets to the AWS Cloud, increasing scalability, durability, security, and compliance.

  • Be the first to comment

  • Be the first to like this

How to backup, restore and archive your data on AWS

  1. 1. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T How to backup, restore, and archive your data on AWS Everett Dolgner AWS Chris Astall Photobox
  2. 2. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Agenda Backup and archive…. AWS storage options Getting data into AWS Hybrid solutions AWS Backup CloudEndure AWS partner solutions Photobox: 9PB migration
  3. 3. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Backup challenges today • IDC predicts that the collective sum of the world’s data will grow from 33 zettabytes this year to a 175 ZBs by 2025, for a CAGR of 61 percent. • The world’s information is doubling every two years. • Backup and restore use cases protect data from logical errors such as system failure, application error, or accidental deletion. IDC: Expect 175 zettabytes of data worldwide by 2025 Network World, Dec 3, 2018
  4. 4. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Backup/Archive before the AWS Cloud
  5. 5. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Backup/Archive: cloud connectors cloud connector HTTP/API Cloud connector Amazon S3 Storage Classes For active backup data For long-term backup dataHTTP/API
  6. 6. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Backup/Archive: gateways AWS Storage Gateway Amazon S3 Amazon S3 Glacier/S3 Glacier Deep Archive Amazon EBS Snapshots
  7. 7. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Five key questions Files Block Application DB 2) Where is the data stored? 4) What is your RPO and RTO? 3) How long do you need to keep it? 5) What about the network? 1) How much data do you have?
  8. 8. It’s not binary. Start somewhere and scale up.
  9. 9. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  10. 10. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Complete AWS storage portfolio Object storage S3 Standard S3 Glacier Deep Archive S3 Glacier S3 Intelligent-Tiering S3 One Zone-IA S3 Standard-IA Block storage Provisioned IOPS SSD Cold HDD Throughput-Optimized HDD NEW! File storage EFS Standard EFS Infrequent Access Elastic Amazon EFS AWS Storage Gateway Family Amazon S3 Amazon FSx for Lustre Amazon EBS Amazon EC2 NEW! Amazon FSx for Windows File Server NEW! Backup & Restore AWS Backup NEW! NEW!
  11. 11. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Amazon S3 Storage Classes Access frequencyFrequent Infrequent • Active, frequently accessed data • Milliseconds access • > 3 AZ • $0.0210/GB • Data with changing access patterns • Milliseconds access • > 3 AZ • $0.0210 to $0.0125/GB • Monitoring fee per Obj. • Min storage duration • Infrequently accessed data • Milliseconds access • > 3 AZ • $0.0125/GB • Retrieval fee per GB • Min storage duration • Min object size • Re-creatable, less accessed data • Milliseconds access • 1 AZ • $0.0100/GB • Retrieval fee per GB • Min storage duration • Min object size • Archive data • Select minutes or hours • > 3 AZ • $0.0040/GB • Retrieval fee per GB • Min storage duration • Min object size S3 Standard S3 Standard-IA S3 One Zone-IA S3 Glacier S3 Intelligent- Tiering S3 Glacier Deep Archive • Archive data • Select hours • > 3 AZ • $0.00099/GB • Retrieval fee per GB • Min storage duration • Min object size
  12. 12. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Multiple Restore Speeds Retrieval tier S3 Glacier S3 Glacier Deep Archive Expedited 1-5 mins* N/A Standard 3-5 hours within 12 hours Bulk 5-12 hours within 48 hours
  13. 13. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Cross-Region Replication Automatically replicate data to any other AWS Region Configure by bucket, prefix, or tag values Replicate data to S3 Glacier Deep Archive Change the object owner in the destination region
  14. 14. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Lifecycle Management Create rules to automatically transition or expire your storage Configure by bucket, prefix, or tag values Lifecycle rules take action based on object age Example policy: Move all objects older than 30 days to Standard–Infrequent Access Move all objects older than 90 days to Glacier-Deep Archive
  15. 15. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Lifecycle Management Create rules to automatically transition or expire your storage Configure by bucket, prefix, or tag values Lifecycle rules take action based on object age Example policy: Move all objects older than 180 days to Glacier Move all objects older than 730 days to Glacier – Deep Archive
  16. 16. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Amazon S3 Storage Classes Access frequencyFrequent Infrequent • Active, frequently accessed data • Milliseconds access • > 3 AZ • $0.0210/GB • Data with changing access patterns • Milliseconds access • > 3 AZ • $0.0210 to $0.0125/GB • Monitoring fee per Obj. • Min storage duration • Infrequently accessed data • Milliseconds access • > 3 AZ • $0.0125/GB • Retrieval fee per GB • Min storage duration • Min object size • Re-creatable, less accessed data • Milliseconds access • 1 AZ • $0.0100/GB • Retrieval fee per GB • Min storage duration • Min object size • Archive data • Select minutes or hours • > 3 AZ • $0.0040/GB • Retrieval fee per GB • Min storage duration • Min object size S3 Standard S3 Standard-IA S3 One Zone-IA S3 Glacier S3 Intelligent- Tiering S3 Glacier Deep Archive • Archive data • Select hours • > 3 AZ • $0.00099/GB • Retrieval fee per GB • Min storage duration • Min object size
  17. 17. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  18. 18. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T AWS Options for Data Transfer AWS Direct Connect Amazon Kinesis Firehose AWS Snowball AWS Snowmobile AWS Storage Gateway Amazon S3 Transfer Acceleration AWS DataSync AWS Transfer for SFTP AWS Snowball Edge Amazon Kinesis Data Streams Amazon Kinesis Video Streams
  19. 19. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T AWS Snowball AWS Snowball Edge Compute or Storage Optimized AWS Snowmobile • 80-TB storage capacity • 10GE networking • Data encryption end-to-end • Rugged 8.5-G impact case • Rain and dust-resistant • 42 or 100-TB storage capacity • Data encryption end-to-end • Rugged 8.5-G impact case • Rain and dust resistant • AWS Greengrass support for local compute, messaging, and caching • EC2/AMI support for edge compute • Optional GPU • Exabyte-scale storage in a 45-ft container • Data encryption end-to-end • Dedicated security personnel • GPS tracking, alarm monitoring, 24/7 surveillance, and optional additional security AWS Snow Family
  20. 20. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Shared file system NFS TLS AWS DataSync On-Premise Amazon S3 bucket AWS Storage resources AWS DataSync Agent deployed on-premises for fast access to local storage Region Amazon EFS file system AWS DataSync agent Data transfer over the WAN via efficient purpose- built protocol Managed from the console or AWS Command Line Interface (AWS CLI) Service in AWS writes or reads data from AWS storage services
  21. 21. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  22. 22. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T File gateway overview Store and access objects in Amazon S3 from file-based applications with local caching Use cases • Backup files and databases as objects • Hybrid workflows for on- premises data (e.g., data piplelines) • Low-latency on-premises access to data in cloud (e.g., media editing) • Content distribution and collaboration File Gateway Objects in your Amazon S3 bucket NFS v3/v4 SMB v2/v3
  23. 23. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Volume gateway overview • Supports iSCSI block protocol • Available in two modes: cached and stored • Compresses and tiers partial or whole volumes to the cloud to reduce on-premises SAN/NAS footprint • Take EBS snapshots of on-premises volumes and store them in cloud Cloud-backed block storage presented on-premises Key Capabilities Application server Amazon EBS snapshots ServiceVolume Gateway Volume stored in Amazon S3 HTTPSiSCSI Customer Premises CACHE
  24. 24. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T • Emulates physical tape device with a media changer and tape drives • Virtual tapes written to Amazon S3 • Ejected virtual tapes are marked “read only” and moved to “tape shelf” in Amazon Glacier • Recovery: Retrieve tapes to library (3–5 hours) and read data from a gateway Tape gateway overview Virtual tapes presented to on-premises backup applications Key Capabilities Backup server Tape library Tape shelf On-Premise iSCSI HTTPS Media changer Tape Gateway Tape drives VTL
  25. 25. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Storage Gateway is available on a variety of form factors Virtual machines Amazon EC2 Hardware appliance Corporate data centers In AWS Remote locations, branch offices WhereWhen Leverage existing VM infrastructure In-cloud data retrieval/access/DR Out of the box simplified experience
  26. 26. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  27. 27. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T What is AWS Backup • Centralized backup management service • Central console and set of APIs for protecting your application data across AWS services • Common way to protect application data in the AWS Cloud and on-premises • Meet business and regulatory backup compliance requirements • Simple and cost-effective AWS Backup
  28. 28. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T AWS Backup: services supported at launch ✓✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ Automated Backup Schedules Automated Retention Management Centralized Backup Monitoring/Logging KMS Integrated backup encryption Lifecycle to Cold Storage Independent Backup Encryption Amazon EFS Amazon EBS Amazon RDS DynamoDB AWS Storage Gateway
  29. 29. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  30. 30. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Better, Faster, More Affordable Disaster Recovery • Improve recovery objectives & reduce TCO • Simple setup lets you start in minutes • Same highly automated process for all workloads (regardless of OS type/version, application, or DBs) • Minimizes complexity and reduces risk • Easily failover and failback Highly Automated Replicate From Any Source Minimal Skill Set Required to Operate Easy, Non- Disruptive DR Tests ReliableFlexible Failback to Cloud/On-Prem Robust, Predictable, Non-Disruptive Continuous Replication Wide Range of OS, Application, and Database Support Protection Against Ransomware, Corruptions, and Human Errors RPO: Sub-Second RTO: Minutes Automated Lightweight Staging Area Reduces TCO
  31. 31. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T How CloudEndure Disaster Recovery Works Lightweight Staging Area in Target Region Source Data Center Target Subnet(s) Continuous data replication traffic (compressed and encrypted) Lightweight Linux Replication Server Low-Cost Block Storage Orchestration & system conversion ready to run workloads in minutes regardless of disk size • CloudEndure continuously replicates any application or database from any source into AWS • Business outcome: Use AWS to significantly reduce DR TCO, while improving recovery objectives
  32. 32. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  33. 33. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Archive Partner Integrations
  34. 34. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved. The AWS Partner Network (APN) helps you seamlessly integrate AWS services into your existing storage environment Backup & Restore Archive Primary Storage BC/DR AWS third-party integrations
  35. 35. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  36. 36. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Group history
  37. 37. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Photobox’s journey to AWS Started moving in 2012 The web platform was 100% on AWS by 2015 The original lift and shift evolved in a cloud native containerized infrastructure Now we are developing new services using AWS Serverless Technologies Today fully in AWS
  38. 38. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Photobox’s journey to AWS Photo storage wasn’t immediately migrated No significant reasons to move Not expected to be cost effective A migration would have been an hard technical challenge On-Premise equipment was still fairly new and supported However there were limitations of our on-prem infrastructure Durability Reliability Both came from our single copy in single DC approach
  39. 39. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  40. 40. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Growth in photos During 2018 uploads peaked at 161 millions a month and 1.1 millions in an hour
  41. 41. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Photobox’s photo library When we started migration 5.7 Billion individual photos / 9 Petabytes Each of them stored in three different resolutions: • Full-size ~ 1.6 MB • Scaled photo < 100 KB • Thumbnail < 10 KB And we are growing at a rate of 1.2 B/year Today it is 7 Billion photos and 12 Petabytes …expected to keep increasing
  42. 42. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Access patterns Only 48% of photos uploaded are printed Of those: • 41% are printed on the day they are uploaded • 83% within 30 days • …and 95% within 180 days
  43. 43. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  44. 44. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Why moving? Cloud first strategy Photobox had already migrated the website into AWS and improved the operating model Cost About to face large capital investments for hardware expansion and replacement Cloud storage cost kept dropping consistently Ecosystem AWS services can be leveraged to enhance the upload and serving workflow
  45. 45. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Why moving? (continued) Improved security Access control in AWS is more flexible and granular than on-premise Improved resilience and flexibility Any given photo was stored in a single datacenter Uploads were “sharded” between the two datacenters (AMS and PAR) Storage arrays took months to scale up
  46. 46. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T How to move? Early dilemmas: network vs Snowball Edge With the existing 4Gbps internet uplink the migration would have taken ¾ of a year Upgrading the production network was slow and expensive There was no need to move everything Scaled photos and thumbnails can be rebuilt directly into AWS …or on the fly Snowball Edge proved to be a good fit 82TB of useable storage per device ~110 of them needed to carry out the migration
  47. 47. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Integrity checking After the first use photos are hardly ever accessed again Impossible to spot corrupted objects The migration strategy/workflow must immediately fix any inconsistency Hashing comes to the rescue! In our use case the ETag of objects always matched their MD5 hash (careful: this is not always true) Easy to query it from S3 Inventory and compare to what has been computed at the very beginning of the process
  48. 48. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T The migration workflow
  49. 49. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  50. 50. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Challenges… at scale Network congestion The on premise network had never been tested at 100% Traffic flows during a migration are “atypical” Handling an huge amount of small objects The overhead of every single operation becomes significant Both the data extraction and loading process had to be optimized with this in mind
  51. 51. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Challenges… at scale (continued) Tracking migration status Needed to follow the status of every single object …this results in a massive database and heavy data flow Logistics in the physical world In EU UPS doesn’t deliver nor collect on weekends Next day delivery …doesn’t always happen next day
  52. 52. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.
  53. 53. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Evolution of the serving architecture Serverless as a natural evolution Low management overhead Can scale instantly Multi-region is the next big step Keeping the live user experience consistent in case of a single region outage S3-ZIA empowers this model
  54. 54. © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.S U M M I T Evolution of the user journey Amazon S3 is only the foundation Now that the photo assets are in AWS they can be retrieved/used right away A world of possibilities Amazon Rekognition can be leveraged to radically change Photobox’s customer journey Magic Book Using AI to curating customer photos to enhance their experience
  55. 55. S U M M I T © 2019, Amazon Web Services, Inc. or its affiliates. All rights reserved.

×