Backup and Recovery for Linux with                    Amazon S3                 Learn how to back up and recover your on-p...
What We’ll Do Today     Move quickly     Examine today’s backup challenges     Justify use of Amazon S3 for backup     Inv...
Today’s Backup Challenges     Cheaper hardware     Abundant bandwidth     The paperless business     Automated data collec...
AMAZON S3 FOR BACKUP© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distri...
Why Should You Back Up to Amazon S3?     Primary features:        •     Offsite storage as a service        •     Fully re...
Why Should You Back Up to Amazon S3?     Additional features:        •     Server-side encryption        •     Transfer da...
More Reasons to Back up to S3!     Pay as you go model:        • No sticker shock        • Costs scale with usage        •...
BACKUP, STORAGE, AND    RECOVERY IN DEPTH© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copie...
Backup, Storage, and Recovery in Depth     There are lots of things to     thing about:        • Backup considerations    ...
Backup Considerations     What do you need back up?        • Files        • Databases        • Which ones (all or selected...
More Backup Considerations     How long will a backup run take?     What about data consistency?     How long is the backu...
Storage Considerations     Where do you store your backups?        • Onsite – What’s your DR plan?        • Offsite – How ...
More Storage Considerations     How do you organize and catalog your backups?        •     How do you locate the right one...
Retrieval Considerations     Which backup do you need?     How long will it take to retrieve     if offsite?     What if i...
Expiration and Rotation Cosiderations     How long do you retain the backups?     What happens when they expire?     What ...
Financial Considerations     How much is your business worth?     How much is your data worth?     How much does down time...
A TECHNICAL INTERLUDE© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distr...
Some Technical Issues     Files and databases may require distinct strategies     Files        •     Many objects of varia...
Consistent Backups     Logical consistency is a must!     Operational Model        • Sync all in-memory data to disk      ...
BACKUP ARCHITECTURES© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distri...
On-Premises to Cloud Backup Architectures     Transparent     Archive & Upload     Backup Tools     Backup Services© 2012 ...
Transparent Backup / File Share     Shared file system     Local writes mirrored to the cloud     “It just works”        •...
Transparent Backup / File Share Examples (1)     Dropbox (www.dropbox.com)        •     Master copy of all files stored in...
Transparent Backup / File Share Examples (2)     S3Fuse (http://code.google.com/p/s3fs/)        • User-space file system  ...
S3FS Configuration and Use     Configure Credentials:     Launch S3FS:     Copy files:© 2012 Amazon.com, Inc. and its affi...
AWS Management Console / Shell© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified...
Archive and Upload     Use existing local tools to create archive locally        • tar, cpio, etc.     Must have sufficien...
Archive and Upload Example     Requirements:        • tar (standard utility)        • S3 Curl (Free download at http://aws...
Archive and Upload Example (1)     Create archive:     Upload to S3:s3curl.pl --id=[…] --key=[…] --put=backup_public_html_...
Archive and Upload Example (2)     File uploads to S3:     Run the tar and upload from a cron© 2012 Amazon.com, Inc. and i...
Archive and Upload Example (3)© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified...
Archive and Upload Example (4)                                                                                            ...
About This Simple Example…     Store as much as you want     For as long as you want     99.999999999% durability     Pay ...
Commercial Backup Tools                                                                                                   ...
Backup Services (1)                                                                                                     • ...
Backup Services (2)     FTP / SFTP / WebDAV Gateway                                                                       ...
CLOUD-BASED BACKUP OPTIONS© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or ...
Cloud-Based Backup Tools     Everything mentioned so far still applies        • Low (internal) latency makes it even faste...
EBS Volume Review     Create volume (1 GB to 1 TB) in an AZ     Attach to an EC2 instance     Format (mkfs) and use     Ba...
Create EBS Snapshot© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distrib...
Catalog EBS Snapshots© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distr...
Restore EBS Volume From Snapshot© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modifi...
EBS Consistent Snapshot     Freeze file system     Lock databases     Take snapshot     Unlock databases     Unfreeze file...
Schedule EBS Backups     Cron and the EC2 command line tools:        ec2-create-snapshot -d "Home Volume Snapshot" vol-f12...
Schedule EBS Backups - Skeddly© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified...
Things We Didn’t Cover     Amazon Relational Database Service        • Back up your RDS (MySQL or Oracle) instances just l...
© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in...
Things To Remember     The cloud makes backup and recovery easy     You can get started for pennies per month     The clou...
For More Information     AWS Backup Site: http://aws.amazon.com/backup-storage/     Slides http://www.slideshare.net/Amazo...
Thanks For Watching!© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distri...
Upcoming SlideShare
Loading in...5
×

Backup and Recovery for Linux With Amazon S3

7,874

Published on

You will learn how to create file archives, upload them to Amazon S3, and manage permissions and lifetimes, giving you the ability to back up any amount of data and to retain it for as long as you'd like. A number of open source and commercial backup and archiving tools will be demonstrated, as time permits.

You will also learn how to use built-in AWS facilities to quickly and easily create and restore snapshots of entire disk volumes.

Published in: Technology, Business
1 Comment
20 Likes
Statistics
Notes
No Downloads
Views
Total Views
7,874
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
1
Comments
1
Likes
20
Embeds 0
No embeds

No notes for slide

Transcript of "Backup and Recovery for Linux With Amazon S3"

  1. 1. Backup and Recovery for Linux with Amazon S3 Learn how to back up and recover your on-premise and cloud-based Linux systems. Jeff Barr (@jeffbarr) Senior Web Services Evangelist© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  2. 2. What We’ll Do Today Move quickly Examine today’s backup challenges Justify use of Amazon S3 for backup Investigate backup, storage, and recovery in depth Review options for on-premise & cloud-based backup Give you enough information to get started today Address your questions© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  3. 3. Today’s Backup Challenges Cheaper hardware Abundant bandwidth The paperless business Automated data collection Data analysis (Big Data) Strategic importance of data Regulatory requirements http://www.datacenterknowledge.com/archives/2012/03/20/trends-driving-data-growth-backup/© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  4. 4. AMAZON S3 FOR BACKUP© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  5. 5. Why Should You Back Up to Amazon S3? Primary features: • Offsite storage as a service • Fully redundant • Data stored on multiple devices in multiple facilities • Designed for 99.999999999% durability • Designed for 99.99% availability • Available in all 8 AWS Regions • Full control over data residency • Continuous price drops© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  6. 6. Why Should You Back Up to Amazon S3? Additional features: • Server-side encryption • Transfer data through an encrypted (SSL) connection • ACLs (Access Control Lists) to regulate access • Object expiration • Access logging • Physical security • Location control© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  7. 7. More Reasons to Back up to S3! Pay as you go model: • No sticker shock • Costs scale with usage • No hardware to buy Broad industry acceptance: • 762 billion objects stored as of Q4, 2011 • Many third-party tools and applications are available© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  8. 8. BACKUP, STORAGE, AND RECOVERY IN DEPTH© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  9. 9. Backup, Storage, and Recovery in Depth There are lots of things to thing about: • Backup considerations • Storage considerations • Recovery considerations© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  10. 10. Backup Considerations What do you need back up? • Files • Databases • Which ones (all or selected) Full or incremental backups? How frequently? How much data? Annual growth rate?© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  11. 11. More Backup Considerations How long will a backup run take? What about data consistency? How long is the backup window? The backup device: • How much does the backup device cost? • What is its useful life? • Do you have a spare one? • For each facility? • What is the cost / GB for the backup medium?© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  12. 12. Storage Considerations Where do you store your backups? • Onsite – What’s your DR plan? • Offsite – How much does that cost? How long do you retain backups? • Physical space limitations? • What about media deterioration? How quickly do you need to recover? • How long (days) until you can retrieve your data?© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  13. 13. More Storage Considerations How do you organize and catalog your backups? • How do you locate the right one? • What data is stored? • When was it stored? • When does it expire?© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  14. 14. Retrieval Considerations Which backup do you need? How long will it take to retrieve if offsite? What if it is missing? What if the data is corrupt? What is the expected retrieval / recovery time?© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  15. 15. Expiration and Rotation Cosiderations How long do you retain the backups? What happens when they expire? What is the media rotation plan? What happens to old media? How do you dispose of old backups?© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  16. 16. Financial Considerations How much is your business worth? How much is your data worth? How much does down time cost you? What is your medium cost / GB? How much does your device cost? • How many do you need? What are the operational costs?© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  17. 17. A TECHNICAL INTERLUDE© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  18. 18. Some Technical Issues Files and databases may require distinct strategies Files • Many objects of variable size • Often date-stamped, allowing incremental backups • Open files, In-flight writes Databases • Generally large, monolithic objects • Open transactions • Locks & timeouts RAID – logical (file system) vs. physical (device) backup© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  19. 19. Consistent Backups Logical consistency is a must! Operational Model • Sync all in-memory data to disk • Freeze database or filesystem • fsfreeze – Linux • mysql stop - MySQL • Run backup • Unfreeze • Continue© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  20. 20. BACKUP ARCHITECTURES© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  21. 21. On-Premises to Cloud Backup Architectures Transparent Archive & Upload Backup Tools Backup Services© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  22. 22. Transparent Backup / File Share Shared file system Local writes mirrored to the cloud “It just works” • Save locally • Mirror to cloud© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  23. 23. Transparent Backup / File Share Examples (1) Dropbox (www.dropbox.com) • Master copy of all files stored in S3 • Install Linux client on premises on cloud • Backup == copy to Dropbox directory • Recover == copy from Dropbox directory© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  24. 24. Transparent Backup / File Share Examples (2) S3Fuse (http://code.google.com/p/s3fs/) • User-space file system • Store files locally, automatically uploaded to S3 Application S3Fuse Amazon S3 Bucket Kernel© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  25. 25. S3FS Configuration and Use Configure Credentials: Launch S3FS: Copy files:© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  26. 26. AWS Management Console / Shell© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  27. 27. Archive and Upload Use existing local tools to create archive locally • tar, cpio, etc. Must have sufficient local free space Upload to S3 Manage S3 archive© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  28. 28. Archive and Upload Example Requirements: • tar (standard utility) • S3 Curl (Free download at http://aws.amazon.com/code/128) • Supply AWS credentials: • From command line • In configuration file© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  29. 29. Archive and Upload Example (1) Create archive: Upload to S3:s3curl.pl --id=[…] --key=[…] --put=backup_public_html_2012_03_28.tar -- http://s3.amazonaws.com/jeffbarr-backup/backup_public_html_2012_03_28.tar© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  30. 30. Archive and Upload Example (2) File uploads to S3: Run the tar and upload from a cron© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  31. 31. Archive and Upload Example (3)© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  32. 32. Archive and Upload Example (4) Encrypt backup file! Delete backups after 45 days!© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  33. 33. About This Simple Example… Store as much as you want For as long as you want 99.999999999% durability Pay only for what you store (This file: $0.04 / month) Encrypted Automatic expiration All tooling was free Restore on premises or to cloud© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  34. 34. Commercial Backup Tools • Amanda Enterprise • www.zmanda.com • Backup agents • Data to S3© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  35. 35. Backup Services (1) • www.tarsnap.com • Local encryption • Pay by byte/month© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  36. 36. Backup Services (2) FTP / SFTP / WebDAV Gateway http://www.maluke.com/gate© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  37. 37. CLOUD-BASED BACKUP OPTIONS© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  38. 38. Cloud-Based Backup Tools Everything mentioned so far still applies • Low (internal) latency makes it even faster • No charge to transfer between EC2 and S3 Cloud adds even more options: • EBS snapshot backups • RDS snapshot backups© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  39. 39. EBS Volume Review Create volume (1 GB to 1 TB) in an AZ Attach to an EC2 instance Format (mkfs) and use Backup options: • Traditional • Snapshot AFR 0.1 – 0.5% (4% for commodity drive)© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  40. 40. Create EBS Snapshot© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  41. 41. Catalog EBS Snapshots© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  42. 42. Restore EBS Volume From Snapshot© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  43. 43. EBS Consistent Snapshot Freeze file system Lock databases Take snapshot Unlock databases Unfreeze file system© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  44. 44. Schedule EBS Backups Cron and the EC2 command line tools: ec2-create-snapshot -d "Home Volume Snapshot" vol-f12a6f9c Third-party tools: • Skeddly© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  45. 45. Schedule EBS Backups - Skeddly© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  46. 46. Things We Didn’t Cover Amazon Relational Database Service • Back up your RDS (MySQL or Oracle) instances just like EBS General Database Backup • Oracle RMAN • MySQL Backup AWS Storage Gateway …Stay Tuned…© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  47. 47. © 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  48. 48. Things To Remember The cloud makes backup and recovery easy You can get started for pennies per month The cloud will scale to accommodate all of your data You retain visibility and control© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  49. 49. For More Information AWS Backup Site: http://aws.amazon.com/backup-storage/ Slides http://www.slideshare.net/AmazonWebServices Webinars http://aws.amazon.com/resources/webinars Videos http://www.youtube.com/user/AmazonWebServices Website http://aws.amazon.com Blog http://aws.typepad.com White papers http://aws.amazon.com/whitepapers© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  50. 50. Thanks For Watching!© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

×