• Save
Backup and Recovery for Linux With Amazon S3
 

Backup and Recovery for Linux With Amazon S3

on

  • 8,082 views

You will learn how to create file archives, upload them to Amazon S3, and manage permissions and lifetimes, giving you the ability to back up any amount of data and to retain it for as long as you'd ...

You will learn how to create file archives, upload them to Amazon S3, and manage permissions and lifetimes, giving you the ability to back up any amount of data and to retain it for as long as you'd like. A number of open source and commercial backup and archiving tools will be demonstrated, as time permits.

You will also learn how to use built-in AWS facilities to quickly and easily create and restore snapshots of entire disk volumes.

Statistics

Views

Total Views
8,082
Views on SlideShare
7,378
Embed Views
704

Actions

Likes
18
Downloads
1
Comments
1

3 Embeds 704

http://understeer.hatenablog.com 692
http://webcache.googleusercontent.com 10
http://www.twylah.com 2

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Backup and Recovery for Linux With Amazon S3 Backup and Recovery for Linux With Amazon S3 Presentation Transcript

  • Backup and Recovery for Linux with Amazon S3 Learn how to back up and recover your on-premise and cloud-based Linux systems. Jeff Barr (@jeffbarr) Senior Web Services Evangelist© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • What We’ll Do Today Move quickly Examine today’s backup challenges Justify use of Amazon S3 for backup Investigate backup, storage, and recovery in depth Review options for on-premise & cloud-based backup Give you enough information to get started today Address your questions© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • Today’s Backup Challenges Cheaper hardware Abundant bandwidth The paperless business Automated data collection Data analysis (Big Data) Strategic importance of data Regulatory requirements http://www.datacenterknowledge.com/archives/2012/03/20/trends-driving-data-growth-backup/© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • AMAZON S3 FOR BACKUP© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • Why Should You Back Up to Amazon S3? Primary features: • Offsite storage as a service • Fully redundant • Data stored on multiple devices in multiple facilities • Designed for 99.999999999% durability • Designed for 99.99% availability • Available in all 8 AWS Regions • Full control over data residency • Continuous price drops© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • Why Should You Back Up to Amazon S3? Additional features: • Server-side encryption • Transfer data through an encrypted (SSL) connection • ACLs (Access Control Lists) to regulate access • Object expiration • Access logging • Physical security • Location control© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • More Reasons to Back up to S3! Pay as you go model: • No sticker shock • Costs scale with usage • No hardware to buy Broad industry acceptance: • 762 billion objects stored as of Q4, 2011 • Many third-party tools and applications are available© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • BACKUP, STORAGE, AND RECOVERY IN DEPTH© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • Backup, Storage, and Recovery in Depth There are lots of things to thing about: • Backup considerations • Storage considerations • Recovery considerations© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • Backup Considerations What do you need back up? • Files • Databases • Which ones (all or selected) Full or incremental backups? How frequently? How much data? Annual growth rate?© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • More Backup Considerations How long will a backup run take? What about data consistency? How long is the backup window? The backup device: • How much does the backup device cost? • What is its useful life? • Do you have a spare one? • For each facility? • What is the cost / GB for the backup medium?© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • Storage Considerations Where do you store your backups? • Onsite – What’s your DR plan? • Offsite – How much does that cost? How long do you retain backups? • Physical space limitations? • What about media deterioration? How quickly do you need to recover? • How long (days) until you can retrieve your data?© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • More Storage Considerations How do you organize and catalog your backups? • How do you locate the right one? • What data is stored? • When was it stored? • When does it expire?© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • Retrieval Considerations Which backup do you need? How long will it take to retrieve if offsite? What if it is missing? What if the data is corrupt? What is the expected retrieval / recovery time?© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • Expiration and Rotation Cosiderations How long do you retain the backups? What happens when they expire? What is the media rotation plan? What happens to old media? How do you dispose of old backups?© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • Financial Considerations How much is your business worth? How much is your data worth? How much does down time cost you? What is your medium cost / GB? How much does your device cost? • How many do you need? What are the operational costs?© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • A TECHNICAL INTERLUDE© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • Some Technical Issues Files and databases may require distinct strategies Files • Many objects of variable size • Often date-stamped, allowing incremental backups • Open files, In-flight writes Databases • Generally large, monolithic objects • Open transactions • Locks & timeouts RAID – logical (file system) vs. physical (device) backup© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • Consistent Backups Logical consistency is a must! Operational Model • Sync all in-memory data to disk • Freeze database or filesystem • fsfreeze – Linux • mysql stop - MySQL • Run backup • Unfreeze • Continue© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • BACKUP ARCHITECTURES© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • On-Premises to Cloud Backup Architectures Transparent Archive & Upload Backup Tools Backup Services© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • Transparent Backup / File Share Shared file system Local writes mirrored to the cloud “It just works” • Save locally • Mirror to cloud© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • Transparent Backup / File Share Examples (1) Dropbox (www.dropbox.com) • Master copy of all files stored in S3 • Install Linux client on premises on cloud • Backup == copy to Dropbox directory • Recover == copy from Dropbox directory© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • Transparent Backup / File Share Examples (2) S3Fuse (http://code.google.com/p/s3fs/) • User-space file system • Store files locally, automatically uploaded to S3 Application S3Fuse Amazon S3 Bucket Kernel© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • S3FS Configuration and Use Configure Credentials: Launch S3FS: Copy files:© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • AWS Management Console / Shell© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • Archive and Upload Use existing local tools to create archive locally • tar, cpio, etc. Must have sufficient local free space Upload to S3 Manage S3 archive© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • Archive and Upload Example Requirements: • tar (standard utility) • S3 Curl (Free download at http://aws.amazon.com/code/128) • Supply AWS credentials: • From command line • In configuration file© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • Archive and Upload Example (1) Create archive: Upload to S3:s3curl.pl --id=[…] --key=[…] --put=backup_public_html_2012_03_28.tar -- http://s3.amazonaws.com/jeffbarr-backup/backup_public_html_2012_03_28.tar© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • Archive and Upload Example (2) File uploads to S3: Run the tar and upload from a cron© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • Archive and Upload Example (3)© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • Archive and Upload Example (4) Encrypt backup file! Delete backups after 45 days!© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • About This Simple Example… Store as much as you want For as long as you want 99.999999999% durability Pay only for what you store (This file: $0.04 / month) Encrypted Automatic expiration All tooling was free Restore on premises or to cloud© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • Commercial Backup Tools • Amanda Enterprise • www.zmanda.com • Backup agents • Data to S3© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • Backup Services (1) • www.tarsnap.com • Local encryption • Pay by byte/month© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • Backup Services (2) FTP / SFTP / WebDAV Gateway http://www.maluke.com/gate© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • CLOUD-BASED BACKUP OPTIONS© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • Cloud-Based Backup Tools Everything mentioned so far still applies • Low (internal) latency makes it even faster • No charge to transfer between EC2 and S3 Cloud adds even more options: • EBS snapshot backups • RDS snapshot backups© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • EBS Volume Review Create volume (1 GB to 1 TB) in an AZ Attach to an EC2 instance Format (mkfs) and use Backup options: • Traditional • Snapshot AFR 0.1 – 0.5% (4% for commodity drive)© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • Create EBS Snapshot© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • Catalog EBS Snapshots© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • Restore EBS Volume From Snapshot© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • EBS Consistent Snapshot Freeze file system Lock databases Take snapshot Unlock databases Unfreeze file system© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • Schedule EBS Backups Cron and the EC2 command line tools: ec2-create-snapshot -d "Home Volume Snapshot" vol-f12a6f9c Third-party tools: • Skeddly© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • Schedule EBS Backups - Skeddly© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • Things We Didn’t Cover Amazon Relational Database Service • Back up your RDS (MySQL or Oracle) instances just like EBS General Database Backup • Oracle RMAN • MySQL Backup AWS Storage Gateway …Stay Tuned…© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • © 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • Things To Remember The cloud makes backup and recovery easy You can get started for pennies per month The cloud will scale to accommodate all of your data You retain visibility and control© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • For More Information AWS Backup Site: http://aws.amazon.com/backup-storage/ Slides http://www.slideshare.net/AmazonWebServices Webinars http://aws.amazon.com/resources/webinars Videos http://www.youtube.com/user/AmazonWebServices Website http://aws.amazon.com Blog http://aws.typepad.com White papers http://aws.amazon.com/whitepapers© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.
  • Thanks For Watching!© 2012 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.