Journey Through the AWS Cloud; Storage and Archiving
Upcoming SlideShare
Loading in...5
×
 

Journey Through the AWS Cloud; Storage and Archiving

on

  • 2,291 views

The third in our series of webinars, 'Journey Through the AWS Cloud', this complimentary presentation discusses the use of AWS as a storage and archive platform. We introduce some key mechanisms that ...

The third in our series of webinars, 'Journey Through the AWS Cloud', this complimentary presentation discusses the use of AWS as a storage and archive platform. We introduce some key mechanisms that will help you use AWS as a flexible deployment environment, talk about customers who are using AWS for development and test, and provide some tips and tricks to help you manage your AWS infrastructure and keep it cost effective.

Statistics

Views

Total Views
2,291
Views on SlideShare
2,223
Embed Views
68

Actions

Likes
7
Downloads
124
Comments
1

1 Embed 68

http://www.scoop.it 68

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • recording can be found here: http://www.youtube.com/watch?v=Y0fTokZDcG8
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Journey Through the AWS Cloud; Storage and Archiving Journey Through the AWS Cloud; Storage and Archiving Presentation Transcript

  • Journey through the Cloud: Storage & Archive Ryan Shuttleworth – Technical Evangelist @ryanAWS
  • Journey through the cloudCommon use cases & stepping stones into the AWS cloud Learning from customer journeys Best practices to bootstrap your projects
  • Storage & ArchiveBenefit from cloud economics with simple to implement use cases Simplify the management of data assets Eliminate technologies & processes Gain performance and reliability improvements
  • AgendaWhy AWS for storage & archiveAWS fundamental servicesStorage & archive – examples & patternsWhere to go next
  • Why AWS for storage & archive?
  • Storage & Archive AWS is used in a variety of ways…Powers applications that allows customers to access historical Store its vast repository of music to stock price information feed to over 15 million active users Estimates it has saved $500,000 Digital assets and usage data behind in storage expenditures and cut publication sites and mobileits disk storage array costs in half applications
  • Business & technical drivers You might be able to: Reduce costs Reduce on-premiseSlash storage & archive budgets Eliminate on premise equipment to manage archives Change processes Remove aging technologiesRemove the need to do capacity Eliminate tape for backup and archive planning
  • Business & technical drivers You might be able to: Reduce costs Reduce on-premise Reduce CAPEX while dramaticallySlash storage & archive budgets by Eliminate on premise equipment to increasing scalability up to 50% manage archivesEliminate the need for secondary sites Change processes Remove aging technologies Remove the need to do capacity Eliminate tape for backup and archive planning
  • Business & technical drivers You might be able to: Reduce costs Reduce on-premise Reduce CAPEX while dramatically Eliminate 30%+ of your storageSlash storage & archive budgets by Eliminate on premise equipment to increasing scalability footprint up to 50% manage archivesEliminate the need for secondary Consolidate on-premise and sites augment with cloud Change processes Remove aging technologies Remove the need to do capacity Eliminate tape for backup and archive planning
  • Business & technical drivers You might be able to: Reduce costs Reduce on-premise Reduce CAPEX while dramatically Eliminate 30%+ of your storageSlash storage & archive budgets by Eliminate on premise equipment to increasing scalability footprint up to 50% manage archivesEliminate the need for secondary Consolidate on-premise and sites augment with cloud Change processes Remove aging technologies Remove the need to do capacity Eliminate capacity planning Eliminate tape for backup and archive planning Eliminate provisioning for peak demand
  • Business & technical drivers You might be able to: Reduce costs Reduce on-premise Reduce CAPEX while dramatically Eliminate 30%+ of your storageSlash storage & archive budgets by Eliminate on premise equipment to increasing scalability footprint up to 50% manage archivesEliminate the need for secondary Consolidate on-premise and sites augment with cloud Change processes Remove aging technologies Remove the need to do capacity Eliminate capacity planning Eliminate tape for backup and planning Remove tape archives Eliminate provisioning for peak Cycle out aging disk arrays demand
  • AWS fundamental services
  • Fundamental Storage Options Elastic Block Store, S3 and Glacier Elastic Block Store Simple Storage Service GlacierHigh performance block storage device Highly scalable object storage Long term object archive 1GB to 1TB in size 1 byte to 5TB in size Extremely low cost per gigabyte Mount as drives to instances with 99.999999999% durability 99.999999999% durability snapshot/cloning functionalities
  • Fundamental Storage Options Elastic Block Store, S3 and Glacier Elastic Block Store Simple Storage Service GlacierHigh performance block storage device Highly scalable object storage Long term object archive 1GB to 1TB in size 1 byte to 5TB in size Extremely low cost per gigabyte Mount as drives to instances with 99.999999999% durability 99.999999999% durability snapshot/cloning functionalities Very fast Fast web object Slow, rare access ‘instance’ disks storage
  • Fundamental Storage Options Elastic Block Store, S3 and Glacier Elastic Block Store Simple Storage Service GlacierHigh performance block storage device Highly scalable object storage Long term object archive 1GB to 1TB in size 1 byte to 5TB in size Extremely low cost per gigabyte Mount as drives to instances with 99.999999999% durability 99.999999999% durability snapshot/cloning functionalities
  • Fundamental Storage Options Elastic Block Store, S3 and Glacier Elastic Block Store Simple Storage Service GlacierHigh performance block storage device Persistent storage Highly scalablelifetime is independent of any particular EC2 instance. archive Volume object storage Long term object 1GB to 1TB in size 1 byte to 5TB in size Extremely low cost per gigabyte Mount as drives to instances with General purpose snapshot/cloning functionalities 99.999999999% durability 99.999999999% durability Raw, unformatted, block device. Use from Linux, Solaris or Windows. High performance Equal to or better than local EC2 drive. Provisioned IOPS High reliability Built-in redundancy within availability zone. AFR (Annual Failure Rate) between 0.1% and 1%. Scalable Volume sizes ranging from 1 GB to 1 TB. IMAGE Easy Easy to create, attach, back up, restore, and delete volumes.
  • Fundamental Storage Options Elastic Block Store, S3 and Glacier Elastic Block Store Simple Storage Service GlacierHigh performance block storage device Highly scalable object storage Long term object archive 1GB to 1TB in size Paradigm in size 1 byte to 5TB Extremely low cost per gigabyte File system Mount as drives to instances with snapshot/cloning functionalities Very, very fast (~100 IOPs per durability 99.999999999% durability Performance 99.999999999% volume) Redundancy Within data center Security Visible only to your EC2 instances . Pricing $0.10/GB/Mo. allocated Access from the Net? No Typical use case IMAGE It’s a disk drive
  • Fundamental Storage Options Elastic Block Store, S3 and Glacier Elastic Block Store Simple Storage Service GlacierHigh performance block storage device Highly scalable object storage Long term object archive 1GB to 1TB in size 1 byte to 5TB in size Extremely low cost per gigabyte Mount as drives to instances with 99.999999999% durability 99.999999999% durability snapshot/cloning functionalities
  • Fundamental Storage Options Elastic Block Store, S3 and Glacier ElasticAmazon S3 Block Store Simple Storage Service Glacier Highly scalable object storageHigh performance block Service device Simple Storage storage Paradigm term object archive Long Object store 1 byte to 5TB in size 1GB to 1TB in size Highly scalable Extremely low cost per gigabyte 99.999999999% durability Performance Very fast Mount data storage in-the-cloud with as drives to instances 99.999999999% durability snapshot/cloning functionalities Redundancy Across data centers Programmatic access via web services API Security Public Key / Private Key Is a Web Store Pricing $0.125/GB/month stored Not a file system Access from Yes Optimized for WORM Eventually consistent the Net? IMAGE Typical use Write once, read many Fast, highly available case Durable Economical
  • Fundamental Storage Options Elastic Block Store, S3 and Glacier Elastic Block Store Simple Storage Service GlacierHigh performance block storage device Highly scalable object storage Long term object archive 1GB to 1TB in size 1 byte to 5TB in size Extremely low cost per gigabyte Mount as drives to instances with 99.999999999% durability 99.999999999% durability snapshot/cloning functionalities
  • Fundamental Storage Options Elastic Block Store, S3 and Glacier Elastic Block Store Archive Backup Simple Storage Service DR GlacierHigh performance block storage device Highly scalable object storage Long term object archive Data accessed ~>10% Snapshots 1GB to 1TB in size 1 byte to 5TB in size Rapid RTO Extremely low cost per gigabyte / monthAmazon as drives to instances with Mount 99.999999999% durability Shorter term data99.999999999% durability S3 Expiration policies snapshot/cloning functionalities backup with rapid 11 9s durability RTOAmazon Lower cost when 11 9s Lower cost Lower costS3 RRS not required Long term archiving Use policies to move Retain write once readAmazon Infrequent data cold backup data for never copy in case of worstGlacier access (~<10% long term retention case scenario data/month)
  • Putting it together Storage & archive
  • Use case journey On-premise On-instance Object level Long term Locallyaccessible file systems Workloadswith local data
  • Use case journey On-premise On-instance Object level Long term Locallyaccessible file systems AWS Workloadswith local data
  • Use case journey On-premise On-instance Object level Long term Locally EC2 based Data System imagesaccessible file applications distribution Database systems DR Durable media backups Workloads deployments storage Data archiveswith local data
  • Use case journey On-premise On-instance Object level Long term Locally EC2 based Data System imagesaccessible file applications distribution Database systems DR Durable media backups Workloads deployments storage Data archiveswith local data High IO High IO performance Good Very low price performance Provisioned IOPS performance High durability High network Backup & Restore High durability Slow access performance Scalability
  • Use case journey On-premise On-instance Object level Long term Locally EC2 based Data System imagesaccessible file applications distribution Database systems DR Durable media backups Workloads deployments storage Data archiveswith local data High IO High IO performance Good Very low price performance Provisioned IOPS performance High durability High network Backup & Restore High durability Slow access performance Scalability
  • Use case journey On-premise On-instance Object level Long term Locally 1 EC2 based Data System imagesaccessible file applications distribution Database systems DR Durable media backups Workloads deployments storage Data archiveswith local data Getting data into the cloud High IO High IO performance Good Very low price performance Provisioned IOPS performance High durability High network Backup & Restore High durability Slow access performance Scalability
  • Getting data into the cloud Direct connect, import/export and storage gatewayAWS Direct Connect AWS Import/Export Amazon Storage GatewayDedicated bandwidth between you Physical transfer of media into and Shrink-wrapped gateway for volume site and AWS out of AWS synchronization
  • Getting data into the cloud Storage gatewayRestorationfrom snapshots Snapshot of local volumes
  • “Amazon Web Services and AWS Storage Gateway are great assets that help us scale fast, store data in an ultra-secure environment, spend more time on productdevelopment (rather than disaster recovery & backup), and achieve faster time-to- market with minimal investment… …By using AWS Storage Gateway, we wentto just hours instead of days to restore from backup.” Craig Link, Glympse Technology Manager
  • Use case journey On-premise On-instance Object level Long term Locally 1 EC2 based Data System imagesaccessible file applications distribution Database systems DR Durable media backups Workloads deployments storage Data archiveswith local data Getting data into the cloud High IO High IO performance Good Very low price performance Provisioned IOPS performance High durability High network Backup & Restore High durability Slow access performance Scalability
  • Use case journey On-premise On-instance Object level Long term Locally 1 EC2 based Data System imagesaccessible file applications and distribution Disks Database systems data DR Durable media backups Workloads deployments storage Data archiveswith local data Getting data into 2 the cloud High IO High IO performance Good Very low price performance Provisioned IOPS performance High durability High network Backup & Restore High durability Slow access performance Scalability
  • EBS attached volume…
  • …transportable between instances…
  • …many volumes per instance
  • Package data on EBS boot instances from EBS volume
  • Curiosity
  • Curiosity The mars.jpl.nasa.gov website is based on the open-source Content Management System (CMS) Railo, running on Amazon EC2 Shared storage for Railo is provided by Amazon EC2 instances running Gluster on a pool of Amazon Elastic Block Store (EBS) volumes for consistently high performance disk I/O.
  • Use case journey On-premise On-instance Object level Long term Locally 1 EC2 based Data System imagesaccessible file applications and distribution Disks Database systems data DR Durable media backups Workloads deployments storage Data archiveswith local data Getting data into 2 the cloud High IO High IO performance Good Very low price performance Provisioned IOPS performance High durability High network Backup & Restore High durability Slow access performance Scalability
  • Use case journey On-premise On-instance Object level Long term Locally 1 EC2 based Data System imagesaccessible file applications and distribution Disks Database systems data DR Durable media backups Workloads deployments storage Data archiveswith local data Getting data into 2 the cloud High IO High IO performance Good Very low price performance Provisioned IOPS performance High durability High network Backup & Restore High durability Slow access Database performance as a service 3 Scalability
  • Database services RDS, SimpleDB, DynamoDBRelational Database SimpleDB DynamoDB Schemaless NoSQL, Schemaless, Service Smaller datasets Provisioned throughput Fully managed database database (MySQL, Oracle, MSSQL)
  • Traditional self managed database…
  • …with Relational Database Service
  • Backup management with RDS
  • Restoration…
  • …and switchover
  • Database services DynamoDB & NoSQL Requirement: predictable, consistent performancePerformance Scalability
  • Database services DynamoDB & NoSQL Requirement: predictable, consistent performancePerformance Reality: performance degrades with scale Scalability
  • Database services DynamoDB & NoSQL Requirement: predictable, consistent performancePerformance Hardware provisioning Data sharding Data caching Reality: performance Cluster management degrades with scale Fault management Scalability
  • Traditionally…
  • DynamoDB…
  • DynamoDB Feature DetailsProvisioned throughput NoSQL Provisioned Dial up or down provisioned read/write throughput capacitydatabase Predictable Average single digit millisecond latenciesFast, predictable performance performance from SSD backed infrastructureFully distributed, fault tolerant Strong consistency Be sure you are reading the most up toarchitecture date values Fault tolerant Data replicated across availability zones Monitoring Integrated to Cloud Watch Secure Integrates with AWS Identity and Access Management (IAM) Elastic Integrates with Elastic MapReduce for MapReduce complex analytics on large datasets
  • “AWS gave us the flexibility to bring a massive amount of capacity online in a short period of DynamoDB: time and allowed us to do so in an operationally over 500,000 writes per straightforward way. second AWS is now Shazam’s cloud provider of choice,” Amazon EMR:more than 1 million writes Jason Titus, per second CTO
  • Use case journey On-premise On-instance Object level Long term Locally 1 EC2 based Data System imagesaccessible file applications and distribution Disks Database systems data DR Durable media backups Workloads deployments storage Data archiveswith local data Getting data into 2 the cloud High IO High IO performance Good Very low price performance Provisioned IOPS performance High durability High network Backup & Restore High durability Slow access Database performance as a service 3 Scalability
  • Use case journey On-premise On-instance Object level Long term Locally 1 EC2 based Data System imagesaccessible file applications and distribution Disks Database systems data DR Durable media backups Workloads deployments storage Data archiveswith local data Getting data into 2 4 the cloud High IO High IO performance Good Very low price performance Provisioned IOPS performance High durability High network Backup & Restore High durability Slow access Database performance Object as a service 3 Scalability serving and storage
  • Web accessible S3 storage…
  • You put in it S3AWS stores with 99.999999999% durability
  • Highly scalable web access to objects You put in it S3AWS stores with 99.999999999% durabilityMultiple redundantcopies in a region
  • …not so simple CloudFront integration Logging Access control lists Requestor Pays Server side encryption Signed URLs Object expiry Bittorrent support Website support IAM Versioning Meta-data Browser Upload to S3 Multi-object delete
  • “Spotify needed a storage solution that could scale very quickly without incurringlong lead times for upgrades. This led us to cloud storage, and in that market, Amazon Simple Storage Service (Amazon S3) is the most mature large-scale product. Amazon S3 gives us confidence in ourability to expand storage quickly while also providing high data durability.” Emil Fredriksson, Operations Director
  • Need to store ‘something’? S3 is a foundation building block
  • Use case journey On-premise On-instance Object level Long term Locally 1 EC2 based Data System imagesaccessible file applications and distribution Disks Database systems data DR Durable media backups Workloads deployments storage Data archiveswith local data Getting data into 2 4 the cloud High IO High IO performance Good Very low price performance Provisioned IOPS performance High durability High network Backup & Restore High durability Slow access Database performance Object as a service 3 Scalability serving and storage
  • Use case journey On-premise On-instance Object level Long term Locally 1 EC2 based Data Cold System imagesaccessible file applications and distribution storage & Database Disks systems data DR Durable mediaarchiving backups Workloads deployments storage Data archiveswith local data Getting data into 2 4 5 the cloud High IO High IO performance Good Very low price performance Provisioned IOPS performance High durability High network Backup & Restore High durability Slow access Database performance Object as a service 3 Scalability serving and storage
  • Reliable and cheap storage of data for: Data with long retention periods Multi-PB, infrequently accessed Glacier data sets Long term cold storageFrom $0.01 per GB/Month99.999999999% durability
  • Glacier allows you to cost-effectively and securely storeOffsite archive enterprise data offsite, making it simple, inexpensive and safe to retain archived data for as long as desired. Common use cases include enterprise data, media assets, and research and scientific data
  • Glacier allows you to cost-effectively and securely storeOffsite archive enterprise data offsite, making it simple, inexpensive and safe to retain archived data for as long as desired. Common use cases include enterprise data, media assets, and research and scientific data Libraries, historical societies, non-profit organizations andDigital preservation governments are increasing their efforts to preserve valuable but aging digital content such as websites, software source code, video games, user-generated content and other digital artifacts
  • Glacier allows you to cost-effectively and securely storeOffsite archive enterprise data offsite, making it simple, inexpensive and safe to retain archived data for as long as desired. Common use cases include enterprise data, media assets, and research and scientific data Libraries, historical societies, non-profit organizations andDigital preservation governments are increasing their efforts to preserve valuable but aging digital content such as websites, software source code, video games, user-generated content and other digital artifacts Amazon Glacier is cost competitive, even at scale, andTape replacement eliminates pain points like capacity planning, capital budgeting and investments, media formats, hardware refreshes, and off-site storage costs, shipping and retrieving
  • “Every day our genome sequencers produceterabytes of data. As our company moves into the clinical space, we face a legal requirement to archive patient data for years that would drastically raise the cost of storage. Thanks to Amazon Glacier’s secure and scalablesolution, we will be able to provide cost-effective,long-term storage and thereby eliminate a barrier to providing whole genome sequencing for medical treatment of cancer and other genetic diseases.” Keith Raffel, Senior Vice President and Chief Commercial Officer, Complete Genomics
  • “An organization like ours thinks in centuries when it comes to content retention, and long term preservation of our Master Archives is a critical part our mission here at NYPR. Storing these core assets on traditional mediasuch as local disk and off-site tape exposes us to corruption and even outright-loss of data. We are excited to move our archives to Amazon Glacier, which will be a better long-term solution.” Steve Shultis, CTO, New York Public Radio
  • Where to go next
  • Rich partner ecosystem Technology and services organisations
  • http://aws.typepad.comhttp://aws.amazon.com/whitepapers
  • Summary
  • A wide range of use cases AWS supports archive & storage across many application types…Customer facing online App Storage Big Data storage Smartphone apps Log files Files, photos, downloads Facebook Apps Customer Data Streaming Media File Sharing Usage Data EC2 Instance Storage Backup and Archive On Premise Storage File Storage Data Retention NAS Storage Block Storage Tape Replacement SAN Storage Usage Data Offsite Backup Offsite Backups
  • AWS is a cost effective place to manage digital assetsThere are many options for storing data based upon requirementsOn-premise data assets can integrated with cloud servicesAWS storage and archive revolutionizes the technology behind long term data
  • aws.amazon.com