AWS Summit 2011: Data Storage Solutions in the AWS Cloud

Storage Solutions in the AWS Cloud Miles Ward – Enterprise Solutions Architect

Traditional Storage On-Premise Storage Options SAN – network-attached block devices: LUNs DAS – local block devices (disks) NAS – network attached file systems, filers Differ in, Performance, Durability, and Cost Often used to form a Storage Hierarchy

Cloud Storage AWS offers multiple cloud-based storage options. • Amazon S3 (Simple Storage Service) • Amazon EBS (Elastic Block Storage) • Amazon EC2 Local Instance Store (Ephemeral volumes) Each has a unique combination of performance, durability, cost, and interface Often used to form a Storage Hierarchy

Other Storage options? Amazon S3, EBS, and EC2 Instance Store volumes are the basic storage building blocks, but for more specialized uses, or managed resources, we have other options as well: Cloudfront Amazon RDS SimpleDB SQS Relational Databases on EC2

Storage by the Numbers S3 EBS LocalScale Unlimited Storage 1TB per volume 147 – 1690GB 5TB objects ~44TB per server 1-5 volumes 100 name spaces (buckets)Durability 99.999999999% (standard) .1-.5% AFR Ephemeral design 99.99% requires application- (RRS) level redundancyLatency High Low Very LowManagement Zero Maintenance Snapshots, RAID, Manual Redundancy ConsistencyLocality Region-Centric AZ-Specific In-machineInterface REST or SOAP Block Device Block Device Web Services

S3 (Simple Storage Service) Pay for exactly what you use Highly Durable Limitlessly scalable Natively On-Line

S3 Best Practices Replicate Metadata Parallel IO for max speed  Multipart Upload Sorted Writes Resource-level IAM permissions

S3 Use Cases and Users Media Serving Media Sharing File Storage / Backup Static Content Big Data

Amazon EBS Pay for what you provision POSIX compliant network attached block device Native Redundancy and Write Cache; optimized for random writes and reads.

EBS Best Practices Pre-warming Snapshots Striping (RAID 0) Block-device-mapping

EBS Use Cases and Users Application hosting Envoy Media Group Case Stud Storage for multimedia content delivery on Amazon W Data Processing Introd Establi that of Database media and po channe implem Large Data Envoy moved their IT infrastructure from a traditional co-location advantage of -as-you- to share, and promptly delivered, Envoy selected Gluster in conce Challenges Envoy currently hosts their entire web presence on Amazon Web which has helped them meet the needs of their growing business base. One of their biggest challenges was ensuring the IT infrastru to respond to the highly dynamic nature of demand such as when launch and then grow exponentially. From a data storage standpo needed a solution that integrated easily with AWS and provided h Storing the data locally on EC2 instances is not suffcient since that persistent and would be lost if a server reboots. Amazon EBS volu to one instance at a time a problem if your business requires hu needed to ensure their data was easily shared.

EC2 Instance Store or Ephemeral Volumes Free with your EC2 Instance Zero Network Overhead; local, direct attached resource. High performance for sequential reads and writes Temporary; goes away when your instance does

Instance Store Best Practices App-level replication Striping (RAID 0, 1) Block-device-mapping

Instance Store Use Cases and Users Stateless Web Hosts Transcoding Caching HPC DNFS

Better Together: Cross-system patterns EBS <-> S3: Snapshots S3 <-> SimpleDB: meta-data store S3 <-> Local Storage: Caching

Short Detour: S3 vs EBS AMIs Two types of EC2 AMIs and Instances: Instance store (“S3-backed”) Original EC2, boot from ephemeral storage Can Start and Terminate only All data is ephemeral (unless separate EBS volume attached) EBS root (“boot from EBS”) Newer, boot from EBS volume Can Start, Stop, Create Image, and Terminate Ephemeral storage still available, but not exposed by default Use instance Block Device Mapping (command line and API only) Additional EBS volumes can be attached to either type

So, which should I choose? Might be all of them! S3: Static Online Content, Media Serving, Backups, Big Data, Centralized Resources EBS: Applications that require POSIX, Databases, Data Warehouses Ephemeral: Cache/Swap/Temp, Applications that require minimum latency

Test! It costs pennies to evaluate these technologies against your use-case, and making the right choice could save you millions.

How do I get it to the Cloud? AWS Import/Export Faster than Internet transfer and more cost effective than upgrading your connectivity Use cases: data migration, offsite backup, direct data interchange, disaster recovery WAN Optimization Asperatech, Riverbed, Tsunami-UDP, etc

Read the originals! Storage Options in the AWS Cloud:https://media.amazonwebservices.com/AWS_Storage_Options.pdf Storage Use Cases in the AWS Cloud:https://media.amazonwebservices.com/AWS_Storage_Use_Cases.pdf

Thanks! http://aws.amazon.com

AWS Summit 2011: Data Storage Solutions in the AWS Cloud

by Amazon Web Services on Jul 05, 2011

Accessibility

Categories

Upload Details

Usage Rights

Statistics

AWS Summit 2011: Data Storage Solutions in the AWS Cloud Presentation Transcript