Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

AWS Storage Options


Published on

This is a presentation that covers AWS's storage options and how they compare to traditional on-premises storage options.

Published in: Technology, Business

AWS Storage Options

  1. 1. Storage Options in the AWS Cloud Joe Baron, Solutions Architect
  2. 2. What is Amazon Web Services?Amazon Web Services is a cloud computing platform that providesflexible, scalable, and cost-effective IT infrastructure for businesses of allsizes around the world……built on the same reliable, secure technologyplatform used to power’s globalweb properties for over a decade.
  3. 3. Storage in the Amazon Cloud Tools to access services Cross Service features Platform building blocks Infrastructure building blocks
  4. 4. Amazon EC2 Regions and Availability Zones US East (Northern Virginia) EU (Dublin) Availability Availability Zone A Zone B Availability Availability Zone A Zone B Availability Availability Zone C Zone DAmazon EC2 Regions: US East (Northern Virginia) US West (Northern California) EU (Dublin) Asia Pacific (Singapore) Asia Pacific (Tokyo)
  5. 5. Traditional StorageTraditional Storage Options Memory – caches, RAM disks Message Queues – FIFOs, MOM SAN – network-attached block devices: LUNs DAS – local block devices (disks) NAS – network attached file systems, filers Databases – structured storage (on SAN, DAS or NAS) Off-line – backup, archival, DRDiffer in Performance Durability CostOften used to form a Storage Hierarchy
  6. 6. AWS Cloud Storage OptionsMultiple Storage Options EC2 Elastic Block Storage (EBS) EC2 Local Instance Store (Ephemeral disk) Simple Storage Service (S3) Simple Queue Service (SQS) SimpleDB Relational Database Service (RDS) EC2 Database AMIsDiffer in Performance Durability CostPlus cloud attributes Elasticity Availability ScalabilityAlso often used together as a Storage Hierarchy
  7. 7. AWS Cloud Storage OptionsExamine each Description Ideal use cases Characteristics (durability, scalability, etc.) Anti-Patterns
  8. 8. Amazon Elastic Block (EBS) Storage Block-level persistent storage for EC2 instances Each EBS volume is like a hard drive on a physical server Off-instance, network-attached storage Data lifetime independent of EC2 instance lifetime Can attach multiple EBS volumes to an EC2 instance Can only attach any single EBS volume to one EC2 instance at a time Cost: $.10 per GB-month (provisioned) Ideal use case -- primary storage for: File systems Databases Raw block devices
  9. 9. Amazon EBS CharacteristicsDurable off-instance storage Mirrored (replicated) within a single Availability Zone EBS Snapshots provide enhanced durability, other benefits .1% - .5% annual failure rate (using snapshots)Optimized for random I/O, expect: About 100 IOPS for 8K random reads Up to 80 MB/sec sequential access Some level of variability – network-attached, shared resourceCan be striped using RAID 0 or LVM Aggregate IOPS Generally cannot aggregate throughput
  10. 10. Amazon EBS SnapshotsSnapshots provide point-in-time incremental backupsfor EBS volumes Saves only the data changed since the last snapshot Any single snapshot represents a complete backupSnapshots enhance durability of EBS EBS volumes “live” (are mirrored) in a single Availability Zone (AZ) Snapshots “live” in S3, are replicated across multiple AZs Durability of EBS volume is proportional to amount changed data – Snapshot frequently!Snapshots allow: Cloning an EBS volume in different AZ Sharing an EBS volume with another AWS account Resizing EBS volumes
  11. 11. Amazon EBS Anti-PatternsTemporary storage Consider EC2 Instance StorageVery high-durability storage Consider Amazon S3 S3 design point eleven 9’s annual EBS design point 99.5 to 99.9% annual durabilityStoring static web content Consider Amazon S3Storing structured data or Key-Value pairs Consider Amazon SimpleDB or Amazon RDS
  12. 12. Amazon Local Instance StorageBlock-level temporary storage for EC2 instances Also called “Ephemeral” storage or “Local drives”“In the box” storage No network variability No persistence -- All data disappears on instance termination!Cost: No additional chargeIdeal use cases: Scratch disks Temporary files, Buffers, Caches Easily replicated data – web server’s DocumentRoot or equivalent
  13. 13. Instance Storage CharacteristicsNumber and size of instance store devices variesby EC2 instance type: Larger instances have larger / more volumes c1.xlarge has four 450GB drives Micro instances have none Local storage may be available but not exposed See next slideNot optimized for random I/O EBS generally better for random I/O Instance stores generally better for sequential I/OCan be striped using RAID 0 Aggregate IOPS Aggregate throughput (bandwidth)
  14. 14. Short Detour: S3 vs EBS AMIsTwo types of EC2 AMIs and Instances: Instance store (“S3-backed”) Original EC2, boot from ephemeral storage Can Start and Terminate only All data is ephemeral (unless separate EBS volume attached) EBS root (“boot from EBS”) Newer, boot from EBS volume Can Start, Stop, Create Image, and Terminate Ephemeral storage still available, but not exposed by default Use instance Block Device Mapping (command line and API only)Additional EBS volumes can be attached toeither type
  15. 15. Amazon Instance Store Anti-Patterns Persistent storage Consider EBS Database storage Generally need persistence beyond lifecycle of single EC2 instance Consider EBS Shared storage Local instance stores cannot be moved / shared Consider EBS Snapshots Need point-in-time backups, shareability Consider EBS
  16. 16. Amazon Simple Storage Service (S3) Scalable and durable data storage in the cloud Read / write / delete objects (files) from 1 Byte up to 5TBs Concurrent reads / writes to single S3 bucket or object Store nearly any kind of data Pay-as-you-go tiered pricing: $0.14/GB (first TB) to $0.055/GB (over 5PB) Plus data transfer and requests Ideal use cases Static web content – often used with CloudFront CDN Source and working storage for large-scale computation or analytics Backup, archival, and DR storage
  17. 17. Amazon S3 CharacteristicsTwo layer hierarchy: Buckets and ObjectsEvery object has a unique URLSimple Get, Put, Delete API using HTTPSupports ACLs & Access policies Versioning and Reduced Redundancy StorageExtreme durability Automatic replication to multiple devices and facilities in the selected Region Design point is eleven 9’s durability and four 9’s availabilityScale Unlimited number of objects per bucket Web-scale concurrent read / writes
  18. 18. Amazon S3 Anti-PatternsFile systems S3 is an object store, not a POSIX file system Can emulate a folder/file hierarchy For a true filesystem, use EBSStructured data with query S3 does not support query Must know the bucket name and the key Use in conjunction with SimpleDB or databaseRapidly-changing, fine-grained changes S3 generally reads / writes whole files “Web-like” rather than “disk-like” latencies Use EBS for fine-grained changes, lower latencies
  19. 19. Amazon CloudFrontEasy-to-use Content Delivery Network (CDN)Same pay-as-you-go pricing as all of AWS No up-front contracts No long-term commitments Self-service sign-upOften uses Amazon S3 as the origin store Also can use non-S3 origins, such as web server on EC2Worldwide network of edge locations Seattle, Palo Alto, Los Angeles, St. Louis, Dallas, Ashburn, Newark, New York, Miami Amsterdam, Dublin, Frankfurt, London Hong Kong, Tokyo, Singapore
  20. 20. Amazon Simple Queue Service (SQS) Reliable, highly-scalable hosted message queuing service SQS is storage? I thought this was an asynchronous communication protocol… SQS provides temporary storage (and delivery) of short messages Avoids use of other storage, such as temp files Cost: First 100K requests free, $.01 per 10K Ideal use case: “Software glue” to enable loose coupling Store and move data between servers or application components Often used with data stored in S3
  21. 21. Amazon SQSLoose coupling, Producer-Consumer modelScale up / down Producers and Consumers asneeded
  22. 22. Amazon SQS CharacteristicsSQS provides Durable but Temporary storage Messages must be text-only Messages stay in queue for up to 14 daysHighly Reliable Messages are stored redundantly on multiple servers and data centers in a RegionHighly Scalable Unlimited number of clients reading / writing an Unlimited number of messagesNot designed to maximize single-threadperformance 5 - 50 messages per second per thread Higher performance with multiple messages per call
  23. 23. Amazon SQS Anti-PatternsBinary data orLarge data (>64KB) Store the data in S3 or RDS Store a pointer to the data in SQSLong-term storage – over 14 days Consider S3 or other storage
  24. 24. Amazon SimpleDBNew approach to store, manage, and querystructured data“NoSQL” offering: Flexible spreadsheet-like database Domains, Items, Attributes, and ValuesSupports SQL-like queries using SELECTEliminates the overhead of managing a relationaldatabaseCost: First 1GB-Month and ~2M queries freeIdeal use cases: Structured, fine-grained data needing query Data needing flexible schema Metadata storage -- often used in conjunction with S3
  25. 25. Amazon SimpleDB Core database functionality for data storage and querying No schema, no data modeling, no DBAitem description color material123 Sweater Blue, Red789 Shoes Black LeatherStore:PUT (item, 123), (description, Sweater), (color, Blue), (color, Red)Query:Domain = MyStore[‘description’ = ‘Sweater’]
  26. 26. Amazon SimpleDB CharacteristicsText-based data, 10GB per domain Can “shard” larger data over multiple domainsHighly durable Data automatically replicated in multiple geographic locations in a RegionFast Near-LAN latencies for queries from EC2Flexible and low-touch “Schema-less” design, add attributes as needed Items are indexed automaticallySupports both: Eventually-consistent reads (for speed) Consistent reads (for transactional updates)
  27. 27. Amazon SimpleDB Anti-PatternsIf your application: Is tied to a traditional relational database Needs Joins and Complex transactions Needs BLObs (Binary Large Objects) support Needs Numeric data Needs very large data (>>10GB)Then consider Amazon RDS or a traditionalrelational database
  28. 28. Amazon Relational Database Service (RDS) Fully-functional relational database provided as a managed, cloud-based service. Automates: Patching Backups Failover (Multi-AZ deployment) Read Replicas Cost: $.11 to $2.60 per hour (standard deployment), plus storage and data transfer costs Ideal use cases: Any application that needs a full native capabilities of traditional relational database With minimal administrative overhead
  29. 29. Amazon RDS CharacteristicsFully-managed, tuned MySQL* database Compatible with all normal tools, drivers, & applications mysqldump, mysqlimport, etc. (* Oracle 11g coming to RDS in second quarter)Simple to deploy Make a few clicks or API calls, get a connection stringScalable Scale vertically (increase / decrease compute, increase storage 5GB to 1TB) Scale horizontally (add read replicas)Reliable Automated backups of DB and logs, restore to any point in time Multi-AZ deployments Synchronous replication and automated failover User-initiated DB Snapshots
  30. 30. Amazon RDS Anti-PatternsSimple index-and-query focused data If you don’t need joins and complex transactions, consider using SimpleDBHeavy use of BLObs, or very large BLObs Consider storing the BLObs in S3, with pointers and metadata in RDSScaling beyond RDS vertical / horizontal scalinglimits Consider using S3 and SimpleDB togetherYour application requires a specific RDBMS notsupported by RDS, orYou need complete administrative control Consider using EC2 Relational Database AMIs
  31. 31. Amazon Relational Database AMIsEC2 instances and EBS storage provide aplatform to run many relational databasesReady-to-use AMIs for Oracle, SQLServer, DB2, Informix, PostgreSQL, Sybase, Vertica, etc.Costs and license terms vary by vendorIdeal use cases: Need the full native capabilities of one particular relational database With full administrative control
  32. 32. Amazon Relational Database AMIsPerformance characteristics depend on Database software Number / size of EBS storage volumes Configuration of DB …Benchmark your application
  33. 33. Relational Database AMIs – Anti-Patterns Index-and-query focused data Consider SimpleDB Have BLObs Consider moving BLObs to S3 Need more Automation, Scaling etc. Consider RDS, SimpleDB
  34. 34. Choosing an AWS Database Solution Amazon EC2 Relational Amazon Relational Amazon SimpleDB Database AMIs Database ServiceFeatures Choose your own database Native access to a relational Schema-less data store server (IBM DB2, Microsoft database engine, with key SQL Server, MySQL, features of a relational Oracle, and more) database, such as joins or complex transactionsAdministration Exert complete Gain a managed experience Zero administrative administrative control over and offload common overhead (automatic your database administrative tasks, such handling of geo-redundant as provisioning, backup replication, index creation, database tuning)Flexibility Easy migration path Easy migration path Easy to use web service (existing code, tools, (existing code, tools, APIs applications are compatible) applications are compatible)Scalability Employ many of the Quickly scale up resources Automatic and elastic scalability features of with only an API call scaling of resources to meet Amazon EC2 request loadBackup Store snapshots to Amazon Use automatic backup High availability (multiple S3 service at an interval you copies of data for reliability specify and failover)
  35. 35. Questions?
  36. 36. Next Steps: Amazon Web Services White Papers: AWS Security: AWS Economics:   AWS Summit 2011: summit-2011 Contact me: 
  37. 37. Thank You!
  38. 38. AWS Import/ExportGet your data into AWS faster - load it onto aportable storage device and ship it to an Amazondata centerFaster than Internet transfer and more costeffective than upgrading your connectivityUse cases: data migration, offsite backup, directdata interchange, disaster recovery