Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

(STG402) Amazon EBS Deep Dive


Published on

Amazon Elastic Block Store (Amazon EBS) provides persistent block level storage volumes for use with Amazon EC2 instances. In this technical session, we conduct a detailed analysis of the differences among the three types of Amazon EBS block storage: General Purpose (SSD), Provisioned IOPS (SSD), and Magnetic. We discuss how to maximize Amazon EBS performance, with a special eye towards low-latency, high-throughput applications like databases. We discuss Amazon EBS encryption and share best practices for Amazon EBS snapshot management. Throughout, we share tips for success.

Published in: Technology
  • Be the first to comment

(STG402) Amazon EBS Deep Dive

  1. 1. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Dougal Ballantyne, HPC Solutions Architect October 7th, 2015 Amazon EBS Deep Dive STG402
  2. 2. What to Expect from the Session Amazon EBS • Volumes • Snapshots Encryption Q&A
  3. 3. Amazon EBS overview
  4. 4. For most builders AWS is get in and go! Source:
  5. 5. A “normal” hard drive
  6. 6. EBS =
  7. 7. What is Amazon EBS? • Network block storage as a service • Designed for five nines of availability • EBS volumes attach to any EC2 instance in the same Availability Zone • Provides point-in-time snapshots to Amazon S3
  8. 8. More about Amazon EBS • It’s a service! • It’s independent of Amazon EC2 • It has regional and AZ availability goals • All EBS volumes are designed for 99.999% availability • Over 2 million volumes are created per day
  9. 9. A few definitions… • IOPS: Input/output operations per second (#) • Throughput: Read/write rate to storage (MB/s) • Latency: Delay between request and completion (ms) • Capacity: Volume of data that can be stored (GB) • Block size: Size of each I/O (KB)
  10. 10. Volume management • Volume management, use tags • Delete on termination flags, can be changed anytime • It is persistent storage, it will stay around! • Every customer should have an approach, example tag scratch volumes
  11. 11. Volume Initialization • Newly created volumes • Just attach, mount and go! • Pre-warming not recommended • Volumes restored from snapshots • EBS implements lazy loading from Amazon S3 so that you can begin using the volume right away. • For performance sensitive applications we recommend reading the entire volume to accelerate the loading of data from S3 and avoid potential latency increase.
  12. 12. EBS Volume Types Magnetic General Purpose (SSD) Provisioned IOPS (SSD)
  13. 13. EBS Volume Types IOPS Baseline: 100-10,000 (3 / GiB) IOPS Burst: 30 minutes @ 3,000 Throughput: Up to 160 MB/s Latency: Single-digit ms Performance Consistency: 99% Most workloadsGeneral Purpose (SSD)
  14. 14. General Purpose (SSD) – Burst & baseline 16 KiB I/O size
  15. 15. (2) Max I/O credit per bucket is 5.4M (1) Always accumulating 3 IOPS per GiB per second (3) You can spend up to 3000 IOPS per second Understanding General Purpose (SSD) bursting Baseline performance = 3 IOPS per GiB or 100 IOPS
  16. 16. Minutes to empty a full I/O credit bucket for various volume sizes The larger the volume, the longer it takes to empty the I/O credit bucket 1 TB or larger volume will never exhaust its I/O credit bucket
  17. 17. Minutes to empty a full I/O credit bucket for various volume sizes The larger the volume, the longer it takes to empty the I/O credit bucket 1 TB or larger volume will never exhaust its I/O credit bucket
  18. 18. General Purpose (SSD) volumes example Microsoft Windows 30 GiB boot volume: Gets initial I/O credit of 5.4M Could burst for up to 30 mins @ 3000 IOPS Always accumulating 90 I/O credits per second
  19. 19. Improved instance boot time m3.medium Volume type Access time OS GP2 4:33 Windows Server 2012 Magnetic 7:16 Windows Server 2012 GP2 0:45 CentOS6 Magnetic 1:16 CentOS6 40% Reduction in access times by using General Purpose SSD
  20. 20. Database volume 1 TB PIOPS volume with 4K IOPS = $526.40 per month per volume GP2 1 TB volume with 3000 IOPS = $102.40 GP2 2 x 500 GB volume at 3K, Burst to 6K = $102.40 80% cost savings, 50% more peak I/O with General Purpose SSD
  21. 21. Guidelines for sizing General Purpose (SSD) volumes Generic boot, developer, test/dev, and web apps: Provision GB required for your applications Database apps: 1. Calculate the IOPS required in steady state 2. Perform this calculation: (steady state IOPS) ÷ 3 = GB to provision Note: I/O bursts will support: • Database load or table scan operations • Spike in I/O workload
  22. 22. EBS Volume Types IOPS: 100-20,000 (Customer Provisioned) Throughput: Up to 320 MB/s Latency: Single-digit ms Performance Consistency: 99.9% Mission Critical workloadsProvisioned IOPS (SSD)
  23. 23. Provisioned IOPS (SSD) 16 KiB I/O size
  24. 24. EBS Volume Types IOPS: typically 100, best effort Throughput: 40-90 MB/s Latency: Read 10-40ms, Write 2-10ms Best for infrequently accessed data Magnetic
  25. 25. EBS volume types - summary General Purpose (SSD) Provisioned IOPS (SSD) Magnetic Recommend use cases Boot volumes Small to med DBs Dev and test I/O-intensive workloads Large DBs Cold storage Storage media SSD-backed SSD-backed Magnetic-backed Volume size 1 GB - 16 TB 4 GB - 16 TB 1 GB - 1 TB Max IOPS per volume 10,000 IOPS 20,000 IOPS ~100 IOPS Burst < 1 TB to 3000 IOPS baseline baseline Read and write peak throughput 160 MB/s 320 MB/s ~50-90 MBps Max IOPS per node (16k) 48,000 48,000 48,000 Peak throughput node 800 MB/s 800 MB/s 800 MB/s Latency (random read) 1-2 ms 1-2 ms 20-40 ms API Name gp2 io1 standard Price* $.10/GB-month $.125/GB-month $.065/provisioned IOPS $.05/GB-month $.05/ 1M I/O
  26. 26. Why is General Purpose SSD the default? High baseline level of performance Burst to higher level of IOPS Single, capacity-based pricing dimension • Makes forecasting very easy • Eliminates sizing complexity Attractive price/GB/price/IOPS density
  27. 27. Always use General Purpose (SSD) for boot volumes
  28. 28. Migrating to General Purpose (SSD) volumes Change volume type during launch Use EBS snapshots You may be able to resize the file system Use General Purpose (SSD) sizing guide
  29. 29. Benefits of using EBS snapshots More durable than an EBS volume • Stored in Amazon S3 Incremental (space-efficient) • First snapshot is a clone • Pay only for what you use Availability Zone-independent • Clone into any AZ Can be copied efficiently across regions
  30. 30. What happens? • EBS volume is made up of blocks • Only blocks written to are marked as updated • At the point of create snapshot, we make a list of blocks to copy • Snapshots copy only updated blocks to S3 • No need to wait for snapshot to complete • Future snapshots copy only what has changed since last snapshot
  31. 31. Tagging snapshots Use tags to add metadata to snapshots: • Type (daily, weekly) • Version • Instance Id • Volume Id • Application stack
  32. 32. Tools to manage snapshots • Customers have told us these work: • Skeddly – • ec2-consistent-snapshot – snapshot/
  33. 33. EBS optimized instances • Most instance families support the EBS-optimized flag • EBS-optimized instances now support up to 4 GB/s • Drive 32,000 16K IOPS or 500 MB/s • Available by default on newer instance types • EC2 *.8xlarge instances support 10 Gb/s network • Max IOPS per node supported is ~48,000 IOPS @ 16K I/O
  34. 34. Encryption
  35. 35. Why encrypt data volumes? Security: Protects against someone who might gain unauthorized physical access to the volume Can help with internal or external compliance efforts: • Chief Information Security Officer wants encryption to protect sensitive corporate data • Third-party auditors want to see evidence that sensitive customer data is encrypted Ease of use and operating cost reduction: Unlike open-source or third-party solutions, such as Trend Micro SecureCloud, SafeNet ProtectV, etc., EBS encryption offers: • “Checkbox” encryption at no extra cost • Automated, secure key management
  36. 36. AWS Key Management Service Managed service simplifies creation, control, rotation, and use of encryption keys in your applications Integrated with AWS Server-side encryption • Amazon S3, EBS, Amazon RDS, Amazon Redshift, Amazon WorkMail, and Amazon Elastic Transcoder Integrated with Client-side encryption • AWS SDKs, S3 Encryption Client, DynamoDB Encryption Client Integrated with AWS CloudTrail to provide auditable logs for regulatory and compliance activities Available in all commercial regions except China
  37. 37. AWS Key Management Service Integrated with AWS IAM Console
  38. 38. Server-side encryption in AWS Amazon EBS
  39. 39. Your Application or AWS Service + Data Key Encrypted Data Key Encrypted Data Master Key(s) in Customer’s Account AWS Key Management Service 1. Application requests encryption key to use to encrypt data, passes reference to master key in account. 2. Client request authenticated based on master key permissions. 3. New data encryption key created - copy encrypted under master key. 4. Plaintext and encrypted data key returned to the client. 5. Plaintext data key used to encrypt data and then deleted. 6. Encrypted data key stored for later use and sent back to AWS KMS for when decryption occurs. AWS Key Management Service How Keys are Used to Protect Your Data
  40. 40. Summary Use encryption if you need it Take snapshotsSelect the right instance for your workload Select the right volume for your workload
  41. 41. Q&A
  42. 42. Remember to complete your evaluations!
  43. 43. Thank you!