Amazon Elastic Block Store (Amazon EBS) provides persistent block level storage volumes for use with Amazon EC2 instances. In this technical session, we discuss how to maximize Amazon EBS performance, with a special eye toward low-latency, high-throughput applications like databases. We explain how to monitor your application and share real-world examples.

1. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Marc Olson, Senior Software Engineer
October 2015
Amazon Elastic Block Store
Designing for Performance
STG403

2. Amazon EBS overview

3. EBS =

4. What is EBS?
• Network block storage as a service
• EBS volumes attach to any Amazon EC2 instance in the
same Availability Zone
• Designed for five nines of availability
• 2 million volumes created every day

5. EBS volume types
Magnetic General purpose (SSD) Provisioned IOPS (SSD)

6. EBS volume types
IOPS: Typically 100, best effort
Throughput: 40-90 MB/s
Latency: Read 10-40ms, Write 2-10ms
Best for infrequently accessed data
Magnetic

7. EBS volume types
IOPS Baseline: 100-10,000 (3 / GiB)
IOPS Burst: 30 minutes @ 3,000
Throughput: Up to 160 MB/s
Latency: Single-digit ms
Performance consistency: 99%
Most workloadsGeneral purpose (SSD)

8. EBS volume types
IOPS: 100-20,000 (customer provisioned)
Throughput: Up to 320 MB/s
Latency: Single-digit ms
Performance consistency: 99.9%
Mission Critical workloadsProvisioned IOPS (SSD)

9. Performance

10. Queuing theory
Little’s law is the foundation for performance tuning theory
• Mathematically proven by John Little in 1961
𝑾 =
𝑳
𝑨
W = Wait time = average wait time per request
L = Queue length = average number of requests waiting
A = Arrival rate = the rate of requests arriving
EBS performance is related to this law

11. Performance optimization is measured by:
IOPS: Read/write I/O rate (IOPS)
Latency: Time between I/O submission
and completion (ms)
Throughput: Read/write transfer rate
(MB/s); throughput = IOPS X I/O size

12. Key components to performance
EC2 instance
I/O
EBS
Network link

13. A day in the life of an I/O

14. A day in the life: I/O
All I/O must pass through I/O domain
Requires “grant mapping” prior to 3.8.0
Grant mappings are expensive operations due to TLB flushes
EBS
Grant
mappingread(fd, buffer,
BLOCK_SIZE)
I/O domainInstance

15. A day in the life: I/O (continued)
Responses
Requests
Instance I/O domain
READ
8KB @ 1234
Request queue is a single memory page
Each I/O request has 11 grant references (4KiB/reference)
Maximum data in queue = 1408 KiB

16. 3.8.0+ Kernels – Persistent grants
Grant mappings are setup in a pool once
Data is copied in and out of the grant pool
Copying is significantly faster than remapping
EBS
Grant
poolread(fd, buffer,
BLOCK_SIZE)
I/O domainInstance

17. 3.8.0+ Kernels – Indirect grants
Responses
Requests
Instance
READ
8KB @ 1234
I/O domain
Each I/O request has grant references that contain grant references
Maximum data in queue = 4096 KiB (default)

18. Instance I/O: Before 3.8.0
0 1 2 3 4 5 30 31
128KiB
44KiB 44KiB 40KiB

19. Instance I/O: Linux 3.8.0+
0 1 2 3 4 5 30 31
128KiB

20. Tip: Use 3.8+ kernel
Amazon Linux 2013.09 or later
Ubuntu 14.04 or later
RHEL7 or later
Etc.

21. Queue depth
An I/O operation
EBS
After it’s gone, it’s gone
EC2
Queue depth is the pending I/O for a volume

22. Workload/
software
Typical block
size
Random/
Seq?
Max EBS @ 500
MB/s instances
Max EBS @
1 GB/s instances
Max EBS @ 10 GB/s
instances
Oracle DB Configurable:2 KB –
16 KB
Default 8 KB
random ~7,800 IOPS ~15,600 IOPS ~48,000 IOPS
Microsoft SQL
Server
8 KB w/ 64 KB
extents
random ~7,800 IOPS ~15,600 IOPS ~48,000 IOPS
MySQL 16 KB random ~4,000 IOPS ~7,800 IOPS ~48,000 IOPS
PostgreSQL 8 KB random ~7,800 IOPS ~15,600 IOPS ~48,000 IOPS
MongoDB 4 KB sequential ~15,600 IOPS ~31,000 IOPS ~48,000 IOPS
Apache
Cassandra
4 KB random ~15,600 IOPS ~31,000 IOPS ~48,000 IOPS
GlusterFS 128 KB sequential ~500 IOPS ~1,000 IOPS ~6,000 IOPS
Cheat sheet: Sample storage workloads on AWS

23. Example workload
Transaction (OLTP)
Examples: eCommerce website, metadata storage
Benchmark: MySQL + sysbench

24. Tip: Workload
Where possible, use real
production workloads for
performance testing

25. Baseline configuration
Availability Zone: US West (Oregon)
Instance type: m2.4xlarge
vCPU: 8
Memory: 68.4GiB
EBS-optimized
Data volume: 500GiB EBS magnetic
OS: Amazon Linux 2015.03.1

26. Optimization: Increase parallelism
MySQL threads
Transactions(n)
Baseline
2 n

27. Tip: Parallelism
Increase parallelism of your system

28. Key components to performance
EC2 instance
I/O
EBS
Network link

29. m2.4xlarge
CPU: Intel Xeon
vCPU: 8
Memory: 68.4 GiB
Price: $0.98/hour*
Instance selection
r3.2xlarge
CPU: Intel Xeon E5-2670 v2
vCPU: 8
Memory: 61 GiB
Enhanced networking
Price: $0.70/hour*
* All pricing from US West (Oregon)

30. EBS optimized instances
• Most instance families support the EBS-optimized flag
• EBS-optimized instances now support up to 4 Gb/s
• Drive 32,000 16K IOPS or 500 MB/s
• Available by default on newer instance types
• EC2 *.8xlarge instances support 10 Gb/s network
• Max IOPS per node supported is ~48,000 IOPS @ 16K I/O

31. Tip: Use EBS-optimized instances
Use EBS-optimized instances
for consistent EBS performance

32. Updated configuration: Instance type
Availability Zone: US West (Oregon)
Instance type: r3.2xlarge
vCPU: 8
Memory: 61 GiB
EBS-optimized
EBS volume: 500GiB magnetic
OS: Amazon Linux 2015.03.1

33. 25%
Optimization: Current generation instances
MySQL threads
Transactions(n)
Baseline
r3.2xlarge
2 n

34. Tip: Instance selection
Use the right instance family for
your workload
Use current generation instances

35. Key components to performance
EC2 instance
I/O
EBS
Network link

36. Volume selection
EBS magnetic
Latency:
Read: 10-40ms
Write: 2-10ms
SSD backed
Latency:
Read/Write: Single-digit ms

37. File systems
Use a modern, journaled filesystem
ext4, xfs, etc.
Ensure partitions are aligned on 4KiB boundaries

38. Pre-warming

39. Volume initialization
Newly created volumes
• Just attach, mount, and go!
• Pre-warming is no longer recommended
Volumes restored from snapshots
• You can use your volume right away
• Accelerate data loading by reading

40. Updated configuration: EBS volumes
Availability Zone: US West (Oregon)
Instance type: r3.2xlarge
vCPU: 8
Memory: 61 GiB
EBS-optimized
Boot volume: 8 GiB – EBS general purpose
Data volume: 500 GiB – EBS general purpose
OS: Amazon Linux 2015.03.1

41. Optimization: Volume selection
Transactions(n)
19% 50%
MySQL threads
Baseline
r3.2xlarge
r3.2xlarge gp2
2 n

42. Tip: Volume selection
Use SSD backed volumes when
performance matters

43. EBS IOPS vs. Throughput
20,000 IOPS
PIOPS volume
20,000 IOPS
320 MB/s
throughput
You can achieve 20,000 IOPS when
driving smaller I/O operations
You can achieve up to 320 MB/s
when driving larger I/O operations

44. EBS IOPS vs. Throughput
8,000 IOPS
PIOPS volume
8,000 IOPS
320 MB/s
throughput
8,000 x 8 KB = 64 MB/s
8,000 x 16 KB = 128 MB/s
8,000 x 32 KB = 256 MB/s
16,000 x 8 KB = 128 MB/s
8,000 x 64 KB=512 MB/s
5,000 x 64 KB = 320 MB/s

45. Striping
Increases performance, or capacity, or both
Don’t mix volume types
Typically RAID 0 or LVM stripe
Avoid RAID for redundancy
EBS
EC2

46. Striping: Snapshots
Quiesce I/O
1. Database: FLUSH and LOCK tables
2. Filesystem: sync and fsfreeze
3. EBS: snapshot all volumes
When snapshot API returns,
it is safe to resume

47. EBS-optimized instance
Four key components: Balanced
EC2
A “boatload” of I/O
Right-sized EBS

48. Monitoring

49. Amazon CloudWatch
Important Amazon CloudWatch metrics:
• IOPS and bandwidth
• Latency
• Queue depth
All EBS metrics are prefixed with “Volume”

50. CloudWatch: Instance bandwidth
m4.2xlarge
Instance: 128MB/s
m4.4xlarge
Instance: 256MB/s
m4.10xlarge
Volume: 320MB/s

51. Iostat
Don’t trust svctm or %util on parallel storage devices and systems!

52. Perfmon

53. Summary
Monitor
performance
Increase I/O
parallelism
Select the right
instance for your
workload
Select the right
volume for your
workload

54. Related Sessions
BDT323 - Amazon EBS and Cassandra:
1 Million Writes Per Second on 60 Nodes

55. Thank you!