Learn tips and techniques that will improve the performance of your applications and databases running on Amazon EC2 instance storage and/or Amazon Elastic Block Store (EBS). This advanced session discusses when to use HI1, HS1, and Amazon EBS. We will share an "under the hood" view to tune the performance of your Elastic Block Store and best practices for running workloads on Amazon EBS, such as relational databases (MySQL, Oracle, SQL Server, Postgres) and NoSQL data stores, such as MongoDB and Riak.
3. What is Amazon EBS?
• Network block storage
• Attach to EC2 within the same AZ
• Point-in-time snapshots to S3
• Optional managed encryption
• Provision and pay just for what you need
4. EBS Volume Types
• General Purpose (SSD)
• Provisioned IOPS (SSD)
• Magnetic
When performance matters, use SSD-backed volumes
5. Amazon EBS General Purpose (SSD)
• New default volume type for EBS
• Every volume can burst up to 3,000 IOPS
• Larger volumes can burst for longer periods
• Token bucket burst algorithm
• 3 IOPS per GB baseline performance
• Maximum of 3,000 on a 1TB volume
• Throughput up to 128MB/s per volume
• Designed to deliver within 10% of performance 99% of the time
• Latency: Single digit milliseconds
6. Token Bucket
Bucket Size
3000 IOPS in 30 mins =
5.4M tokens
Fill rate
3 IOPS * volume size in GB
Bucket starts full!
7. Amazon EBS PIOPS (SSD)
• Provision up to 4,000 IOPS per volume
• Supports IOPS to GB ratio of 30 (>133GB to achieve 4000 IOPS)
• Designed to deliver within 10% of performance 99.9% of the time
• Throughput: up to 128MB/s per volume
• Latency: Single digit millisecond
NEW
• Best for IO intensive databases that require highest consistency
8. Amazon EBS Magnetic
• IOPS: ~100 IOPS steady-state, with best-effort bursts
• Throughput: variable by workload, best effort to 10s of MB/s.
• Latency: Varies, reads typically ~20-40 ms, writes typically ~2-10ms
• Best for cold workloads
9. EBS Volume Types (Cheat Sheet)
General Purpose (SSD) Provisioned IOPS (SSD) Magnetic
Recommend Use Cases
Boot Volumes
Low to Med Perf DBs
Dev and test
I/O intensive
High Perf DBs
Cold Storage
Storage Media SSD-backed SSD-backed Magnetic-backed
Volume Size 1GB-1TB 1GB-1TB 1GB-1TB
Max IOPS per volume
3 IOPS/GB
Burst up to 3000 IOPS
4,000 IOPS ~100 IOPS
Read and Write Peak
Throughput
128 MB/s 128 MB/s ~50-90 MB/s
Max I/O per node 48,000 48,000 ~1000-5000
Peak Throughput Node 800 MB/s 800 MB/s ~800 MB/s
Latency (random read) 1-2ms 1-2 ms 20-40 ms
API Name gp2 io1 standard
Price* $.10/GB-Month
$.125/GB-Month
$.065/provisioned IOPS
$.05/GB-Month
$.05/ 1M I/O
17. 3. Workload
• I/O size
– Data size of read/write
• I/O pattern
– Sequential and random
• I/O type
– Read and write
• I/O concurrency
– Number of concurrent IO
EBS SSD-backed volumes measure IO size up to 256KB
EBS SSD-backed delivers same performance for read and write
NEW
18. IOPS and Throughput limits on EBS
4,000 IOPS
PIOPS volume
4,000 IOPS
You can achieve 4,000 IOPS
128MB/s throughput
when driving smaller IO
4000 IOPS x 32 KB = 128 MB/s
You can achieve up to 128MB/s
when driving larger IO
500 IOPS x 256 KB = 128 MB/s
19. Block (IO) size determines whether your application is IOPS bound or
throughput bound
20. 4. Queue Depth
Queue depth is the pending IO in flight.
An I/O
Amazon
EBS
After it’s gone, it’s gone
Amazon
EC2
21. Random Read Latency
0.075
35.1
16 KB random READ IOPS, Latency across various QDs
2.09
1,865
4,152
3,851
4,500
4,000
3,500
3,000
2,500
2,000
1,500
1,000
500
-
35
30
25
20
15
10
5
0
1 4 8 12 16 20 24 28 32
Latency TP90 ( ms)
Queue Depth
Latency (TP90) Avg Read IOPS
IOPS
Queue depth of one has the lowest latency, but has the lowest IOPS
22. Random Read Latency
0.075
35.1
16 KB random READ IOPS, Latency across various QDs
2.09
1,865
4,152
3,851
4,500
4,000
3,500
3,000
2,500
2,000
1,500
1,000
500
-
35
30
25
20
15
10
5
0
1 4 8 12 16 20 24 28 32
Latency TP90 ( ms)
Queue Depth
Latency (TP90) Avg Read IOPS
IOPS
Queue depth between 4-8 has the optimal IOPS and latency performance
23. Random Read Latency
0.075
35.1
16 KB random READ IOPS, Latency across various QDs
2.09
1,865
4,152
3,851
4,500
4,000
3,500
3,000
2,500
2,000
1,500
1,000
500
-
35
30
25
20
15
10
5
0
1 4 8 12 16 20 24 28 32
Latency TP90 ( ms)
Queue Depth
Latency (TP90) Avg Read IOPS
IOPS
Higher queue depths have negative impact on IOPS and latency
24. Optimal queue depth to achieve lower latency and highest IOPS is
between 4-8; 1 QD per 500 IOPS
EBS-Optimized offers consistent latency experience
26. Maximum performance per instance
12×400 GB PIOPS, RAID 0 LVM, Stripe size 64 KB, attached to CR1 instance
• How should you think about taking snapshots on a striped volume?
– Quiesce file systems and take snapshot
– Unmount file system and take snapshot
– Use OS-specific tools
30. Four Key Pieces: Balanced Oh, YEAH!!
EBS-optimized
A “boatload” of I/O
Amazon
EC2
Right sized EBS
31. Thank you!
Maximizing EC2 and
Elastic Block Store Disk Performance
Martin Elwin, Manager, Solutions Architecture, AWS
Editor's Notes
1 Welcome and thank you for taking the time to attend the AWS NYC Summit and this session.
2. I’m Vinay Kumar, Product Manager for Amazon Elastic Block Storage.
3 For the next 50 minutes or so we will be discussing – How to extract the maximum amount of performance from the combination of EC2 and EBS.
For most builders – AWS is get in and go. We have made it even easier with our recent announcement of General Purpose (SSD) storage.
Attach General purpose (SSD) volumes ..you should be just fine. (What is it)
the rest, let’s talk about optimizing for performance.
So – what is EBS?
Persistent block level storage - storage volumes that persist independently from the life of the instance.
Automatically replicated within the same AZ ..designed for five 9s
Do things such as take snapshots (more on that later)
Has certain reliability design goals
Follows the general AWS approach of choice and pay as you go
EBS now has 3 volume types –
You need to carefully choose the volume type to optimize performance and cost for your applications.
Magnetic Standard
Guidanace is true for both GP(SSD) and PIOPS
Newest volume type added
Making SSD compelling use case for general purpose use cases. $0.10 per GB …4 months
Every volume ..irrespevie of the size can burst to 3000 IOPS for smaller IO size; or 128MB?s for larger block size use cases
Smaller volumes can burst up to 30 mins …larger volumes can burst for longer durations.
Baseline performance of 3 IOPS per GB. For example, 500GB ..1500 but burst to 3000
99% consitency
Latency expectionai is in single digit millit seconds.
One of the design goal is to make boot faster. ~6000 Ios linux
For database, if your database requires
Take away: IF you need a cost effective
Smaller volumes burst up to 30 mins ..larger volumes burst for longer periods. For example ..750GB volumes can burst …2Hours.
The second is EBS PIOPS – Made available in Mid-2012
Backed by SSD
Used for heavy-use workloads needing very consistent performance
DATABASES!
What can I expect from EBS PIOPS ?
Specify up front amount from 100 to 4K Provisioned IOPS (PIOPS)
30 IOPS per GB (4K PIOP volume as small as 134 GB)
Can achieve single digit millisecond latencies when paired with EBS-Optimized
Designed to deliver within 10% of the provisioned IOPS performance 99.9% of the time.
Throughput occurs in 16K blocks in PIOPS
as a service – that’s a throttle used to constrain the performance you will receive which allows us to maintain the consistency.
Which means up to 64 MB/s, as provisioned
4K (PIOP/second) x 16K (Bytes) = 64,000,000 or 64 Mega (million) Bytes/second
Latency(much lower than EBS Standard)
1 over the PIOPS or 1 Second/The number of Provisioned –IOPS = Latency
For example 1K PIOP volume would be 1/1K or one ms (1/1,000)
Or, for example a 4K PIOP volume would be ¼ of a ms latency
Price
Again, from a technical perspective, request exactly what you need. In TCO discussions – want to make to finance folks head spin – because it is hard to quantify. When we don’t need it, we’ll make it go away and we’ll stop paying for it. When we need it again we’ll make it re-appear and we’ll begin paying for it. If we need it to have different characteristics we’ll tell it to change.
There are two “Product” choices when it comes to EBS.
The first we will review is EBS Standard - Our original block storage as a service
Behaviors are:
Bursty I/O – swap to disk, load OS and App
Heavy sequential READ – FAST!
Original design consideration was as AMI image for EC2
What can I expect from Standard EBS?
~100 IOPS steady state
Best effort bursts to hundreds of IOPS (maybe 700 up to 1,100)
Provided on a “best effort” basis (steady state loads will normalize to the ~100 number)
You’ll observe short-term bursts of accepted work but then will return to the wall rate of ~100 IOPS
Throughput
As a result of variable IOPS Standard EBS has a variable throughput
Latency
Reads < 20 ms
Writes < 10 ms
Capacity
As provisioned, up to 1 TB
Use case for EBS Standard?
Well-suited for use as boot volumes or applications where I/O can be bursty.
- it is designed as a root volume service designed for EC2 to start and stop instances.
Putting it all together –
Use SSD-backed for performance
GP2 offers the best price performance
Always use gp2 volumes for boot.
Customers generally use
Breeze through ..
Level set ..so that we are talking the same there.
I’m
To achieve end-to-end performance optimization, you need to look at these 4 components.
For many of you this is review – but let’s take a moment and get everyone on the same page.
Key pieces are:
1-TheEC2 instance
2-The EBS Volume
3-The network link that connects the EBS volume and the EC2 instance
4-The Input and Output - The individual transits back and forth between EBS and EC2
These four key components work together.
So, to maximize performance it is important to arrange them into a balanced system.
There are 6 available tools to balance the systems and we are going ..
EBS is a network attached block storage. Network connectivity is super important for performance
Bandwidth:
Instances are important:
Reminder that EBS is a network block store
Customers generally use memory optimized insances R3
Network resources available per instance is very very important Low – moderate – high
Higher the resource ..higher the IOPS.
First – Choosing the right instance is important. EC2 is classified into memory, compute and general purpose
Instance network resource = low, medium, and high
1,000 Mbits/sec = 128MB/s
Key take-aways from this slide: (This table is a sample)
1-Maximum throughput delivered by EBS is limited by EC2 bandwidth (see this fairly regularly as an SA)
2-Different EC2 instances have different levels of bandwidth capacity (review the AWS site details)
3-You can compute the max MB/s for a given configuration
4-There is a CPU factor
6-Cluster compute instances (list) are on the “Cluster Compute Network” which means:
10 Gigabit
Full bi-section
Non-blocking
Not linear in relationship to cost
Does not include option to select “EBS Optimized” – that’s already done.
You’ll hear this many times from me and my AWS colleagues - Use PIOPS and EBS Optimized for your databases
Key Messages:
Use EBS SSD backed for consistency of latency and throughput
Use EBS Optimized to ensure that network traffic isn’t interfering with storage traffic
Every application IO workload is different. They are characterized by
Consistency is the keyword
PIOPS, because of its “throttled” say computer-generated behavior is going to provide a very consistent behavior – unlike many physical devices it has the same:
1- performance for random and sequential workloads. (Physical devices don’t do that).
2- The same number of IOPS for reads and writes
3- The same I/O size – 16K – All the time
Oracle, MySQL, mongo
PIOPS is Optimized for database workloads.
MySQL default block size is 16k, SQL Server default block size is 8k, MongoDB default block size is 4K, and Oracle default block size is 8k
Cloudera and MapR has a default block size of 64MB, DWH has block size in 1MB-2MB range
Queue depth is the pending IO in flight. Another simple way to think about it is ..you need to keep EBS volume busy all the time. Hence, you need to have IO in queue to be processed by EBS.
You can’t put lost IOPS back on the clock.
Discuss the CloudWatch metrics for Queue Depth
RAID is a useful tool
It allows you to aggregate the performance of multiple EBS volumes to deliver whatever IOPS number you need to get to. Again, think end-to-end and avoiding constraints.
Considerations:
If your application (such as MS SQL Server) has the ability to manage storage – typically will do a better job at this. Just give your software EBS volumes.
RAID 5 and RAID 6 are not recommended (strong statement) – please do not use.
Parity information is distributed among the drives. Which means a write to every disk. A write occurs for every volume in RAID 5 and RAID 6. And contention potentially occurs with the other writes increasing latency.
RAID 0, RAID 1 and in extreme cases RAID 10 make sense.
Both Standard EBS and PIOPS EBS both operate at much better Annual Failure Rate (AFR) than standard physical disk at point 1 percent (0.1%) to point 4 percent (0.4%) per year. Which compares very favorably to physical disk at 4.0%.
Certainly the design consideration should be that “things fail”. RAID in AWS is on top of an already existing improved redundancy level.
We have team using LVM direct to disk. No problem there.
More users are using MDADM. Lower volume of support requests on that side.
We also have teams using MDADM to manage the storage and LVM above that to manage the volume you construct above that storage.
For example – MongoDB: You create one big RAID so that anyone any one of the logical storage volumes you construct can use the IOPS you have available.
You can also stripe multiple volumes together to achieve up to 48,000 IOPS when attached to larger EC2 instances.
Key Messages
1-On the larger instances you can drive pretty awesome read performance – in the range of 50K READ IOPS.
The volume need not be attached to a running instance in order to take a snapshot.
These snapshots can be used to create multiple new Amazon EBS volumes, expand the size of a volume, or move volumes across Availability Zones.
The snapshots can be shared with specific AWS accounts or made public.
Considerations:
When you do RAID the single biggest operational consideration is executing snapshots. Because its possible to have a write that’s happening to multiple different EBS volumes at slightly different latencies, then taking a snapshot from those volumes has be done in such a way that is more consistent. Which means that the file system cannot be in a state of change when a snapshot is taken.
Discussion points
1-This is a preview of work that is underway and planned for publishing
2-Different workloads with different default characteristics
3-Optimized configuration for each workload
Zeroing in on just one – Mongo
1-Typical block size – 4 KB
2-Typical I/O Type – serialized (interesting, we’ll get to that shortly)
3-Recommended EBS Product – EBS PIOPS
4-Throughput on different instance link capacity (linear with PIOPS
4 volumes and 8 volumes both at RAID 0
CC though jumps off the chart at 3x the IOPS
So to re-cap
1-EC2 instances have different workload focus and network config – select one that fits your goals
2-EBS Optimized, in the context of this discussion is almost always a “yes please”
3-PIOPS is built for database workloads – enough said
4-Manage your queue depth
5-Understand and match your workload blocksize to PIOPS
6-Yor workload has its own tuning options and behavios – those matter too
Now we have everything optimized and working as a “system”
Which results in Maximized EC2 and Elastic Block Store Disk Performance
Oh, YEAH!
1 Welcome and thank you for taking the time to attend the AWS NYC Summit and this session.
2. I’m Vinay Kumar, Product Manager for Amazon Elastic Block Storage.
3 For the next 50 minutes or so we will be discussing – How to extract the maximum amount of performance from the combination of EC2 and EBS.