Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

(SDD416) Amazon EBS Deep Dive | AWS re:Invent 2014

33,658 views

Published on

Amazon Elastic Block Store (Amazon EBS) provides persistent block level storage volumes for use with Amazon EC2 instances. In this technical session, we conduct a detailed analysis of the differences among the three types of Amazon EBS block storage: General Purpose (SSD), Provisioned IOPS (SSD), and Magnetic. We discuss how to maximize Amazon EBS performance, with a special eye towards low-latency, high-throughput applications like databases. We discuss Amazon EBS encryption and share best practices for Amazon EBS snapshot management. Throughout, we share tips for success.

Published in: Technology
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://soo.gd/irt2 } ......................................................................................................................... Download Full EPUB Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download Full doc Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download PDF EBOOK here { https://soo.gd/irt2 } ......................................................................................................................... Download EPUB Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download doc Ebook here { https://soo.gd/irt2 } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book THIS can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer THIS is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story THIS Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money THIS the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths THIS Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://soo.gd/irt2 } ......................................................................................................................... Download Full EPUB Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download Full doc Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download PDF EBOOK here { https://soo.gd/irt2 } ......................................................................................................................... Download EPUB Ebook here { https://soo.gd/irt2 } ......................................................................................................................... Download doc Ebook here { https://soo.gd/irt2 } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book THIS can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer THIS is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story THIS Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money THIS the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths THIS Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download Full EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... ,DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ,DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Great job on that, also I've had unbelievable success using a stock trading system called Tryita. Check it out: www.tryita.com
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

(SDD416) Amazon EBS Deep Dive | AWS re:Invent 2014

  1. 1. © 2014 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
  2. 2. Amazon Elastic Block Store Deep Dive •Larger and faster volumes
  3. 3. EBS !=
  4. 4. A few definitions… •IOPS: Input/output operations per second (#) •Throughput: Read/write rate to storage (MB/s) •Latency: Delay between request and completion (ms) •Capacity: Volume of data that can be stored (GB) •Block size: Size of each I/O (k)
  5. 5. When performance matters, use SSD-backed volumes
  6. 6. •New default volume type for Amazon EBS •Every volume can burst up to 3,000 IOPS -Larger volumes can burst for longer periods •3 IOPS per GB baseline performance -Maximum of 3,000 on a 1TB volume •99% performance consistency
  7. 7. (2) Max IO credit per bucket is 5.4M (1) Always accumulating 3 IOPS per GB per second (3) You can spend up to 3000 IOPS per second 13
  8. 8. 14
  9. 9. 15
  10. 10. 16
  11. 11. VolumeType BootTime AccessTime OS GP2 3:31 4:33 WindowsServer 2012 Standard 4:30 7:16 WindowsServer 2012 GP2 0:36 0:45 CentOS6 Standard 0:57 1:16 CentOS6 40% Reduction in boot times by using General Purpose (SSD)
  12. 12. 80% Cost Savings. 50% more peak I/O with General Purpose (SSD)
  13. 13. 19
  14. 14. •Best for I/O intensive databases that require highest consistency •Provision up to 4,000 IOPS per volume •Supports IOPS to GB ratio of 30 •Designed to offer 99.9% consistency
  15. 15. •Best for cold workloads •Rarely accessed data that needs always on access •IOPS: ~100 IOPS steady-state, with best-effort bursts •Throughput: variable by workload, best effort to 10s of MB/s •Latency: Varies, reads typically ~20-40 ms, writes typically ~2-10ms
  16. 16. General Purpose (SSD) Provisioned IOPS (SSD) Magnetic Recommend use cases Boot volumes Small to med DBs Dev and test I/O intensive Large DBs Cold storage Storage media SSD-backed SSD-backed Magnetic-backed Volume size 1GB-1TB 1GB-1TB 1GB-1TB Max IOPS per volume 3 IOPS/GB Burst up to 3000 IOPS 4,000 IOPS ~100 IOPS Read and write peak throughput 128 MB/s 128 MB/s ~50-90 MBps Max I/O per node(16k) 48,000 48,000 48,000 Peak throughput node 800 MB/s 800 MB/s 800MB/s Latency (random read) 1-2ms 1-2 ms 20-40 ms API name gp2 io1 standard Price* $.10/GB-Month $.125/GB-Month $.065/provisioned IOPS $.05/GB-Month $.05/ 1M I/O
  17. 17. Always use General Purpose (SSD) for boot volumes
  18. 18. •LAW –L –A –W
  19. 19. 1.IOPS:Read/write I/O rate (IOPS) 2.Latency:Time between I/O submission and completion (ms) 3.Throughput:Read/write transfer rate (MB/s) –Throughput = IOPS X I/O Size
  20. 20. (1) Amazon EC2 (2) I/O (4) Amazon EBS (3) Network link
  21. 21. 1.EC2 instance:Network bandwidth(Mbps) 2.EBS-optimized:EC2 instance option (On/Off) 3.Workload:Block size, read/write ratio, serialization 4.Queue depth:The number of outstanding I/Os 5.RAID:Stripe volumes to maximize performance 6.Pre-warm:Eliminates first touch penalty
  22. 22. 1. Amazon EC2 instance • Compute optimized– C3 • Memory optimized- R3 • General Purpose – M3 Amazon EBS Amazon EC2 Select Amazon EC2 instance that has the right network, RAM, and CPU resources for your applications.
  23. 23. 2. Amazon EBS-optimized b
  24. 24. Use EBS-optimized instances for consistent EBS performance
  25. 25. 3. Workload EBS SSD-backed volumes measure I/O size up to 256KB EBS SSD-backed delivers same performance for read and write NEW
  26. 26. 4,000 IOPS PIOPS volume 4,000 IOPS 128MB/s throughput You can achieve 4,000 IOPS when driving smaller I/O You can achieve up to 128MB/s when driving larger I/O
  27. 27. 4,000 IOPS PIOPS volume 4,000 IOPS 128MB/s throughput 4,000 X 4KB = 16MB/s 4,000 X 32KB =128MB/s 8,000 x 8KB = 64MB/s 4,000 x 8KB = 32MB/s 2,000 x 128KB=256MB/s 1,000 x 128KB =128MB/s
  28. 28. Block (I/O) size determines whether your application is IOPS bound or throughput bound
  29. 29. 4. Queue depth An I/O Amazon EBS After it’s gone, it’s gone Amazon EC2 Queue depth is the pending I/O in flight
  30. 30. 41
  31. 31. I/O Latency
  32. 32. 0.075 35.1 0 5 10 15 20 25 30 35 40 1 4 8 12 16 20 24 28 32 Latency TP90 ( ms) Queue Depth Random READ latency across various QDs Latency (TP90) Read latency linearly increases with increase in queue depth
  33. 33. 0.075 35.1 2.09 1,865 4,152 3,851 - 500 1,000 1,500 2,000 2,500 3,000 3,500 4,000 4,500 0 5 10 15 20 25 30 35 1 4 8 12 16 20 24 28 32 Latency TP90 ( ms) Queue Depth 16 KB random READ IOPS, Latency across various QDs Latency (TP90) Avg Read IOPS IOPS Queue depth of one has the lowest latency, but has the lowest IOPS
  34. 34. 0.075 35.1 2.09 1,865 4,152 3,851 - 500 1,000 1,500 2,000 2,500 3,000 3,500 4,000 4,500 0 5 10 15 20 25 30 35 1 4 8 12 16 20 24 28 32 Latency TP90 ( ms) Queue Depth 16 KB random READ IOPS, Latency across various QDs Latency (TP90) Avg Read IOPS IOPS Queue depth between 4-8 has the optimal IOPS and latency performance
  35. 35. 0.075 35.1 2.09 1,865 4,152 3,851 - 500 1,000 1,500 2,000 2,500 3,000 3,500 4,000 4,500 0 5 10 15 20 25 30 35 1 4 8 12 16 20 24 28 32 Latency TP90 ( ms) Queue Depth 16 KB random READ IOPS, Latency across various QDs Latency (TP90) Avg Read IOPS IOPS Higher queue depths have negative impact on IOPS and latency
  36. 36. 0.08 7.71 845 4,152 0 500 1,000 1,500 2,000 2,500 3,000 3,500 4,000 4,500 0 1 2 3 4 5 6 7 8 9 10 1 4 8 12 16 20 24 28 32 Latency TP90 ( ms) Queue Depth 16 KB random WRITE IOPS, Latency across various QDs Latency (TP90) AvgIOPS IOPS Write latency, queue depth and IOPS interaction is similar to that of read latency
  37. 37. Optimal queue depth to achieve lower latency and highest IOPS is typically between 4-8; ~1 QD per 500 IOPS Amazon EBS-optimized offers consistent latency experience
  38. 38. 5. RAID • Increase performance or capacity or both • Stripe multiple volumes to achieve 10’s of TB and up-to 48,000 IOPS per Instance • Don’t mix volume types • Typically RAID 0 or LVM stripe • Avoid RAID for redundancy Amazon EBS Amazon EC2
  39. 39. Maximum performance per instance 12×400 GB PIOPS, pre-warmed, RAID 0 LVM, Stripe size 128 KB, attached to CR1 instance
  40. 40. Use stripe size of 128KB or 256KB
  41. 41. 6. Pre-warminghttp://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-prewarm.html
  42. 42. Use large block size to speed up your pre-warming. ex: sudoddif=/dev/xvdfof=/dev/xvdfconv=notruncbs=1M
  43. 43. Final tips •Ext4 or XFS (understand journal impact!) •Amazon EBS encryption has no visible IOPS performance impact for many Amazon EC2 instances •Alignment can matter, check tools use 4k
  44. 44. Workload/ Software Typical block size Random/ Seq? Max EBS @ 500 Mb/s Max EBS @ 1 Gb/s MaxEBS @ 10Gb/s instances Oracle DB Configurable:2 KB–16 KB Default 8 KB random ~7800 IOPS ~15,600 IOPS ~96,000 IOPS Microsoft SQL Server 8 KB w/ 64 KB extents random ~7800 IOPS ~15,600 IOPS ~80,000 IOPS MySQL 16 KB random ~4000 IOPS ~7,800 IOPS ~48,000 IOPS PostgreSQL 8 KB random ~7,800 IOPS ~15,600 IOPS ~96,000 IOPS MongoDB 4 KB serialized ~15,600 IOPS ~31,000 IOPS ~96,000 IOPS Cassandra 4 KB random ~15,600 IOPS ~31,000 IOPS ~96,000 IOPS GlusterFS 128 KB sequential ~500 IOPS ~1000 IOPS ~6,000 IOPS
  45. 45. EBS-optimized Oh, YEAH!! Amazon EC2 A “boatload” of I/O Right sized EBS
  46. 46. 1.EC2 instance:Network transfer rate (Mbps) 2.EBS-optimized:EC2 instance option (On/Off) 3.Workload:Block size, read/write ratio, serialization 4.Queue depth:The number of outstanding I/Os(#) 5.RAID:Stripe volumes to maximize performance 6.Pre-warm:Eliminates first touch penalty
  47. 47. Performance (per volume) GP2 PIOPS GP2 PIOPS Max Size 1 TB 1 TB 16 TB 16 TB Max IOPS 3,000 4,000 10,000 20,000 Max Throughput 128 Mbps 128 Mbps 160 MBps 320 MBps PREVIOUSLY W/ LARGER & FASTER VOLUMES
  48. 48. Amazon EBS larger and faster volume types General Purpose (SSD) Provisioned IOPS (SSD) Magnetic Recommend use cases Boot volumes Small to med DBs Dev and test I/O intensive Large DBs Cold storage Storage media SSD-backed SSD-backed Magnetic-backed Volume size 1GB-16TB 4GB-16TB 1GB-1TB Max IOPS per volume 10,000 IOPS 20,000 IOPS ~100 IOPS burst < 1TB to3000 IOPS baseline baseline Read and write peak throughput 160 MB/s 320 MB/s ~50-90 MBps Max IOPS per node (16k) 48,000 48,000 48,000 Peak throughput node 800 MB/s 800 MB/s 800MB/s Latency (random read) 1-2ms 1-2 ms 20-40 ms API name gp2 io1 standard Price* $.10/GB-Month $.125/GB-Month $.065/provisioned IOPS $.05/GB-Month $.05/ 1M I/O
  49. 49. 64
  50. 50. 66
  51. 51. • 2-tiered key hierarchy using envelope encryption • Unique data key encrypt customer data • AWS KMS master keys encrypt data keys • Benefits of envelope encryption: • Limits risk of a compromised data key • Better performance for encrypting large data • Easier to manage a small number of master keys than millions of data keys Master Key(s) Data Key 1 S3 object EBS volume Redshift cluster Data Key 2 Data Key 3 Data Key 4 Custom application AWS KMS
  52. 52. Select the right volume for your workload Select the right instance for your workload Take snapshots Use encryption if you need it
  53. 53. Please give us your feedback on this session. Complete session evaluations and earn re:Invent swag. http://bit.ly/awsevals

×