Your SlideShare is downloading. ×
0
Load-Testing &
        Benchmarking Riak
           Sean Cribbs       Jon Meredith
        Developer Advocate    Developer...
Why Benchmark?




basho
Why Benchmark?
        •Minimize surprises during deployment




basho
Why Benchmark?
        •Minimize surprises during deployment
        •Test expected load




basho
Why Benchmark?
        •Minimize surprises during deployment
        •Test expected load
        •Test peak load



basho
Why Benchmark?
        •Minimize surprises during deployment
        •Test expected load
        •Test peak load
        •...
Why Benchmark?
        •Minimize surprises during deployment
        •Test expected load
        •Test peak load
        •...
Why Benchmark?
        •Minimize surprises during deployment
        •Test expected load
        •Test peak load
        •...
Types of Benchmarks




basho
Types of Benchmarks
        •Throughput




basho
Types of Benchmarks
        •Throughput
         •operations / sec




basho
Types of Benchmarks
        •Throughput
         •operations / sec
         •systems experience



basho
Types of Benchmarks
        • Throughput           May be multiple
                                 requests!
         •op...
Types of Benchmarks
        • Throughput           May be multiple
                                 requests!
         •op...
Types of Benchmarks
        • Throughput           May be multiple
                                 requests!
         •op...
Types of Benchmarks
        • Throughput           May be multiple
                                 requests!
         •op...
Types of Benchmarks
        • Throughput           May be multiple
                                 requests!
         •op...
Benchmarking
          Overview




basho
Benchmarking
                Overview
        •Determine your application’s needs




basho
Benchmarking
                Overview
        •Determine your application’s needs
        •Test typical and worst-case sce...
Benchmarking
                Overview
        •Determine your application’s needs
        •Test typical and worst-case sce...
Benchmarking
                Overview
        •Determine your application’s needs
        •Test typical and worst-case sce...
Benchmarking Steps




basho
Benchmarking Steps

        •Start up your test cluster




basho
Benchmarking Steps

        •Start up your test cluster
        •Configure a test




basho
Benchmarking Steps

        •Start up your test cluster
        •Configure a test
        •Run the test against the cluster...
Benchmarking Steps

        •Start up your test cluster
        •Configure a test
        •Run the test against the cluster...
Benchmarking Steps

        •Start up your test cluster
        •Configure a test
        •Run the test against the cluster...
basho_bench




basho
basho_bench
        •Dave “Dizzy” Smith’s experience testing
          instant messaging and database systems




basho
basho_bench
        •Dave “Dizzy” Smith’s experience testing
          instant messaging and database systems

        •Cr...
basho_bench
        •Dave “Dizzy” Smith’s experience testing
          instant messaging and database systems

        •Cr...
basho_bench
        •Dave “Dizzy” Smith’s experience testing
          instant messaging and database systems

        •Cr...
basho_bench
        •Dave “Dizzy” Smith’s experience testing
          instant messaging and database systems

        •Cr...
basho_bench
basho
worker   worker   worker   worker




                 basho_bench
basho
Event Generator
                   get    update
                  80%      20%




        worker   worker   worker   wor...
Event Generator
                    get       update
                   80%         20%




        worker    worker      ...
Event Generator
                    get       update
                   80%         20%

                                 ...
Event Generator
                    get       update
                   80%         20%

                                 ...
Event Generator
                    get       update
                   80%         20%

                                 ...
Demo



basho
Microbenchmarks
            Are Bad




basho
Microbenchmarks
              Are Bad
        •Benchmarks should be long-running




basho
Microbenchmarks
              Are Bad
        •Benchmarks should be long-running
         •Cache warmup




basho
Microbenchmarks
              Are Bad
        •Benchmarks should be long-running
         •Cache warmup
         •Page flus...
Microbenchmarks
              Are Bad
        •Benchmarks should be long-running
         •Cache warmup
         •Page flus...
Microbenchmarks
              Are Bad
        •Benchmarks should be long-running
         •Cache warmup
         •Page flus...
basho
Key Generation

        •sequential_int_bin
        •uniform_int_bin
        •pareto_int_bin


basho
Uniform Distribution
                         {uniform_int_bin, 10000}
                                                   ...
Pareto Distribution
                        {pareto_int_bin, 10000}




                                                  ...
Value Generation


        •fixed_bin
        •exponential_bin


basho
Exponential Distrib.
                        {exponential_bin, 5000, 2500}
                                               ...
Benchmarking is Hard




basho
Benchmarking is Hard

        •Design an accurate test




basho
Benchmarking is Hard

        •Design an accurate test
        •Tool and system limits




basho
Benchmarking is Hard

        •Design an accurate test
        •Tool and system limits
        •Testing a multi-variate sp...
Benchmarking is Hard

        •Design an accurate test
        •Tool and system limits
        •Testing a multi-variate sp...
Benchmarking is Hard

        •Design an accurate test
        •Tool and system limits
        •Testing a multi-variate sp...
Gotchas




basho
Gotchas

        •Running out of file handles




basho
Gotchas

        •Running out of file handles
        •Swapping / thrashing



basho
Gotchas

        •Running out of file handles
        •Swapping / thrashing
        •Running benchmark on same machine


ba...
Conduct your own




basho
Conduct your own
        •Take metrics from existing app




basho
Conduct your own
        •Take metrics from existing app
         •Mixture of get/put/delete operations




basho
Conduct your own
        •Take metrics from existing app
         •Mixture of get/put/delete operations
         •Value si...
Conduct your own
        •Take metrics from existing app
         •Mixture of get/put/delete operations
         •Value si...
Conduct your own
        •Take metrics from existing app
         •Mixture of get/put/delete operations
         •Value si...
Conduct your own
        •Take metrics from existing app
         •Mixture of get/put/delete operations
         •Value si...
Plug
        Interested in learning about support,
        consulting, or Enterprise features?
          
        Email in...
Upcoming SlideShare
Loading in...5
×

Using Basho Bench to Load Test Distributed Applications

4,000

Published on

Slides from the Basho webinar held on

Published in: Technology, Business
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
4,000
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
72
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

  • Why do you benchmark
     - minimize suprises during deployment
     - test expected load
     - test peak load
     - see how much room there is to scale
     - find failure points - in your application or in the data store
  • Why do you benchmark
     - minimize suprises during deployment
     - test expected load
     - test peak load
     - see how much room there is to scale
     - find failure points - in your application or in the data store
  • Why do you benchmark
     - minimize suprises during deployment
     - test expected load
     - test peak load
     - see how much room there is to scale
     - find failure points - in your application or in the data store
  • Why do you benchmark
     - minimize suprises during deployment
     - test expected load
     - test peak load
     - see how much room there is to scale
     - find failure points - in your application or in the data store
  • Why do you benchmark
     - minimize suprises during deployment
     - test expected load
     - test peak load
     - see how much room there is to scale
     - find failure points - in your application or in the data store
  • Why do you benchmark
     - minimize suprises during deployment
     - test expected load
     - test peak load
     - see how much room there is to scale
     - find failure points - in your application or in the data store
  • Goals of benchmarking
     - Two basic kinds of tests
     - Throughput tests - how much can you throw at it
     - Latency tests - how well does it respond at a fixed load

  • Goals of benchmarking
     - Two basic kinds of tests
     - Throughput tests - how much can you throw at it
     - Latency tests - how well does it respond at a fixed load

  • Goals of benchmarking
     - Two basic kinds of tests
     - Throughput tests - how much can you throw at it
     - Latency tests - how well does it respond at a fixed load

  • Goals of benchmarking
     - Two basic kinds of tests
     - Throughput tests - how much can you throw at it
     - Latency tests - how well does it respond at a fixed load

  • Goals of benchmarking
     - Two basic kinds of tests
     - Throughput tests - how much can you throw at it
     - Latency tests - how well does it respond at a fixed load

  • Goals of benchmarking
     - Two basic kinds of tests
     - Throughput tests - how much can you throw at it
     - Latency tests - how well does it respond at a fixed load

  • Goals of benchmarking
     - Two basic kinds of tests
     - Throughput tests - how much can you throw at it
     - Latency tests - how well does it respond at a fixed load

  • Goals of benchmarking
     - Two basic kinds of tests
     - Throughput tests - how much can you throw at it
     - Latency tests - how well does it respond at a fixed load










  • Introduce basho_bench
     - Credit for Dizzy - created after years of experience testing instant messaging and
       database software.
     - Created for internal benchmarking
     - Designed for us to evaluate anything shaped like a key/value store
     - Used it heavily in our development and testing of innostore and bitcask.
     - Extensible

  • Introduce basho_bench
     - Credit for Dizzy - created after years of experience testing instant messaging and
       database software.
     - Created for internal benchmarking
     - Designed for us to evaluate anything shaped like a key/value store
     - Used it heavily in our development and testing of innostore and bitcask.
     - Extensible

  • Introduce basho_bench
     - Credit for Dizzy - created after years of experience testing instant messaging and
       database software.
     - Created for internal benchmarking
     - Designed for us to evaluate anything shaped like a key/value store
     - Used it heavily in our development and testing of innostore and bitcask.
     - Extensible

  • Introduce basho_bench
     - Credit for Dizzy - created after years of experience testing instant messaging and
       database software.
     - Created for internal benchmarking
     - Designed for us to evaluate anything shaped like a key/value store
     - Used it heavily in our development and testing of innostore and bitcask.
     - Extensible

  • Introduce basho_bench
     - Credit for Dizzy - created after years of experience testing instant messaging and
       database software.
     - Created for internal benchmarking
     - Designed for us to evaluate anything shaped like a key/value store
     - Used it heavily in our development and testing of innostore and bitcask.
     - Extensible

  • How basho bench works
     - event generator with weights
     - scheduler to trigger the events
     - worker threads calls driver to execute the events
     - key generators
     - value generators
  • How basho bench works
     - event generator with weights
     - scheduler to trigger the events
     - worker threads calls driver to execute the events
     - key generators
     - value generators
  • How basho bench works
     - event generator with weights
     - scheduler to trigger the events
     - worker threads calls driver to execute the events
     - key generators
     - value generators
  • How basho bench works
     - event generator with weights
     - scheduler to trigger the events
     - worker threads calls driver to execute the events
     - key generators
     - value generators
  • How basho bench works
     - event generator with weights
     - scheduler to trigger the events
     - worker threads calls driver to execute the events
     - key generators
     - value generators
  • How basho bench works
     - event generator with weights
     - scheduler to trigger the events
     - worker threads calls driver to execute the events
     - key generators
     - value generators

  • Say why this is a bad bad thing
     - benchmarks should be long running - make sure you saturate things
       otherwise you don't see the effect of a cache being filled, or pauses
       due to pages being flushedm, trees rebalanced etc.
  • Say why this is a bad bad thing
     - benchmarks should be long running - make sure you saturate things
       otherwise you don't see the effect of a cache being filled, or pauses
       due to pages being flushedm, trees rebalanced etc.
  • Say why this is a bad bad thing
     - benchmarks should be long running - make sure you saturate things
       otherwise you don't see the effect of a cache being filled, or pauses
       due to pages being flushedm, trees rebalanced etc.
  • Say why this is a bad bad thing
     - benchmarks should be long running - make sure you saturate things
       otherwise you don't see the effect of a cache being filled, or pauses
       due to pages being flushedm, trees rebalanced etc.
  • Say why this is a bad bad thing
     - benchmarks should be long running - make sure you saturate things
       otherwise you don't see the effect of a cache being filled, or pauses
       due to pages being flushedm, trees rebalanced etc.






  • Why benchmarking is hard
     - You will have to iterate a lot
     - Has the benchmarking tool hit the limit or is it the system under test?
     - Run a second copy from another server
  • Why benchmarking is hard
     - You will have to iterate a lot
     - Has the benchmarking tool hit the limit or is it the system under test?
     - Run a second copy from another server
  • Why benchmarking is hard
     - You will have to iterate a lot
     - Has the benchmarking tool hit the limit or is it the system under test?
     - Run a second copy from another server
  • Why benchmarking is hard
     - You will have to iterate a lot
     - Has the benchmarking tool hit the limit or is it the system under test?
     - Run a second copy from another server
  • Why benchmarking is hard
     - You will have to iterate a lot
     - Has the benchmarking tool hit the limit or is it the system under test?
     - Run a second copy from another server
  • Troublespots to look for
     - running out of file handles
     - running low on socket connections
     - swapping
  • Troublespots to look for
     - running out of file handles
     - running low on socket connections
     - swapping
  • Troublespots to look for
     - running out of file handles
     - running low on socket connections
     - swapping
  • Recommendations
     - Take metrics from your application - measure the mix of get/put/delete operations
     - Provide a map/reduce job to find out data distributions
  • Recommendations
     - Take metrics from your application - measure the mix of get/put/delete operations
     - Provide a map/reduce job to find out data distributions
  • Recommendations
     - Take metrics from your application - measure the mix of get/put/delete operations
     - Provide a map/reduce job to find out data distributions
  • Recommendations
     - Take metrics from your application - measure the mix of get/put/delete operations
     - Provide a map/reduce job to find out data distributions
  • Recommendations
     - Take metrics from your application - measure the mix of get/put/delete operations
     - Provide a map/reduce job to find out data distributions
  • Recommendations
     - Take metrics from your application - measure the mix of get/put/delete operations
     - Provide a map/reduce job to find out data distributions

  • Transcript of "Using Basho Bench to Load Test Distributed Applications"

    1. 1. Load-Testing & Benchmarking Riak Sean Cribbs Jon Meredith Developer Advocate Developer basho
    2. 2. Why Benchmark? basho
    3. 3. Why Benchmark? •Minimize surprises during deployment basho
    4. 4. Why Benchmark? •Minimize surprises during deployment •Test expected load basho
    5. 5. Why Benchmark? •Minimize surprises during deployment •Test expected load •Test peak load basho
    6. 6. Why Benchmark? •Minimize surprises during deployment •Test expected load •Test peak load •Assess headroom basho
    7. 7. Why Benchmark? •Minimize surprises during deployment •Test expected load •Test peak load •Assess headroom •Find failure points basho
    8. 8. Why Benchmark? •Minimize surprises during deployment •Test expected load •Test peak load •Assess headroom •Find failure points •You must measure to know! basho
    9. 9. Types of Benchmarks basho
    10. 10. Types of Benchmarks •Throughput basho
    11. 11. Types of Benchmarks •Throughput •operations / sec basho
    12. 12. Types of Benchmarks •Throughput •operations / sec •systems experience basho
    13. 13. Types of Benchmarks • Throughput May be multiple requests! •operations / sec •systems experience basho
    14. 14. Types of Benchmarks • Throughput May be multiple requests! •operations / sec •systems experience •Latency basho
    15. 15. Types of Benchmarks • Throughput May be multiple requests! •operations / sec •systems experience •Latency •time per operation basho
    16. 16. Types of Benchmarks • Throughput May be multiple requests! •operations / sec •systems experience •Latency •time per operation •user experience basho
    17. 17. Types of Benchmarks • Throughput May be multiple requests! •operations / sec •systems experience •Latency •time per operation Need mean, •user experience median, percentiles basho
    18. 18. Benchmarking Overview basho
    19. 19. Benchmarking Overview •Determine your application’s needs basho
    20. 20. Benchmarking Overview •Determine your application’s needs •Test typical and worst-case scenarios basho
    21. 21. Benchmarking Overview •Determine your application’s needs •Test typical and worst-case scenarios •Minimize variables changed between scenarios basho
    22. 22. Benchmarking Overview •Determine your application’s needs •Test typical and worst-case scenarios •Minimize variables changed between scenarios •Run early, run often basho
    23. 23. Benchmarking Steps basho
    24. 24. Benchmarking Steps •Start up your test cluster basho
    25. 25. Benchmarking Steps •Start up your test cluster •Configure a test basho
    26. 26. Benchmarking Steps •Start up your test cluster •Configure a test •Run the test against the cluster basho
    27. 27. Benchmarking Steps •Start up your test cluster •Configure a test •Run the test against the cluster •Generate pretty graphs (requires R) basho
    28. 28. Benchmarking Steps •Start up your test cluster •Configure a test •Run the test against the cluster •Generate pretty graphs (requires R) •Interpret graphs, tweak, re-test basho
    29. 29. basho_bench basho
    30. 30. basho_bench •Dave “Dizzy” Smith’s experience testing instant messaging and database systems basho
    31. 31. basho_bench •Dave “Dizzy” Smith’s experience testing instant messaging and database systems •Created for internal benchmarking basho
    32. 32. basho_bench •Dave “Dizzy” Smith’s experience testing instant messaging and database systems •Created for internal benchmarking •Benchmark anything key/value-like basho
    33. 33. basho_bench •Dave “Dizzy” Smith’s experience testing instant messaging and database systems •Created for internal benchmarking •Benchmark anything key/value-like •Used heavily on innostore and bitcask basho
    34. 34. basho_bench •Dave “Dizzy” Smith’s experience testing instant messaging and database systems •Created for internal benchmarking •Benchmark anything key/value-like •Used heavily on innostore and bitcask •Simple, extensible Erlang API basho
    35. 35. basho_bench basho
    36. 36. worker worker worker worker basho_bench basho
    37. 37. Event Generator get update 80% 20% worker worker worker worker basho_bench basho
    38. 38. Event Generator get update 80% 20% worker worker worker worker Driver (HTTP, Protobuffs, etc) basho_bench basho
    39. 39. Event Generator get update 80% 20% Key Generator uniform worker worker worker worker integer Driver (HTTP, Protobuffs, etc) basho_bench basho
    40. 40. Event Generator get update 80% 20% Key Generator uniform worker worker worker worker integer Driver (HTTP, Protobuffs, etc) Value Generator fixed 8KB basho_bench basho
    41. 41. Event Generator get update 80% 20% Key Generator uniform worker worker worker worker integer Driver (HTTP, Protobuffs, etc) Value Generator fixed Riak 8KB basho_bench basho
    42. 42. Demo basho
    43. 43. Microbenchmarks Are Bad basho
    44. 44. Microbenchmarks Are Bad •Benchmarks should be long-running basho
    45. 45. Microbenchmarks Are Bad •Benchmarks should be long-running •Cache warmup basho
    46. 46. Microbenchmarks Are Bad •Benchmarks should be long-running •Cache warmup •Page flushes basho
    47. 47. Microbenchmarks Are Bad •Benchmarks should be long-running •Cache warmup •Page flushes •RAM vs. Disk basho
    48. 48. Microbenchmarks Are Bad •Benchmarks should be long-running •Cache warmup •Page flushes •RAM vs. Disk •Test the limits of the infrastructure basho
    49. 49. basho
    50. 50. Key Generation •sequential_int_bin •uniform_int_bin •pareto_int_bin basho
    51. 51. Uniform Distribution {uniform_int_bin, 10000} CDF Uniform 1.0 0.8 • Equal chance 0.6 of selecting cdf any key 0.4 0.2 0.0 0 2000 4000 6000 8000 10000 key basho
    52. 52. Pareto Distribution {pareto_int_bin, 10000} 1.0 0.8 • Hard coded so ppareto(x, 2000, 1.5) 0.6 20% of keys receive 80% 0.4 of requests 0.2 0.0 0 2000 4000 6000 8000 10000 x basho
    53. 53. Value Generation •fixed_bin •exponential_bin basho
    54. 54. Exponential Distrib. {exponential_bin, 5000, 2500} Exponential CDF 1.0 • Value size: 0.8 Min + Exp(Lambda) 0.6 P(Size <= size) • Set Lambda 0.4 1 / (Mean - Min) 0.2 0.0 0 5000 10000 15000 20000 size basho
    55. 55. Benchmarking is Hard basho
    56. 56. Benchmarking is Hard •Design an accurate test basho
    57. 57. Benchmarking is Hard •Design an accurate test •Tool and system limits basho
    58. 58. Benchmarking is Hard •Design an accurate test •Tool and system limits •Testing a multi-variate space basho
    59. 59. Benchmarking is Hard •Design an accurate test •Tool and system limits •Testing a multi-variate space •Easy to take results out of context basho
    60. 60. Benchmarking is Hard •Design an accurate test •Tool and system limits •Testing a multi-variate space •Easy to take results out of context •Everything is relative! basho
    61. 61. Gotchas basho
    62. 62. Gotchas •Running out of file handles basho
    63. 63. Gotchas •Running out of file handles •Swapping / thrashing basho
    64. 64. Gotchas •Running out of file handles •Swapping / thrashing •Running benchmark on same machine basho
    65. 65. Conduct your own basho
    66. 66. Conduct your own •Take metrics from existing app basho
    67. 67. Conduct your own •Take metrics from existing app •Mixture of get/put/delete operations basho
    68. 68. Conduct your own •Take metrics from existing app •Mixture of get/put/delete operations •Value size distribution basho
    69. 69. Conduct your own •Take metrics from existing app •Mixture of get/put/delete operations •Value size distribution •Key distribution basho
    70. 70. Conduct your own •Take metrics from existing app •Mixture of get/put/delete operations •Value size distribution •Key distribution •“Hot” and “cold” keys basho
    71. 71. Conduct your own •Take metrics from existing app •Mixture of get/put/delete operations •Value size distribution •Key distribution •“Hot” and “cold” keys •Configure a test, run, RINSE & REPEAT basho
    72. 72. Plug Interested in learning about support, consulting, or Enterprise features?   Email info@basho.com or go to http://www.basho.com/contact.html to talk with us. www.basho.com basho
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×