Benchmarking (RICON 2014)
•
3 likes•1,127 views
Knowledge of how to set up good benchmarks is invaluable in understanding performance of the system. Writing correct and useful benchmarks is hard, and verification of the results is difficult and prone to errors. When done right, benchmarks guide teams to improve the performance of their systems. When done wrong, hours of effort may result in a worse performing application, upset customers or worse! In this talk, we will discuss what you need to know to write better benchmarks for distributed systems. We will look at examples of bad benchmarks and learn about what biases can invalidate the measurements, in the hope of correctly applying our new-found skills and avoiding such pitfalls in the future.
Recommended
Recommended
More Related Content
Similar to Benchmarking (RICON 2014)
Similar to Benchmarking (RICON 2014) (20)
More from Aysylu Greenberg
More from Aysylu Greenberg (20)
Recently uploaded
Recently uploaded (20)
Benchmarking (RICON 2014)
- 1. Benchmarking: You’re Doing It Wrong Aysylu Greenberg @aysylu22
- 3. To Write Good Benchmarks… Need to be Full Stack
- 4. Benchmark = How Fast? your process vs Goal your process vs Best PracCces
- 5. Today • How Not to Write Benchmarks • Benchmark Setup & Results: - You’re wrong about machines - You’re wrong about stats - You’re wrong about what maLers • Becoming Less Wrong • Having Fun with Riak
- 6. HOW NOT TO WRITE BENCHMARKS
- 7. Website Serving Images • Access 1 image 1000 Cmes • Latency measured for each access • Start measuring immediately • 3 runs • Find mean • Dev environment Web Request Server Cache S3
- 8. WHAT’S WRONG WITH THIS BENCHMARK?
- 9. YOU’RE WRONG ABOUT THE MACHINE
- 10. Wrong About the Machine • Cache, cache, cache, cache!
- 11. It’s Caches All The Way Down Web Request Server Cache S3
- 12. It’s Caches All The Way Down
- 13. Caches in Benchmarks Prof. Saman Amarasinghe, MIT 2009
- 14. Caches in Benchmarks Prof. Saman Amarasinghe, MIT 2009
- 15. Caches in Benchmarks Prof. Saman Amarasinghe, MIT 2009
- 16. Caches in Benchmarks Prof. Saman Amarasinghe, MIT 2009
- 17. Caches in Benchmarks Prof. Saman Amarasinghe, MIT 2009
- 18. Website Serving Images • Access 1 image 1000 Cmes • Latency measured for each access • Start measuring immediately • 3 runs • Find mean • Dev environment Web Request Server Cache S3
- 19. Wrong About the Machine • Cache, cache, cache, cache! • Warmup & Cming
- 20. Website Serving Images • Access 1 image 1000 Cmes • Latency measured for each access • Start measuring immediately • 3 runs • Find mean • Dev environment Web Request Server Cache S3
- 21. Wrong About the Machine • Cache, cache, cache, cache! • Warmup & Cming • Periodic interference
- 22. Website Serving Images • Access 1 image 1000 Cmes • Latency measured for each access • Start measuring immediately • 3 runs • Find mean • Dev environment Web Request Server Cache S3
- 23. Wrong About the Machine • Cache, cache, cache, cache! • Warmup & Cming • Periodic interference • Test != Prod
- 24. Website Serving Images • Access 1 image 1000 Cmes • Latency measured for each access • Start measuring immediately • 3 runs • Find mean • Dev environment Web Request Server Cache S3
- 25. Wrong About the Machine • Cache, cache, cache, cache! • Warmup & Cming • Periodic interference • Test != Prod • Power mode changes
- 26. YOU’RE WRONG ABOUT THE STATS
- 27. Wrong About Stats • Too few samples
- 28. Wrong About Stats 120 100 80 60 40 20 0 Convergence of Median on Samples 0 10 20 30 40 50 60 Latency Time Stable Samples Stable Median Decaying Samples Decaying Median
- 29. Website Serving Images • Access 1 image 1000 Cmes • Latency measured for each access • Start measuring immediately • 3 runs • Find mean • Dev machine Web Request Server Cache S3
- 30. Wrong About Stats • Too few samples • Gaussian (not)
- 31. Website Serving Images • Access 1 image 1000 Cmes • Latency measured for each access • Start measuring immediately • 3 runs • Find mean • Dev machine Web Request Server Cache S3
- 32. Wrong About Stats • Too few samples • Gaussian (not) • MulCmodal distribuCon
- 33. MulCmodal DistribuCon 50% 99% # occurrences Latency 5 ms 10 ms
- 34. Wrong About Stats • Too few samples • Gaussian (not) • MulCmodal distribuCon • Outliers
- 35. YOU’RE WRONG ABOUT WHAT MATTERS
- 36. Wrong About What MaLers • Premature opCmizaCon
- 37. “Programmers waste enormous amounts of Cme thinking about … the speed of noncriCcal parts of their programs ... Forget about small efficiencies …97% of the Cme: premature opHmizaHon is the root of all evil. Yet we should not pass up our opportuniCes in that criCcal 3%.” -‐-‐ Donald Knuth
- 38. Wrong About What MaLers • Premature opCmizaCon • UnrepresentaCve workloads
- 39. Wrong About What MaLers • Premature opCmizaCon • UnrepresentaCve workloads • Memory pressure
- 40. Wrong About What MaLers • Premature opCmizaCon • UnrepresentaCve workloads • Memory pressure • Load balancing
- 41. Wrong About What MaLers • Premature opCmizaCon • UnrepresentaCve workloads • Memory pressure • Load balancing • Reproducibility of measurements
- 43. User AcCons MaLer X > Y for workload Z with trade offs A, B, and C -‐ hLp://www.toomuchcode.org/
- 44. Profiling Code instrumentaCon Aggregate over logs Traces
- 45. Microbenchmarking: Blessing & Curse + Quick & cheap + Answers narrow ?s well - Osen misleading results - Not representaCve of the program
- 46. Microbenchmarking: Blessing & Curse • Choose your N wisely
- 47. Choose Your N Wisely Prof. Saman Amarasinghe, MIT 2009
- 48. Microbenchmarking: Blessing & Curse • Choose your N wisely • Measure side effects
- 49. Microbenchmarking: Blessing & Curse • Choose your N wisely • Measure side effects • Beware of clock resoluCon
- 50. Microbenchmarking: Blessing & Curse • Choose your N wisely • Measure side effects • Beware of clock resoluCon • Dead Code EliminaCon
- 51. Microbenchmarking: Blessing & Curse • Choose your N wisely • Measure side effects • Beware of clock resoluCon • Dead Code EliminaCon • Constant work per iteraCon
- 52. Non-‐Constant Work Per IteraCon
- 53. Follow-‐up Material • How NOT to Measure Latency by Gil Tene – hLp://www.infoq.com/presentaCons/latency-‐piualls • Taming the Long Latency Tail on highscalability.com – hLp://highscalability.com/blog/2012/3/12/google-‐taming-‐ the-‐long-‐latency-‐tail-‐when-‐more-‐machines-‐equal.html • Performance Analysis Methodology by Brendan Gregg – hLp://www.brendangregg.com/methodology.html • Silverman’s Mode Detec@on Method by MaL Adereth – hLp://adereth.github.io/blog/2014/10/12/silvermans-‐ mode-‐detecCon-‐method-‐explained/
- 54. HAVING FUN WITH
- 55. Setup • SSD 30 GB • M3 large • Riak version 1.4.2-‐0-‐g61ac9d8 • Ubuntu 12.04.5 LTS • 4 byte keys, 10 KB values
- 56. 2350 2300 2250 2200 2150 2100 2050 2000 1950 1900 1850 Latency (usec) Get Latency L3 Number of Keys
- 60. Benchmarking: You’re Doing It Wrong Aysylu Greenberg @aysylu22