The document discusses common mistakes made when benchmarking systems and provides recommendations for improving benchmarking methodology. Some key issues covered include failing to account for hardware caches, using too few samples which can miss outliers, and testing with unrealistic workloads not representative of real-world usage. The document recommends approaches like profiling tools to better understand system behavior, ensuring workloads are representative, eliminating sources of noise, and reproducing results.

1. Benchmarking:-
You’re Doing It Wrong
Aysylu-Greenberg-
@aysylu22-
October-2015-

2. Aysylu-Greenberg-
--------@aysylu22-
-
-

4. To-Write-Good-Benchmarks…-
Need-to-be-Full-Stack-

5. -
-
your-process-vs-goal-
your-process-vs-best-pracFces-
-
Benchmark-=-How-Fast?-

6. Today-
• How-Not-to-Write-Benchmarks-
• Benchmark-Setup-&-Results:-
- -You’re-wrong-about-machines-
- -You’re-wrong-about-stats-
- -You’re-wrong-about-what-maOers-
• Becoming-Less-Wrong-

7. HOW$NOT$TO$WRITE$BENCHMARKS$

8. Website-Serving-Images-
• Access-1-image-1000-Fmes-
• Latency-measured-for-each-access-
• Start-measuring-immediately-
• 3-runs-
• Find-mean-
• Dev-environment-
Web-Request-
Server-
S3-Cache-

9. WHAT’S$WRONG$WITH$THIS$
BENCHMARK?$$

10. YOU’RE$WRONG$ABOUT$THE$MACHINE$
$

11. Wrong-About-the-Machine-
• Cache,-cache,-cache,-cache!-

12. It’s-Caches-All-The-Way-Down-
Web-Request-
Server-
S3-Cache-

13. It’s-Caches-All-The-Way-Down-

14. Prefetching:-Program-

16. Prefetching:-Disabled-

17. Prefetching:-Enabled-

19. Caches-in-Benchmarks-
Prof.-Saman-Amarasinghe,-MIT-2009--

20. Caches-in-Benchmarks-
Prof.-Saman-Amarasinghe,-MIT-2009--

21. Caches-in-Benchmarks-
Prof.-Saman-Amarasinghe,-MIT-2009--

22. Caches-in-Benchmarks-
Prof.-Saman-Amarasinghe,-MIT-2009--

23. Caches-in-Benchmarks-
Prof.-Saman-Amarasinghe,-MIT-2009--

24. Website-Serving-Images-
• Access-1-image-1000-Fmes-
• Latency-measured-for-each-access-
• Start-measuring-immediately-
• 3-runs-
• Find-mean-
• Dev-environment-
Web-Request-
Server-
S3-Cache-

25. Wrong-About-the-Machine-
• Cache,-cache,-cache,-cache!-
• Warmup-&-Fming-

26. Website-Serving-Images-
• Access-1-image-1000-Fmes-
• Latency-measured-for-each-access-
• Start-measuring-immediately-
• 3-runs-
• Find-mean-
• Dev-environment-
Web-Request-
Server-
S3-Cache-

27. Wrong-About-the-Machine-
• Cache,-cache,-cache,-cache!-
• Warmup-&-Fming-
• Periodic-interference-

28. Periodic-Interference-

29. Periodic-Interference-

30. Website-Serving-Images-
• Access-1-image-1000-Fmes-
• Latency-measured-for-each-access-
• Start-measuring-immediately-
• 3-runs-
• Find-mean-
• Dev-environment-
Web-Request-
Server-
S3-Cache-

31. Wrong-About-the-Machine-
• Cache,-cache,-cache,-cache!-
• Warmup-&-Fming-
• Periodic-interference-
• Test-!=-Prod-

32. Website-Serving-Images-
• Access-1-image-1000-Fmes-
• Latency-measured-for-each-access-
• Start-measuring-immediately-
• 3-runs-
• Find-mean-
• Dev-environment-
Web-Request-
Server-
S3-Cache-

33. Wrong-About-the-Machine-
• Cache,-cache,-cache,-cache!-
• Warmup-&-Fming-
• Periodic-interference-
• Test-!=-Prod-
• Power-mode-changes-

34. Power-Modes-
$-cat-/sys/devices/system/cpu/*/cpufreq/scaling_governor-
“ondemand”-OR-“performance”-
-
Current-CPU-frequencies:-
$-grep-"MHz"-/proc/cpuinfo-

35. YOU’RE$WRONG$ABOUT$THE$STATS$
$

36. Wrong-About-Stats-
• Too-few-samples-
-

37. 0-
20-
40-
60-
80-
100-
120-
0- 10- 20- 30- 40- 50- 60-
Latency$
#$Runs$
Convergence$of$Median$on$Samples$
Stable-Samples-
Stable-Median-
Decaying-Samples-
Decaying-Median-

38. Website-Serving-Images-
• Access-1-image-1000-Fmes-
• Latency-measured-for-each-access-
• Start-measuring-immediately-
• 3-runs-
• Find-mean-
• Dev-machine-
Web-Request-
Server-
S3-Cache-

39. Wrong-About-Stats-
• Too-few-samples-
• Gaussian-(not)-

40. Website-Serving-Images-
• Access-1-image-1000-Fmes-
• Latency-measured-for-each-access-
• Start-measuring-immediately-
• 3-runs-
• Find-mean-
• Dev-machine-
Web-Request-
Server-
S3-Cache-

41. Wrong-About-Stats-
• Too-few-samples-
• Gaussian-(not)-
• MulFmodal-distribuFon-

42. MulFmodal-DistribuFon-
50%-
99%-
#-occurrences-
Latency- 5-ms- 10-ms-

43. MulFmodal-DistribuFon-

44. Wrong-About-Stats-
• Too-few-samples-
• Gaussian-(not)-
• MulFmodal-distribuFon-
• Outliers-

45. Coordinated-Omission-
0-
request-
response-
request-
response-
10-
request-
20- 30- 40- 50- 60- 70- 80-
response-
Fme-
request-
response-
request-

46. Wrong-About-Stats-
• Too-few-samples-
• Gaussian-(not)-
• MulFmodal-distribuFon-
• Outliers-

47. YOU’RE$WRONG$ABOUT$WHAT$MATTERS$
$

48. Wrong-About-What-MaOers-
• Premature-opFmizaFon-

49. “Programmers-waste-enormous-amounts-of-
Fme-thinking-about-…-the-speed-of-
noncriFcal-parts-of-their-programs-...-Forget-
about-small-efficiencies-…97%-of-the-Fme:-
premature$opImizaIon$is$
the$root$of$all$evil.-Yet-we-
should-not-pass-up-our-
opportuniFes-in-that-criFcal-
3%.”-
-
pp-Donald-Knuth-

50. Wrong-About-What-MaOers-
• Premature-opFmizaFon-
• UnrepresentaFve-workloads-

51. Wrong-About-What-MaOers-
• Premature-opFmizaFon-
• UnrepresentaFve-workloads-
• Memory-pressure-

52. Wrong-About-What-MaOers-
• Premature-opFmizaFon-
• UnrepresentaFve-workloads-
• Memory-pressure-
• Hidden-components-

53. Wrong-About-What-MaOers-
• Premature-opFmizaFon-
• UnrepresentaFve-workloads-
• Memory-pressure-
• Hidden-components-
• Reproducibility-of-measurements-

54. BECOMING$LESS$WRONG$

55. User-AcFons-MaOer-
-
X->-Y-for-workload-Z-
with-trade-offs-A,-B,-and-C-
p-hOp://www.toomuchcode.org/-

56. Profiling-
-

57. Profiling-
perf-
gprof-&-
Oprofile-
YourKit-&-
jProfiler-
jVisualVM-
cProfile-

58. Profiling-
perf-
gprof-&-
Oprofile-
YourKit-&-
jProfiler-
jVisualVM-
cProfile-

59. perf-
#-Various-basic-CPU-staFsFcs,-system-wide,-for-10-seconds-
perf-stat-pe-cycles,instrucFons,cachepmisses-pa-sleep-10-
#-Count-system-calls-for-the-enFre-system,-for-5-seconds-
perf-stat-pe-'syscalls:sys_enter_*'-pa-sleep-5-
#-Sample-CPU-stack-traces,-once-every-10,000-Level-1-data-
cache-misses,-for-5-seconds-
perf-record-pe-L1pdcacheploadpmisses-pc-10000-pag-pp-sleep-5-
hOp://www.brendangregg.com/perf.html-

60. perf-
hOp://www.brendangregg.com/perf.html-

61. Profiling-
perf-
gprof-&-
Oprofile-
YourKit-&-
jProfiler-
jVisualVM-
cProfile-

62. gprof:-Where-Does-It-Spend-Its-Time?-
• Compile-with-profiling-
-
• Execute-the-code-
-
• Run-the-gprof-
hOp://www.thegeekstuff.com/2012/08/gprofptutorial/-

63. gprof:-Where-Does-It-Spend-Its-Time?-
hOp://www.thegeekstuff.com/2012/08/gprofptutorial/-

64. Profiling-
perf-
gprof-&-
Oprofile-
YourKit-&-
jProfiler-
jVisualVM-
cProfile-

65. hOp://www.brendangregg.com/linuxperf.html-

66. Profiling-
perf-
gprof-&-
Oprofile-
YourKit-&-
jProfiler-
jVisualVM-
cProfile-

67. Profiling-
perf-
gprof-&-
Oprofile-
YourKit-&-
jProfiler-
jVisualVM-
cProfile-

68. Profiling-
perf-
gprof-&-
Oprofile-
YourKit-&-
jProfiler-
jVisualVM-
cProfile-

69. Profiling-
perf-
gprof-&-
Oprofile-
YourKit-&-
jProfiler-
jVisualVM-
cProfile-

70. Profiling-
Code-instrumentaFon-
Aggregate-over-logs-
Traces-
-

71. Microbenchmarking:-Blessing-&-Curse-
+ Quick-&-cheap-
+ Answers-narrow-?s-well-
- O|en-misleading-results-
- Not-representaFve-of-the-program-

72. Microbenchmarking:-Blessing-&-Curse-
• Choose-your-N-wisely-
-

73. Choose-Your-N-Wisely-
Prof.-Saman-Amarasinghe,-MIT-2009--

74. Microbenchmarking:-Blessing-&-Curse-
• Choose-your-N-wisely-
• Measure-side-effects-

75. Microbenchmarking:-Blessing-&-Curse-
• Choose-your-N-wisely-
• Measure-side-effects-
• Beware-of-clock-resoluFon-

76. Microbenchmarking:-Blessing-&-Curse-
• Choose-your-N-wisely-
• Measure-side-effects-
• Beware-of-clock-resoluFon-
• Dead-Code-EliminaFon-

77. Microbenchmarking:-Blessing-&-Curse-
• Choose-your-N-wisely-
• Measure-side-effects-
• Beware-of-clock-resoluFon-
• Dead-Code-EliminaFon-
• Constant-work-per-iteraFon-

78. NonpConstant-Work-Per-IteraFon-

79. What-Should-a-Benchmark-Do?-
Measure-behavior-of-system-
-
Represent-realisFc-workload-
-
Run-for-sufficiently-long-Fme-
-
Compare-in-the-same-context-
-
Output-predictable-and-reproducible-results-

80. Followpup-Material-
• How$NOT$to$Measure$Latency$by-Gil-Tene-
– hOp://www.infoq.com/presentaFons/latencyppi}alls-
• Taming$the$Long$Latency$Tail-on-highscalability.com-
– hOp://highscalability.com/blog/2012/3/12/googleptamingptheplongplatencyp
tailpwhenpmorepmachinespequal.html-
• Performance$Analysis$Methodology$by-Brendan-Gregg-
– hOp://www.brendangregg.com/methodology.html-
• Silverman’s$Mode$Detec@on$Method-by-MaO-Adereth-
– hOp://adereth.github.io/blog/2014/10/12/silvermanspmodepdetecFonp
methodpexplained/-
• How$Not$To$Measure$System$Performance-by-James-Bornholt$
– hOps://homes.cs.washington.edu/~bornholt/post/performancep
evaluaFon.html-
• Trust$No$One,$Not$Even$Performance$Counters-by-Paul-Khuong$
– hDp://www.pvk.ca/Blog/2014/10/19/performancePop@misa@onP~Pwri@ngPanP
essay/#trustPnoPone$

81. Followpup-Material-
hOp://wwwpplan.cs.colorado.edu/diwan/asplos09.pdf-

82. Followpup-Material-
• List-of-media-for-learning-more-about-measurement-bias-in-
system-benchmarks:-
hOps://gist.github.com/aysylu/58ab5d67314d684a7f4c-
-

83. Takeaway-#1:-Cache-

84. Takeaway-#2:-Outliers-

85. Takeaway-#3:-Workload-

86. Benchmarking:-
You’re Doing It Wrong
Aysylu-Greenberg-
@aysylu22-