Software is often improved incrementally. Each software optimization should be assessed with microbenchmarks. In a microbenchmark, we record performance measures such as elapsed time or instruction counts during specific tasks, often in idealized conditions. In principle, the process is easy: if the new code is faster, we adopt it. Unfortunately, there are many pitfalls, such as unrealistic statistical assumptions and poorly designed benchmarks. Abstractions like cloud computing add further challenges. We illustrate effective benchmarking practices with examples.

1. Accurateandefficientsoftwaremicrobenchmarks
DanielLemire
professor,DataScienceResearchCenter
UniversitéduQuébec(TÉLUQ)
Montreal
blog:https://lemire.me
twitter:@lemire
GitHub:https://github.com/lemire/

2. Background
FastestJSONparserintheworld(oncommodityprocessors):
https://github.com/simdjson/simdjson
FirsttoparseJSONfilesatgigabytespersecond

3. Whereisthecode?
Allcodeforthistalkisonline(reproducible!!!)
https://github.com/lemire/talks/tree/master/2023/performance/code

4. Howfastisyourdisk?
PCIe4drives:5GB/sreadingspeed(sequential)
PCIe5drives:10GB/sreadingspeed(sequential)

5. CPUFrequenciesarestagnating
architecture availability max.frequency
IntelSkylake 2015 4.5GHz
IntelIceLake 2019 4.1GHz

6. Fact
Single-coreprocessesareoftenCPUbound

7. Solution?
Optimizethesoftware.
Incrementaloptimization,howdoyouknowthatyouareontherighttrack?

8. Hypothesis
Thissoftwarechange(commit)improvesourperformance.

9. Simple
Measuretimeelapsedbefore,timeelapsedafter.

10. Complexsystem
Softwaresystemsarecomplexsystems:changescanhaveunexpectedconsequences.

11. JIT
VirtualMachineWarmupBlowsHotandCold

12. Systemcalls
Ssystemcalls(especiallyIO)maydominate,assumethattheyremainconstant.

13. Dataaccess
datastructurelayoutchangescantriggerexpensiveloads,assumethatwekeepthat
constant.

14. Tinyfunctions
Incertitudeprinciple:bymeasuringyouareaffectingtheexecutionsothatyoucannot
measuresafelytinyfunctions.

15. Takestaticallycompiledcode
TranscodingUTF-16toUTF-8ofan80kBArabicstringusingthesimdutflibrary(NEON
kernel).

16. Usetheaverage?
Let bethetruevalueandlet bethenoisedistribution(variance ).
Weseek .

17. Repeatedmeasuresincreaseaccuracy
Measuresare
Sumis .Varianceis .
Averageis .Varianceis .Standarddeviationof .

18. Simulation
mu, sigma = 10000, 5000
for N in range(20, 2000+1):
s = [sum(np.random.default_rng().normal(mu, sigma, N))/N for i in range(30)]
print(N,np.std(s))

20. Actualmeasurements
// returns the average
double transcode(const std::string& source, size_t iterations);
...
for(size_t i = iterations_start; i <= iterations_end; i+=step) {
std::vector<double> averages;
for(size_t j = 0; j < 30; j++) { averages.push_back(transcode(source, i)); }
std::cout << i << "t" << compute_std_dev(averages) << std::endl;
}

22. Sigmaevents

23. 1-sigmais32%
2-sigmais5%
3-sigmais0.3%(onceever300trials)
4-sigmais0.00669%(onceevery15000trials)
5-sigmais5.9e-05%(onceevery1,700,000trials)
6-sigmais2e-07%(onceevery500,000,000)
for

24. Measuringsigmaevents
Take300measuresafterwarmup,andmeasuretheworstrelativedeviation
$ for i in {1..10}; do sudo ./sigma_test; done
4.56151
4.904
7.43446
5.73425
9.89544
12.975
3.92584
3.14633
4.91766
5.3699

25. Whatifwedealtwithlog-normaldistributions?

26. for N in range(20, 2000+1):
s = [sum(np.random.default_rng().lognormal(1, 4, N))/N for i in range(30)]
print(N,np.std(s))

28. Whatifwemeasuredtheminimum?
Relativestandarddeviation( )
N average minimum
200 3.44% 1.38%
2000 2.66% 1.19%
10000 2.95% 1.27%

29. Theminimumiseasiertomeasureto1%accuracy.

30. CPUperformancecounters
Processorshavezero-overheadcountersrecordinginstructionretired,actualcycles,and
soforth.
NoneedtofreezetheCPUfrequency:youcanmeasureit.

31. Limitations
Youcanonlymeasuresomanythings(2,4metrics,not25)
Requiredprivilegedaccess(e.g.,root)

32. Countersinthecloud
x64:RequiresatleastafullCPU
ARMGraviton:generallyavailablebutlimitednumber(e.g.,2counters)

33. Instructioncountsareaccurate

34. Usingperformancecounters
Javainstructioncounters:https://github.com/jvm-profiling-tools/async-profiler
C/C++:instructioncountersareavailablethroughtheLinuxkernel
Goinstructioncounters

35. Generally,fewerinstructionsmeansfastercode
Someinstructionsaremoreexpensivethanothers(e.g.,division).
Datadependencycanmakeinstructioncountslessrelevant.
Branchingcanartificiallylowerinstructioncount.

36. Ifyouareaddingspeculativebranching,makesureyourtestinputislarge.
while (howmany != 0) {
val = random();
if( val is an odd integer ) {
out[index] = val;
index += 1;
}
howmany--;
}

37. 2000'random'elements,AMDRome
trial mispredictedbranches
1 50%
2 18%
3 6%
4 2%
5 1%
6 0.3%
7 0.15%
8 0.15%

38. Takeaway1
Computationalmicrobenchmarkscanhavelog-normaldistributions.
Considermeasuringtheminimuminsteadoftheaverage.

39. Takeaway2
Benchmarkingoftenisgood
Long-runningbenchmarksarenotnecessarilymoreaccurate.
Prefercheap,well-designedbenchmarks.

40. Links
Bloghttps://lemire.me/blog/
Twitter:@lemire
GitHub:https://github.com/lemire