This document provides tips for benchmarking MongoDB databases. It recommends creating realistic benchmarks that mimic real workloads, executing multiple runs to eliminate outliers, and methodically making single changes between runs such as varying memory or compression settings. Key steps are to measure system metrics, publish detailed results and methods so others can reproduce them, and draw conclusions by comparing different database versions or configurations. Overall it presents benchmarking as a way to gain recognition in MongoDB "bug hunts" and provides resources for learning to benchmark databases professionally.

1. Benchmarking
MongoDB
For Fame and Fortune
Tim Callaghan : @acmebench

2. Benchmarking
MongoDB
For Fame and Fortune
Tim Callaghan : @acmebench

3. Who am I?
● Professional technical benchmarker at
VoltDB (09-11) and Tokutek (11-15)
● Amateur technical benchmarker at www.
acmebenchmarking.com
● Lifelong fan of fast databases and
performance engineering
● Apologies for the number of

4. Where is the Fortune?
MongoDB “bug hunts” for each major release
- Free MongoDB/World Pass
- Reserved front-row keynote seat
- Amazon gift card

5. Google AdWords/Blog?

6. 3 Steps of Benchmarking
1. Create the benchmark
2. Execute the benchmark
3. Publish your results (and explain them)
www.acmebenchmarking.com/2015/02/how-to-benchmark-mongodb.html

7. Create the Benchmark
● Should be something people can understand
○ “Real-world”
○ Better yet, mimic your workload
■ We need a lot more of this!
○ Be prepared to explain in detail
● Open source it!
○ www.github.com/tmcallaghan
○ Sysbench and iiBench for MongoDB

8. Execute the Benchmark
● Measure everything
○ CPU, RSS, IO, mongostat, db.serverStatus()
● Multiple runs are key
○ 2 out of 3 (eliminate bad results)
● Make 1 change, rerun
○ Memory, compression type, data size
● Lather, rinse, repeat

9. Pro-tips for Benchmarking
● Use Linux (sorry Windows)
● Get good at bash
● Learn from others
○ www.acmebenchmarking.com/2015/02/how-to-benchmark-mongodb.html

10. Publish Your Results
● Start a blog
● Describe the benchmark in detail
● Include steps to reproduce
● Come to a conclusion
○ Make some sort of comparison
○ MongoDB v2.6 vs. 3.0
○ WiredTiger vs. MMAPv1
○ DirectIO vs. BufferedIO

11. Getting Started
● Good blogs
○ www.acmebenchmarking.com
○ http://smalldatum.blogspot.com/
○ https://www.percona.com/blog/
● Google “mongodb benchmark” or “mysql benchmark”
● Get some good (and cheap) hardware
○ Desktops are fine
○ www.acmebenchmarking.com/2015/04/how-to-purchase-benchmarking-hardware.html

12. Questions?
Tim Callaghan
@acmebench
Deck available at http://www.slideshare.net/tmcallaghan