The document discusses benchmarking tools for databases and introduces a new benchmarking tool called Firehose. It describes some issues with existing tools like Mongoimport and YCSB in that they do not adequately model real-world workloads. Firehose is presented as a new multi-threaded benchmarking tool that aims to have high relevance to real applications. It measures performance metrics like operation durations and supports features like configurable load levels and integration with monitoring systems.

1. Instrument Your Code
{ Name: ‘Bryan Reinero’,
Title: ‘Developer Advocate’,
Twitter: ‘@blimpyacht’,
Email: ‘bryan@mongdb.com’ }

2. 2
Benchmarking
http://photos1.blogger.com/blogger/4051/403/1600/Better%20Shockwave.1.jpg

3. 3
Benchmarking
Mongoimport
- Single threaded
- Poor type conversions
- Only intended to be a convenience tool
- little relevance to real world workload
http://photos1.blogger.com/blogger/4051/403/1600/Better%20Shockwave.1.jpg

4. 4
Benchmarking
Mongoimport
- Single threaded
- Poor type conversions
- Only intended to be a convenience tool
- little relevance to real world workload
YCSB
- Pseudo de-facto benchmark
- Runs batch and report style
- Little relevance to real world workload
workload
http://photos1.blogger.com/blogger/4051/403/1600/Better%20Shockwave.1.jpg

5. 5
Benchmarking
Mongoimport
- Single threaded
- Poor type conversions
- Only intended to be a convenience tool
- little relevance to real world workload
YCSB
- Pseudo de-facto benchmark
- Runs batch and report style
- Little relevance to real world workload
Roll Your Own
- Coded goofy
- Prove of Concept only
- Lots of hardcoded cruft
- little relevancy to real world workload
http://photos1.blogger.com/blogger/4051/403/1600/Better%20Shockwave.1.jpg

6. 6
Load Testing Requirements
- Multi-threaded
- Easy to implement
- Easy to aggregate metrics
- Easy to change test detents
- High relevance to real world
workload

7. 7
Introducing Firehose!

8. 8
https://github.com/breinero/Firehose

9. 9
https://github.com/breinero/Firehose
Components
• WorkerPool
• Command Line Interface
• Markov
• Metrics

10. 10
https://github.com/breinero/Firehose
Components
• Worker Pool
• Command Line Interface
• Markov
• Metrics

11. 11
Instrumenting a Task
Interval insertDuration = samples.set("insert");
dao.insert( object );
insertDuration.mark();

12. 12
Instrumenting a Task
• Start the clock
Interval insertDuration = samples.set("insert");
dao.insert( object );
insertDuration.mark();

13. 13
Instrumenting a Task
• Start the clock
• Perform the work
Interval insertDuration = samples.set("insert");
dao.insert( object );
insertDuration.mark();

14. 14
Instrumenting a Task
• Start the clock
• Perform the work
• Stop the clock
Interval insertDuration = samples.set("insert");
dao.insert( object );
insertDuration.mark();

15. 15
Interval insertDuration =
samples.set("insert");
//Get an Interval
public Interval set( String name ) {
return new Interval( name, this );
}

16. 16
Interval insertDuration =
samples.set("insert");
//Get an Interval
public Interval set( String name ) {
return new Interval( name, this );
}
// Interval Constructor
Interval ( String name, SampleSet set ) {
this.name = name;
this.set = set;
}

17. 17
insertDuration.mark();
//Get an Interval
public Interval set( String name ) {
return new Interval( name, this );
}
// Interval Constructor
Interval ( String name, SampleSet set ) {
this.name = name;
this.set = set;
}
//Stop the clock
public void mark() {
end = System.nanoTime();
inception = System.currentTimeMillis();
set.add(this);
}

18. 18
insertDuration.mark();
//Get an Interval
public Interval set( String name ) {
return new Interval( name, this );
}
// Interval Constructor
Interval ( String name, SampleSet set ) {
this.name = name;
this.set = set;
}
//Stop the clock
public void mark() {
end = System.nanoTime();
inception = System.currentTimeMillis();
set.add(this);
}
• Time of completion

19. 19
insertDuration.mark();
//Get an Interval
public Interval set( String name ) {
return new Interval( name, this );
}
// Interval Constructor
Interval ( String name, SampleSet set ) {
this.name = name;
this.set = set;
}
//Stop the clock
public void mark() {
end = System.nanoTime();
inception = System.currentTimeMillis();
set.add(this);
}
• Time of completion
private final Long start
= System.nanoTime();

20. 20
insertDuration.mark();
//Get an Interval
public Interval set( String name ) {
return new Interval( name, this );
}
// Interval Constructor
Interval ( String name, SampleSet set ) {
this.name = name;
this.set = set;
}
//Stop the clock
public void mark() {
end = System.nanoTime();
inception = System.currentTimeMillis();
set.add(this);
}

21. 21
SampleSet Class
public class SampleSet {
private final DelayQueue<Interval> intervals = new DelayQueue<Interval>();
private final Cleanser cleanser = new Cleanser();
private AtomicBoolean running = new AtomicBoolean(true);
private Long timeToLive = 1000L;

22. 40 µs
42 µs
38 µs
39 µs
42 µs
40 µs
38 µs
38 µs
41 µs
39 µs
40 µs
42 µs
Sampling

23. 40 µs
42 µs
38 µs
39 µs
42 µs
40 µs
38 µs
38 µs
41 µs
39 µs
40 µs
42 µs
Sampling
insert: {
mean: 170.2857,
median: 158.0,
std: 39.3636,
count: 49,
total: 8344.0
}

24. 40 µs
42 µs
38 µs
39 µs
42 µs
40 µs
38 µs
38 µs
41 µs
39 µs
40 µs
42 µs
Sampling

25. 40 µs
42 µs
38 µs
39 µs
42 µs
40 µs
38 µs
38 µs
41 µs
39 µs
40 µs
42 µs
Sampling Interval
38 µs
37 µs
Time to live
controls
sampling
interval

26. Composite Operations
{
threads: 2,
"lines read": 100000,
samples: {
units: "microseconds",
"reporting interval ms": 5,
total: {
mean: 221.32,
median: 181.5,
std: 82.68293015354203,
count: 50,
total: 11066.0
},
readline: {
mean: 4.160000000000002,
median: 1.0,
std: 14.412806097073416,
count: 50,
total: 208.0
},
build: {
mean: 14.291666666666666,
median: 15.0,
std: 3.439188012365297,
count: 48,
total: 686.0
},
insert: {
mean: 170.28571428571428,
median: 158.0,
std: 39.36368885152915,
count: 49,
total: 8344.0
}
}
}
t
total
read build insert

27. Demo

28. 28
Save your load tests!
Load Generators
Application
Metrics
• Firehose
• Top
Database
Metrics
• Mongostat
• iostat

29. 29
Save your load tests [properly] !
Load Generators
Application
Metrics
• Firehose
• Top
Database
Metrics
• Mongostat
• iostat

30. 30
Monitoring with JMX
Application
Servers
L.B.
JConsole
Java Management Extensions (JMX)

31. 31
Activating JMX
$ java -Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9001
-Dcom.sun.management.jmxremote.authenticate=false
-jar target/Firehose-0.1.1.one-jar.jar
-ri 500 -n 1000 -ns test.firehose

32. 32
Activating JMX
$ java -Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9001
-Dcom.sun.management.jmxremote.authenticate=false
-jar target/Firehose-0.1.1.one-jar.jar
-ri 500 -n 1000 -ns test.firehose

33. 33
Mbean Interface
public interface StatisticsMBean {
public String report();
public String report( String metric );
public void setReportingInterval( Long milliseconds );
public Long getReportingInterval();
}

34. 34
Another Demo

36. Sizing
By Evan-Amos (Own work) [CC BY-SA 3.0 (http://creativecommons.org/licenses/by-sa/3.0) or GFDL (http://www.gnu.org/copyleft/fdl.html)], via Wikimedia Commons

37. Sizing
By Evan-Amos (Own work) [CC BY-SA 3.0 (http://creativecommons.org/licenses/by-sa/3.0) or GFDL (http://www.gnu.org/copyleft/fdl.html)], via Wikimedia Commons

38. Sizing

39. Sizing
Photograph © Andrew Dunn, 9 November 2005. Website: http://www.andrewdunnphoto.com/

40. Sizing
What is the expected workload?
• Read or write heavy?
• Access patterns?
• Frequency of operations?
• Required service level agreements?
SCHEMA DESIGN

41. Always Be Baselining
http://goruco.com/speakers/2015/eileen-uchitelle/

42. 42
Planning for Growth
0
1
2
3
4
5
6
7
8
9
1 2 3 4 5 6 7 8 9 10
Load
1K Ops / Second
t

43. 43
Planning for Growth
0
2
4
6
8
10
12
1 2 3 4 5 6 7 8 9 10
Saturation
Load
1K Ops / Second
t

44. 44
Planning for Growth
0
2
4
6
8
10
12
1 2 3 4 5 6 7 8 9 10
Saturation
Load
1K Ops / Second
t
• Test it like you use it. Benchmarks don’t count
• Test to failure

45. 45
Planning for Growth
0
2
4
6
8
10
12
1 2 3 4 5 6 7 8 9 10
Saturation
Warn
Load
• Test it like you use it. Benchmarks don’t count
• Test to failure
1K Ops / Second

46. 46
Production Ready Architecture
L.B.

47. 47
Production Ready Architecture
L.B.
Unindexed queries

48. 48
Production Ready Architecture
L.B.
Unindexed queries Leads to collection scans

49. 49
Production Ready Architecture
L.B.
Unindexed queries Leads to collection scans
Results in high latencies

50. 50
Classic Failure Scenario
L.B.
Unindexed queries Leads to collection scans
Results in high latenciesCauses memory exhaustion

51. 51
Production Ready Architecture
L.B.
Unindexed queries Leads to collection scans
Results in high latenciesCauses memory exhaustion
CASCADING FAILURE

52. 52
https://github.com/breinero/Firehose
Components
• WorkerPool
• Command Line Interface
• Markov
• Metrics
• Circuit Breaker

53. 53
Circuit Breaker
Trigger Conditions
• Latency stats.getMean() >= max
• OpsPerSecond stats.getN() >= max
• ConcurrentOperations stats.getN()*stats.getMean() >= max

54. 54
Circuit Breaker
Trigger Conditions
• Latency stats.getMean() >= max
• OpsPerSecond stats.getN() >= max
• ConcurrentOperations stats.getN()*stats.getMean() >= max
Use in conjunction with MongoDB Settings

55. 55
{ breakers: [
{ name: "foo", tripped: false,
thresholds: [
{ opsPerSec: 5000.0 },
{ latency: 1000.0 },
{ concurrency: 50.0 }
]
},
{ name: "bar", tripped: false,
thresholds: [
{ latency: 1000.0 },
{ concurrency: 50.0 }
]
}
]
}
Circuit Breaker Configuration

56. 56
Circuit Breaker Configuration
{ breakers: [
{ name: "foo", tripped: false,
thresholds: [
{ opsPerSec: 5000.0 },
{ latency: 1000.0 },
{ concurrency: 50.0 }
]
},
{ name: "bar", tripped: false,
thresholds: [
{ latency: 1000.0 },
{ concurrency: 50.0 }
]
}
]
}
// Start the clock
Interval fooOp = samples.set(”foo");
// foo does it’s work
dao.insert( someObject );
// stop the clock
fooOp.mark();

57. 57
Yet Another Demo

58. 58
Think about YOUR requirements & Specs
Make a DevOps Contract
• Database Access Requirements
• Database Access Fulfillment Specification
• Cluster Configuration
• Monitoring and Alerting Specification

59. 59
Required Reading
http://www.amazon.com/Release-It-Production-Ready-Pragmatic-Programmers/dp/0978739213

60. 60
Thanks!
{ name: ‘Bryan Reinero’,
title: ‘Developer Advocate’,
twitter: ‘@blimpyacht’,
code: ‘github.com/breinero’
email: ‘bryan@mongdb.com’ }