The document summarizes a benchmarking study conducted by Altoros Systems to compare the performance of Couchbase Server, MongoDB, and Cassandra. It outlines the benchmark goals of having a reproducible workload, using a realistic scenario, and comparing latency and throughput. It describes the benchmarking tools, scenario details involving data size, operations, and hardware configuration. Configuration details are provided for each database, including cluster specifications and parameter settings.

1. Benchmarking Couchbase Server
Renat Khasanshyn
CEO, Altoros Systems, Inc.
CouchConf 2012
September 21, 2012
Copyright © Altoros Systems, Inc. | CONFIDENTIAL

2. Presentation Outline
• Benchmark Goals
• Benchmark Design and Scenario
• Benchmarking Tools
• Benchmark Results
2
Copyright © Altoros Systems, Inc. | CONFIDENTIAL

3. 3
Copyright © Altoros Systems, Inc. | CONFIDENTIAL

4. About Altoros
• Software delivery acceleration specialist for big data application
implementation services
• 200+ employees globally (Eastern Europe, US, UK, Denmark, Norway)
• Big data practice areas:
 Advertising analytics
 Automated device analytics
 Big data warehouse
Customers
Partners
Implementation Partner
4
Copyright © Altoros Systems, Inc. | CONFIDENTIAL

5. Why Benchmark NoSQL technologies?
• All NoSQL technologies say they are “high
performance and scalable”
But this isn’t helpful to end users
• Performance needs to be measured for meaning full
workloads
⇒ To help users understand the performance characteristics of
databases those workloads
• So we decided to compare the commonly used
NoSQL databases
• MongoDB 2.2RC
• Cassandra 1.1.2
• Couchbase Server 2.0 - Recent Build
5
Copyright © Altoros Systems, Inc. | CONFIDENTIAL

6. Benchmark Goals
• Reproducible by anyone
– Open Source workload generator
• Focus on use case for which NoSQL typically
selected
• Use a realistic workload
– Simulate steady state of application running
– Meaningful data amounts & runtime
• Compare latency vs throughput
• Measure max throughput (for given scenario)
6
Copyright © Altoros Systems, Inc. | CONFIDENTIAL

7. Benchmarking Scenario
• For interactive web application
• Scalability and performance are the most common
requirements
• Typically leads to users selecting NoSQL over RDBMS
• The working set of data changes with time
• End users using the application change over time
• Example: every few hours, every few days, every few weeks
• There is more data available than memory (RAM)
• Replication is used for fault tolerance
• Real world data sizes
• Use EC2 as deployment platform
– Commonly used
– Easy to replicate results
7
Copyright © Altoros Systems, Inc. | CONFIDENTIAL

8. Benchmarking Scenario Details
Hardware
• 4 Amazon m1.xlarge instances for the NoSQL DBs
• 1 instance used as the client
Workload details
• Operations are a mix of C:R:U:D in the ratio 5:60:33:2
• Each document roughly 1.5-2K in size (15 fields * 100 bytes)
• 15 million active and 15 million replica documents
• Workload with sliding working set
• Load phase, warm-up phase, access phase
• Runtime of the access phase ~1 hour
• Latency measured for varying throughput - 3 times for each run
• Focus on transaction performance
– Latency
– Throughput
8
Copyright © Altoros Systems, Inc. | CONFIDENTIAL

9. What was measured?
• Latency • Throughput
• Round trip time taken • Throughput was varied
for a request to execute from 1K ops/sec to 25K
from the client to the ops/sec depending on
server and back NoSQL database
• Average, 95th and 99th • Max throughput was
percentile measured measured
• Why is this important? • Why is this important?
• You want your users to • You want your app to
have a great experience support hundreds of
• Not just an “average” thousands of users
one
Workloads are not rate limited, focused on
max throughput.
9
Copyright © Altoros Systems, Inc. | CONFIDENTIAL

10. YCSB
10
Copyright © Altoros Systems, Inc. | CONFIDENTIAL

11. Benchmark Implementation: YCSB
• Yahoo! team offered a “standard” benchmark
• Yahoo! Cloud Serving Benchmark (YCSB)
– Focus on database
– Focus on performance
• YCSB Client consists of 2 parts
– Workload generator
– Workload scenarios
11
Copyright © Altoros Systems, Inc. | CONFIDENTIAL

12. Why YCSB
• Open source
• Extensible
• Rich selection of connectors
• Azure, BigTable, Cassandra, CouchDB,
• Dynomite, GemFire, HBase, Hypertable,
• Infinispan, MongoDB, PNUTS, Redis,
• Connector for Sharded RDBMS (i.e. MySQL),
• Voldemort, GigaSpaces XAP
• We developed a few connectors
• Accumulo, Couchbase, Riak,
• Connector for Shared Nothing RDBMS (i.e. MySQL Cluster)
12
Copyright © Altoros Systems, Inc. | CONFIDENTIAL

13. How YCSB Works
13
Copyright © Altoros Systems, Inc. | CONFIDENTIAL

14. THE CONFIGURATIONS
14
Copyright © Altoros Systems, Inc. | CONFIDENTIAL

15. Cluster specification
Amazon m1.xlarge Instance
15 GB memory
4 virtual cores
4 EBS 50 GB volumes in RAID0
YCSB Client 64-bit Amazon Linux (CentOS binary compatible)
Amazon m1.xlarge Instances * 4
15 GB memory
4 virtual cores
4 EBS 50 GB volumes in RAID0
64-bit Amazon Linux
* Extra nodes for masters, routers, etc
15
Copyright © Altoros Systems, Inc. | CONFIDENTIAL

16. Couchbase Configuration
• 4 node Couchbase cluster
• 1 replica setting
• Each node has some active and some replica
data
• 12GB used as the (12288 MB) Couchbase
bucket size per node
16
Copyright © Altoros Systems, Inc. | CONFIDENTIAL

17. MongoDB Configation
• 4 shards each has 1 replica (replication factor – 1),
where each shard is a set of 2 nodes - primary and
secondary
• Journaling disabled (trying to maximize performance)
• var shards = [
"shard1/ycsb-node1:27017,ycsb-node2:27018",
"shard2/ycsb-node2:27017,ycsb-node1:27018",
"shard3/ycsb-node3:27017,ycsb-node4:27018",
"shard4/ycsb-node4:27017,ycsb-node3:27018"];
Each node running
• 2 mongod processes (all together 8 mongod
processes on 4 nodes)
• 4 mongos processes, which is the MongoDB router,
process on 27019 port
17
Copyright © Altoros Systems, Inc. | CONFIDENTIAL

18. Cassandra Configuration
• Cassandra JVM settings:
• 1.1) MAX_HEAP_SIZE, which is a total amount of
memory dedicated to the Java heap - 6G
• 1.2) HEAP_NEWSIZE, total amount of memory for the
new generation of objects - 400M
• Cassandra settings:
• 2.1) RandomPartitioner was used which distributes
rows across the cluster evenly by MD5
• 2.2) Memtable size 4048 MB
18
Copyright © Altoros Systems, Inc. | CONFIDENTIAL

19. THE RESULTS
19
Copyright © Altoros Systems, Inc. | CONFIDENTIAL

20. Reads (Average time)
Read latencies against throughput
7
6
Cassandra
5
Average Latency [ms]
4
MongoDB
3
2
1
Couchbase
0
0 2000 4000 6000 8000 10000 12000 14000 16000 18000 20000 22000
Operations per Second
20
Copyright © Altoros Systems, Inc. | CONFIDENTIAL

21. Reads (95th percentile)
18
Read latencies against throughput
16
14
Cassandra
12
95th Percentile Latency [ms]
10
8
6
4 Couchbase
2
MongoDB
0
0 2000 4000 6000 8000 10000 12000 14000 16000 18000 20000 22000
Operations per Second
21
Copyright © Altoros Systems, Inc. | CONFIDENTIAL

22. Reads (99th percentile)
60
Read latencies against throughput
50
Cassandra
40
99th Percentile Latency [ms]
30 MongoDB
20
10
Couchbase
0
0 2000 4000 6000 8000 10000 12000 14000 16000 18000 20000 22000
Operations per Second
22
Copyright © Altoros Systems, Inc. | CONFIDENTIAL

23. Mongo Replica Reads
• MongoDB setup had 4 shards
• By default only masters will service reads
• To allow replica reads and still be comparable, need to
ensure that replica data is up-to-date
• This was done using write-concern (REPLICAS_SAFE)
• Tests showed that results did not improve
• This includes results for writes
23
Copyright © Altoros Systems, Inc. | CONFIDENTIAL

24. Writes (Average time)
5
Insert and Update latencies against throughput
4.5
4
MongoDB
3.5
Average Latency [ms]
3 Cassandra
2.5
2
1.5
1
Couchbase
0.5
0
0 2000 4000 6000 8000 10000 12000 14000 16000 18000 20000 22000
Operations per second
24
Copyright © Altoros Systems, Inc. | CONFIDENTIAL

25. Writes (95th percentile)
30
Insert and update latencies against throughput
25
MongoDB
95th Percentile Latency [ms]
20
15
Cassandra
10
Couchbase
5
0
0 2000 4000 6000 8000 10000 12000 14000 16000 18000 20000 22000
Operations per Second
25
Copyright © Altoros Systems, Inc. | CONFIDENTIAL

26. Writes (99th percentile)
50
Insert and update latencies against throughput
45
40
MongoDB
35
99th Percentile Latency [ms]
30
25
Cassandra
20
15
10
5
Couchbase
0
0 2000 4000 6000 8000 10000 12000 14000 16000 18000 20000 22000
Operations per Second
26
Copyright © Altoros Systems, Inc. | CONFIDENTIAL

27. Results Analysis
• Couchbase
• Showed the lowest latencies & highest throughput
• Latency was independent of throughput for up to 3/4th the max
achievable throughput (for both reads and write)
• Cassandra
• Had the highest latencies of all the databases
• Showed higher max throughput compared with mongoDB but only
60% of the throughput achieved by Couchbase
• Latencies rose fast as throughput was increased
• MongoDB
• Read latencies were better than Cassandra but higher than
Couchbase
• Max throughput for read and writes was the lowest of all the
databases
– Particularly for writes, high latencies seen for average throughput
– Coarse write lock seems to have a big impact on performance
27
Copyright © Altoros Systems, Inc. | CONFIDENTIAL

28. Other Thoughts
• You decide who is a winner
• NoSQL is a “different horses for different courses”
• Evaluate before choosing the “horse”
• Construct your own or use existing workloads
• Benchmark it
• Tune database!
• Benchmark it again
Amazon EC2 observations
• Scales perfectly for NoSQL
• EBS slows down database on reads
• RAID0 it! Use 4 disk in array (good choice), some reported
performance degraded with higher number (6 and >)
28
Copyright © Altoros Systems, Inc. | CONFIDENTIAL

29. What are we missing in our benchmarking scenario?
Load phase workload
• Working set is created
• 15 million records
• 1.5 KB record (15 fields by 100 Bytes)
• 45GB total or ≈12GB per node
Ideas, anyone?
29
Copyright © Altoros Systems, Inc. | CONFIDENTIAL

30. YCSB Connectors
github.com/Altoros/YCSB
30
Copyright © Altoros Systems, Inc. | CONFIDENTIAL

31. Workload Generator Specs
Hotspot generator with sliding window:
hotspotslidingspeed=10
Speed of the hot set window movement measured in keys per second, with a
default value of 10 keys/sec (can be overridden in workload properties file).
hotspotdatafraction=0.2
Proportion of the hot data set to the whole dataset, default is 0.2
hotspotoperationfraction=0.9
Value specifying how often hot dataset will be queried comparing to cold
dataset, default is 0.8, used 0.9
lowerbound=0
The minimal key value allowed to be queried. Set to 0
upperbound=15000000
The maximum key value allowed to be queried. Set to 15 million
Also specification of the client process, which drives workload:
6) threadcount=30
Number of parallel threads spawned on the client node to drive benchmark
31
Copyright © Altoros Systems, Inc. | CONFIDENTIAL

32. Thank you!
Thank You!
renat.k@altoros.com
@renatkhasanshyn
Tel. (650) 395-7002
32
Copyright © Altoros Systems, Inc. | CONFIDENTIAL