This document summarizes the results of a benchmark comparing the performance of several cloud database systems, including Cassandra, HBase, Sherpa, and MySQL. The benchmark uses a standard workload and measures key metrics like latency and throughput. Overall, Cassandra showed strong write performance but weaker reads. Sherpa delivered good read and write latency as well as high throughput. HBase read latency was poor. Later versions of Cassandra showed performance improvements over earlier versions.

1. Yahoo! Cloud Serving Benchmark
Overview and results – February 3, 2010
Brian F. Cooper
cooperb@yahoo-inc.com
Joint work with Adam Silberstein, Erwin Tam, Raghu Ramakrishnan and Russell Sears
System setup and tuning assistance from members of the Cassandra and HBase
committers, and the Sherpa engineering team
1

2. Versions of this deck
• V4.1 – Original set of results from
benchmark
• V4.2 – added Cassandra 0.5 versus 0.4.2
comparison, Cassandra range query
results, and vary scan size results
2

3. Motivation
• There are many “cloud DB” and “nosql” systems out there
– Sherpa/PNUTS
– BigTable
• HBase, Hypertable, HTable
– Megastore
– Azure
– Cassandra
– Amazon Web Services
• S3, SimpleDB, EBS
– CouchDB
– Voldemort
– Dynomite
– Etc: Tokyo, Redis, MongoDB
• How do they compare?
– Feature tradeoffs
– Performance tradeoffs
– Not clear!
3

4. Goal
• Implement a standard benchmark
– Evaluate different systems on common workloads
– Focus on performance and scale out
• Future additions – availability, replication
• Artifacts
– Open source workload generator
– Experimental study comparing several systems
4

5. Benchmark tool
• Java application
– Many systems have Java APIs
– Other systems via HTTP/REST, JNI or some other solution
Command-line parameters
• DB to use
• Target throughput
• Number of threads
•…
Workload
YCSB client
Cloud DB
parameter file
DB client
• R/W mix
Client
• Record size
Workload threads
• Data set
executor
•…
Stats
Extensible: define new workloads
Extensible: define new workloads
Extensible: plug in new clients
Extensible: plug in new clients
5

6. Workloads
• Workload – particular combination of workload parameters, defining
one workload
– Defines read/write mix, request distribution, record size, …
– Two ways to define workloads:
• Adjust parameters to an existing workload (via properties file)
• Define a new kind of workload (by writing Java code)
• Experiment – running a particular workload on a particular hardware
setup to produce a single graph for 1 or N systems
– Example – vary throughput and measure latency while running a
workload against Cassandra and HBase
• Workload package – A collection of related workloads
– Example: CoreWorkload – a set of basic read/write workloads
6

7. Benchmark tiers
• Tier 1 – Performance
– For constant hardware, increase offered throughput
until saturation
– Measure resulting latency/throughput curve
– “Sizeup” in Wisconsin benchmark terminology
• Tier 2 – Scalability
– Scaleup – Increase hardware, data size and workload
proportionally. Measure latency; should be constant
– Elastic speedup – Run workload against N servers;
while workload is running att N+1th server; measure
timeseries of latencies (should drop after adding
server)
7

8. Test setup
• Setup
– Six server-class machines
• 8 cores (2 x quadcore) 2.5 GHz CPUs, 8 GB RAM, 6 x 146GB 15K RPM SAS drives in RAID 1+0,
Gigabit ethernet, RHEL 4
– Plus extra machines for clients, routers, controllers, etc.
– Cassandra 0.4.2
– HBase 0.20.2
– MySQL 5.1.32 organized into a sharded configuration
– Sherpa 1.8
– No replication; force updates to disk (except HBase, which does not yet support this)
• Workloads
– 120 million 1 KB records = 20 GB per server
– Reads retrieve whole record; updates write a single field
– 100 or more client threads
• Caveats
– Write performance would be improved for Sherpa, sharded MySQL and Cassandra with a
dedicated log disk
– We tuned each system as well as we knew how, with assistance from the teams of
developers
8

9. Workload A – Update heavy
• 50/50 Read/update
Workload A - Read latency Workload A - Update latency
90 80
80 70
Average read latency (ms)
70
Update latency (ms)
60
60
50
50
40
40
30
30
20 20
10 10
0 0
0 2000 4000 6000 8000 0 2000 4000 6000 8000
Throughput (ops/sec) Throughput (ops/sec)
Cassandra Hbase Sherpa MySQL Cassandra Hbase Sherpa MySQL
Comment: Cassandra is optimized for writes, and has better write latency. However, Sherpa
has pretty good write latency, comparable read latency, and comparable peak
throughput. HBase has good write latency because it does not sync updates to disk, at
the cost of lower durability; but read latency is very bad 9

10. Workload B – Read heavy
• 95/5 Read/update
Workload B - Read latency Workload B - Update latency
60 40
Average update latency (ms)
35
Average read latency (ms)
50
30
40
25
30 20
15
20
10
10
5
0 0
0 2000 4000 6000 8000 10000 0 2000 4000 6000 8000 10000
Throughput (operations/sec) Throughput (operations/sec)
Cassandra HBase Sherpa MySQL Cassandra Hbase Sherpa MySQL
Comment: Sherpa does very well here, with better read and write latency and peak
throughput than Cassandra, and better read latency and peak throughput than HBase.
Again HBase write latency is very low because of no disk syncs. Buffer pool architecture
is good for random reads. 10

11. Workload E – short scans
• Scans of 1-100 records of size 1KB
Workload E - Scan latency
120
100
Average scan latency (ms)
80
60
40
20
0
0 200 400 600 800 1000 1200 1400 1600
Throughput (operations/sec)
Hbase Sherpa Cassandra
Comment: HBase and Sherpa are roughly equivalent for latency and peak throughput,
even though HBase is “meant” for scans. Cassandra’s performance is poor, but the
development team notes that many optimizations still need to be done.
11

12. Workload E – range size
• Vary size of range scans
Range size versus latency (Workload E)
500
Average range scan latency (ms)
450
400
350
300
250
200
150
100
50
0
0 200 400 600 800 1000 1200 1400 1600 1800
Max range size (records)
Hbase Sherpa
Comment: For small ranges, queries are similar to random lookups; Sherpa is efficient for
random lokoups and does well. As range increases, HBase begins to perform better
since it is optimized for large scans 12

13. Scale-up
• Read heavy workload with varying hardware
Read latency during scale-up
35
30
Average read latency (ms)
25
20
15
10
5
0
0 2 4 6 8 10 12 14
Number of servers
Cassandra Hbase Sherpa
Comment: Sherpa scales well, with flat latency as system size increases.
Cassandra scales less well, with more P2P communication. HBase is very
unstable; 3 servers or less performs very poorly. More experiments are
needed to get more data points on these curves.
13

14. Elasticity
• Run a read-heavy workload on 3 servers; add a 4th
server after 5 minutes
Cassandra elastic read performance
8.2
8
7.8
Average read latency (ms)
7.6
7.4
7.2
7
6.8
6.6
0 10 20 30 40 50 60 70
Time (min)
Comment: Cassandra shows nice elasticity; after a fourth server is added,
average latency of requests quickly drops by 11% with little or no
disruption. 14

15. Elasticity
• Run a read-heavy workload on 3 servers; add a 4th
server after 5 minutes
Hbase elastic read performance (detail)
70
65
60
Average read latency (ms)
55
50
45
40
35
30
0 10 20 30 40 50 60 70
Time (min)
Comment: HBase initially exhibits a large latency spike, with some requests
taking as much as 1000 ms; then, latency settles down and eventually
becomes 12% lower than latency before adding the server. 15

16. Cassandra 0.5 Results
Workload A - Update heavy
90
80
70
Average latency (ms)
60
50
40
30
20
10
0
0 2000 4000 6000 8000 10000 12000 14000
Throughput (operations/sec)
Cas 0.5 Read Cas 0.5 Update Cas 0.4.2 Read Cas 0.4.2 Update
16

17. Cassandra 0.5 Results
Workload B - Read heavy
60
50
Average latency (ms)
40
30
20
10
0
0 1000 2000 3000 4000 5000 6000 7000 8000 9000
Throughput (operations/sec)
Cas 0.5 Read Cas 0.5 Update Cas 0.4.2 Read Cas 0.4.2 Update
17

18. For more information
• Contact: Brian Cooper (cooperb@yahoo-inc.com)
• Detailed writeup of benchmark:
http://www.brianfrankcooper.net/pubs/ycsb.pdf
• Open source YCSB tool coming soon
18