Yahoo Cloud Serving Benchmark

Yahoo! Cloud Serving Benchmark Overview and results – February 3, 2010 Brian F. Cooper cooperb@yahoo-inc.com Joint work with Adam Silberstein, Erwin Tam, Raghu Ramakrishnan and Russell Sears System setup and tuning assistance from members of the Cassandra and HBase committers, and the Sherpa engineering team 1

Versions of this deck • V4.1 – Original set of results from benchmark • V4.2 – added Cassandra 0.5 versus 0.4.2 comparison, Cassandra range query results, and vary scan size results 2

Motivation • There are many “cloud DB” and “nosql” systems out there – Sherpa/PNUTS – BigTable • HBase, Hypertable, HTable – Megastore – Azure – Cassandra – Amazon Web Services • S3, SimpleDB, EBS – CouchDB – Voldemort – Dynomite – Etc: Tokyo, Redis, MongoDB • How do they compare? – Feature tradeoffs – Performance tradeoffs – Not clear! 3

Goal • Implement a standard benchmark – Evaluate different systems on common workloads – Focus on performance and scale out • Future additions – availability, replication • Artifacts – Open source workload generator – Experimental study comparing several systems 4

Benchmark tool • Java application – Many systems have Java APIs – Other systems via HTTP/REST, JNI or some other solution Command-line parameters • DB to use • Target throughput • Number of threads •… Workload YCSB client Cloud DB parameter file DB client • R/W mix Client • Record size Workload threads • Data set executor •… Stats Extensible: define new workloads Extensible: define new workloads Extensible: plug in new clients Extensible: plug in new clients 5

Workloads • Workload – particular combination of workload parameters, defining one workload – Defines read/write mix, request distribution, record size, … – Two ways to define workloads: • Adjust parameters to an existing workload (via properties file) • Define a new kind of workload (by writing Java code) • Experiment – running a particular workload on a particular hardware setup to produce a single graph for 1 or N systems – Example – vary throughput and measure latency while running a workload against Cassandra and HBase • Workload package – A collection of related workloads – Example: CoreWorkload – a set of basic read/write workloads 6

Benchmark tiers • Tier 1 – Performance – For constant hardware, increase offered throughput until saturation – Measure resulting latency/throughput curve – “Sizeup” in Wisconsin benchmark terminology • Tier 2 – Scalability – Scaleup – Increase hardware, data size and workload proportionally. Measure latency; should be constant – Elastic speedup – Run workload against N servers; while workload is running att N+1th server; measure timeseries of latencies (should drop after adding server) 7

Test setup • Setup – Six server-class machines • 8 cores (2 x quadcore) 2.5 GHz CPUs, 8 GB RAM, 6 x 146GB 15K RPM SAS drives in RAID 1+0, Gigabit ethernet, RHEL 4 – Plus extra machines for clients, routers, controllers, etc. – Cassandra 0.4.2 – HBase 0.20.2 – MySQL 5.1.32 organized into a sharded configuration – Sherpa 1.8 – No replication; force updates to disk (except HBase, which does not yet support this) • Workloads – 120 million 1 KB records = 20 GB per server – Reads retrieve whole record; updates write a single field – 100 or more client threads • Caveats – Write performance would be improved for Sherpa, sharded MySQL and Cassandra with a dedicated log disk – We tuned each system as well as we knew how, with assistance from the teams of developers 8

Workload A – Update heavy • 50/50 Read/update Workload A - Read latency Workload A - Update latency 90 80 80 70 Average read latency (ms) 70 Update latency (ms) 60 60 50 50 40 40 30 30 20 20 10 10 0 0 0 2000 4000 6000 8000 0 2000 4000 6000 8000 Throughput (ops/sec) Throughput (ops/sec) Cassandra Hbase Sherpa MySQL Cassandra Hbase Sherpa MySQL Comment: Cassandra is optimized for writes, and has better write latency. However, Sherpa has pretty good write latency, comparable read latency, and comparable peak throughput. HBase has good write latency because it does not sync updates to disk, at the cost of lower durability; but read latency is very bad 9

Workload B – Read heavy • 95/5 Read/update Workload B - Read latency Workload B - Update latency 60 40 Average update latency (ms) 35 Average read latency (ms) 50 30 40 25 30 20 15 20 10 10 5 0 0 0 2000 4000 6000 8000 10000 0 2000 4000 6000 8000 10000 Throughput (operations/sec) Throughput (operations/sec) Cassandra HBase Sherpa MySQL Cassandra Hbase Sherpa MySQL Comment: Sherpa does very well here, with better read and write latency and peak throughput than Cassandra, and better read latency and peak throughput than HBase. Again HBase write latency is very low because of no disk syncs. Buffer pool architecture is good for random reads. 10

Workload E – short scans • Scans of 1-100 records of size 1KB Workload E - Scan latency 120 100 Average scan latency (ms) 80 60 40 20 0 0 200 400 600 800 1000 1200 1400 1600 Throughput (operations/sec) Hbase Sherpa Cassandra Comment: HBase and Sherpa are roughly equivalent for latency and peak throughput, even though HBase is “meant” for scans. Cassandra’s performance is poor, but the development team notes that many optimizations still need to be done. 11

Workload E – range size • Vary size of range scans Range size versus latency (Workload E) 500 Average range scan latency (ms) 450 400 350 300 250 200 150 100 50 0 0 200 400 600 800 1000 1200 1400 1600 1800 Max range size (records) Hbase Sherpa Comment: For small ranges, queries are similar to random lookups; Sherpa is efficient for random lokoups and does well. As range increases, HBase begins to perform better since it is optimized for large scans 12

Scale-up • Read heavy workload with varying hardware Read latency during scale-up 35 30 Average read latency (ms) 25 20 15 10 5 0 0 2 4 6 8 10 12 14 Number of servers Cassandra Hbase Sherpa Comment: Sherpa scales well, with flat latency as system size increases. Cassandra scales less well, with more P2P communication. HBase is very unstable; 3 servers or less performs very poorly. More experiments are needed to get more data points on these curves. 13

Elasticity • Run a read-heavy workload on 3 servers; add a 4th server after 5 minutes Cassandra elastic read performance 8.2 8 7.8 Average read latency (ms) 7.6 7.4 7.2 7 6.8 6.6 0 10 20 30 40 50 60 70 Time (min) Comment: Cassandra shows nice elasticity; after a fourth server is added, average latency of requests quickly drops by 11% with little or no disruption. 14

Elasticity • Run a read-heavy workload on 3 servers; add a 4th server after 5 minutes Hbase elastic read performance (detail) 70 65 60 Average read latency (ms) 55 50 45 40 35 30 0 10 20 30 40 50 60 70 Time (min) Comment: HBase initially exhibits a large latency spike, with some requests taking as much as 1000 ms; then, latency settles down and eventually becomes 12% lower than latency before adding the server. 15

Cassandra 0.5 Results Workload A - Update heavy 90 80 70 Average latency (ms) 60 50 40 30 20 10 0 0 2000 4000 6000 8000 10000 12000 14000 Throughput (operations/sec) Cas 0.5 Read Cas 0.5 Update Cas 0.4.2 Read Cas 0.4.2 Update 16

Cassandra 0.5 Results Workload B - Read heavy 60 50 Average latency (ms) 40 30 20 10 0 0 1000 2000 3000 4000 5000 6000 7000 8000 9000 Throughput (operations/sec) Cas 0.5 Read Cas 0.5 Update Cas 0.4.2 Read Cas 0.4.2 Update 17

For more information • Contact: Brian Cooper (cooperb@yahoo-inc.com) • Detailed writeup of benchmark: http://www.brianfrankcooper.net/pubs/ycsb.pdf • Open source YCSB tool coming soon 18

Yahoo Cloud Serving Benchmark

by Han Kevin on Mar 18, 2010

Accessibility

Tags

Upload Details

Usage Rights

1 Embed 33

Statistics

Yahoo Cloud Serving Benchmark — Presentation Transcript