Yahoo Cloud Serving Benchmark
Upcoming SlideShare
Loading in...5
×
 

Yahoo Cloud Serving Benchmark

on

  • 6,244 views

 

Statistics

Views

Total Views
6,244
Views on SlideShare
6,113
Embed Views
131

Actions

Likes
6
Downloads
88
Comments
0

2 Embeds 131

http://www.scoop.it 98
http://www.slideshare.net 33

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Yahoo Cloud Serving Benchmark Yahoo Cloud Serving Benchmark Presentation Transcript

  • Yahoo! Cloud Serving Benchmark Overview and results – February 3, 2010 Brian F. Cooper cooperb@yahoo-inc.com Joint work with Adam Silberstein, Erwin Tam, Raghu Ramakrishnan and Russell Sears System setup and tuning assistance from members of the Cassandra and HBase committers, and the Sherpa engineering team 1
  • Versions of this deck • V4.1 – Original set of results from benchmark • V4.2 – added Cassandra 0.5 versus 0.4.2 comparison, Cassandra range query results, and vary scan size results 2
  • Motivation • There are many “cloud DB” and “nosql” systems out there – Sherpa/PNUTS – BigTable • HBase, Hypertable, HTable – Megastore – Azure – Cassandra – Amazon Web Services • S3, SimpleDB, EBS – CouchDB – Voldemort – Dynomite – Etc: Tokyo, Redis, MongoDB • How do they compare? – Feature tradeoffs – Performance tradeoffs – Not clear! 3
  • Goal • Implement a standard benchmark – Evaluate different systems on common workloads – Focus on performance and scale out • Future additions – availability, replication • Artifacts – Open source workload generator – Experimental study comparing several systems 4
  • Benchmark tool • Java application – Many systems have Java APIs – Other systems via HTTP/REST, JNI or some other solution Command-line parameters • DB to use • Target throughput • Number of threads •… Workload YCSB client Cloud DB parameter file DB client • R/W mix Client • Record size Workload threads • Data set executor •… Stats Extensible: define new workloads Extensible: define new workloads Extensible: plug in new clients Extensible: plug in new clients 5
  • Workloads • Workload – particular combination of workload parameters, defining one workload – Defines read/write mix, request distribution, record size, … – Two ways to define workloads: • Adjust parameters to an existing workload (via properties file) • Define a new kind of workload (by writing Java code) • Experiment – running a particular workload on a particular hardware setup to produce a single graph for 1 or N systems – Example – vary throughput and measure latency while running a workload against Cassandra and HBase • Workload package – A collection of related workloads – Example: CoreWorkload – a set of basic read/write workloads 6
  • Benchmark tiers • Tier 1 – Performance – For constant hardware, increase offered throughput until saturation – Measure resulting latency/throughput curve – “Sizeup” in Wisconsin benchmark terminology • Tier 2 – Scalability – Scaleup – Increase hardware, data size and workload proportionally. Measure latency; should be constant – Elastic speedup – Run workload against N servers; while workload is running att N+1th server; measure timeseries of latencies (should drop after adding server) 7
  • Test setup • Setup – Six server-class machines • 8 cores (2 x quadcore) 2.5 GHz CPUs, 8 GB RAM, 6 x 146GB 15K RPM SAS drives in RAID 1+0, Gigabit ethernet, RHEL 4 – Plus extra machines for clients, routers, controllers, etc. – Cassandra 0.4.2 – HBase 0.20.2 – MySQL 5.1.32 organized into a sharded configuration – Sherpa 1.8 – No replication; force updates to disk (except HBase, which does not yet support this) • Workloads – 120 million 1 KB records = 20 GB per server – Reads retrieve whole record; updates write a single field – 100 or more client threads • Caveats – Write performance would be improved for Sherpa, sharded MySQL and Cassandra with a dedicated log disk – We tuned each system as well as we knew how, with assistance from the teams of developers 8
  • Workload A – Update heavy • 50/50 Read/update Workload A - Read latency Workload A - Update latency 90 80 80 70 Average read latency (ms) 70 Update latency (ms) 60 60 50 50 40 40 30 30 20 20 10 10 0 0 0 2000 4000 6000 8000 0 2000 4000 6000 8000 Throughput (ops/sec) Throughput (ops/sec) Cassandra Hbase Sherpa MySQL Cassandra Hbase Sherpa MySQL Comment: Cassandra is optimized for writes, and has better write latency. However, Sherpa has pretty good write latency, comparable read latency, and comparable peak throughput. HBase has good write latency because it does not sync updates to disk, at the cost of lower durability; but read latency is very bad 9
  • Workload B – Read heavy • 95/5 Read/update Workload B - Read latency Workload B - Update latency 60 40 Average update latency (ms) 35 Average read latency (ms) 50 30 40 25 30 20 15 20 10 10 5 0 0 0 2000 4000 6000 8000 10000 0 2000 4000 6000 8000 10000 Throughput (operations/sec) Throughput (operations/sec) Cassandra HBase Sherpa MySQL Cassandra Hbase Sherpa MySQL Comment: Sherpa does very well here, with better read and write latency and peak throughput than Cassandra, and better read latency and peak throughput than HBase. Again HBase write latency is very low because of no disk syncs. Buffer pool architecture is good for random reads. 10
  • Workload E – short scans • Scans of 1-100 records of size 1KB Workload E - Scan latency 120 100 Average scan latency (ms) 80 60 40 20 0 0 200 400 600 800 1000 1200 1400 1600 Throughput (operations/sec) Hbase Sherpa Cassandra Comment: HBase and Sherpa are roughly equivalent for latency and peak throughput, even though HBase is “meant” for scans. Cassandra’s performance is poor, but the development team notes that many optimizations still need to be done. 11
  • Workload E – range size • Vary size of range scans Range size versus latency (Workload E) 500 Average range scan latency (ms) 450 400 350 300 250 200 150 100 50 0 0 200 400 600 800 1000 1200 1400 1600 1800 Max range size (records) Hbase Sherpa Comment: For small ranges, queries are similar to random lookups; Sherpa is efficient for random lokoups and does well. As range increases, HBase begins to perform better since it is optimized for large scans 12
  • Scale-up • Read heavy workload with varying hardware Read latency during scale-up 35 30 Average read latency (ms) 25 20 15 10 5 0 0 2 4 6 8 10 12 14 Number of servers Cassandra Hbase Sherpa Comment: Sherpa scales well, with flat latency as system size increases. Cassandra scales less well, with more P2P communication. HBase is very unstable; 3 servers or less performs very poorly. More experiments are needed to get more data points on these curves. 13
  • Elasticity • Run a read-heavy workload on 3 servers; add a 4th server after 5 minutes Cassandra elastic read performance 8.2 8 7.8 Average read latency (ms) 7.6 7.4 7.2 7 6.8 6.6 0 10 20 30 40 50 60 70 Time (min) Comment: Cassandra shows nice elasticity; after a fourth server is added, average latency of requests quickly drops by 11% with little or no disruption. 14
  • Elasticity • Run a read-heavy workload on 3 servers; add a 4th server after 5 minutes Hbase elastic read performance (detail) 70 65 60 Average read latency (ms) 55 50 45 40 35 30 0 10 20 30 40 50 60 70 Time (min) Comment: HBase initially exhibits a large latency spike, with some requests taking as much as 1000 ms; then, latency settles down and eventually becomes 12% lower than latency before adding the server. 15
  • Cassandra 0.5 Results Workload A - Update heavy 90 80 70 Average latency (ms) 60 50 40 30 20 10 0 0 2000 4000 6000 8000 10000 12000 14000 Throughput (operations/sec) Cas 0.5 Read Cas 0.5 Update Cas 0.4.2 Read Cas 0.4.2 Update 16
  • Cassandra 0.5 Results Workload B - Read heavy 60 50 Average latency (ms) 40 30 20 10 0 0 1000 2000 3000 4000 5000 6000 7000 8000 9000 Throughput (operations/sec) Cas 0.5 Read Cas 0.5 Update Cas 0.4.2 Read Cas 0.4.2 Update 17
  • For more information • Contact: Brian Cooper (cooperb@yahoo-inc.com) • Detailed writeup of benchmark: http://www.brianfrankcooper.net/pubs/ycsb.pdf • Open source YCSB tool coming soon 18