Scaling with Apache Cassandra

Scaling with Cassandra Peter Schuller peter.schuller@inﬁdyne.com @scode

Apache Cassandra http://cassandra.apache.org A distributed fault-tolerant database

• Data model inspired by Google’s BigTable • Distribution model inspired by Amazon’s Dynamo • Open-sourced by Facebook in 2008 • Now developed by a team of committers with members and contributions from several companies • Used by Digg, Facebook, Twitter, Reddit, Rackspace, Cloudkick and others.

• No single point of failure • Add/replace/remove nodes without downtime • Durability is a choice • Consistency level is a choice (eventual -> strong) • Multiple data centers supported • Data model richer than key/value store • Whole-cluster reliability and operative ease

• Writes are faster than reads(!) • Designed to scale to data sizes that do not ﬁt in memory • Active set for reads primarily determines I/O load

• No ACID transactions • No ad-hoc queries • ... but would you get those if you sharded your relational database? • Less tool maturity

For the application programmer a major difference between a relational database and Cassandra is the data model RDBMS Cassandra server cluster database keyspace table column family row row column column (ad-hoc, per row)

• Each column is a key/value pair with a timestamp • Given two column inserts with the same key, greatest timestamp wins • Keep your clocks synchronized • Timestamp based conﬂict resolution integral to replication and eventual consistency • Writes are idempotent

Data model driven in part by performance characteristics of underlying storage

B-tree indexed table performance • B-tree access is seek bound • Writes require reads • Performance degrades considerably when tree no longer ﬁts in memory • Where is the data, if not in the leaf node? • Data scans in B-tree order is still random with respect to the data (unless data in leaf nodes) • INSERT INTO posts (id,post_time,author, body) VALUES (...) - how many seeks does it take?

The cost of seeking • Assume 7 ms avg seek time • Assume 80 MB/sec sequential write speed • Amortized time required for disk to write? • 100byte/8kb random: 7/7.1 ms (140 per second) • 100byte/8 kb sequential: 0.1 ms (800000/10000 per second) • What if we could write sequentially?

write memtable background ﬂush commit log sstable sstable sstable sstable

read write memtable background ﬂush commit log sstable sstable sstable sstable

read write memtable background ﬂush commit log sstable sstable sstable sstable background compaction sstable

read write memtable background ﬂush commit log sstable

sorted index, sample in memory <key 127> <row data 0> <key 255> <row data 1> ... ... <row data 127> ... <row data 255> ... sorted data ﬁle

D S Helen K

0 2^127*3/4 2^127/4 2^127/2

0 2^127*3/4 md5(“Helen”) = x 2^127/4 2^127/2

www. D S K

• All nodes keep token->node mappings • Communicated by gossiping • Changes originate with the nodes themselves (no central authority/arbiter)

bin/nodetool -host 10.176.0.146 ring Address Status Load Range Ring 10.176.0.146 Up 459.27 MB 75603446264197340449435394672681112420 |<--| 10.176.1.161 Up 382.53 MB 137462771597874153173150284137310597304 | | 10.176.1.162 Up 511.34 MB 63538518574533451921556363897953848387 |-->|

write req

+hint write req

How many nodes may be down before a write (or read) fails?

• Replication factor (RF) determines number of replicas (copies of data) • RF is per keyspace • Eventually consistent • Unless you choose stronger consistency • R +W > N • CAP theorem (Concistency, Availability, Partition tolerance) • ConsistencyLevel

Write consistency levels ANY At least one node, included hinted handoff ONE At least one replica QUOROM At least a majority of replicas DCQUOROM At least a majority of replicas in the current DC DCQUOROMSYNC At least a majority of replicas in each DC ALL All replicas

Read consistency levels ONE At least one replicate QUOROM At least a majority of replicas DCQUOROM At least a majority of replicas in the current DC DCQUOROMSYNC At least a majority of replicas in each DC ALL All replicas

Read repair read req

Read repair data read read req

Read repair data read digest read read req

Read repair data read digest read digest read read req

Read repair • Without read-repair, reads only go to so many replicas required according to consistency level • Probabilistically performed (conﬁgurable percentage) • Affects read performance and scalability with RF

Snitches and replication strategies • Snitches inform Cassandra of network topology • Tries to pick closest replica on reads • Dynamic snitch • Replication strategy determines where replicas are located • Strategy can take advantage of snitch • Rack and DC awareness! (NetworkTopologyStrategy)

How eventual is eventual? • Hinted hand-off • Read repair • Anti-entropy

Anti-entropy Uses Merkle Trees

Streaming • Used to send data in bulk between nodes • anti-entropy • node addition/removal/move • Sequential I/O; pipe a bunch of bytes over a socket. • sstables sorted on token, not key (matters for random partitioner)

Talking to Cassandra • The wire protocol is currently thrift • High-level client for your programming language highly recommended • With plain thrift, you have to consider load balancing, automatic reconnect/retry, timeouts and fairly verbose code

Talking to Cassandra • Python (Telephus/Pycassa) • Java (Hector/Pelops/Kundera) • .NET (Aquiles) • Ruby (Cassandra) • PHP (phpcassa - experimental)

Resources • http://cassandra.apache.org • http://www.riptano.com • http://wiki.apache.org/cassandra/ClientOptions • user@cassandra.apache.org (mailing list) • Dynamo: http://s3.amazonaws.com/AllThingsDistributed/sosp/amazon- dynamo-sosp2007.pdf • BigTable: http://labs.google.com/papers/bigtable.html

Scaling with Apache Cassandra

by scode

Accessibility

Categories

Upload Details

Usage Rights

Statistics