Cassandra Replication & Consistency Benjamin
Black, b@b3k.us 2010-04-28

Dynamo BigTable Cluster Sparse, management,
columnar data replication, fault model, storage tolerance architecture Cassandra

Dynamo-like Features Symmetric, P2P architecture
No special nodes/SPOFs Gossip-based cluster management Distributed hash table for data placement Pluggable partitioning Pluggable topology discovery Pluggable placement strategies Tunable, eventual consistency

BigTable-like Features Sparse, “columnar” data
model Optional, 2-level maps called Super Column Families SSTable disk storage Append-only commit log Memtable (buffer and sort) Immutable SSTable files Hadoop integration

Topic(s) for Today Replication &
Consistency

[1]

Replication How many copies of
each piece of data do we want in the system? N=3

Consistency Level How many replicas
must respond to declare success? W=2 R=2 ?

CL.Options WRITE READ Level Description
Level Description ZERO Cross fingers ANY WEAK 1st Response (including HH) ONE 1st Response ONE 1st Response STRONG QUORUM N/2 + 1 replicas QUORUM N/2 + 1 replicas ALL All replicas ALL All replicas

A Side Note on CL
Consistency Level is based on Replication Factor (N), not on the number of nodes in the system.

A Question of Time row
column column column column column value value value value value timestamp timestamp timestamp timestamp timestamp All columns have a value and a timestamp Timestamps provided by clients usec resolution by convention Latest timestamp wins Vector clocks may be introduced in 0.7

Read Repair ? Query all
replicas on every read Data from one replica Checksum/timestamps from all others If there is a mismatch: Pull all data and merge Write back to out of sync replicas

Weak vs. Strong Weak Consistency
(reads)Perform repair after returning results Strong Consistency (reads) Perform repair before returning results

R+W>N Please imagine this inequality
has huge fangs, dripping with the blood of innocent, enterprise developers so you can best appreciate the terror it inspires.

Our Guarantee R+W>N guarantees overlap
of read and write quorums W=2 R=2 N=3

A Matter of Perspective View
consistency Replica consistency

[2]

The Ring 0 range 113
375 125 312 250

Tokens A TOKEN is a
partitioner-dependent element on the ring Each NODE has a single, unique TOKEN Each NODE claims a RANGE of the ring from its TOKEN to the token of the previous node on the ring

Partitioning Map from Key Space
to Token RandomPartitioner Tokens are integers in the range 0-2127 MD5(Key) -> Token Good: Even key distribution, Bad: Inefficient range queries OrderPreservingPartitioner Tokens are UTF8 strings in the range ‘’-∞ Key -> Token Good: Efficient range queries, Bad: Uneven key distribution

Snitching Map from Nodes to
Physical Location EndpointSnitch Guess at rack and datacenter based on IP address octets. DatacenterEndpointSnitch Specify IP subnets for racks, grouped per datacenter. PropertySnitch Specify arbitrary mappings from individual IP addresses to racks and datacenters. Or write your own!

Placement Map from Token Space
to Nodes The first replica is always placed on the node that claims the range in which the token falls. Strategies determine where the rest of the replicas are placed.

RackUnaware Place replicas on the
N-1 subsequent nodes around the ring, ignoring topology. datacenter A datacenter B rack 1 rack 2 rack 1 rack 2

RackAware Place the second replica
in another datacenter, and the remaining N-2 replicas on nodes in other racks in the same datacenter. datacenter A datacenter B rack 1 rack 2 rack 1 rack 2

DatacenterShard Place M of the
N replicas in another datacenter, and the remaining N - (M + 1) replicas on nodes in other racks in the same datacenter. datacenter A datacenter B rack 1 rack 2 rack 1 rack 2

Or write your own!

[fin]

Cassandra
http://cassandra.apache.org

Amazon Dynamo http://www.allthingsdistributed.com/2007/10/amazons_dynamo.html Google BigTable
http://labs.google.com/papers/bigtable.html Facebook Cassandra http://www.cs.cornell.edu/projects/ladis2009/papers/lakshman-ladis2009.pdf

Thank you!
Questions?