Building a distributed Key-Value store with Cassandra

Building a Key-Value Store with Cassandra Kiwi PyCon 2010 Aaron Morton @aaronmorton Weta Digital 1

Why Cassandra? • Part of a larger project started earlier this year to build new systems for code running on the render farm of 35,000 cores • Larger project goals were Scalability, Reliability, Flexible Schema 2

How about MySQL ? • It works. But... • Schema changes • Write redundancy • Query language mismatch • So went looking for the right tool for the job 3

Redis ? • Fast, ﬂexible. But... • Single core limit • Replication, but no cluster (itʼs coming) • Limited support options 4

Couch DB ? • Schema free, scalable (sort of), redundant (sort of). But... • Single write thread limit • Replication, but no cluster (itʼs coming) • Low consistency with asynchronous replication 5

Cassandra ? • Just right, perhaps. Letʼs see... • Highly available • Tuneable synchronous replication • Scalable writes and reads • Schema free (sort of) • Lots of new mistakes to be made 6

Availability • Row data is kept together and replicated around the cluster • Replication Factor is conﬁgurable • Partitioner determines the position of a row key in the distributed hash table • Replication Strategy determines where in the cluster to place the replicas 7

Consistency • Each read or write request speciﬁes a Consistency Level • Individual nodes may be inconsistent with respect to others • Reads may give consistent results while some nodes have inconsistent values • The entire cluster will eventually mode to a state where there is one version of each 8

Consistency • R+W>N • R = Read Consistency • W = Write Consistency • N = Replication Factor 9

Scale • Distributed hash table • Scale throughput and capacity with more nodes, more disk, more memory • Adding or removing nodes is an online operation • Gossip based protocol for discovery 10

Data Model • Column orientated • Denormalise • Cassandra in an index building machine • Simple explanation: a row has a key and stores an ordered hash in one or more Column Families 11

Data Model • Keyspace • Row / Key • Column Family or Super Column Family • Column 12

Data Model User CF Posts SCF post_1:{ email:fred@... Fred title: foo, dob:04/03 body: bar} post_100:{ Bob email:bob title: monkeys, body: naughty} 13

API • Thrift • Avro (beta) • Auto generated bindings for many languages • Stateful connections • Python wrappers pycassa, Telephus (twisted) 14

API • Client supplied time stamp for all mutations • Client supplied Consistency Level for all mutations and reads 15

API • insert (key, column_family, super_column, column, value) • get(key, column_family, super_column, column) • remove(key, column_family, super_column, column) 16

API • Slicing columns or super columns • list of names • start, ﬁnish, count, reversed • get_slice() to slice one row • multiget_slice() to slice multiple rows • get_range_slices() to slice rows and columns 17

API • Slicing keys • start key, ﬁnish key, count • Partitioner effects key order • get_range_slices() to slice rows and columns 18

API • batch_mutate() • multiple rows and CFʼs • delete or insert / update • Individual mutations are atomic • Request is not atomic, no rollback 19

Our Application Varnish Nginx Tornado Cassandra Rabbit MQ 20

Our Application • Similar to Amazon S3. • REST API. • Databases, Buckets, Keys+Values. 21

Our Column Families • Database (super) • Bucket (super) • Bucket Index • Object • Object Index (super) 22

Our API http:// db_name.wetafx.co.nz/bucket/key 23

PUT Object • /bucket/object • batch_mutate() • one row in Objects CF with columns for meta and the body • one column in ObjectIndex CF row for the bucket 24

List Objects • /bucket_name?start=foo • get_slice() • for the bucket row in ObjectIndex CF • if needed, multiget_slice() to “join” to the Object CF 25

Delete Bucket • /bucket_name • get_slice() on ObjectIndex CF • batch_mutate() to delete Object CF and ObjectIndex CF • delete Bucket CF row 26

Thanks • http://wetafx.co.nz • http://cassandra.apache.org/ • 27

Building a distributed Key-Value store with Cassandra

by aaronmorton on Nov 22, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

Statistics

Building a distributed Key-Value store with Cassandra — Presentation Transcript