Apache Cassandra Overview, presented @GenevaJUG 2013.02.26

1. Apache Cassandra
http://cassandra.apache.org
Benoit Perroud
Software Engineer @Verisign
& Apache Committer
Geneva JUG, 26.02.2013

2. Agenda
• NoSQL Quick Overview
• Apache Cassandra Fundamentals
– Design principles
– Data & Query Model
• Real Life Uses Cases
– Illustrated in CQL3
• What‟s new in 1.2
• Conclusion
• Q&A
2

3. NoSQL
• [Wikipedia] NoSQL is a term used to designate database
management systems that differ from classic relational
database management systems (RDBMS) in some way.
These data stores may not require fixed table schemas,
usually avoid join operations, do not attempt to provide
ACID properties and typically scale horizontally.
• Pioneers : Google BigTable, Amazon Dynamo, etc.
3

4. Scalability
• [Wikipedia] Scalability is a desirable property of a
system, a network, or a process, which indicates its
ability to either handle growing amounts of work in a
graceful manner or to be readily enlarged.
• Scalability in two dimensions :
– Scale up → scale vertically (increase RAM in an existing node)
– Scale out → scale horizontally (add a node to the cluster)
• In summary : handle load and peaks.
4

5. Availability
• [Wikipedia] Availability refers to the ability of the users to
access and use the system. If a user cannot access the
system, it is said to be unavailable. Generally, the term
downtime is used to refer to periods when a system is
unavailable.
• In summary : minimize downtime.
5

6. CAP Theorem
• Consistency : all nodes see the same data at the same
time
• Availability : node failures do not prevent survivors from
continuing to operate
• Partition Tolerance : the system continues to operate
despite arbitrary message loss
• According to the theorem, a distributed system can
satisfy any two of these guarantees at the same time, but
not all three.
6

7. NoSQL Promises
• Scale horizontally
– Double computational power or storage by doubling size of the
cluster. Cluster shrinking should also be true (tight provisioning)
– Adding nodes to the cluster in constant time
• High availability
– No / few / under control SPoF
• On commodity hardware
– 32 cores, 64GB RAM, 12x2TB HDD IS commodity hardware 
• Let see how Cassandra achieves all of these
7

8. Apache Cassandra
• Apache Cassandra is could be simplified as a scalable,
distributed, sparse and eventually consistent hash
map. But it's actually way more.
• Originally developed by Facebook, hit AFS incubator
early 2008, version 1.0 in 2010, version 1.2 early 2013
• Inspired from Amazon Dynamo and Google BigTable
• Version at time of speaking 1.2.2
• Under high development by several startups : Datastax,
Acunu, Netflix, Twitter, Rackspace, …
8

9. Apache Cassandra is a scalable distributed,
sparse, eventually consistent hash map
• Gossip protocol (spreading states like a rumor)
• Consistent hashing
– Node responsible for key range and replica sets
• Replication factor (RF) to achieve persistence
• No single point of failure
100% keyspace
0
• Key space is 2^128 bits 87 12
? ?
More on this later 75 Take half of key range 25
with VNodes ?
of most loaded node ?
62 37
? ?
50
Take half of key range 9
of most loaded node

10. Apache Cassandra is a scalable distributed,
sparse, eventually consistent hash map
• Schemaless
– A schema (metadata) may be determined for convenience
– Column names are stored for every rows
• [Wikipedia] Bloom filter is a space-efficient probabilistic
data structure that is used to test whether an element is a
member of a set.
10

11. Apache Cassandra is a scalable distributed,
sparse, eventually consistent hash map
• [Wikipedia] A quorum is the minimum number of votes
that a distributed transaction has to obtain in order to be
allowed to perform an operation in a distributed system.
A quorum-based technique is implemented to enforce
consistent operation in a distributed system.
• Quorum : W + R > N
– N : number of replica, R : number of node read, W : number of
node written.
– Condition met when:
• R = 1, W = N
• R = N, W = 1
• R = N/2, W = N/2 (+1 if N is even) 11

12. Apache Cassandra is a scalable distributed,
sparse, eventually consistent hash map
• Key space [0,99], previously put(22, 1, t1)
• Replication factor 2
• Consistency : ONE
coordinator 0
Put (22, 2, t2)
80 20
Async put(22,2, t2)
60 40
owner
replica 12

13. Apache Cassandra is a scalable distributed,
sparse, eventually consistent hash map
• Key space [0,99], previously put(13, 1, t1)
• Replication factor 3
• Consistency : QUORUM (R = 2, W = 2)
0
Read(13) = 2, t2
Put (13, 2, t2) Put (13, 2, t2)
80 20
Read(13) = 1, t1
Read repair
60 40
13

14. Apache Cassandra is a scalable distributed,
sparse, eventually consistent hash map
• Can be seen as a multilevel map :
Map of SortedMap of Objects
• Keyspace > ColumnFamily > row > column name = value
– # use keyspace1;
– # set ColumnFamily1['key1']['columName1'] = 'value1';
– # get ColumnFamily1['key1']['columName1'];
14

15. Data Model : Keyspace
Keyspace > ColumnFamily > row > column name = value
• Equivalent to database name in SQL world
• Define replication factor and network topology
– Network topology include multi datacenters topology
– Replication factor can be defined per datacenters
15

16. Data Model : Column Family
Keyspace > ColumnFamily > row > column name = value
• Equivalent to table name in SQL world
– Term may change in upcoming releases to stop confusing users
• Define
– Type of the keys
– Column name comparator
– Additional metadata (types of certain known columns)
16

17. Data Model : Row
Keyspace > ColumnFamily > row > column name = value
• Defined by the key.
– Eventually stored to a node and it's replicas
• Keys are typed
• 2 strategies of key partitioner on the key space
– Random partitioner
• md5(key), murmur3(key), evenly distribute keys on nodes
– Byte Ordered partitioner
• Keep order while iterating through the keys, may lead to hot spots
17

18. Data Model : Column Name
Keyspace > ColumnFamily > row > column name = value
• Could be seen as column in SQL world
• Not mandatory to be declared
– If declared, their corresponding values have types
– Or secondary index
• Ordered
• Column Names are often used as values
Column names
Event1
Column
Family 24.04.2012 07:00 08:00
239 255
18
Row key Values

19. Data Model : Value
Keyspace > ColumnFamily > row > column name = value
• Can be typed, seen as array of bytes otherwise
• Existing types include
– Bytes
– Strings (ASCII or UTF-8 strings)
– Integer, Long, Float, Double, Decimal
– UUID, dates
– Counters (of long)
• Can expire
• No foreign keys (!)
19

20. Write path
1. Write to commit log
Memory
2. Update MemTable CF1
MemTable
CF2
MemTable
CFn
MemTable
…
3. Acknowledge the client
4. When MemTable reaches a Disks
CF1 CFn
Commit log
threshold, flush to disk as Bloom filter … SSTable
SSTable Index
Data
…
SSTable SSTable
20

21. Read path
• Versions of the same column
Memory
can be spread at the same time CF1 CF2 CFn
MemTable MemTable MemTable
…
– In the MemTable
– In the MemTable being flushed Disks
– In one or multiple SSTable Commit log
CF1 CFn
…
• All versions read, and resolved /
Bloom filter
SSTable
Index
merged using timestamp Data
…
– Keys and Rows cache
SSTable SSTable
– Bloom filters allow to skip reading
unnecessary SSTables
– SSTables are indexed
– Compaction keep things
reasonable 21

22. Compaction
• Runs regularly as a background operation
• Merge SSTables together
• Remove expired and deleted values
• Has impact on general I/O availability (and thus
performance)
– This is where most of tuning happens
– Can be throttled
• Two type of compaction
– Size-tiered
• Fewer I/O consumption  write-heavy workload
– Leveled
• Guarantee to read from fewer SSTables  read-heavy workload
• See http://www.datastax.com/dev/blog/leveled-compaction-in-apache-cassandra for complete details. 22

23. Query Model
• Thrift API
– CLI
– Higher level third-party libraries
• Hector
• Pycassa
• Phpyandra
• Astyanax
• Helenus
• CQL (Cassandra Query Language)
– And newly CQL3 released with C*1.2
23

24. Query Model
• Cassandra is more than a key – value store.
– Get
– Put
– Delete
– Update
– But also various range queries
• Key range
• Column range (slice)
– Secondary indexes
24

25. Query Model : Get
• Get single key
– Give me key „a‟
• Get multiple keys
– Give me rows for keys „a‟, „c‟, „d‟ and „f‟
Ordered regarding column name comparator
‘1’ ‘2’ ‘3’ ‘4’ ‘5’
„c‟ 8 9 10 11
„e‟ 12 13 14
RandomPartitionner
„f‟ 15 16 17
„a‟ 18
„b‟ 19 20 20
„d‟ 22 23 24 25 26
25

26. Query Model : Get Range
• Range
– Query for a range of key
• Give me all rows with keys between „c‟ and „f‟.
• Mind the partitioner.
‘1’ ‘2’ ‘3’ ‘4’ ‘5’
„c‟ 8 9 10 11
„e‟ 12 13 14
„f‟ 15 16 17
„a‟ 18
„b‟ 19 20 20
„d‟ 22 23 24 25 26
26

27. Query Model : Get Slice
• Slice
– Query for a slice of columns
• For key „c‟, give me all columns between „3‟ and „5‟
• For key „d‟, give me all columns between „3‟ and „5‟
‘1’ ‘2’ ‘3’ ‘4’ ‘5’
„c‟ 8 9 10 11
„e‟ 12 13 14
„f‟ 15 16 17
„a‟ 18
„b‟ 19 20 20
„d‟ 22 23 24 25 26
27

28. Query Model : Get Range Slice
• Range and Slice can be combined : rangeSliceQuery
– For keys between „b‟ and „d‟, give me columns between „2‟ and „4‟
‘1’ ‘2’ ‘3’ ‘4’ ‘5’
„a‟ 8 9 10 11
„b‟ 12 13 14
„c‟ 15 16 17
„d‟ 18
„e‟ 19 20 20
„f‟ 22 23 24 25 26
28

29. Query Model : Secondary Index
• Secondary Index
– Give me all rows where value for column „2‟ is „12‟
‘1’ ‘2’ ‘3’ ‘4’ ‘5’
„a‟ 8 9 10 11
„b‟ 12 13 14
„c‟ 15 16 17
„d‟ 18
„e‟ 19 20 20
„f‟ 22 23 24 25 26
29

30. Real Life Use Case : Doodle Clone
• Living demo http://doodle.noisette.ch
Data model
Polls { id, label, [choices], email, limit, [ subscribers ] }
• Id generation
– TimeUUID is your friend
• Avoid super column families
– Use composite, or CQL3 
• Subscriber‟s name uniqueness per poll ?
– Cassandra anti-pattern (read after write)
• Limit to n subscribers per option ?
– Cassandra anti-pattern (read after write)
31

31. Real Life Use Case : Doodle Clone
CREATE KEYSPACE Doodle
WITH replication = {'class': 'SimpleStrategy', 'replication_factor' : 1};
USE doodle;
CREATE TABLE Polls (
id uuid,
label text,
choices list<text>,
email text,
maxChoices int,
subscribers list<text>,
PRIMARY KEY (id)
) WITH compaction = { 'class' : 'LeveledCompactionStrategy' }
AND read_repair_chance = 0.0;
INSERT INTO Polls (id, label, email, choices) VALUES (eba080a0-8011-11e2-9e96-0800200c9a66,
'Test poll1', 'benoit@noisette.ch', ['Monday', 'Tuesday', 'Wednesday', 'Thursday',
'Friday']);
UPDATE Polls SET subscribers = subscribers + [ 'Benoit' ] WHERE id = eba080a0-8011-11e2-9e96-
0800200c9a66;
UPDATE Polls SET subscribers = subscribers + [ 'Maxime', 'Nicolas' ] WHERE id = eba080a0-8011-
11e2-9e96-0800200c9a66;
DELETE subscribers[0] FROM Polls WHERE id = eba080a0-8011-11e2-9e96-0800200c9a66;
32

32. Real Life Use Case : Heavy Writes
• Cassandra is a really good fit when the ratio read / write
is close to 0
– Event logging / redo logs
– Time series
• Best practice to write data in its raw format
AND in aggregated forms at the same time
• But need compation tuning
– {min,max}_compaction_threshold
– memtable_flush_writers
– … no magic solution here, only pragmatic approach
• change configuration in one node, and mesure the difference (load, latency, …)
33

33. Real Life Use Case : Counters
• Cassandra >= 0.8 (CASSANDRA-1072)
CREATE TABLE Events (id uuid, count counter, PRIMARY KEY (id));
UPDATE Events SET count = count + 1 WHERE id = 95b64d72-8014-11e2-9e96-0800200c9a66;
• Example
counterCF['entity1'][2012-06-14 18:30:00]
counterCF['entity1'][2012-06-14 18:30:05] Query per entity
counterCF['entity1'][2012-06-14 18:30:10] number of hits for „entity1‟
… between 18:30:00 and 19:00:00
counterCF['entity2'][2012-06-14 18:30:05]
counterCF[2012-06-14 18:30:00]['entity1']
counterCF[2012-06-14 18:30:00]['entity2'] Query per date range
counterCF[2012-06-14 18:30:00]['entity3'] all entities being hit between
… 18:30:00 and 19:00:00
counterCF[2012-06-14 18:30:05]['entity1'] ! need complete date enumeration
34

34. Real Life Use Case : Bulk Loading
• Data is transformed (e.g. using MapReduce)
• Data is bulk loaded
– ColumFamilyOutputFormat (< v1.1)
• Not real bulk loading
– BulkOutputFormat (>= v1.1)
• SSTable generated during the tranformation, and streamed
• Prefer Leveled Compaction Strategy
– Reduce read latency
– Size sstable_size_in_mb to your data
35

35. Real Life Use Case : Bulk Loading
• Data is transformed (e.g. using MapReduce)
• Data is bulk loaded
– ColumFamilyOutputFormat (< v1.1)
• Not real bulk loading
– BulkOutputFormat (>= v1.1)
• SSTable generated during the tranformation, and streamed
• Prefer Leveled Compaction Strategy
– Reduce read latency
– Size sstable_size_in_mb to your data
36

36. Real Life Use Case : λ Architecture
• Enabling real-time queries to end-users
– “Hybrid Approach to Enable Real-Time Queries to End-Users”,
Software Developer Journal February 2013
37

37. What‟s New in 1.2
• CQL3
– http://cassandra.apache.org/doc/cql3/CQL.html
• Virtual Nodes (vnodes)
• Atomic batches
• Murmur3Partitioner
• Off-heap SSTable metadata
• Query tracing
• … a lot more … 38
Illustrations credits to Datastax, http://www.datastax.com/dev/blog/upgrading-an-existing-cluster-to-vnodes

38. Conclusion
• Cassandra is not a general purpose solution
• But Cassandra is doing a really good job if used
accordingly
– Really good scalability
• Netflix‟s 1M w/s on AWS
http://techblog.netflix.com/2011/11/benchmarking-cassandra-
scalability-on.html
– Low operational cost
• Admin friendly, no SPoF, Vnodes, snapshot, …
– Advanced data and query model
39

39. Thanks for your attention
• Questions?
benoit@noisette.ch
@killerwhile
• No? Cool … Apéro 
40