Göteborg Distributed: Eventual Consistency in Apache Cassandra


Published on

A brief introduction to Cassandra and an overview of eventual consistency in Cassandra.

Published in: Software, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Göteborg Distributed: Eventual Consistency in Apache Cassandra

  1. 1. ©2013 DataStax Confidential. Do not distribute without consent. Jeremy Hanna Support Engineer Eventual Consistency in Apache Cassandra
  2. 2. Cassandra Design •Massive scalability •High Performance •Reliability/Availability •Ease of use
  3. 3. Developer friendly •CQL3 •Collections (List, Map, Set) •User defined types (2.1) •Cassandra native drivers •Native paging •Tracing •DataStax DevCenter tool •Atomic batches •Lightweight transactions •Triggers
  4. 4. CQL3 examples CREATE KEYSPACE shire WITH REPLICATION = {'class': 'NetworkTopologyStrategy', 'eu' : 3, 'us-east' : 2}; SELECT * FROM emp WHERE empID IN (130,104) ORDER BY deptID DESC; INSERT INTO excelsior.clicks (userid, url, date, name)
 ‘2013-10-09', ‘Mary')
 USING TTL 86400; UPDATE users SET email = ‘charlie@wonka.com’ WHERE login = ‘cbucket64' IF email = ‘cbucket@wonka.com’ CREATE USER bombadil WITH PASSWORD 'goldberry4ever' SUPERUSER; GRANT ALTER ON KEYSPACE shire TO gandalf;
  5. 5. Ops Friendly •Simple design •no special role, no single point of failure •Lots of exposed metrics via JMX •Nodes and entire datacenters can go down with no loss of service •Rapid read protection •DataStax OpsCenter •Visual monitoring tool •REST interface to metric data •Free version •Hands-off services
  6. 6. Some C* Users
  7. 7. Cassandra Design •Massive scalability •Multi-datacenter •High Performance •Reliability/Availability •no SPOF, no special roles •Ease of Use
  8. 8. Fully Distributed •Distributed systems introduce complex problems •What is “down”? •Individual server is down •Network link is down •Long server pause (e.g. GC pause) •Variable network latency •What do I do when a server is overloaded? •How can I stay available/reliable in such circumstances? •How can I maintain consistency? •How do I reconcile differences?
  9. 9. CAP Theorem •Select two Consistency Availability Partition Tolerance
  10. 10. Eventual Consistency •Individual server durability •Write to commitlog (batch or periodic sync) •Write to memtable (which gets flushed to disk) •Achieving consistency level •ONE, QUORUM, ALL •LOCAL_ONE, LOCAL_QUORUM •ANY, EACH_QUORUM (for writes) •Important to note: •All replicas always get a copy of the write
  11. 11. Stuff happens •Overloaded node •“Down” node(s) •Network partition •Datacenter down •Outcome: inconsistency among replicas
  12. 12. Continually cleaning •Hinted handoff •valid for a window of time •replays back to node restored to service •Read repair •after a read, check that data for agreement (digest) •read_repair_chance defaults to 0.1 •also dclocal_read_repair_chance •Anti-entropy service (manual repair) •Check for agreement for all data for range A-B •Run manual repair every gc grace seconds
  13. 13. Advanced Repair •Manual repairs have limited resolution •“There is something different in these 1000 rows” •Therefore you have to stream all 1000 rows •Leads to overstreaming, waste •You can specify start/end keys •Get row level precision •More complicated to execute •DataStax has a repair service to help
  14. 14. Safely consistent? •(LOCAL_)QUORUM reads/writes to be safe? •Ultimately depends on your requirements •Theoretical versus empirical
  15. 15. Netflix Study •Two datacenters (US-East and US-West) •Wrote 500,000 records in each datacenter •50k write operations per second in each DC •Wrote at consistency level ONE •All data read back correctly in other DC •Tried 5 different runs, introduced failures along the way See planetcassandra.org/blog/post/a-netflix-experiment-eventual-consistency-hopeful-consistency-by-christos-kalantzis/
  16. 16. Practical Consistency •ONE is not suitable for all cases •Review your requirements, SLA •Do your own testing to get comfortable •Flexibility translates into the best performance for your use case
  17. 17. Questions?