Consistency in Distributed Systems, Part 2

949 views

Published on

Building on the introductory work of a past webinar, we take a deep dive in the locking, replication, and failure modes of leading NoSQL databases. We focus on three main areas critical for modern developers and architects:

Industry survey of ACID compliance.
Best practices to store 1-many and many-many relationships in Riak, Cassandra, MongoDB, DyanamoDB, and Cloudant.
Consistency between primary and secondary indexes (an often neglected subject) and implications for immutable data models.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
949
On SlideShare
0
From Embeds
0
Number of Embeds
68
Actions
Shares
0
Downloads
119
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Consistency in Distributed Systems, Part 2

  1. 1. Consistency in Distributed Systems: II Mike Miller Co-Founder, Chief Scientist @mlmilleratmit
  2. 2. 2014-07-31 2 AMP on Consistency https://amplab.cs.berkeley.edu/tag/consistency/
  3. 3. 2014-07-31 Topics For Today ‣ Brief review of Part I (2014-06-12) ‣ Why is Consistency hard? ‣ What should you really care about? • Single object/row/document operations • Multi-part transactions • Primary/secondary indexes ‣ Dirty little (ACID) secrets: Results from industry survey ‣ Failure modes, strategies, and gotchas 3
  4. 4. 2014-07-31 Motivation 4
  5. 5. 2014-07-31 5 MobileBig Data => Stress models for consistency, transactional reasoning
  6. 6. 2014-07-31 This is your problem when… ! … data doesn’t fit on one server. … data replicated between servers (e.g. read slaves). … data spread between data centers. … state spread across more than one device (mobile!) … mixed workloads with concurrency. … state spread across more than one process. 6
  7. 7. 2014-07-31 This is now everyone’s problem 7
  8. 8. 2014-07-31 Good news — market response: NewSQL, NoSQL, Cloud, … 8
  9. 9. 2014-07-31 9 ships with a mobile strategy
  10. 10. 2014-07-31 {Write: ‘Local’, Sync: ‘Later’} Embedded, Edge, Satellites Desktop, Browser Cloud 10
  11. 11. 2014-07-31 NoSQL Taxonomy 11
  12. 12. 2014-07-31 12
  13. 13. 2014-07-31 13 … … http://www.bailis.org/papers/ramp-sigmod2014.pdf Fundamental reason: CAP Theorem
  14. 14. 2014-07-31 You do need to understand your datastore. 14
  15. 15. 2014-07-31 Why is Consistency Hard? 15
  16. 16. 2014-07-31 1. The network is reliable. 2. Latency is zero. (Fallacies of Distributed Computing, P. Deutsch) 16
  17. 17. MySQL, MongoDB, CouchDB, SOLR, … Dynamo, Cloudant, Cassandra, Riak, …
  18. 18. Primary Secondary Client Replication w(x=1) success Client r(x) x=1 time Perfect Network
  19. 19. Primary Secondary Client Replication w(x=1) success Client r(x) x=Null time Network Partition: Primary Only Available, temporarily inconsistent
  20. 20. Primary Secondary Client Replication w(x=1) Client success time Network Partition: Primary+Secondary Consistent
  21. 21. Primary Secondary Client Replication w(x=1) failure time Network Partition: Primary+Secondary Not Available
  22. 22. 2014-07-31 Partition Failures Dominate ‣ 2011 (AWS): • misconfiguration => 12 hour outage ‣ 2011 Survey (Microsoft): • 13,300 customer impacting network failures • Median 60,000 packts lost per failure • mean 41 link failures per day (95% of 136) • median time to repair of 5 minutes (up to a week) • Redundant networks only reduce failure impact by 40% ‣ HP Managed Enterprise Networks • 28% of customer tickets due to network problems • 39% of all support tickets due to network problems • Median incident duration: 114-188 minutes 22 http://queue.acm.org/detail.cfm?id=2655736
  23. 23. 2014-07-31 Latency Network health really depends on your latency tolerance. A slow network can be just as bad as a broken network. The tails matter. 23
  24. 24. 2014-07-31 Median Latencies 24 Same AZ Different AZs Different Regions http://www.bailis.org/blog/communication-costs-in-real-world-networks/
  25. 25. 2014-07-31 99.99% Latencies 25 Same AZ Different AZs Different Regions http://www.bailis.org/blog/communication-costs-in-real-world-networks/
  26. 26. 2014-07-31 Latency Summary ‣ Distributed, coordinated operations: ‣ rate ~ 1/latency ‣ Real world latencies are substantial, with long tails ‣ At scale, 0.01% events happen constantly ‣ Picture actually much worse due to systematic fluctuations ‣ 99.99% Latencies: ‣ Same AZ: ~50 ms ‣ Same Region: ~80 ms ‣ Inter-Region: 200-400 ms! 26
  27. 27. 2014-07-31 Thank god for ACID (New)SQL, right? ! … not so fast 27
  28. 28. 2014-07-31 ACID in the Wild 28 http://arxiv-web3.library.cornell.edu/abs/1302.0309v1
  29. 29. 2014-07-31 Beware the Marketing 29 http://arxiv-web3.library.cornell.edu/abs/1302.0309v1
  30. 30. 2014-07-31 30 Wow!
  31. 31. 2014-07-31 So… What do we use? What should we worry about? 31
  32. 32. 2014-07-31 1. Locks / Concurrency 2. Relationships / Foreign Keys 3. Inter-index consistency 32 Distinguishing Characteristics
  33. 33. 2014-07-31 Subjective Classification 33 Cassandra Cloudant MongoDB Riak Locking Minimal None Writes and Reads Minimal Consistency Quorum, Optional Paxos Quorum Single document Locks Quorum, Optional Paxos Relationships, “JOINs” De-normalize, Materialized Views Normalize, Materialized Views De-normalize, Application Joins De-normalize or Link Walking Leading Strategies Immutability Immutability Fat Documents Immutability “Intention” HA, Shared Nothing, Many Servers HA, Shared Nothing, Many Servers Master/Slave, Single Server HA, Shared Nothing, Many Servers
  34. 34. 2014-07-31 It happens in all no-SQL systems. Is it the application's responsibility or the DB? 34 De-normalization
  35. 35. 2014-07-31 Relationships as Single Documents 35 Natural fit for some applications http://www.sarahmei.com/blog/2013/11/11/why-you-should-never-use-mongodb/
  36. 36. 2014-07-31 Relationships as Single Documents 36 Duplication sucks, pathologically http://www.sarahmei.com/blog/2013/11/11/why-you-should-never-use-mongodb/
  37. 37. 2014-07-31 Materialized Views Rule 37 Cassandra, Cloudant: “JOINs” via materialized views
  38. 38. 2014-07-31 Review: Cassandra ‣ Highly Available ‣ CQL eases pain of de-normalization ‣ 1-many, many-many relationships via inserts into multiple column families at update ‣ Eventual consistency as those updates propagate ‣ Can appeal to Paxos API with latency, availability hit 38
  39. 39. 2014-07-31 Review: Cloudant ‣ Highly Available ‣ Normalize document structure, include foreign keys to other documents. ‣ Manage foreign key integrity yourself ‣ 1-many, many-many relationships via materialized views ‣ Eventual consistency between primary-index and (batch updated) materialized view 39
  40. 40. 2014-07-31 Review: MongoDB ‣ Understand when MongoDB locks ‣ Go as far as you can with “fat”, de-normalized documents ‣ Beware the consistency subtleties of replica sets, de-normalization 40
  41. 41. 2014-07-31 Review: Riak ‣ Highly Available ‣ Include foreign keys to other documents. ‣ Manage foreign key integrity yourself ‣ one-way (“graphy”) relationships via link-walking API ‣ Can appeal to Paxos API with latency, availability hit 41
  42. 42. 2014-07-31 My Final $0.02 ‣ Time to market should be your #1 concern. ‣ You will probably run both SQL and NoSQL. ‣ We’ve focused on the database, but all new apps need a mobile strategy. ‣ You’ll never engineer a perfect network • Focus on Availability and Partition Tolerance ‣ You will need to become advanced/expert in data modeling for your choice of DB 42
  43. 43. 2014-07-31 cloudant.com mike@cloudant.com @mlmilleratmit #Cloudant Thanks! 43 IRC
  44. 44. 2014-07-31 44

×