Consistency in Distributed Systems, Part 2
 

Consistency in Distributed Systems, Part 2

on

  • 328 views

Building on the introductory work of a past webinar, we take a deep dive in the locking, replication, and failure modes of leading NoSQL databases. We focus on three main areas critical for modern ...

Building on the introductory work of a past webinar, we take a deep dive in the locking, replication, and failure modes of leading NoSQL databases. We focus on three main areas critical for modern developers and architects:

Industry survey of ACID compliance.
Best practices to store 1-many and many-many relationships in Riak, Cassandra, MongoDB, DyanamoDB, and Cloudant.
Consistency between primary and secondary indexes (an often neglected subject) and implications for immutable data models.

Statistics

Views

Total Views
328
Views on SlideShare
292
Embed Views
36

Actions

Likes
0
Downloads
32
Comments
0

1 Embed 36

http://www.dataversity.net 36

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

CC Attribution-NonCommercial-NoDerivs LicenseCC Attribution-NonCommercial-NoDerivs LicenseCC Attribution-NonCommercial-NoDerivs License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Consistency in Distributed Systems, Part 2 Presentation Transcript

  • 1. Consistency in Distributed Systems: II Mike Miller Co-Founder, Chief Scientist @mlmilleratmit
  • 2. 2014-07-31 2 AMP on Consistency https://amplab.cs.berkeley.edu/tag/consistency/
  • 3. 2014-07-31 Topics For Today ‣ Brief review of Part I (2014-06-12) ‣ Why is Consistency hard? ‣ What should you really care about? • Single object/row/document operations • Multi-part transactions • Primary/secondary indexes ‣ Dirty little (ACID) secrets: Results from industry survey ‣ Failure modes, strategies, and gotchas 3
  • 4. 2014-07-31 Motivation 4
  • 5. 2014-07-31 5 MobileBig Data => Stress models for consistency, transactional reasoning
  • 6. 2014-07-31 This is your problem when… ! … data doesn’t fit on one server. … data replicated between servers (e.g. read slaves). … data spread between data centers. … state spread across more than one device (mobile!) … mixed workloads with concurrency. … state spread across more than one process. 6
  • 7. 2014-07-31 This is now everyone’s problem 7
  • 8. 2014-07-31 Good news — market response: NewSQL, NoSQL, Cloud, … 8
  • 9. 2014-07-31 9 ships with a mobile strategy
  • 10. 2014-07-31 {Write: ‘Local’, Sync: ‘Later’} Embedded, Edge, Satellites Desktop, Browser Cloud 10
  • 11. 2014-07-31 NoSQL Taxonomy 11
  • 12. 2014-07-31 12
  • 13. 2014-07-31 13 … … http://www.bailis.org/papers/ramp-sigmod2014.pdf Fundamental reason: CAP Theorem
  • 14. 2014-07-31 You do need to understand your datastore. 14
  • 15. 2014-07-31 Why is Consistency Hard? 15
  • 16. 2014-07-31 1. The network is reliable. 2. Latency is zero. (Fallacies of Distributed Computing, P. Deutsch) 16
  • 17. MySQL, MongoDB, CouchDB, SOLR, … Dynamo, Cloudant, Cassandra, Riak, …
  • 18. Primary Secondary Client Replication w(x=1) success Client r(x) x=1 time Perfect Network
  • 19. Primary Secondary Client Replication w(x=1) success Client r(x) x=Null time Network Partition: Primary Only Available, temporarily inconsistent
  • 20. Primary Secondary Client Replication w(x=1) Client success time Network Partition: Primary+Secondary Consistent
  • 21. Primary Secondary Client Replication w(x=1) failure time Network Partition: Primary+Secondary Not Available
  • 22. 2014-07-31 Partition Failures Dominate ‣ 2011 (AWS): • misconfiguration => 12 hour outage ‣ 2011 Survey (Microsoft): • 13,300 customer impacting network failures • Median 60,000 packts lost per failure • mean 41 link failures per day (95% of 136) • median time to repair of 5 minutes (up to a week) • Redundant networks only reduce failure impact by 40% ‣ HP Managed Enterprise Networks • 28% of customer tickets due to network problems • 39% of all support tickets due to network problems • Median incident duration: 114-188 minutes 22 http://queue.acm.org/detail.cfm?id=2655736
  • 23. 2014-07-31 Latency Network health really depends on your latency tolerance. A slow network can be just as bad as a broken network. The tails matter. 23
  • 24. 2014-07-31 Median Latencies 24 Same AZ Different AZs Different Regions http://www.bailis.org/blog/communication-costs-in-real-world-networks/
  • 25. 2014-07-31 99.99% Latencies 25 Same AZ Different AZs Different Regions http://www.bailis.org/blog/communication-costs-in-real-world-networks/
  • 26. 2014-07-31 Latency Summary ‣ Distributed, coordinated operations: ‣ rate ~ 1/latency ‣ Real world latencies are substantial, with long tails ‣ At scale, 0.01% events happen constantly ‣ Picture actually much worse due to systematic fluctuations ‣ 99.99% Latencies: ‣ Same AZ: ~50 ms ‣ Same Region: ~80 ms ‣ Inter-Region: 200-400 ms! 26
  • 27. 2014-07-31 Thank god for ACID (New)SQL, right? ! … not so fast 27
  • 28. 2014-07-31 ACID in the Wild 28 http://arxiv-web3.library.cornell.edu/abs/1302.0309v1
  • 29. 2014-07-31 Beware the Marketing 29 http://arxiv-web3.library.cornell.edu/abs/1302.0309v1
  • 30. 2014-07-31 30 Wow!
  • 31. 2014-07-31 So… What do we use? What should we worry about? 31
  • 32. 2014-07-31 1. Locks / Concurrency 2. Relationships / Foreign Keys 3. Inter-index consistency 32 Distinguishing Characteristics
  • 33. 2014-07-31 Subjective Classification 33 Cassandra Cloudant MongoDB Riak Locking Minimal None Writes and Reads Minimal Consistency Quorum, Optional Paxos Quorum Single document Locks Quorum, Optional Paxos Relationships, “JOINs” De-normalize, Materialized Views Normalize, Materialized Views De-normalize, Application Joins De-normalize or Link Walking Leading Strategies Immutability Immutability Fat Documents Immutability “Intention” HA, Shared Nothing, Many Servers HA, Shared Nothing, Many Servers Master/Slave, Single Server HA, Shared Nothing, Many Servers
  • 34. 2014-07-31 It happens in all no-SQL systems. Is it the application's responsibility or the DB? 34 De-normalization
  • 35. 2014-07-31 Relationships as Single Documents 35 Natural fit for some applications http://www.sarahmei.com/blog/2013/11/11/why-you-should-never-use-mongodb/
  • 36. 2014-07-31 Relationships as Single Documents 36 Duplication sucks, pathologically http://www.sarahmei.com/blog/2013/11/11/why-you-should-never-use-mongodb/
  • 37. 2014-07-31 Materialized Views Rule 37 Cassandra, Cloudant: “JOINs” via materialized views
  • 38. 2014-07-31 Review: Cassandra ‣ Highly Available ‣ CQL eases pain of de-normalization ‣ 1-many, many-many relationships via inserts into multiple column families at update ‣ Eventual consistency as those updates propagate ‣ Can appeal to Paxos API with latency, availability hit 38
  • 39. 2014-07-31 Review: Cloudant ‣ Highly Available ‣ Normalize document structure, include foreign keys to other documents. ‣ Manage foreign key integrity yourself ‣ 1-many, many-many relationships via materialized views ‣ Eventual consistency between primary-index and (batch updated) materialized view 39
  • 40. 2014-07-31 Review: MongoDB ‣ Understand when MongoDB locks ‣ Go as far as you can with “fat”, de-normalized documents ‣ Beware the consistency subtleties of replica sets, de-normalization 40
  • 41. 2014-07-31 Review: Riak ‣ Highly Available ‣ Include foreign keys to other documents. ‣ Manage foreign key integrity yourself ‣ one-way (“graphy”) relationships via link-walking API ‣ Can appeal to Paxos API with latency, availability hit 41
  • 42. 2014-07-31 My Final $0.02 ‣ Time to market should be your #1 concern. ‣ You will probably run both SQL and NoSQL. ‣ We’ve focused on the database, but all new apps need a mobile strategy. ‣ You’ll never engineer a perfect network • Focus on Availability and Partition Tolerance ‣ You will need to become advanced/expert in data modeling for your choice of DB 42
  • 43. 2014-07-31 cloudant.com mike@cloudant.com @mlmilleratmit #Cloudant Thanks! 43 IRC
  • 44. 2014-07-31 44