Your SlideShare is downloading. ×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Consistency in Distributed Systems, Part 2

359
views

Published on

Building on the introductory work of a past webinar, we take a deep dive in the locking, replication, and failure modes of leading NoSQL databases. We focus on three main areas critical for modern …

Building on the introductory work of a past webinar, we take a deep dive in the locking, replication, and failure modes of leading NoSQL databases. We focus on three main areas critical for modern developers and architects:

Industry survey of ACID compliance.
Best practices to store 1-many and many-many relationships in Riak, Cassandra, MongoDB, DyanamoDB, and Cloudant.
Consistency between primary and secondary indexes (an often neglected subject) and implications for immutable data models.

Published in: Technology

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
359
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
37
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Consistency in Distributed Systems: II Mike Miller Co-Founder, Chief Scientist @mlmilleratmit
  • 2. 2014-07-31 2 AMP on Consistency https://amplab.cs.berkeley.edu/tag/consistency/
  • 3. 2014-07-31 Topics For Today ‣ Brief review of Part I (2014-06-12) ‣ Why is Consistency hard? ‣ What should you really care about? • Single object/row/document operations • Multi-part transactions • Primary/secondary indexes ‣ Dirty little (ACID) secrets: Results from industry survey ‣ Failure modes, strategies, and gotchas 3
  • 4. 2014-07-31 Motivation 4
  • 5. 2014-07-31 5 MobileBig Data => Stress models for consistency, transactional reasoning
  • 6. 2014-07-31 This is your problem when… ! … data doesn’t fit on one server. … data replicated between servers (e.g. read slaves). … data spread between data centers. … state spread across more than one device (mobile!) … mixed workloads with concurrency. … state spread across more than one process. 6
  • 7. 2014-07-31 This is now everyone’s problem 7
  • 8. 2014-07-31 Good news — market response: NewSQL, NoSQL, Cloud, … 8
  • 9. 2014-07-31 9 ships with a mobile strategy
  • 10. 2014-07-31 {Write: ‘Local’, Sync: ‘Later’} Embedded, Edge, Satellites Desktop, Browser Cloud 10
  • 11. 2014-07-31 NoSQL Taxonomy 11
  • 12. 2014-07-31 12
  • 13. 2014-07-31 13 … … http://www.bailis.org/papers/ramp-sigmod2014.pdf Fundamental reason: CAP Theorem
  • 14. 2014-07-31 You do need to understand your datastore. 14
  • 15. 2014-07-31 Why is Consistency Hard? 15
  • 16. 2014-07-31 1. The network is reliable. 2. Latency is zero. (Fallacies of Distributed Computing, P. Deutsch) 16
  • 17. MySQL, MongoDB, CouchDB, SOLR, … Dynamo, Cloudant, Cassandra, Riak, …
  • 18. Primary Secondary Client Replication w(x=1) success Client r(x) x=1 time Perfect Network
  • 19. Primary Secondary Client Replication w(x=1) success Client r(x) x=Null time Network Partition: Primary Only Available, temporarily inconsistent
  • 20. Primary Secondary Client Replication w(x=1) Client success time Network Partition: Primary+Secondary Consistent
  • 21. Primary Secondary Client Replication w(x=1) failure time Network Partition: Primary+Secondary Not Available
  • 22. 2014-07-31 Partition Failures Dominate ‣ 2011 (AWS): • misconfiguration => 12 hour outage ‣ 2011 Survey (Microsoft): • 13,300 customer impacting network failures • Median 60,000 packts lost per failure • mean 41 link failures per day (95% of 136) • median time to repair of 5 minutes (up to a week) • Redundant networks only reduce failure impact by 40% ‣ HP Managed Enterprise Networks • 28% of customer tickets due to network problems • 39% of all support tickets due to network problems • Median incident duration: 114-188 minutes 22 http://queue.acm.org/detail.cfm?id=2655736
  • 23. 2014-07-31 Latency Network health really depends on your latency tolerance. A slow network can be just as bad as a broken network. The tails matter. 23
  • 24. 2014-07-31 Median Latencies 24 Same AZ Different AZs Different Regions http://www.bailis.org/blog/communication-costs-in-real-world-networks/
  • 25. 2014-07-31 99.99% Latencies 25 Same AZ Different AZs Different Regions http://www.bailis.org/blog/communication-costs-in-real-world-networks/
  • 26. 2014-07-31 Latency Summary ‣ Distributed, coordinated operations: ‣ rate ~ 1/latency ‣ Real world latencies are substantial, with long tails ‣ At scale, 0.01% events happen constantly ‣ Picture actually much worse due to systematic fluctuations ‣ 99.99% Latencies: ‣ Same AZ: ~50 ms ‣ Same Region: ~80 ms ‣ Inter-Region: 200-400 ms! 26
  • 27. 2014-07-31 Thank god for ACID (New)SQL, right? ! … not so fast 27
  • 28. 2014-07-31 ACID in the Wild 28 http://arxiv-web3.library.cornell.edu/abs/1302.0309v1
  • 29. 2014-07-31 Beware the Marketing 29 http://arxiv-web3.library.cornell.edu/abs/1302.0309v1
  • 30. 2014-07-31 30 Wow!
  • 31. 2014-07-31 So… What do we use? What should we worry about? 31
  • 32. 2014-07-31 1. Locks / Concurrency 2. Relationships / Foreign Keys 3. Inter-index consistency 32 Distinguishing Characteristics
  • 33. 2014-07-31 Subjective Classification 33 Cassandra Cloudant MongoDB Riak Locking Minimal None Writes and Reads Minimal Consistency Quorum, Optional Paxos Quorum Single document Locks Quorum, Optional Paxos Relationships, “JOINs” De-normalize, Materialized Views Normalize, Materialized Views De-normalize, Application Joins De-normalize or Link Walking Leading Strategies Immutability Immutability Fat Documents Immutability “Intention” HA, Shared Nothing, Many Servers HA, Shared Nothing, Many Servers Master/Slave, Single Server HA, Shared Nothing, Many Servers
  • 34. 2014-07-31 It happens in all no-SQL systems. Is it the application's responsibility or the DB? 34 De-normalization
  • 35. 2014-07-31 Relationships as Single Documents 35 Natural fit for some applications http://www.sarahmei.com/blog/2013/11/11/why-you-should-never-use-mongodb/
  • 36. 2014-07-31 Relationships as Single Documents 36 Duplication sucks, pathologically http://www.sarahmei.com/blog/2013/11/11/why-you-should-never-use-mongodb/
  • 37. 2014-07-31 Materialized Views Rule 37 Cassandra, Cloudant: “JOINs” via materialized views
  • 38. 2014-07-31 Review: Cassandra ‣ Highly Available ‣ CQL eases pain of de-normalization ‣ 1-many, many-many relationships via inserts into multiple column families at update ‣ Eventual consistency as those updates propagate ‣ Can appeal to Paxos API with latency, availability hit 38
  • 39. 2014-07-31 Review: Cloudant ‣ Highly Available ‣ Normalize document structure, include foreign keys to other documents. ‣ Manage foreign key integrity yourself ‣ 1-many, many-many relationships via materialized views ‣ Eventual consistency between primary-index and (batch updated) materialized view 39
  • 40. 2014-07-31 Review: MongoDB ‣ Understand when MongoDB locks ‣ Go as far as you can with “fat”, de-normalized documents ‣ Beware the consistency subtleties of replica sets, de-normalization 40
  • 41. 2014-07-31 Review: Riak ‣ Highly Available ‣ Include foreign keys to other documents. ‣ Manage foreign key integrity yourself ‣ one-way (“graphy”) relationships via link-walking API ‣ Can appeal to Paxos API with latency, availability hit 41
  • 42. 2014-07-31 My Final $0.02 ‣ Time to market should be your #1 concern. ‣ You will probably run both SQL and NoSQL. ‣ We’ve focused on the database, but all new apps need a mobile strategy. ‣ You’ll never engineer a perfect network • Focus on Availability and Partition Tolerance ‣ You will need to become advanced/expert in data modeling for your choice of DB 42
  • 43. 2014-07-31 cloudant.com mike@cloudant.com @mlmilleratmit #Cloudant Thanks! 43 IRC
  • 44. 2014-07-31 44