0
Consistency in
Distributed Systems: II
Mike Miller
Co-Founder, Chief Scientist
@mlmilleratmit
2014-07-31 2
AMP on Consistency
https://amplab.cs.berkeley.edu/tag/consistency/
2014-07-31
Topics For Today
‣ Brief review of Part I (2014-06-12)
‣ Why is Consistency hard?
‣ What should you really care...
2014-07-31
Motivation
4
2014-07-31 5
MobileBig Data
=> Stress models for consistency, transactional reasoning
2014-07-31
This is your problem when…
!
… data doesn’t fit on one server.
… data replicated between servers (e.g. read sla...
2014-07-31
This is now everyone’s problem
7
2014-07-31
Good news — market response:
NewSQL, NoSQL, Cloud, …
8
2014-07-31 9
ships with a mobile strategy
2014-07-31
{Write: ‘Local’, Sync: ‘Later’}
Embedded,
Edge, Satellites
Desktop,
Browser
Cloud
10
2014-07-31
NoSQL Taxonomy
11
2014-07-31 12
2014-07-31 13
…
…
http://www.bailis.org/papers/ramp-sigmod2014.pdf
Fundamental reason: CAP Theorem
2014-07-31
You do need to understand your
datastore.
14
2014-07-31
Why is Consistency Hard?
15
2014-07-31
1. The network is reliable.
2. Latency is zero.
(Fallacies of Distributed Computing, P. Deutsch)
16
MySQL, MongoDB,
CouchDB, SOLR, …
Dynamo, Cloudant,
Cassandra, Riak, …
Primary
Secondary
Client
Replication
w(x=1)
success
Client
r(x)
x=1
time
Perfect Network
Primary
Secondary
Client
Replication
w(x=1)
success
Client
r(x)
x=Null
time
Network Partition: Primary Only
Available, tem...
Primary
Secondary
Client
Replication
w(x=1)
Client
success
time
Network Partition: Primary+Secondary
Consistent
Primary
Secondary
Client
Replication
w(x=1)
failure
time
Network Partition: Primary+Secondary
Not Available
2014-07-31
Partition Failures Dominate
‣ 2011 (AWS):
• misconfiguration => 12 hour outage
‣ 2011 Survey (Microsoft):
• 13,...
2014-07-31
Latency
Network health really depends on your latency tolerance.
A slow network can be just as bad as a broken ...
2014-07-31
Median Latencies
24
Same AZ Different
AZs
Different
Regions
http://www.bailis.org/blog/communication-costs-in-rea...
2014-07-31
99.99% Latencies
25
Same AZ Different
AZs
Different
Regions
http://www.bailis.org/blog/communication-costs-in-rea...
2014-07-31
Latency Summary
‣ Distributed, coordinated operations:
‣ rate ~ 1/latency
‣ Real world latencies are substantia...
2014-07-31
Thank god for ACID (New)SQL, right?
!
… not so fast
27
2014-07-31
ACID in the Wild
28
http://arxiv-web3.library.cornell.edu/abs/1302.0309v1
2014-07-31
Beware the Marketing
29
http://arxiv-web3.library.cornell.edu/abs/1302.0309v1
2014-07-31 30
Wow!
2014-07-31
So…
What do we use?
What should we worry
about?
31
2014-07-31
1. Locks / Concurrency
2. Relationships / Foreign Keys
3. Inter-index consistency
32
Distinguishing Characteris...
2014-07-31
Subjective Classification
33
Cassandra Cloudant MongoDB Riak
Locking Minimal None
Writes and
Reads
Minimal
Cons...
2014-07-31
It happens in all no-SQL systems.
Is it the application's responsibility or the DB?
34
De-normalization
2014-07-31
Relationships as Single Documents
35
Natural fit for some applications
http://www.sarahmei.com/blog/2013/11/11/...
2014-07-31
Relationships as Single Documents
36
Duplication sucks, pathologically
http://www.sarahmei.com/blog/2013/11/11/...
2014-07-31
Materialized Views Rule
37
Cassandra, Cloudant: “JOINs” via materialized views
2014-07-31
Review: Cassandra
‣ Highly Available
‣ CQL eases pain of de-normalization
‣ 1-many, many-many relationships via...
2014-07-31
Review: Cloudant
‣ Highly Available
‣ Normalize document structure, include foreign keys to other documents.
‣ ...
2014-07-31
Review: MongoDB
‣ Understand when MongoDB locks
‣ Go as far as you can with “fat”, de-normalized documents
‣ Be...
2014-07-31
Review: Riak
‣ Highly Available
‣ Include foreign keys to other documents.
‣ Manage foreign key integrity yours...
2014-07-31
My Final $0.02
‣ Time to market should be your #1 concern.
‣ You will probably run both SQL and NoSQL.
‣ We’ve ...
2014-07-31
cloudant.com
mike@cloudant.com
@mlmilleratmit
#Cloudant
Thanks!
43
IRC
2014-07-31 44
Upcoming SlideShare
Loading in...5
×

Consistency in Distributed Systems, Part 2

428

Published on

Building on the introductory work of a past webinar, we take a deep dive in the locking, replication, and failure modes of leading NoSQL databases. We focus on three main areas critical for modern developers and architects:

Industry survey of ACID compliance.
Best practices to store 1-many and many-many relationships in Riak, Cassandra, MongoDB, DyanamoDB, and Cloudant.
Consistency between primary and secondary indexes (an often neglected subject) and implications for immutable data models.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
428
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
40
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Consistency in Distributed Systems, Part 2"

  1. 1. Consistency in Distributed Systems: II Mike Miller Co-Founder, Chief Scientist @mlmilleratmit
  2. 2. 2014-07-31 2 AMP on Consistency https://amplab.cs.berkeley.edu/tag/consistency/
  3. 3. 2014-07-31 Topics For Today ‣ Brief review of Part I (2014-06-12) ‣ Why is Consistency hard? ‣ What should you really care about? • Single object/row/document operations • Multi-part transactions • Primary/secondary indexes ‣ Dirty little (ACID) secrets: Results from industry survey ‣ Failure modes, strategies, and gotchas 3
  4. 4. 2014-07-31 Motivation 4
  5. 5. 2014-07-31 5 MobileBig Data => Stress models for consistency, transactional reasoning
  6. 6. 2014-07-31 This is your problem when… ! … data doesn’t fit on one server. … data replicated between servers (e.g. read slaves). … data spread between data centers. … state spread across more than one device (mobile!) … mixed workloads with concurrency. … state spread across more than one process. 6
  7. 7. 2014-07-31 This is now everyone’s problem 7
  8. 8. 2014-07-31 Good news — market response: NewSQL, NoSQL, Cloud, … 8
  9. 9. 2014-07-31 9 ships with a mobile strategy
  10. 10. 2014-07-31 {Write: ‘Local’, Sync: ‘Later’} Embedded, Edge, Satellites Desktop, Browser Cloud 10
  11. 11. 2014-07-31 NoSQL Taxonomy 11
  12. 12. 2014-07-31 12
  13. 13. 2014-07-31 13 … … http://www.bailis.org/papers/ramp-sigmod2014.pdf Fundamental reason: CAP Theorem
  14. 14. 2014-07-31 You do need to understand your datastore. 14
  15. 15. 2014-07-31 Why is Consistency Hard? 15
  16. 16. 2014-07-31 1. The network is reliable. 2. Latency is zero. (Fallacies of Distributed Computing, P. Deutsch) 16
  17. 17. MySQL, MongoDB, CouchDB, SOLR, … Dynamo, Cloudant, Cassandra, Riak, …
  18. 18. Primary Secondary Client Replication w(x=1) success Client r(x) x=1 time Perfect Network
  19. 19. Primary Secondary Client Replication w(x=1) success Client r(x) x=Null time Network Partition: Primary Only Available, temporarily inconsistent
  20. 20. Primary Secondary Client Replication w(x=1) Client success time Network Partition: Primary+Secondary Consistent
  21. 21. Primary Secondary Client Replication w(x=1) failure time Network Partition: Primary+Secondary Not Available
  22. 22. 2014-07-31 Partition Failures Dominate ‣ 2011 (AWS): • misconfiguration => 12 hour outage ‣ 2011 Survey (Microsoft): • 13,300 customer impacting network failures • Median 60,000 packts lost per failure • mean 41 link failures per day (95% of 136) • median time to repair of 5 minutes (up to a week) • Redundant networks only reduce failure impact by 40% ‣ HP Managed Enterprise Networks • 28% of customer tickets due to network problems • 39% of all support tickets due to network problems • Median incident duration: 114-188 minutes 22 http://queue.acm.org/detail.cfm?id=2655736
  23. 23. 2014-07-31 Latency Network health really depends on your latency tolerance. A slow network can be just as bad as a broken network. The tails matter. 23
  24. 24. 2014-07-31 Median Latencies 24 Same AZ Different AZs Different Regions http://www.bailis.org/blog/communication-costs-in-real-world-networks/
  25. 25. 2014-07-31 99.99% Latencies 25 Same AZ Different AZs Different Regions http://www.bailis.org/blog/communication-costs-in-real-world-networks/
  26. 26. 2014-07-31 Latency Summary ‣ Distributed, coordinated operations: ‣ rate ~ 1/latency ‣ Real world latencies are substantial, with long tails ‣ At scale, 0.01% events happen constantly ‣ Picture actually much worse due to systematic fluctuations ‣ 99.99% Latencies: ‣ Same AZ: ~50 ms ‣ Same Region: ~80 ms ‣ Inter-Region: 200-400 ms! 26
  27. 27. 2014-07-31 Thank god for ACID (New)SQL, right? ! … not so fast 27
  28. 28. 2014-07-31 ACID in the Wild 28 http://arxiv-web3.library.cornell.edu/abs/1302.0309v1
  29. 29. 2014-07-31 Beware the Marketing 29 http://arxiv-web3.library.cornell.edu/abs/1302.0309v1
  30. 30. 2014-07-31 30 Wow!
  31. 31. 2014-07-31 So… What do we use? What should we worry about? 31
  32. 32. 2014-07-31 1. Locks / Concurrency 2. Relationships / Foreign Keys 3. Inter-index consistency 32 Distinguishing Characteristics
  33. 33. 2014-07-31 Subjective Classification 33 Cassandra Cloudant MongoDB Riak Locking Minimal None Writes and Reads Minimal Consistency Quorum, Optional Paxos Quorum Single document Locks Quorum, Optional Paxos Relationships, “JOINs” De-normalize, Materialized Views Normalize, Materialized Views De-normalize, Application Joins De-normalize or Link Walking Leading Strategies Immutability Immutability Fat Documents Immutability “Intention” HA, Shared Nothing, Many Servers HA, Shared Nothing, Many Servers Master/Slave, Single Server HA, Shared Nothing, Many Servers
  34. 34. 2014-07-31 It happens in all no-SQL systems. Is it the application's responsibility or the DB? 34 De-normalization
  35. 35. 2014-07-31 Relationships as Single Documents 35 Natural fit for some applications http://www.sarahmei.com/blog/2013/11/11/why-you-should-never-use-mongodb/
  36. 36. 2014-07-31 Relationships as Single Documents 36 Duplication sucks, pathologically http://www.sarahmei.com/blog/2013/11/11/why-you-should-never-use-mongodb/
  37. 37. 2014-07-31 Materialized Views Rule 37 Cassandra, Cloudant: “JOINs” via materialized views
  38. 38. 2014-07-31 Review: Cassandra ‣ Highly Available ‣ CQL eases pain of de-normalization ‣ 1-many, many-many relationships via inserts into multiple column families at update ‣ Eventual consistency as those updates propagate ‣ Can appeal to Paxos API with latency, availability hit 38
  39. 39. 2014-07-31 Review: Cloudant ‣ Highly Available ‣ Normalize document structure, include foreign keys to other documents. ‣ Manage foreign key integrity yourself ‣ 1-many, many-many relationships via materialized views ‣ Eventual consistency between primary-index and (batch updated) materialized view 39
  40. 40. 2014-07-31 Review: MongoDB ‣ Understand when MongoDB locks ‣ Go as far as you can with “fat”, de-normalized documents ‣ Beware the consistency subtleties of replica sets, de-normalization 40
  41. 41. 2014-07-31 Review: Riak ‣ Highly Available ‣ Include foreign keys to other documents. ‣ Manage foreign key integrity yourself ‣ one-way (“graphy”) relationships via link-walking API ‣ Can appeal to Paxos API with latency, availability hit 41
  42. 42. 2014-07-31 My Final $0.02 ‣ Time to market should be your #1 concern. ‣ You will probably run both SQL and NoSQL. ‣ We’ve focused on the database, but all new apps need a mobile strategy. ‣ You’ll never engineer a perfect network • Focus on Availability and Partition Tolerance ‣ You will need to become advanced/expert in data modeling for your choice of DB 42
  43. 43. 2014-07-31 cloudant.com mike@cloudant.com @mlmilleratmit #Cloudant Thanks! 43 IRC
  44. 44. 2014-07-31 44
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×