Embrace NoSQL and Eventual Consistency with Ripple

2,687 views
2,616 views

Published on

So, there's this "NoSQL" thing you may have heard of, and this related thing called "eventual consistency". Supposedly, they help you scale, but no one has ever explained why! Well, wonder no more! This talk will demystify NoSQL, eventual consistency, how they might help you scale, and -- most importantly -- why you should care.

We'll look closely at how Riak, a linearly-scalable, distributed and fault-tolerant NoSQL datastore, implements eventual consistency, and how you can harness it from Ruby via the slick Ripple client/ORM. When the talk is finished, you'll have the tools both to understand eventual consistency and to handle it like a pro inside your next Ruby application.

Published in: Technology
0 Comments
6 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,687
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
46
Comments
0
Likes
6
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Embrace NoSQL and Eventual Consistency with Ripple

    1. 1. Embrace NoSQL and Eventual Consistency with Ripple Sean Cribbs Basho Technologies @seancribbsbasho
    2. 2. UnderstandingEventual Consistency
    3. 3. Internet
    4. 4. Internet
    5. 5. Internet
    6. 6. Internet
    7. 7. InternetCDN
    8. 8. InternetCDN
    9. 9. InternetCDN M S S
    10. 10. InternetCDN
    11. 11. Is this system consistent?
    12. 12. You sacrificed ACID as soon as you added caching.
    13. 13. How does yourapplication cope?
    14. 14. Inconsistency Creep
    15. 15. Inconsistency Creep Cache staleness
    16. 16. Inconsistency Creep Cache staleness Master-slave replication lag
    17. 17. Inconsistency Creep Cache staleness Master-slave replication lag CDN propagation lag
    18. 18. Inconsistency Creep Cache staleness Master-slave replication lag CDN propagation lag Race-conditions
    19. 19. Inconsistency Creep Cache staleness Master-slave replication lag CDN propagation lag Race-conditions Service failures
    20. 20. Eventual Consistency
    21. 21. Eventual ConsistencyTolerates temporary inconsistency
    22. 22. Eventual ConsistencyTolerates temporary inconsistency Moves toward consistency automatically
    23. 23. Eventual ConsistencyTolerates temporary inconsistency Moves toward consistency automatically Resilient, maintains availability
    24. 24. Eventual ConsistencyTolerates temporary inconsistency Moves toward consistency automatically Resilient, maintains availability “Eventual” in seconds, not days
    25. 25. Eventual ConsistencyTolerates temporary inconsistency Moves toward consistency automatically Resilient, maintains availability “Eventual” in seconds, not days
    26. 26. Real-WorldEventual Consistency
    27. 27. Real-WorldEventual Consistency DNS
    28. 28. Real-WorldEventual Consistency DNS LDAP
    29. 29. Real-WorldEventual Consistency DNS LDAP The Web
    30. 30. Harvest & Yield
    31. 31. Harvest & Yield CAP “For Dummies”
    32. 32. Harvest & Yield CAP “For Dummies”Harvest: how much of dataset is reflected in a response (C)
    33. 33. Harvest & Yield CAP “For Dummies”Harvest: how much of dataset is reflected in a response (C)Yield: how likely is the datastore to complete request (A)
    34. 34. Harvest & Yieldhttp://codahale.com/you-cant-sacrifice-partition- tolerance/ CAP “For Dummies” Harvest: how much of dataset is reflected in a response (C) Yield: how likely is the datastore to complete request (A)
    35. 35. # Do I need it?def eventual_consistency? @money === @uptimeend
    36. 36. # Do I need it?def eventual_consistency? @money === @uptimeend# “Any sufficiently large# system is in a constant# state of failure.”# --Justin Sheehy, Basho CTO
    37. 37. Ok, I’ll use EventualConsistency. How?
    38. 38. Oh, those NoSQL things.
    39. 39. NoSQL Flavors
    40. 40. NoSQL FlavorsGraph (Neo4J, InfiniteGraph)
    41. 41. NoSQL FlavorsGraph (Neo4J, InfiniteGraph)Document (Mongo, Couch)
    42. 42. NoSQL FlavorsGraph (Neo4J, InfiniteGraph)Document (Mongo, Couch)Column (HBase, Cassandra)
    43. 43. NoSQL FlavorsGraph (Neo4J, InfiniteGraph)Document (Mongo, Couch)Column (HBase, Cassandra)Key-Value (Riak,Voldemort)
    44. 44. NoSQL Flavors structured Graph (Neo4J, InfiniteGraph) Document (Mongo, Couch) Column (HBase, Cassandra) Key-Value (Riak, Voldemort)unstructured
    45. 45. Not all NoSQLs areEventually Consistent.
    46. 46. Riak implementsEventual Consistency.
    47. 47. What is Riak?
    48. 48. Riak is...
    49. 49. Riak is...a scalable
    50. 50. Riak is... a scalablehighly-available
    51. 51. Riak is... a scalable highly-availablenetworked/distributed
    52. 52. Riak is... a scalable highly-availablenetworked/distributed key-value store
    53. 53. Riak’s Data Model
    54. 54. Riak’s Data ModelStores values against keys
    55. 55. Riak’s Data ModelStores values against keys Values are opaque
    56. 56. Riak’s Data Model Stores values against keys Values are opaqueKeys are grouped into buckets
    57. 57. Values Have Metadata
    58. 58. Values Have Metadata Content-Type
    59. 59. Values Have Metadata Content-Type Last-Modified
    60. 60. Values Have Metadata Content-Type Last-Modified Link
    61. 61. Values Have Metadata Content-Type Last-Modified Link Secondary index values
    62. 62. Values Have Metadata Content-Type Last-Modified Link Secondary index values Custom metadata
    63. 63. Values Have Metadata Content-Type Last-Modified Link Secondary index values Custom metadata Value + Metadata == “Object”
    64. 64. Basic Operations
    65. 65. Basic Operations GET /riak/bucket/key
    66. 66. Basic Operations GET /riak/bucket/key PUT /riak/bucket/key
    67. 67. Basic Operations GET /riak/bucket/key PUT /riak/bucket/keyDELETE /riak/bucket/key
    68. 68. Basic Operations GET /riak/bucket/key PUT /riak/bucket/keyDELETE /riak/bucket/key HTTP or Protocol Buffers
    69. 69. Other Query Methods
    70. 70. Other Query Methods Link-traversal (“walking”)
    71. 71. Other Query Methods Link-traversal (“walking”) Full-text Search
    72. 72. Other Query Methods Link-traversal (“walking”) Full-text Search Secondary Indexes
    73. 73. Other Query Methods Link-traversal (“walking”) Full-text Search Secondary Indexes MapReduce
    74. 74. Ripple connects Ruby to Riak.
    75. 75. # gem install riak-clientrequire ‘riak’# Connect to Riak on localhostclient = Riak::Client.new# Fetch something, write it to a fileo = client[‘slide-decks’][‘RubyConfAR’]File.open(‘slides.key’,‘wb’) do |f| f.write o.raw_dataend
    76. 76. # Create a new keyv = client[‘venues’].new(‘RubyConfAR’)# Set the value, JSON by defaultv.data = { name: ‘Ciudad Cultural Konex’, address: { street: ‘Sarmiento 3131’, city: ‘Buenos Aires’, zip: ‘C1196AAG’ }}# Write it to Riakv.store
    77. 77. # Create a new key for an imagepic = client[‘pictures’].new(‘bue1.jpg’)# Read the file, set the content-typepic.raw_data = File.read(‘bue1.jpg’)pic.content_type = ‘image/jpeg’# Set some metadatapic.meta[‘camera’] = ‘Nikon’pic.indexes[‘date_int’] << Time.now.to_i# Write it to Riakpic.store
    78. 78. But I want models! Ok, no problem.
    79. 79. require ‘ripple’ # gem install rippleclass Person include Ripple::Document property :name, String, presence: true key_on :name many :addressesendclass Address include Ripple::EmbeddedDocument property :street, String property :city, String property :zip, Stringend
    80. 80. # Create my infome = Person.new( name: ‘Sean Cribbs’, addresses: [{ street: ‘929 Market St’, city: ‘San Francisco’, zip: ‘94103’ }] )me.save! # Store as JSON under # “people/Sean Cribbs”me == Person.find(‘Sean Cribbs’) # true
    81. 81. That’s kind of cool. But what aboutEventual Consistency?
    82. 82. Riak uses CausalEventual Consistency. also called Vector Clocks
    83. 83. Causal Consistency
    84. 84. Causal Consistency Each read includes a token of the current state (vector clock)
    85. 85. Causal Consistency Each read includes a token of the current state (vector clock)Clients send the token back with writes
    86. 86. Causal Consistency Each read includes a token of the current state (vector clock)Clients send the token back with writesRiak updates the token and detects stale versions and conflicts by comparison
    87. 87. Causal Consistency Each read includes a token of the current state (vector clock)Clients send the token back with writesRiak updates the token and detects stale versions and conflicts by comparison Conflicts are exposed to your application
    88. 88. Conflicts? That sounds bad.
    89. 89. It happens frequently without strict consistency. For example, havetwo threads write at the same time. Riak detects this conflict for you.
    90. 90. Your application is also part of the Eventually Consistent system. Itmakes the hard decisions that Riak can’t.
    91. 91. Ok. So how do Iresolve conflicts?
    92. 92. Riak will give you all values that can’t be resolved automatically. Just write back the resolved value.
    93. 93. Think of it like merging or rebasing git branches. Sometimes you MUST usegit mergetool to fix things.
    94. 94. Two Ways to Resolve
    95. 95. Two Ways to Resolve Semantic Resolution
    96. 96. Two Ways to Resolve Semantic Resolution CRDTs
    97. 97. Semantic Resolution
    98. 98. Semantic Resolution Given any number of values,compute the resolved version using business logic.
    99. 99. Example
    100. 100. Example If your value is a shopping cart,merge all items and sum quantities into a single cart.
    101. 101. Example If your value is a shopping cart,merge all items and sum quantities into a single cart. Maybe the customer buys more that way!
    102. 102. Ripple supportsSemantic Resolution.
    103. 103. # A cart is { product_id => quantity }Riak::RObject.on_conflict do |obj| if obj.bucket.name == ‘carts’ obj.content_type = ‘application/json’ obj.data = {} obj.siblings.each do |val| val.data.each do |product,q| obj.data[product] ||= 0 obj.data[product] += q end end obj endend
    104. 104. # Documents can be resolved tooPerson.on_conflict(:addresses) do |vals| vals.each do |doc| self.addresses &= doc.addresses endend
    105. 105. CommutativeReplicated Data-Typeshttp://hal.inria.fr/inria-00397981/en/
    106. 106. CommutativeReplicated Data-Types Recent area of researchhttp://hal.inria.fr/inria-00397981/en/
    107. 107. CommutativeReplicated Data-Types Recent area of research Data-type defines commutative operationshttp://hal.inria.fr/inria-00397981/en/
    108. 108. CommutativeReplicated Data-Types Recent area of research Data-type defines commutative operations Value contains a state and limited history of operationshttp://hal.inria.fr/inria-00397981/en/
    109. 109. CommutativeReplicated Data-Types Recent area of research Data-type defines commutative operations Value contains a state and limited history of operations Roll-back, merge and replay to resolvehttp://hal.inria.fr/inria-00397981/en/
    110. 110. Currently, only Erlanghas a CRDT for Riak, called statebox.
    111. 111. We’re researching howto bring CRDTs to Ruby and other languages.
    112. 112. Conclusion
    113. 113. Conclusion Any sufficiently large system isinconsistent and constantly failing
    114. 114. Conclusion Any sufficiently large system is inconsistent and constantly failingRiak remains available during failures, progresses toward consistency
    115. 115. Conclusion Any sufficiently large system is inconsistent and constantly failingRiak remains available during failures, progresses toward consistency Riak & Ripple help your application recover from inconsistency
    116. 116. Abrazámonos Riak

    ×