Embrace NoSQL and        Eventual Consistency             with Ripple               Sean Cribbs            Basho Technolog...
UnderstandingEventual Consistency
Internet
Internet
Internet
Internet
InternetCDN
InternetCDN
InternetCDN      M     S            S
InternetCDN
Is this system consistent?
You sacrificed ACID  as soon as you added caching.
How does yourapplication cope?
Inconsistency Creep
Inconsistency Creep     Cache staleness
Inconsistency Creep       Cache staleness  Master-slave replication lag
Inconsistency Creep       Cache staleness  Master-slave replication lag     CDN propagation lag
Inconsistency Creep       Cache staleness  Master-slave replication lag     CDN propagation lag       Race-conditions
Inconsistency Creep       Cache staleness  Master-slave replication lag     CDN propagation lag       Race-conditions     ...
Eventual Consistency
Eventual ConsistencyTolerates temporary inconsistency
Eventual ConsistencyTolerates temporary inconsistency    Moves toward consistency          automatically
Eventual ConsistencyTolerates temporary inconsistency    Moves toward consistency          automatically  Resilient, maint...
Eventual ConsistencyTolerates temporary inconsistency    Moves toward consistency          automatically  Resilient, maint...
Eventual ConsistencyTolerates temporary inconsistency    Moves toward consistency          automatically  Resilient, maint...
Real-WorldEventual Consistency
Real-WorldEventual Consistency        DNS
Real-WorldEventual Consistency        DNS        LDAP
Real-WorldEventual Consistency        DNS        LDAP       The Web
Harvest & Yield
Harvest & Yield CAP “For Dummies”
Harvest & Yield      CAP “For Dummies”Harvest: how much of dataset is  reflected in a response (C)
Harvest & Yield      CAP “For Dummies”Harvest: how much of dataset is  reflected in a response (C)Yield: how likely is the ...
Harvest & Yieldhttp://codahale.com/you-cant-sacrifice-partition-                   tolerance/          CAP “For Dummies” Ha...
# Do I need it?def eventual_consistency?  @money === @uptimeend
# Do I need it?def eventual_consistency?  @money === @uptimeend# “Any sufficiently large# system is in a constant# state o...
Ok, I’ll use EventualConsistency. How?
Oh, those NoSQL     things.
NoSQL Flavors
NoSQL FlavorsGraph (Neo4J, InfiniteGraph)
NoSQL FlavorsGraph (Neo4J, InfiniteGraph)Document (Mongo, Couch)
NoSQL FlavorsGraph (Neo4J, InfiniteGraph)Document (Mongo, Couch)Column (HBase, Cassandra)
NoSQL FlavorsGraph (Neo4J, InfiniteGraph)Document (Mongo, Couch)Column (HBase, Cassandra)Key-Value (Riak,Voldemort)
NoSQL Flavors structured         Graph (Neo4J, InfiniteGraph)         Document (Mongo, Couch)         Column (HBase, Cassan...
Not all NoSQLs areEventually Consistent.
Riak implementsEventual Consistency.
What is Riak?
Riak is...
Riak is...a scalable
Riak is...   a scalablehighly-available
Riak is...      a scalable   highly-availablenetworked/distributed
Riak is...      a scalable   highly-availablenetworked/distributed   key-value store
Riak’s Data Model
Riak’s Data ModelStores values against keys
Riak’s Data ModelStores values against keys   Values are opaque
Riak’s Data Model  Stores values against keys     Values are opaqueKeys are grouped into buckets
Values Have Metadata
Values Have Metadata      Content-Type
Values Have Metadata      Content-Type     Last-Modified
Values Have Metadata      Content-Type     Last-Modified          Link
Values Have Metadata       Content-Type       Last-Modified            Link    Secondary index values
Values Have Metadata       Content-Type       Last-Modified            Link    Secondary index values      Custom metadata
Values Have Metadata        Content-Type       Last-Modified            Link    Secondary index values       Custom metada...
Basic Operations
Basic Operations GET /riak/bucket/key
Basic Operations GET /riak/bucket/key PUT /riak/bucket/key
Basic Operations GET /riak/bucket/key PUT /riak/bucket/keyDELETE /riak/bucket/key
Basic Operations GET /riak/bucket/key PUT /riak/bucket/keyDELETE /riak/bucket/key HTTP or Protocol Buffers
Other Query Methods
Other Query Methods Link-traversal (“walking”)
Other Query Methods Link-traversal (“walking”)     Full-text Search
Other Query Methods Link-traversal (“walking”)     Full-text Search    Secondary Indexes
Other Query Methods Link-traversal (“walking”)     Full-text Search    Secondary Indexes        MapReduce
Ripple connects  Ruby to Riak.
# gem install riak-clientrequire ‘riak’# Connect to Riak on localhostclient = Riak::Client.new# Fetch something, write it ...
# Create a new keyv = client[‘venues’].new(‘RubyConfAR’)# Set the value, JSON by defaultv.data = {  name: ‘Ciudad Cultural...
# Create a new key for an imagepic = client[‘pictures’].new(‘bue1.jpg’)# Read the file, set the content-typepic.raw_data =...
But I want models! Ok, no problem.
require ‘ripple’ # gem install rippleclass Person  include Ripple::Document  property :name, String, presence: true  key_o...
# Create my infome = Person.new(        name: ‘Sean Cribbs’,        addresses: [{             street: ‘929 Market St’,    ...
That’s kind of cool.    But what aboutEventual Consistency?
Riak uses CausalEventual Consistency.   also called Vector Clocks
Causal Consistency
Causal Consistency Each read includes a token of the    current state (vector clock)
Causal Consistency   Each read includes a token of the      current state (vector clock)Clients send the token back with w...
Causal Consistency   Each read includes a token of the      current state (vector clock)Clients send the token back with w...
Causal Consistency   Each read includes a token of the      current state (vector clock)Clients send the token back with w...
Conflicts? That sounds bad.
It happens frequently without strict   consistency. For example, havetwo threads write at the same time.   Riak detects th...
Your application is also  part of the Eventually  Consistent system. Itmakes the hard decisions    that Riak can’t.
Ok. So how do Iresolve conflicts?
Riak will give you all values   that can’t be resolved  automatically. Just write back the resolved value.
Think of it like merging or  rebasing git branches. Sometimes you MUST usegit mergetool to fix things.
Two Ways to Resolve
Two Ways to Resolve Semantic Resolution
Two Ways to Resolve Semantic Resolution       CRDTs
Semantic Resolution
Semantic Resolution Given any number of values,compute the resolved version    using business logic.
Example
Example If your value is a shopping cart,merge all items and sum quantities         into a single cart.
Example If your value is a shopping cart,merge all items and sum quantities         into a single cart.      Maybe the cus...
Ripple supportsSemantic Resolution.
# A cart is { product_id => quantity }Riak::RObject.on_conflict do |obj|  if obj.bucket.name == ‘carts’    obj.content_typ...
# Documents can be resolved tooPerson.on_conflict(:addresses) do |vals|  vals.each do |doc|    self.addresses &= doc.addre...
CommutativeReplicated Data-Typeshttp://hal.inria.fr/inria-00397981/en/
CommutativeReplicated Data-Types        Recent area of researchhttp://hal.inria.fr/inria-00397981/en/
CommutativeReplicated Data-Types        Recent area of research          Data-type defines       commutative operationshttp...
CommutativeReplicated Data-Types        Recent area of research          Data-type defines       commutative operations    ...
CommutativeReplicated Data-Types        Recent area of research          Data-type defines       commutative operations    ...
Currently, only Erlanghas a CRDT for Riak, called statebox.
We’re researching howto bring CRDTs to Ruby and other languages.
Conclusion
Conclusion   Any sufficiently large system isinconsistent and constantly failing
Conclusion    Any sufficiently large system is inconsistent and constantly failingRiak remains available during failures, ...
Conclusion    Any sufficiently large system is inconsistent and constantly failingRiak remains available during failures, ...
Abrazámonos Riak
Embrace NoSQL and Eventual Consistency with Ripple
Embrace NoSQL and Eventual Consistency with Ripple
Embrace NoSQL and Eventual Consistency with Ripple
Upcoming SlideShare
Loading in...5
×

Embrace NoSQL and Eventual Consistency with Ripple

2,453

Published on

So, there's this "NoSQL" thing you may have heard of, and this related thing called "eventual consistency". Supposedly, they help you scale, but no one has ever explained why! Well, wonder no more! This talk will demystify NoSQL, eventual consistency, how they might help you scale, and -- most importantly -- why you should care.

We'll look closely at how Riak, a linearly-scalable, distributed and fault-tolerant NoSQL datastore, implements eventual consistency, and how you can harness it from Ruby via the slick Ripple client/ORM. When the talk is finished, you'll have the tools both to understand eventual consistency and to handle it like a pro inside your next Ruby application.

Published in: Technology
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,453
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
46
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Embrace NoSQL and Eventual Consistency with Ripple

    1. 1. Embrace NoSQL and Eventual Consistency with Ripple Sean Cribbs Basho Technologies @seancribbsbasho
    2. 2. UnderstandingEventual Consistency
    3. 3. Internet
    4. 4. Internet
    5. 5. Internet
    6. 6. Internet
    7. 7. InternetCDN
    8. 8. InternetCDN
    9. 9. InternetCDN M S S
    10. 10. InternetCDN
    11. 11. Is this system consistent?
    12. 12. You sacrificed ACID as soon as you added caching.
    13. 13. How does yourapplication cope?
    14. 14. Inconsistency Creep
    15. 15. Inconsistency Creep Cache staleness
    16. 16. Inconsistency Creep Cache staleness Master-slave replication lag
    17. 17. Inconsistency Creep Cache staleness Master-slave replication lag CDN propagation lag
    18. 18. Inconsistency Creep Cache staleness Master-slave replication lag CDN propagation lag Race-conditions
    19. 19. Inconsistency Creep Cache staleness Master-slave replication lag CDN propagation lag Race-conditions Service failures
    20. 20. Eventual Consistency
    21. 21. Eventual ConsistencyTolerates temporary inconsistency
    22. 22. Eventual ConsistencyTolerates temporary inconsistency Moves toward consistency automatically
    23. 23. Eventual ConsistencyTolerates temporary inconsistency Moves toward consistency automatically Resilient, maintains availability
    24. 24. Eventual ConsistencyTolerates temporary inconsistency Moves toward consistency automatically Resilient, maintains availability “Eventual” in seconds, not days
    25. 25. Eventual ConsistencyTolerates temporary inconsistency Moves toward consistency automatically Resilient, maintains availability “Eventual” in seconds, not days
    26. 26. Real-WorldEventual Consistency
    27. 27. Real-WorldEventual Consistency DNS
    28. 28. Real-WorldEventual Consistency DNS LDAP
    29. 29. Real-WorldEventual Consistency DNS LDAP The Web
    30. 30. Harvest & Yield
    31. 31. Harvest & Yield CAP “For Dummies”
    32. 32. Harvest & Yield CAP “For Dummies”Harvest: how much of dataset is reflected in a response (C)
    33. 33. Harvest & Yield CAP “For Dummies”Harvest: how much of dataset is reflected in a response (C)Yield: how likely is the datastore to complete request (A)
    34. 34. Harvest & Yieldhttp://codahale.com/you-cant-sacrifice-partition- tolerance/ CAP “For Dummies” Harvest: how much of dataset is reflected in a response (C) Yield: how likely is the datastore to complete request (A)
    35. 35. # Do I need it?def eventual_consistency? @money === @uptimeend
    36. 36. # Do I need it?def eventual_consistency? @money === @uptimeend# “Any sufficiently large# system is in a constant# state of failure.”# --Justin Sheehy, Basho CTO
    37. 37. Ok, I’ll use EventualConsistency. How?
    38. 38. Oh, those NoSQL things.
    39. 39. NoSQL Flavors
    40. 40. NoSQL FlavorsGraph (Neo4J, InfiniteGraph)
    41. 41. NoSQL FlavorsGraph (Neo4J, InfiniteGraph)Document (Mongo, Couch)
    42. 42. NoSQL FlavorsGraph (Neo4J, InfiniteGraph)Document (Mongo, Couch)Column (HBase, Cassandra)
    43. 43. NoSQL FlavorsGraph (Neo4J, InfiniteGraph)Document (Mongo, Couch)Column (HBase, Cassandra)Key-Value (Riak,Voldemort)
    44. 44. NoSQL Flavors structured Graph (Neo4J, InfiniteGraph) Document (Mongo, Couch) Column (HBase, Cassandra) Key-Value (Riak, Voldemort)unstructured
    45. 45. Not all NoSQLs areEventually Consistent.
    46. 46. Riak implementsEventual Consistency.
    47. 47. What is Riak?
    48. 48. Riak is...
    49. 49. Riak is...a scalable
    50. 50. Riak is... a scalablehighly-available
    51. 51. Riak is... a scalable highly-availablenetworked/distributed
    52. 52. Riak is... a scalable highly-availablenetworked/distributed key-value store
    53. 53. Riak’s Data Model
    54. 54. Riak’s Data ModelStores values against keys
    55. 55. Riak’s Data ModelStores values against keys Values are opaque
    56. 56. Riak’s Data Model Stores values against keys Values are opaqueKeys are grouped into buckets
    57. 57. Values Have Metadata
    58. 58. Values Have Metadata Content-Type
    59. 59. Values Have Metadata Content-Type Last-Modified
    60. 60. Values Have Metadata Content-Type Last-Modified Link
    61. 61. Values Have Metadata Content-Type Last-Modified Link Secondary index values
    62. 62. Values Have Metadata Content-Type Last-Modified Link Secondary index values Custom metadata
    63. 63. Values Have Metadata Content-Type Last-Modified Link Secondary index values Custom metadata Value + Metadata == “Object”
    64. 64. Basic Operations
    65. 65. Basic Operations GET /riak/bucket/key
    66. 66. Basic Operations GET /riak/bucket/key PUT /riak/bucket/key
    67. 67. Basic Operations GET /riak/bucket/key PUT /riak/bucket/keyDELETE /riak/bucket/key
    68. 68. Basic Operations GET /riak/bucket/key PUT /riak/bucket/keyDELETE /riak/bucket/key HTTP or Protocol Buffers
    69. 69. Other Query Methods
    70. 70. Other Query Methods Link-traversal (“walking”)
    71. 71. Other Query Methods Link-traversal (“walking”) Full-text Search
    72. 72. Other Query Methods Link-traversal (“walking”) Full-text Search Secondary Indexes
    73. 73. Other Query Methods Link-traversal (“walking”) Full-text Search Secondary Indexes MapReduce
    74. 74. Ripple connects Ruby to Riak.
    75. 75. # gem install riak-clientrequire ‘riak’# Connect to Riak on localhostclient = Riak::Client.new# Fetch something, write it to a fileo = client[‘slide-decks’][‘RubyConfAR’]File.open(‘slides.key’,‘wb’) do |f| f.write o.raw_dataend
    76. 76. # Create a new keyv = client[‘venues’].new(‘RubyConfAR’)# Set the value, JSON by defaultv.data = { name: ‘Ciudad Cultural Konex’, address: { street: ‘Sarmiento 3131’, city: ‘Buenos Aires’, zip: ‘C1196AAG’ }}# Write it to Riakv.store
    77. 77. # Create a new key for an imagepic = client[‘pictures’].new(‘bue1.jpg’)# Read the file, set the content-typepic.raw_data = File.read(‘bue1.jpg’)pic.content_type = ‘image/jpeg’# Set some metadatapic.meta[‘camera’] = ‘Nikon’pic.indexes[‘date_int’] << Time.now.to_i# Write it to Riakpic.store
    78. 78. But I want models! Ok, no problem.
    79. 79. require ‘ripple’ # gem install rippleclass Person include Ripple::Document property :name, String, presence: true key_on :name many :addressesendclass Address include Ripple::EmbeddedDocument property :street, String property :city, String property :zip, Stringend
    80. 80. # Create my infome = Person.new( name: ‘Sean Cribbs’, addresses: [{ street: ‘929 Market St’, city: ‘San Francisco’, zip: ‘94103’ }] )me.save! # Store as JSON under # “people/Sean Cribbs”me == Person.find(‘Sean Cribbs’) # true
    81. 81. That’s kind of cool. But what aboutEventual Consistency?
    82. 82. Riak uses CausalEventual Consistency. also called Vector Clocks
    83. 83. Causal Consistency
    84. 84. Causal Consistency Each read includes a token of the current state (vector clock)
    85. 85. Causal Consistency Each read includes a token of the current state (vector clock)Clients send the token back with writes
    86. 86. Causal Consistency Each read includes a token of the current state (vector clock)Clients send the token back with writesRiak updates the token and detects stale versions and conflicts by comparison
    87. 87. Causal Consistency Each read includes a token of the current state (vector clock)Clients send the token back with writesRiak updates the token and detects stale versions and conflicts by comparison Conflicts are exposed to your application
    88. 88. Conflicts? That sounds bad.
    89. 89. It happens frequently without strict consistency. For example, havetwo threads write at the same time. Riak detects this conflict for you.
    90. 90. Your application is also part of the Eventually Consistent system. Itmakes the hard decisions that Riak can’t.
    91. 91. Ok. So how do Iresolve conflicts?
    92. 92. Riak will give you all values that can’t be resolved automatically. Just write back the resolved value.
    93. 93. Think of it like merging or rebasing git branches. Sometimes you MUST usegit mergetool to fix things.
    94. 94. Two Ways to Resolve
    95. 95. Two Ways to Resolve Semantic Resolution
    96. 96. Two Ways to Resolve Semantic Resolution CRDTs
    97. 97. Semantic Resolution
    98. 98. Semantic Resolution Given any number of values,compute the resolved version using business logic.
    99. 99. Example
    100. 100. Example If your value is a shopping cart,merge all items and sum quantities into a single cart.
    101. 101. Example If your value is a shopping cart,merge all items and sum quantities into a single cart. Maybe the customer buys more that way!
    102. 102. Ripple supportsSemantic Resolution.
    103. 103. # A cart is { product_id => quantity }Riak::RObject.on_conflict do |obj| if obj.bucket.name == ‘carts’ obj.content_type = ‘application/json’ obj.data = {} obj.siblings.each do |val| val.data.each do |product,q| obj.data[product] ||= 0 obj.data[product] += q end end obj endend
    104. 104. # Documents can be resolved tooPerson.on_conflict(:addresses) do |vals| vals.each do |doc| self.addresses &= doc.addresses endend
    105. 105. CommutativeReplicated Data-Typeshttp://hal.inria.fr/inria-00397981/en/
    106. 106. CommutativeReplicated Data-Types Recent area of researchhttp://hal.inria.fr/inria-00397981/en/
    107. 107. CommutativeReplicated Data-Types Recent area of research Data-type defines commutative operationshttp://hal.inria.fr/inria-00397981/en/
    108. 108. CommutativeReplicated Data-Types Recent area of research Data-type defines commutative operations Value contains a state and limited history of operationshttp://hal.inria.fr/inria-00397981/en/
    109. 109. CommutativeReplicated Data-Types Recent area of research Data-type defines commutative operations Value contains a state and limited history of operations Roll-back, merge and replay to resolvehttp://hal.inria.fr/inria-00397981/en/
    110. 110. Currently, only Erlanghas a CRDT for Riak, called statebox.
    111. 111. We’re researching howto bring CRDTs to Ruby and other languages.
    112. 112. Conclusion
    113. 113. Conclusion Any sufficiently large system isinconsistent and constantly failing
    114. 114. Conclusion Any sufficiently large system is inconsistent and constantly failingRiak remains available during failures, progresses toward consistency
    115. 115. Conclusion Any sufficiently large system is inconsistent and constantly failingRiak remains available during failures, progresses toward consistency Riak & Ripple help your application recover from inconsistency
    116. 116. Abrazámonos Riak
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×