Embrace NoSQL and Eventual Consistency with Ripple

  • 2,271 views
Uploaded on

So, there's this "NoSQL" thing you may have heard of, and this related thing called "eventual consistency". Supposedly, they help you scale, but no one has ever explained why! Well, wonder no more! …

So, there's this "NoSQL" thing you may have heard of, and this related thing called "eventual consistency". Supposedly, they help you scale, but no one has ever explained why! Well, wonder no more! This talk will demystify NoSQL, eventual consistency, how they might help you scale, and -- most importantly -- why you should care.

We'll look closely at how Riak, a linearly-scalable, distributed and fault-tolerant NoSQL datastore, implements eventual consistency, and how you can harness it from Ruby via the slick Ripple client/ORM. When the talk is finished, you'll have the tools both to understand eventual consistency and to handle it like a pro inside your next Ruby application.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
2,271
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
32
Comments
0
Likes
5

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n

Transcript

  • 1. Embrace NoSQL and Eventual Consistency with Ripple Sean Cribbs Basho Technologies @seancribbsbasho
  • 2. UnderstandingEventual Consistency
  • 3. Internet
  • 4. Internet
  • 5. Internet
  • 6. Internet
  • 7. InternetCDN
  • 8. InternetCDN
  • 9. InternetCDN M S S
  • 10. InternetCDN
  • 11. Is this system consistent?
  • 12. You sacrificed ACID as soon as you added caching.
  • 13. How does yourapplication cope?
  • 14. Inconsistency Creep
  • 15. Inconsistency Creep Cache staleness
  • 16. Inconsistency Creep Cache staleness Master-slave replication lag
  • 17. Inconsistency Creep Cache staleness Master-slave replication lag CDN propagation lag
  • 18. Inconsistency Creep Cache staleness Master-slave replication lag CDN propagation lag Race-conditions
  • 19. Inconsistency Creep Cache staleness Master-slave replication lag CDN propagation lag Race-conditions Service failures
  • 20. Eventual Consistency
  • 21. Eventual ConsistencyTolerates temporary inconsistency
  • 22. Eventual ConsistencyTolerates temporary inconsistency Moves toward consistency automatically
  • 23. Eventual ConsistencyTolerates temporary inconsistency Moves toward consistency automatically Resilient, maintains availability
  • 24. Eventual ConsistencyTolerates temporary inconsistency Moves toward consistency automatically Resilient, maintains availability “Eventual” in seconds, not days
  • 25. Eventual ConsistencyTolerates temporary inconsistency Moves toward consistency automatically Resilient, maintains availability “Eventual” in seconds, not days
  • 26. Real-WorldEventual Consistency
  • 27. Real-WorldEventual Consistency DNS
  • 28. Real-WorldEventual Consistency DNS LDAP
  • 29. Real-WorldEventual Consistency DNS LDAP The Web
  • 30. Harvest & Yield
  • 31. Harvest & Yield CAP “For Dummies”
  • 32. Harvest & Yield CAP “For Dummies”Harvest: how much of dataset is reflected in a response (C)
  • 33. Harvest & Yield CAP “For Dummies”Harvest: how much of dataset is reflected in a response (C)Yield: how likely is the datastore to complete request (A)
  • 34. Harvest & Yieldhttp://codahale.com/you-cant-sacrifice-partition- tolerance/ CAP “For Dummies” Harvest: how much of dataset is reflected in a response (C) Yield: how likely is the datastore to complete request (A)
  • 35. # Do I need it?def eventual_consistency? @money === @uptimeend
  • 36. # Do I need it?def eventual_consistency? @money === @uptimeend# “Any sufficiently large# system is in a constant# state of failure.”# --Justin Sheehy, Basho CTO
  • 37. Ok, I’ll use EventualConsistency. How?
  • 38. Oh, those NoSQL things.
  • 39. NoSQL Flavors
  • 40. NoSQL FlavorsGraph (Neo4J, InfiniteGraph)
  • 41. NoSQL FlavorsGraph (Neo4J, InfiniteGraph)Document (Mongo, Couch)
  • 42. NoSQL FlavorsGraph (Neo4J, InfiniteGraph)Document (Mongo, Couch)Column (HBase, Cassandra)
  • 43. NoSQL FlavorsGraph (Neo4J, InfiniteGraph)Document (Mongo, Couch)Column (HBase, Cassandra)Key-Value (Riak,Voldemort)
  • 44. NoSQL Flavors structured Graph (Neo4J, InfiniteGraph) Document (Mongo, Couch) Column (HBase, Cassandra) Key-Value (Riak, Voldemort)unstructured
  • 45. Not all NoSQLs areEventually Consistent.
  • 46. Riak implementsEventual Consistency.
  • 47. What is Riak?
  • 48. Riak is...
  • 49. Riak is...a scalable
  • 50. Riak is... a scalablehighly-available
  • 51. Riak is... a scalable highly-availablenetworked/distributed
  • 52. Riak is... a scalable highly-availablenetworked/distributed key-value store
  • 53. Riak’s Data Model
  • 54. Riak’s Data ModelStores values against keys
  • 55. Riak’s Data ModelStores values against keys Values are opaque
  • 56. Riak’s Data Model Stores values against keys Values are opaqueKeys are grouped into buckets
  • 57. Values Have Metadata
  • 58. Values Have Metadata Content-Type
  • 59. Values Have Metadata Content-Type Last-Modified
  • 60. Values Have Metadata Content-Type Last-Modified Link
  • 61. Values Have Metadata Content-Type Last-Modified Link Secondary index values
  • 62. Values Have Metadata Content-Type Last-Modified Link Secondary index values Custom metadata
  • 63. Values Have Metadata Content-Type Last-Modified Link Secondary index values Custom metadata Value + Metadata == “Object”
  • 64. Basic Operations
  • 65. Basic Operations GET /riak/bucket/key
  • 66. Basic Operations GET /riak/bucket/key PUT /riak/bucket/key
  • 67. Basic Operations GET /riak/bucket/key PUT /riak/bucket/keyDELETE /riak/bucket/key
  • 68. Basic Operations GET /riak/bucket/key PUT /riak/bucket/keyDELETE /riak/bucket/key HTTP or Protocol Buffers
  • 69. Other Query Methods
  • 70. Other Query Methods Link-traversal (“walking”)
  • 71. Other Query Methods Link-traversal (“walking”) Full-text Search
  • 72. Other Query Methods Link-traversal (“walking”) Full-text Search Secondary Indexes
  • 73. Other Query Methods Link-traversal (“walking”) Full-text Search Secondary Indexes MapReduce
  • 74. Ripple connects Ruby to Riak.
  • 75. # gem install riak-clientrequire ‘riak’# Connect to Riak on localhostclient = Riak::Client.new# Fetch something, write it to a fileo = client[‘slide-decks’][‘RubyConfAR’]File.open(‘slides.key’,‘wb’) do |f| f.write o.raw_dataend
  • 76. # Create a new keyv = client[‘venues’].new(‘RubyConfAR’)# Set the value, JSON by defaultv.data = { name: ‘Ciudad Cultural Konex’, address: { street: ‘Sarmiento 3131’, city: ‘Buenos Aires’, zip: ‘C1196AAG’ }}# Write it to Riakv.store
  • 77. # Create a new key for an imagepic = client[‘pictures’].new(‘bue1.jpg’)# Read the file, set the content-typepic.raw_data = File.read(‘bue1.jpg’)pic.content_type = ‘image/jpeg’# Set some metadatapic.meta[‘camera’] = ‘Nikon’pic.indexes[‘date_int’] << Time.now.to_i# Write it to Riakpic.store
  • 78. But I want models! Ok, no problem.
  • 79. require ‘ripple’ # gem install rippleclass Person include Ripple::Document property :name, String, presence: true key_on :name many :addressesendclass Address include Ripple::EmbeddedDocument property :street, String property :city, String property :zip, Stringend
  • 80. # Create my infome = Person.new( name: ‘Sean Cribbs’, addresses: [{ street: ‘929 Market St’, city: ‘San Francisco’, zip: ‘94103’ }] )me.save! # Store as JSON under # “people/Sean Cribbs”me == Person.find(‘Sean Cribbs’) # true
  • 81. That’s kind of cool. But what aboutEventual Consistency?
  • 82. Riak uses CausalEventual Consistency. also called Vector Clocks
  • 83. Causal Consistency
  • 84. Causal Consistency Each read includes a token of the current state (vector clock)
  • 85. Causal Consistency Each read includes a token of the current state (vector clock)Clients send the token back with writes
  • 86. Causal Consistency Each read includes a token of the current state (vector clock)Clients send the token back with writesRiak updates the token and detects stale versions and conflicts by comparison
  • 87. Causal Consistency Each read includes a token of the current state (vector clock)Clients send the token back with writesRiak updates the token and detects stale versions and conflicts by comparison Conflicts are exposed to your application
  • 88. Conflicts? That sounds bad.
  • 89. It happens frequently without strict consistency. For example, havetwo threads write at the same time. Riak detects this conflict for you.
  • 90. Your application is also part of the Eventually Consistent system. Itmakes the hard decisions that Riak can’t.
  • 91. Ok. So how do Iresolve conflicts?
  • 92. Riak will give you all values that can’t be resolved automatically. Just write back the resolved value.
  • 93. Think of it like merging or rebasing git branches. Sometimes you MUST usegit mergetool to fix things.
  • 94. Two Ways to Resolve
  • 95. Two Ways to Resolve Semantic Resolution
  • 96. Two Ways to Resolve Semantic Resolution CRDTs
  • 97. Semantic Resolution
  • 98. Semantic Resolution Given any number of values,compute the resolved version using business logic.
  • 99. Example
  • 100. Example If your value is a shopping cart,merge all items and sum quantities into a single cart.
  • 101. Example If your value is a shopping cart,merge all items and sum quantities into a single cart. Maybe the customer buys more that way!
  • 102. Ripple supportsSemantic Resolution.
  • 103. # A cart is { product_id => quantity }Riak::RObject.on_conflict do |obj| if obj.bucket.name == ‘carts’ obj.content_type = ‘application/json’ obj.data = {} obj.siblings.each do |val| val.data.each do |product,q| obj.data[product] ||= 0 obj.data[product] += q end end obj endend
  • 104. # Documents can be resolved tooPerson.on_conflict(:addresses) do |vals| vals.each do |doc| self.addresses &= doc.addresses endend
  • 105. CommutativeReplicated Data-Typeshttp://hal.inria.fr/inria-00397981/en/
  • 106. CommutativeReplicated Data-Types Recent area of researchhttp://hal.inria.fr/inria-00397981/en/
  • 107. CommutativeReplicated Data-Types Recent area of research Data-type defines commutative operationshttp://hal.inria.fr/inria-00397981/en/
  • 108. CommutativeReplicated Data-Types Recent area of research Data-type defines commutative operations Value contains a state and limited history of operationshttp://hal.inria.fr/inria-00397981/en/
  • 109. CommutativeReplicated Data-Types Recent area of research Data-type defines commutative operations Value contains a state and limited history of operations Roll-back, merge and replay to resolvehttp://hal.inria.fr/inria-00397981/en/
  • 110. Currently, only Erlanghas a CRDT for Riak, called statebox.
  • 111. We’re researching howto bring CRDTs to Ruby and other languages.
  • 112. Conclusion
  • 113. Conclusion Any sufficiently large system isinconsistent and constantly failing
  • 114. Conclusion Any sufficiently large system is inconsistent and constantly failingRiak remains available during failures, progresses toward consistency
  • 115. Conclusion Any sufficiently large system is inconsistent and constantly failingRiak remains available during failures, progresses toward consistency Riak & Ripple help your application recover from inconsistency
  • 116. Abrazámonos Riak