Introducing Riak and Ripple

  • 11,702 views
Uploaded on

An introduction to Basho's Riak distributed data store, and the Ripple client in Ruby. Code samples from the demos are here: http://gist.github.com/365791

An introduction to Basho's Riak distributed data store, and the Ripple client in Ruby. Code samples from the demos are here: http://gist.github.com/365791

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
11,702
On Slideshare
0
From Embeds
0
Number of Embeds
2

Actions

Shares
Downloads
275
Comments
0
Likes
24

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Introducing Riak and Ripple Sean Cribbs Wednesday, April 14, 2010
  • 2. system “whoami” Wednesday, April 14, 2010
  • 3. system “whoami” BCS, three years as a musician Wednesday, April 14, 2010
  • 4. system “whoami” BCS, three years as a musician Started with Ruby in early 2006 Wednesday, April 14, 2010
  • 5. system “whoami” BCS, three years as a musician Started with Ruby in early 2006 Rails freelancer 2007-2010 Wednesday, April 14, 2010
  • 6. system “whoami” BCS, three years as a musician Started with Ruby in early 2006 Rails freelancer 2007-2010 Radiant CMS Wednesday, April 14, 2010
  • 7. system “whoami” BCS, three years as a musician Started with Ruby in early 2006 Rails freelancer 2007-2010 Radiant CMS March 2010 - Developer Advocate basho Wednesday, April 14, 2010
  • 8. Wednesday, April 14, 2010
  • 9. Wednesday, April 14, 2010
  • 10. Rails Can’t Scale Wednesday, April 14, 2010
  • 11. Rails Can’t Scale what does it mean? Wednesday, April 14, 2010
  • 12. Ruby Rails Can’t Scale what does it mean? Wednesday, April 14, 2010
  • 13. MySQL Ruby Rails Can’t Scale what does it mean? Wednesday, April 14, 2010
  • 14. insert straw man here MySQL Ruby Rails Can’t Scale what does it mean? Wednesday, April 14, 2010
  • 15. Mythbusting “scalability” Wednesday, April 14, 2010
  • 16. Mythbusting “scalability” Scalability is not a yes/no question Wednesday, April 14, 2010
  • 17. Mythbusting “scalability” Scalability is not a yes/no question It’s a ratio of benefit to cost Wednesday, April 14, 2010
  • 18. Mythbusting “scalability” Scalability is not a yes/no question It’s a ratio of benefit to cost Benefits: low latency, throughput, uptime, concurrency, reliability Wednesday, April 14, 2010
  • 19. Mythbusting “scalability” Scalability is not a yes/no question It’s a ratio of benefit to cost Benefits: low latency, throughput, uptime, concurrency, reliability Costs: CPU, RAM, disk, bandwidth, power, hw/sw Wednesday, April 14, 2010
  • 20. Mythbusting “scalability” Scalability is not a yes/no question It’s a ratio of benefit to cost Benefits: low latency, throughput, uptime, concurrency, reliability Costs: CPU, RAM, disk, bandwidth, power, hw/sw scalability = bang for your buck (ROI) Wednesday, April 14, 2010
  • 21. Scaling goes both ways Wednesday, April 14, 2010
  • 22. Scaling goes both ways Up - high traffic, “Big Data”, multi-datacenter Wednesday, April 14, 2010
  • 23. Scaling goes both ways Up - high traffic, “Big Data”, multi-datacenter Down - laptop, set-top, mobile Wednesday, April 14, 2010
  • 24. How do you scale? Wednesday, April 14, 2010
  • 25. How do you scale? Vertically - get bigger, expensive machines Wednesday, April 14, 2010
  • 26. How do you scale? Vertically - get bigger, expensive machines Horizontally - get more commodity machines Wednesday, April 14, 2010
  • 27. What is Riak? (the horizontal scaling bits) Wednesday, April 14, 2010
  • 28. What is Riak? (the horizontal scaling bits) Based on Amazon’s Dynamo (2007) Wednesday, April 14, 2010
  • 29. What is Riak? (the horizontal scaling bits) Based on Amazon’s Dynamo (2007) key-value storage Wednesday, April 14, 2010
  • 30. What is Riak? (the horizontal scaling bits) Based on Amazon’s Dynamo (2007) key-value storage masterless, peer-to-peer replication Wednesday, April 14, 2010
  • 31. What is Riak? (the horizontal scaling bits) Based on Amazon’s Dynamo (2007) key-value storage masterless, peer-to-peer replication consistent hashing Wednesday, April 14, 2010
  • 32. What is Riak? (the horizontal scaling bits) Based on Amazon’s Dynamo (2007) key-value storage masterless, peer-to-peer replication consistent hashing eventual consistency Wednesday, April 14, 2010
  • 33. What is Riak? (the horizontal scaling bits) Based on Amazon’s Dynamo (2007) key-value storage masterless, peer-to-peer replication consistent hashing eventual consistency failover - quorums, hinted handoff Wednesday, April 14, 2010
  • 34. “O/RM is the Vietnam of Computer Science.” -- Ted Neward, 2004/6 Wednesday, April 14, 2010
  • 35. “O/RM is the Vietnam of Computer Science.” -- Ted Neward, 2004/6 slippery slope Wednesday, April 14, 2010
  • 36. “O/RM is the Vietnam of Computer Science.” -- Ted Neward, 2004/6 slippery slope diminishing returns Wednesday, April 14, 2010
  • 37. Impedance mismatch Wednesday, April 14, 2010
  • 38. Impedance mismatch Relational = predicate logic, truth statements Wednesday, April 14, 2010
  • 39. Impedance mismatch Relational = predicate logic, truth statements Object = identity, state, behavior, encapsulation Wednesday, April 14, 2010
  • 40. Pick your poison Wednesday, April 14, 2010
  • 41. Pick your poison Make the objects fit the database Wednesday, April 14, 2010
  • 42. Pick your poison Make the objects fit the database Active Record Wednesday, April 14, 2010
  • 43. Pick your poison Make the objects fit the database Active Record Make the database fit the objects Wednesday, April 14, 2010
  • 44. Pick your poison Make the objects fit the database Active Record Make the database fit the objects OODBMS, Graph DBs Wednesday, April 14, 2010
  • 45. Middle ground Wednesday, April 14, 2010
  • 46. Middle ground Document databases (identity and state) Wednesday, April 14, 2010
  • 47. Middle ground Document databases (identity and state) No JOINs - denormalized / composed Wednesday, April 14, 2010
  • 48. Middle ground Document databases (identity and state) No JOINs - denormalized / composed Loosely structured Wednesday, April 14, 2010
  • 49. Middle ground Document databases (identity and state) No JOINs - denormalized / composed Loosely structured Friendly transport formats, e.g. JSON Wednesday, April 14, 2010
  • 50. What is Riak? (the document database bits) Wednesday, April 14, 2010
  • 51. What is Riak? (the document database bits) Store your objects as JSON (or any format) Wednesday, April 14, 2010
  • 52. What is Riak? (the document database bits) Store your objects as JSON (or any format) Link between objects (like hypertext) Wednesday, April 14, 2010
  • 53. What is Riak? (the document database bits) Store your objects as JSON (or any format) Link between objects (like hypertext) No SQL - query with Javascript Map-Reduce Wednesday, April 14, 2010
  • 54. “DevOps” - agile infrastructure Wednesday, April 14, 2010
  • 55. “DevOps” - agile infrastructure programming your infrastructure Wednesday, April 14, 2010
  • 56. “DevOps” - agile infrastructure programming your infrastructure rapid “cloud” deployments Wednesday, April 14, 2010
  • 57. Befriend your sysadmin Wednesday, April 14, 2010
  • 58. Befriend your sysadmin Integrate with existing infrastructure Wednesday, April 14, 2010
  • 59. Befriend your sysadmin Integrate with existing infrastructure Reduce the amount of hands-on work Wednesday, April 14, 2010
  • 60. Befriend your sysadmin Integrate with existing infrastructure Reduce the amount of hands-on work Have a growth plan Wednesday, April 14, 2010
  • 61. What is Riak? (the ops-friendly bits) Wednesday, April 14, 2010
  • 62. What is Riak? (the ops-friendly bits) Web-shaped storage Wednesday, April 14, 2010
  • 63. What is Riak? (the ops-friendly bits) Web-shaped storage Store data in its original format Wednesday, April 14, 2010
  • 64. What is Riak? (the ops-friendly bits) Web-shaped storage Store data in its original format It’s just HTTP - same techniques apply Wednesday, April 14, 2010
  • 65. What is Riak? (the ops-friendly bits) Web-shaped storage Store data in its original format It’s just HTTP - same techniques apply load balancing, proxy caches, round-robin DNS Wednesday, April 14, 2010
  • 66. What is Riak? (the ops-friendly bits) Web-shaped storage Store data in its original format It’s just HTTP - same techniques apply load balancing, proxy caches, round-robin DNS No node is special - grow horizontally Wednesday, April 14, 2010
  • 67. What is Riak? (the ops-friendly bits) Web-shaped storage Store data in its original format It’s just HTTP - same techniques apply load balancing, proxy caches, round-robin DNS No node is special - grow horizontally Get some sleep, fix it in the morning Wednesday, April 14, 2010
  • 68. To review, Riak is... Wednesday, April 14, 2010
  • 69. To review, Riak is... Magical Horizontally- Scaling Unicorns Wednesday, April 14, 2010
  • 70. To review, Riak is... Magical Horizontally- Scaling Unicorns Application-Friendly Rainbows in the Cloud Wednesday, April 14, 2010
  • 71. To review, Riak is... Magical Horizontally- Scaling Unicorns Application-Friendly Rainbows in the Cloud Data-Shredding MapReduce Ninjas Wednesday, April 14, 2010
  • 72. http://downloads.basho.com/riak/ hg clone http://hg.basho.com/riak/ Wednesday, April 14, 2010
  • 73. gem install ripple git clone git://github.com/seancribbs/ripple.git Wednesday, April 14, 2010
  • 74. CRUD in Riak Wednesday, April 14, 2010
  • 75. CRUD in Riak PUT /riak/bucket/key (C,U) (also POST) Wednesday, April 14, 2010
  • 76. CRUD in Riak PUT /riak/bucket/key (C,U) (also POST) GET /riak/bucket/key (R) Wednesday, April 14, 2010
  • 77. CRUD in Riak PUT /riak/bucket/key (C,U) (also POST) GET /riak/bucket/key (R) DELETE /riak/bucket/key (D) Wednesday, April 14, 2010
  • 78. CRUD Demo Wednesday, April 14, 2010
  • 79. Links Wednesday, April 14, 2010
  • 80. Links One-way, qualified “pointers” to other objects Wednesday, April 14, 2010
  • 81. Links One-way, qualified “pointers” to other objects “Link” header in HTTP Wednesday, April 14, 2010
  • 82. Links One-way, qualified “pointers” to other objects “Link” header in HTTP </riak/bucket/key>; riaktag=”tag” Wednesday, April 14, 2010
  • 83. Link-walking Wednesday, April 14, 2010
  • 84. Link-walking GET /riak/bucket/key/[bucket,tag,keep]* Wednesday, April 14, 2010
  • 85. Link-walking GET /riak/bucket/key/[bucket,tag,keep]* bucket = scope to a bucket Wednesday, April 14, 2010
  • 86. Link-walking GET /riak/bucket/key/[bucket,tag,keep]* bucket = scope to a bucket tag = scope to a tag Wednesday, April 14, 2010
  • 87. Link-walking GET /riak/bucket/key/[bucket,tag,keep]* bucket = scope to a bucket tag = scope to a tag keep = return results (last one always) Wednesday, April 14, 2010
  • 88. Link-walking GET /riak/bucket/key/[bucket,tag,keep]* bucket = scope to a bucket tag = scope to a tag keep = return results (last one always) “_” means match anything Wednesday, April 14, 2010
  • 89. Link-walking GET /riak/bucket/key/[bucket,tag,keep]* bucket = scope to a bucket tag = scope to a tag keep = return results (last one always) “_” means match anything Example: Wednesday, April 14, 2010
  • 90. Link-walking GET /riak/bucket/key/[bucket,tag,keep]* bucket = scope to a bucket tag = scope to a tag keep = return results (last one always) “_” means match anything Example: GET /riak/artists/TheBeatles/albums,_,1/tracks,_,1 Wednesday, April 14, 2010
  • 91. Links Demo Wednesday, April 14, 2010
  • 92. Riak Map-Reduce I believe I did, Bob. Wednesday, April 14, 2010
  • 93. Riak Map-Reduce Wednesday, April 14, 2010
  • 94. Riak Map-Reduce Based on Google’s Map-Reduce idea Wednesday, April 14, 2010
  • 95. Riak Map-Reduce Based on Google’s Map-Reduce idea Some similarities to Hadoop and CouchDB Wednesday, April 14, 2010
  • 96. Riak Map-Reduce Based on Google’s Map-Reduce idea Some similarities to Hadoop and CouchDB High data-locality Wednesday, April 14, 2010
  • 97. Riak Map-Reduce Based on Google’s Map-Reduce idea Some similarities to Hadoop and CouchDB High data-locality Phases can be Javascript or Erlang Wednesday, April 14, 2010
  • 98. Map-Reduce Terminology Wednesday, April 14, 2010
  • 99. Map-Reduce Terminology Job: a query, composed of inputs and phases Wednesday, April 14, 2010
  • 100. Map-Reduce Terminology Job: a query, composed of inputs and phases Phase: a single map or reduce function application Wednesday, April 14, 2010
  • 101. Map-Reduce Terminology Job: a query, composed of inputs and phases Phase: a single map or reduce function application Map: phase applied to each item - filter, transform Wednesday, April 14, 2010
  • 102. Map-Reduce Terminology Job: a query, composed of inputs and phases Phase: a single map or reduce function application Map: phase applied to each item - filter, transform Reduce: phase applied to the collection - collate, aggregate, compute Wednesday, April 14, 2010
  • 103. Map Phases Wednesday, April 14, 2010
  • 104. Map Phases Inputs are bucket-key pairs (with optional key-specific data) Wednesday, April 14, 2010
  • 105. Map Phases Inputs are bucket-key pairs (with optional key-specific data) Must return a list Wednesday, April 14, 2010
  • 106. Map Phases Inputs are bucket-key pairs (with optional key-specific data) Must return a list Parallel results are aggregated Wednesday, April 14, 2010
  • 107. Map Built-ins Wednesday, April 14, 2010
  • 108. Map Built-ins Riak.mapValues Wednesday, April 14, 2010
  • 109. Map Built-ins Riak.mapValues Riak.mapValuesJson Wednesday, April 14, 2010
  • 110. Reduce Phases Wednesday, April 14, 2010
  • 111. Reduce Phases Performed on a single node Wednesday, April 14, 2010
  • 112. Reduce Phases Performed on a single node Two processes per reduce phase increase parallelism Wednesday, April 14, 2010
  • 113. Reduce Phases Performed on a single node Two processes per reduce phase increase parallelism Must return a list Wednesday, April 14, 2010
  • 114. Reduce Built-ins Wednesday, April 14, 2010
  • 115. Reduce Built-ins Riak.reduceMin Wednesday, April 14, 2010
  • 116. Reduce Built-ins Riak.reduceMin Riak.reduceMax Wednesday, April 14, 2010
  • 117. Reduce Built-ins Riak.reduceMin Riak.reduceMax Riak.reduceSort Wednesday, April 14, 2010
  • 118. Build a M/R Job Wednesday, April 14, 2010
  • 119. Build a M/R Job Specify inputs Wednesday, April 14, 2010
  • 120. Build a M/R Job Specify inputs bucket/key, bucket/key/key-specific-arg, bucket Wednesday, April 14, 2010
  • 121. Build a M/R Job Specify inputs bucket/key, bucket/key/key-specific-arg, bucket Specify phases Wednesday, April 14, 2010
  • 122. Build a M/R Job Specify inputs bucket/key, bucket/key/key-specific-arg, bucket Specify phases Each can receive a static argument Wednesday, April 14, 2010
  • 123. Build a M/R Job Specify inputs bucket/key, bucket/key/key-specific-arg, bucket Specify phases Each can receive a static argument Each can return intermediate results Wednesday, April 14, 2010
  • 124. Map-Reduce on HTTP Wednesday, April 14, 2010
  • 125. Map-Reduce on HTTP Job is a JSON object Wednesday, April 14, 2010
  • 126. Map-Reduce on HTTP Job is a JSON object POST /mapred Wednesday, April 14, 2010
  • 127. A simple M/R job {“inputs”: “goog”, “query”: [{“map”: {“language”:”javascript”, “name”: “Riak.mapValuesJson”, “keep”: true}]} Wednesday, April 14, 2010
  • 128. A simple M/R job {“inputs”: “goog”, “query”: [{“map”: {“language”:”javascript”, “name”: “Riak.mapValuesJson”, “keep”: true}]} Riak::MapReduce.new(c).add(‘goog’). map(‘Riak.mapValuesJson’, :keep => true).run Wednesday, April 14, 2010
  • 129. Another M/R Job {“inputs”: “stocks”, “query”: [{“map”:{“language”:”javascript”, “name”: “App.extractTickers”, “arg”: “GOOG”, “keep”: false}, {“reduce”:{“language”:”javascript, “name”: “Riak.reduceMin”, “keep”: true}]} Wednesday, April 14, 2010
  • 130. Another M/R Job {“inputs”: “stocks”, “query”: [{“map”:{“language”:”javascript”, “name”: “App.extractTickers”, “arg”: “GOOG”, “keep”: false}, {“reduce”:{“language”:”javascript, “name”: “Riak.reduceMin”, “keep”: true}]} Riak::MapReduce.new(c).add(‘stocks’). map(‘App.extractTickers’, :arg =>”GOOG”). reduce(“Riak.reduceMin”, :keep => true).run Wednesday, April 14, 2010
  • 131. Map-Reduce Demo Wednesday, April 14, 2010
  • 132. Ripple Wednesday, April 14, 2010
  • 133. Ripple Document-style persistence (like MongoMapper) Wednesday, April 14, 2010
  • 134. Ripple Document-style persistence (like MongoMapper) ActiveModel niceties Wednesday, April 14, 2010
  • 135. Ripple Document-style persistence (like MongoMapper) ActiveModel niceties Callbacks Wednesday, April 14, 2010
  • 136. Ripple Document-style persistence (like MongoMapper) ActiveModel niceties Callbacks Validations Wednesday, April 14, 2010
  • 137. Ripple Document-style persistence (like MongoMapper) ActiveModel niceties Callbacks Validations ActionPack compatibility Wednesday, April 14, 2010
  • 138. Ripple TODO Wednesday, April 14, 2010
  • 139. Ripple TODO Associations Wednesday, April 14, 2010
  • 140. Ripple TODO Associations Indexes Wednesday, April 14, 2010
  • 141. Ripple TODO Associations Indexes Use a property/method as the key Wednesday, April 14, 2010
  • 142. Ripple TODO Associations Indexes Use a property/method as the key Dynamic finders Wednesday, April 14, 2010
  • 143. Ripple TODO Associations Indexes Use a property/method as the key Dynamic finders Session/cache stores Wednesday, April 14, 2010
  • 144. Ripple Demo Wednesday, April 14, 2010
  • 145. sean@basho.com @seancribbs “seancribbs” on Freenode IRC #riak Questions? http://wiki.basho.com riak-users@lists.basho.com Wednesday, April 14, 2010