Introducing Riak

11,778 views

Published on

Introducing Riak

  1. 1. Introducing Riak Kevin A. Smith Senior Developer Basho Technologies
  2. 2. What Is Riak?
  3. 3. What Is Riak? • Key/Value store
  4. 4. What Is Riak? • Key/Value store • Document-oriented database
  5. 5. What Is Riak? • Key/Value store • Document-oriented database • Web-shaped storage
  6. 6. Key/Value Store
  7. 7. Key/Value Store • Data organized by bucket/key pairs
  8. 8. Key/Value Store • Data organized by bucket/key pairs • Simple REST API (GET, PUT, DELETE)
  9. 9. Document Store
  10. 10. Document Store • Store values as JSON
  11. 11. Document Store • Store values as JSON • Many clients support automatic JSON encoding/decoding
  12. 12. Document Store • Store values as JSON • Many clients support automatic JSON encoding/decoding • Javascript Map/Reduce built on top of JSON
  13. 13. Web-Shaped Storage
  14. 14. Web-Shaped Storage • Content neutral
  15. 15. Web-Shaped Storage • Content neutral • Highly distributed
  16. 16. Web-Shaped Storage • Content neutral • Highly distributed • Replicated
  17. 17. Web-Shaped Storage • Content neutral • Highly distributed • Replicated • Fault-tolerant
  18. 18. What Is Riak?
  19. 19. What Is Riak? A flexible storage engine...
  20. 20. What Is Riak? A flexible storage engine... ...with a REST API...
  21. 21. What Is Riak? A flexible storage engine... ...with a REST API... ...and map/reduce capability...
  22. 22. What Is Riak? A flexible storage engine... ...with a REST API... ...and map/reduce capability... ....designed to be fault-tolerant...
  23. 23. What Is Riak? A flexible storage engine... ...with a REST API... ...and map/reduce capability... ....designed to be fault-tolerant... ...distributed...
  24. 24. What Is Riak? A flexible storage engine... ...with a REST API... ...and map/reduce capability... ....designed to be fault-tolerant... ...distributed... ...and ops friendly
  25. 25. Influences
  26. 26. Influences • CAP Theorem
  27. 27. Influences • CAP Theorem • Amazon’s Dynamo Paper
  28. 28. Influences • CAP Theorem • Amazon’s Dynamo Paper • Experience running large networks (Akamai)
  29. 29. CAP Theorem
  30. 30. CAP Theorem Consistent Reads and writes reflect a globally consistent system state
  31. 31. CAP Theorem Consistent Reads and writes reflect a globally consistent system state
  32. 32. CAP Theorem Consistent Reads and writes reflect a globally consistent system state Available System is available for reads and writes
  33. 33. CAP Theorem Consistent Reads and writes reflect a globally consistent system state Available System is available for reads and writes
  34. 34. CAP Theorem Consistent Reads and writes reflect a globally consistent system state Available System is available for reads and writes Partition Tolerant System can handle the failure of individual parts
  35. 35. Common Wisdom
  36. 36. Common Wisdom Pick two.
  37. 37. The Riak Way
  38. 38. The Riak Way Pick Two.
  39. 39. The Riak Way Pick Two. For each operation.
  40. 40. Dynamo Influences
  41. 41. Dynamo Influences • N = The number of replicas
  42. 42. Dynamo Influences • N = The number of replicas • R = The number of replicas needed for a successful read
  43. 43. Dynamo Influences • N = The number of replicas • R = The number of replicas needed for a successful read • W = The number of replicas needed for a successful write
  44. 44. Dynamo Math
  45. 45. Dynamo Math N - R = read fault tolerance
  46. 46. Dynamo Math N - R = read fault tolerance N - W = write fault tolerance
  47. 47. Dynamo Math
  48. 48. Dynamo Math N = 4, W = 2, R = 1
  49. 49. Dynamo Math N = 4, W = 2, R = 1
  50. 50. Dynamo Math N = 4, W = 2, R = 1 4 - 2 = 2 hosts can be down and Riak can still perform writes.
  51. 51. Dynamo Math N = 4, W = 2, R = 1 4 - 2 = 2 hosts can be down and Riak can still perform writes. 4 - 1 = 3 hosts can be down and Riak can still perform reads.
  52. 52. Riak Improvements
  53. 53. Riak Improvements • N can vary per bucket
  54. 54. Riak Improvements • N can vary per bucket • R and W can vary per operation
  55. 55. Riak Improvements • N can vary per bucket • R and W can vary per operation Choose your own fault tolerance/performance tradeoff
  56. 56. Consistent Hashing 2160 0 node 0 node 1 2160/4 node 2 node 3 hash(<<"artist">>,<<"REM">>) 2160/2
  57. 57. R value get(<<"artist">>,<<"REM">>, R=2) (N=3) {ok, Object} X
  58. 58. W value put(<<"artist">>,<<"REM">>, W=2) (N=3) ok X
  59. 59. N=10, R/W=2 get/put("artist", "REM", R/W=2) (N=10) {ok, Object} X X X X X X X X
  60. 60. Resolving Conflicts
  61. 61. Resolving Conflicts • Riak focuses on the AP of CAP
  62. 62. Resolving Conflicts • Riak focuses on the AP of CAP • Data could be briefly inconsistent
  63. 63. Resolving Conflicts • Riak focuses on the AP of CAP • Data could be briefly inconsistent • Inconsistency must be resolved
  64. 64. Detecting & Resolving Conflicts 0 1 Object v0 2 3
  65. 65. Detecting & Resolving Conflicts Object 0 1 v0 Object 2 3 v0
  66. 66. Detecting & Resolving Conflicts Object 0 1 v1 Object 2 3 v0
  67. 67. Detecting & Resolving Conflicts 0 1 Object v1 2 3
  68. 68. Detecting & Resolving Conflicts Object 0 1 v1 Object 2 3 v1
  69. 69. Client Resolution
  70. 70. Client Resolution • Can be set per-bucket or server-wide
  71. 71. Client Resolution • Can be set per-bucket or server-wide • Conflicting data is “bubbled up” to the client
  72. 72. Client Resolution • Can be set per-bucket or server-wide • Conflicting data is “bubbled up” to the client • Client picks the winner
  73. 73. Server Resolution
  74. 74. Server Resolution • “Last write wins”
  75. 75. Server Resolution • “Last write wins” • Enabled by default
  76. 76. Server Resolution • “Last write wins” • Enabled by default • What most apps need 80% of the time
  77. 77. Live Demo!
  78. 78. Linking Objects
  79. 79. Linking Objects • Objects can store pointers, or links, to other objects
  80. 80. Linking Objects • Objects can store pointers, or links, to other objects • Doesn’t have to be the same bucket
  81. 81. Linking Objects • Objects can store pointers, or links, to other objects • Doesn’t have to be the same bucket • Object links described in a Link header
  82. 82. Link Header Format Object URL </riak/demo/test1>; riaktag="userinfo" Link tag
  83. 83. Link Walking
  84. 84. Link Walking • Ask Riak to “walk” a sequence of links
  85. 85. Link Walking • Ask Riak to “walk” a sequence of links • Optionally, collect objects along the walk and return them
  86. 86. Link Walking • Ask Riak to “walk” a sequence of links • Optionally, collect objects along the walk and return them • Can be arbitrarily deep
  87. 87. Link Walking Examples
  88. 88. Link Walking Examples /riak/demo/test1/_,_,1
  89. 89. Link Walking Examples /riak/demo/test1/_,_,1 Start walking at /demo/test1 and return all linked objects
  90. 90. Link Walking Examples
  91. 91. Link Walking Examples /riak/demo/test1/demo,_,1
  92. 92. Link Walking Examples /riak/demo/test1/demo,_,1 Start walking at /demo/test1 and return all linked objects contained in the demo bucket
  93. 93. Link Walking Examples
  94. 94. Link Walking Examples /riak/demo/test1/_,_,0/_,_,1
  95. 95. Link Walking Examples /riak/demo/test1/_,_,0/_,_,1 Start walking at /demo/test1, find any linked objects, then find and return any objects linked to those
  96. 96. Link Walking Examples
  97. 97. Link Walking Examples /riak/demo/test1/_,child,0/_,_,1
  98. 98. Link Walking Examples /riak/demo/test1/_,child,0/_,_,1 Start walking at /demo/test1, find any linked objects with the link tag “child”, then find and return any objects linked to those
  99. 99. Map/Reduce Terms
  100. 100. Map/Reduce Terms • Phase: A step within a job
  101. 101. Map/Reduce Terms • Phase: A step within a job • Job: A sequence of phases and inputs
  102. 102. Map/Reduce Terms • Phase: A step within a job • Job: A sequence of phases and inputs • Map: Data collection phase
  103. 103. Map/Reduce Terms • Phase: A step within a job • Job: A sequence of phases and inputs • Map: Data collection phase • Reduce: Data collation or processing phase
  104. 104. Map/Reduce Overview
  105. 105. Map/Reduce Overview • Map phases execute in parallel w/data locality
  106. 106. Map/Reduce Overview • Map phases execute in parallel w/data locality • Reduce phases execute in parallel on the node where job was submitted
  107. 107. Map/Reduce Overview • Map phases execute in parallel w/data locality • Reduce phases execute in parallel on the node where job was submitted • Results are not cached or stored
  108. 108. Map/Reduce Overview • Map phases execute in parallel w/data locality • Reduce phases execute in parallel on the node where job was submitted • Results are not cached or stored • Phases can be written in Erlang or Javascript
  109. 109. Map Phase
  110. 110. Map Phase • Inputs must be bucket/key pairs
  111. 111. Map Phase • Inputs must be bucket/key pairs • Must return a list
  112. 112. Map Phase • Inputs must be bucket/key pairs • Must return a list • Parallel results are aggregated into a single list
  113. 113. Parallel Map
  114. 114. Parallel Map
  115. 115. Parallel Map
  116. 116. Parallel Map
  117. 117. Erlang Map Phase
  118. 118. Erlang Map Phase • Two types: modfun and qfun
  119. 119. Erlang Map Phase • Two types: modfun and qfun • modfuns reference the module and name of the Erlang function to call
  120. 120. Erlang Map Phase • Two types: modfun and qfun • modfuns reference the module and name of the Erlang function to call • qfuns are anonymous Erlang functions*
  121. 121. Erlang Map Phase • Two types: modfun and qfun • modfuns reference the module and name of the Erlang function to call • qfuns are anonymous Erlang functions* *Must be on the server-side codepath
  122. 122. Erlang Map Phase
  123. 123. Erlang Map Phase map_object_value(Obj, _KeyData, _Arg) ->
  124. 124. Erlang Map Phase map_object_value(Obj, _KeyData, _Arg) -> [riak_object:get_value(Obj)].
  125. 125. Erlang Map Phase map_object_value(Obj, _KeyData, _Arg) -> [riak_object:get_value(Obj)].
  126. 126. Erlang Map Phase map_object_value(Obj, _KeyData, _Arg) -> [riak_object:get_value(Obj)]. • Obj:riak_object retrieved from bucket/key
  127. 127. Erlang Map Phase map_object_value(Obj, _KeyData, _Arg) -> [riak_object:get_value(Obj)]. • Obj:riak_object retrieved from bucket/key • KeyData: Static argument specified with the bucket/ key
  128. 128. Erlang Map Phase map_object_value(Obj, _KeyData, _Arg) -> [riak_object:get_value(Obj)]. • Obj:riak_object retrieved from bucket/key • KeyData: Static argument specified with the bucket/ key • Arg: Static argument specified with the phase
  129. 129. Erlang Map Built-Ins
  130. 130. Erlang Map Built-Ins riak_mapreduce:map_object_value/3
  131. 131. Erlang Map Built-Ins riak_mapreduce:map_object_value/3 • Returns object value wrapped in a list
  132. 132. Erlang Map Built-Ins riak_mapreduce:map_object_value/3 • Returns object value wrapped in a list riak_mapreduce:map_object_value_list/3
  133. 133. Erlang Map Built-Ins riak_mapreduce:map_object_value/3 • Returns object value wrapped in a list riak_mapreduce:map_object_value_list/3 • Returns object value. Object value must already be a list
  134. 134. Javascript Map Phase
  135. 135. Javascript Map Phase • Two types: jsanon and jsfun
  136. 136. Javascript Map Phase • Two types: jsanon and jsfun • jsanons are anonymous JS functions:
  137. 137. Javascript Map Phase • Two types: jsanon and jsfun • jsanons are anonymous JS functions: function(value) { return [value]; }
  138. 138. Javascript Map Phase • Two types: jsanon and jsfun • jsanons are anonymous JS functions: function(value) { return [value]; } • jsfuns are named JS functions:
  139. 139. Javascript Map Phase • Two types: jsanon and jsfun • jsanons are anonymous JS functions: function(value) { return [value]; } • jsfuns are named JS functions: Riak.mapValuesJson
  140. 140. Erlang & Javascript
  141. 141. Erlang & Javascript • Same environment as Firefox minus browser bits
  142. 142. Erlang & Javascript • Same environment as Firefox minus browser bits • Erlang to Javascript data is JSON encoded
  143. 143. Erlang & Javascript • Same environment as Firefox minus browser bits • Erlang to Javascript data is JSON encoded • Javascript to Erlang data is JSON decoded
  144. 144. Javascript Map Phase
  145. 145. Javascript Map Phase function(value, keyData, arg)
  146. 146. Javascript Map Phase function(value, keyData, arg)
  147. 147. Javascript Map Phase function(value, keyData, arg) • value: JSON-encoded version of riak_object
  148. 148. Javascript Map Phase function(value, keyData, arg) • value: JSON-encoded version of riak_object • keyData: Same as Erlang
  149. 149. Javascript Map Phase function(value, keyData, arg) • value: JSON-encoded version of riak_object • keyData: Same as Erlang • arg: Same as Erlang
  150. 150. Javascript Map Built-Ins
  151. 151. Javascript Map Built-Ins Riak.mapValues
  152. 152. Javascript Map Built-Ins Riak.mapValues • Returns object values. Handles detecting when/if to use list wrapping.
  153. 153. Javascript Map Built-Ins Riak.mapValues • Returns object values. Handles detecting when/if to use list wrapping. Riak.mapValuesJson
  154. 154. Javascript Map Built-Ins Riak.mapValues • Returns object values. Handles detecting when/if to use list wrapping. Riak.mapValuesJson • Returns JSON parsed object values. Also performs list wrapping, if needed.
  155. 155. Reduce Phase
  156. 156. Reduce Phase • Performed on the node coordinating the map/reduce job
  157. 157. Reduce Phase • Performed on the node coordinating the map/reduce job • Two processes per reduce phase to add minor parallelism
  158. 158. Reduce Phase • Performed on the node coordinating the map/reduce job • Two processes per reduce phase to add minor parallelism • Must return a list
  159. 159. Erlang Reduce Built-Ins
  160. 160. Erlang Reduce Built-Ins riak_mapreduce:reduce_set_union/2
  161. 161. Erlang Reduce Built-Ins riak_mapreduce:reduce_set_union/2 • Returns unique set of values
  162. 162. Erlang Reduce Built-Ins riak_mapreduce:reduce_set_union/2 • Returns unique set of values riak_mapreduce:reduce_sum/2
  163. 163. Erlang Reduce Built-Ins riak_mapreduce:reduce_set_union/2 • Returns unique set of values riak_mapreduce:reduce_sum/2 • Returns the sum of inputs
  164. 164. Erlang Reduce Built-Ins riak_mapreduce:reduce_set_union/2 • Returns unique set of values riak_mapreduce:reduce_sum/2 • Returns the sum of inputs riak_mapreduce:reduce_sort/2
  165. 165. Erlang Reduce Built-Ins riak_mapreduce:reduce_set_union/2 • Returns unique set of values riak_mapreduce:reduce_sum/2 • Returns the sum of inputs riak_mapreduce:reduce_sort/2 • Returns the sorted list of inputs
  166. 166. Javascript Reduce Built-Ins
  167. 167. Javascript Reduce Built-Ins Riak.reduceMin
  168. 168. Javascript Reduce Built-Ins Riak.reduceMin • Returns the minimum value of the input set
  169. 169. Javascript Reduce Built-Ins Riak.reduceMin • Returns the minimum value of the input set Riak.reduceMax
  170. 170. Javascript Reduce Built-Ins Riak.reduceMin • Returns the minimum value of the input set Riak.reduceMax • Returns the maximum value of the input set
  171. 171. Javascript Reduce Built-Ins Riak.reduceMin • Returns the minimum value of the input set Riak.reduceMax • Returns the maximum value of the input set Riak.reduceSort
  172. 172. Javascript Reduce Built-Ins Riak.reduceMin • Returns the minimum value of the input set Riak.reduceMax • Returns the maximum value of the input set Riak.reduceSort • Returns a sorted list of the input set
  173. 173. Building M/R Job
  174. 174. Building M/R Job • Job is a list of phases and starting inputs
  175. 175. Building M/R Job • Job is a list of phases and starting inputs • Each phase can:
  176. 176. Building M/R Job • Job is a list of phases and starting inputs • Each phase can: • Receive a static argument
  177. 177. Building M/R Job • Job is a list of phases and starting inputs • Each phase can: • Receive a static argument • Accumulate and return results
  178. 178. Submitting Jobs via HTTP
  179. 179. Submitting Jobs via HTTP • Riak exposes M/R via its REST API
  180. 180. Submitting Jobs via HTTP • Riak exposes M/R via its REST API • Job is described in JSON
  181. 181. Submitting Jobs via HTTP • Riak exposes M/R via its REST API • Job is described in JSON • Submitted via POST
  182. 182. Submitting Jobs via HTTP • Riak exposes M/R via its REST API • Job is described in JSON • Submitted via POST • Default URL is /mapred
  183. 183. Erlang Phase (JSON)
  184. 184. Erlang Phase (JSON) {Type:{“language”:”erlang”, “module”: Module,
  185. 185. Erlang Phase (JSON) {Type:{“language”:”erlang”, “module”: Module, “function”: Function, “keep”:Flag}}
  186. 186. Erlang Phase (JSON) {Type:{“language”:”erlang”, “module”: Module, “function”: Function, “keep”:Flag}}
  187. 187. Erlang Phase (JSON) {Type:{“language”:”erlang”, “module”: Module, “function”: Function, “keep”:Flag}} • Type: “map” or “reduce”
  188. 188. Erlang Phase (JSON) {Type:{“language”:”erlang”, “module”: Module, “function”: Function, “keep”:Flag}} • Type: “map” or “reduce” • Module: String name of Erlang module
  189. 189. Erlang Phase (JSON) {Type:{“language”:”erlang”, “module”: Module, “function”: Function, “keep”:Flag}} • Type: “map” or “reduce” • Module: String name of Erlang module • Function: String name of Erlang function
  190. 190. Erlang Phase (JSON) {Type:{“language”:”erlang”, “module”: Module, “function”: Function, “keep”:Flag}} • Type: “map” or “reduce” • Module: String name of Erlang module • Function: String name of Erlang function • Flag: Boolean accumulation toggle
  191. 191. Javascript Phase (JSON)
  192. 192. Javascript Phase (JSON) {Type:{“language”:”javascript”,
  193. 193. Javascript Phase (JSON) {Type:{“language”:”javascript”, “source”: Source,“keep”:Flag}}
  194. 194. Javascript Phase (JSON) {Type:{“language”:”javascript”, “source”: Source,“keep”:Flag}}
  195. 195. Javascript Phase (JSON) {Type:{“language”:”javascript”, “source”: Source,“keep”:Flag}} • Type: “map” or “reduce”
  196. 196. Javascript Phase (JSON) {Type:{“language”:”javascript”, “source”: Source,“keep”:Flag}} • Type: “map” or “reduce” • Source: Source for anonymous function
  197. 197. Javascript Phase (JSON) {Type:{“language”:”javascript”, “source”: Source,“keep”:Flag}} • Type: “map” or “reduce” • Source: Source for anonymous function • Flag: Boolean accumulation toggle
  198. 198. Javascript Phase (JSON)
  199. 199. Javascript Phase (JSON) {Type:{“language”:”javascript”,
  200. 200. Javascript Phase (JSON) {Type:{“language”:”javascript”, “name”:Name,“keep”:Flag}}
  201. 201. Javascript Phase (JSON) {Type:{“language”:”javascript”, “name”:Name,“keep”:Flag}}
  202. 202. Javascript Phase (JSON) {Type:{“language”:”javascript”, “name”:Name,“keep”:Flag}} • Type: “map” or “reduce”
  203. 203. Javascript Phase (JSON) {Type:{“language”:”javascript”, “name”:Name,“keep”:Flag}} • Type: “map” or “reduce” • Name: String name of Javascript function
  204. 204. Javascript Phase (JSON) {Type:{“language”:”javascript”, “name”:Name,“keep”:Flag}} • Type: “map” or “reduce” • Name: String name of Javascript function • Flag: Boolean accumulation toggle
  205. 205. Putting It Together {“inputs”: [[“stocks”, “goog”]], “query”: [{“map”:{“language”:”javascript”, “name”: “Riak.mapValuesJson”}, “keep”: true}]}
  206. 206. Putting It Together {“inputs”: [[“stocks”, “goog”], [“stocks”, “csco”]], “query”: [{“map”:{“language”:”javascript”, “name”: “Riak.mapValuesJson”}, “keep”: true}]}
  207. 207. Putting It Together {“inputs”: “stocks”, “query”: [{“map”:{“language”:”javascript”, “name”: “App.extractTickers”, “arg”: “GOOG”}, “keep”: false}, {“reduce”:{“language”:”javascript, “name”: “Riak.reduceMin”}, “keep”: true}]}
  208. 208. Live Demo!
  209. 209. Thank You Kevin A. Smith Email: ksmith@basho.com Twitter: @kevsmith

×