Your SlideShare is downloading. ×
Introducing Riak
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Introducing Riak

9,847
views

Published on


0 Comments
19 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
9,847
On Slideshare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
438
Comments
0
Likes
19
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide









































































































































































  • Transcript

    • 1. Introducing Riak Kevin A. Smith Senior Developer Basho Technologies
    • 2. What Is Riak?
    • 3. What Is Riak? • Key/Value store
    • 4. What Is Riak? • Key/Value store • Document-oriented database
    • 5. What Is Riak? • Key/Value store • Document-oriented database • Web-shaped storage
    • 6. Key/Value Store
    • 7. Key/Value Store • Data organized by bucket/key pairs
    • 8. Key/Value Store • Data organized by bucket/key pairs • Simple REST API (GET, PUT, DELETE)
    • 9. Document Store
    • 10. Document Store • Store values as JSON
    • 11. Document Store • Store values as JSON • Many clients support automatic JSON encoding/decoding
    • 12. Document Store • Store values as JSON • Many clients support automatic JSON encoding/decoding • Javascript Map/Reduce built on top of JSON
    • 13. Web-Shaped Storage
    • 14. Web-Shaped Storage • Content neutral
    • 15. Web-Shaped Storage • Content neutral • Highly distributed
    • 16. Web-Shaped Storage • Content neutral • Highly distributed • Replicated
    • 17. Web-Shaped Storage • Content neutral • Highly distributed • Replicated • Fault-tolerant
    • 18. What Is Riak?
    • 19. What Is Riak? A flexible storage engine...
    • 20. What Is Riak? A flexible storage engine... ...with a REST API...
    • 21. What Is Riak? A flexible storage engine... ...with a REST API... ...and map/reduce capability...
    • 22. What Is Riak? A flexible storage engine... ...with a REST API... ...and map/reduce capability... ....designed to be fault-tolerant...
    • 23. What Is Riak? A flexible storage engine... ...with a REST API... ...and map/reduce capability... ....designed to be fault-tolerant... ...distributed...
    • 24. What Is Riak? A flexible storage engine... ...with a REST API... ...and map/reduce capability... ....designed to be fault-tolerant... ...distributed... ...and ops friendly
    • 25. Influences
    • 26. Influences • CAP Theorem
    • 27. Influences • CAP Theorem • Amazon’s Dynamo Paper
    • 28. Influences • CAP Theorem • Amazon’s Dynamo Paper • Experience running large networks (Akamai)
    • 29. CAP Theorem
    • 30. CAP Theorem Consistent Reads and writes reflect a globally consistent system state
    • 31. CAP Theorem Consistent Reads and writes reflect a globally consistent system state
    • 32. CAP Theorem Consistent Reads and writes reflect a globally consistent system state Available System is available for reads and writes
    • 33. CAP Theorem Consistent Reads and writes reflect a globally consistent system state Available System is available for reads and writes
    • 34. CAP Theorem Consistent Reads and writes reflect a globally consistent system state Available System is available for reads and writes Partition Tolerant System can handle the failure of individual parts
    • 35. Common Wisdom
    • 36. Common Wisdom Pick two.
    • 37. The Riak Way
    • 38. The Riak Way Pick Two.
    • 39. The Riak Way Pick Two. For each operation.
    • 40. Dynamo Influences
    • 41. Dynamo Influences • N = The number of replicas
    • 42. Dynamo Influences • N = The number of replicas • R = The number of replicas needed for a successful read
    • 43. Dynamo Influences • N = The number of replicas • R = The number of replicas needed for a successful read • W = The number of replicas needed for a successful write
    • 44. Dynamo Math
    • 45. Dynamo Math N - R = read fault tolerance
    • 46. Dynamo Math N - R = read fault tolerance N - W = write fault tolerance
    • 47. Dynamo Math
    • 48. Dynamo Math N = 4, W = 2, R = 1
    • 49. Dynamo Math N = 4, W = 2, R = 1
    • 50. Dynamo Math N = 4, W = 2, R = 1 4 - 2 = 2 hosts can be down and Riak can still perform writes.
    • 51. Dynamo Math N = 4, W = 2, R = 1 4 - 2 = 2 hosts can be down and Riak can still perform writes. 4 - 1 = 3 hosts can be down and Riak can still perform reads.
    • 52. Riak Improvements
    • 53. Riak Improvements • N can vary per bucket
    • 54. Riak Improvements • N can vary per bucket • R and W can vary per operation
    • 55. Riak Improvements • N can vary per bucket • R and W can vary per operation Choose your own fault tolerance/performance tradeoff
    • 56. Consistent Hashing 2160 0 node 0 node 1 2160/4 node 2 node 3 hash(<<"artist">>,<<"REM">>) 2160/2
    • 57. R value get(<<"artist">>,<<"REM">>, R=2) (N=3) {ok, Object} X
    • 58. W value put(<<"artist">>,<<"REM">>, W=2) (N=3) ok X
    • 59. N=10, R/W=2 get/put("artist", "REM", R/W=2) (N=10) {ok, Object} X X X X X X X X
    • 60. Resolving Conflicts
    • 61. Resolving Conflicts • Riak focuses on the AP of CAP
    • 62. Resolving Conflicts • Riak focuses on the AP of CAP • Data could be briefly inconsistent
    • 63. Resolving Conflicts • Riak focuses on the AP of CAP • Data could be briefly inconsistent • Inconsistency must be resolved
    • 64. Detecting & Resolving Conflicts 0 1 Object v0 2 3
    • 65. Detecting & Resolving Conflicts Object 0 1 v0 Object 2 3 v0
    • 66. Detecting & Resolving Conflicts Object 0 1 v1 Object 2 3 v0
    • 67. Detecting & Resolving Conflicts 0 1 Object v1 2 3
    • 68. Detecting & Resolving Conflicts Object 0 1 v1 Object 2 3 v1
    • 69. Client Resolution
    • 70. Client Resolution • Can be set per-bucket or server-wide
    • 71. Client Resolution • Can be set per-bucket or server-wide • Conflicting data is “bubbled up” to the client
    • 72. Client Resolution • Can be set per-bucket or server-wide • Conflicting data is “bubbled up” to the client • Client picks the winner
    • 73. Server Resolution
    • 74. Server Resolution • “Last write wins”
    • 75. Server Resolution • “Last write wins” • Enabled by default
    • 76. Server Resolution • “Last write wins” • Enabled by default • What most apps need 80% of the time
    • 77. Live Demo!
    • 78. Linking Objects
    • 79. Linking Objects • Objects can store pointers, or links, to other objects
    • 80. Linking Objects • Objects can store pointers, or links, to other objects • Doesn’t have to be the same bucket
    • 81. Linking Objects • Objects can store pointers, or links, to other objects • Doesn’t have to be the same bucket • Object links described in a Link header
    • 82. Link Header Format Object URL </riak/demo/test1>; riaktag="userinfo" Link tag
    • 83. Link Walking
    • 84. Link Walking • Ask Riak to “walk” a sequence of links
    • 85. Link Walking • Ask Riak to “walk” a sequence of links • Optionally, collect objects along the walk and return them
    • 86. Link Walking • Ask Riak to “walk” a sequence of links • Optionally, collect objects along the walk and return them • Can be arbitrarily deep
    • 87. Link Walking Examples
    • 88. Link Walking Examples /riak/demo/test1/_,_,1
    • 89. Link Walking Examples /riak/demo/test1/_,_,1 Start walking at /demo/test1 and return all linked objects
    • 90. Link Walking Examples
    • 91. Link Walking Examples /riak/demo/test1/demo,_,1
    • 92. Link Walking Examples /riak/demo/test1/demo,_,1 Start walking at /demo/test1 and return all linked objects contained in the demo bucket
    • 93. Link Walking Examples
    • 94. Link Walking Examples /riak/demo/test1/_,_,0/_,_,1
    • 95. Link Walking Examples /riak/demo/test1/_,_,0/_,_,1 Start walking at /demo/test1, find any linked objects, then find and return any objects linked to those
    • 96. Link Walking Examples
    • 97. Link Walking Examples /riak/demo/test1/_,child,0/_,_,1
    • 98. Link Walking Examples /riak/demo/test1/_,child,0/_,_,1 Start walking at /demo/test1, find any linked objects with the link tag “child”, then find and return any objects linked to those
    • 99. Map/Reduce Terms
    • 100. Map/Reduce Terms • Phase: A step within a job
    • 101. Map/Reduce Terms • Phase: A step within a job • Job: A sequence of phases and inputs
    • 102. Map/Reduce Terms • Phase: A step within a job • Job: A sequence of phases and inputs • Map: Data collection phase
    • 103. Map/Reduce Terms • Phase: A step within a job • Job: A sequence of phases and inputs • Map: Data collection phase • Reduce: Data collation or processing phase
    • 104. Map/Reduce Overview
    • 105. Map/Reduce Overview • Map phases execute in parallel w/data locality
    • 106. Map/Reduce Overview • Map phases execute in parallel w/data locality • Reduce phases execute in parallel on the node where job was submitted
    • 107. Map/Reduce Overview • Map phases execute in parallel w/data locality • Reduce phases execute in parallel on the node where job was submitted • Results are not cached or stored
    • 108. Map/Reduce Overview • Map phases execute in parallel w/data locality • Reduce phases execute in parallel on the node where job was submitted • Results are not cached or stored • Phases can be written in Erlang or Javascript
    • 109. Map Phase
    • 110. Map Phase • Inputs must be bucket/key pairs
    • 111. Map Phase • Inputs must be bucket/key pairs • Must return a list
    • 112. Map Phase • Inputs must be bucket/key pairs • Must return a list • Parallel results are aggregated into a single list
    • 113. Parallel Map
    • 114. Parallel Map
    • 115. Parallel Map
    • 116. Parallel Map
    • 117. Erlang Map Phase
    • 118. Erlang Map Phase • Two types: modfun and qfun
    • 119. Erlang Map Phase • Two types: modfun and qfun • modfuns reference the module and name of the Erlang function to call
    • 120. Erlang Map Phase • Two types: modfun and qfun • modfuns reference the module and name of the Erlang function to call • qfuns are anonymous Erlang functions*
    • 121. Erlang Map Phase • Two types: modfun and qfun • modfuns reference the module and name of the Erlang function to call • qfuns are anonymous Erlang functions* *Must be on the server-side codepath
    • 122. Erlang Map Phase
    • 123. Erlang Map Phase map_object_value(Obj, _KeyData, _Arg) ->
    • 124. Erlang Map Phase map_object_value(Obj, _KeyData, _Arg) -> [riak_object:get_value(Obj)].
    • 125. Erlang Map Phase map_object_value(Obj, _KeyData, _Arg) -> [riak_object:get_value(Obj)].
    • 126. Erlang Map Phase map_object_value(Obj, _KeyData, _Arg) -> [riak_object:get_value(Obj)]. • Obj:riak_object retrieved from bucket/key
    • 127. Erlang Map Phase map_object_value(Obj, _KeyData, _Arg) -> [riak_object:get_value(Obj)]. • Obj:riak_object retrieved from bucket/key • KeyData: Static argument specified with the bucket/ key
    • 128. Erlang Map Phase map_object_value(Obj, _KeyData, _Arg) -> [riak_object:get_value(Obj)]. • Obj:riak_object retrieved from bucket/key • KeyData: Static argument specified with the bucket/ key • Arg: Static argument specified with the phase
    • 129. Erlang Map Built-Ins
    • 130. Erlang Map Built-Ins riak_mapreduce:map_object_value/3
    • 131. Erlang Map Built-Ins riak_mapreduce:map_object_value/3 • Returns object value wrapped in a list
    • 132. Erlang Map Built-Ins riak_mapreduce:map_object_value/3 • Returns object value wrapped in a list riak_mapreduce:map_object_value_list/3
    • 133. Erlang Map Built-Ins riak_mapreduce:map_object_value/3 • Returns object value wrapped in a list riak_mapreduce:map_object_value_list/3 • Returns object value. Object value must already be a list
    • 134. Javascript Map Phase
    • 135. Javascript Map Phase • Two types: jsanon and jsfun
    • 136. Javascript Map Phase • Two types: jsanon and jsfun • jsanons are anonymous JS functions:
    • 137. Javascript Map Phase • Two types: jsanon and jsfun • jsanons are anonymous JS functions: function(value) { return [value]; }
    • 138. Javascript Map Phase • Two types: jsanon and jsfun • jsanons are anonymous JS functions: function(value) { return [value]; } • jsfuns are named JS functions:
    • 139. Javascript Map Phase • Two types: jsanon and jsfun • jsanons are anonymous JS functions: function(value) { return [value]; } • jsfuns are named JS functions: Riak.mapValuesJson
    • 140. Erlang & Javascript
    • 141. Erlang & Javascript • Same environment as Firefox minus browser bits
    • 142. Erlang & Javascript • Same environment as Firefox minus browser bits • Erlang to Javascript data is JSON encoded
    • 143. Erlang & Javascript • Same environment as Firefox minus browser bits • Erlang to Javascript data is JSON encoded • Javascript to Erlang data is JSON decoded
    • 144. Javascript Map Phase
    • 145. Javascript Map Phase function(value, keyData, arg)
    • 146. Javascript Map Phase function(value, keyData, arg)
    • 147. Javascript Map Phase function(value, keyData, arg) • value: JSON-encoded version of riak_object
    • 148. Javascript Map Phase function(value, keyData, arg) • value: JSON-encoded version of riak_object • keyData: Same as Erlang
    • 149. Javascript Map Phase function(value, keyData, arg) • value: JSON-encoded version of riak_object • keyData: Same as Erlang • arg: Same as Erlang
    • 150. Javascript Map Built-Ins
    • 151. Javascript Map Built-Ins Riak.mapValues
    • 152. Javascript Map Built-Ins Riak.mapValues • Returns object values. Handles detecting when/if to use list wrapping.
    • 153. Javascript Map Built-Ins Riak.mapValues • Returns object values. Handles detecting when/if to use list wrapping. Riak.mapValuesJson
    • 154. Javascript Map Built-Ins Riak.mapValues • Returns object values. Handles detecting when/if to use list wrapping. Riak.mapValuesJson • Returns JSON parsed object values. Also performs list wrapping, if needed.
    • 155. Reduce Phase
    • 156. Reduce Phase • Performed on the node coordinating the map/reduce job
    • 157. Reduce Phase • Performed on the node coordinating the map/reduce job • Two processes per reduce phase to add minor parallelism
    • 158. Reduce Phase • Performed on the node coordinating the map/reduce job • Two processes per reduce phase to add minor parallelism • Must return a list
    • 159. Erlang Reduce Built-Ins
    • 160. Erlang Reduce Built-Ins riak_mapreduce:reduce_set_union/2
    • 161. Erlang Reduce Built-Ins riak_mapreduce:reduce_set_union/2 • Returns unique set of values
    • 162. Erlang Reduce Built-Ins riak_mapreduce:reduce_set_union/2 • Returns unique set of values riak_mapreduce:reduce_sum/2
    • 163. Erlang Reduce Built-Ins riak_mapreduce:reduce_set_union/2 • Returns unique set of values riak_mapreduce:reduce_sum/2 • Returns the sum of inputs
    • 164. Erlang Reduce Built-Ins riak_mapreduce:reduce_set_union/2 • Returns unique set of values riak_mapreduce:reduce_sum/2 • Returns the sum of inputs riak_mapreduce:reduce_sort/2
    • 165. Erlang Reduce Built-Ins riak_mapreduce:reduce_set_union/2 • Returns unique set of values riak_mapreduce:reduce_sum/2 • Returns the sum of inputs riak_mapreduce:reduce_sort/2 • Returns the sorted list of inputs
    • 166. Javascript Reduce Built-Ins
    • 167. Javascript Reduce Built-Ins Riak.reduceMin
    • 168. Javascript Reduce Built-Ins Riak.reduceMin • Returns the minimum value of the input set
    • 169. Javascript Reduce Built-Ins Riak.reduceMin • Returns the minimum value of the input set Riak.reduceMax
    • 170. Javascript Reduce Built-Ins Riak.reduceMin • Returns the minimum value of the input set Riak.reduceMax • Returns the maximum value of the input set
    • 171. Javascript Reduce Built-Ins Riak.reduceMin • Returns the minimum value of the input set Riak.reduceMax • Returns the maximum value of the input set Riak.reduceSort
    • 172. Javascript Reduce Built-Ins Riak.reduceMin • Returns the minimum value of the input set Riak.reduceMax • Returns the maximum value of the input set Riak.reduceSort • Returns a sorted list of the input set
    • 173. Building M/R Job
    • 174. Building M/R Job • Job is a list of phases and starting inputs
    • 175. Building M/R Job • Job is a list of phases and starting inputs • Each phase can:
    • 176. Building M/R Job • Job is a list of phases and starting inputs • Each phase can: • Receive a static argument
    • 177. Building M/R Job • Job is a list of phases and starting inputs • Each phase can: • Receive a static argument • Accumulate and return results
    • 178. Submitting Jobs via HTTP
    • 179. Submitting Jobs via HTTP • Riak exposes M/R via its REST API
    • 180. Submitting Jobs via HTTP • Riak exposes M/R via its REST API • Job is described in JSON
    • 181. Submitting Jobs via HTTP • Riak exposes M/R via its REST API • Job is described in JSON • Submitted via POST
    • 182. Submitting Jobs via HTTP • Riak exposes M/R via its REST API • Job is described in JSON • Submitted via POST • Default URL is /mapred
    • 183. Erlang Phase (JSON)
    • 184. Erlang Phase (JSON) {Type:{“language”:”erlang”, “module”: Module,
    • 185. Erlang Phase (JSON) {Type:{“language”:”erlang”, “module”: Module, “function”: Function, “keep”:Flag}}
    • 186. Erlang Phase (JSON) {Type:{“language”:”erlang”, “module”: Module, “function”: Function, “keep”:Flag}}
    • 187. Erlang Phase (JSON) {Type:{“language”:”erlang”, “module”: Module, “function”: Function, “keep”:Flag}} • Type: “map” or “reduce”
    • 188. Erlang Phase (JSON) {Type:{“language”:”erlang”, “module”: Module, “function”: Function, “keep”:Flag}} • Type: “map” or “reduce” • Module: String name of Erlang module
    • 189. Erlang Phase (JSON) {Type:{“language”:”erlang”, “module”: Module, “function”: Function, “keep”:Flag}} • Type: “map” or “reduce” • Module: String name of Erlang module • Function: String name of Erlang function
    • 190. Erlang Phase (JSON) {Type:{“language”:”erlang”, “module”: Module, “function”: Function, “keep”:Flag}} • Type: “map” or “reduce” • Module: String name of Erlang module • Function: String name of Erlang function • Flag: Boolean accumulation toggle
    • 191. Javascript Phase (JSON)
    • 192. Javascript Phase (JSON) {Type:{“language”:”javascript”,
    • 193. Javascript Phase (JSON) {Type:{“language”:”javascript”, “source”: Source,“keep”:Flag}}
    • 194. Javascript Phase (JSON) {Type:{“language”:”javascript”, “source”: Source,“keep”:Flag}}
    • 195. Javascript Phase (JSON) {Type:{“language”:”javascript”, “source”: Source,“keep”:Flag}} • Type: “map” or “reduce”
    • 196. Javascript Phase (JSON) {Type:{“language”:”javascript”, “source”: Source,“keep”:Flag}} • Type: “map” or “reduce” • Source: Source for anonymous function
    • 197. Javascript Phase (JSON) {Type:{“language”:”javascript”, “source”: Source,“keep”:Flag}} • Type: “map” or “reduce” • Source: Source for anonymous function • Flag: Boolean accumulation toggle
    • 198. Javascript Phase (JSON)
    • 199. Javascript Phase (JSON) {Type:{“language”:”javascript”,
    • 200. Javascript Phase (JSON) {Type:{“language”:”javascript”, “name”:Name,“keep”:Flag}}
    • 201. Javascript Phase (JSON) {Type:{“language”:”javascript”, “name”:Name,“keep”:Flag}}
    • 202. Javascript Phase (JSON) {Type:{“language”:”javascript”, “name”:Name,“keep”:Flag}} • Type: “map” or “reduce”
    • 203. Javascript Phase (JSON) {Type:{“language”:”javascript”, “name”:Name,“keep”:Flag}} • Type: “map” or “reduce” • Name: String name of Javascript function
    • 204. Javascript Phase (JSON) {Type:{“language”:”javascript”, “name”:Name,“keep”:Flag}} • Type: “map” or “reduce” • Name: String name of Javascript function • Flag: Boolean accumulation toggle
    • 205. Putting It Together {“inputs”: [[“stocks”, “goog”]], “query”: [{“map”:{“language”:”javascript”, “name”: “Riak.mapValuesJson”}, “keep”: true}]}
    • 206. Putting It Together {“inputs”: [[“stocks”, “goog”], [“stocks”, “csco”]], “query”: [{“map”:{“language”:”javascript”, “name”: “Riak.mapValuesJson”}, “keep”: true}]}
    • 207. Putting It Together {“inputs”: “stocks”, “query”: [{“map”:{“language”:”javascript”, “name”: “App.extractTickers”, “arg”: “GOOG”}, “keep”: false}, {“reduce”:{“language”:”javascript, “name”: “Riak.reduceMin”}, “keep”: true}]}
    • 208. Live Demo!
    • 209. Thank You Kevin A. Smith Email: ksmith@basho.com Twitter: @kevsmith