• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Introducing Riak
 

Introducing Riak

on

  • 1,765 views

 

Statistics

Views

Total Views
1,765
Views on SlideShare
1,620
Embed Views
145

Actions

Likes
2
Downloads
65
Comments
0

1 Embed 145

http://www.scoop.it 145

Accessibility

Categories

Upload Details

Uploaded via as Apple Keynote

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />

Introducing Riak Introducing Riak Presentation Transcript

  • Introducing Riak Kevin A. Smith Senior Developer Basho Technologies
  • What Is Riak?
  • What Is Riak? • Key/Value store
  • What Is Riak? • Key/Value store • Document-oriented database
  • What Is Riak? • Key/Value store • Document-oriented database • Web-shaped storage
  • Key/Value Store
  • Key/Value Store • Data organized by bucket/key pairs
  • Key/Value Store • Data organized by bucket/key pairs • Simple REST API (GET, PUT, DELETE)
  • Document Store
  • Document Store • Store values as JSON
  • Document Store • Store values as JSON • Many clients support automatic JSON encoding/decoding
  • Document Store • Store values as JSON • Many clients support automatic JSON encoding/decoding • Javascript Map/Reduce built on top of JSON
  • Web-Shaped Storage
  • Web-Shaped Storage • Content neutral
  • Web-Shaped Storage • Content neutral • Highly distributed
  • Web-Shaped Storage • Content neutral • Highly distributed • Replicated
  • Web-Shaped Storage • Content neutral • Highly distributed • Replicated • Fault-tolerant
  • What Is Riak?
  • What Is Riak? A flexible storage engine...
  • What Is Riak? A flexible storage engine... ...with a REST API...
  • What Is Riak? A flexible storage engine... ...with a REST API... ...and map/reduce capability...
  • What Is Riak? A flexible storage engine... ...with a REST API... ...and map/reduce capability... ....designed to be fault-tolerant...
  • What Is Riak? A flexible storage engine... ...with a REST API... ...and map/reduce capability... ....designed to be fault-tolerant... ...distributed...
  • What Is Riak? A flexible storage engine... ...with a REST API... ...and map/reduce capability... ....designed to be fault-tolerant... ...distributed... ...and ops friendly
  • Influences
  • Influences • CAP Theorem
  • Influences • CAP Theorem • Amazon’s Dynamo Paper
  • Influences • CAP Theorem • Amazon’s Dynamo Paper • Experience running large networks (Akamai)
  • CAP Theorem
  • CAP Theorem Consistent Reads and writes reflect a globally consistent system state
  • CAP Theorem Consistent Reads and writes reflect a globally consistent system state
  • CAP Theorem Consistent Reads and writes reflect a globally consistent system state Available System is available for reads and writes
  • CAP Theorem Consistent Reads and writes reflect a globally consistent system state Available System is available for reads and writes
  • CAP Theorem Consistent Reads and writes reflect a globally consistent system state Available System is available for reads and writes Partition Tolerant System can handle the failure of individual parts
  • Common Wisdom
  • Common Wisdom Pick two.
  • The Riak Way
  • The Riak Way Pick Two.
  • The Riak Way Pick Two. For each operation.
  • Dynamo Influences
  • Dynamo Influences • N = The number of replicas
  • Dynamo Influences • N = The number of replicas • R = The number of replicas needed for a successful read
  • Dynamo Influences • N = The number of replicas • R = The number of replicas needed for a successful read • W = The number of replicas needed for a successful write
  • Dynamo Math
  • Dynamo Math N - R = read fault tolerance
  • Dynamo Math N - R = read fault tolerance N - W = write fault tolerance
  • Dynamo Math
  • Dynamo Math N = 4, W = 2, R = 1
  • Dynamo Math N = 4, W = 2, R = 1
  • Dynamo Math N = 4, W = 2, R = 1 4 - 2 = 2 hosts can be down and Riak can still perform writes.
  • Dynamo Math N = 4, W = 2, R = 1 4 - 2 = 2 hosts can be down and Riak can still perform writes. 4 - 1 = 3 hosts can be down and Riak can still perform reads.
  • Riak Improvements
  • Riak Improvements • N can vary per bucket
  • Riak Improvements • N can vary per bucket • R and W can vary per operation
  • Riak Improvements • N can vary per bucket • R and W can vary per operation Choose your own fault tolerance/performance tradeoff
  • Consistent Hashing 2160 0 node 0 node 1 2160/4 node 2 node 3 hash(<<"artist">>,<<"REM">>) 2160/2
  • R value get(<<"artist">>,<<"REM">>, R=2) (N=3) {ok, Object} X
  • W value put(<<"artist">>,<<"REM">>, W=2) (N=3) ok X
  • N=10, R/W=2 get/put("artist", "REM", R/W=2) (N=10) {ok, Object} X X X X X X X X
  • Resolving Conflicts
  • Resolving Conflicts • Riak focuses on the AP of CAP
  • Resolving Conflicts • Riak focuses on the AP of CAP • Data could be briefly inconsistent
  • Resolving Conflicts • Riak focuses on the AP of CAP • Data could be briefly inconsistent • Inconsistency must be resolved
  • Detecting & Resolving Conflicts 0 1 Object v0 2 3
  • Detecting & Resolving Conflicts Object 0 1 v0 Object 2 3 v0
  • Detecting & Resolving Conflicts Object 0 1 v1 Object 2 3 v0
  • Detecting & Resolving Conflicts 0 1 Object v1 2 3
  • Detecting & Resolving Conflicts Object 0 1 v1 Object 2 3 v1
  • Client Resolution
  • Client Resolution • Can be set per-bucket or server-wide
  • Client Resolution • Can be set per-bucket or server-wide • Conflicting data is “bubbled up” to the client
  • Client Resolution • Can be set per-bucket or server-wide • Conflicting data is “bubbled up” to the client • Client picks the winner
  • Server Resolution
  • Server Resolution • “Last write wins”
  • Server Resolution • “Last write wins” • Enabled by default
  • Server Resolution • “Last write wins” • Enabled by default • What most apps need 80% of the time
  • Live Demo!
  • Linking Objects
  • Linking Objects • Objects can store pointers, or links, to other objects
  • Linking Objects • Objects can store pointers, or links, to other objects • Doesn’t have to be the same bucket
  • Linking Objects • Objects can store pointers, or links, to other objects • Doesn’t have to be the same bucket • Object links described in a Link header
  • Link Header Format Object URL </riak/demo/test1>; riaktag="userinfo" Link tag
  • Link Walking
  • Link Walking • Ask Riak to “walk” a sequence of links
  • Link Walking • Ask Riak to “walk” a sequence of links • Optionally, collect objects along the walk and return them
  • Link Walking • Ask Riak to “walk” a sequence of links • Optionally, collect objects along the walk and return them • Can be arbitrarily deep
  • Link Walking Examples
  • Link Walking Examples /riak/demo/test1/_,_,1
  • Link Walking Examples /riak/demo/test1/_,_,1 Start walking at /demo/test1 and return all linked objects
  • Link Walking Examples
  • Link Walking Examples /riak/demo/test1/demo,_,1
  • Link Walking Examples /riak/demo/test1/demo,_,1 Start walking at /demo/test1 and return all linked objects contained in the demo bucket
  • Link Walking Examples
  • Link Walking Examples /riak/demo/test1/_,_,0/_,_,1
  • Link Walking Examples /riak/demo/test1/_,_,0/_,_,1 Start walking at /demo/test1, find any linked objects, then find and return any objects linked to those
  • Link Walking Examples
  • Link Walking Examples /riak/demo/test1/_,child,0/_,_,1
  • Link Walking Examples /riak/demo/test1/_,child,0/_,_,1 Start walking at /demo/test1, find any linked objects with the link tag “child”, then find and return any objects linked to those
  • Map/Reduce Terms
  • Map/Reduce Terms • Phase: A step within a job
  • Map/Reduce Terms • Phase: A step within a job • Job: A sequence of phases and inputs
  • Map/Reduce Terms • Phase: A step within a job • Job: A sequence of phases and inputs • Map: Data collection phase
  • Map/Reduce Terms • Phase: A step within a job • Job: A sequence of phases and inputs • Map: Data collection phase • Reduce: Data collation or processing phase
  • Map/Reduce Overview
  • Map/Reduce Overview • Map phases execute in parallel w/data locality
  • Map/Reduce Overview • Map phases execute in parallel w/data locality • Reduce phases execute in parallel on the node where job was submitted
  • Map/Reduce Overview • Map phases execute in parallel w/data locality • Reduce phases execute in parallel on the node where job was submitted • Results are not cached or stored
  • Map/Reduce Overview • Map phases execute in parallel w/data locality • Reduce phases execute in parallel on the node where job was submitted • Results are not cached or stored • Phases can be written in Erlang or Javascript
  • Map Phase
  • Map Phase • Inputs must be bucket/key pairs
  • Map Phase • Inputs must be bucket/key pairs • Must return a list
  • Map Phase • Inputs must be bucket/key pairs • Must return a list • Parallel results are aggregated into a single list
  • Parallel Map
  • Parallel Map
  • Parallel Map
  • Parallel Map
  • Erlang Map Phase
  • Erlang Map Phase • Two types: modfun and qfun
  • Erlang Map Phase • Two types: modfun and qfun • modfuns reference the module and name of the Erlang function to call
  • Erlang Map Phase • Two types: modfun and qfun • modfuns reference the module and name of the Erlang function to call • qfuns are anonymous Erlang functions*
  • Erlang Map Phase • Two types: modfun and qfun • modfuns reference the module and name of the Erlang function to call • qfuns are anonymous Erlang functions* *Must be on the server-side codepath
  • Erlang Map Phase
  • Erlang Map Phase map_object_value(Obj, _KeyData, _Arg) ->
  • Erlang Map Phase map_object_value(Obj, _KeyData, _Arg) -> [riak_object:get_value(Obj)].
  • Erlang Map Phase map_object_value(Obj, _KeyData, _Arg) -> [riak_object:get_value(Obj)].
  • Erlang Map Phase map_object_value(Obj, _KeyData, _Arg) -> [riak_object:get_value(Obj)]. • Obj:riak_object retrieved from bucket/key
  • Erlang Map Phase map_object_value(Obj, _KeyData, _Arg) -> [riak_object:get_value(Obj)]. • Obj:riak_object retrieved from bucket/key • KeyData: Static argument specified with the bucket/ key
  • Erlang Map Phase map_object_value(Obj, _KeyData, _Arg) -> [riak_object:get_value(Obj)]. • Obj:riak_object retrieved from bucket/key • KeyData: Static argument specified with the bucket/ key • Arg: Static argument specified with the phase
  • Erlang Map Built-Ins
  • Erlang Map Built-Ins riak_mapreduce:map_object_value/3
  • Erlang Map Built-Ins riak_mapreduce:map_object_value/3 • Returns object value wrapped in a list
  • Erlang Map Built-Ins riak_mapreduce:map_object_value/3 • Returns object value wrapped in a list riak_mapreduce:map_object_value_list/3
  • Erlang Map Built-Ins riak_mapreduce:map_object_value/3 • Returns object value wrapped in a list riak_mapreduce:map_object_value_list/3 • Returns object value. Object value must already be a list
  • Javascript Map Phase
  • Javascript Map Phase • Two types: jsanon and jsfun
  • Javascript Map Phase • Two types: jsanon and jsfun • jsanons are anonymous JS functions:
  • Javascript Map Phase • Two types: jsanon and jsfun • jsanons are anonymous JS functions: function(value) { return [value]; }
  • Javascript Map Phase • Two types: jsanon and jsfun • jsanons are anonymous JS functions: function(value) { return [value]; } • jsfuns are named JS functions:
  • Javascript Map Phase • Two types: jsanon and jsfun • jsanons are anonymous JS functions: function(value) { return [value]; } • jsfuns are named JS functions: Riak.mapValuesJson
  • Erlang & Javascript
  • Erlang & Javascript • Same environment as Firefox minus browser bits
  • Erlang & Javascript • Same environment as Firefox minus browser bits • Erlang to Javascript data is JSON encoded
  • Erlang & Javascript • Same environment as Firefox minus browser bits • Erlang to Javascript data is JSON encoded • Javascript to Erlang data is JSON decoded
  • Javascript Map Phase
  • Javascript Map Phase function(value, keyData, arg)
  • Javascript Map Phase function(value, keyData, arg)
  • Javascript Map Phase function(value, keyData, arg) • value: JSON-encoded version of riak_object
  • Javascript Map Phase function(value, keyData, arg) • value: JSON-encoded version of riak_object • keyData: Same as Erlang
  • Javascript Map Phase function(value, keyData, arg) • value: JSON-encoded version of riak_object • keyData: Same as Erlang • arg: Same as Erlang
  • Javascript Map Built-Ins
  • Javascript Map Built-Ins Riak.mapValues
  • Javascript Map Built-Ins Riak.mapValues • Returns object values. Handles detecting when/if to use list wrapping.
  • Javascript Map Built-Ins Riak.mapValues • Returns object values. Handles detecting when/if to use list wrapping. Riak.mapValuesJson
  • Javascript Map Built-Ins Riak.mapValues • Returns object values. Handles detecting when/if to use list wrapping. Riak.mapValuesJson • Returns JSON parsed object values. Also performs list wrapping, if needed.
  • Reduce Phase
  • Reduce Phase • Performed on the node coordinating the map/reduce job
  • Reduce Phase • Performed on the node coordinating the map/reduce job • Two processes per reduce phase to add minor parallelism
  • Reduce Phase • Performed on the node coordinating the map/reduce job • Two processes per reduce phase to add minor parallelism • Must return a list
  • Erlang Reduce Built-Ins
  • Erlang Reduce Built-Ins riak_mapreduce:reduce_set_union/2
  • Erlang Reduce Built-Ins riak_mapreduce:reduce_set_union/2 • Returns unique set of values
  • Erlang Reduce Built-Ins riak_mapreduce:reduce_set_union/2 • Returns unique set of values riak_mapreduce:reduce_sum/2
  • Erlang Reduce Built-Ins riak_mapreduce:reduce_set_union/2 • Returns unique set of values riak_mapreduce:reduce_sum/2 • Returns the sum of inputs
  • Erlang Reduce Built-Ins riak_mapreduce:reduce_set_union/2 • Returns unique set of values riak_mapreduce:reduce_sum/2 • Returns the sum of inputs riak_mapreduce:reduce_sort/2
  • Erlang Reduce Built-Ins riak_mapreduce:reduce_set_union/2 • Returns unique set of values riak_mapreduce:reduce_sum/2 • Returns the sum of inputs riak_mapreduce:reduce_sort/2 • Returns the sorted list of inputs
  • Javascript Reduce Built-Ins
  • Javascript Reduce Built-Ins Riak.reduceMin
  • Javascript Reduce Built-Ins Riak.reduceMin • Returns the minimum value of the input set
  • Javascript Reduce Built-Ins Riak.reduceMin • Returns the minimum value of the input set Riak.reduceMax
  • Javascript Reduce Built-Ins Riak.reduceMin • Returns the minimum value of the input set Riak.reduceMax • Returns the maximum value of the input set
  • Javascript Reduce Built-Ins Riak.reduceMin • Returns the minimum value of the input set Riak.reduceMax • Returns the maximum value of the input set Riak.reduceSort
  • Javascript Reduce Built-Ins Riak.reduceMin • Returns the minimum value of the input set Riak.reduceMax • Returns the maximum value of the input set Riak.reduceSort • Returns a sorted list of the input set
  • Building M/R Job
  • Building M/R Job • Job is a list of phases and starting inputs
  • Building M/R Job • Job is a list of phases and starting inputs • Each phase can:
  • Building M/R Job • Job is a list of phases and starting inputs • Each phase can: • Receive a static argument
  • Building M/R Job • Job is a list of phases and starting inputs • Each phase can: • Receive a static argument • Accumulate and return results
  • Submitting Jobs via HTTP
  • Submitting Jobs via HTTP • Riak exposes M/R via its REST API
  • Submitting Jobs via HTTP • Riak exposes M/R via its REST API • Job is described in JSON
  • Submitting Jobs via HTTP • Riak exposes M/R via its REST API • Job is described in JSON • Submitted via POST
  • Submitting Jobs via HTTP • Riak exposes M/R via its REST API • Job is described in JSON • Submitted via POST • Default URL is /mapred
  • Erlang Phase (JSON)
  • Erlang Phase (JSON) {Type:{“language”:”erlang”, “module”: Module,
  • Erlang Phase (JSON) {Type:{“language”:”erlang”, “module”: Module, “function”: Function, “keep”:Flag}}
  • Erlang Phase (JSON) {Type:{“language”:”erlang”, “module”: Module, “function”: Function, “keep”:Flag}}
  • Erlang Phase (JSON) {Type:{“language”:”erlang”, “module”: Module, “function”: Function, “keep”:Flag}} • Type: “map” or “reduce”
  • Erlang Phase (JSON) {Type:{“language”:”erlang”, “module”: Module, “function”: Function, “keep”:Flag}} • Type: “map” or “reduce” • Module: String name of Erlang module
  • Erlang Phase (JSON) {Type:{“language”:”erlang”, “module”: Module, “function”: Function, “keep”:Flag}} • Type: “map” or “reduce” • Module: String name of Erlang module • Function: String name of Erlang function
  • Erlang Phase (JSON) {Type:{“language”:”erlang”, “module”: Module, “function”: Function, “keep”:Flag}} • Type: “map” or “reduce” • Module: String name of Erlang module • Function: String name of Erlang function • Flag: Boolean accumulation toggle
  • Javascript Phase (JSON)
  • Javascript Phase (JSON) {Type:{“language”:”javascript”,
  • Javascript Phase (JSON) {Type:{“language”:”javascript”, “source”: Source,“keep”:Flag}}
  • Javascript Phase (JSON) {Type:{“language”:”javascript”, “source”: Source,“keep”:Flag}}
  • Javascript Phase (JSON) {Type:{“language”:”javascript”, “source”: Source,“keep”:Flag}} • Type: “map” or “reduce”
  • Javascript Phase (JSON) {Type:{“language”:”javascript”, “source”: Source,“keep”:Flag}} • Type: “map” or “reduce” • Source: Source for anonymous function
  • Javascript Phase (JSON) {Type:{“language”:”javascript”, “source”: Source,“keep”:Flag}} • Type: “map” or “reduce” • Source: Source for anonymous function • Flag: Boolean accumulation toggle
  • Javascript Phase (JSON)
  • Javascript Phase (JSON) {Type:{“language”:”javascript”,
  • Javascript Phase (JSON) {Type:{“language”:”javascript”, “name”:Name,“keep”:Flag}}
  • Javascript Phase (JSON) {Type:{“language”:”javascript”, “name”:Name,“keep”:Flag}}
  • Javascript Phase (JSON) {Type:{“language”:”javascript”, “name”:Name,“keep”:Flag}} • Type: “map” or “reduce”
  • Javascript Phase (JSON) {Type:{“language”:”javascript”, “name”:Name,“keep”:Flag}} • Type: “map” or “reduce” • Name: String name of Javascript function
  • Javascript Phase (JSON) {Type:{“language”:”javascript”, “name”:Name,“keep”:Flag}} • Type: “map” or “reduce” • Name: String name of Javascript function • Flag: Boolean accumulation toggle
  • Putting It Together {“inputs”: [[“stocks”, “goog”]], “query”: [{“map”:{“language”:”javascript”, “name”: “Riak.mapValuesJson”}, “keep”: true}]}
  • Putting It Together {“inputs”: [[“stocks”, “goog”], [“stocks”, “csco”]], “query”: [{“map”:{“language”:”javascript”, “name”: “Riak.mapValuesJson”}, “keep”: true}]}
  • Putting It Together {“inputs”: “stocks”, “query”: [{“map”:{“language”:”javascript”, “name”: “App.extractTickers”, “arg”: “GOOG”}, “keep”: false}, {“reduce”:{“language”:”javascript, “name”: “Riak.reduceMin”}, “keep”: true}]}
  • Live Demo!
  • Thank You Kevin A. Smith Email: ksmith@basho.com Twitter: @kevsmith