Introducing Riak

Introducing Riak Kevin A. Smith Senior Developer Basho Technologies

What Is Riak?

What Is Riak? • Key/Value store

What Is Riak? • Key/Value store • Document-oriented database

What Is Riak? • Key/Value store • Document-oriented database • Web-shaped storage

Key/Value Store

Key/Value Store • Data organized by bucket/key pairs

Key/Value Store • Data organized by bucket/key pairs • Simple REST API (GET, PUT, DELETE)

Document Store

Document Store • Store values as JSON

Document Store • Store values as JSON • Many clients support automatic JSON encoding/decoding

Document Store • Store values as JSON • Many clients support automatic JSON encoding/decoding • Javascript Map/Reduce built on top of JSON

Web-Shaped Storage

Web-Shaped Storage • Content neutral

Web-Shaped Storage • Content neutral • Highly distributed

Web-Shaped Storage • Content neutral • Highly distributed • Replicated

Web-Shaped Storage • Content neutral • Highly distributed • Replicated • Fault-tolerant

What Is Riak?

What Is Riak? A ﬂexible storage engine...

What Is Riak? A ﬂexible storage engine... ...with a REST API...

What Is Riak? A ﬂexible storage engine... ...with a REST API... ...and map/reduce capability...

What Is Riak? A ﬂexible storage engine... ...with a REST API... ...and map/reduce capability... ....designed to be fault-tolerant...

What Is Riak? A ﬂexible storage engine... ...with a REST API... ...and map/reduce capability... ....designed to be fault-tolerant... ...distributed...

What Is Riak? A ﬂexible storage engine... ...with a REST API... ...and map/reduce capability... ....designed to be fault-tolerant... ...distributed... ...and ops friendly

Inﬂuences

Inﬂuences • CAP Theorem

Inﬂuences • CAP Theorem • Amazon’s Dynamo Paper

Inﬂuences • CAP Theorem • Amazon’s Dynamo Paper • Experience running large networks (Akamai)

CAP Theorem

CAP Theorem Consistent Reads and writes reﬂect a globally consistent system state

CAP Theorem Consistent Reads and writes reﬂect a globally consistent system state Available System is available for reads and writes

CAP Theorem Consistent Reads and writes reﬂect a globally consistent system state Available System is available for reads and writes Partition Tolerant System can handle the failure of individual parts

Common Wisdom

Common Wisdom Pick two.

The Riak Way

The Riak Way Pick Two.

The Riak Way Pick Two. For each operation.

Dynamo Inﬂuences

Dynamo Inﬂuences • N = The number of replicas

Dynamo Inﬂuences • N = The number of replicas • R = The number of replicas needed for a successful read

Dynamo Inﬂuences • N = The number of replicas • R = The number of replicas needed for a successful read • W = The number of replicas needed for a successful write

Dynamo Math

Dynamo Math N - R = read fault tolerance

Dynamo Math N - R = read fault tolerance N - W = write fault tolerance

Dynamo Math

Dynamo Math N = 4, W = 2, R = 1

Dynamo Math N = 4, W = 2, R = 1 4 - 2 = 2 hosts can be down and Riak can still perform writes.

Dynamo Math N = 4, W = 2, R = 1 4 - 2 = 2 hosts can be down and Riak can still perform writes. 4 - 1 = 3 hosts can be down and Riak can still perform reads.

Riak Improvements

Riak Improvements • N can vary per bucket

Riak Improvements • N can vary per bucket • R and W can vary per operation

Riak Improvements • N can vary per bucket • R and W can vary per operation Choose your own fault tolerance/performance tradeoff

Consistent Hashing 2160 0 node 0 node 1 2160/4 node 2 node 3 hash(<<"artist">>,<<"REM">>) 2160/2

R value get(<<"artist">>,<<"REM">>, R=2) (N=3) {ok, Object} X

W value put(<<"artist">>,<<"REM">>, W=2) (N=3) ok X

N=10, R/W=2 get/put("artist", "REM", R/W=2) (N=10) {ok, Object} X X X X X X X X

Resolving Conﬂicts

Resolving Conﬂicts • Riak focuses on the AP of CAP

Resolving Conﬂicts • Riak focuses on the AP of CAP • Data could be brieﬂy inconsistent

Resolving Conﬂicts • Riak focuses on the AP of CAP • Data could be brieﬂy inconsistent • Inconsistency must be resolved

Detecting & Resolving Conﬂicts 0 1 Object v0 2 3

Detecting & Resolving Conﬂicts Object 0 1 v0 Object 2 3 v0

Detecting & Resolving Conﬂicts 0 1 Object v1 2 3

Client Resolution

Client Resolution • Can be set per-bucket or server-wide

Client Resolution • Can be set per-bucket or server-wide • Conﬂicting data is “bubbled up” to the client

Client Resolution • Can be set per-bucket or server-wide • Conﬂicting data is “bubbled up” to the client • Client picks the winner

Server Resolution

Server Resolution • “Last write wins”

Server Resolution • “Last write wins” • Enabled by default

Server Resolution • “Last write wins” • Enabled by default • What most apps need 80% of the time

Live Demo!

Linking Objects

Linking Objects • Objects can store pointers, or links, to other objects

Linking Objects • Objects can store pointers, or links, to other objects • Doesn’t have to be the same bucket

Linking Objects • Objects can store pointers, or links, to other objects • Doesn’t have to be the same bucket • Object links described in a Link header

Link Header Format Object URL </riak/demo/test1>; riaktag="userinfo" Link tag

Link Walking

Link Walking • Ask Riak to “walk” a sequence of links

Link Walking • Ask Riak to “walk” a sequence of links • Optionally, collect objects along the walk and return them

Link Walking • Ask Riak to “walk” a sequence of links • Optionally, collect objects along the walk and return them • Can be arbitrarily deep

Link Walking Examples

Link Walking Examples /riak/demo/test1/_,_,1

Link Walking Examples /riak/demo/test1/_,_,1 Start walking at /demo/test1 and return all linked objects

Link Walking Examples /riak/demo/test1/demo,_,1

Link Walking Examples /riak/demo/test1/demo,_,1 Start walking at /demo/test1 and return all linked objects contained in the demo bucket

Link Walking Examples /riak/demo/test1/_,_,0/_,_,1

Link Walking Examples /riak/demo/test1/_,_,0/_,_,1 Start walking at /demo/test1, ﬁnd any linked objects, then ﬁnd and return any objects linked to those

Link Walking Examples /riak/demo/test1/_,child,0/_,_,1

Link Walking Examples /riak/demo/test1/_,child,0/_,_,1 Start walking at /demo/test1, ﬁnd any linked objects with the link tag “child”, then ﬁnd and return any objects linked to those

Map/Reduce Terms

Map/Reduce Terms • Phase: A step within a job

Map/Reduce Terms • Phase: A step within a job • Job: A sequence of phases and inputs

Map/Reduce Terms • Phase: A step within a job • Job: A sequence of phases and inputs • Map: Data collection phase

Map/Reduce Terms • Phase: A step within a job • Job: A sequence of phases and inputs • Map: Data collection phase • Reduce: Data collation or processing phase

Map/Reduce Overview

Map/Reduce Overview • Map phases execute in parallel w/data locality

Map/Reduce Overview • Map phases execute in parallel w/data locality • Reduce phases execute in parallel on the node where job was submitted

Map/Reduce Overview • Map phases execute in parallel w/data locality • Reduce phases execute in parallel on the node where job was submitted • Results are not cached or stored

Map/Reduce Overview • Map phases execute in parallel w/data locality • Reduce phases execute in parallel on the node where job was submitted • Results are not cached or stored • Phases can be written in Erlang or Javascript

Map Phase

Map Phase • Inputs must be bucket/key pairs

Map Phase • Inputs must be bucket/key pairs • Must return a list

Map Phase • Inputs must be bucket/key pairs • Must return a list • Parallel results are aggregated into a single list

Parallel Map

Erlang Map Phase

Erlang Map Phase • Two types: modfun and qfun

Erlang Map Phase • Two types: modfun and qfun • modfuns reference the module and name of the Erlang function to call

Erlang Map Phase • Two types: modfun and qfun • modfuns reference the module and name of the Erlang function to call • qfuns are anonymous Erlang functions*

Erlang Map Phase • Two types: modfun and qfun • modfuns reference the module and name of the Erlang function to call • qfuns are anonymous Erlang functions* *Must be on the server-side codepath

Erlang Map Phase

Erlang Map Phase map_object_value(Obj, _KeyData, _Arg) ->

Erlang Map Phase map_object_value(Obj, _KeyData, _Arg) -> [riak_object:get_value(Obj)].

Erlang Map Phase map_object_value(Obj, _KeyData, _Arg) -> [riak_object:get_value(Obj)]. • Obj:riak_object retrieved from bucket/key

Erlang Map Phase map_object_value(Obj, _KeyData, _Arg) -> [riak_object:get_value(Obj)]. • Obj:riak_object retrieved from bucket/key • KeyData: Static argument speciﬁed with the bucket/ key

Erlang Map Phase map_object_value(Obj, _KeyData, _Arg) -> [riak_object:get_value(Obj)]. • Obj:riak_object retrieved from bucket/key • KeyData: Static argument speciﬁed with the bucket/ key • Arg: Static argument speciﬁed with the phase

Erlang Map Built-Ins

Erlang Map Built-Ins riak_mapreduce:map_object_value/3

Erlang Map Built-Ins riak_mapreduce:map_object_value/3 • Returns object value wrapped in a list

Erlang Map Built-Ins riak_mapreduce:map_object_value/3 • Returns object value wrapped in a list riak_mapreduce:map_object_value_list/3

Erlang Map Built-Ins riak_mapreduce:map_object_value/3 • Returns object value wrapped in a list riak_mapreduce:map_object_value_list/3 • Returns object value. Object value must already be a list

Javascript Map Phase

Javascript Map Phase • Two types: jsanon and jsfun

Javascript Map Phase • Two types: jsanon and jsfun • jsanons are anonymous JS functions:

Javascript Map Phase • Two types: jsanon and jsfun • jsanons are anonymous JS functions: function(value) { return [value]; }

Javascript Map Phase • Two types: jsanon and jsfun • jsanons are anonymous JS functions: function(value) { return [value]; } • jsfuns are named JS functions:

Javascript Map Phase • Two types: jsanon and jsfun • jsanons are anonymous JS functions: function(value) { return [value]; } • jsfuns are named JS functions: Riak.mapValuesJson

Erlang & Javascript

Erlang & Javascript • Same environment as Firefox minus browser bits

Erlang & Javascript • Same environment as Firefox minus browser bits • Erlang to Javascript data is JSON encoded

Erlang & Javascript • Same environment as Firefox minus browser bits • Erlang to Javascript data is JSON encoded • Javascript to Erlang data is JSON decoded

Javascript Map Phase

Javascript Map Phase function(value, keyData, arg)

Javascript Map Phase function(value, keyData, arg) • value: JSON-encoded version of riak_object

Javascript Map Phase function(value, keyData, arg) • value: JSON-encoded version of riak_object • keyData: Same as Erlang

Javascript Map Phase function(value, keyData, arg) • value: JSON-encoded version of riak_object • keyData: Same as Erlang • arg: Same as Erlang

Javascript Map Built-Ins

Javascript Map Built-Ins Riak.mapValues

Javascript Map Built-Ins Riak.mapValues • Returns object values. Handles detecting when/if to use list wrapping.

Javascript Map Built-Ins Riak.mapValues • Returns object values. Handles detecting when/if to use list wrapping. Riak.mapValuesJson

Javascript Map Built-Ins Riak.mapValues • Returns object values. Handles detecting when/if to use list wrapping. Riak.mapValuesJson • Returns JSON parsed object values. Also performs list wrapping, if needed.

Reduce Phase

Reduce Phase • Performed on the node coordinating the map/reduce job

Reduce Phase • Performed on the node coordinating the map/reduce job • Two processes per reduce phase to add minor parallelism

Reduce Phase • Performed on the node coordinating the map/reduce job • Two processes per reduce phase to add minor parallelism • Must return a list

Erlang Reduce Built-Ins

Erlang Reduce Built-Ins riak_mapreduce:reduce_set_union/2

Erlang Reduce Built-Ins riak_mapreduce:reduce_set_union/2 • Returns unique set of values

Erlang Reduce Built-Ins riak_mapreduce:reduce_set_union/2 • Returns unique set of values riak_mapreduce:reduce_sum/2

Erlang Reduce Built-Ins riak_mapreduce:reduce_set_union/2 • Returns unique set of values riak_mapreduce:reduce_sum/2 • Returns the sum of inputs

Erlang Reduce Built-Ins riak_mapreduce:reduce_set_union/2 • Returns unique set of values riak_mapreduce:reduce_sum/2 • Returns the sum of inputs riak_mapreduce:reduce_sort/2

Erlang Reduce Built-Ins riak_mapreduce:reduce_set_union/2 • Returns unique set of values riak_mapreduce:reduce_sum/2 • Returns the sum of inputs riak_mapreduce:reduce_sort/2 • Returns the sorted list of inputs

Javascript Reduce Built-Ins

Javascript Reduce Built-Ins Riak.reduceMin

Javascript Reduce Built-Ins Riak.reduceMin • Returns the minimum value of the input set

Javascript Reduce Built-Ins Riak.reduceMin • Returns the minimum value of the input set Riak.reduceMax

Javascript Reduce Built-Ins Riak.reduceMin • Returns the minimum value of the input set Riak.reduceMax • Returns the maximum value of the input set

Javascript Reduce Built-Ins Riak.reduceMin • Returns the minimum value of the input set Riak.reduceMax • Returns the maximum value of the input set Riak.reduceSort

Javascript Reduce Built-Ins Riak.reduceMin • Returns the minimum value of the input set Riak.reduceMax • Returns the maximum value of the input set Riak.reduceSort • Returns a sorted list of the input set

Building M/R Job

Building M/R Job • Job is a list of phases and starting inputs

Building M/R Job • Job is a list of phases and starting inputs • Each phase can:

Building M/R Job • Job is a list of phases and starting inputs • Each phase can: • Receive a static argument

Building M/R Job • Job is a list of phases and starting inputs • Each phase can: • Receive a static argument • Accumulate and return results

Submitting Jobs via HTTP

Submitting Jobs via HTTP • Riak exposes M/R via its REST API

Submitting Jobs via HTTP • Riak exposes M/R via its REST API • Job is described in JSON

Submitting Jobs via HTTP • Riak exposes M/R via its REST API • Job is described in JSON • Submitted via POST

Submitting Jobs via HTTP • Riak exposes M/R via its REST API • Job is described in JSON • Submitted via POST • Default URL is /mapred

Erlang Phase (JSON)

Erlang Phase (JSON) {Type:{“language”:”erlang”, “module”: Module,

Erlang Phase (JSON) {Type:{“language”:”erlang”, “module”: Module, “function”: Function, “keep”:Flag}}

Erlang Phase (JSON) {Type:{“language”:”erlang”, “module”: Module, “function”: Function, “keep”:Flag}} • Type: “map” or “reduce”

Erlang Phase (JSON) {Type:{“language”:”erlang”, “module”: Module, “function”: Function, “keep”:Flag}} • Type: “map” or “reduce” • Module: String name of Erlang module

Erlang Phase (JSON) {Type:{“language”:”erlang”, “module”: Module, “function”: Function, “keep”:Flag}} • Type: “map” or “reduce” • Module: String name of Erlang module • Function: String name of Erlang function

Erlang Phase (JSON) {Type:{“language”:”erlang”, “module”: Module, “function”: Function, “keep”:Flag}} • Type: “map” or “reduce” • Module: String name of Erlang module • Function: String name of Erlang function • Flag: Boolean accumulation toggle

Javascript Phase (JSON)

Javascript Phase (JSON) {Type:{“language”:”javascript”,

Javascript Phase (JSON) {Type:{“language”:”javascript”, “source”: Source,“keep”:Flag}}

Javascript Phase (JSON) {Type:{“language”:”javascript”, “source”: Source,“keep”:Flag}} • Type: “map” or “reduce”

Javascript Phase (JSON) {Type:{“language”:”javascript”, “source”: Source,“keep”:Flag}} • Type: “map” or “reduce” • Source: Source for anonymous function

Javascript Phase (JSON) {Type:{“language”:”javascript”, “source”: Source,“keep”:Flag}} • Type: “map” or “reduce” • Source: Source for anonymous function • Flag: Boolean accumulation toggle

Javascript Phase (JSON)

Javascript Phase (JSON) {Type:{“language”:”javascript”,

Javascript Phase (JSON) {Type:{“language”:”javascript”, “name”:Name,“keep”:Flag}}

Javascript Phase (JSON) {Type:{“language”:”javascript”, “name”:Name,“keep”:Flag}} • Type: “map” or “reduce”

Javascript Phase (JSON) {Type:{“language”:”javascript”, “name”:Name,“keep”:Flag}} • Type: “map” or “reduce” • Name: String name of Javascript function

Javascript Phase (JSON) {Type:{“language”:”javascript”, “name”:Name,“keep”:Flag}} • Type: “map” or “reduce” • Name: String name of Javascript function • Flag: Boolean accumulation toggle

Putting It Together {“inputs”: [[“stocks”, “goog”]], “query”: [{“map”:{“language”:”javascript”, “name”: “Riak.mapValuesJson”}, “keep”: true}]}

Putting It Together {“inputs”: [[“stocks”, “goog”], [“stocks”, “csco”]], “query”: [{“map”:{“language”:”javascript”, “name”: “Riak.mapValuesJson”}, “keep”: true}]}

Putting It Together {“inputs”: “stocks”, “query”: [{“map”:{“language”:”javascript”, “name”: “App.extractTickers”, “arg”: “GOOG”}, “keep”: false}, {“reduce”:{“language”:”javascript, “name”: “Riak.reduceMin”}, “keep”: true}]}

Live Demo!

Thank You Kevin A. Smith Email: ksmith@basho.com Twitter: @kevsmith

http://nosql.mypopescu.com	550
http://acovado.com	89
http://www.slideshare.net	54
http://scalabilitypatterns.com	30
http://www.acovado.com	12
http://scalabilitypatterns.wordpress.com	7
http://us-w1.rockmelt.com	2
http://static.slidesharecdn.com	2
http://xianguo.com	1
http://www.iweb34.com	1
http://www.zhuaxia.com	1
http://www.awesomehighlighter.com	1
http://awurl.com	1

Introducing Riak

by Kevin Smith on Apr 07, 2010

Accessibility

Upload Details

Usage Rights

13 Embeds 751

Statistics

Introducing Riak Presentation Transcript