Introducing
    Riak
     Kevin A. Smith
   Senior Developer
  Basho Technologies
What Is Riak?
What Is Riak?

• Key/Value store
What Is Riak?

• Key/Value store
• Document-oriented database
What Is Riak?

• Key/Value store
• Document-oriented database
• Web-shaped storage
Key/Value Store
Key/Value Store


• Data organized by bucket/key pairs
Key/Value Store


• Data organized by bucket/key pairs
• Simple REST API (GET, PUT, DELETE)
Document Store
Document Store

• Store values as JSON
Document Store

• Store values as JSON
• Many clients support automatic JSON
  encoding/decoding
Document Store

• Store values as JSON
• Many clients support automatic JSON
  encoding/decoding
• Javascript Map/Reduce b...
Web-Shaped
 Storage
Web-Shaped
         Storage

• Content neutral
Web-Shaped
          Storage

• Content neutral
• Highly distributed
Web-Shaped
          Storage

• Content neutral
• Highly distributed
• Replicated
Web-Shaped
          Storage

• Content neutral
• Highly distributed
• Replicated
• Fault-tolerant
What Is Riak?
What Is Riak?
A flexible storage engine...
What Is Riak?
A flexible storage engine...
   ...with a REST API...
What Is Riak?
 A flexible storage engine...
     ...with a REST API...
...and map/reduce capability...
What Is Riak?
   A flexible storage engine...
       ...with a REST API...
 ...and map/reduce capability...
....designed to...
What Is Riak?
   A flexible storage engine...
       ...with a REST API...
 ...and map/reduce capability...
....designed to...
What Is Riak?
   A flexible storage engine...
       ...with a REST API...
 ...and map/reduce capability...
....designed to...
Influences
Influences

• CAP Theorem
Influences

• CAP Theorem
• Amazon’s Dynamo Paper
Influences

• CAP Theorem
• Amazon’s Dynamo Paper
• Experience running large networks
  (Akamai)
CAP Theorem
CAP Theorem
Consistent Reads and writes reflect a
globally consistent system state
CAP Theorem
Consistent Reads and writes reflect a
globally consistent system state
CAP Theorem
Consistent Reads and writes reflect a
globally consistent system state

Available System is available for reads...
CAP Theorem
Consistent Reads and writes reflect a
globally consistent system state

Available System is available for reads...
CAP Theorem
Consistent Reads and writes reflect a
globally consistent system state

Available System is available for reads...
Common Wisdom
Common Wisdom


   Pick two.
The Riak Way
The Riak Way

  Pick Two.
The Riak Way

     Pick Two.

For each operation.
Dynamo
Influences
Dynamo
           Influences
• N = The number of replicas
Dynamo
           Influences
• N = The number of replicas
• R = The number of replicas needed for a
  successful read
Dynamo
            Influences
• N = The number of replicas
• R = The number of replicas needed for a
  successful read
• W ...
Dynamo Math
Dynamo Math


N - R = read fault tolerance
Dynamo Math


N - R = read fault tolerance
N - W = write fault tolerance
Dynamo Math
Dynamo Math
N = 4, W = 2, R = 1
Dynamo Math
N = 4, W = 2, R = 1
Dynamo Math
N = 4, W = 2, R = 1


4 - 2 = 2 hosts can be down and Riak can
still perform writes.
Dynamo Math
N = 4, W = 2, R = 1


4 - 2 = 2 hosts can be down and Riak can
still perform writes.
4 - 1 = 3 hosts can be do...
Riak Improvements
Riak Improvements

• N can vary per bucket
Riak Improvements

• N can vary per bucket
• R and W can vary per operation
Riak Improvements

• N can vary per bucket
• R and W can vary per operation
   Choose your own fault tolerance/performance...
Consistent Hashing
2160            0

                                 node 0
                                 node 1
    ...
R value
            get(<<"artist">>,<<"REM">>,
                        R=2)

(N=3)
                            {ok, Objec...
W value
            put(<<"artist">>,<<"REM">>,
                        W=2)

(N=3)
                            ok




   ...
N=10, R/W=2
                                 get/put("artist", "REM",
                                          R/W=2)
   ...
Resolving Conflicts
Resolving Conflicts

• Riak focuses on the AP of CAP
Resolving Conflicts

• Riak focuses on the AP of CAP
• Data could be briefly inconsistent
Resolving Conflicts

• Riak focuses on the AP of CAP
• Data could be briefly inconsistent
• Inconsistency must be resolved
Detecting & Resolving
      Conflicts
    0   1
             Object
              v0
    2   3
Detecting & Resolving
      Conflicts
             Object
   0    1
              v0

             Object
   2    3
       ...
Detecting & Resolving
      Conflicts
             Object
   0    1
              v1

             Object
   2    3
       ...
Detecting & Resolving
      Conflicts
   0    1
             Object
              v1
   2    3
Detecting & Resolving
      Conflicts
             Object
   0    1
              v1

             Object
   2    3
       ...
Client Resolution
Client Resolution

• Can be set per-bucket or server-wide
Client Resolution

• Can be set per-bucket or server-wide
• Conflicting data is “bubbled up” to the
  client
Client Resolution

• Can be set per-bucket or server-wide
• Conflicting data is “bubbled up” to the
  client
• Client picks...
Server Resolution
Server Resolution

• “Last write wins”
Server Resolution

• “Last write wins”
• Enabled by default
Server Resolution

• “Last write wins”
• Enabled by default
• What most apps need 80% of the time
Live Demo!
Linking Objects
Linking Objects

• Objects can store pointers, or links, to
  other objects
Linking Objects

• Objects can store pointers, or links, to
  other objects
• Doesn’t have to be the same bucket
Linking Objects

• Objects can store pointers, or links, to
  other objects
• Doesn’t have to be the same bucket
• Object ...
Link Header Format

    Object URL


</riak/demo/test1>; riaktag="userinfo"


                              Link tag
Link Walking
Link Walking

• Ask Riak to “walk” a sequence of links
Link Walking

• Ask Riak to “walk” a sequence of links
• Optionally, collect objects along the walk
  and return them
Link Walking

• Ask Riak to “walk” a sequence of links
• Optionally, collect objects along the walk
  and return them
• Ca...
Link Walking Examples
Link Walking Examples


   /riak/demo/test1/_,_,1
Link Walking Examples


      /riak/demo/test1/_,_,1
Start walking at /demo/test1 and return all
linked objects
Link Walking Examples
Link Walking Examples


  /riak/demo/test1/demo,_,1
Link Walking Examples


    /riak/demo/test1/demo,_,1
Start walking at /demo/test1 and return all
linked objects contained...
Link Walking Examples
Link Walking Examples


 /riak/demo/test1/_,_,0/_,_,1
Link Walking Examples


     /riak/demo/test1/_,_,0/_,_,1
Start walking at /demo/test1, find any linked objects,
then find a...
Link Walking Examples
Link Walking Examples

/riak/demo/test1/_,child,0/_,_,1
Link Walking Examples

  /riak/demo/test1/_,child,0/_,_,1
Start walking at /demo/test1, find any linked objects
with the li...
Map/Reduce
  Terms
Map/Reduce
           Terms
• Phase: A step within a job
Map/Reduce
           Terms
• Phase: A step within a job
• Job: A sequence of phases and inputs
Map/Reduce
           Terms
• Phase: A step within a job
• Job: A sequence of phases and inputs
• Map: Data collection pha...
Map/Reduce
            Terms
• Phase: A step within a job
• Job: A sequence of phases and inputs
• Map: Data collection ph...
Map/Reduce
 Overview
Map/Reduce
              Overview
• Map phases execute in parallel w/data
  locality
Map/Reduce
              Overview
• Map phases execute in parallel w/data
  locality
• Reduce phases execute in parallel o...
Map/Reduce
              Overview
• Map phases execute in parallel w/data
  locality
• Reduce phases execute in parallel o...
Map/Reduce
              Overview
• Map phases execute in parallel w/data
  locality
• Reduce phases execute in parallel o...
Map Phase
Map Phase

• Inputs must be bucket/key pairs
Map Phase

• Inputs must be bucket/key pairs
• Must return a list
Map Phase

• Inputs must be bucket/key pairs
• Must return a list
• Parallel results are aggregated into a single
  list
Parallel Map
Parallel Map
Parallel Map
Parallel Map
Erlang Map Phase
Erlang Map Phase

• Two types: modfun and qfun
Erlang Map Phase

• Two types: modfun and qfun
• modfuns reference the module and name
  of the Erlang function to call
Erlang Map Phase

• Two types: modfun and qfun
• modfuns reference the module and name
    of the Erlang function to call
...
Erlang Map Phase

• Two types: modfun and qfun
• modfuns reference the module and name
    of the Erlang function to call
...
Erlang Map Phase
Erlang Map Phase
map_object_value(Obj, _KeyData, _Arg) ->
Erlang Map Phase
map_object_value(Obj, _KeyData, _Arg) ->
 [riak_object:get_value(Obj)].
Erlang Map Phase
map_object_value(Obj, _KeyData, _Arg) ->
 [riak_object:get_value(Obj)].
Erlang Map Phase
map_object_value(Obj, _KeyData, _Arg) ->
 [riak_object:get_value(Obj)].


  •   Obj:riak_object retrieved...
Erlang Map Phase
map_object_value(Obj, _KeyData, _Arg) ->
 [riak_object:get_value(Obj)].


  •   Obj:riak_object retrieved...
Erlang Map Phase
map_object_value(Obj, _KeyData, _Arg) ->
 [riak_object:get_value(Obj)].


  •   Obj:riak_object retrieved...
Erlang Map
 Built-Ins
Erlang Map
             Built-Ins
riak_mapreduce:map_object_value/3
Erlang Map
                Built-Ins
riak_mapreduce:map_object_value/3

•   Returns object value wrapped in a list
Erlang Map
                Built-Ins
riak_mapreduce:map_object_value/3

•   Returns object value wrapped in a list

riak_m...
Erlang Map
                Built-Ins
riak_mapreduce:map_object_value/3

•   Returns object value wrapped in a list

riak_m...
Javascript
Map Phase
Javascript
           Map Phase
• Two types: jsanon and jsfun
Javascript
            Map Phase
• Two types: jsanon and jsfun
• jsanons are anonymous JS functions:
Javascript
            Map Phase
• Two types: jsanon and jsfun
• jsanons are anonymous JS functions:
  function(value) { r...
Javascript
             Map Phase
• Two types: jsanon and jsfun
• jsanons are anonymous JS functions:
  function(value) { ...
Javascript
             Map Phase
• Two types: jsanon and jsfun
• jsanons are anonymous JS functions:
  function(value) { ...
Erlang & Javascript
Erlang & Javascript

• Same environment as Firefox minus
  browser bits
Erlang & Javascript

• Same environment as Firefox minus
  browser bits
• Erlang to Javascript data is JSON encoded
Erlang & Javascript

• Same environment as Firefox minus
  browser bits
• Erlang to Javascript data is JSON encoded
• Java...
Javascript Map Phase
Javascript Map Phase
function(value, keyData, arg)
Javascript Map Phase
function(value, keyData, arg)
Javascript Map Phase
function(value, keyData, arg)


• value: JSON-encoded version of
  riak_object
Javascript Map Phase
function(value, keyData, arg)


• value: JSON-encoded version of
  riak_object

• keyData: Same as Er...
Javascript Map Phase
function(value, keyData, arg)


• value: JSON-encoded version of
  riak_object

• keyData: Same as Er...
Javascript Map
   Built-Ins
Javascript Map
        Built-Ins
Riak.mapValues
Javascript Map
          Built-Ins
Riak.mapValues

•   Returns object values. Handles detecting
    when/if to use list wr...
Javascript Map
          Built-Ins
Riak.mapValues

•   Returns object values. Handles detecting
    when/if to use list wr...
Javascript Map
          Built-Ins
Riak.mapValues

•   Returns object values. Handles detecting
    when/if to use list wr...
Reduce Phase
Reduce Phase

• Performed on the node coordinating the
  map/reduce job
Reduce Phase

• Performed on the node coordinating the
  map/reduce job
• Two processes per reduce phase to add
  minor pa...
Reduce Phase

• Performed on the node coordinating the
  map/reduce job
• Two processes per reduce phase to add
  minor pa...
Erlang Reduce
   Built-Ins
Erlang Reduce
           Built-Ins
riak_mapreduce:reduce_set_union/2
Erlang Reduce
               Built-Ins
riak_mapreduce:reduce_set_union/2
•   Returns unique set of values
Erlang Reduce
               Built-Ins
riak_mapreduce:reduce_set_union/2
•   Returns unique set of values
riak_mapreduce:r...
Erlang Reduce
               Built-Ins
riak_mapreduce:reduce_set_union/2
•   Returns unique set of values
riak_mapreduce:r...
Erlang Reduce
               Built-Ins
riak_mapreduce:reduce_set_union/2
•   Returns unique set of values
riak_mapreduce:r...
Erlang Reduce
               Built-Ins
riak_mapreduce:reduce_set_union/2
•   Returns unique set of values
riak_mapreduce:r...
Javascript Reduce
     Built-Ins
Javascript Reduce
        Built-Ins
Riak.reduceMin
Javascript Reduce
           Built-Ins
  Riak.reduceMin

• Returns the minimum value of the input set
Javascript Reduce
           Built-Ins
  Riak.reduceMin

• Returns the minimum value of the input set
  Riak.reduceMax
Javascript Reduce
           Built-Ins
  Riak.reduceMin

• Returns the minimum value of the input set
  Riak.reduceMax

• ...
Javascript Reduce
           Built-Ins
  Riak.reduceMin

• Returns the minimum value of the input set
  Riak.reduceMax

• ...
Javascript Reduce
           Built-Ins
  Riak.reduceMin

• Returns the minimum value of the input set
  Riak.reduceMax

• ...
Building
M/R Job
Building
               M/R Job

• Job is a list of phases and starting inputs
Building
               M/R Job

• Job is a list of phases and starting inputs
• Each phase can:
Building
               M/R Job

• Job is a list of phases and starting inputs
• Each phase can:
 • Receive a static argum...
Building
               M/R Job

• Job is a list of phases and starting inputs
• Each phase can:
 • Receive a static argum...
Submitting Jobs
  via HTTP
Submitting Jobs
        via HTTP
• Riak exposes M/R via its REST API
Submitting Jobs
        via HTTP
• Riak exposes M/R via its REST API
• Job is described in JSON
Submitting Jobs
        via HTTP
• Riak exposes M/R via its REST API
• Job is described in JSON
• Submitted via POST
Submitting Jobs
        via HTTP
• Riak exposes M/R via its REST API
• Job is described in JSON
• Submitted via POST
• Def...
Erlang Phase
   (JSON)
Erlang Phase
             (JSON)
{Type:{“language”:”erlang”, “module”: Module,
Erlang Phase
             (JSON)
{Type:{“language”:”erlang”, “module”: Module,
       “function”: Function, “keep”:Flag}}
Erlang Phase
             (JSON)
{Type:{“language”:”erlang”, “module”: Module,
       “function”: Function, “keep”:Flag}}
Erlang Phase
                 (JSON)
    {Type:{“language”:”erlang”, “module”: Module,
           “function”: Function, “k...
Erlang Phase
                 (JSON)
    {Type:{“language”:”erlang”, “module”: Module,
           “function”: Function, “k...
Erlang Phase
                 (JSON)
    {Type:{“language”:”erlang”, “module”: Module,
           “function”: Function, “k...
Erlang Phase
                 (JSON)
    {Type:{“language”:”erlang”, “module”: Module,
           “function”: Function, “k...
Javascript Phase
    (JSON)
Javascript Phase
          (JSON)
{Type:{“language”:”javascript”,
Javascript Phase
          (JSON)
{Type:{“language”:”javascript”,
       “source”: Source,“keep”:Flag}}
Javascript Phase
          (JSON)
{Type:{“language”:”javascript”,
       “source”: Source,“keep”:Flag}}
Javascript Phase
              (JSON)
    {Type:{“language”:”javascript”,
           “source”: Source,“keep”:Flag}}


•   ...
Javascript Phase
              (JSON)
    {Type:{“language”:”javascript”,
           “source”: Source,“keep”:Flag}}


•   ...
Javascript Phase
              (JSON)
    {Type:{“language”:”javascript”,
           “source”: Source,“keep”:Flag}}


•   ...
Javascript Phase
    (JSON)
Javascript Phase
          (JSON)
{Type:{“language”:”javascript”,
Javascript Phase
          (JSON)
{Type:{“language”:”javascript”,
       “name”:Name,“keep”:Flag}}
Javascript Phase
          (JSON)
{Type:{“language”:”javascript”,
       “name”:Name,“keep”:Flag}}
Javascript Phase
              (JSON)
    {Type:{“language”:”javascript”,
           “name”:Name,“keep”:Flag}}


•   Type:...
Javascript Phase
              (JSON)
    {Type:{“language”:”javascript”,
           “name”:Name,“keep”:Flag}}


•   Type:...
Javascript Phase
              (JSON)
    {Type:{“language”:”javascript”,
           “name”:Name,“keep”:Flag}}


•   Type:...
Putting It
              Together

{“inputs”: [[“stocks”, “goog”]],

 “query”: [{“map”:{“language”:”javascript”,

        ...
Putting It
              Together

{“inputs”: [[“stocks”, “goog”],

            [“stocks”, “csco”]],

 “query”: [{“map”:{“...
Putting It
              Together
{“inputs”: “stocks”,

 “query”: [{“map”:{“language”:”javascript”,

                   “n...
Live Demo!
Thank You

     Kevin A. Smith
Email: ksmith@basho.com
  Twitter: @kevsmith
Upcoming SlideShare
Loading in...5
×

Introducing Riak

1,476

Published on

0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,476
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
67
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide









































































































































































  • Introducing Riak

    1. 1. Introducing Riak Kevin A. Smith Senior Developer Basho Technologies
    2. 2. What Is Riak?
    3. 3. What Is Riak? • Key/Value store
    4. 4. What Is Riak? • Key/Value store • Document-oriented database
    5. 5. What Is Riak? • Key/Value store • Document-oriented database • Web-shaped storage
    6. 6. Key/Value Store
    7. 7. Key/Value Store • Data organized by bucket/key pairs
    8. 8. Key/Value Store • Data organized by bucket/key pairs • Simple REST API (GET, PUT, DELETE)
    9. 9. Document Store
    10. 10. Document Store • Store values as JSON
    11. 11. Document Store • Store values as JSON • Many clients support automatic JSON encoding/decoding
    12. 12. Document Store • Store values as JSON • Many clients support automatic JSON encoding/decoding • Javascript Map/Reduce built on top of JSON
    13. 13. Web-Shaped Storage
    14. 14. Web-Shaped Storage • Content neutral
    15. 15. Web-Shaped Storage • Content neutral • Highly distributed
    16. 16. Web-Shaped Storage • Content neutral • Highly distributed • Replicated
    17. 17. Web-Shaped Storage • Content neutral • Highly distributed • Replicated • Fault-tolerant
    18. 18. What Is Riak?
    19. 19. What Is Riak? A flexible storage engine...
    20. 20. What Is Riak? A flexible storage engine... ...with a REST API...
    21. 21. What Is Riak? A flexible storage engine... ...with a REST API... ...and map/reduce capability...
    22. 22. What Is Riak? A flexible storage engine... ...with a REST API... ...and map/reduce capability... ....designed to be fault-tolerant...
    23. 23. What Is Riak? A flexible storage engine... ...with a REST API... ...and map/reduce capability... ....designed to be fault-tolerant... ...distributed...
    24. 24. What Is Riak? A flexible storage engine... ...with a REST API... ...and map/reduce capability... ....designed to be fault-tolerant... ...distributed... ...and ops friendly
    25. 25. Influences
    26. 26. Influences • CAP Theorem
    27. 27. Influences • CAP Theorem • Amazon’s Dynamo Paper
    28. 28. Influences • CAP Theorem • Amazon’s Dynamo Paper • Experience running large networks (Akamai)
    29. 29. CAP Theorem
    30. 30. CAP Theorem Consistent Reads and writes reflect a globally consistent system state
    31. 31. CAP Theorem Consistent Reads and writes reflect a globally consistent system state
    32. 32. CAP Theorem Consistent Reads and writes reflect a globally consistent system state Available System is available for reads and writes
    33. 33. CAP Theorem Consistent Reads and writes reflect a globally consistent system state Available System is available for reads and writes
    34. 34. CAP Theorem Consistent Reads and writes reflect a globally consistent system state Available System is available for reads and writes Partition Tolerant System can handle the failure of individual parts
    35. 35. Common Wisdom
    36. 36. Common Wisdom Pick two.
    37. 37. The Riak Way
    38. 38. The Riak Way Pick Two.
    39. 39. The Riak Way Pick Two. For each operation.
    40. 40. Dynamo Influences
    41. 41. Dynamo Influences • N = The number of replicas
    42. 42. Dynamo Influences • N = The number of replicas • R = The number of replicas needed for a successful read
    43. 43. Dynamo Influences • N = The number of replicas • R = The number of replicas needed for a successful read • W = The number of replicas needed for a successful write
    44. 44. Dynamo Math
    45. 45. Dynamo Math N - R = read fault tolerance
    46. 46. Dynamo Math N - R = read fault tolerance N - W = write fault tolerance
    47. 47. Dynamo Math
    48. 48. Dynamo Math N = 4, W = 2, R = 1
    49. 49. Dynamo Math N = 4, W = 2, R = 1
    50. 50. Dynamo Math N = 4, W = 2, R = 1 4 - 2 = 2 hosts can be down and Riak can still perform writes.
    51. 51. Dynamo Math N = 4, W = 2, R = 1 4 - 2 = 2 hosts can be down and Riak can still perform writes. 4 - 1 = 3 hosts can be down and Riak can still perform reads.
    52. 52. Riak Improvements
    53. 53. Riak Improvements • N can vary per bucket
    54. 54. Riak Improvements • N can vary per bucket • R and W can vary per operation
    55. 55. Riak Improvements • N can vary per bucket • R and W can vary per operation Choose your own fault tolerance/performance tradeoff
    56. 56. Consistent Hashing 2160 0 node 0 node 1 2160/4 node 2 node 3 hash(<<"artist">>,<<"REM">>) 2160/2
    57. 57. R value get(<<"artist">>,<<"REM">>, R=2) (N=3) {ok, Object} X
    58. 58. W value put(<<"artist">>,<<"REM">>, W=2) (N=3) ok X
    59. 59. N=10, R/W=2 get/put("artist", "REM", R/W=2) (N=10) {ok, Object} X X X X X X X X
    60. 60. Resolving Conflicts
    61. 61. Resolving Conflicts • Riak focuses on the AP of CAP
    62. 62. Resolving Conflicts • Riak focuses on the AP of CAP • Data could be briefly inconsistent
    63. 63. Resolving Conflicts • Riak focuses on the AP of CAP • Data could be briefly inconsistent • Inconsistency must be resolved
    64. 64. Detecting & Resolving Conflicts 0 1 Object v0 2 3
    65. 65. Detecting & Resolving Conflicts Object 0 1 v0 Object 2 3 v0
    66. 66. Detecting & Resolving Conflicts Object 0 1 v1 Object 2 3 v0
    67. 67. Detecting & Resolving Conflicts 0 1 Object v1 2 3
    68. 68. Detecting & Resolving Conflicts Object 0 1 v1 Object 2 3 v1
    69. 69. Client Resolution
    70. 70. Client Resolution • Can be set per-bucket or server-wide
    71. 71. Client Resolution • Can be set per-bucket or server-wide • Conflicting data is “bubbled up” to the client
    72. 72. Client Resolution • Can be set per-bucket or server-wide • Conflicting data is “bubbled up” to the client • Client picks the winner
    73. 73. Server Resolution
    74. 74. Server Resolution • “Last write wins”
    75. 75. Server Resolution • “Last write wins” • Enabled by default
    76. 76. Server Resolution • “Last write wins” • Enabled by default • What most apps need 80% of the time
    77. 77. Live Demo!
    78. 78. Linking Objects
    79. 79. Linking Objects • Objects can store pointers, or links, to other objects
    80. 80. Linking Objects • Objects can store pointers, or links, to other objects • Doesn’t have to be the same bucket
    81. 81. Linking Objects • Objects can store pointers, or links, to other objects • Doesn’t have to be the same bucket • Object links described in a Link header
    82. 82. Link Header Format Object URL </riak/demo/test1>; riaktag="userinfo" Link tag
    83. 83. Link Walking
    84. 84. Link Walking • Ask Riak to “walk” a sequence of links
    85. 85. Link Walking • Ask Riak to “walk” a sequence of links • Optionally, collect objects along the walk and return them
    86. 86. Link Walking • Ask Riak to “walk” a sequence of links • Optionally, collect objects along the walk and return them • Can be arbitrarily deep
    87. 87. Link Walking Examples
    88. 88. Link Walking Examples /riak/demo/test1/_,_,1
    89. 89. Link Walking Examples /riak/demo/test1/_,_,1 Start walking at /demo/test1 and return all linked objects
    90. 90. Link Walking Examples
    91. 91. Link Walking Examples /riak/demo/test1/demo,_,1
    92. 92. Link Walking Examples /riak/demo/test1/demo,_,1 Start walking at /demo/test1 and return all linked objects contained in the demo bucket
    93. 93. Link Walking Examples
    94. 94. Link Walking Examples /riak/demo/test1/_,_,0/_,_,1
    95. 95. Link Walking Examples /riak/demo/test1/_,_,0/_,_,1 Start walking at /demo/test1, find any linked objects, then find and return any objects linked to those
    96. 96. Link Walking Examples
    97. 97. Link Walking Examples /riak/demo/test1/_,child,0/_,_,1
    98. 98. Link Walking Examples /riak/demo/test1/_,child,0/_,_,1 Start walking at /demo/test1, find any linked objects with the link tag “child”, then find and return any objects linked to those
    99. 99. Map/Reduce Terms
    100. 100. Map/Reduce Terms • Phase: A step within a job
    101. 101. Map/Reduce Terms • Phase: A step within a job • Job: A sequence of phases and inputs
    102. 102. Map/Reduce Terms • Phase: A step within a job • Job: A sequence of phases and inputs • Map: Data collection phase
    103. 103. Map/Reduce Terms • Phase: A step within a job • Job: A sequence of phases and inputs • Map: Data collection phase • Reduce: Data collation or processing phase
    104. 104. Map/Reduce Overview
    105. 105. Map/Reduce Overview • Map phases execute in parallel w/data locality
    106. 106. Map/Reduce Overview • Map phases execute in parallel w/data locality • Reduce phases execute in parallel on the node where job was submitted
    107. 107. Map/Reduce Overview • Map phases execute in parallel w/data locality • Reduce phases execute in parallel on the node where job was submitted • Results are not cached or stored
    108. 108. Map/Reduce Overview • Map phases execute in parallel w/data locality • Reduce phases execute in parallel on the node where job was submitted • Results are not cached or stored • Phases can be written in Erlang or Javascript
    109. 109. Map Phase
    110. 110. Map Phase • Inputs must be bucket/key pairs
    111. 111. Map Phase • Inputs must be bucket/key pairs • Must return a list
    112. 112. Map Phase • Inputs must be bucket/key pairs • Must return a list • Parallel results are aggregated into a single list
    113. 113. Parallel Map
    114. 114. Parallel Map
    115. 115. Parallel Map
    116. 116. Parallel Map
    117. 117. Erlang Map Phase
    118. 118. Erlang Map Phase • Two types: modfun and qfun
    119. 119. Erlang Map Phase • Two types: modfun and qfun • modfuns reference the module and name of the Erlang function to call
    120. 120. Erlang Map Phase • Two types: modfun and qfun • modfuns reference the module and name of the Erlang function to call • qfuns are anonymous Erlang functions*
    121. 121. Erlang Map Phase • Two types: modfun and qfun • modfuns reference the module and name of the Erlang function to call • qfuns are anonymous Erlang functions* *Must be on the server-side codepath
    122. 122. Erlang Map Phase
    123. 123. Erlang Map Phase map_object_value(Obj, _KeyData, _Arg) ->
    124. 124. Erlang Map Phase map_object_value(Obj, _KeyData, _Arg) -> [riak_object:get_value(Obj)].
    125. 125. Erlang Map Phase map_object_value(Obj, _KeyData, _Arg) -> [riak_object:get_value(Obj)].
    126. 126. Erlang Map Phase map_object_value(Obj, _KeyData, _Arg) -> [riak_object:get_value(Obj)]. • Obj:riak_object retrieved from bucket/key
    127. 127. Erlang Map Phase map_object_value(Obj, _KeyData, _Arg) -> [riak_object:get_value(Obj)]. • Obj:riak_object retrieved from bucket/key • KeyData: Static argument specified with the bucket/ key
    128. 128. Erlang Map Phase map_object_value(Obj, _KeyData, _Arg) -> [riak_object:get_value(Obj)]. • Obj:riak_object retrieved from bucket/key • KeyData: Static argument specified with the bucket/ key • Arg: Static argument specified with the phase
    129. 129. Erlang Map Built-Ins
    130. 130. Erlang Map Built-Ins riak_mapreduce:map_object_value/3
    131. 131. Erlang Map Built-Ins riak_mapreduce:map_object_value/3 • Returns object value wrapped in a list
    132. 132. Erlang Map Built-Ins riak_mapreduce:map_object_value/3 • Returns object value wrapped in a list riak_mapreduce:map_object_value_list/3
    133. 133. Erlang Map Built-Ins riak_mapreduce:map_object_value/3 • Returns object value wrapped in a list riak_mapreduce:map_object_value_list/3 • Returns object value. Object value must already be a list
    134. 134. Javascript Map Phase
    135. 135. Javascript Map Phase • Two types: jsanon and jsfun
    136. 136. Javascript Map Phase • Two types: jsanon and jsfun • jsanons are anonymous JS functions:
    137. 137. Javascript Map Phase • Two types: jsanon and jsfun • jsanons are anonymous JS functions: function(value) { return [value]; }
    138. 138. Javascript Map Phase • Two types: jsanon and jsfun • jsanons are anonymous JS functions: function(value) { return [value]; } • jsfuns are named JS functions:
    139. 139. Javascript Map Phase • Two types: jsanon and jsfun • jsanons are anonymous JS functions: function(value) { return [value]; } • jsfuns are named JS functions: Riak.mapValuesJson
    140. 140. Erlang & Javascript
    141. 141. Erlang & Javascript • Same environment as Firefox minus browser bits
    142. 142. Erlang & Javascript • Same environment as Firefox minus browser bits • Erlang to Javascript data is JSON encoded
    143. 143. Erlang & Javascript • Same environment as Firefox minus browser bits • Erlang to Javascript data is JSON encoded • Javascript to Erlang data is JSON decoded
    144. 144. Javascript Map Phase
    145. 145. Javascript Map Phase function(value, keyData, arg)
    146. 146. Javascript Map Phase function(value, keyData, arg)
    147. 147. Javascript Map Phase function(value, keyData, arg) • value: JSON-encoded version of riak_object
    148. 148. Javascript Map Phase function(value, keyData, arg) • value: JSON-encoded version of riak_object • keyData: Same as Erlang
    149. 149. Javascript Map Phase function(value, keyData, arg) • value: JSON-encoded version of riak_object • keyData: Same as Erlang • arg: Same as Erlang
    150. 150. Javascript Map Built-Ins
    151. 151. Javascript Map Built-Ins Riak.mapValues
    152. 152. Javascript Map Built-Ins Riak.mapValues • Returns object values. Handles detecting when/if to use list wrapping.
    153. 153. Javascript Map Built-Ins Riak.mapValues • Returns object values. Handles detecting when/if to use list wrapping. Riak.mapValuesJson
    154. 154. Javascript Map Built-Ins Riak.mapValues • Returns object values. Handles detecting when/if to use list wrapping. Riak.mapValuesJson • Returns JSON parsed object values. Also performs list wrapping, if needed.
    155. 155. Reduce Phase
    156. 156. Reduce Phase • Performed on the node coordinating the map/reduce job
    157. 157. Reduce Phase • Performed on the node coordinating the map/reduce job • Two processes per reduce phase to add minor parallelism
    158. 158. Reduce Phase • Performed on the node coordinating the map/reduce job • Two processes per reduce phase to add minor parallelism • Must return a list
    159. 159. Erlang Reduce Built-Ins
    160. 160. Erlang Reduce Built-Ins riak_mapreduce:reduce_set_union/2
    161. 161. Erlang Reduce Built-Ins riak_mapreduce:reduce_set_union/2 • Returns unique set of values
    162. 162. Erlang Reduce Built-Ins riak_mapreduce:reduce_set_union/2 • Returns unique set of values riak_mapreduce:reduce_sum/2
    163. 163. Erlang Reduce Built-Ins riak_mapreduce:reduce_set_union/2 • Returns unique set of values riak_mapreduce:reduce_sum/2 • Returns the sum of inputs
    164. 164. Erlang Reduce Built-Ins riak_mapreduce:reduce_set_union/2 • Returns unique set of values riak_mapreduce:reduce_sum/2 • Returns the sum of inputs riak_mapreduce:reduce_sort/2
    165. 165. Erlang Reduce Built-Ins riak_mapreduce:reduce_set_union/2 • Returns unique set of values riak_mapreduce:reduce_sum/2 • Returns the sum of inputs riak_mapreduce:reduce_sort/2 • Returns the sorted list of inputs
    166. 166. Javascript Reduce Built-Ins
    167. 167. Javascript Reduce Built-Ins Riak.reduceMin
    168. 168. Javascript Reduce Built-Ins Riak.reduceMin • Returns the minimum value of the input set
    169. 169. Javascript Reduce Built-Ins Riak.reduceMin • Returns the minimum value of the input set Riak.reduceMax
    170. 170. Javascript Reduce Built-Ins Riak.reduceMin • Returns the minimum value of the input set Riak.reduceMax • Returns the maximum value of the input set
    171. 171. Javascript Reduce Built-Ins Riak.reduceMin • Returns the minimum value of the input set Riak.reduceMax • Returns the maximum value of the input set Riak.reduceSort
    172. 172. Javascript Reduce Built-Ins Riak.reduceMin • Returns the minimum value of the input set Riak.reduceMax • Returns the maximum value of the input set Riak.reduceSort • Returns a sorted list of the input set
    173. 173. Building M/R Job
    174. 174. Building M/R Job • Job is a list of phases and starting inputs
    175. 175. Building M/R Job • Job is a list of phases and starting inputs • Each phase can:
    176. 176. Building M/R Job • Job is a list of phases and starting inputs • Each phase can: • Receive a static argument
    177. 177. Building M/R Job • Job is a list of phases and starting inputs • Each phase can: • Receive a static argument • Accumulate and return results
    178. 178. Submitting Jobs via HTTP
    179. 179. Submitting Jobs via HTTP • Riak exposes M/R via its REST API
    180. 180. Submitting Jobs via HTTP • Riak exposes M/R via its REST API • Job is described in JSON
    181. 181. Submitting Jobs via HTTP • Riak exposes M/R via its REST API • Job is described in JSON • Submitted via POST
    182. 182. Submitting Jobs via HTTP • Riak exposes M/R via its REST API • Job is described in JSON • Submitted via POST • Default URL is /mapred
    183. 183. Erlang Phase (JSON)
    184. 184. Erlang Phase (JSON) {Type:{“language”:”erlang”, “module”: Module,
    185. 185. Erlang Phase (JSON) {Type:{“language”:”erlang”, “module”: Module, “function”: Function, “keep”:Flag}}
    186. 186. Erlang Phase (JSON) {Type:{“language”:”erlang”, “module”: Module, “function”: Function, “keep”:Flag}}
    187. 187. Erlang Phase (JSON) {Type:{“language”:”erlang”, “module”: Module, “function”: Function, “keep”:Flag}} • Type: “map” or “reduce”
    188. 188. Erlang Phase (JSON) {Type:{“language”:”erlang”, “module”: Module, “function”: Function, “keep”:Flag}} • Type: “map” or “reduce” • Module: String name of Erlang module
    189. 189. Erlang Phase (JSON) {Type:{“language”:”erlang”, “module”: Module, “function”: Function, “keep”:Flag}} • Type: “map” or “reduce” • Module: String name of Erlang module • Function: String name of Erlang function
    190. 190. Erlang Phase (JSON) {Type:{“language”:”erlang”, “module”: Module, “function”: Function, “keep”:Flag}} • Type: “map” or “reduce” • Module: String name of Erlang module • Function: String name of Erlang function • Flag: Boolean accumulation toggle
    191. 191. Javascript Phase (JSON)
    192. 192. Javascript Phase (JSON) {Type:{“language”:”javascript”,
    193. 193. Javascript Phase (JSON) {Type:{“language”:”javascript”, “source”: Source,“keep”:Flag}}
    194. 194. Javascript Phase (JSON) {Type:{“language”:”javascript”, “source”: Source,“keep”:Flag}}
    195. 195. Javascript Phase (JSON) {Type:{“language”:”javascript”, “source”: Source,“keep”:Flag}} • Type: “map” or “reduce”
    196. 196. Javascript Phase (JSON) {Type:{“language”:”javascript”, “source”: Source,“keep”:Flag}} • Type: “map” or “reduce” • Source: Source for anonymous function
    197. 197. Javascript Phase (JSON) {Type:{“language”:”javascript”, “source”: Source,“keep”:Flag}} • Type: “map” or “reduce” • Source: Source for anonymous function • Flag: Boolean accumulation toggle
    198. 198. Javascript Phase (JSON)
    199. 199. Javascript Phase (JSON) {Type:{“language”:”javascript”,
    200. 200. Javascript Phase (JSON) {Type:{“language”:”javascript”, “name”:Name,“keep”:Flag}}
    201. 201. Javascript Phase (JSON) {Type:{“language”:”javascript”, “name”:Name,“keep”:Flag}}
    202. 202. Javascript Phase (JSON) {Type:{“language”:”javascript”, “name”:Name,“keep”:Flag}} • Type: “map” or “reduce”
    203. 203. Javascript Phase (JSON) {Type:{“language”:”javascript”, “name”:Name,“keep”:Flag}} • Type: “map” or “reduce” • Name: String name of Javascript function
    204. 204. Javascript Phase (JSON) {Type:{“language”:”javascript”, “name”:Name,“keep”:Flag}} • Type: “map” or “reduce” • Name: String name of Javascript function • Flag: Boolean accumulation toggle
    205. 205. Putting It Together {“inputs”: [[“stocks”, “goog”]], “query”: [{“map”:{“language”:”javascript”, “name”: “Riak.mapValuesJson”}, “keep”: true}]}
    206. 206. Putting It Together {“inputs”: [[“stocks”, “goog”], [“stocks”, “csco”]], “query”: [{“map”:{“language”:”javascript”, “name”: “Riak.mapValuesJson”}, “keep”: true}]}
    207. 207. Putting It Together {“inputs”: “stocks”, “query”: [{“map”:{“language”:”javascript”, “name”: “App.extractTickers”, “arg”: “GOOG”}, “keep”: false}, {“reduce”:{“language”:”javascript, “name”: “Riak.reduceMin”}, “keep”: true}]}
    208. 208. Live Demo!
    209. 209. Thank You Kevin A. Smith Email: ksmith@basho.com Twitter: @kevsmith
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×