riak
            A friendly key/value store for the web.



       ION
 EV NAT

     010D
D
    2 N TLA
    POR


        ...
D PO
                          EV R
                            N TLA
                              A N
                  ...
D PO
                         EV R
                           N TLA
                             A N
                     ...
D PO
                             EV R
                               N TLA
                                 A N
         ...
D PO
                                                                          EV R
                                      ...
D PO
                              EV R
                                N TLA
                                  A N
      ...
You may find the following
 conspicuously missing in
        this talk:
                   r y!
               o r
       ...
D PO
                                    EV R
                                      N TLA
     I will not be




         ...
D PO
                                   EV R
                                     N TLA
                                  ...
D PO
                               EV R
                                 N TLA
                                   A N
   ...
D PO
                                 EV R
                                   N TLA
                                     A...
What is Riak?
D PO
                      EV R
                        N TLA
                          A N
                           TI ...
D PO
                           EV R
                             N TLA
                               A N
               ...
D PO
                             EV R
                               N TLA
                                 A N
         ...
D PO
                              EV R
                                N TLA
                                  A N
      ...
D PO
                                 EV R
                                   N TLA
                                     A...
D PO
                                 EV R
                                   N TLA
                                     A...
D PO
                                EV R
                                  N TLA
                                    A N
...
D PO
                          EV R
                            N TLA
                              A N
                  ...
D PO
                           EV R
                             N TLA
                               A N
               ...
D PO
                         EV R
                           N TLA
                             A N
                     ...
D PO
                     EV R
                       N TLA
                         A N
                          TI D
  ...
Getting
Up & Running
N
        O
      TI D
     A N
   N TLA
              http://riak.basho.com




 EV R
D PO
N
        O
      TI D
     A N
   N TLA
 EV R
D PO
              hg & git
D PO
                                                   EV R
A Quick Local Cluster




                                   ...
Your Data
D PO
                                 EV R
       Object




                                   N TLA
                    ...
D PO
                                      EV R
                         ca
           Key




                           ...
D PO
                                   EV R
        Bucket




                                     N TLA
               ...
D PO
                                     EV R
    Addressability




                                       N TLA
       ...
D PO
                                                     EV R
        Example




                                       ...
D PO
                                                       EV R
         Example




                                    ...
D PO
                                    EV R
            Links




                                      N TLA
          ...
D PO
                                                           EV R
                Example




                         ...
Hooks

pre-commit
reject or transform an object to be committed

post-commit
notify external services, build your own inde...
Where does it go?
D PO
                           EV R
    The Ring




                             N TLA
                               A ...
D PO
                                  EV R
          The Ring




                                    N TLA
             ...
N
        O
      TI D
     A N
   N TLA
 EV R
D PO
                                                              st more ...
D PO
                              EV R
     The Ring




                                N TLA
                          ...
D PO
                       EV R
  The Ring




                         N TLA
                           A N
            ...
D PO
                                EV R
        The Ring




                                  N TLA
                   ...
D PO
                                     EV R
     Replication




                                       N TLA
         ...
,
                                  ils
          N
        O
      TI D
     A N
   N TLA                       fa
 EV R ...
D PO
                                                EV R
    Persistence




                                            ...
CAP Tuning
D PO
                                                EV R
                    GET




                                    ...
D PO
                                                 EV R
                     PUT




                                  ...
(Map|Link)*Reduce
D PO
                                          EV R
              Map




                                            N TL...
Data-locality is important in Riak.
Map phases are run where the data is
               stored.

 You can have multiple ma...
D PO
                                             EV R
              Link




                                            ...
D PO
                                             EV R
                 Reduce




                                       ...
The reduce phase occurs on the
      “coordinating node.”

Reduces may be run multiple times
  as more input comes in (eg,...
D PO
                                                  EV R
         Example




                                         ...
D PO
                                                                 EV R
                Example




                   ...
Hurdles
D PO
                                         EV R
                                           N TLA
 No range queries.



...
D PO
                                       EV R
                                         N TLA
                          ...
D PO
                                      EV R
                                        N TLA
                            ...
sy
a
E Questions?
N
        O
      TI D
     A N
   N TLA
 EV R
D PO
                        @wbruce
              Thanks!
Upcoming SlideShare
Loading in...5
×

Riak: A friendly key/value store for the web.

13,242

Published on

Talk at DevNation Portland, July 10th, 2010.

Keynote available for download at:
http://dl.dropbox.com/u/458036/presentations/DevNation%20Portland.zip

Sources and additional information in the presenter notes.

Published in: Technology
0 Comments
17 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
13,242
On Slideshare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
156
Comments
0
Likes
17
Embeds 0
No embeds

No notes for slide

Riak: A friendly key/value store for the web.

  1. 1. riak A friendly key/value store for the web. ION EV NAT 010D D 2 N TLA POR A primer by Bruce Williams
  2. 2. D PO EV R N TLA A N TI D O N My name is Bruce Williams. ct ed di g ad din I’m lee a nd e b th e. to e dg
  3. 3. D PO EV R N TLA A N TI D O N 2001 - Present Day wa yyy before it was a viable job choice.
  4. 4. D PO EV R N TLA A N TI D O N But I use other languages, too. rom . y f ms all ig ci ad es pe ar rp o the
  5. 5. D PO EV R N TLA A N TI D O N Photo by oddsteph - http://flic.kr/p/6vWPBU me su of as e Le t’s on ll a is ba J av base t he ats. b Choose the Right Weapon
  6. 6. D PO EV R N TLA A N TI D O N Based in the D.C. area. (but I’m not.)
  7. 7. You may find the following conspicuously missing in this talk: r y! o r S
  8. 8. D PO EV R N TLA I will not be A N TI D O N presenting a paper on Dynamo, the CAP theorem, vector clocks, merkle trees, etc. These are explained elsewhere by my alg orithmic betters.
  9. 9. D PO EV R N TLA A N TI D O N I will not be dwelling on performance or redundancy. Expect some vague statements like “very fa st” and “very robust.”
  10. 10. D PO EV R N TLA A N TI D O N I will not try to convince you that “NoSQL” is the messiah. I t’s an alternative that m akes sense in some situations.
  11. 11. D PO EV R N TLA A N TI D O N I will not be conducting a large-scale comparison of competing technologies. b ut I’d love to hear abou t what you use, and why
  12. 12. What is Riak?
  13. 13. D PO EV R N TLA A N TI D O N NoSQL and of the Dynamo persuasion.
  14. 14. D PO EV R N TLA A N TI D O N Open Source & a commercial “EnterpriseDS” version with some proprietary pieces
  15. 15. D PO EV R N TLA A N TI D O N Key/Value Store With some metadata.
  16. 16. D PO EV R N TLA A N TI D O N Schema-less Great for sparse data, but requires more discipline.
  17. 17. D PO EV R N TLA A N TI D O N Datatype Agnostic Con tent-Type is King.
  18. 18. D PO EV R N TLA A N TI D O N Language Agnostic REST & PBC Erlang, Javascript, Java, PHP, Python, Ruby, ...
  19. 19. D PO EV R N TLA A N TI D O N Distributed It’s [mostly] Erlang, what did you expect?
  20. 20. D PO EV R N TLA A N TI D O N Masterless All nodes are equal
  21. 21. D PO EV R N TLA A N TI D O N Scalable o r “easy to scale.”
  22. 22. D PO EV R N TLA A N TI D O N Eventually Consistent and CAP tunable.
  23. 23. D PO EV R N TLA A N TI D O N Uses Map/Reduce and “Link.”
  24. 24. Getting Up & Running
  25. 25. N O TI D A N N TLA http://riak.basho.com EV R D PO
  26. 26. N O TI D A N N TLA EV R D PO hg & git
  27. 27. D PO EV R A Quick Local Cluster N TLA A N TI D O N $ ./riak1/bin/riak start $ ./riak2/bin/riak start $ ./riak3/bin/riak start Start three “nodes” $ ./riak2/bin/riak-admin join riak1@127.0.0.1 $ ./riak3/bin/riak-admin join riak1@127.0.0.1 Join them in to a cluster
  28. 28. Your Data
  29. 29. D PO EV R Object N TLA A N TI D O N Content Type Body + Links The thing you’re storing.
  30. 30. D PO EV R ca Key N TLA n A N TI D de be O N au fi to ne use ge ma d o r- ne tic r ra a te lly d pic1 The identifier for the object.
  31. 31. D PO EV R Bucket N TLA A N TI D O N “p thin ic wi 1” “im is ag un es iq ” pic1 ue pic2 pic3 images The type or category of object.
  32. 32. D PO EV R Addressability N TLA A N TI D O N <i ma ge images s/ pi c1 > pic1 Refer to objects by bucket and key.
  33. 33. D PO EV R Example N TLA A N TI D O N require 'riak' client = Riak::Client.new client.bucket('images').new('pic1').tap do |pic1| pic1.content_type = 'image/jpeg' pic1.data = File.read('/path/to/jpg') pic1.store end $g em install riak-client
  34. 34. D PO EV R Example N TLA A N TI D O N client.bucket('people').new('bruce').tap do |bruce| bruce.data = { name: 'Bruce Williams', email: 'bruce@codefluency.com' } bruce.store end puts client['people']['bruce'].data['name'] “application/json” is the d efault for riak-client
  35. 35. D PO EV R Links N TLA A N TI D O N st or ed images people he re pic1 bruce can also be “tagged” Connect objects
  36. 36. D PO EV R Example N TLA A N TI D O N client['people']['bruce'].tap do |bruce| bruce.links << client['images']['pic1'].to_link('avatar') bruce.store end client['people']['bruce'].walk(:tag => 'avatar')
  37. 37. Hooks pre-commit reject or transform an object to be committed post-commit notify external services, build your own indexe
  38. 38. Where does it go?
  39. 39. D PO EV R The Ring N TLA A N TI D O N A 160-bit integer space
  40. 40. D PO EV R The Ring N TLA A N TI D O N broken into equal sized partitions.
  41. 41. N O TI D A N N TLA EV R D PO st more functional) looks kinda like this The Ring (it’s ju It Photo by marchdoe - http://flickr.com/photos/marchdoe/457741149
  42. 42. D PO EV R The Ring N TLA A N TI D O N Each partition is managed by a vnode (virtual node),
  43. 43. D PO EV R The Ring N TLA A N TI D O N Each vnode runs on a [physical] node.
  44. 44. D PO EV R The Ring N TLA A N TI D O N 1 2 3 4 Each node owns an equal share of vnodes (& partitions)
  45. 45. D PO EV R Replication N TLA A N TI D O N 3 is th e de fa ult n_val = 3 Objects are written to multiple partitions.
  46. 46. , ils N O TI D A N N TLA fa EV R ” up “2 ck Uses Hinted Handoff to deal with D PO de pi k. no s n er lac he th s W e o the th Availability node failures. 4 2 3 1
  47. 47. D PO EV R Persistence N TLA A N TI D O N dets ets fs gb_trees innostore bitcask multi + Supports pluggable backends
  48. 48. CAP Tuning
  49. 49. D PO EV R GET N TLA A N TI D O N r how many replicas need to agree (default: 2)
  50. 50. D PO EV R PUT N TLA A N TI D O N r how many replicas need to agree when retrieving an existing object before the write (default: 2) w how many replicas to write to before returning a successful response (default: 2). dw how many replicas to commit to durable storage before returning a successful response (default: 0)
  51. 51. (Map|Link)*Reduce
  52. 52. D PO EV R Map N TLA A N TI D O N obj [result, ...] your function Map functions take one piece of data as input, and produce zero or more results as output.
  53. 53. Data-locality is important in Riak. Map phases are run where the data is stored. You can have multiple map phases. The input to a map definition is a series of [bucket, key] names. unlike CouchDB
  54. 54. D PO EV R Link N TLA A N TI D O N obj [linked_obj, ...] link walk, using a pattern A special kind of map phase; links matching a pattern are “walked” to find objects to be output.
  55. 55. D PO EV R Reduce N TLA A N TI D O N [obj, ...] [result] your function Reduce functions combine the output of many "map" step evaluations, into one result
  56. 56. The reduce phase occurs on the “coordinating node.” Reduces may be run multiple times as more input comes in (eg, re- reduce)
  57. 57. D PO EV R Example N TLA A N TI D O N bruce = client['people']['bruce'] melissa = client['people']['melissa'] lets assume these have ages addy = client['addresses'].new('123fake') addy.data = { street: '123 Fake St', city: 'Portland', state: 'OR', zip: '97214' } addy.links << bruce.to_link('resident') addy.links << melissa.to_link('resident') addy.store
  58. 58. D PO EV R Example N TLA A N TI D O N Riak::MapReduce.new(client).add(addy). link(tag: 'resident'). map("function (v) { return [Riak.mapValuesJson(v)[0]['age'] || 0] }"). reduce(function: 'Riak.reduceSum', keep: true). run We should get an array with one value
  59. 59. Hurdles
  60. 60. D PO EV R N TLA No range queries. A N TI D O N Sorry, Cassandra fans Things like time series data require creative approaches. like bucket and key naming, etc
  61. 61. D PO EV R N TLA A N Don’t list keys. TI D O N ever, if you can avoid it. Processing an entire bucket is more expensive than you might think. because it lists keys
  62. 62. D PO EV R N TLA A N TI D O N Watch your encoding. MapReduce Javascript phases need your data to be in valid Unicode. you’ll get a “bad encoding” error
  63. 63. sy a E Questions?
  64. 64. N O TI D A N N TLA EV R D PO @wbruce Thanks!
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×