OverviewMårten Gustafson   Stockholm Riak Meetup #1          2011-03-31
What is Riak         and what’s the agenda?Decentralized key-value storeA database ideally suited for web applicationsA fle...
What is Riak         and what’s the agenda?Decentralized key-value storeA database ideally suited for web applicationsA fle...
http://www.flickr.com/photos/linneberg/4481309196/
Operations              HTTP Client                                              PUT    /bucket/key                       ...
An entry• lives in a bucket• has a key• has a value
An entry                 arbitrary name• lives in a bucket             bucket• has a key• has a value
An entry                 arbitrary name• lives in a bucket             bucket• has a key arbitrary name         key• has a...
An entry                 arbitrary name                                  forms the path• lives in a bucket             buc...
An entry                  arbitrary name                                   forms the path• lives in a bucket             b...
An entry                  arbitrary name                                        forms the path• lives in a bucket         ...
Store anything   bucket meetup
Store anything                       bucket meetupkey foo.html mime text/html         <html><body>              <h1>   <a ...
Store anything                       bucket meetupkey foo.html mime text/html       key bar.html         <html><body>     ...
Store anything                       bucket meetupkey foo.html mime text/html              key bar.html         <html><bod...
What is Riak         and what’s the agenda?Decentralized key-value storeA database ideally suited for web applicationsA fle...
What is Riak         and what’s the agenda?Decentralized key-value storeA database ideally suited for web applicationsA fle...
StorageRiak Key/Value Store
Storage                 Riak Key/Value StoreBitcask InnoDB DETS File system   Balanced trees   LRU ETS
Storage                 Riak Key/Value StoreBitcask InnoDB DETS File system   Balanced trees   LRU ETS             disk ba...
Storagedefault                           Riak Key/Value Store          Bitcask InnoDB DETS File system   Balanced trees   ...
What is Riak         and what’s the agenda?Decentralized key-value storeA database ideally suited for web applicationsA fle...
What is Riak         and what’s the agenda?Decentralized key-value storeA database ideally suited for web applicationsA fle...
The Ring
The Ring
The Ring            12     1     11                    210                             39                              4  ...
Consistent Hashing            12   1       11            2  10                     3   9                     4       8    ...
Consistent Hashing            12   1       11            2  10                     3   9                     4       8    ...
Read“I want “requires us to know:where is on the ring?
Read“I want “requires us to know:where is on the ring?
ClusterOne Ring size to rule them all, One Ring size tofind them, One Ring size to bring them all and inthe cluster bind th...
Cluster                                            node A                                            node B               ...
Cluster                                            node A                                            node B               ...
Cluster - Readnode A   node B   node C
Cluster - ReadI can haz     ?                        node A   node B   node CHm,       hashes to aslice of the ringowned b...
Cluster - ReadI can haz   ?                     node A   node B   node C                                                Ok...
Cluster - ReadI can haz   ?                node A   node B   node C
So what about...
...network partitions?  node A   node B   node C
...network partitions?  node A   node B   X   node C
...failed nodes?node A   node B   node C
...failed nodes?node A   node B
...concurrent writes? node A     node B   node C client 1            client 2
...concurrent writes? node A                       ?                      node B   node C                 hey, A!         ...
What is Riak         and what’s the agenda?Decentralized key-value storeA database ideally suited for web applicationsA fle...
Livin’ on the web• concurrent users• unreliable networks• node failures
The N to the R to the W to the DW and the RW
N_VALClient          BucketNode A    Node B   Node C
N_VAL 1Client          BucketNode A    Node B   Node C
N_VAL 2Client          BucketNode A    Node B   Node C
N_VAL 3Client          BucketNode A    Node B   Node C
RClient   GET /bucket/key?R=1          BucketNode A    Node B   Node C
RClient   GET /bucket/key?R=1          BucketNode A    Node B   Node C
RClient   GET /bucket/key?R=2          BucketNode A    Node B   Node C
RClient   GET /bucket/key?R=2          BucketNode A    Node B   Node C
RClient   GET /bucket/key?R=2         Agree?           BucketNode A    Node B    Node C
RClient   GET /bucket/key?R=2          BucketNode A    Node B   Node C
The N to the R to the W to the DW and the RW
The N to the R to the W to the DW and the RWNumber of copiesie. distribute to N nodes
Read      ie. have R nodes agree           The N to the R to the W to the DW and the RWNumber of copiesie. distribute to N...
Read      ie. have R nodes agree           The N to the R to the W to the DW and the RW                               Writ...
Durable write      Read                                    ie. persistently written by      ie. have R nodes agree        ...
Durable write      Read                                    ie. persistently written by      ie. have R nodes agree        ...
The Quorum([node count] / 2) + 1
Bucket propertieshttp://127.0.0.1:8088/riak/meetup/
Conflicts
When worlds collide
I don’t care! allow_mult = falselast_write_wins = true
I do care!allow_mult = true
I do care!•   Resolve conflicts in application logic•   Conflicts exposed as siblings beneath a key•   Response is HTTP 300 ...
ExampleHTTP/1.1 300 Multiple ChoicesX-Riak-Vclock: a85hYGBgzWDKBVIsrLnh3BlMiYx5rAymfeeO8EGFWRLl30G==Content-Type: multipar...
What is Riak         and what’s the agenda?Decentralized key-value storeA database ideally suited for web applicationsA fle...
Map / Reduce• Javascript or Erlang• Exposed in the HTTP API
Map / Reduce      count words     function(v) {    var words = v.values[0].data.toLowerCase().match(w*,g);    var counts =...
Map / Reduce      count wordsfunction(values) {  var result = {};  for (var value in values) {    for(var word in values[v...
Map & Reduce          count words Put this in your POST request and let Riak smoke it{"inputs":"bucket", "query":[{"map":{...
Map & Reduce          count wordsfunction(v) {  var words = v.values[0].data.toLowerCase().match(w*,g);  var counts = []; ...
Map & Reduce          count wordsfunction(values) {  var result = {};  for (var value in values) {    for(var word in valu...
Map / Reduce  Demo
Links• Non-enforced• Traversable
LinksDemo
The whole enchilada     HTTP API             Riak Search                                 Luwak                     Riak Ke...
Try it          downloads.basho.com            brew install riakWeb admin @ github.com/gmaurice/Riaktive
ResourcesRiak Fast Track @ wiki.basho.com       #riak @ freenode       github.com/basho/
Thanks for listeningMårten Gustafson@martengustafsonhttp://marten.gustafson.pp.se/marten.gustafson@gmail.com
Upcoming SlideShare
Loading in...5
×

2011 03-31 Riak Stockholm Meetup

996

Published on

Published in: Sports
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
996
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
14
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

2011 03-31 Riak Stockholm Meetup

  1. 1. OverviewMårten Gustafson Stockholm Riak Meetup #1 2011-03-31
  2. 2. What is Riak and what’s the agenda?Decentralized key-value storeA database ideally suited for web applicationsA flexible map/reduce engine
  3. 3. What is Riak and what’s the agenda?Decentralized key-value storeA database ideally suited for web applicationsA flexible map/reduce engine
  4. 4. http://www.flickr.com/photos/linneberg/4481309196/
  5. 5. Operations HTTP Client PUT /bucket/key GET /bucket/key DELETE /bucket/key Item ItemItem Item Item Item Item Item Bucket Bucket Bucket Riak Key/Value Store
  6. 6. An entry• lives in a bucket• has a key• has a value
  7. 7. An entry arbitrary name• lives in a bucket bucket• has a key• has a value
  8. 8. An entry arbitrary name• lives in a bucket bucket• has a key arbitrary name key• has a value
  9. 9. An entry arbitrary name forms the path• lives in a bucket bucket to the value• has a key arbitrary name key• has a value
  10. 10. An entry arbitrary name forms the path• lives in a bucket bucket to the value• has a key arbitrary name key• has a value value a binary blob and mime type
  11. 11. An entry arbitrary name forms the path• lives in a bucket bucket to the value• has a key arbitrary name key• has a value value a binary blob and mime type = Store anything, yay!
  12. 12. Store anything bucket meetup
  13. 13. Store anything bucket meetupkey foo.html mime text/html <html><body> <h1> <a href=”bar.html”>foo</a> </h1> </body></html>
  14. 14. Store anything bucket meetupkey foo.html mime text/html key bar.html <html><body> mime text/html <h1> <a href=”bar.html”>foo</a> <html><body> </h1> <h1>bar!</h1> </body></html> </body></html>
  15. 15. Store anything bucket meetupkey foo.html mime text/html key bar.html <html><body> mime text/html <h1> <a href=”bar.html”>foo</a> <html><body> </h1> <h1>bar!</h1> </body></html> </body></html> http://127.0.0.1:8088/riak/meetup/foo.html
  16. 16. What is Riak and what’s the agenda?Decentralized key-value storeA database ideally suited for web applicationsA flexible map/reduce engine
  17. 17. What is Riak and what’s the agenda?Decentralized key-value storeA database ideally suited for web applicationsA flexible map/reduce engine
  18. 18. StorageRiak Key/Value Store
  19. 19. Storage Riak Key/Value StoreBitcask InnoDB DETS File system Balanced trees LRU ETS
  20. 20. Storage Riak Key/Value StoreBitcask InnoDB DETS File system Balanced trees LRU ETS disk based, ram based, durable not durable
  21. 21. Storagedefault Riak Key/Value Store Bitcask InnoDB DETS File system Balanced trees LRU ETS commo disk based, ram based, durable not durable
  22. 22. What is Riak and what’s the agenda?Decentralized key-value storeA database ideally suited for web applicationsA flexible map/reduce engine
  23. 23. What is Riak and what’s the agenda?Decentralized key-value storeA database ideally suited for web applicationsA flexible map/reduce engine
  24. 24. The Ring
  25. 25. The Ring
  26. 26. The Ring 12 1 11 210 39 4 8 5 7 6 ring size = 12
  27. 27. Consistent Hashing 12 1 11 2 10 3 9 4 8 5 7 6
  28. 28. Consistent Hashing 12 1 11 2 10 3 9 4 8 5 7 6
  29. 29. Read“I want “requires us to know:where is on the ring?
  30. 30. Read“I want “requires us to know:where is on the ring?
  31. 31. ClusterOne Ring size to rule them all, One Ring size tofind them, One Ring size to bring them all and inthe cluster bind them...
  32. 32. Cluster node A node B node Cring size = 12instances = 3ring size / nodes = ~slices per instances
  33. 33. Cluster node A node B node Cring size = 12instances = 3ring size / nodes = ~slices per instances
  34. 34. Cluster - Readnode A node B node C
  35. 35. Cluster - ReadI can haz ? node A node B node CHm, hashes to aslice of the ringowned by node C.
  36. 36. Cluster - ReadI can haz ? node A node B node C Okidoki, now Hey C! I need where’s he...a yeah in my fourth slice
  37. 37. Cluster - ReadI can haz ? node A node B node C
  38. 38. So what about...
  39. 39. ...network partitions? node A node B node C
  40. 40. ...network partitions? node A node B X node C
  41. 41. ...failed nodes?node A node B node C
  42. 42. ...failed nodes?node A node B
  43. 43. ...concurrent writes? node A node B node C client 1 client 2
  44. 44. ...concurrent writes? node A ? node B node C hey, A! save for me client 1 client 2 hey, C! save for me
  45. 45. What is Riak and what’s the agenda?Decentralized key-value storeA database ideally suited for web applicationsA flexible map/reduce engine
  46. 46. Livin’ on the web• concurrent users• unreliable networks• node failures
  47. 47. The N to the R to the W to the DW and the RW
  48. 48. N_VALClient BucketNode A Node B Node C
  49. 49. N_VAL 1Client BucketNode A Node B Node C
  50. 50. N_VAL 2Client BucketNode A Node B Node C
  51. 51. N_VAL 3Client BucketNode A Node B Node C
  52. 52. RClient GET /bucket/key?R=1 BucketNode A Node B Node C
  53. 53. RClient GET /bucket/key?R=1 BucketNode A Node B Node C
  54. 54. RClient GET /bucket/key?R=2 BucketNode A Node B Node C
  55. 55. RClient GET /bucket/key?R=2 BucketNode A Node B Node C
  56. 56. RClient GET /bucket/key?R=2 Agree? BucketNode A Node B Node C
  57. 57. RClient GET /bucket/key?R=2 BucketNode A Node B Node C
  58. 58. The N to the R to the W to the DW and the RW
  59. 59. The N to the R to the W to the DW and the RWNumber of copiesie. distribute to N nodes
  60. 60. Read ie. have R nodes agree The N to the R to the W to the DW and the RWNumber of copiesie. distribute to N nodes
  61. 61. Read ie. have R nodes agree The N to the R to the W to the DW and the RW WriteNumber of copies ie. ack’d byie. distribute to N nodes W nodes
  62. 62. Durable write Read ie. persistently written by ie. have R nodes agree DW nodes The N to the R to the W to the DW and the RW WriteNumber of copies ie. ack’d byie. distribute to N nodes W nodes
  63. 63. Durable write Read ie. persistently written by ie. have R nodes agree DW nodes The N to the R to the W to the DW and the RW Write Read-writeNumber of copies ie. ack’d by ie. persistently deleted byie. distribute to N nodes W nodes RW nodes
  64. 64. The Quorum([node count] / 2) + 1
  65. 65. Bucket propertieshttp://127.0.0.1:8088/riak/meetup/
  66. 66. Conflicts
  67. 67. When worlds collide
  68. 68. I don’t care! allow_mult = falselast_write_wins = true
  69. 69. I do care!allow_mult = true
  70. 70. I do care!• Resolve conflicts in application logic• Conflicts exposed as siblings beneath a key• Response is HTTP 300 Multiple Choice• Served as mime/multipart
  71. 71. ExampleHTTP/1.1 300 Multiple ChoicesX-Riak-Vclock: a85hYGBgzWDKBVIsrLnh3BlMiYx5rAymfeeO8EGFWRLl30G==Content-Type: multipart/mixed; boundary=ZZ3eyjUllBi7GXRRMJsUublFxjnContent-Length: 368--ZZ3eyjUllBi7GXRRMJsUublFxjnContent-Type: text/plainTuesday--ZZ3eyjUllBi7GXRRMJsUublFxjnContent-Type: text/plainThursday--ZZ3eyjUllBi7GXRRMJsUublFxjn--
  72. 72. What is Riak and what’s the agenda?Decentralized key-value storeA database ideally suited for web applicationsA flexible map/reduce engine
  73. 73. Map / Reduce• Javascript or Erlang• Exposed in the HTTP API
  74. 74. Map / Reduce count words function(v) { var words = v.values[0].data.toLowerCase().match(w*,g); var counts = []; for(var word in words) if (words[word] != ) { var count = {}; count[words[word]] = 1; counts.push(count); } return counts;}
  75. 75. Map / Reduce count wordsfunction(values) { var result = {}; for (var value in values) { for(var word in values[value]) { if (word in result) result[word] += values[value][word]; else result[word] = values[value][word]; } } return [result];}
  76. 76. Map & Reduce count words Put this in your POST request and let Riak smoke it{"inputs":"bucket", "query":[{"map":{"language":"javascript","source":"function(v) { var words = v.values[0].data.toLowerCase().match(/w*/g); var counts = []; for(var word in words) if (words[word] != ){ var count = {}; count[words[word]] = 1; counts.push(count); } returncounts; }"}},{"reduce":{"language":"javascript", "source":"function(values){ var result = {}; for (var value in values) { for(var word invalues[value]) { if (word in result) result[word] += values[value][word];else result[word] = values[value][word]; } } return [result]; }"}}]}
  77. 77. Map & Reduce count wordsfunction(v) { var words = v.values[0].data.toLowerCase().match(w*,g); var counts = []; for(var word in words) if (words[word] != ) { var count = {}; count[words[word]] = 1; counts.push(count); } return counts;}{"inputs":"bucket", "query":[{"map":{"language":"javascript","source":"function(v) { var words = v.values[0].data.toLowerCase().match(/w*/g); var counts = []; for(var word in words) if (words[word] != ){ var count = {}; count[words[word]] = 1; counts.push(count); } returncounts; }"}},{"reduce":{"language":"javascript", "source":"function(values){ var result = {}; for (var value in values) { for(var word invalues[value]) { if (word in result) result[word] += values[value][word];else result[word] = values[value][word]; } } return [result]; }"}}]}
  78. 78. Map & Reduce count wordsfunction(values) { var result = {}; for (var value in values) { for(var word in values[value]) { if (word in result) result[word] += values[value][word]; else result[word] = values[value][word]; } } return [result];}{"inputs":"bucket", "query":[{"map":{"language":"javascript","source":"function(v) { var words = v.values[0].data.toLowerCase().match(/w*/g); var counts = []; for(var word in words) if (words[word] != ){ var count = {}; count[words[word]] = 1; counts.push(count); } returncounts; }"}},{"reduce":{"language":"javascript", "source":"function(values){ var result = {}; for (var value in values) { for(var word invalues[value]) { if (word in result) result[word] += values[value][word];else result[word] = values[value][word]; } } return [result]; }"}}]}
  79. 79. Map / Reduce Demo
  80. 80. Links• Non-enforced• Traversable
  81. 81. LinksDemo
  82. 82. The whole enchilada HTTP API Riak Search Luwak Riak Key/Value Store Bitcask InnoDB DETS File system Balanced trees LRU ETS Riak Core Membership management Work distribution leave/join Cluster state Partitioning gossip protocol (consistent hashing, hinted handoff) Erlang / OTP
  83. 83. Try it downloads.basho.com brew install riakWeb admin @ github.com/gmaurice/Riaktive
  84. 84. ResourcesRiak Fast Track @ wiki.basho.com #riak @ freenode github.com/basho/
  85. 85. Thanks for listeningMårten Gustafson@martengustafsonhttp://marten.gustafson.pp.se/marten.gustafson@gmail.com
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×