Consistency or Bust                      Breaking a Riak ClusterSunday, July 31, 11
Jeff Kirkell      email: jeff.kirkell@gmail.com      twitter: @jeffkirkell      Consistency or Bust: Breaking a Riak Clust...
NoSQL Primer      Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
Riak Sales Pitch      Dynamo inspired      AP of CAP      key / value store      scales consistently      fault toler...
Riak Deep Dive      Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
Cluster Defined      cluster      # nodes      64 / # nodes      vnodes      Consistency or Bust: Breaking a Riak Clust...
Important Bits      gossiping      hintful handoff      n # copies of data      r # nodes that must respond for read  ...
Tools of the Trade      terminal      curl      rebar      rekon      Consistency or Bust: Breaking a Riak ClusterSund...
The Workshop                                   setup instructions for cluster                            https://github.co...
Start Your Engines     $ ./dev/dev1/bin/riak start     $ ./dev/dev2/bin/riak start     $ ./dev/dev3/bin/riak start     $ ....
Come Together     $ ./dev/dev2/bin/riak-admin join dev1@127.0.0.1     $ ./dev/dev3/bin/riak-admin join dev1@127.0.0.1     ...
Default Settings                                           quorum = floor(5 / 2 + 1) = 3      Consistency or Bust: Breaking...
Add Data     $ curl -v -d inserted data -H "Content-Type: text/plain"       http://127.0.0.1:8091/riak/default      Consis...
Confirm Data Exists     $ curl -i http://127.0.0.1:8091/riak/default?keys=true      Consistency or Bust: Breaking a Riak Cl...
Find Your Data     $ find ./dev/dev*/data/bitcask/*/*.data      Consistency or Bust: Breaking a Riak ClusterSunday, July 3...
Stop Primary Node     $ dev[n]/bin/riak stop     $ curl -i http://127.0.0.1:8091/riak/default/[key]      Consistency or Bu...
Stop Another Primary Node     $ dev[n]/bin/riak stop     $ curl -i http://127.0.0.1:8091/riak/default/[key]      Consisten...
Query Again     $ curl -i http://127.0.0.1:8091/riak/default/[key]      Consistency or Bust: Breaking a Riak ClusterSunday...
Check What’s Happening     $ find ./dev/dev*/data/bitcask/*/*.data     $ ./dev/dev[n]/bin/riak-admin transfers      Consist...
Update Data     $ curl -v -d updated data -H "Content-Type: text/plain"       http://127.0.0.1:8091/riak/defaults/[key]   ...
Start Node     $ dev[n]/bin/riak start     $ ./dev/dev[n]/bin/riak-admin transfers      Consistency or Bust: Breaking a Ri...
Review Logs     $ less ./dev2/log/*.1     $ less ./dev5/log/*.1      Consistency or Bust: Breaking a Riak ClusterSunday, J...
Start Other Node     $ dev[n]/bin/riak start     $ curl -i http://127.0.0.1:8091/riak/default/[key]      Consistency or Bu...
Summary      consistently fail 1st read with 2 node down      2 node outage max.      hintful-handoff takes time      ...
Clean Your Cluster     $        dev1/bin/riak stop     $        dev2/bin/riak stop     $        dev3/bin/riak stop     $  ...
Low Value Data                                                     n = 2, r = 1, w = 1      Consistency or Bust: Breaking ...
Set Bucket Properties     $ curl -v -X PUT -H "Content-Type: application/json" -d       {"props":{"n_val":2,"r":1,"w":1}} ...
Add Data     $ curl -v -d inserted data -H "Content-Type: text/plain"       http://127.0.0.1:8091/riak/low_value      Cons...
Confirm Data Exists     $ curl -i http://127.0.0.1:8091/riak/low_value?keys=true      Consistency or Bust: Breaking a Riak ...
Find Your Data     $ find ./dev/dev*/data/bitcask/*/*.data      Consistency or Bust: Breaking a Riak ClusterSunday, July 3...
Stop Primary Node     $ dev[n]/bin/riak stop     $ curl -i http://127.0.0.1:8091/riak/default/[key]      Consistency or Bu...
Find Your Data Again     $ find ./dev/dev*/data/bitcask/*/*.data      Consistency or Bust: Breaking a Riak ClusterSunday, ...
Stop Another Primary Node     $ dev[n]/bin/riak stop     $ curl -i http://127.0.0.1:8091/riak/default/[key]      Consisten...
Query Again     $ curl -i http://127.0.0.1:8091/riak/default/[key]      Consistency or Bust: Breaking a Riak ClusterSunday...
Check What’s Happening     $ find ./dev/dev*/data/bitcask/*/*.data     $ ./dev/dev[n]/bin/riak-admin transfers      Consist...
Add More Data     $ curl -v -d new data -H "Content-Type: text/plain" http://       127.0.0.1:8091/riak/low_value      Con...
Confirm Data Exists     $ curl -i http://127.0.0.1:8091/riak/low_value?keys=true      Consistency or Bust: Breaking a Riak ...
Start Down Nodes     $ dev[n]/bin/riak start     $ dev[n]/bin/riak start     $ ./dev/dev1/bin/riak-admin transfers      Co...
Confirm Data Exists     $ curl -i http://127.0.0.1:8091/riak/low_value?keys=true      Consistency or Bust: Breaking a Riak ...
Summary      hintful-handoff does not happen *      1 node outage max for reads      4 node outage max for writes     ...
Clean Your Cluster     $        dev1/bin/riak stop     $        dev2/bin/riak stop     $        dev3/bin/riak stop     $  ...
Mission Critical Data                                              n = 5, r = 1, w = 5, dw = 5      Consistency or Bust: B...
Set Bucket Properties      Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
Set Bucket Properties     $ curl -v -X PUT -H "Content-Type: application/json" -d       {"props":{"n_val":5,"r":1,"w":5,"d...
Add Data     $ curl -v -d inserted data -H "Content-Type: text/plain"       http://127.0.0.1:8091/riak/high_value      Con...
Confirm Data Exists     $ curl -i http://127.0.0.1:8091/riak/high_value?keys=true      Consistency or Bust: Breaking a Riak...
Find Your Data     $ find ./dev/dev*/data/bitcask/*/*.data      Consistency or Bust: Breaking a Riak ClusterSunday, July 3...
Stop Any 2 Nodes     $ dev[n]/bin/riak stop     $ dev[n]/bin/riak stop     $ curl -i http://127.0.0.1:8091/riak/high_value...
Check What’s Happening     $ find ./dev/dev*/data/bitcask/*/*.data     $ ./dev/dev[n]/bin/riak-admin transfers      Consist...
Stop Another Node     $ dev[n]/bin/riak stop     $ curl -i http://127.0.0.1:8091/riak/high_value/[key]      Consistency or...
Check What’s Happening     $ find ./dev/dev*/data/bitcask/*/*.data     $ ./dev/dev[n]/bin/riak-admin transfers      Consist...
Update Data     $ curl -v -d updated data -H "Content-Type: text/plain"       http://127.0.0.1:8091/riak/high_value/[key] ...
Query Data     $ curl -i http://127.0.0.1:8091/riak/high_value/[key]      Consistency or Bust: Breaking a Riak ClusterSund...
Start Down Nodes     $ dev[n]/bin/riak start     $ dev[n]/bin/riak start     $ dev[n]/bin/riak start     $ ./dev/dev1/bin/...
Confirm Data Exists     $ curl -i http://127.0.0.1:8091/riak/high_value?keys=true      Consistency or Bust: Breaking a Riak...
Summary      data will duplicate on available nodes to ensure 5 copies      4 node outage max for reads      4 node out...
Total Destruction      Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
Total Destruction      Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
Summary      easy to get up and going      scales as you need it      flexible configuration for consistency sake      e...
Great Reading      http://www.basho.com      http://bit.ly/eG08u0      http://bit.ly/eFfcjb      http://bit.ly/ejCClJ ...
Upcoming SlideShare
Loading in...5
×

Breaking a riak cluster

4,278

Published on

slides from OSCON data 2011 workshop Consistency or Bust: Breaking a RIak Cluster

0 Comments
8 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
4,278
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
70
Comments
0
Likes
8
Embeds 0
No embeds

No notes for slide

Transcript of "Breaking a riak cluster"

  1. 1. Consistency or Bust Breaking a Riak ClusterSunday, July 31, 11
  2. 2. Jeff Kirkell email: jeff.kirkell@gmail.com twitter: @jeffkirkell Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  3. 3. NoSQL Primer Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  4. 4. Riak Sales Pitch  Dynamo inspired  AP of CAP  key / value store  scales consistently  fault tolerance  true high availability  built-in replication  lot’s of clients  simple to implement  low $$$ to use  multiple license options Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  5. 5. Riak Deep Dive Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  6. 6. Cluster Defined  cluster  # nodes  64 / # nodes  vnodes Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  7. 7. Important Bits  gossiping  hintful handoff  n # copies of data  r # nodes that must respond for read  w # nodes that must respond for write  dw # replicas written on write for success  rw # replicas deleted for success Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  8. 8. Tools of the Trade  terminal  curl  rebar  rekon Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  9. 9. The Workshop setup instructions for cluster https://github.com/jkirkell/riak_workshop Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  10. 10. Start Your Engines $ ./dev/dev1/bin/riak start $ ./dev/dev2/bin/riak start $ ./dev/dev3/bin/riak start $ ./dev/dev4/bin/riak start $ ./dev/dev5/bin/riak start $ ps aux | grep erl Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  11. 11. Come Together $ ./dev/dev2/bin/riak-admin join dev1@127.0.0.1 $ ./dev/dev3/bin/riak-admin join dev1@127.0.0.1 $ ./dev/dev4/bin/riak-admin join dev1@127.0.0.1 $ ./dev/dev5/bin/riak-admin join dev1@127.0.0.1 $ ./dev/dev3/bin/riak-admin status Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  12. 12. Default Settings quorum = floor(5 / 2 + 1) = 3 Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  13. 13. Add Data $ curl -v -d inserted data -H "Content-Type: text/plain" http://127.0.0.1:8091/riak/default Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  14. 14. Confirm Data Exists $ curl -i http://127.0.0.1:8091/riak/default?keys=true Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  15. 15. Find Your Data $ find ./dev/dev*/data/bitcask/*/*.data Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  16. 16. Stop Primary Node $ dev[n]/bin/riak stop $ curl -i http://127.0.0.1:8091/riak/default/[key] Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  17. 17. Stop Another Primary Node $ dev[n]/bin/riak stop $ curl -i http://127.0.0.1:8091/riak/default/[key] Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  18. 18. Query Again $ curl -i http://127.0.0.1:8091/riak/default/[key] Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  19. 19. Check What’s Happening $ find ./dev/dev*/data/bitcask/*/*.data $ ./dev/dev[n]/bin/riak-admin transfers Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  20. 20. Update Data $ curl -v -d updated data -H "Content-Type: text/plain" http://127.0.0.1:8091/riak/defaults/[key] Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  21. 21. Start Node $ dev[n]/bin/riak start $ ./dev/dev[n]/bin/riak-admin transfers Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  22. 22. Review Logs $ less ./dev2/log/*.1 $ less ./dev5/log/*.1 Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  23. 23. Start Other Node $ dev[n]/bin/riak start $ curl -i http://127.0.0.1:8091/riak/default/[key] Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  24. 24. Summary  consistently fail 1st read with 2 node down  2 node outage max.  hintful-handoff takes time  default settings good for general use Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  25. 25. Clean Your Cluster $ dev1/bin/riak stop $ dev2/bin/riak stop $ dev3/bin/riak stop $ dev4/bin/riak stop $ dev5/bin/riak stop $ cd .. && rm -rf ./dev/* $ make devrel $ dev1/bin/riak start $ dev2/bin/riak start $ dev3/bin/riak start $ dev4/bin/riak start $ dev5/bin/riak start Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  26. 26. Low Value Data n = 2, r = 1, w = 1 Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  27. 27. Set Bucket Properties $ curl -v -X PUT -H "Content-Type: application/json" -d {"props":{"n_val":2,"r":1,"w":1}} http://127.0.0.1:8091/ riak/low_value Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  28. 28. Add Data $ curl -v -d inserted data -H "Content-Type: text/plain" http://127.0.0.1:8091/riak/low_value Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  29. 29. Confirm Data Exists $ curl -i http://127.0.0.1:8091/riak/low_value?keys=true Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  30. 30. Find Your Data $ find ./dev/dev*/data/bitcask/*/*.data Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  31. 31. Stop Primary Node $ dev[n]/bin/riak stop $ curl -i http://127.0.0.1:8091/riak/default/[key] Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  32. 32. Find Your Data Again $ find ./dev/dev*/data/bitcask/*/*.data Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  33. 33. Stop Another Primary Node $ dev[n]/bin/riak stop $ curl -i http://127.0.0.1:8091/riak/default/[key] Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  34. 34. Query Again $ curl -i http://127.0.0.1:8091/riak/default/[key] Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  35. 35. Check What’s Happening $ find ./dev/dev*/data/bitcask/*/*.data $ ./dev/dev[n]/bin/riak-admin transfers Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  36. 36. Add More Data $ curl -v -d new data -H "Content-Type: text/plain" http:// 127.0.0.1:8091/riak/low_value Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  37. 37. Confirm Data Exists $ curl -i http://127.0.0.1:8091/riak/low_value?keys=true Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  38. 38. Start Down Nodes $ dev[n]/bin/riak start $ dev[n]/bin/riak start $ ./dev/dev1/bin/riak-admin transfers Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  39. 39. Confirm Data Exists $ curl -i http://127.0.0.1:8091/riak/low_value?keys=true Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  40. 40. Summary  hintful-handoff does not happen *  1 node outage max for reads  4 node outage max for writes  great for fire and forget data like logs * Look at the logs for the nodes you brought down. It will show data transfer from hand-off even though the available nodes do not serve the read requests. This may or may not happen randomly due to Eventual Consistency. One other note is that R + W = N cannot guarantee consistency. Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  41. 41. Clean Your Cluster $ dev1/bin/riak stop $ dev2/bin/riak stop $ dev3/bin/riak stop $ dev4/bin/riak stop $ dev5/bin/riak stop $ cd .. && rm -rf ./dev/* $ make devrel $ dev1/bin/riak start $ dev2/bin/riak start $ dev3/bin/riak start $ dev4/bin/riak start $ dev5/bin/riak start Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  42. 42. Mission Critical Data n = 5, r = 1, w = 5, dw = 5 Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  43. 43. Set Bucket Properties Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  44. 44. Set Bucket Properties $ curl -v -X PUT -H "Content-Type: application/json" -d {"props":{"n_val":5,"r":1,"w":5,"dw":5}} http:// 127.0.0.1:8091/riak/high_value Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  45. 45. Add Data $ curl -v -d inserted data -H "Content-Type: text/plain" http://127.0.0.1:8091/riak/high_value Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  46. 46. Confirm Data Exists $ curl -i http://127.0.0.1:8091/riak/high_value?keys=true Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  47. 47. Find Your Data $ find ./dev/dev*/data/bitcask/*/*.data Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  48. 48. Stop Any 2 Nodes $ dev[n]/bin/riak stop $ dev[n]/bin/riak stop $ curl -i http://127.0.0.1:8091/riak/high_value/[key] Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  49. 49. Check What’s Happening $ find ./dev/dev*/data/bitcask/*/*.data $ ./dev/dev[n]/bin/riak-admin transfers Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  50. 50. Stop Another Node $ dev[n]/bin/riak stop $ curl -i http://127.0.0.1:8091/riak/high_value/[key] Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  51. 51. Check What’s Happening $ find ./dev/dev*/data/bitcask/*/*.data $ ./dev/dev[n]/bin/riak-admin transfers Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  52. 52. Update Data $ curl -v -d updated data -H "Content-Type: text/plain" http://127.0.0.1:8091/riak/high_value/[key] Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  53. 53. Query Data $ curl -i http://127.0.0.1:8091/riak/high_value/[key] Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  54. 54. Start Down Nodes $ dev[n]/bin/riak start $ dev[n]/bin/riak start $ dev[n]/bin/riak start $ ./dev/dev1/bin/riak-admin transfers Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  55. 55. Confirm Data Exists $ curl -i http://127.0.0.1:8091/riak/high_value?keys=true Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  56. 56. Summary  data will duplicate on available nodes to ensure 5 copies  4 node outage max for reads  4 node outage max for writes  slower due to all data writes  great for high availability data Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  57. 57. Total Destruction Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  58. 58. Total Destruction Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  59. 59. Summary  easy to get up and going  scales as you need it  flexible configuration for consistency sake  eventual consistency doesn’t always do what you expect  just scratched the surface Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  60. 60. Great Reading  http://www.basho.com  http://bit.ly/eG08u0  http://bit.ly/eFfcjb  http://bit.ly/ejCClJ  http://slidesha.re/g3hQfg  http://bit.ly/n6hWMK Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×