Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Consistency or Bust                      Breaking a Riak ClusterSunday, July 31, 11
Jeff Kirkell      email: jeff.kirkell@gmail.com      twitter: @jeffkirkell      Consistency or Bust: Breaking a Riak Clust...
NoSQL Primer      Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
Riak Sales Pitch      Dynamo inspired      AP of CAP      key / value store      scales consistently      fault toler...
Riak Deep Dive      Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
Cluster Defined      cluster      # nodes      64 / # nodes      vnodes      Consistency or Bust: Breaking a Riak Clust...
Important Bits      gossiping      hintful handoff      n # copies of data      r # nodes that must respond for read  ...
Tools of the Trade      terminal      curl      rebar      rekon      Consistency or Bust: Breaking a Riak ClusterSund...
The Workshop                                   setup instructions for cluster                            https://github.co...
Start Your Engines     $ ./dev/dev1/bin/riak start     $ ./dev/dev2/bin/riak start     $ ./dev/dev3/bin/riak start     $ ....
Come Together     $ ./dev/dev2/bin/riak-admin join dev1@127.0.0.1     $ ./dev/dev3/bin/riak-admin join dev1@127.0.0.1     ...
Default Settings                                           quorum = floor(5 / 2 + 1) = 3      Consistency or Bust: Breaking...
Add Data     $ curl -v -d inserted data -H "Content-Type: text/plain"       http://127.0.0.1:8091/riak/default      Consis...
Confirm Data Exists     $ curl -i http://127.0.0.1:8091/riak/default?keys=true      Consistency or Bust: Breaking a Riak Cl...
Find Your Data     $ find ./dev/dev*/data/bitcask/*/*.data      Consistency or Bust: Breaking a Riak ClusterSunday, July 3...
Stop Primary Node     $ dev[n]/bin/riak stop     $ curl -i http://127.0.0.1:8091/riak/default/[key]      Consistency or Bu...
Stop Another Primary Node     $ dev[n]/bin/riak stop     $ curl -i http://127.0.0.1:8091/riak/default/[key]      Consisten...
Query Again     $ curl -i http://127.0.0.1:8091/riak/default/[key]      Consistency or Bust: Breaking a Riak ClusterSunday...
Check What’s Happening     $ find ./dev/dev*/data/bitcask/*/*.data     $ ./dev/dev[n]/bin/riak-admin transfers      Consist...
Update Data     $ curl -v -d updated data -H "Content-Type: text/plain"       http://127.0.0.1:8091/riak/defaults/[key]   ...
Start Node     $ dev[n]/bin/riak start     $ ./dev/dev[n]/bin/riak-admin transfers      Consistency or Bust: Breaking a Ri...
Review Logs     $ less ./dev2/log/*.1     $ less ./dev5/log/*.1      Consistency or Bust: Breaking a Riak ClusterSunday, J...
Start Other Node     $ dev[n]/bin/riak start     $ curl -i http://127.0.0.1:8091/riak/default/[key]      Consistency or Bu...
Summary      consistently fail 1st read with 2 node down      2 node outage max.      hintful-handoff takes time      ...
Clean Your Cluster     $        dev1/bin/riak stop     $        dev2/bin/riak stop     $        dev3/bin/riak stop     $  ...
Low Value Data                                                     n = 2, r = 1, w = 1      Consistency or Bust: Breaking ...
Set Bucket Properties     $ curl -v -X PUT -H "Content-Type: application/json" -d       {"props":{"n_val":2,"r":1,"w":1}} ...
Add Data     $ curl -v -d inserted data -H "Content-Type: text/plain"       http://127.0.0.1:8091/riak/low_value      Cons...
Confirm Data Exists     $ curl -i http://127.0.0.1:8091/riak/low_value?keys=true      Consistency or Bust: Breaking a Riak ...
Find Your Data     $ find ./dev/dev*/data/bitcask/*/*.data      Consistency or Bust: Breaking a Riak ClusterSunday, July 3...
Stop Primary Node     $ dev[n]/bin/riak stop     $ curl -i http://127.0.0.1:8091/riak/default/[key]      Consistency or Bu...
Find Your Data Again     $ find ./dev/dev*/data/bitcask/*/*.data      Consistency or Bust: Breaking a Riak ClusterSunday, ...
Stop Another Primary Node     $ dev[n]/bin/riak stop     $ curl -i http://127.0.0.1:8091/riak/default/[key]      Consisten...
Query Again     $ curl -i http://127.0.0.1:8091/riak/default/[key]      Consistency or Bust: Breaking a Riak ClusterSunday...
Check What’s Happening     $ find ./dev/dev*/data/bitcask/*/*.data     $ ./dev/dev[n]/bin/riak-admin transfers      Consist...
Add More Data     $ curl -v -d new data -H "Content-Type: text/plain" http://       127.0.0.1:8091/riak/low_value      Con...
Confirm Data Exists     $ curl -i http://127.0.0.1:8091/riak/low_value?keys=true      Consistency or Bust: Breaking a Riak ...
Start Down Nodes     $ dev[n]/bin/riak start     $ dev[n]/bin/riak start     $ ./dev/dev1/bin/riak-admin transfers      Co...
Confirm Data Exists     $ curl -i http://127.0.0.1:8091/riak/low_value?keys=true      Consistency or Bust: Breaking a Riak ...
Summary      hintful-handoff does not happen *      1 node outage max for reads      4 node outage max for writes     ...
Clean Your Cluster     $        dev1/bin/riak stop     $        dev2/bin/riak stop     $        dev3/bin/riak stop     $  ...
Mission Critical Data                                              n = 5, r = 1, w = 5, dw = 5      Consistency or Bust: B...
Set Bucket Properties      Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
Set Bucket Properties     $ curl -v -X PUT -H "Content-Type: application/json" -d       {"props":{"n_val":5,"r":1,"w":5,"d...
Add Data     $ curl -v -d inserted data -H "Content-Type: text/plain"       http://127.0.0.1:8091/riak/high_value      Con...
Confirm Data Exists     $ curl -i http://127.0.0.1:8091/riak/high_value?keys=true      Consistency or Bust: Breaking a Riak...
Find Your Data     $ find ./dev/dev*/data/bitcask/*/*.data      Consistency or Bust: Breaking a Riak ClusterSunday, July 3...
Stop Any 2 Nodes     $ dev[n]/bin/riak stop     $ dev[n]/bin/riak stop     $ curl -i http://127.0.0.1:8091/riak/high_value...
Check What’s Happening     $ find ./dev/dev*/data/bitcask/*/*.data     $ ./dev/dev[n]/bin/riak-admin transfers      Consist...
Stop Another Node     $ dev[n]/bin/riak stop     $ curl -i http://127.0.0.1:8091/riak/high_value/[key]      Consistency or...
Check What’s Happening     $ find ./dev/dev*/data/bitcask/*/*.data     $ ./dev/dev[n]/bin/riak-admin transfers      Consist...
Update Data     $ curl -v -d updated data -H "Content-Type: text/plain"       http://127.0.0.1:8091/riak/high_value/[key] ...
Query Data     $ curl -i http://127.0.0.1:8091/riak/high_value/[key]      Consistency or Bust: Breaking a Riak ClusterSund...
Start Down Nodes     $ dev[n]/bin/riak start     $ dev[n]/bin/riak start     $ dev[n]/bin/riak start     $ ./dev/dev1/bin/...
Confirm Data Exists     $ curl -i http://127.0.0.1:8091/riak/high_value?keys=true      Consistency or Bust: Breaking a Riak...
Summary      data will duplicate on available nodes to ensure 5 copies      4 node outage max for reads      4 node out...
Total Destruction      Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
Total Destruction      Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
Summary      easy to get up and going      scales as you need it      flexible configuration for consistency sake      e...
Great Reading      http://www.basho.com      http://bit.ly/eG08u0      http://bit.ly/eFfcjb      http://bit.ly/ejCClJ ...
Upcoming SlideShare
Loading in …5
×

Breaking a riak cluster

8,734 views

Published on

slides from OSCON data 2011 workshop Consistency or Bust: Breaking a RIak Cluster

  • Be the first to comment

Breaking a riak cluster

  1. Consistency or Bust Breaking a Riak ClusterSunday, July 31, 11
  2. Jeff Kirkell email: jeff.kirkell@gmail.com twitter: @jeffkirkell Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  3. NoSQL Primer Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  4. Riak Sales Pitch  Dynamo inspired  AP of CAP  key / value store  scales consistently  fault tolerance  true high availability  built-in replication  lot’s of clients  simple to implement  low $$$ to use  multiple license options Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  5. Riak Deep Dive Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  6. Cluster Defined  cluster  # nodes  64 / # nodes  vnodes Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  7. Important Bits  gossiping  hintful handoff  n # copies of data  r # nodes that must respond for read  w # nodes that must respond for write  dw # replicas written on write for success  rw # replicas deleted for success Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  8. Tools of the Trade  terminal  curl  rebar  rekon Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  9. The Workshop setup instructions for cluster https://github.com/jkirkell/riak_workshop Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  10. Start Your Engines $ ./dev/dev1/bin/riak start $ ./dev/dev2/bin/riak start $ ./dev/dev3/bin/riak start $ ./dev/dev4/bin/riak start $ ./dev/dev5/bin/riak start $ ps aux | grep erl Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  11. Come Together $ ./dev/dev2/bin/riak-admin join dev1@127.0.0.1 $ ./dev/dev3/bin/riak-admin join dev1@127.0.0.1 $ ./dev/dev4/bin/riak-admin join dev1@127.0.0.1 $ ./dev/dev5/bin/riak-admin join dev1@127.0.0.1 $ ./dev/dev3/bin/riak-admin status Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  12. Default Settings quorum = floor(5 / 2 + 1) = 3 Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  13. Add Data $ curl -v -d inserted data -H "Content-Type: text/plain" http://127.0.0.1:8091/riak/default Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  14. Confirm Data Exists $ curl -i http://127.0.0.1:8091/riak/default?keys=true Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  15. Find Your Data $ find ./dev/dev*/data/bitcask/*/*.data Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  16. Stop Primary Node $ dev[n]/bin/riak stop $ curl -i http://127.0.0.1:8091/riak/default/[key] Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  17. Stop Another Primary Node $ dev[n]/bin/riak stop $ curl -i http://127.0.0.1:8091/riak/default/[key] Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  18. Query Again $ curl -i http://127.0.0.1:8091/riak/default/[key] Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  19. Check What’s Happening $ find ./dev/dev*/data/bitcask/*/*.data $ ./dev/dev[n]/bin/riak-admin transfers Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  20. Update Data $ curl -v -d updated data -H "Content-Type: text/plain" http://127.0.0.1:8091/riak/defaults/[key] Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  21. Start Node $ dev[n]/bin/riak start $ ./dev/dev[n]/bin/riak-admin transfers Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  22. Review Logs $ less ./dev2/log/*.1 $ less ./dev5/log/*.1 Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  23. Start Other Node $ dev[n]/bin/riak start $ curl -i http://127.0.0.1:8091/riak/default/[key] Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  24. Summary  consistently fail 1st read with 2 node down  2 node outage max.  hintful-handoff takes time  default settings good for general use Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  25. Clean Your Cluster $ dev1/bin/riak stop $ dev2/bin/riak stop $ dev3/bin/riak stop $ dev4/bin/riak stop $ dev5/bin/riak stop $ cd .. && rm -rf ./dev/* $ make devrel $ dev1/bin/riak start $ dev2/bin/riak start $ dev3/bin/riak start $ dev4/bin/riak start $ dev5/bin/riak start Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  26. Low Value Data n = 2, r = 1, w = 1 Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  27. Set Bucket Properties $ curl -v -X PUT -H "Content-Type: application/json" -d {"props":{"n_val":2,"r":1,"w":1}} http://127.0.0.1:8091/ riak/low_value Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  28. Add Data $ curl -v -d inserted data -H "Content-Type: text/plain" http://127.0.0.1:8091/riak/low_value Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  29. Confirm Data Exists $ curl -i http://127.0.0.1:8091/riak/low_value?keys=true Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  30. Find Your Data $ find ./dev/dev*/data/bitcask/*/*.data Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  31. Stop Primary Node $ dev[n]/bin/riak stop $ curl -i http://127.0.0.1:8091/riak/default/[key] Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  32. Find Your Data Again $ find ./dev/dev*/data/bitcask/*/*.data Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  33. Stop Another Primary Node $ dev[n]/bin/riak stop $ curl -i http://127.0.0.1:8091/riak/default/[key] Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  34. Query Again $ curl -i http://127.0.0.1:8091/riak/default/[key] Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  35. Check What’s Happening $ find ./dev/dev*/data/bitcask/*/*.data $ ./dev/dev[n]/bin/riak-admin transfers Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  36. Add More Data $ curl -v -d new data -H "Content-Type: text/plain" http:// 127.0.0.1:8091/riak/low_value Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  37. Confirm Data Exists $ curl -i http://127.0.0.1:8091/riak/low_value?keys=true Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  38. Start Down Nodes $ dev[n]/bin/riak start $ dev[n]/bin/riak start $ ./dev/dev1/bin/riak-admin transfers Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  39. Confirm Data Exists $ curl -i http://127.0.0.1:8091/riak/low_value?keys=true Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  40. Summary  hintful-handoff does not happen *  1 node outage max for reads  4 node outage max for writes  great for fire and forget data like logs * Look at the logs for the nodes you brought down. It will show data transfer from hand-off even though the available nodes do not serve the read requests. This may or may not happen randomly due to Eventual Consistency. One other note is that R + W = N cannot guarantee consistency. Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  41. Clean Your Cluster $ dev1/bin/riak stop $ dev2/bin/riak stop $ dev3/bin/riak stop $ dev4/bin/riak stop $ dev5/bin/riak stop $ cd .. && rm -rf ./dev/* $ make devrel $ dev1/bin/riak start $ dev2/bin/riak start $ dev3/bin/riak start $ dev4/bin/riak start $ dev5/bin/riak start Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  42. Mission Critical Data n = 5, r = 1, w = 5, dw = 5 Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  43. Set Bucket Properties Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  44. Set Bucket Properties $ curl -v -X PUT -H "Content-Type: application/json" -d {"props":{"n_val":5,"r":1,"w":5,"dw":5}} http:// 127.0.0.1:8091/riak/high_value Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  45. Add Data $ curl -v -d inserted data -H "Content-Type: text/plain" http://127.0.0.1:8091/riak/high_value Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  46. Confirm Data Exists $ curl -i http://127.0.0.1:8091/riak/high_value?keys=true Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  47. Find Your Data $ find ./dev/dev*/data/bitcask/*/*.data Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  48. Stop Any 2 Nodes $ dev[n]/bin/riak stop $ dev[n]/bin/riak stop $ curl -i http://127.0.0.1:8091/riak/high_value/[key] Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  49. Check What’s Happening $ find ./dev/dev*/data/bitcask/*/*.data $ ./dev/dev[n]/bin/riak-admin transfers Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  50. Stop Another Node $ dev[n]/bin/riak stop $ curl -i http://127.0.0.1:8091/riak/high_value/[key] Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  51. Check What’s Happening $ find ./dev/dev*/data/bitcask/*/*.data $ ./dev/dev[n]/bin/riak-admin transfers Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  52. Update Data $ curl -v -d updated data -H "Content-Type: text/plain" http://127.0.0.1:8091/riak/high_value/[key] Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  53. Query Data $ curl -i http://127.0.0.1:8091/riak/high_value/[key] Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  54. Start Down Nodes $ dev[n]/bin/riak start $ dev[n]/bin/riak start $ dev[n]/bin/riak start $ ./dev/dev1/bin/riak-admin transfers Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  55. Confirm Data Exists $ curl -i http://127.0.0.1:8091/riak/high_value?keys=true Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  56. Summary  data will duplicate on available nodes to ensure 5 copies  4 node outage max for reads  4 node outage max for writes  slower due to all data writes  great for high availability data Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  57. Total Destruction Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  58. Total Destruction Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  59. Summary  easy to get up and going  scales as you need it  flexible configuration for consistency sake  eventual consistency doesn’t always do what you expect  just scratched the surface Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  60. Great Reading  http://www.basho.com  http://bit.ly/eG08u0  http://bit.ly/eFfcjb  http://bit.ly/ejCClJ  http://slidesha.re/g3hQfg  http://bit.ly/n6hWMK Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11

×