Breaking a riak cluster
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Breaking a riak cluster

on

  • 4,650 views

slides from OSCON data 2011 workshop Consistency or Bust: Breaking a RIak Cluster

slides from OSCON data 2011 workshop Consistency or Bust: Breaking a RIak Cluster

Statistics

Views

Total Views
4,650
Views on SlideShare
4,546
Embed Views
104

Actions

Likes
7
Downloads
67
Comments
0

5 Embeds 104

http://nosql.io 60
http://lanyrd.com 41
http://twitter.com 1
http://thinkery.me 1
http://www.linkedin.com 1

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Breaking a riak cluster Presentation Transcript

  • 1. Consistency or Bust Breaking a Riak ClusterSunday, July 31, 11
  • 2. Jeff Kirkell email: jeff.kirkell@gmail.com twitter: @jeffkirkell Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 3. NoSQL Primer Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 4. Riak Sales Pitch  Dynamo inspired  AP of CAP  key / value store  scales consistently  fault tolerance  true high availability  built-in replication  lot’s of clients  simple to implement  low $$$ to use  multiple license options Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 5. Riak Deep Dive Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 6. Cluster Defined  cluster  # nodes  64 / # nodes  vnodes Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 7. Important Bits  gossiping  hintful handoff  n # copies of data  r # nodes that must respond for read  w # nodes that must respond for write  dw # replicas written on write for success  rw # replicas deleted for success Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 8. Tools of the Trade  terminal  curl  rebar  rekon Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 9. The Workshop setup instructions for cluster https://github.com/jkirkell/riak_workshop Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 10. Start Your Engines $ ./dev/dev1/bin/riak start $ ./dev/dev2/bin/riak start $ ./dev/dev3/bin/riak start $ ./dev/dev4/bin/riak start $ ./dev/dev5/bin/riak start $ ps aux | grep erl Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 11. Come Together $ ./dev/dev2/bin/riak-admin join dev1@127.0.0.1 $ ./dev/dev3/bin/riak-admin join dev1@127.0.0.1 $ ./dev/dev4/bin/riak-admin join dev1@127.0.0.1 $ ./dev/dev5/bin/riak-admin join dev1@127.0.0.1 $ ./dev/dev3/bin/riak-admin status Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 12. Default Settings quorum = floor(5 / 2 + 1) = 3 Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 13. Add Data $ curl -v -d inserted data -H "Content-Type: text/plain" http://127.0.0.1:8091/riak/default Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 14. Confirm Data Exists $ curl -i http://127.0.0.1:8091/riak/default?keys=true Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 15. Find Your Data $ find ./dev/dev*/data/bitcask/*/*.data Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 16. Stop Primary Node $ dev[n]/bin/riak stop $ curl -i http://127.0.0.1:8091/riak/default/[key] Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 17. Stop Another Primary Node $ dev[n]/bin/riak stop $ curl -i http://127.0.0.1:8091/riak/default/[key] Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 18. Query Again $ curl -i http://127.0.0.1:8091/riak/default/[key] Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 19. Check What’s Happening $ find ./dev/dev*/data/bitcask/*/*.data $ ./dev/dev[n]/bin/riak-admin transfers Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 20. Update Data $ curl -v -d updated data -H "Content-Type: text/plain" http://127.0.0.1:8091/riak/defaults/[key] Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 21. Start Node $ dev[n]/bin/riak start $ ./dev/dev[n]/bin/riak-admin transfers Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 22. Review Logs $ less ./dev2/log/*.1 $ less ./dev5/log/*.1 Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 23. Start Other Node $ dev[n]/bin/riak start $ curl -i http://127.0.0.1:8091/riak/default/[key] Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 24. Summary  consistently fail 1st read with 2 node down  2 node outage max.  hintful-handoff takes time  default settings good for general use Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 25. Clean Your Cluster $ dev1/bin/riak stop $ dev2/bin/riak stop $ dev3/bin/riak stop $ dev4/bin/riak stop $ dev5/bin/riak stop $ cd .. && rm -rf ./dev/* $ make devrel $ dev1/bin/riak start $ dev2/bin/riak start $ dev3/bin/riak start $ dev4/bin/riak start $ dev5/bin/riak start Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 26. Low Value Data n = 2, r = 1, w = 1 Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 27. Set Bucket Properties $ curl -v -X PUT -H "Content-Type: application/json" -d {"props":{"n_val":2,"r":1,"w":1}} http://127.0.0.1:8091/ riak/low_value Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 28. Add Data $ curl -v -d inserted data -H "Content-Type: text/plain" http://127.0.0.1:8091/riak/low_value Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 29. Confirm Data Exists $ curl -i http://127.0.0.1:8091/riak/low_value?keys=true Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 30. Find Your Data $ find ./dev/dev*/data/bitcask/*/*.data Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 31. Stop Primary Node $ dev[n]/bin/riak stop $ curl -i http://127.0.0.1:8091/riak/default/[key] Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 32. Find Your Data Again $ find ./dev/dev*/data/bitcask/*/*.data Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 33. Stop Another Primary Node $ dev[n]/bin/riak stop $ curl -i http://127.0.0.1:8091/riak/default/[key] Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 34. Query Again $ curl -i http://127.0.0.1:8091/riak/default/[key] Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 35. Check What’s Happening $ find ./dev/dev*/data/bitcask/*/*.data $ ./dev/dev[n]/bin/riak-admin transfers Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 36. Add More Data $ curl -v -d new data -H "Content-Type: text/plain" http:// 127.0.0.1:8091/riak/low_value Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 37. Confirm Data Exists $ curl -i http://127.0.0.1:8091/riak/low_value?keys=true Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 38. Start Down Nodes $ dev[n]/bin/riak start $ dev[n]/bin/riak start $ ./dev/dev1/bin/riak-admin transfers Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 39. Confirm Data Exists $ curl -i http://127.0.0.1:8091/riak/low_value?keys=true Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 40. Summary  hintful-handoff does not happen *  1 node outage max for reads  4 node outage max for writes  great for fire and forget data like logs * Look at the logs for the nodes you brought down. It will show data transfer from hand-off even though the available nodes do not serve the read requests. This may or may not happen randomly due to Eventual Consistency. One other note is that R + W = N cannot guarantee consistency. Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 41. Clean Your Cluster $ dev1/bin/riak stop $ dev2/bin/riak stop $ dev3/bin/riak stop $ dev4/bin/riak stop $ dev5/bin/riak stop $ cd .. && rm -rf ./dev/* $ make devrel $ dev1/bin/riak start $ dev2/bin/riak start $ dev3/bin/riak start $ dev4/bin/riak start $ dev5/bin/riak start Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 42. Mission Critical Data n = 5, r = 1, w = 5, dw = 5 Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 43. Set Bucket Properties Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 44. Set Bucket Properties $ curl -v -X PUT -H "Content-Type: application/json" -d {"props":{"n_val":5,"r":1,"w":5,"dw":5}} http:// 127.0.0.1:8091/riak/high_value Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 45. Add Data $ curl -v -d inserted data -H "Content-Type: text/plain" http://127.0.0.1:8091/riak/high_value Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 46. Confirm Data Exists $ curl -i http://127.0.0.1:8091/riak/high_value?keys=true Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 47. Find Your Data $ find ./dev/dev*/data/bitcask/*/*.data Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 48. Stop Any 2 Nodes $ dev[n]/bin/riak stop $ dev[n]/bin/riak stop $ curl -i http://127.0.0.1:8091/riak/high_value/[key] Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 49. Check What’s Happening $ find ./dev/dev*/data/bitcask/*/*.data $ ./dev/dev[n]/bin/riak-admin transfers Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 50. Stop Another Node $ dev[n]/bin/riak stop $ curl -i http://127.0.0.1:8091/riak/high_value/[key] Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 51. Check What’s Happening $ find ./dev/dev*/data/bitcask/*/*.data $ ./dev/dev[n]/bin/riak-admin transfers Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 52. Update Data $ curl -v -d updated data -H "Content-Type: text/plain" http://127.0.0.1:8091/riak/high_value/[key] Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 53. Query Data $ curl -i http://127.0.0.1:8091/riak/high_value/[key] Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 54. Start Down Nodes $ dev[n]/bin/riak start $ dev[n]/bin/riak start $ dev[n]/bin/riak start $ ./dev/dev1/bin/riak-admin transfers Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 55. Confirm Data Exists $ curl -i http://127.0.0.1:8091/riak/high_value?keys=true Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 56. Summary  data will duplicate on available nodes to ensure 5 copies  4 node outage max for reads  4 node outage max for writes  slower due to all data writes  great for high availability data Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 57. Total Destruction Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 58. Total Destruction Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 59. Summary  easy to get up and going  scales as you need it  flexible configuration for consistency sake  eventual consistency doesn’t always do what you expect  just scratched the surface Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11
  • 60. Great Reading  http://www.basho.com  http://bit.ly/eG08u0  http://bit.ly/eFfcjb  http://bit.ly/ejCClJ  http://slidesha.re/g3hQfg  http://bit.ly/n6hWMK Consistency or Bust: Breaking a Riak ClusterSunday, July 31, 11