Squashing the Heisenbugs

1,547 views
1,442 views

Published on

Squashing the Heisenbugs (really Bohrbugs) is a talk that I gave at the Chariot CI Event on December 1st, 2010. Feel free to ask me any questions about it via email or twitter. Also, please hire us if you're looking for any devops help.

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,547
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
12
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Squashing the Heisenbugs

  1. 1. SQUASHINGTHE HEISENBUGS Trotter Cashion / Chariot CI Event 2010 http://www.flickr.com/photos/gliuoo/2539512435/sizes/z/in/photostream/
  2. 2. SQUASHINGTHE HEISENBUGS Trotter Cashion / Chariot CI Event 2010 http://www.flickr.com/photos/gliuoo/2539512435/sizes/z/in/photostream/ BOHRBUGS
  3. 3. SQUASHINGTHE BOHRBUGS Trotter Cashion / Chariot CI Event 2010 http://www.flickr.com/photos/gliuoo/2539512435/sizes/z/in/photostream/
  4. 4. THIS WILL BE ON THE INTERNET!
  5. 5. CO-FOUNDER http://mashion.net
  6. 6. http://www.charlesheflin.com/wp-content/uploads/2010/06/fail-whale.jpg @CASHION
  7. 7. http://gems.github.com/octocat.png github.com/trotter
  8. 8. AGENDA
  9. 9. HEISENBUGS AND BOHRBUGS
  10. 10. THE MANUAL PROBLEM
  11. 11. CHEF
  12. 12. TYING CHEF INTO CI
  13. 13. QUESTIONS (AQAFT)
  14. 14. HEISENBUGS http://upload.wikimedia.org/wikipedia/commons/8/84/Hindenburg_burning.jpg
  15. 15. OBSERVER EFFECT EventYou
  16. 16. http://en.wikipedia.org/wiki/File:Bohr-atom-PAR.svg BOHRBUGS
  17. 17. MANDELBUGS
  18. 18. SCHRÖDINBUGS
  19. 19. NOOBBUGS
  20. 20. THE MANUAL PROBLEM
  21. 21. http://www.dolphinsc.com/images/Dolphin.jpg MYSQL
  22. 22. http://images.buycostumes.com/mgen/merchandiser/32494.jpg POSTGRES
  23. 23. http://www.wallpapers-free.org/34/-/Apache_Longbow_Helicopter/ APACHE
  24. 24. http://www.allbestwallpapers.com/travel-kremlin_and_red_square_fireworks,_moscow,_russia_wallpapers.html NGINX
  25. 25. Memcached
  26. 26. Memcached Redis
  27. 27. Memcached Redis Mongodb
  28. 28. Memcached Redis Mongodb Riak
  29. 29. Memcached Redis Mongodb Riak Cassandra
  30. 30. Memcached Redis Mongodb Riak Cassandra ProjectVoldemort
  31. 31. Memcached Redis Mongodb Riak Cassandra ProjectVoldemort Ubuntu
  32. 32. Memcached Redis Mongodb Riak Cassandra ProjectVoldemort Ubuntu Redhat
  33. 33. Memcached Redis Mongodb Riak Cassandra ProjectVoldemort Ubuntu Redhat Solaris
  34. 34. Memcached Redis Mongodb Riak Cassandra ProjectVoldemort Ubuntu Redhat Solaris iptables
  35. 35. Memcached Redis Mongodb Riak Cassandra ProjectVoldemort Ubuntu Redhat Solaris iptables sshd
  36. 36. Memcached Redis Mongodb Riak Cassandra ProjectVoldemort Ubuntu Redhat Solaris iptables sshd DNS Servers
  37. 37. Memcached Redis Mongodb Riak Cassandra ProjectVoldemort Ubuntu Redhat Solaris iptables sshd DNS Servers TCP/IP internals
  38. 38. We Idolize the Artisan Sysadmin
  39. 39. ... but what do we end up with?
  40. 40. http://www.enviro-bc.ca/wp-content/uploads/2009/04/bowlake-aval_clair-israelson.jpg SNOWFLAKES ... that become avalanches
  41. 41. http://upload.wikimedia.org/wikipedia/commons/6/6d/Niels_Bohr.jpg BOHRBUGS
  42. 42. http://assets.troubledteenswizard.com/uploaded/WoW/nomoney1.jpg NO MORE MONEY!
  43. 43. We’re really left with two choices
  44. 44. Automate... or die.
  45. 45. Automation is not hard!
  46. 46. BASH
  47. 47. http://hackedgadgets.com/wp-content/2/LEGO_V8_Engine_1.jpg CFENGINE
  48. 48. http://www.flickr.com/photos/willemvelthoven/381608678/sizes/l/ PUPPET
  49. 49. http://www.flickr.com/photos/venndiagram/4891510333/sizes/l/ CHEF!
  50. 50. http://www.opscode.com/ WRITTEN BY
  51. 51. http://www.prestonlee.com/wp-content/uploads/2008/09/ruby.png WRITTEN IN
  52. 52. Two Delicious Flavors
  53. 53. http://upload.wikimedia.org/wikipedia/commons/e/e0/Google%E2%80%99s_First_Production_Server.jpg SERVER
  54. 54. HOW IT WORKS
  55. 55. HOW IT WORKS Your Computer
  56. 56. HOW IT WORKS Your Computer Box to Provision
  57. 57. HOW IT WORKS Your Computer Chef Server Box to Provision
  58. 58. HOW IT WORKS Your Computer Chef Server Box to Provision Prepare
  59. 59. HOW IT WORKS Your Computer Chef Server Box to Provision Prepare What now?
  60. 60. HOW IT WORKS Your Computer Chef Server Box to Provision Prepare What now? Cookbooks
  61. 61. HOW IT WORKS Your Computer Chef Server Box to Provision Prepare What now? Cookbooks Run!
  62. 62. http://www.pbs.org/parents/supersisters/han_solo-thumb-454x654.jpg SOLO
  63. 63. HOW IT WORKS
  64. 64. HOW IT WORKS Your Computer
  65. 65. HOW IT WORKS Your Computer Box to Provision
  66. 66. HOW IT WORKS Your Computer Box to Provision Prepare
  67. 67. HOW IT WORKS Your Computer Box to Provision Cookbooks Prepare
  68. 68. HOW IT WORKS Your Computer Box to Provision Cookbooks Prepare Go!
  69. 69. Chef Structure
  70. 70. CHEF HEIRARCHY Node Roles Cookbooks Cookbooks Recipes Attributes Templates Definitions
  71. 71. ROLES ARE REUSABLE! config/dev.json config/db.json roles/db.rb
  72. 72. http://www.flickr.com/photos/seandreilinger/959864706/sizes/o/ GETTING GOING
  73. 73. WHAT YOU NEED
  74. 74. WHAT YOU NEED Ruby & Rubygems
  75. 75. WHAT YOU NEED Ruby & Rubygems `gem install chef` Chef Gem
  76. 76. WHAT YOU NEED Ruby & Rubygems `gem install chef` Chef Gem `git clone http://github.com/ opscode/chef-repo.git` Chef Repo
  77. 77. http://cookbooks.opscode.com/ COOKBOOKS.OPSCODE.COM
  78. 78. TERMINAL JUNKIES
  79. 79. TERMINAL JUNKIES •spatula search mysql
  80. 80. TERMINAL JUNKIES •spatula search mysql •spatula install mysql
  81. 81. TERMINAL JUNKIES •spatula search mysql •spatula install mysql •spatula show mysql
  82. 82. TERMINAL JUNKIES •spatula search mysql •spatula install mysql •spatula show mysql •... knife has similar functionality
  83. 83. GET THE BOX READY spatula prepare db-one.host ssh db-one.host “install ruby” ssh db-one.host “install chef” =
  84. 84. GO GO GO! spatula cook db-one.host db ssh db-one.host “sudo chef-solo -c config/solo.rb -j config/db.json” rsync repo to db-one.host =
  85. 85. WITH CHEF SERVER install dependencies turn on client send cookbooks to client
  86. 86. http://media.photobucket.com/image/minecraft/bluekabal/roma/minesofroma.jpg DIGGING DEEPER
  87. 87. CHEF HEIRARCHY Node Roles Cookbooks Cookbooks Recipes Attributes Templates Definitions
  88. 88. // ./config/db_one.json { "run_list": ["role[db]"] } A NODE Recipes and Roles go here
  89. 89. CHEF HEIRARCHY Node Roles Cookbooks Cookbooks Recipes Attributes Templates Definitions
  90. 90. # ./roles/db.rb name "database" description "Be a DB" run_list [ "apache2", "mysql::server" ] A ROLE
  91. 91. # ./roles/db.rb name "database" description "Be a DB" run_list [ "apache2", "mysql::server" ] A ROLE Recipes to run
  92. 92. # ./roles/db.rb name "database" description "Be a DB" run_list [ "apache2", "mysql::server" ] A ROLE ./cookbooks/apache2/recipes/default.rb
  93. 93. # ./roles/db.rb name "database" description "Be a DB" run_list [ "apache2", "mysql::server" ] A ROLE ./cookbooks/mysql/recipes/server.rb
  94. 94. CHEF HEIRARCHY Node Roles Cookbooks Cookbooks Recipes Attributes Templates Definitions
  95. 95. # ./cookbooks/mysql/attributes/server.rb default[:mysql][:server_debian_password] = secure_password default[:mysql][:server_root_password] = secure_password default[:mysql][:server_repl_password] = secure_password default[:mysql][:bind_address] = ipaddress default[:mysql][:datadir] = "/var/lib/mysql" # Tunables default[:mysql][:tunable][:key_buffer] = "250M" default[:mysql][:tunable][:max_connections] = "800" default[:mysql][:tunable][:wait_timeout] = "180" ATTRIBUTES
  96. 96. // ./config/db.json { "mysql": { "server_root_password": "beefcake" },   "recipes": [ "mysql" ] } OVERRIDING ATTRIBUTES # ./roles/db.rb name "database" description "Be a DB" run_list [ "mysql::server" ] default_attributes :mysql => { :server_root_password => "beefcake" } Node Role
  97. 97. CHEF HEIRARCHY Node Roles Cookbooks Cookbooks Recipes Attributes Templates Definitions
  98. 98. # ./cookbooks/mysql/recipes/server.rb include_recipe "mysql::client" THE RECIPE... PART 1
  99. 99. package "mysql-server" do action :install end THE RECIPE... PART 2
  100. 100. template "/etc/mysql/my.cnf" do source "my.cnf.erb" owner "root" group "root" mode "0644" notifies :restart, resources(:service => "mysql"), :immediately end THE RECIPE... PART 3
  101. 101. CHEF HEIRARCHY Node Roles Cookbooks Cookbooks Recipes Attributes Templates Definitions
  102. 102. user = mysql pid-file = /var/run/mysqld/mysqld.pid socket = /var/run/mysqld/mysqld.sock datadir = <%= @datadir %> bind-address = <%= @node[:mysql][:bind_address] %> A TEMPLATE SAMPLE
  103. 103. user = mysql pid-file = /var/run/mysqld/mysqld.pid socket = /var/run/mysqld/mysqld.sock datadir = <%= @datadir %> bind-address = <%= @node[:mysql][:bind_address] %> A TEMPLATE SAMPLE OMG, ERB!!
  104. 104. user = mysql pid-file = /var/run/mysqld/mysqld.pid socket = /var/run/mysqld/mysqld.sock datadir = <%= @datadir %> bind-address = <%= @node[:mysql][:bind_address] %> A TEMPLATE SAMPLE From Attributes
  105. 105. user = mysql pid-file = /var/run/mysqld/mysqld.pid socket = /var/run/mysqld/mysqld.sock datadir = <%= @datadir %> bind-address = <%= @node[:mysql][:bind_address] %> A TEMPLATE SAMPLE FromTemplate
  106. 106. template "/etc/mysql/my.cnf" do source "my.cnf.erb" owner "root" group "root" mode "0644" variables :datadir => “/var/lib/mysql” end ASSIGNING VARIABLES
  107. 107. CHEF HEIRARCHY Node Roles Cookbooks Cookbooks Recipes Attributes Templates Definitions
  108. 108. define :apache_conf do dir = node[:apache][:dir] template "#{dir}/mods-available/#{params[:name]}.conf" do source "mods/#{params[:name]}.conf.erb" notifies :restart, resources(:service => "apache2") end end # In your recipe apache_conf "mod_ssl" DEFINITIONS
  109. 109. TYING IT INTO CI
  110. 110. USE CHEFTO PROVISION CI http://images.nationalgeographic.com/wpf/media-live/photos/000/148/custom/heart-ngk0207_14824_470x300.jpg
  111. 111. attribution CHEF AS PART OF TEST SCRIPT
  112. 112. CONTINUOUS DEPLOYMENT
  113. 113. http://www.flickr.com/photos/andresthor/3946773501/sizes/l/in/photostream/ LET’S WRAP IT UP
  114. 114. http://www.flickr.com/photos/kenfagerdotcom/3409813881/ AUTOMATE EVERYTHING!!
  115. 115. Deploying == Provisioning
  116. 116. http://www.flickr.com/photos/martin_borjesson/4083726805/sizes/o/ USE CHEF!!!!!
  117. 117. ONTHE INTERNETS
  118. 118. ONTHE INTERNETS •twitter - @cashion
  119. 119. ONTHE INTERNETS •twitter - @cashion •github - http://github.com/trotter
  120. 120. ONTHE INTERNETS •twitter - @cashion •github - http://github.com/trotter •email - cashion on the gmailz
  121. 121. http://www.flickr.com/photos/dancoulter/21042744/sizes/o/ THANK YOU!
  122. 122. RESOURCES
  123. 123. RESOURCES •http://wiki.opscode.com/display/chef/Resources
  124. 124. RESOURCES •http://wiki.opscode.com/display/chef/Resources •#chef on freenode
  125. 125. RESOURCES •http://wiki.opscode.com/display/chef/Resources •#chef on freenode •cashion@gmail.com

×