SQUASHINGTHE HEISENBUGS
Trotter Cashion / Chariot CI Event 2010
http://www.flickr.com/photos/gliuoo/2539512435/sizes/z/in/p...
SQUASHINGTHE HEISENBUGS
Trotter Cashion / Chariot CI Event 2010
http://www.flickr.com/photos/gliuoo/2539512435/sizes/z/in/p...
SQUASHINGTHE BOHRBUGS
Trotter Cashion / Chariot CI Event 2010
http://www.flickr.com/photos/gliuoo/2539512435/sizes/z/in/pho...
THIS WILL BE ON THE INTERNET!
CO-FOUNDER
http://mashion.net
http://www.charlesheflin.com/wp-content/uploads/2010/06/fail-whale.jpg
@CASHION
http://gems.github.com/octocat.png
github.com/trotter
AGENDA
HEISENBUGS AND BOHRBUGS
THE MANUAL PROBLEM
CHEF
TYING CHEF INTO CI
QUESTIONS (AQAFT)
HEISENBUGS
http://upload.wikimedia.org/wikipedia/commons/8/84/Hindenburg_burning.jpg
OBSERVER EFFECT
EventYou
http://en.wikipedia.org/wiki/File:Bohr-atom-PAR.svg
BOHRBUGS
MANDELBUGS
SCHRÖDINBUGS
NOOBBUGS
THE MANUAL PROBLEM
http://www.dolphinsc.com/images/Dolphin.jpg
MYSQL
http://images.buycostumes.com/mgen/merchandiser/32494.jpg
POSTGRES
http://www.wallpapers-free.org/34/-/Apache_Longbow_Helicopter/
APACHE
http://www.allbestwallpapers.com/travel-kremlin_and_red_square_fireworks,_moscow,_russia_wallpapers.html
NGINX
Memcached
Memcached
Redis
Memcached
Redis
Mongodb
Memcached
Redis
Mongodb Riak
Memcached
Redis
Mongodb Riak
Cassandra
Memcached
Redis
Mongodb Riak
Cassandra
ProjectVoldemort
Memcached
Redis
Mongodb Riak
Cassandra
ProjectVoldemort
Ubuntu
Memcached
Redis
Mongodb Riak
Cassandra
ProjectVoldemort
Ubuntu
Redhat
Memcached
Redis
Mongodb Riak
Cassandra
ProjectVoldemort
Ubuntu
Redhat
Solaris
Memcached
Redis
Mongodb Riak
Cassandra
ProjectVoldemort
Ubuntu
Redhat
Solaris
iptables
Memcached
Redis
Mongodb Riak
Cassandra
ProjectVoldemort
Ubuntu
Redhat
Solaris
iptables
sshd
Memcached
Redis
Mongodb Riak
Cassandra
ProjectVoldemort
Ubuntu
Redhat
Solaris
iptables
sshd
DNS Servers
Memcached
Redis
Mongodb Riak
Cassandra
ProjectVoldemort
Ubuntu
Redhat
Solaris
iptables
sshd
DNS Servers
TCP/IP internals
We Idolize the Artisan Sysadmin
... but what do we end up with?
http://www.enviro-bc.ca/wp-content/uploads/2009/04/bowlake-aval_clair-israelson.jpg
SNOWFLAKES
... that become avalanches
http://upload.wikimedia.org/wikipedia/commons/6/6d/Niels_Bohr.jpg
BOHRBUGS
http://assets.troubledteenswizard.com/uploaded/WoW/nomoney1.jpg
NO MORE MONEY!
We’re really left with two choices
Automate... or die.
Automation is not hard!
BASH
http://hackedgadgets.com/wp-content/2/LEGO_V8_Engine_1.jpg
CFENGINE
http://www.flickr.com/photos/willemvelthoven/381608678/sizes/l/
PUPPET
http://www.flickr.com/photos/venndiagram/4891510333/sizes/l/
CHEF!
http://www.opscode.com/
WRITTEN BY
http://www.prestonlee.com/wp-content/uploads/2008/09/ruby.png
WRITTEN IN
Two Delicious Flavors
http://upload.wikimedia.org/wikipedia/commons/e/e0/Google%E2%80%99s_First_Production_Server.jpg
SERVER
HOW IT WORKS
HOW IT WORKS
Your
Computer
HOW IT WORKS
Your
Computer
Box to
Provision
HOW IT WORKS
Your
Computer
Chef Server
Box to
Provision
HOW IT WORKS
Your
Computer
Chef Server
Box to
Provision
Prepare
HOW IT WORKS
Your
Computer
Chef Server
Box to
Provision
Prepare
What now?
HOW IT WORKS
Your
Computer
Chef Server
Box to
Provision
Prepare
What now?
Cookbooks
HOW IT WORKS
Your
Computer
Chef Server
Box to
Provision
Prepare
What now?
Cookbooks
Run!
http://www.pbs.org/parents/supersisters/han_solo-thumb-454x654.jpg
SOLO
HOW IT WORKS
HOW IT WORKS
Your
Computer
HOW IT WORKS
Your
Computer
Box to
Provision
HOW IT WORKS
Your
Computer
Box to
Provision
Prepare
HOW IT WORKS
Your
Computer
Box to
Provision
Cookbooks
Prepare
HOW IT WORKS
Your
Computer
Box to
Provision
Cookbooks
Prepare
Go!
Chef Structure
CHEF HEIRARCHY
Node
Roles
Cookbooks
Cookbooks
Recipes Attributes Templates Definitions
ROLES ARE REUSABLE!
config/dev.json config/db.json
roles/db.rb
http://www.flickr.com/photos/seandreilinger/959864706/sizes/o/
GETTING
GOING
WHAT YOU NEED
WHAT YOU NEED
Ruby &
Rubygems
WHAT YOU NEED
Ruby &
Rubygems
`gem install chef` Chef Gem
WHAT YOU NEED
Ruby &
Rubygems
`gem install chef` Chef Gem
`git clone http://github.com/
opscode/chef-repo.git`
Chef Repo
http://cookbooks.opscode.com/
COOKBOOKS.OPSCODE.COM
TERMINAL JUNKIES
TERMINAL JUNKIES
•spatula search mysql
TERMINAL JUNKIES
•spatula search mysql
•spatula install mysql
TERMINAL JUNKIES
•spatula search mysql
•spatula install mysql
•spatula show mysql
TERMINAL JUNKIES
•spatula search mysql
•spatula install mysql
•spatula show mysql
•... knife has similar functionality
GET THE BOX READY
spatula prepare db-one.host
ssh db-one.host “install ruby”
ssh db-one.host “install chef”
=
GO GO GO!
spatula cook db-one.host db ssh db-one.host “sudo 
chef-solo -c config/solo.rb 
-j config/db.json”
rsync repo to d...
WITH CHEF SERVER
install dependencies
turn on client
send cookbooks to client
http://media.photobucket.com/image/minecraft/bluekabal/roma/minesofroma.jpg
DIGGING DEEPER
CHEF HEIRARCHY
Node
Roles
Cookbooks
Cookbooks
Recipes Attributes Templates Definitions
// ./config/db_one.json
{ "run_list": ["role[db]"] }
A NODE
Recipes and Roles go here
CHEF HEIRARCHY
Node
Roles
Cookbooks
Cookbooks
Recipes Attributes Templates Definitions
# ./roles/db.rb
name "database"
description "Be a DB"
run_list [ "apache2", "mysql::server" ]
A ROLE
# ./roles/db.rb
name "database"
description "Be a DB"
run_list [ "apache2", "mysql::server" ]
A ROLE
Recipes to run
# ./roles/db.rb
name "database"
description "Be a DB"
run_list [ "apache2", "mysql::server" ]
A ROLE
./cookbooks/apache2/r...
# ./roles/db.rb
name "database"
description "Be a DB"
run_list [ "apache2", "mysql::server" ]
A ROLE
./cookbooks/mysql/rec...
CHEF HEIRARCHY
Node
Roles
Cookbooks
Cookbooks
Recipes Attributes Templates Definitions
# ./cookbooks/mysql/attributes/server.rb
default[:mysql][:server_debian_password] = secure_password
default[:mysql][:serve...
// ./config/db.json
{ "mysql": { "server_root_password": "beefcake" },
  "recipes": [ "mysql" ] }
OVERRIDING ATTRIBUTES
# ...
CHEF HEIRARCHY
Node
Roles
Cookbooks
Cookbooks
Recipes Attributes Templates Definitions
# ./cookbooks/mysql/recipes/server.rb
include_recipe "mysql::client"
THE RECIPE... PART 1
package "mysql-server" do
action :install
end
THE RECIPE... PART 2
template "/etc/mysql/my.cnf" do
source "my.cnf.erb"
owner "root"
group "root"
mode "0644"
notifies :restart,
resources(:se...
CHEF HEIRARCHY
Node
Roles
Cookbooks
Cookbooks
Recipes Attributes Templates Definitions
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
datadir = <%= @datadir %>
bind-add...
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
datadir = <%= @datadir %>
bind-add...
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
datadir = <%= @datadir %>
bind-add...
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
datadir = <%= @datadir %>
bind-add...
template "/etc/mysql/my.cnf" do
source "my.cnf.erb"
owner "root"
group "root"
mode "0644"
variables :datadir => “/var/lib/...
CHEF HEIRARCHY
Node
Roles
Cookbooks
Cookbooks
Recipes Attributes Templates Definitions
define :apache_conf do
dir = node[:apache][:dir]
template "#{dir}/mods-available/#{params[:name]}.conf" do
source "mods/#{...
TYING IT INTO CI
USE CHEFTO PROVISION CI
http://images.nationalgeographic.com/wpf/media-live/photos/000/148/custom/heart-ngk0207_14824_470x...
attribution
CHEF AS PART OF TEST SCRIPT
CONTINUOUS DEPLOYMENT
http://www.flickr.com/photos/andresthor/3946773501/sizes/l/in/photostream/
LET’S
WRAP
IT UP
http://www.flickr.com/photos/kenfagerdotcom/3409813881/
AUTOMATE EVERYTHING!!
Deploying == Provisioning
http://www.flickr.com/photos/martin_borjesson/4083726805/sizes/o/
USE CHEF!!!!!
ONTHE INTERNETS
ONTHE INTERNETS
•twitter - @cashion
ONTHE INTERNETS
•twitter - @cashion
•github - http://github.com/trotter
ONTHE INTERNETS
•twitter - @cashion
•github - http://github.com/trotter
•email - cashion on the gmailz
http://www.flickr.com/photos/dancoulter/21042744/sizes/o/
THANK YOU!
RESOURCES
RESOURCES
•http://wiki.opscode.com/display/chef/Resources
RESOURCES
•http://wiki.opscode.com/display/chef/Resources
•#chef on freenode
RESOURCES
•http://wiki.opscode.com/display/chef/Resources
•#chef on freenode
•cashion@gmail.com
Squashing the Heisenbugs
Upcoming SlideShare
Loading in …5
×

Squashing the Heisenbugs

1,523 views
1,435 views

Published on

Squashing the Heisenbugs (really Bohrbugs) is a talk that I gave at the Chariot CI Event on December 1st, 2010. Feel free to ask me any questions about it via email or twitter. Also, please hire us if you're looking for any devops help.

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,523
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
12
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Squashing the Heisenbugs

  1. 1. SQUASHINGTHE HEISENBUGS Trotter Cashion / Chariot CI Event 2010 http://www.flickr.com/photos/gliuoo/2539512435/sizes/z/in/photostream/
  2. 2. SQUASHINGTHE HEISENBUGS Trotter Cashion / Chariot CI Event 2010 http://www.flickr.com/photos/gliuoo/2539512435/sizes/z/in/photostream/ BOHRBUGS
  3. 3. SQUASHINGTHE BOHRBUGS Trotter Cashion / Chariot CI Event 2010 http://www.flickr.com/photos/gliuoo/2539512435/sizes/z/in/photostream/
  4. 4. THIS WILL BE ON THE INTERNET!
  5. 5. CO-FOUNDER http://mashion.net
  6. 6. http://www.charlesheflin.com/wp-content/uploads/2010/06/fail-whale.jpg @CASHION
  7. 7. http://gems.github.com/octocat.png github.com/trotter
  8. 8. AGENDA
  9. 9. HEISENBUGS AND BOHRBUGS
  10. 10. THE MANUAL PROBLEM
  11. 11. CHEF
  12. 12. TYING CHEF INTO CI
  13. 13. QUESTIONS (AQAFT)
  14. 14. HEISENBUGS http://upload.wikimedia.org/wikipedia/commons/8/84/Hindenburg_burning.jpg
  15. 15. OBSERVER EFFECT EventYou
  16. 16. http://en.wikipedia.org/wiki/File:Bohr-atom-PAR.svg BOHRBUGS
  17. 17. MANDELBUGS
  18. 18. SCHRÖDINBUGS
  19. 19. NOOBBUGS
  20. 20. THE MANUAL PROBLEM
  21. 21. http://www.dolphinsc.com/images/Dolphin.jpg MYSQL
  22. 22. http://images.buycostumes.com/mgen/merchandiser/32494.jpg POSTGRES
  23. 23. http://www.wallpapers-free.org/34/-/Apache_Longbow_Helicopter/ APACHE
  24. 24. http://www.allbestwallpapers.com/travel-kremlin_and_red_square_fireworks,_moscow,_russia_wallpapers.html NGINX
  25. 25. Memcached
  26. 26. Memcached Redis
  27. 27. Memcached Redis Mongodb
  28. 28. Memcached Redis Mongodb Riak
  29. 29. Memcached Redis Mongodb Riak Cassandra
  30. 30. Memcached Redis Mongodb Riak Cassandra ProjectVoldemort
  31. 31. Memcached Redis Mongodb Riak Cassandra ProjectVoldemort Ubuntu
  32. 32. Memcached Redis Mongodb Riak Cassandra ProjectVoldemort Ubuntu Redhat
  33. 33. Memcached Redis Mongodb Riak Cassandra ProjectVoldemort Ubuntu Redhat Solaris
  34. 34. Memcached Redis Mongodb Riak Cassandra ProjectVoldemort Ubuntu Redhat Solaris iptables
  35. 35. Memcached Redis Mongodb Riak Cassandra ProjectVoldemort Ubuntu Redhat Solaris iptables sshd
  36. 36. Memcached Redis Mongodb Riak Cassandra ProjectVoldemort Ubuntu Redhat Solaris iptables sshd DNS Servers
  37. 37. Memcached Redis Mongodb Riak Cassandra ProjectVoldemort Ubuntu Redhat Solaris iptables sshd DNS Servers TCP/IP internals
  38. 38. We Idolize the Artisan Sysadmin
  39. 39. ... but what do we end up with?
  40. 40. http://www.enviro-bc.ca/wp-content/uploads/2009/04/bowlake-aval_clair-israelson.jpg SNOWFLAKES ... that become avalanches
  41. 41. http://upload.wikimedia.org/wikipedia/commons/6/6d/Niels_Bohr.jpg BOHRBUGS
  42. 42. http://assets.troubledteenswizard.com/uploaded/WoW/nomoney1.jpg NO MORE MONEY!
  43. 43. We’re really left with two choices
  44. 44. Automate... or die.
  45. 45. Automation is not hard!
  46. 46. BASH
  47. 47. http://hackedgadgets.com/wp-content/2/LEGO_V8_Engine_1.jpg CFENGINE
  48. 48. http://www.flickr.com/photos/willemvelthoven/381608678/sizes/l/ PUPPET
  49. 49. http://www.flickr.com/photos/venndiagram/4891510333/sizes/l/ CHEF!
  50. 50. http://www.opscode.com/ WRITTEN BY
  51. 51. http://www.prestonlee.com/wp-content/uploads/2008/09/ruby.png WRITTEN IN
  52. 52. Two Delicious Flavors
  53. 53. http://upload.wikimedia.org/wikipedia/commons/e/e0/Google%E2%80%99s_First_Production_Server.jpg SERVER
  54. 54. HOW IT WORKS
  55. 55. HOW IT WORKS Your Computer
  56. 56. HOW IT WORKS Your Computer Box to Provision
  57. 57. HOW IT WORKS Your Computer Chef Server Box to Provision
  58. 58. HOW IT WORKS Your Computer Chef Server Box to Provision Prepare
  59. 59. HOW IT WORKS Your Computer Chef Server Box to Provision Prepare What now?
  60. 60. HOW IT WORKS Your Computer Chef Server Box to Provision Prepare What now? Cookbooks
  61. 61. HOW IT WORKS Your Computer Chef Server Box to Provision Prepare What now? Cookbooks Run!
  62. 62. http://www.pbs.org/parents/supersisters/han_solo-thumb-454x654.jpg SOLO
  63. 63. HOW IT WORKS
  64. 64. HOW IT WORKS Your Computer
  65. 65. HOW IT WORKS Your Computer Box to Provision
  66. 66. HOW IT WORKS Your Computer Box to Provision Prepare
  67. 67. HOW IT WORKS Your Computer Box to Provision Cookbooks Prepare
  68. 68. HOW IT WORKS Your Computer Box to Provision Cookbooks Prepare Go!
  69. 69. Chef Structure
  70. 70. CHEF HEIRARCHY Node Roles Cookbooks Cookbooks Recipes Attributes Templates Definitions
  71. 71. ROLES ARE REUSABLE! config/dev.json config/db.json roles/db.rb
  72. 72. http://www.flickr.com/photos/seandreilinger/959864706/sizes/o/ GETTING GOING
  73. 73. WHAT YOU NEED
  74. 74. WHAT YOU NEED Ruby & Rubygems
  75. 75. WHAT YOU NEED Ruby & Rubygems `gem install chef` Chef Gem
  76. 76. WHAT YOU NEED Ruby & Rubygems `gem install chef` Chef Gem `git clone http://github.com/ opscode/chef-repo.git` Chef Repo
  77. 77. http://cookbooks.opscode.com/ COOKBOOKS.OPSCODE.COM
  78. 78. TERMINAL JUNKIES
  79. 79. TERMINAL JUNKIES •spatula search mysql
  80. 80. TERMINAL JUNKIES •spatula search mysql •spatula install mysql
  81. 81. TERMINAL JUNKIES •spatula search mysql •spatula install mysql •spatula show mysql
  82. 82. TERMINAL JUNKIES •spatula search mysql •spatula install mysql •spatula show mysql •... knife has similar functionality
  83. 83. GET THE BOX READY spatula prepare db-one.host ssh db-one.host “install ruby” ssh db-one.host “install chef” =
  84. 84. GO GO GO! spatula cook db-one.host db ssh db-one.host “sudo chef-solo -c config/solo.rb -j config/db.json” rsync repo to db-one.host =
  85. 85. WITH CHEF SERVER install dependencies turn on client send cookbooks to client
  86. 86. http://media.photobucket.com/image/minecraft/bluekabal/roma/minesofroma.jpg DIGGING DEEPER
  87. 87. CHEF HEIRARCHY Node Roles Cookbooks Cookbooks Recipes Attributes Templates Definitions
  88. 88. // ./config/db_one.json { "run_list": ["role[db]"] } A NODE Recipes and Roles go here
  89. 89. CHEF HEIRARCHY Node Roles Cookbooks Cookbooks Recipes Attributes Templates Definitions
  90. 90. # ./roles/db.rb name "database" description "Be a DB" run_list [ "apache2", "mysql::server" ] A ROLE
  91. 91. # ./roles/db.rb name "database" description "Be a DB" run_list [ "apache2", "mysql::server" ] A ROLE Recipes to run
  92. 92. # ./roles/db.rb name "database" description "Be a DB" run_list [ "apache2", "mysql::server" ] A ROLE ./cookbooks/apache2/recipes/default.rb
  93. 93. # ./roles/db.rb name "database" description "Be a DB" run_list [ "apache2", "mysql::server" ] A ROLE ./cookbooks/mysql/recipes/server.rb
  94. 94. CHEF HEIRARCHY Node Roles Cookbooks Cookbooks Recipes Attributes Templates Definitions
  95. 95. # ./cookbooks/mysql/attributes/server.rb default[:mysql][:server_debian_password] = secure_password default[:mysql][:server_root_password] = secure_password default[:mysql][:server_repl_password] = secure_password default[:mysql][:bind_address] = ipaddress default[:mysql][:datadir] = "/var/lib/mysql" # Tunables default[:mysql][:tunable][:key_buffer] = "250M" default[:mysql][:tunable][:max_connections] = "800" default[:mysql][:tunable][:wait_timeout] = "180" ATTRIBUTES
  96. 96. // ./config/db.json { "mysql": { "server_root_password": "beefcake" },   "recipes": [ "mysql" ] } OVERRIDING ATTRIBUTES # ./roles/db.rb name "database" description "Be a DB" run_list [ "mysql::server" ] default_attributes :mysql => { :server_root_password => "beefcake" } Node Role
  97. 97. CHEF HEIRARCHY Node Roles Cookbooks Cookbooks Recipes Attributes Templates Definitions
  98. 98. # ./cookbooks/mysql/recipes/server.rb include_recipe "mysql::client" THE RECIPE... PART 1
  99. 99. package "mysql-server" do action :install end THE RECIPE... PART 2
  100. 100. template "/etc/mysql/my.cnf" do source "my.cnf.erb" owner "root" group "root" mode "0644" notifies :restart, resources(:service => "mysql"), :immediately end THE RECIPE... PART 3
  101. 101. CHEF HEIRARCHY Node Roles Cookbooks Cookbooks Recipes Attributes Templates Definitions
  102. 102. user = mysql pid-file = /var/run/mysqld/mysqld.pid socket = /var/run/mysqld/mysqld.sock datadir = <%= @datadir %> bind-address = <%= @node[:mysql][:bind_address] %> A TEMPLATE SAMPLE
  103. 103. user = mysql pid-file = /var/run/mysqld/mysqld.pid socket = /var/run/mysqld/mysqld.sock datadir = <%= @datadir %> bind-address = <%= @node[:mysql][:bind_address] %> A TEMPLATE SAMPLE OMG, ERB!!
  104. 104. user = mysql pid-file = /var/run/mysqld/mysqld.pid socket = /var/run/mysqld/mysqld.sock datadir = <%= @datadir %> bind-address = <%= @node[:mysql][:bind_address] %> A TEMPLATE SAMPLE From Attributes
  105. 105. user = mysql pid-file = /var/run/mysqld/mysqld.pid socket = /var/run/mysqld/mysqld.sock datadir = <%= @datadir %> bind-address = <%= @node[:mysql][:bind_address] %> A TEMPLATE SAMPLE FromTemplate
  106. 106. template "/etc/mysql/my.cnf" do source "my.cnf.erb" owner "root" group "root" mode "0644" variables :datadir => “/var/lib/mysql” end ASSIGNING VARIABLES
  107. 107. CHEF HEIRARCHY Node Roles Cookbooks Cookbooks Recipes Attributes Templates Definitions
  108. 108. define :apache_conf do dir = node[:apache][:dir] template "#{dir}/mods-available/#{params[:name]}.conf" do source "mods/#{params[:name]}.conf.erb" notifies :restart, resources(:service => "apache2") end end # In your recipe apache_conf "mod_ssl" DEFINITIONS
  109. 109. TYING IT INTO CI
  110. 110. USE CHEFTO PROVISION CI http://images.nationalgeographic.com/wpf/media-live/photos/000/148/custom/heart-ngk0207_14824_470x300.jpg
  111. 111. attribution CHEF AS PART OF TEST SCRIPT
  112. 112. CONTINUOUS DEPLOYMENT
  113. 113. http://www.flickr.com/photos/andresthor/3946773501/sizes/l/in/photostream/ LET’S WRAP IT UP
  114. 114. http://www.flickr.com/photos/kenfagerdotcom/3409813881/ AUTOMATE EVERYTHING!!
  115. 115. Deploying == Provisioning
  116. 116. http://www.flickr.com/photos/martin_borjesson/4083726805/sizes/o/ USE CHEF!!!!!
  117. 117. ONTHE INTERNETS
  118. 118. ONTHE INTERNETS •twitter - @cashion
  119. 119. ONTHE INTERNETS •twitter - @cashion •github - http://github.com/trotter
  120. 120. ONTHE INTERNETS •twitter - @cashion •github - http://github.com/trotter •email - cashion on the gmailz
  121. 121. http://www.flickr.com/photos/dancoulter/21042744/sizes/o/ THANK YOU!
  122. 122. RESOURCES
  123. 123. RESOURCES •http://wiki.opscode.com/display/chef/Resources
  124. 124. RESOURCES •http://wiki.opscode.com/display/chef/Resources •#chef on freenode
  125. 125. RESOURCES •http://wiki.opscode.com/display/chef/Resources •#chef on freenode •cashion@gmail.com

×