Michelin StarredCooking with Chef    Jon Cowie, Etsy.com      jcowie@etsy.com          @jonlives
What?
What?• Chef at Etsy
What?• Chef at Etsy• Familiarity and Understanding
What?• Chef at Etsy• Familiarity and Understanding• Critical Approach and Experimentation
What?• Chef at Etsy• Familiarity and Understanding• Critical Approach and Experimentation• Use The Source
What?• Chef at Etsy• Familiarity and Understanding• Critical Approach and Experimentation• Use The Source• A liberal sprin...
What?• Chef at Etsy• Familiarity and Understanding• Critical Approach and Experimentation• Use The Source• A liberal sprin...
What?• Chef at Etsy• Familiarity and Understanding• Critical Approach and Experimentation• Use The Source• A liberal sprin...
Opscode is Orange,  Velocity is Blue.  In Soviet Russia,Cookbook writes you.
Chef at Etsy
Our Setup
Our Setup• Open Source chef server 0.10.4
Our Setup• Open Source chef server 0.10.4• Backup to Opscode Platform
Our Setup• Open Source chef server 0.10.4• Backup to Opscode Platform• ~800 self-hosted nodes (Mainly CentOS,  some RHEL &...
Our Setup• Open Source chef server 0.10.4• Backup to Opscode Platform• ~800 self-hosted nodes (Mainly CentOS,  some RHEL &...
Our Setup• Open Source chef server 0.10.4• Backup to Opscode Platform• ~800 self-hosted nodes (Mainly CentOS,  some RHEL &...
Our Setup• Open Source chef server 0.10.4• Backup to Opscode Platform• ~800 self-hosted nodes (Mainly CentOS,  some RHEL &...
Our Setup• Open Source chef server 0.10.4• Backup to Opscode Platform• ~800 self-hosted nodes (Mainly CentOS,  some RHEL &...
Familiarity andUnderstanding
nothttp://www.flickr.com/photos/photo_secessionist/5555167113/         (C) Alexander McQueen
• Insight
• Insight• Simplicity
• Insight• Simplicity• Standards
Insight
Insight• You should never have to say “I don’t  know.”
Insight• You should never have to say “I don’t  know.”• What, where, when, why, how long?
Insight• You should never have to say “I don’t  know.”• What, where, when, why, how long?• Easy, and I’ll show you how!
Chef Dashboard
Chef Dashboard•   Chef handler sends metrics to graphite [4]
Chef Dashboard•   Chef handler sends metrics to graphite [4]    • git clone git://github.com/etsy/chef-handlers.git
Chef Dashboard•   Chef handler sends metrics to graphite [4]    • git clone git://github.com/etsy/chef-handlers.git    •  ...
Chef Dashboard•   Chef handler sends metrics to graphite [4]    • git clone git://github.com/etsy/chef-handlers.git    •  ...
Chef Dashboard•   Chef handler sends metrics to graphite [4]    • git clone git://github.com/etsy/chef-handlers.git    •  ...
Chef Dashboard•   Chef handler sends metrics to graphite [4]    • git clone git://github.com/etsy/chef-handlers.git    •  ...
[12:54:01] <irccat> Chef run failed on gfernandez.vm.ny4dev.etsy.com[12:54:02] <irccat> https://github.etsycorp.com/gist/3...
Chef irccat Alerts
Chef irccat Alerts•   Chef handler send fails to irc[4] via irccat [6]
Chef irccat Alerts•   Chef handler send fails to irc[4] via irccat [6]    • git clone git://github.com/etsy/chef-handlers....
Chef irccat Alerts•   Chef handler send fails to irc[4] via irccat [6]    • git clone git://github.com/etsy/chef-handlers....
Chef irccat Alerts•   Chef handler send fails to irc[4] via irccat [6]    • git clone git://github.com/etsy/chef-handlers....
Chef irccat Alerts•   Chef handler send fails to irc[4] via irccat [6]    • git clone git://github.com/etsy/chef-handlers....
~ > knife node lastrun buildtest11.ny4dev.etsy.comStatus                     failedElapsed Time               4.628171438S...
~ > knife search node lastrun_debug_formatted_exception:Chef::Exceptions::Package* -a lastrun.debug.formatted_exception5 i...
Chef lastrun Info
Chef lastrun Info•   Chef handler and knife plugin [7]
Chef lastrun Info•   Chef handler and knife plugin [7]    •   gem install knife-lastrun
Chef lastrun Info•   Chef handler and knife plugin [7]    •   gem install knife-lastrun    •   Add the following to client...
Chef lastrun Info•   Chef handler and knife plugin [7]    •   gem install knife-lastrun    •   Add the following to client...
Chef lastrun Info•   Chef handler and knife plugin [7]    •   gem install knife-lastrun    •   Add the following to client...
Simplicity
Simplicity• Think of yourself at 3AM!
Simplicity• Think of yourself at 3AM!• Please, won’t you think of the new guy?
Simplicity• Think of yourself at 3AM!• Please, won’t you think of the new guy?• Minimize the logics!
Simplicity• Think of yourself at 3AM!• Please, won’t you think of the new guy?• Minimize the logics! • As few logical step...
Simplicity - Not!   Date: Mon Dec 05 2011 23:07:18 GMT+0000 (GMT)Subject: so close to death# Dont install v2 on search or ...
Simplicity - Better!if node.chef_environment == "libmemcached_upgrade"  package "libmemcached" do       version "1.0.4-1" ...
Simplicity - Complexity
Simplicity - Complexity• Sometimes you need complex behaviour
Simplicity - Complexity• Sometimes you need complex behaviour• Don’t fight it, try to abstract it.
Simplicity - Complexity• Sometimes you need complex behaviour• Don’t fight it, try to abstract it.• Case in point: Syslog-n...
Case Study: Syslog-ng
Case Study: Syslog-ng• 36 recipes
Case Study: Syslog-ng• 36 recipes• 30 versions of syslog-ng.conf
Case Study: Syslog-ng• 36 recipes• 30 versions of syslog-ng.conf• 27 manually configured files in /etc/syslog-  ng.d on cent...
Case Study: Syslog-ng• 36 recipes• 30 versions of syslog-ng.conf• 27 manually configured files in /etc/syslog-  ng.d on cent...
Case Study: Syslog-ng
Case Study: Syslog-ng• Down to:
Case Study: Syslog-ng• Down to: • 2 recipes (one client, one server)
Case Study: Syslog-ng• Down to: • 2 recipes (one client, one server) • 2 templates (one for syslog-ng.conf, one    for stu...
Case Study: Syslog-ng• Down to: • 2 recipes (one client, one server) • 2 templates (one for syslog-ng.conf, one    for stu...
Case Study: Syslog-ng• Down to: • 2 recipes (one client, one server) • 2 templates (one for syslog-ng.conf, one    for stu...
Case Study: Syslog-ng"syslog": {    "group": "preprod_web",    "items": {         "web_apache_access_log": {              ...
Remember, No  Panacea!
Remember, No       Panacea!• A new package hits the repo.
Remember, No       Panacea!• A new package hits the repo. • Are you in control of when it goes out?
Remember, No       Panacea!• A new package hits the repo. • Are you in control of when it goes out?   • Memcached Outage
Remember, No       Panacea!• A new package hits the repo. • Are you in control of when it goes out?   • Memcached Outage• ...
Remember, No       Panacea!• A new package hits the repo. • Are you in control of when it goes out?   • Memcached Outage• ...
Standards
Standards• Not going to talk about testing [8]!
Standards• Not going to talk about testing [8]!• But I don’t have time for standards!
Standards - No Time!
Standards - No Time!• I won’t say “Make Time”, but you should...
Standards - No Time!• I won’t say “Make Time”, but you should...• For a quick win, try Foodcritic
Standards - No Time!• I won’t say “Make Time”, but you should...• For a quick win, try Foodcritic • Good out of the box ru...
Standards - No Time!• I won’t say “Make Time”, but you should...• For a quick win, try Foodcritic • Good out of the box ru...
Standards - No Time!• I won’t say “Make Time”, but you should...• For a quick win, try Foodcritic • Good out of the box ru...
Standards - No Time!• I won’t say “Make Time”, but you should...• For a quick win, try Foodcritic • Good out of the box ru...
Foodcritic
Foodcritic•gem install foodcritic
Foodcritic•gem install foodcritic•foodcritic <cookbook_repo>
Foodcritic•gem install foodcritic•foodcritic <cookbook_repo>•Simple Jenkins job:
Foodcritic•gem install foodcritic•foodcritic <cookbook_repo>•Simple Jenkins job:#!/usr/bin/env rvm-shell 1.9.3foodcritic -...
Standards at Etsy
Standards at Etsy• “style” not “correctness”[9]
Standards at Etsy• “style” not “correctness”[9] •   ETSY001 - Package or yum_package resource used with :upgrade action
Standards at Etsy• “style” not “correctness”[9] •   ETSY001 - Package or yum_package resource used with :upgrade action • ...
Standards at Etsy• “style” not “correctness”[9] •   ETSY001 - Package or yum_package resource used with :upgrade action • ...
Standards at Etsy• “style” not “correctness”[9] •   ETSY001 - Package or yum_package resource used with :upgrade action • ...
Standards at Etsy• “style” not “correctness”[9] •   ETSY001 - Package or yum_package resource used with :upgrade action • ...
Standards at Etsy• “style” not “correctness”[9] •   ETSY001 - Package or yum_package resource used with :upgrade action • ...
Standards at Etsy• “style” not “correctness”[9] •   ETSY001 - Package or yum_package resource used with :upgrade action • ...
Standards at Etsy• “style” not “correctness”[9] •   ETSY001 - Package or yum_package resource used with :upgrade action • ...
Standards at Etsy
Standards at Etsy• ETSY001 - Written after Memcached  Outage
Standards at Etsy• ETSY001 - Written after Memcached  Outage• Package or yum_package resource used  with :upgrade action
Standards at Etsy• ETSY001 - Written after Memcached  Outage• Package or yum_package resource used  with :upgrade action• ...
Standards at Etsy
Standards at Etsy• ETSY005 - Written after a total Image  service outage
Standards at Etsy• ETSY005 - Written after a total Image  service outage• Action :restart sent to a core service
Standards at Etsy• ETSY005 - Written after a total Image    service outage• Action :restart sent to a core service•   cook...
Critical Approach and  Experimentation
CA&E
CA&E• Chef is by necessity generic
CA&E• Chef is by necessity generic• ...so don’t take Opscode’s word for it.
CA&E• Chef is by necessity generic• ...so don’t take Opscode’s word for it.• If it doesn’t work well for you, change it!
CA&E• Chef is by necessity generic• ...so don’t take Opscode’s word for it.• If it doesn’t work well for you, change it!• ...
Environments Rollout
Environments Rollout• We knew we needed them
Environments Rollout• We knew we needed them• Simple enough, right?
Environments Rollout• We knew we needed them• Simple enough, right?• Let’s look at the workflow...
Env: Standard Workflow
Env: Standard Workflow • knife cookbook show php
Env: Standard Workflow • knife cookbook show php • Change version number in metadata.rb
Env: Standard Workflow • knife cookbook show php • Change version number in metadata.rb • Change version constraint in foo....
Env: Standard Workflow • knife cookbook show php • Change version number in metadata.rb • Change version constraint in foo....
Env: Standard Workflow • knife cookbook show php • Change version number in metadata.rb • Change version constraint in foo....
Env: Standard Workflow • knife cookbook show php • Change version number in metadata.rb • Change version constraint in foo....
Env: Our Issues
Env: Our Issues• 41 people with Chef repo access
Env: Our Issues• 41 people with Chef repo access• Most with knife keys
Env: Our Issues• 41 people with Chef repo access• Most with knife keys• All editing the same 2 files with every  change...
Env: Solutions?
Env: Solutions?• Go with it and hope for the best?
Env: Solutions?• Go with it and hope for the best?• Don’t use environments?
Env: Solutions?• Go with it and hope for the best?• Don’t use environments?• Write a totally new workflow?
Env: Solutions?• Go with it and hope for the best?• Don’t use environments?• Write a totally new workflow?• Tweak the exist...
Env: Solution!
Env: Solution!• Tweak with some tooling!
Env: Solution!• Tweak with some tooling!• Presenting knife-spork [10]...
Spork: Workflow
Spork: Workflow• Wrapper around standard environments  workflow
Spork: Workflow• Wrapper around standard environments  workflow • check - cookbook versioning
Spork: Workflow• Wrapper around standard environments  workflow • check - cookbook versioning • bump - increment version com...
Spork: Workflow• Wrapper around standard environments  workflow • check - cookbook versioning • bump - increment version com...
Spork: Workflow• Wrapper around standard environments  workflow • check - cookbook versioning • bump - increment version com...
Spork: Check$> knife spork check apache2Checking versions for cookbook apache2...Current local version: 1.0.6Remote versio...
Spork: Bump$> knife spork bump apache2 <major|minor|patch|manual>Bumping patch level of the apache2 cookbook from 1.0.6to ...
Spork: Upload$> knife spork upload apache2Uploading and freezing apache2   [1.0.7]upload complete
Spork: Promote$> knife spork promote foo phpAdding version constraint php = 1.0.6Saving changes into foo.jsonPromotion com...
Spork
Spork• Worked well, avoided the issues it was  designed for
Spork• Worked well, avoided the issues it was  designed for• Subsequently evolved
Spork• Worked well, avoided the issues it was  designed for• Subsequently evolved • A lot of input & work came from Devs
Spork• Worked well, avoided the issues it was  designed for• Subsequently evolved • A lot of input & work came from Devs •...
Spork• Worked well, avoided the issues it was  designed for• Subsequently evolved • A lot of input & work came from Devs •...
Spork: Evolution
Spork: Evolution• Safety Checks
Spork: Evolution• Safety Checks• Extra Features
Spork: Safety Checks
Spork: Safety Checks• Before promoting, check version is  uploaded...
Spork: Safety Checks$> knife spork promote php --remote<snip>WARNING: It looks like you have multiple cookbook pathsdefine...
Spork: Safety Checks• Before promoting, check version is  uploaded...• Check if you’re promoting changes to more  than you...
Spork: Safety ChecksWARNING: Youre about to promote changes to severalcookbooks:WARNING:ganglia: = 0.1.26 changed to = 0.1...
Spork: Features
Spork: Features• Default Environments
Spork: Features• Default Environments• Git support
Spork: Features• Default Environments• Git support• Chat notifications + Gist (upload and  promote)
Spork: Features• Default Environments• Git support• Chat notifications + Gist (upload and  promote) • IRCCat
Spork: Features• Default Environments• Git support• Chat notifications + Gist (upload and  promote) • IRCCat • Hipchat (cou...
Features: Chat          Notifications[19:43:46] <irccat> CHEF: pmcdonnell uploaded and frozecookbook immount version 0.0.24...
Spork: Features• Default Environments• Git support• Chat notifications + Gist (upload and  promote) • IRCCat • Hipchat (cou...
Features: Graphite
Spork: Features• Default Environments• Git support• Chat notifications + Gist (upload and  promote) • IRCCat • Hipchat (cou...
Features: Foodcritic$> knife spork upload system<snip>Lint checking system...ERROR: Lint check failed. Halting upload.ERRO...
Use the Source!tiny.cc/velocity2012
We’re hiring!     BoF on Tuesdayor just come and say Hi :)
Michelin Starred Cooking with Chef
Michelin Starred Cooking with Chef
Upcoming SlideShare
Loading in...5
×

Michelin Starred Cooking with Chef

9,949

Published on

The slides from my Velocity 2012 tutorial, Michelin Starred Cooking with Chef

Published in: Technology, Business
0 Comments
12 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
9,949
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
86
Comments
0
Likes
12
Embeds 0
No embeds

No notes for slide
  • \n
  • \n\n
  • \n\n
  • \n\n
  • \n\n
  • \n\n
  • \n\n
  • \n\n
  • \nTHERE IS NO PANACEA. You&amp;#x2019;ll go mad if you go looking for one.\n
  • \n
  • \nWe prefer bare metal for user facing stuff, but we do use &amp;#x201C;the cloud&amp;#x201D; when the shoe fits\n
  • \nWe prefer bare metal for user facing stuff, but we do use &amp;#x201C;the cloud&amp;#x201D; when the shoe fits\n
  • \nWe prefer bare metal for user facing stuff, but we do use &amp;#x201C;the cloud&amp;#x201D; when the shoe fits\n
  • \nWe prefer bare metal for user facing stuff, but we do use &amp;#x201C;the cloud&amp;#x201D; when the shoe fits\n
  • \nWe prefer bare metal for user facing stuff, but we do use &amp;#x201C;the cloud&amp;#x201D; when the shoe fits\n
  • \nWe prefer bare metal for user facing stuff, but we do use &amp;#x201C;the cloud&amp;#x201D; when the shoe fits\n
  • \nWe prefer bare metal for user facing stuff, but we do use &amp;#x201C;the cloud&amp;#x201D; when the shoe fits\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • METRIC: defaults to chef.runs.hostname.&lt;metric&gt;, totally configurable\n\nWORKS WITH OPSCODE PLATFORM!\n
  • METRIC: defaults to chef.runs.hostname.&lt;metric&gt;, totally configurable\n\nWORKS WITH OPSCODE PLATFORM!\n
  • METRIC: defaults to chef.runs.hostname.&lt;metric&gt;, totally configurable\n\nWORKS WITH OPSCODE PLATFORM!\n
  • METRIC: defaults to chef.runs.hostname.&lt;metric&gt;, totally configurable\n\nWORKS WITH OPSCODE PLATFORM!\n
  • METRIC: defaults to chef.runs.hostname.&lt;metric&gt;, totally configurable\n\nWORKS WITH OPSCODE PLATFORM!\n
  • METRIC: defaults to chef.runs.hostname.&lt;metric&gt;, totally configurable\n\nWORKS WITH OPSCODE PLATFORM!\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n\n
  • \n\n
  • \n\n
  • \n\n
  • \n\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Filter name clashes\n
  • Filter name clashes\n
  • Filter name clashes\n
  • Filter name clashes\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Michelin Starred Cooking with Chef

    1. 1. Michelin StarredCooking with Chef Jon Cowie, Etsy.com jcowie@etsy.com @jonlives
    2. 2. What?
    3. 3. What?• Chef at Etsy
    4. 4. What?• Chef at Etsy• Familiarity and Understanding
    5. 5. What?• Chef at Etsy• Familiarity and Understanding• Critical Approach and Experimentation
    6. 6. What?• Chef at Etsy• Familiarity and Understanding• Critical Approach and Experimentation• Use The Source
    7. 7. What?• Chef at Etsy• Familiarity and Understanding• Critical Approach and Experimentation• Use The Source• A liberal sprinkling of screwups
    8. 8. What?• Chef at Etsy• Familiarity and Understanding• Critical Approach and Experimentation• Use The Source• A liberal sprinkling of screwups• Open Sourced Goodness - We’re all here!
    9. 9. What?• Chef at Etsy• Familiarity and Understanding• Critical Approach and Experimentation• Use The Source• A liberal sprinkling of screwups• Open Sourced Goodness - We’re all here! • [x] = http://tiny.cc/velocity2012
    10. 10. Opscode is Orange, Velocity is Blue. In Soviet Russia,Cookbook writes you.
    11. 11. Chef at Etsy
    12. 12. Our Setup
    13. 13. Our Setup• Open Source chef server 0.10.4
    14. 14. Our Setup• Open Source chef server 0.10.4• Backup to Opscode Platform
    15. 15. Our Setup• Open Source chef server 0.10.4• Backup to Opscode Platform• ~800 self-hosted nodes (Mainly CentOS, some RHEL & mac)
    16. 16. Our Setup• Open Source chef server 0.10.4• Backup to Opscode Platform• ~800 self-hosted nodes (Mainly CentOS, some RHEL & mac)• KVM & lxc virts, self hosted too.
    17. 17. Our Setup• Open Source chef server 0.10.4• Backup to Opscode Platform• ~800 self-hosted nodes (Mainly CentOS, some RHEL & mac)• KVM & lxc virts, self hosted too.• Never test in production!
    18. 18. Our Setup• Open Source chef server 0.10.4• Backup to Opscode Platform• ~800 self-hosted nodes (Mainly CentOS, some RHEL & mac)• KVM & lxc virts, self hosted too.• Never test in production!• Many chefs don’t spoil our soup
    19. 19. Our Setup• Open Source chef server 0.10.4• Backup to Opscode Platform• ~800 self-hosted nodes (Mainly CentOS, some RHEL & mac)• KVM & lxc virts, self hosted too.• Never test in production!• Many chefs don’t spoil our soup
    20. 20. Familiarity andUnderstanding
    21. 21. nothttp://www.flickr.com/photos/photo_secessionist/5555167113/ (C) Alexander McQueen
    22. 22. • Insight
    23. 23. • Insight• Simplicity
    24. 24. • Insight• Simplicity• Standards
    25. 25. Insight
    26. 26. Insight• You should never have to say “I don’t know.”
    27. 27. Insight• You should never have to say “I don’t know.”• What, where, when, why, how long?
    28. 28. Insight• You should never have to say “I don’t know.”• What, where, when, why, how long?• Easy, and I’ll show you how!
    29. 29. Chef Dashboard
    30. 30. Chef Dashboard• Chef handler sends metrics to graphite [4]
    31. 31. Chef Dashboard• Chef handler sends metrics to graphite [4] • git clone git://github.com/etsy/chef-handlers.git
    32. 32. Chef Dashboard• Chef handler sends metrics to graphite [4] • git clone git://github.com/etsy/chef-handlers.git • Set your graphite server’s URL in graphite.rb
    33. 33. Chef Dashboard• Chef handler sends metrics to graphite [4] • git clone git://github.com/etsy/chef-handlers.git • Set your graphite server’s URL in graphite.rb • Add the following to client.rb
    34. 34. Chef Dashboard• Chef handler sends metrics to graphite [4] • git clone git://github.com/etsy/chef-handlers.git • Set your graphite server’s URL in graphite.rb • Add the following to client.rb • require "<clonedir>/graphite.rb" graphite_handler = GraphiteReporting.new report_handlers << graphite_handler exception_handlers << graphite_handler
    35. 35. Chef Dashboard• Chef handler sends metrics to graphite [4] • git clone git://github.com/etsy/chef-handlers.git • Set your graphite server’s URL in graphite.rb • Add the following to client.rb • require "<clonedir>/graphite.rb" graphite_handler = GraphiteReporting.new report_handlers << graphite_handler exception_handlers << graphite_handler• Etsy dashboards framework [5]
    36. 36. [12:54:01] <irccat> Chef run failed on gfernandez.vm.ny4dev.etsy.com[12:54:02] <irccat> https://github.etsycorp.com/gist/384228[12:54:02] <irccat>[12:54:07] <irccat> Chef run failed on buildtest11.ny4dev.etsy.com[12:54:07] <irccat> https://github.etsycorp.com/gist/384227[12:54:07] <irccat>
    37. 37. Chef irccat Alerts
    38. 38. Chef irccat Alerts• Chef handler send fails to irc[4] via irccat [6]
    39. 39. Chef irccat Alerts• Chef handler send fails to irc[4] via irccat [6] • git clone git://github.com/etsy/chef-handlers.git
    40. 40. Chef irccat Alerts• Chef handler send fails to irc[4] via irccat [6] • git clone git://github.com/etsy/chef-handlers.git • Set your irccat[6] server’s URL in logtoirc.rb
    41. 41. Chef irccat Alerts• Chef handler send fails to irc[4] via irccat [6] • git clone git://github.com/etsy/chef-handlers.git • Set your irccat[6] server’s URL in logtoirc.rb • Add the following to client.rb
    42. 42. Chef irccat Alerts• Chef handler send fails to irc[4] via irccat [6] • git clone git://github.com/etsy/chef-handlers.git • Set your irccat[6] server’s URL in logtoirc.rb • Add the following to client.rb • require "<clonedir>/logtoirc.rb" exception_handlers << Etsy::LogToIRC.new
    43. 43. ~ > knife node lastrun buildtest11.ny4dev.etsy.comStatus failedElapsed Time 4.628171438Start Time 2012-06-18 10:06:28 +0000End Time 2012-06-18 10:06:32 +0000Recipe Action Resource Type ResourceBacktrace<snip>ExceptionChef::Exceptions::Package: package[php] (php::buildtest line 20) hadan error: Version 5.3.10-1.el5 of php not found. Did you specify bothversion and release? (version-release, e.g. 1.84-10.fc6)
    44. 44. ~ > knife search node lastrun_debug_formatted_exception:Chef::Exceptions::Package* -a lastrun.debug.formatted_exception5 items foundid: masterrestore.ny4.etsy.comlastrun.debug.formatted_exception: Chef::Exceptions::Package:package[postgresql-server] (postgresql::server-8.3 line 1) had anerror: Installed package postgresql-server-8.3.16-1PGDG_id is newerthan candidate package postgresql-server-8.3.11-1PGDG_id.rhel5id: buildtest11.ny4dev.etsy.comlastrun.debug.formatted_exception: Chef::Exceptions::Package:package[php] (php::buildtest line 20) had an error: Version5.3.10-1.el5 of php not found. Did you specify both version andrelease? (version-release, e.g. 1.84-10.fc6)<snip>
    45. 45. Chef lastrun Info
    46. 46. Chef lastrun Info• Chef handler and knife plugin [7]
    47. 47. Chef lastrun Info• Chef handler and knife plugin [7] • gem install knife-lastrun
    48. 48. Chef lastrun Info• Chef handler and knife plugin [7] • gem install knife-lastrun • Add the following to client.rb
    49. 49. Chef lastrun Info• Chef handler and knife plugin [7] • gem install knife-lastrun • Add the following to client.rb • require "lastrun_update" handler = LastRunUpdateHandler.new report_handlers << handler exception_handlers << handler
    50. 50. Chef lastrun Info• Chef handler and knife plugin [7] • gem install knife-lastrun • Add the following to client.rb • require "lastrun_update" handler = LastRunUpdateHandler.new report_handlers << handler exception_handlers << handler • knife node lastrun <nodename>
    51. 51. Simplicity
    52. 52. Simplicity• Think of yourself at 3AM!
    53. 53. Simplicity• Think of yourself at 3AM!• Please, won’t you think of the new guy?
    54. 54. Simplicity• Think of yourself at 3AM!• Please, won’t you think of the new guy?• Minimize the logics!
    55. 55. Simplicity• Think of yourself at 3AM!• Please, won’t you think of the new guy?• Minimize the logics! • As few logical steps from start to finish as possible.
    56. 56. Simplicity - Not! Date: Mon Dec 05 2011 23:07:18 GMT+0000 (GMT)Subject: so close to death# Dont install v2 on search or Cent 5.6 nodes-if node[:fqdn] !~ /b(^(preprod-)?search[0-9]{2}|ny4dev.etsy.com|^(preprod-)?giftsweb[0-9]{2}|^db(shard|spare|data)[0-9]{2}|^qa-web01|^devsearch[0-9]{2}|^nagios01|^webnest[0-9]{2}|^prodking[0-9]{2}|^sandboxweb[0-9]{2}|^virt((0[5-9])|(1[0-9]))|^msysmgr[0-9]{2}|^msysmta[0-9]{2}|^dbconvo[0-9]{2}|^dbshowcase01|atlasweb[0-9]{2}|devnagios[0-9]{2}|cimaster02|worker[0-9]{2}|^ganglia[0-9]{2}|^imgcache[0-9]{2}|imgconvert[0-9]{2}|^imgwriter[0-9]{2}|dev-img02|^datacache04|^graphite01|^graphite03|^webutil03|^webutil04|^statsd01|^maintweb[0-9]{2}|^(dev-|preprod-)?convosearch[0-9]{2}|deployinator[0-9]{2}|^wpadmin01|^(preprod-)?dbtasks[0-9]{2})b/ and node.role?("Web56") ==false and node.role?("Preprodweb56") == false and node.role?("Princess53") == false+if node[:fqdn] !~ /b(^(preprod-)?search[0-9]{2}|ny4dev.etsy.com|^(preprod-)?giftsweb[0-9]{2}|^db(shard|spare|data)[0-9]{2}|^qa-web01|^devsearch[0-9]{2}|^nagios01|^webnest[0-9]{2}|^prodking[0-9]{2}|^sandboxweb[0-9]{2}|^virt((0[5-9])|(1[0-9]))|^msysmgr[0-9]{2}|^msysmta[0-9]{2}|^dbconvo[0-9]{2}|^dbshowcase01|atlasweb[0-9]{2}|devnagios[0-9]{2}|cimaster02|worker[0-9]{2}|^ganglia[0-9]{2}|^imgcache[0-9]{2}|imgconvert[0-9]{2}|^imgwriter[0-9]{2}|dev-img02|^datacache04|^graphite01|^graphite03|^webutil03|^webutil04|^statsd01|^maintweb[0-9]{2}|^(dev-|preprod-)?convosearch[0-9]{2}|deployinator[0-9]{2}|^wpadmin01|^(preprod-)?dbtasks[0-9]{2})b/ and node.role?("Web56") ==false and node.role?("Preprodweb56") == false and node.role?("Princess53") == false andnode.role?("Auth") == false
    57. 57. Simplicity - Better!if node.chef_environment == "libmemcached_upgrade" package "libmemcached" do version "1.0.4-1" action :install end <snip>else package "libmemcached" do version "0.53-1.1" action :install end <snip>end
    58. 58. Simplicity - Complexity
    59. 59. Simplicity - Complexity• Sometimes you need complex behaviour
    60. 60. Simplicity - Complexity• Sometimes you need complex behaviour• Don’t fight it, try to abstract it.
    61. 61. Simplicity - Complexity• Sometimes you need complex behaviour• Don’t fight it, try to abstract it.• Case in point: Syslog-ng refactor
    62. 62. Case Study: Syslog-ng
    63. 63. Case Study: Syslog-ng• 36 recipes
    64. 64. Case Study: Syslog-ng• 36 recipes• 30 versions of syslog-ng.conf
    65. 65. Case Study: Syslog-ng• 36 recipes• 30 versions of syslog-ng.conf• 27 manually configured files in /etc/syslog- ng.d on central server
    66. 66. Case Study: Syslog-ng• 36 recipes• 30 versions of syslog-ng.conf• 27 manually configured files in /etc/syslog- ng.d on central server• Edge cases and exceptions galore
    67. 67. Case Study: Syslog-ng
    68. 68. Case Study: Syslog-ng• Down to:
    69. 69. Case Study: Syslog-ng• Down to: • 2 recipes (one client, one server)
    70. 70. Case Study: Syslog-ng• Down to: • 2 recipes (one client, one server) • 2 templates (one for syslog-ng.conf, one for stuff in /etc/syslog-ng.d)
    71. 71. Case Study: Syslog-ng• Down to: • 2 recipes (one client, one server) • 2 templates (one for syslog-ng.conf, one for stuff in /etc/syslog-ng.d) • Attributes in roles
    72. 72. Case Study: Syslog-ng• Down to: • 2 recipes (one client, one server) • 2 templates (one for syslog-ng.conf, one for stuff in /etc/syslog-ng.d) • Attributes in roles • Not open sourced yet, sorry :(
    73. 73. Case Study: Syslog-ng"syslog": { "group": "preprod_web", "items": { "web_apache_access_log": { "source": "/var/log/httpd/access_log", "source_program_override": "APACHEACCESS: ", "destination": "<snip>/access.log", "destination_filters": [ "host(^preprod-web)", "match(APACHEACCESS)" ], "destination_options": [ "template_escape(no)" ] }, }}
    74. 74. Remember, No Panacea!
    75. 75. Remember, No Panacea!• A new package hits the repo.
    76. 76. Remember, No Panacea!• A new package hits the repo. • Are you in control of when it goes out?
    77. 77. Remember, No Panacea!• A new package hits the repo. • Are you in control of when it goes out? • Memcached Outage
    78. 78. Remember, No Panacea!• A new package hits the repo. • Are you in control of when it goes out? • Memcached Outage• Do you know what services are going to restart and when?
    79. 79. Remember, No Panacea!• A new package hits the repo. • Are you in control of when it goes out? • Memcached Outage• Do you know what services are going to restart and when? • Image Service Outage
    80. 80. Standards
    81. 81. Standards• Not going to talk about testing [8]!
    82. 82. Standards• Not going to talk about testing [8]!• But I don’t have time for standards!
    83. 83. Standards - No Time!
    84. 84. Standards - No Time!• I won’t say “Make Time”, but you should...
    85. 85. Standards - No Time!• I won’t say “Make Time”, but you should...• For a quick win, try Foodcritic
    86. 86. Standards - No Time!• I won’t say “Make Time”, but you should...• For a quick win, try Foodcritic • Good out of the box rules
    87. 87. Standards - No Time!• I won’t say “Make Time”, but you should...• For a quick win, try Foodcritic • Good out of the box rules • Jenkins integration in seconds
    88. 88. Standards - No Time!• I won’t say “Make Time”, but you should...• For a quick win, try Foodcritic • Good out of the box rules • Jenkins integration in seconds • Supports custom rules
    89. 89. Standards - No Time!• I won’t say “Make Time”, but you should...• For a quick win, try Foodcritic • Good out of the box rules • Jenkins integration in seconds • Supports custom rules • Plays well with others
    90. 90. Foodcritic
    91. 91. Foodcritic•gem install foodcritic
    92. 92. Foodcritic•gem install foodcritic•foodcritic <cookbook_repo>
    93. 93. Foodcritic•gem install foodcritic•foodcritic <cookbook_repo>•Simple Jenkins job:
    94. 94. Foodcritic•gem install foodcritic•foodcritic <cookbook_repo>•Simple Jenkins job:#!/usr/bin/env rvm-shell 1.9.3foodcritic -f correctness .
    95. 95. Standards at Etsy
    96. 96. Standards at Etsy• “style” not “correctness”[9]
    97. 97. Standards at Etsy• “style” not “correctness”[9] • ETSY001 - Package or yum_package resource used with :upgrade action
    98. 98. Standards at Etsy• “style” not “correctness”[9] • ETSY001 - Package or yum_package resource used with :upgrade action • ETSY002 - Execute resource used to run git commands
    99. 99. Standards at Etsy• “style” not “correctness”[9] • ETSY001 - Package or yum_package resource used with :upgrade action • ETSY002 - Execute resource used to run git commands • ETSY003 - Execute resource used to run curl or wget commands
    100. 100. Standards at Etsy• “style” not “correctness”[9] • ETSY001 - Package or yum_package resource used with :upgrade action • ETSY002 - Execute resource used to run git commands • ETSY003 - Execute resource used to run curl or wget commands • ETSY004 - Execute resource defined without conditional or action :nothing
    101. 101. Standards at Etsy• “style” not “correctness”[9] • ETSY001 - Package or yum_package resource used with :upgrade action • ETSY002 - Execute resource used to run git commands • ETSY003 - Execute resource used to run curl or wget commands • ETSY004 - Execute resource defined without conditional or action :nothing • ETSY005 - Action :restart sent to a core service
    102. 102. Standards at Etsy• “style” not “correctness”[9] • ETSY001 - Package or yum_package resource used with :upgrade action • ETSY002 - Execute resource used to run git commands • ETSY003 - Execute resource used to run curl or wget commands • ETSY004 - Execute resource defined without conditional or action :nothing • ETSY005 - Action :restart sent to a core service • ETSY006 - Execute resource used to run chef-provided command
    103. 103. Standards at Etsy• “style” not “correctness”[9] • ETSY001 - Package or yum_package resource used with :upgrade action • ETSY002 - Execute resource used to run git commands • ETSY003 - Execute resource used to run curl or wget commands • ETSY004 - Execute resource defined without conditional or action :nothing • ETSY005 - Action :restart sent to a core service • ETSY006 - Execute resource used to run chef-provided command • ETSY007 - Package or yum_package resource used to install core package without specific version number
    104. 104. Standards at Etsy• “style” not “correctness”[9] • ETSY001 - Package or yum_package resource used with :upgrade action • ETSY002 - Execute resource used to run git commands • ETSY003 - Execute resource used to run curl or wget commands • ETSY004 - Execute resource defined without conditional or action :nothing • ETSY005 - Action :restart sent to a core service • ETSY006 - Execute resource used to run chef-provided command • ETSY007 - Package or yum_package resource used to install core package without specific version number
    105. 105. Standards at Etsy
    106. 106. Standards at Etsy• ETSY001 - Written after Memcached Outage
    107. 107. Standards at Etsy• ETSY001 - Written after Memcached Outage• Package or yum_package resource used with :upgrade action
    108. 108. Standards at Etsy• ETSY001 - Written after Memcached Outage• Package or yum_package resource used with :upgrade action• package "memcached" do action :upgrade end
    109. 109. Standards at Etsy
    110. 110. Standards at Etsy• ETSY005 - Written after a total Image service outage
    111. 111. Standards at Etsy• ETSY005 - Written after a total Image service outage• Action :restart sent to a core service
    112. 112. Standards at Etsy• ETSY005 - Written after a total Image service outage• Action :restart sent to a core service• cookbook_file "/etc/httpd/conf.d/myvhost.conf" do source "myvhost.conf" notifies :restart, resources(:service => "httpd") end
    113. 113. Critical Approach and Experimentation
    114. 114. CA&E
    115. 115. CA&E• Chef is by necessity generic
    116. 116. CA&E• Chef is by necessity generic• ...so don’t take Opscode’s word for it.
    117. 117. CA&E• Chef is by necessity generic• ...so don’t take Opscode’s word for it.• If it doesn’t work well for you, change it!
    118. 118. CA&E• Chef is by necessity generic• ...so don’t take Opscode’s word for it.• If it doesn’t work well for you, change it!• Case Study - Etsy Environments rollout
    119. 119. Environments Rollout
    120. 120. Environments Rollout• We knew we needed them
    121. 121. Environments Rollout• We knew we needed them• Simple enough, right?
    122. 122. Environments Rollout• We knew we needed them• Simple enough, right?• Let’s look at the workflow...
    123. 123. Env: Standard Workflow
    124. 124. Env: Standard Workflow • knife cookbook show php
    125. 125. Env: Standard Workflow • knife cookbook show php • Change version number in metadata.rb
    126. 126. Env: Standard Workflow • knife cookbook show php • Change version number in metadata.rb • Change version constraint in foo.json
    127. 127. Env: Standard Workflow • knife cookbook show php • Change version number in metadata.rb • Change version constraint in foo.json • Commit and push changes to git
    128. 128. Env: Standard Workflow • knife cookbook show php • Change version number in metadata.rb • Change version constraint in foo.json • Commit and push changes to git • knife cookbook upload php --freeze
    129. 129. Env: Standard Workflow • knife cookbook show php • Change version number in metadata.rb • Change version constraint in foo.json • Commit and push changes to git • knife cookbook upload php --freeze • knife environment from file foo.json
    130. 130. Env: Our Issues
    131. 131. Env: Our Issues• 41 people with Chef repo access
    132. 132. Env: Our Issues• 41 people with Chef repo access• Most with knife keys
    133. 133. Env: Our Issues• 41 people with Chef repo access• Most with knife keys• All editing the same 2 files with every change...
    134. 134. Env: Solutions?
    135. 135. Env: Solutions?• Go with it and hope for the best?
    136. 136. Env: Solutions?• Go with it and hope for the best?• Don’t use environments?
    137. 137. Env: Solutions?• Go with it and hope for the best?• Don’t use environments?• Write a totally new workflow?
    138. 138. Env: Solutions?• Go with it and hope for the best?• Don’t use environments?• Write a totally new workflow?• Tweak the existing one with some tooling?
    139. 139. Env: Solution!
    140. 140. Env: Solution!• Tweak with some tooling!
    141. 141. Env: Solution!• Tweak with some tooling!• Presenting knife-spork [10]...
    142. 142. Spork: Workflow
    143. 143. Spork: Workflow• Wrapper around standard environments workflow
    144. 144. Spork: Workflow• Wrapper around standard environments workflow • check - cookbook versioning
    145. 145. Spork: Workflow• Wrapper around standard environments workflow • check - cookbook versioning • bump - increment version component
    146. 146. Spork: Workflow• Wrapper around standard environments workflow • check - cookbook versioning • bump - increment version component • upload - upload and freeze
    147. 147. Spork: Workflow• Wrapper around standard environments workflow • check - cookbook versioning • bump - increment version component • upload - upload and freeze • promote - set env constraints
    148. 148. Spork: Check$> knife spork check apache2Checking versions for cookbook apache2...Current local version: 1.0.6Remote versions (Max. 5 most recent only):*1.0.6, frozen1.0.5, frozen<snip>DANGER: Your local cookbook has same version number asthe starred version above!Please bump your local version or you wont be able toupload.
    149. 149. Spork: Bump$> knife spork bump apache2 <major|minor|patch|manual>Bumping patch level of the apache2 cookbook from 1.0.6to 1.0.7
    150. 150. Spork: Upload$> knife spork upload apache2Uploading and freezing apache2 [1.0.7]upload complete
    151. 151. Spork: Promote$> knife spork promote foo phpAdding version constraint php = 1.0.6Saving changes into foo.jsonPromotion complete! Please remember to upload yourchanged Environment file to the Chef Server.---$> knife spork promote foo php --remoteAdding version constraint php = 0.1.0Saving changes into foo.jsonUploading foo to server
    152. 152. Spork
    153. 153. Spork• Worked well, avoided the issues it was designed for
    154. 154. Spork• Worked well, avoided the issues it was designed for• Subsequently evolved
    155. 155. Spork• Worked well, avoided the issues it was designed for• Subsequently evolved • A lot of input & work came from Devs
    156. 156. Spork• Worked well, avoided the issues it was designed for• Subsequently evolved • A lot of input & work came from Devs • Organic evolution
    157. 157. Spork• Worked well, avoided the issues it was designed for• Subsequently evolved • A lot of input & work came from Devs • Organic evolution • Open sourced, of course
    158. 158. Spork: Evolution
    159. 159. Spork: Evolution• Safety Checks
    160. 160. Spork: Evolution• Safety Checks• Extra Features
    161. 161. Spork: Safety Checks
    162. 162. Spork: Safety Checks• Before promoting, check version is uploaded...
    163. 163. Spork: Safety Checks$> knife spork promote php --remote<snip>WARNING: It looks like you have multiple cookbook pathsdefined so I cant tell if youre running inside a gitrepo.Checking that php version 0.1.93 exists on the serverbefore promoting (any error means it hasnt beenuploaded yet)...ERROR: The object you are looking for could not be foundResponse: Cannot find a cookbook named php with version0.1.93
    164. 164. Spork: Safety Checks• Before promoting, check version is uploaded...• Check if you’re promoting changes to more than you thought....
    165. 165. Spork: Safety ChecksWARNING: Youre about to promote changes to severalcookbooks:WARNING:ganglia: = 0.1.26 changed to = 0.1.25installerz: = 0.1.66 changed to = 0.1.65php: = 0.1.92 changed to = 0.1.93Are you sure you want to continue? (Y/N) NYou said no, so Im done here.Would you like to reset your local development.json tomatch the server?? (Y/N) Y<snip>development.json reset.
    166. 166. Spork: Features
    167. 167. Spork: Features• Default Environments
    168. 168. Spork: Features• Default Environments• Git support
    169. 169. Spork: Features• Default Environments• Git support• Chat notifications + Gist (upload and promote)
    170. 170. Spork: Features• Default Environments• Git support• Chat notifications + Gist (upload and promote) • IRCCat
    171. 171. Spork: Features• Default Environments• Git support• Chat notifications + Gist (upload and promote) • IRCCat • Hipchat (courtesy of Secondmarket)
    172. 172. Features: Chat Notifications[19:43:46] <irccat> CHEF: pmcdonnell uploaded and frozecookbook immount version 0.0.24[19:44:00] <irccat> CHEF: pmcdonnell uploaded environmentproduction https://github.etsycorp.com/gist/385043[19:44:01] <irccat> CHEF: pmcdonnell uploaded environmentdevelopment https://github.etsycorp.com/gist/385044
    173. 173. Spork: Features• Default Environments• Git support• Chat notifications + Gist (upload and promote) • IRCCat • Hipchat (courtesy of Secondmarket)• Graphite (on promote)
    174. 174. Features: Graphite
    175. 175. Spork: Features• Default Environments• Git support• Chat notifications + Gist (upload and promote) • IRCCat • Hipchat (courtesy of Secondmarket)• Graphite (on promote)• Foodcritic (on upload)
    176. 176. Features: Foodcritic$> knife spork upload system<snip>Lint checking system...ERROR: Lint check failed. Halting upload.ERROR: Lint check output:ERROR: ETSY003: Execute resource used to run curl orwget commands: /Users/jcowie/dev/etsy/chef/cookbooks/system/recipes/dev-ssl.rb:41
    177. 177. Use the Source!tiny.cc/velocity2012
    178. 178. We’re hiring! BoF on Tuesdayor just come and say Hi :)
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×