Michelin Starred Cooking with Chef

Michelin StarredCooking with Chef Jon Cowie, Etsy.com jcowie@etsy.com @jonlives

What?• Chef at Etsy

What?• Chef at Etsy• Familiarity and Understanding

What?• Chef at Etsy• Familiarity and Understanding• Critical Approach and Experimentation

What?• Chef at Etsy• Familiarity and Understanding• Critical Approach and Experimentation• Use The Source

What?• Chef at Etsy• Familiarity and Understanding• Critical Approach and Experimentation• Use The Source• A liberal sprinkling of screwups

What?• Chef at Etsy• Familiarity and Understanding• Critical Approach and Experimentation• Use The Source• A liberal sprinkling of screwups• Open Sourced Goodness - We’re all here!

What?• Chef at Etsy• Familiarity and Understanding• Critical Approach and Experimentation• Use The Source• A liberal sprinkling of screwups• Open Sourced Goodness - We’re all here! • [x] = http://tiny.cc/velocity2012

Opscode is Orange, Velocity is Blue. In Soviet Russia,Cookbook writes you.

Chef at Etsy

Our Setup

Our Setup• Open Source chef server 0.10.4

Our Setup• Open Source chef server 0.10.4• Backup to Opscode Platform

Our Setup• Open Source chef server 0.10.4• Backup to Opscode Platform• ~800 self-hosted nodes (Mainly CentOS, some RHEL & mac)

Our Setup• Open Source chef server 0.10.4• Backup to Opscode Platform• ~800 self-hosted nodes (Mainly CentOS, some RHEL & mac)• KVM & lxc virts, self hosted too.

Our Setup• Open Source chef server 0.10.4• Backup to Opscode Platform• ~800 self-hosted nodes (Mainly CentOS, some RHEL & mac)• KVM & lxc virts, self hosted too.• Never test in production!

Our Setup• Open Source chef server 0.10.4• Backup to Opscode Platform• ~800 self-hosted nodes (Mainly CentOS, some RHEL & mac)• KVM & lxc virts, self hosted too.• Never test in production!• Many chefs don’t spoil our soup

Familiarity andUnderstanding

nothttp://www.ﬂickr.com/photos/photo_secessionist/5555167113/ (C) Alexander McQueen

• Insight

• Insight• Simplicity

• Insight• Simplicity• Standards

Insight

Insight• You should never have to say “I don’t know.”

Insight• You should never have to say “I don’t know.”• What, where, when, why, how long?

Insight• You should never have to say “I don’t know.”• What, where, when, why, how long?• Easy, and I’ll show you how!

Chef Dashboard

Chef Dashboard• Chef handler sends metrics to graphite [4]

Chef Dashboard• Chef handler sends metrics to graphite [4] • git clone git://github.com/etsy/chef-handlers.git

Chef Dashboard• Chef handler sends metrics to graphite [4] • git clone git://github.com/etsy/chef-handlers.git • Set your graphite server’s URL in graphite.rb

Chef Dashboard• Chef handler sends metrics to graphite [4] • git clone git://github.com/etsy/chef-handlers.git • Set your graphite server’s URL in graphite.rb • Add the following to client.rb

Chef Dashboard• Chef handler sends metrics to graphite [4] • git clone git://github.com/etsy/chef-handlers.git • Set your graphite server’s URL in graphite.rb • Add the following to client.rb • require "<clonedir>/graphite.rb" graphite_handler = GraphiteReporting.new report_handlers << graphite_handler exception_handlers << graphite_handler

Chef Dashboard• Chef handler sends metrics to graphite [4] • git clone git://github.com/etsy/chef-handlers.git • Set your graphite server’s URL in graphite.rb • Add the following to client.rb • require "<clonedir>/graphite.rb" graphite_handler = GraphiteReporting.new report_handlers << graphite_handler exception_handlers << graphite_handler• Etsy dashboards framework [5]

[12:54:01] <irccat> Chef run failed on gfernandez.vm.ny4dev.etsy.com[12:54:02] <irccat> https://github.etsycorp.com/gist/384228[12:54:02] <irccat>[12:54:07] <irccat> Chef run failed on buildtest11.ny4dev.etsy.com[12:54:07] <irccat> https://github.etsycorp.com/gist/384227[12:54:07] <irccat>

Chef irccat Alerts

Chef irccat Alerts• Chef handler send fails to irc[4] via irccat [6]

Chef irccat Alerts• Chef handler send fails to irc[4] via irccat [6] • git clone git://github.com/etsy/chef-handlers.git

Chef irccat Alerts• Chef handler send fails to irc[4] via irccat [6] • git clone git://github.com/etsy/chef-handlers.git • Set your irccat[6] server’s URL in logtoirc.rb

Chef irccat Alerts• Chef handler send fails to irc[4] via irccat [6] • git clone git://github.com/etsy/chef-handlers.git • Set your irccat[6] server’s URL in logtoirc.rb • Add the following to client.rb

Chef irccat Alerts• Chef handler send fails to irc[4] via irccat [6] • git clone git://github.com/etsy/chef-handlers.git • Set your irccat[6] server’s URL in logtoirc.rb • Add the following to client.rb • require "<clonedir>/logtoirc.rb" exception_handlers << Etsy::LogToIRC.new

~ > knife node lastrun buildtest11.ny4dev.etsy.comStatus failedElapsed Time 4.628171438Start Time 2012-06-18 10:06:28 +0000End Time 2012-06-18 10:06:32 +0000Recipe Action Resource Type ResourceBacktrace<snip>ExceptionChef::Exceptions::Package: package[php] (php::buildtest line 20) hadan error: Version 5.3.10-1.el5 of php not found. Did you specify bothversion and release? (version-release, e.g. 1.84-10.fc6)

~ > knife search node lastrun_debug_formatted_exception:Chef::Exceptions::Package* -a lastrun.debug.formatted_exception5 items foundid: masterrestore.ny4.etsy.comlastrun.debug.formatted_exception: Chef::Exceptions::Package:package[postgresql-server] (postgresql::server-8.3 line 1) had anerror: Installed package postgresql-server-8.3.16-1PGDG_id is newerthan candidate package postgresql-server-8.3.11-1PGDG_id.rhel5id: buildtest11.ny4dev.etsy.comlastrun.debug.formatted_exception: Chef::Exceptions::Package:package[php] (php::buildtest line 20) had an error: Version5.3.10-1.el5 of php not found. Did you specify both version andrelease? (version-release, e.g. 1.84-10.fc6)<snip>

Chef lastrun Info

Chef lastrun Info• Chef handler and knife plugin [7]

Chef lastrun Info• Chef handler and knife plugin [7] • gem install knife-lastrun

Chef lastrun Info• Chef handler and knife plugin [7] • gem install knife-lastrun • Add the following to client.rb

Chef lastrun Info• Chef handler and knife plugin [7] • gem install knife-lastrun • Add the following to client.rb • require "lastrun_update" handler = LastRunUpdateHandler.new report_handlers << handler exception_handlers << handler

Chef lastrun Info• Chef handler and knife plugin [7] • gem install knife-lastrun • Add the following to client.rb • require "lastrun_update" handler = LastRunUpdateHandler.new report_handlers << handler exception_handlers << handler • knife node lastrun <nodename>

Simplicity

Simplicity• Think of yourself at 3AM!

Simplicity• Think of yourself at 3AM!• Please, won’t you think of the new guy?

Simplicity• Think of yourself at 3AM!• Please, won’t you think of the new guy?• Minimize the logics!

Simplicity• Think of yourself at 3AM!• Please, won’t you think of the new guy?• Minimize the logics! • As few logical steps from start to ﬁnish as possible.

Simplicity - Not! Date: Mon Dec 05 2011 23:07:18 GMT+0000 (GMT)Subject: so close to death# Dont install v2 on search or Cent 5.6 nodes-if node[:fqdn] !~ /b(^(preprod-)?search[0-9]{2}|ny4dev.etsy.com|^(preprod-)?giftsweb[0-9]{2}|^db(shard|spare|data)[0-9]{2}|^qa-web01|^devsearch[0-9]{2}|^nagios01|^webnest[0-9]{2}|^prodking[0-9]{2}|^sandboxweb[0-9]{2}|^virt((0[5-9])|(1[0-9]))|^msysmgr[0-9]{2}|^msysmta[0-9]{2}|^dbconvo[0-9]{2}|^dbshowcase01|atlasweb[0-9]{2}|devnagios[0-9]{2}|cimaster02|worker[0-9]{2}|^ganglia[0-9]{2}|îmgcache[0-9]{2}|imgconvert[0-9]{2}|îmgwriter[0-9]{2}|dev-img02|^datacache04|^graphite01|^graphite03|^webutil03|^webutil04|^statsd01|^maintweb[0-9]{2}|^(dev-|preprod-)?convosearch[0-9]{2}|deployinator[0-9]{2}|^wpadmin01|^(preprod-)?dbtasks[0-9]{2})b/ and node.role?("Web56") ==false and node.role?("Preprodweb56") == false and node.role?("Princess53") == false+if node[:fqdn] !~ /b(^(preprod-)?search[0-9]{2}|ny4dev.etsy.com|^(preprod-)?giftsweb[0-9]{2}|^db(shard|spare|data)[0-9]{2}|^qa-web01|^devsearch[0-9]{2}|^nagios01|^webnest[0-9]{2}|^prodking[0-9]{2}|^sandboxweb[0-9]{2}|^virt((0[5-9])|(1[0-9]))|^msysmgr[0-9]{2}|^msysmta[0-9]{2}|^dbconvo[0-9]{2}|^dbshowcase01|atlasweb[0-9]{2}|devnagios[0-9]{2}|cimaster02|worker[0-9]{2}|^ganglia[0-9]{2}|îmgcache[0-9]{2}|imgconvert[0-9]{2}|îmgwriter[0-9]{2}|dev-img02|^datacache04|^graphite01|^graphite03|^webutil03|^webutil04|^statsd01|^maintweb[0-9]{2}|^(dev-|preprod-)?convosearch[0-9]{2}|deployinator[0-9]{2}|^wpadmin01|^(preprod-)?dbtasks[0-9]{2})b/ and node.role?("Web56") ==false and node.role?("Preprodweb56") == false and node.role?("Princess53") == false andnode.role?("Auth") == false

Simplicity - Better!if node.chef_environment == "libmemcached_upgrade" package "libmemcached" do version "1.0.4-1" action :install end <snip>else package "libmemcached" do version "0.53-1.1" action :install end <snip>end

Simplicity - Complexity

Simplicity - Complexity• Sometimes you need complex behaviour

Simplicity - Complexity• Sometimes you need complex behaviour• Don’t ﬁght it, try to abstract it.

Simplicity - Complexity• Sometimes you need complex behaviour• Don’t ﬁght it, try to abstract it.• Case in point: Syslog-ng refactor

Case Study: Syslog-ng

Case Study: Syslog-ng• 36 recipes

Case Study: Syslog-ng• 36 recipes• 30 versions of syslog-ng.conf

Case Study: Syslog-ng• 36 recipes• 30 versions of syslog-ng.conf• 27 manually conﬁgured ﬁles in /etc/syslog- ng.d on central server

Case Study: Syslog-ng• 36 recipes• 30 versions of syslog-ng.conf• 27 manually conﬁgured ﬁles in /etc/syslog- ng.d on central server• Edge cases and exceptions galore

Case Study: Syslog-ng

Case Study: Syslog-ng• Down to:

Case Study: Syslog-ng• Down to: • 2 recipes (one client, one server)

Case Study: Syslog-ng• Down to: • 2 recipes (one client, one server) • 2 templates (one for syslog-ng.conf, one for stuff in /etc/syslog-ng.d)

Case Study: Syslog-ng• Down to: • 2 recipes (one client, one server) • 2 templates (one for syslog-ng.conf, one for stuff in /etc/syslog-ng.d) • Attributes in roles

Case Study: Syslog-ng• Down to: • 2 recipes (one client, one server) • 2 templates (one for syslog-ng.conf, one for stuff in /etc/syslog-ng.d) • Attributes in roles • Not open sourced yet, sorry :(

Case Study: Syslog-ng"syslog": { "group": "preprod_web", "items": { "web_apache_access_log": { "source": "/var/log/httpd/access_log", "source_program_override": "APACHEACCESS: ", "destination": "<snip>/access.log", "destination_filters": [ "host(^preprod-web)", "match(APACHEACCESS)" ], "destination_options": [ "template_escape(no)" ] }, }}

Remember, No Panacea!

Remember, No Panacea!• A new package hits the repo.

Remember, No Panacea!• A new package hits the repo. • Are you in control of when it goes out?

Remember, No Panacea!• A new package hits the repo. • Are you in control of when it goes out? • Memcached Outage

Remember, No Panacea!• A new package hits the repo. • Are you in control of when it goes out? • Memcached Outage• Do you know what services are going to restart and when?

Remember, No Panacea!• A new package hits the repo. • Are you in control of when it goes out? • Memcached Outage• Do you know what services are going to restart and when? • Image Service Outage

Standards

Standards• Not going to talk about testing [8]!

Standards• Not going to talk about testing [8]!• But I don’t have time for standards!

Standards - No Time!

Standards - No Time!• I won’t say “Make Time”, but you should...

Standards - No Time!• I won’t say “Make Time”, but you should...• For a quick win, try Foodcritic

Standards - No Time!• I won’t say “Make Time”, but you should...• For a quick win, try Foodcritic • Good out of the box rules

Standards - No Time!• I won’t say “Make Time”, but you should...• For a quick win, try Foodcritic • Good out of the box rules • Jenkins integration in seconds

Standards - No Time!• I won’t say “Make Time”, but you should...• For a quick win, try Foodcritic • Good out of the box rules • Jenkins integration in seconds • Supports custom rules

Standards - No Time!• I won’t say “Make Time”, but you should...• For a quick win, try Foodcritic • Good out of the box rules • Jenkins integration in seconds • Supports custom rules • Plays well with others

Foodcritic

Foodcritic•gem install foodcritic

Foodcritic•gem install foodcritic•foodcritic <cookbook_repo>

Foodcritic•gem install foodcritic•foodcritic <cookbook_repo>•Simple Jenkins job:

Foodcritic•gem install foodcritic•foodcritic <cookbook_repo>•Simple Jenkins job:#!/usr/bin/env rvm-shell 1.9.3foodcritic -f correctness .

Standards at Etsy

Standards at Etsy• “style” not “correctness”[9]

Standards at Etsy• “style” not “correctness”[9] • ETSY001 - Package or yum_package resource used with :upgrade action

Standards at Etsy• “style” not “correctness”[9] • ETSY001 - Package or yum_package resource used with :upgrade action • ETSY002 - Execute resource used to run git commands

Standards at Etsy• “style” not “correctness”[9] • ETSY001 - Package or yum_package resource used with :upgrade action • ETSY002 - Execute resource used to run git commands • ETSY003 - Execute resource used to run curl or wget commands

Standards at Etsy• “style” not “correctness”[9] • ETSY001 - Package or yum_package resource used with :upgrade action • ETSY002 - Execute resource used to run git commands • ETSY003 - Execute resource used to run curl or wget commands • ETSY004 - Execute resource deﬁned without conditional or action :nothing

Standards at Etsy• “style” not “correctness”[9] • ETSY001 - Package or yum_package resource used with :upgrade action • ETSY002 - Execute resource used to run git commands • ETSY003 - Execute resource used to run curl or wget commands • ETSY004 - Execute resource deﬁned without conditional or action :nothing • ETSY005 - Action :restart sent to a core service

Standards at Etsy• “style” not “correctness”[9] • ETSY001 - Package or yum_package resource used with :upgrade action • ETSY002 - Execute resource used to run git commands • ETSY003 - Execute resource used to run curl or wget commands • ETSY004 - Execute resource deﬁned without conditional or action :nothing • ETSY005 - Action :restart sent to a core service • ETSY006 - Execute resource used to run chef-provided command

Standards at Etsy• “style” not “correctness”[9] • ETSY001 - Package or yum_package resource used with :upgrade action • ETSY002 - Execute resource used to run git commands • ETSY003 - Execute resource used to run curl or wget commands • ETSY004 - Execute resource deﬁned without conditional or action :nothing • ETSY005 - Action :restart sent to a core service • ETSY006 - Execute resource used to run chef-provided command • ETSY007 - Package or yum_package resource used to install core package without speciﬁc version number

Standards at Etsy

Standards at Etsy• ETSY001 - Written after Memcached Outage

Standards at Etsy• ETSY001 - Written after Memcached Outage• Package or yum_package resource used with :upgrade action

Standards at Etsy• ETSY001 - Written after Memcached Outage• Package or yum_package resource used with :upgrade action• package "memcached" do action :upgrade end

Standards at Etsy

Standards at Etsy• ETSY005 - Written after a total Image service outage

Standards at Etsy• ETSY005 - Written after a total Image service outage• Action :restart sent to a core service

Standards at Etsy• ETSY005 - Written after a total Image service outage• Action :restart sent to a core service• cookbook_file "/etc/httpd/conf.d/myvhost.conf" do source "myvhost.conf" notifies :restart, resources(:service => "httpd") end

Critical Approach and Experimentation

CA&E• Chef is by necessity generic

CA&E• Chef is by necessity generic• ...so don’t take Opscode’s word for it.

CA&E• Chef is by necessity generic• ...so don’t take Opscode’s word for it.• If it doesn’t work well for you, change it!

CA&E• Chef is by necessity generic• ...so don’t take Opscode’s word for it.• If it doesn’t work well for you, change it!• Case Study - Etsy Environments rollout

Environments Rollout

Environments Rollout• We knew we needed them

Environments Rollout• We knew we needed them• Simple enough, right?

Environments Rollout• We knew we needed them• Simple enough, right?• Let’s look at the workﬂow...

Env: Standard Workﬂow

Env: Standard Workﬂow • knife cookbook show php

Env: Standard Workﬂow • knife cookbook show php • Change version number in metadata.rb

Env: Standard Workﬂow • knife cookbook show php • Change version number in metadata.rb • Change version constraint in foo.json

Env: Standard Workﬂow • knife cookbook show php • Change version number in metadata.rb • Change version constraint in foo.json • Commit and push changes to git

Env: Standard Workﬂow • knife cookbook show php • Change version number in metadata.rb • Change version constraint in foo.json • Commit and push changes to git • knife cookbook upload php --freeze

Env: Standard Workﬂow • knife cookbook show php • Change version number in metadata.rb • Change version constraint in foo.json • Commit and push changes to git • knife cookbook upload php --freeze • knife environment from ﬁle foo.json

Env: Our Issues

Env: Our Issues• 41 people with Chef repo access

Env: Our Issues• 41 people with Chef repo access• Most with knife keys

Env: Our Issues• 41 people with Chef repo access• Most with knife keys• All editing the same 2 ﬁles with every change...

Env: Solutions?

Env: Solutions?• Go with it and hope for the best?

Env: Solutions?• Go with it and hope for the best?• Don’t use environments?

Env: Solutions?• Go with it and hope for the best?• Don’t use environments?• Write a totally new workﬂow?

Env: Solutions?• Go with it and hope for the best?• Don’t use environments?• Write a totally new workﬂow?• Tweak the existing one with some tooling?

Env: Solution!

Env: Solution!• Tweak with some tooling!

Env: Solution!• Tweak with some tooling!• Presenting knife-spork [10]...

Spork: Workﬂow

Spork: Workﬂow• Wrapper around standard environments workﬂow

Spork: Workﬂow• Wrapper around standard environments workﬂow • check - cookbook versioning

Spork: Workﬂow• Wrapper around standard environments workﬂow • check - cookbook versioning • bump - increment version component

Spork: Workﬂow• Wrapper around standard environments workﬂow • check - cookbook versioning • bump - increment version component • upload - upload and freeze

Spork: Workﬂow• Wrapper around standard environments workﬂow • check - cookbook versioning • bump - increment version component • upload - upload and freeze • promote - set env constraints

Spork: Check$> knife spork check apache2Checking versions for cookbook apache2...Current local version: 1.0.6Remote versions (Max. 5 most recent only):*1.0.6, frozen1.0.5, frozen<snip>DANGER: Your local cookbook has same version number asthe starred version above!Please bump your local version or you wont be able toupload.

Spork: Bump$> knife spork bump apache2 <major|minor|patch|manual>Bumping patch level of the apache2 cookbook from 1.0.6to 1.0.7

Spork: Upload$> knife spork upload apache2Uploading and freezing apache2 [1.0.7]upload complete

Spork: Promote$> knife spork promote foo phpAdding version constraint php = 1.0.6Saving changes into foo.jsonPromotion complete! Please remember to upload yourchanged Environment file to the Chef Server.---$> knife spork promote foo php --remoteAdding version constraint php = 0.1.0Saving changes into foo.jsonUploading foo to server

Spork• Worked well, avoided the issues it was designed for

Spork• Worked well, avoided the issues it was designed for• Subsequently evolved

Spork• Worked well, avoided the issues it was designed for• Subsequently evolved • A lot of input & work came from Devs

Spork• Worked well, avoided the issues it was designed for• Subsequently evolved • A lot of input & work came from Devs • Organic evolution

Spork• Worked well, avoided the issues it was designed for• Subsequently evolved • A lot of input & work came from Devs • Organic evolution • Open sourced, of course

Spork: Evolution

Spork: Evolution• Safety Checks

Spork: Evolution• Safety Checks• Extra Features

Spork: Safety Checks

Spork: Safety Checks• Before promoting, check version is uploaded...

Spork: Safety Checks$> knife spork promote php --remote<snip>WARNING: It looks like you have multiple cookbook pathsdefined so I cant tell if youre running inside a gitrepo.Checking that php version 0.1.93 exists on the serverbefore promoting (any error means it hasnt beenuploaded yet)...ERROR: The object you are looking for could not be foundResponse: Cannot find a cookbook named php with version0.1.93

Spork: Safety Checks• Before promoting, check version is uploaded...• Check if you’re promoting changes to more than you thought....

Spork: Safety ChecksWARNING: Youre about to promote changes to severalcookbooks:WARNING:ganglia: = 0.1.26 changed to = 0.1.25installerz: = 0.1.66 changed to = 0.1.65php: = 0.1.92 changed to = 0.1.93Are you sure you want to continue? (Y/N) NYou said no, so Im done here.Would you like to reset your local development.json tomatch the server?? (Y/N) Y<snip>development.json reset.

Spork: Features

Spork: Features• Default Environments

Spork: Features• Default Environments• Git support

Spork: Features• Default Environments• Git support• Chat notiﬁcations + Gist (upload and promote)

Spork: Features• Default Environments• Git support• Chat notiﬁcations + Gist (upload and promote) • IRCCat

Spork: Features• Default Environments• Git support• Chat notiﬁcations + Gist (upload and promote) • IRCCat • Hipchat (courtesy of Secondmarket)

Features: Chat Notiﬁcations[19:43:46] <irccat> CHEF: pmcdonnell uploaded and frozecookbook immount version 0.0.24[19:44:00] <irccat> CHEF: pmcdonnell uploaded environmentproduction https://github.etsycorp.com/gist/385043[19:44:01] <irccat> CHEF: pmcdonnell uploaded environmentdevelopment https://github.etsycorp.com/gist/385044

Spork: Features• Default Environments• Git support• Chat notiﬁcations + Gist (upload and promote) • IRCCat • Hipchat (courtesy of Secondmarket)• Graphite (on promote)

Features: Graphite

Spork: Features• Default Environments• Git support• Chat notiﬁcations + Gist (upload and promote) • IRCCat • Hipchat (courtesy of Secondmarket)• Graphite (on promote)• Foodcritic (on upload)

Features: Foodcritic$> knife spork upload system<snip>Lint checking system...ERROR: Lint check failed. Halting upload.ERROR: Lint check output:ERROR: ETSY003: Execute resource used to run curl orwget commands: /Users/jcowie/dev/etsy/chef/cookbooks/system/recipes/dev-ssl.rb:41

Use the Source!tiny.cc/velocity2012

We’re hiring! BoF on Tuesdayor just come and say Hi :)

http://lanyrd.com	40
https://twimg0-a.akamaihd.net	8
http://snews.rage2.yandex.ru	4
http://coderwall.com	3
http://10.107.123.12	2
http://www.linkedin.com	2
https://si0.twimg.com	1
http://www.klektd.com	1

Michelin Starred Cooking with Chef

by jonlives on Jul 05, 2012

Accessibility

Categories

Upload Details

Usage Rights

8 Embeds 61

Statistics

Michelin Starred Cooking with Chef Presentation Transcript