My talk from DevOpsDays Minneapolis 2015 on the good and bad about of our Chef workflow at Etsy, with a bonus rant on why the tech industry is broken.

1. Cheffing Etsy
Do too many cooks really spoil the soup?

2. @jonlives
Jon Cowie
Staff Operations Engineer

4. @jonlives
So what is Chef, anyway?

5. @jonlives
What is Chef?
• Desired State Configuration Management
• Thin Server
• Datastore, API and Search
• Thick Client
• Does all the work!

6. @jonlives
Chef Vocabulary Primer
• Node
• Your server, state saved on Chef server
• Cookbook
• Main (versioned) artefact type in Chef
• List of recipes and other stuff

7. @jonlives
Chef Vocabulary Primer - Continued
• Environment
• A list of cookbook version constraints
• Knife
• A CLI interface to Chef Server
• Extensible with Plugins

8. @jonlives
There is no
magic pill.

9. @jonlives
You are the
expert.

10. @jonlives
Chef at Etsy

11. @jonlives
Chef at Etsy
• Chef Server
• ~2000 Nodes
• CentOS, some Mac OS X

12. @jonlives
Chef at Etsy - Continued
• Everything from OS to “Below Code”
• Code deployed using “Deployinator”
• Single git repository
• 2 sources of truth…
• So far, so normal!

13. @jonlives
Chef at Etsy

14. @jonlives
Chef at Etsy - Continued
• Translates to ~35 deploys per day
• What exactly is a Chef deploy?
• Updating “production” version constraint!
• Many less-experienced users

15. @jonlives
Cookbook Workflow

16. @jonlives
$> review -r jcowie --cc ops

17. @jonlives
Push Change
• knife-spork
• Helps multiple chefs avoid clashing
• Visibility into changes
• Plugins

18. @jonlives
Push Change - Continued
• knife spork bump
• knife spork upload
• Test change*
• knife spork promote --remote
• git commit and push

19. @jonlives
Test Change
• Move node to unconstrained environment
• knife node flip foo.etsy.com testing
• knife role flip MyRole testing

20. @jonlives
Downsides of Existing Approach
• No unit tests…
• Holding cookbook in testing is blocking
• Testing env affects all cookbooks
• “Upgrade” envs often used
• How to make it more “Etsy”?

21. @jonlives
chef-whitelist
• Driven by JSON data
• Cookbook library
• Feature flags!

22. @jonlives
chef-whitelist
{
"id": "php-5-5-17",
"patterns": [
"statsd*.ny5.etsy.com",
"deploy*.ny5.etsy.com",
<snip>
]
}

23. @jonlives
chef-whitelist
if node.is_in_whitelist? "php-5-5-17"
package "php-pecl-opcache" do
action :remove
end
end

24. @jonlives
Monitoring & Debugging

25. @jonlives
knife-spork & CI Job
<irccat> CHEF: Jon Cowie uploaded pentaho@0.1.8
<irccat> CHEF: Jon Cowie promoted pentaho@0.1.8 to
production <snip>
<irccat> Git PUSH -> Sysops/chef <snip>
<Jenkins> Starting build #5649 for job chef-
server-git-sync
<Jenkins> Project chef-server-git-sync build
#5649: SUCCESS in 2 min 36 sec: http://
ci.etsycorp.com/job/chef-server-git-sync/5649/

26. @jonlives
IRC Handler
<irccat> Chef run failed on
officebackup01.office.etsy.com gist
failed, see /var/log/chef/client.log on
the host
<irccat> Still Failing on
dbnest01.ny4.etsy.com since 2 days ago
https://github.etsycorp.com/gist/
656d8914fbef5a6bd9aa

27. @jonlives
“Lastrun" Data
%
knife
node
lastrun
dbnest01.ny4.etsy.com
Status
failed
Elapsed
Time
29.055892
Start
Time
2014-­‐10-­‐06
12:54:51
+0000
End
Time
2014-­‐10-­‐06
12:55:20
+0000
<snip>
Exception
<snip>
Installed
package
backupd-­‐1.4-­‐1.365657d.el5.centos
is
newer
than
candidate
package
backupd-­‐1.2-­‐1.99ddb8e.el5

28. @jonlives
Dashboards

29. @jonlives
Links - Workflow Tools
• https://github.com/jonlives/knife-spork
• https://github.com/jonlives/knife-flip
• https://github.com/jgoulah/knife-lastrun
• https://github.com/etsy/chef-whitelist

30. @jonlives
Links - Monitoring
• https://github.com/etsy/chef-handlers
• https://github.com/etsy/dashboard
• https://github.com/bmarini/knife-inspect

31. @jonlives
So, how’s it all going?

32. @jonlives
Some Pain Points
• Change Clashes
• Confusion over state of changes
• People forget things
• Testing pains

33. @jonlives
We can rebuild it.
We have the technology.

34. @jonlives
(201)6 Million Dollar Workflow
• Deployinator-based workflow
• Push queue
• Unit tests
• “try” based testing
• More like existing CD workflows

35. @jonlives
Watch this space!

36. @jonlives
http://jonliv.es/book
Discount Code: AUTHD
40% off Print
50% off Digital

37. @jonlives
And now, a brief rant…

38. 38
“Before I [tweet|open
source|go to an event], I
first have to consider my
personal safety.”

39. 39

40. 40
We also have the privilege
to say THIS IS NOT OK!

41. 41
If we don’t speak out, our
inaction says “We see
nothing wrong with this.”

42. 42
“Assuming that an arbitrary woman
wants to do the work of educating you
about sexism is not the most effective
choice, any more than assuming any
random open-source contributor wants to
provide tech support for you, on demand,
on your timeline.”
http://bridgetkromhout.com/blog/2015/05/31/let-me-google-that-for-you/

43. 43
“Seems Hard?
It is. Welcome to being a
minority with an opinion.
Now, do the work.”
http://www.catehuston.com/blog/2015/07/08/pitfalls-for-men-talking-about-diversity/

44. @jonlives
Thanks!
Questions?
@jonlives / http://jonliv.es / jon@etsy.com