At Etsy we run our production stack exclusively on physical hardware and have (including developer and CI VMs) about 1000 nodes managed with a single Chef server running the same cookbooks across development and production nodes. We have built the knife-spork plugin to support our workflow and have 30 engineers making changes about 20 times a day. While this setup keeps configuration changes and package installations consistent across our network, it makes it harder to gradually roll out and test changes on a per node or role basis. In the talk our general workflow will be introduced and explained and how we enable engineers to frequently roll out small changes. Following the general introduction, our workflow for testing changes will be discussed and the library we wrote to enable config flags and gradual roll outs for our infrastructure within our chef recipes.
7. knife-spork
•Work!ow to manage cookbook and environment
changes
•Versioned cookbooks and pinned environments
•Speci"c work!ow, di#erent ways of using plugins
Tuesday, September 10, 13
18. Downsides
•Longer testing blocks others
•Staged cookbooks can accidentally be promoted
•Testing environment a#ects more than one
cookbook
•Used “upgrade” environments to circumvent
Tuesday, September 10, 13
20. Branching in Code
•Well established pattern for “dark launches”
•Used in the Etsy Web stack
•Allows for restricted roll outs
•http://code.!ickr.net/2009/12/02/!ipping-out/
Tuesday, September 10, 13
28. real world
example
{
"id": "php-5-4-19",
"patterns": [
"dschauenberg.vm.dev.etsy.com",
"web0270.etsy.com",
"api04.etsy.com",
"imgcache01.etsy.com",
"imgwriter01.etsy.com",
"worker01.etsy.com",
"beacon01.etsy.com",
"paymentsweb01.etsy.com"
],
"roles": [
]
}
Tuesday, September 10, 13
29. Advantages
•Easy to access list of what gets upgrades
•Upgrades don’t need the spork work!ow
•Pattern already known by all engineers
Tuesday, September 10, 13
30. Downsides
•Changes outside the regular work!ow
•No graphs (yet)
•Less visible cleanup required
Tuesday, September 10, 13
31. Summary
•GitHub Enterprise, Dev VMs, chef-shell as
development environment
•Chef Server and knife-spork as Deployment
System
•Feature !agging with chef-whitelist
•Monitoring, Noti"cations, Graphs
Tuesday, September 10, 13