Your SlideShare is downloading. ×
0
Feature Flagging your
Infrastructure for Fun and
Pro!t
Daniel Schauenberg
dschauenberg@etsy.com
@mrtazz
Tuesday, September...
Tuesday, September 10, 13
LAMMP
Item by TheBackPackShoppe
Tuesday, September 10, 13
Item by FrankelPhotos
Monolithic App
Tuesday, September 10, 13
Etsy Infrastructure
•~1400 nodes
•~30 dev & ops engineers making changes
regularly
•Open Source Chef server + GitHub Enter...
jonlives/knife-spork
Tuesday, September 10, 13
knife-spork
•Work!ow to manage cookbook and environment
changes
•Versioned cookbooks and pinned environments
•Speci"c work...
Tuesday, September 10, 13
% chef-shell
chef > recipe_mode
chef:recipe > echo off
chef:recipe > include_recipe "apache"
chef:recipe > run_chef
Tuesda...
% review -r jcowie --cc ops
Tuesday, September 10, 13
% knife spork check apache
% knife spork bump apache
% git commit
% git push
% knife spork upload apache
Staging Deploy
Tu...
jonlives/knife-!ip
% knife node flip node.etsy.com testing
% knife role flip testRole testing
Tuesday, September 10, 13
% knife spork promote apache
% git commit
% git push
% knife spork promote apache --remote
Production Deploy
Tuesday, Sept...
Monitoring
Tuesday, September 10, 13
19:18:06 irccat | CHEF: Daniel
Schauenberg promoted apache@0.0.2 to
development https://
github.etsycorp.com/gist/12345
Tu...
etsy/chef-handlers
19:20:00 irccat | Chef run failed on test.etsy.com
19:20:00 irccat | https://github.etsycorp.com/
gist/...
Graphs!
Tuesday, September 10, 13
Downsides
•Longer testing blocks others
•Staged cookbooks can accidentally be promoted
•Testing environment a#ects more th...
Feature Flags
Tuesday, September 10, 13
Branching in Code
•Well established pattern for “dark launches”
•Used in the Etsy Web stack
•Allows for restricted roll ou...
Tuesday, September 10, 13
etsy/chef-whitelist
Tuesday, September 10, 13
chef-whitelist
•Data bag driven whitelist
•Library to include in cookbooks
•Easy access to feature !ags
Tuesday, September...
whitelist data bag
{
"id": "my_whitelist",
"patterns": [
"host.example.com",
"*.subdomain.example.com",
"prefix*.example.c...
whitelist data bag{
"id": "my_whitelist",
"patterns": [
"host.example.com",
"*.subdomain.example.com",
"prefix*.example.co...
feature !ags in recipe
if node.is_in_whitelist? "my_whitelist"
# new hawtness
else
# old stuff
end
Tuesday, September 10, ...
Customizable
node.is_in_whitelist? "my_whitelist", "acl", "hosts"
Tuesday, September 10, 13
real world
example
{
"id": "php-5-4-19",
"patterns": [
"dschauenberg.vm.dev.etsy.com",
"web0270.etsy.com",
"api04.etsy.com...
Advantages
•Easy to access list of what gets upgrades
•Upgrades don’t need the spork work!ow
•Pattern already known by all...
Downsides
•Changes outside the regular work!ow
•No graphs (yet)
•Less visible cleanup required
Tuesday, September 10, 13
Summary
•GitHub Enterprise, Dev VMs, chef-shell as
development environment
•Chef Server and knife-spork as Deployment
Syst...
•http://codeascraft.etsy.com/
•http://codeascraft.com/2013/08/02/
infrastructure-upgrades-with-chef/
•http://www.slideshar...
Thank you!
Questions?
Tuesday, September 10, 13
Feature Flagging your
Infrastructure for Fun and
Pro!t
Daniel Schauenberg
dschauenberg@etsy.com
@mrtazz
Tuesday, September...
Upcoming SlideShare
Loading in...5
×

Feature Flagging your Infrastructure for Fun and Profit

1,072

Published on

At Etsy we run our production stack exclusively on physical hardware and have (including developer and CI VMs) about 1000 nodes managed with a single Chef server running the same cookbooks across development and production nodes. We have built the knife-spork plugin to support our workflow and have 30 engineers making changes about 20 times a day. While this setup keeps configuration changes and package installations consistent across our network, it makes it harder to gradually roll out and test changes on a per node or role basis. In the talk our general workflow will be introduced and explained and how we enable engineers to frequently roll out small changes. Following the general introduction, our workflow for testing changes will be discussed and the library we wrote to enable config flags and gradual roll outs for our infrastructure within our chef recipes.

Published in: Technology, Self Improvement
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,072
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
8
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Transcript of "Feature Flagging your Infrastructure for Fun and Profit"

  1. 1. Feature Flagging your Infrastructure for Fun and Pro!t Daniel Schauenberg dschauenberg@etsy.com @mrtazz Tuesday, September 10, 13
  2. 2. Tuesday, September 10, 13
  3. 3. LAMMP Item by TheBackPackShoppe Tuesday, September 10, 13
  4. 4. Item by FrankelPhotos Monolithic App Tuesday, September 10, 13
  5. 5. Etsy Infrastructure •~1400 nodes •~30 dev & ops engineers making changes regularly •Open Source Chef server + GitHub Enterprise •Default environment setup (production, development, testing) Tuesday, September 10, 13
  6. 6. jonlives/knife-spork Tuesday, September 10, 13
  7. 7. knife-spork •Work!ow to manage cookbook and environment changes •Versioned cookbooks and pinned environments •Speci"c work!ow, di#erent ways of using plugins Tuesday, September 10, 13
  8. 8. Tuesday, September 10, 13
  9. 9. % chef-shell chef > recipe_mode chef:recipe > echo off chef:recipe > include_recipe "apache" chef:recipe > run_chef Tuesday, September 10, 13
  10. 10. % review -r jcowie --cc ops Tuesday, September 10, 13
  11. 11. % knife spork check apache % knife spork bump apache % git commit % git push % knife spork upload apache Staging Deploy Tuesday, September 10, 13
  12. 12. jonlives/knife-!ip % knife node flip node.etsy.com testing % knife role flip testRole testing Tuesday, September 10, 13
  13. 13. % knife spork promote apache % git commit % git push % knife spork promote apache --remote Production Deploy Tuesday, September 10, 13
  14. 14. Monitoring Tuesday, September 10, 13
  15. 15. 19:18:06 irccat | CHEF: Daniel Schauenberg promoted apache@0.0.2 to development https:// github.etsycorp.com/gist/12345 Tuesday, September 10, 13
  16. 16. etsy/chef-handlers 19:20:00 irccat | Chef run failed on test.etsy.com 19:20:00 irccat | https://github.etsycorp.com/ gist/12347 jgoulah/knife-lastrun % knife node lastrun test.etsy.com Tuesday, September 10, 13
  17. 17. Graphs! Tuesday, September 10, 13
  18. 18. Downsides •Longer testing blocks others •Staged cookbooks can accidentally be promoted •Testing environment a#ects more than one cookbook •Used “upgrade” environments to circumvent Tuesday, September 10, 13
  19. 19. Feature Flags Tuesday, September 10, 13
  20. 20. Branching in Code •Well established pattern for “dark launches” •Used in the Etsy Web stack •Allows for restricted roll outs •http://code.!ickr.net/2009/12/02/!ipping-out/ Tuesday, September 10, 13
  21. 21. Tuesday, September 10, 13
  22. 22. etsy/chef-whitelist Tuesday, September 10, 13
  23. 23. chef-whitelist •Data bag driven whitelist •Library to include in cookbooks •Easy access to feature !ags Tuesday, September 10, 13
  24. 24. whitelist data bag { "id": "my_whitelist", "patterns": [ "host.example.com", "*.subdomain.example.com", "prefix*.example.com" ] } Tuesday, September 10, 13
  25. 25. whitelist data bag{ "id": "my_whitelist", "patterns": [ "host.example.com", "*.subdomain.example.com", "prefix*.example.com" ], "roles": [ "Webserver", "DatabaseServer" ] } Tuesday, September 10, 13
  26. 26. feature !ags in recipe if node.is_in_whitelist? "my_whitelist" # new hawtness else # old stuff end Tuesday, September 10, 13
  27. 27. Customizable node.is_in_whitelist? "my_whitelist", "acl", "hosts" Tuesday, September 10, 13
  28. 28. real world example { "id": "php-5-4-19", "patterns": [ "dschauenberg.vm.dev.etsy.com", "web0270.etsy.com", "api04.etsy.com", "imgcache01.etsy.com", "imgwriter01.etsy.com", "worker01.etsy.com", "beacon01.etsy.com", "paymentsweb01.etsy.com" ], "roles": [ ] } Tuesday, September 10, 13
  29. 29. Advantages •Easy to access list of what gets upgrades •Upgrades don’t need the spork work!ow •Pattern already known by all engineers Tuesday, September 10, 13
  30. 30. Downsides •Changes outside the regular work!ow •No graphs (yet) •Less visible cleanup required Tuesday, September 10, 13
  31. 31. Summary •GitHub Enterprise, Dev VMs, chef-shell as development environment •Chef Server and knife-spork as Deployment System •Feature !agging with chef-whitelist •Monitoring, Noti"cations, Graphs Tuesday, September 10, 13
  32. 32. •http://codeascraft.etsy.com/ •http://codeascraft.com/2013/08/02/ infrastructure-upgrades-with-chef/ •http://www.slideshare.net/jonlives/michelin- starred-cooking-with-chef •http://www.slideshare.net/mcdonnps/lessons- from-etsy-avoiding-kitchen-nightmares- chefconf-2012 •https://github.com/etsy Further information Tuesday, September 10, 13
  33. 33. Thank you! Questions? Tuesday, September 10, 13
  34. 34. Feature Flagging your Infrastructure for Fun and Pro!t Daniel Schauenberg dschauenberg@etsy.com @mrtazz Tuesday, September 10, 13
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×