Chef Workflow
at SecondMarket


                                     Julian Dunn
              Senior Systems Operations Engineer
                       jdunn@secondmarket.com
What’s Meant by Workflow?
What’s Meant by Workflow?



           How to not trample on others when making
           changes
           Encompasses the entire process of making the
           change
            – cookbook development or modification
            – unit/integration testing
            – deployment
            – notifications



Member FINRA | MSRB | SIPC          © 2012 SecondMarket Holdings, Inc.
Cookbook Development
Development Workflow



            Local development: Vagrant and VirtualBox
            Unit tests: ChefSpec

     RSpec.configure do |config|
      config.before(:all) do ::Chef::Config(:[]).with(:solo).and_return(nil) # or false
      end
     end

            Integration tests: test-kitchen w/ Minitests




Member FINRA | MSRB | SIPC                             © 2012 SecondMarket Holdings, Inc.
A Word About Git Organization



            Break cookbooks into individual repos
            – chef-cookbooks/{name}.git
            Allows:
            – Independent development
            – Tracking & merging from upstream
            – Contributing changes back




Member FINRA | MSRB | SIPC            © 2012 SecondMarket Holdings, Inc.
Roles as Data



            Roles contain only data, not run_lists
            Version the roles cookbook like others




Member FINRA | MSRB | SIPC             © 2012 SecondMarket Holdings, Inc.
Limitations of Local
   Development
Limitations of Local Cookbook Development



            Missing databags and search
            Develop against a Chef server
            – “cheftest” environment
            – vagrant-butcher
            – vagrant-vbguest




Member FINRA | MSRB | SIPC                   © 2012 SecondMarket Holdings, Inc.
Deployment
Deployment: It’s all about communication



            Spork
            – Uploading and freezing stable cookbooks
            – Change versions per environment
            – Notifications to HipChat
            Semantic versioning
            CHANGELOG.md / JIRA




Member FINRA | MSRB | SIPC                      © 2012 SecondMarket Holdings, Inc.
The Future
The Future



            More people touching Chef
            Modularize cookbooks
            Improve unit & integration testing
            Cookbooks in CI
            Code reviews




Member FINRA | MSRB | SIPC              © 2012 SecondMarket Holdings, Inc.
Thanks!




                           t: @julian_dunn
                      e: jdunn@aquezada.com
              g: github.com/secondmarket-cookbooks
                       w: www.juliandunn.net

                             SecondMarket’s hiring!


Member FINRA | MSRB | SIPC                 © 2012 SecondMarket Holdings, Inc.

Chef Workflow Strategies at SecondMarket

  • 1.
    Chef Workflow at SecondMarket Julian Dunn Senior Systems Operations Engineer jdunn@secondmarket.com
  • 2.
  • 3.
    What’s Meant byWorkflow? How to not trample on others when making changes Encompasses the entire process of making the change – cookbook development or modification – unit/integration testing – deployment – notifications Member FINRA | MSRB | SIPC © 2012 SecondMarket Holdings, Inc.
  • 4.
  • 5.
    Development Workflow Local development: Vagrant and VirtualBox Unit tests: ChefSpec RSpec.configure do |config| config.before(:all) do ::Chef::Config(:[]).with(:solo).and_return(nil) # or false end end Integration tests: test-kitchen w/ Minitests Member FINRA | MSRB | SIPC © 2012 SecondMarket Holdings, Inc.
  • 6.
    A Word AboutGit Organization Break cookbooks into individual repos – chef-cookbooks/{name}.git Allows: – Independent development – Tracking & merging from upstream – Contributing changes back Member FINRA | MSRB | SIPC © 2012 SecondMarket Holdings, Inc.
  • 7.
    Roles as Data Roles contain only data, not run_lists Version the roles cookbook like others Member FINRA | MSRB | SIPC © 2012 SecondMarket Holdings, Inc.
  • 8.
  • 9.
    Limitations of LocalCookbook Development Missing databags and search Develop against a Chef server – “cheftest” environment – vagrant-butcher – vagrant-vbguest Member FINRA | MSRB | SIPC © 2012 SecondMarket Holdings, Inc.
  • 10.
  • 11.
    Deployment: It’s allabout communication Spork – Uploading and freezing stable cookbooks – Change versions per environment – Notifications to HipChat Semantic versioning CHANGELOG.md / JIRA Member FINRA | MSRB | SIPC © 2012 SecondMarket Holdings, Inc.
  • 12.
  • 13.
    The Future More people touching Chef Modularize cookbooks Improve unit & integration testing Cookbooks in CI Code reviews Member FINRA | MSRB | SIPC © 2012 SecondMarket Holdings, Inc.
  • 14.
    Thanks! t: @julian_dunn e: jdunn@aquezada.com g: github.com/secondmarket-cookbooks w: www.juliandunn.net SecondMarket’s hiring! Member FINRA | MSRB | SIPC © 2012 SecondMarket Holdings, Inc.

Editor's Notes

  • #2 Take a poll to see how long people have been using ChefNever - new to it< 6 months6 months - 1 year> 1 year
  • #3 What’s meant by workflow?
  • #4 If you’re organized in how you develop and test your cookbooks, you’ll also be organized when it comes to deploy them, by ensuring changes are tested
  • #6 Explain a bit about Vagrant and VirtualBox for those who are newUnit tests: Just started with this using ChefSpec – still somewhat immature in my view. First need to learn Rspec, but many things are not intuitiveIntegration tests: test-kitchen
  • #7 Unit tests: Just started with this using ChefSpec – still somewhat immature in my view. First need to learn Rspec, but many things are not intuitiveIntegration tests:
  • #10 Missing data bags and search: ChefSpec allows some mocking but that’s for unit testing only. When integration testing with test-kitchen, need a real Server
  • #12 When it comes time for deployment – obviously the most important thing is communication. The tools are used as a means to that end, to automate the routine aspects of communication but there is no substitute for talking to someone if it falls outside those bounds!Sporkto upload and freeze stable cookbooks, as well as promoting versions to certain environments.Unlike Etsy we pin all cookbook versions in all environments, because developers need a stable dev environment for their features.All spork notifications are sent to HipChat; we also graph deploys in Graphite (not currently used)Semantic versioning: unstable cookbooks are never frozen
  • #14 Training up some developers to use Chef as well. Could make Spring properties changes for example without involving OpsModularize cookbooks so that SecondMarket’s bespoke applications each have a cookbook. Easier for developers to handle (smaller pieces == smaller blast radius)