Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Cfg mgmtcamp c-dwithchef

3,762 views

Published on

Published in: Technology, Self Improvement
  • Be the first to comment

Cfg mgmtcamp c-dwithchef

  1. 1. So Continuous. Much Delivery. Very Chef. Wow. A Case Study on using Chef to start building a Continuous Delivery Pipeline
  2. 2. About Me • George Miranda • Sr Consultant at Chef Software, Inc. • Unix guy (15+ years)
  3. 3. Minimum Viable Pipeline
  4. 4. What we know • Step 1: Develop a new change • Step 2: ??? • Step 3: Production! • MOAR FASTERZ
  5. 5. Case Study: Requirements • Must utilize existing tools within the company • Git for SCM • Jenkins approved for use • Working in a static VM environment • Just migrated to single cookbook repos • Starting with infrastructure cookbooks • Want a manual go-to-production button (ugh!)
  6. 6. Case Study: Code Review Model • Git PR model: branch from master for any new feature • 4-person team, only 3 active at any time • Code review done manually and informally • Simple communication/reqs (makes it easy!)
  7. 7. Figuring out new workflow • How are developers expected to work locally? • When do they push to remote? How do we verify their work? • Code Review criteria: what does it mean to be ready to merge? • How do we go from merged code to artifact? • How do we get that artifact all the way to Production?
  8. 8. Local Development Work • New branch for every feature • Create a failing test • Write a resource to pass the test • Local commits • Test-Kitchen + guard • Once local tests passed, push to remote
  9. 9. Push to remote • Open a Pull Request (new branch to master) • Triggers a build via Jenkins GHPRB plugin
  10. 10. Push to remote The Verify Build Job • Verify syntax (knife cookbook check) • Foodcritic Rules • Test-Kitchen w/ BATS busser
  11. 11. BATS: Simple Unit Tests @test "My directory is created" {! test -d /foo/bar! }! ! @test "A basharific test" {! if [ foo != bar ]; then! skip "foo isn't bar"! fi! ! run foo! [ "$status" -eq 0 ]! }! ! • • https://github.com/sstephenson/bats Super low learning curve (but also very limited)
  12. 12. Push to remote • If failed, notify • Another commit to the same branch triggers another Verify Build Job • Super easy to track, comment, and approve • If passed, let’s go to Human Code Review
  13. 13. Human Code Review Rules • Only one change per one cookbook at one time • Must have test for feature that changed • One for one: resource unit tests • Consider the smoke test
  14. 14. Unit Test vs Smoke Test • Unit tests: small, fast, check one single concern • • Smoke tests: test multiple things in the course of one concern • • In this context: checking Chef resources In this context: check the intent of a recipe Note: that was testing for this use case
  15. 15. When are we ready to merge? • Only 3 active team members at any given time • • • Submitter cannot approve Merge approval requires 2 approvals Code review can happen at any time, but only merge when you’re ready to fix it.
  16. 16. Merged code to artifact • Freeze your cookbooks! • Semantic versioning: Major.Minor.Patch • • • You own Major.Minor The Pipeline owns .Patch No one gets to knife upload No one.! Ever.! • "git merge" is the new "knife upload"
  17. 17. The Integration Job • Bumps Cookbook version • Re-commits to master • Upload frozen cookbook (via berks) • Pin that new cookbook to the Integration environment • Converge all nodes that use that cookbook
  18. 18. The Integration Job • First sign that things may be broken • These nodes also run smoke tests • serverspec, minitest, etc
  19. 19. The Integration Job • We survived! Trigger the next job(s) • The Jenkins Build Pipelines Plugin allows upstream/downstream definitions to string together jobs • From here out, it’s all the same Promote Job* • After the Integration job, we just run X number of Promote Jobs * (mostly)
  20. 20. Promote Jobs • Pin cookbook to new Chef Environment • Converge all nodes using this cookbook • Run Tests
  21. 21. Pin the cookbook to Env #!/opt/chef/embedded/bin/ruby ! require 'chef/environment' require 'chef' Chef::Config.from_file("/var/lib/jenkins/tools/knife.rb") ! def pin_env(env, cookbook_versions) to = Chef::Environment.load(env) cookbook_versions.each do |cb, version| puts "Pinning #{cb} #{version} in #{env}" to.cookbook_versions[cb] = version end to.save end ! cookbook_data = Array.new ! if File.exists?(File.expand_path(File.join(ENV['WORKSPACE'], 'metadata.rb'))) metadata_file = File.expand_path(File.join(ENV['WORKSPACE'], 'metadata.rb')) File.read(metadata_file).each_line do |line| if line =~ /^names+["'](w+)["'].*$/ cookbook_data << $1 end if line =~ /^versions+["'](d+.d+.d+)["'].*$/ cookbook_data << "= #{$1}" end end end ! cookbook_versions = Hash[*cookbook_data] ! pin_env(ARGV[0], cookbook_versions)
  22. 22. Pin the cookbook to Env $ berks apply <environment>
  23. 23. Converge Nodes $ knife ssh "recipes:mycookbook AND chef_environment:promote-environment” 'sudo chef-client'! … OR … Pushy!
  24. 24. Run Tests • Most testing frameworks have a Report Handler to automatically run tests • chef-serverspec-handler • minitest-handler • Deploy to your nodes by adding ‘chef_handler’ to their run_list • Many community cookbooks are already packaged with tests
  25. 25. Run Tests • In this particular use case: • Build job: BATS (unit tests) • Integration & Promote jobs: serverspec (smoke tests) • UAT: also ran Cucumber tests (acceptance)
  26. 26. Promoting to more environments • Can string together N number of promotions • UAT • Production A • Production B • etc
  27. 27. Push to Production • In production monitoring is the test • Could not queue up changes reliably anyway • There is no spoon
  28. 28. Results • Small incremental deployments led to greater confidence • TDD was pushed to the forefront of priorities • Commitment from Dev group to write application deployment cookbooks • But the biggest lesson learned…
  29. 29. Let’s Go Devop with a CD tool • Continuous Delivery is a practice, not a tool • Small incremental changes in code • Small incremental changes in workflow • Small incremental changes in tooling • You will constantly improve your code, your workflow, your tools, your team, and your skills.
  30. 30. RECAP
  31. 31. What We Wanted • Step 1: Develop a new change • Step 2: ??? • Step 3: Production! • MOAR FASTERZ
  32. 32. Wait… what was Step 2? • (Pre-req) Test Driven Development • 2A. Establish development workflow before submitting changes * • 2B. Auto verification of submission before humans look at it • 2C. Humans Apply Code Review Criteria * • 2D. Don’t merge unless you mean it * • 2E. Merge kicks off an Integration Job • 2F. Followed by a series of Promotion Jobs • 2G. There is no spoon *
  33. 33. What We Got • • Step 1: Develop a new change Step 2: (Pre-req) Test Driven Development 2A. Establish development workflow before submitting changes * 2B. Auto verification of submission before humans look at it ! 2C. Humans Apply Code Review Criteria * 2D. Don’t merge unless you mean it * 2E. Merge kicks off an Integration Job ! 2F. Followed by a series of Promotion Jobs 2G. There is no spoon * • Step 3: Production! • Step 4: Level Up. This is great! • Step 5: MOAR THINGS! Wait. This is hard! • Go to Step 1
  34. 34. Key Chef Ecosystem Tools • Test Kitchen — http://kitchen.ci/ • Guard Plugin for Test Kitchen — https://github.com/test-kitchen/guard-kitchen • Foodcritic — http://acrmp.github.io/foodcritic/ • Berkshelf — http://berkshelf.com/
  35. 35. Helpful Jenkins Plugins • git • github • build-pipeline-plugin • ghprb • warnings • mailer
  36. 36. I want to hear from you! ! @gmiranda23 gmiranda@getchef.com

×