Cfg mgmtcamp c-dwithchef

3,522 views

Published on

Published in: Technology, Self Improvement
0 Comments
10 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,522
On SlideShare
0
From Embeds
0
Number of Embeds
108
Actions
Shares
0
Downloads
33
Comments
0
Likes
10
Embeds 0
No embeds

No notes for slide

Cfg mgmtcamp c-dwithchef

  1. 1. So Continuous. Much Delivery. Very Chef. Wow. A Case Study on using Chef to start building a Continuous Delivery Pipeline
  2. 2. About Me • George Miranda • Sr Consultant at Chef Software, Inc. • Unix guy (15+ years)
  3. 3. Minimum Viable Pipeline
  4. 4. What we know • Step 1: Develop a new change • Step 2: ??? • Step 3: Production! • MOAR FASTERZ
  5. 5. Case Study: Requirements • Must utilize existing tools within the company • Git for SCM • Jenkins approved for use • Working in a static VM environment • Just migrated to single cookbook repos • Starting with infrastructure cookbooks • Want a manual go-to-production button (ugh!)
  6. 6. Case Study: Code Review Model • Git PR model: branch from master for any new feature • 4-person team, only 3 active at any time • Code review done manually and informally • Simple communication/reqs (makes it easy!)
  7. 7. Figuring out new workflow • How are developers expected to work locally? • When do they push to remote? How do we verify their work? • Code Review criteria: what does it mean to be ready to merge? • How do we go from merged code to artifact? • How do we get that artifact all the way to Production?
  8. 8. Local Development Work • New branch for every feature • Create a failing test • Write a resource to pass the test • Local commits • Test-Kitchen + guard • Once local tests passed, push to remote
  9. 9. Push to remote • Open a Pull Request (new branch to master) • Triggers a build via Jenkins GHPRB plugin
  10. 10. Push to remote The Verify Build Job • Verify syntax (knife cookbook check) • Foodcritic Rules • Test-Kitchen w/ BATS busser
  11. 11. BATS: Simple Unit Tests @test "My directory is created" {! test -d /foo/bar! }! ! @test "A basharific test" {! if [ foo != bar ]; then! skip "foo isn't bar"! fi! ! run foo! [ "$status" -eq 0 ]! }! ! • • https://github.com/sstephenson/bats Super low learning curve (but also very limited)
  12. 12. Push to remote • If failed, notify • Another commit to the same branch triggers another Verify Build Job • Super easy to track, comment, and approve • If passed, let’s go to Human Code Review
  13. 13. Human Code Review Rules • Only one change per one cookbook at one time • Must have test for feature that changed • One for one: resource unit tests • Consider the smoke test
  14. 14. Unit Test vs Smoke Test • Unit tests: small, fast, check one single concern • • Smoke tests: test multiple things in the course of one concern • • In this context: checking Chef resources In this context: check the intent of a recipe Note: that was testing for this use case
  15. 15. When are we ready to merge? • Only 3 active team members at any given time • • • Submitter cannot approve Merge approval requires 2 approvals Code review can happen at any time, but only merge when you’re ready to fix it.
  16. 16. Merged code to artifact • Freeze your cookbooks! • Semantic versioning: Major.Minor.Patch • • • You own Major.Minor The Pipeline owns .Patch No one gets to knife upload No one.! Ever.! • "git merge" is the new "knife upload"
  17. 17. The Integration Job • Bumps Cookbook version • Re-commits to master • Upload frozen cookbook (via berks) • Pin that new cookbook to the Integration environment • Converge all nodes that use that cookbook
  18. 18. The Integration Job • First sign that things may be broken • These nodes also run smoke tests • serverspec, minitest, etc
  19. 19. The Integration Job • We survived! Trigger the next job(s) • The Jenkins Build Pipelines Plugin allows upstream/downstream definitions to string together jobs • From here out, it’s all the same Promote Job* • After the Integration job, we just run X number of Promote Jobs * (mostly)
  20. 20. Promote Jobs • Pin cookbook to new Chef Environment • Converge all nodes using this cookbook • Run Tests
  21. 21. Pin the cookbook to Env #!/opt/chef/embedded/bin/ruby ! require 'chef/environment' require 'chef' Chef::Config.from_file("/var/lib/jenkins/tools/knife.rb") ! def pin_env(env, cookbook_versions) to = Chef::Environment.load(env) cookbook_versions.each do |cb, version| puts "Pinning #{cb} #{version} in #{env}" to.cookbook_versions[cb] = version end to.save end ! cookbook_data = Array.new ! if File.exists?(File.expand_path(File.join(ENV['WORKSPACE'], 'metadata.rb'))) metadata_file = File.expand_path(File.join(ENV['WORKSPACE'], 'metadata.rb')) File.read(metadata_file).each_line do |line| if line =~ /^names+["'](w+)["'].*$/ cookbook_data << $1 end if line =~ /^versions+["'](d+.d+.d+)["'].*$/ cookbook_data << "= #{$1}" end end end ! cookbook_versions = Hash[*cookbook_data] ! pin_env(ARGV[0], cookbook_versions)
  22. 22. Pin the cookbook to Env $ berks apply <environment>
  23. 23. Converge Nodes $ knife ssh "recipes:mycookbook AND chef_environment:promote-environment” 'sudo chef-client'! … OR … Pushy!
  24. 24. Run Tests • Most testing frameworks have a Report Handler to automatically run tests • chef-serverspec-handler • minitest-handler • Deploy to your nodes by adding ‘chef_handler’ to their run_list • Many community cookbooks are already packaged with tests
  25. 25. Run Tests • In this particular use case: • Build job: BATS (unit tests) • Integration & Promote jobs: serverspec (smoke tests) • UAT: also ran Cucumber tests (acceptance)
  26. 26. Promoting to more environments • Can string together N number of promotions • UAT • Production A • Production B • etc
  27. 27. Push to Production • In production monitoring is the test • Could not queue up changes reliably anyway • There is no spoon
  28. 28. Results • Small incremental deployments led to greater confidence • TDD was pushed to the forefront of priorities • Commitment from Dev group to write application deployment cookbooks • But the biggest lesson learned…
  29. 29. Let’s Go Devop with a CD tool • Continuous Delivery is a practice, not a tool • Small incremental changes in code • Small incremental changes in workflow • Small incremental changes in tooling • You will constantly improve your code, your workflow, your tools, your team, and your skills.
  30. 30. RECAP
  31. 31. What We Wanted • Step 1: Develop a new change • Step 2: ??? • Step 3: Production! • MOAR FASTERZ
  32. 32. Wait… what was Step 2? • (Pre-req) Test Driven Development • 2A. Establish development workflow before submitting changes * • 2B. Auto verification of submission before humans look at it • 2C. Humans Apply Code Review Criteria * • 2D. Don’t merge unless you mean it * • 2E. Merge kicks off an Integration Job • 2F. Followed by a series of Promotion Jobs • 2G. There is no spoon *
  33. 33. What We Got • • Step 1: Develop a new change Step 2: (Pre-req) Test Driven Development 2A. Establish development workflow before submitting changes * 2B. Auto verification of submission before humans look at it ! 2C. Humans Apply Code Review Criteria * 2D. Don’t merge unless you mean it * 2E. Merge kicks off an Integration Job ! 2F. Followed by a series of Promotion Jobs 2G. There is no spoon * • Step 3: Production! • Step 4: Level Up. This is great! • Step 5: MOAR THINGS! Wait. This is hard! • Go to Step 1
  34. 34. Key Chef Ecosystem Tools • Test Kitchen — http://kitchen.ci/ • Guard Plugin for Test Kitchen — https://github.com/test-kitchen/guard-kitchen • Foodcritic — http://acrmp.github.io/foodcritic/ • Berkshelf — http://berkshelf.com/
  35. 35. Helpful Jenkins Plugins • git • github • build-pipeline-plugin • ghprb • warnings • mailer
  36. 36. I want to hear from you! ! @gmiranda23 gmiranda@getchef.com

×