1. So Continuous.
Much Delivery.
Very Chef.
Wow.
A Case Study on using Chef to start building a
Continuous Delivery Pipeline

2. About Me
•
George Miranda
•
Sr Consultant at Chef Software, Inc.
•
Unix guy (15+ years)

6. Minimum Viable Pipeline

7. What we know
•
Step 1: Develop a new change
•
Step 2: ???
•
Step 3: Production!
•
MOAR FASTERZ

8. Case Study: Requirements
•
Must utilize existing tools within the company
•
Git for SCM
•
Jenkins approved for use
•
Working in a static VM environment
•
Just migrated to single cookbook repos
•
Starting with infrastructure cookbooks
•
Want a manual go-to-production button (ugh!)

9. Case Study: Code Review Model
•
Git PR model: branch from master for any
new feature
•
4-person team, only 3 active at any time
•
Code review done manually and informally
•
Simple communication/reqs (makes it easy!)

10. Figuring out new workflow
•
How are developers expected to work locally?
•
When do they push to remote? How do we
verify their work?
•
Code Review criteria: what does it mean to be
ready to merge?
•
How do we go from merged code to artifact?
•
How do we get that artifact all the way to
Production?

11. Local Development Work
•
New branch for every feature
•
Create a failing test
•
Write a resource to pass the test
•
Local commits
•
Test-Kitchen + guard
•
Once local tests passed, push to remote

12. Push to remote
•
Open a Pull Request (new branch to master)
•
Triggers a build via Jenkins GHPRB plugin

13. Push to remote
The Verify Build Job
•
Verify syntax (knife cookbook check)
•
Foodcritic Rules
•
Test-Kitchen w/ BATS busser

14. BATS: Simple Unit Tests
@test "My directory is created" {!
test -d /foo/bar!
}!
!
@test "A basharific test" {!
if [ foo != bar ]; then!
skip "foo isn't bar"!
fi!
!
run foo!
[ "$status" -eq 0 ]!
}!
!
•
•
https://github.com/sstephenson/bats
Super low learning curve (but also very limited)

15. Push to remote
•
If failed, notify
•
Another commit to the same branch
triggers another Verify Build Job
•
Super easy to track, comment, and approve
•
If passed, let’s go to Human Code Review

16. Human Code Review Rules
•
Only one change per one cookbook at one
time
•
Must have test for feature that changed
•
One for one: resource unit tests
•
Consider the smoke test

17. Unit Test vs Smoke Test
•
Unit tests: small, fast, check one single
concern
•
•
Smoke tests: test multiple things in the course
of one concern
•
•
In this context: checking Chef resources
In this context: check the intent of a recipe
Note: that was testing for this use case

18. When are we ready to merge?
•
Only 3 active team members at any given
time
•
•
•
Submitter cannot approve
Merge approval requires 2 approvals
Code review can happen at any time, but
only merge when you’re ready to fix it.

19. Merged code to artifact
•
Freeze your cookbooks!
•
Semantic versioning: Major.Minor.Patch
•
•
•
You own Major.Minor
The Pipeline owns .Patch
No one gets to knife upload
No one.!
Ever.!
•
"git merge" is the new "knife upload"

21. The Integration Job
•
Bumps Cookbook version
•
Re-commits to master
•
Upload frozen cookbook (via berks)
•
Pin that new cookbook to the Integration
environment
•
Converge all nodes that use that cookbook

22. The Integration Job
•
First sign that things may be broken
•
These nodes also run smoke tests
•
serverspec, minitest, etc

23. The Integration Job
•
We survived! Trigger the next job(s)
•
The Jenkins Build Pipelines Plugin allows
upstream/downstream definitions to string
together jobs
•
From here out, it’s all the same Promote Job*
•
After the Integration job, we just run X number
of Promote Jobs
* (mostly)

27. Promote Jobs
•
Pin cookbook to new Chef Environment
•
Converge all nodes using this cookbook
•
Run Tests

28. Pin the cookbook to Env
#!/opt/chef/embedded/bin/ruby
!
require 'chef/environment'
require 'chef'
Chef::Config.from_file("/var/lib/jenkins/tools/knife.rb")
!
def pin_env(env, cookbook_versions)
to = Chef::Environment.load(env)
cookbook_versions.each do |cb, version|
puts "Pinning #{cb} #{version} in #{env}"
to.cookbook_versions[cb] = version
end
to.save
end
!
cookbook_data = Array.new
!
if File.exists?(File.expand_path(File.join(ENV['WORKSPACE'], 'metadata.rb')))
metadata_file = File.expand_path(File.join(ENV['WORKSPACE'], 'metadata.rb'))
File.read(metadata_file).each_line do |line|
if line =~ /^names+["'](w+)["'].*$/
cookbook_data << $1
end
if line =~ /^versions+["'](d+.d+.d+)["'].*$/
cookbook_data << "= #{$1}"
end
end
end
!
cookbook_versions = Hash[*cookbook_data]
!
pin_env(ARGV[0], cookbook_versions)

29. Pin the cookbook to Env
$ berks apply <environment>

30. Converge Nodes
$ knife ssh "recipes:mycookbook AND
chef_environment:promote-environment”
'sudo chef-client'!
… OR …
Pushy!

31. Run Tests
•
Most testing frameworks have a Report
Handler to automatically run tests
•
chef-serverspec-handler
•
minitest-handler
•
Deploy to your nodes by adding
‘chef_handler’ to their run_list
•
Many community cookbooks are already
packaged with tests

32. Run Tests
•
In this particular use case:
•
Build job: BATS (unit tests)
•
Integration & Promote jobs: serverspec
(smoke tests)
•
UAT: also ran Cucumber tests (acceptance)

34. Promoting to more environments
•
Can string together N number of promotions
•
UAT
•
Production A
•
Production B
•
etc

37. Push to Production
•
In production monitoring is the test
•
Could not queue up changes reliably anyway
•
There is no spoon

38. Results
•
Small incremental deployments led to greater
confidence
•
TDD was pushed to the forefront of priorities
•
Commitment from Dev group to write
application deployment cookbooks
•
But the biggest lesson learned…

39. Let’s Go Devop with a CD tool
•
Continuous Delivery is a practice, not a tool
•
Small incremental changes in code
•
Small incremental changes in workflow
•
Small incremental changes in tooling
•
You will constantly improve your code, your
workflow, your tools, your team, and your
skills.

40. RECAP

41. What We Wanted
•
Step 1: Develop a new change
•
Step 2: ???
•
Step 3: Production!
•
MOAR FASTERZ

42. Wait… what was Step 2?
•
(Pre-req) Test Driven Development
•
2A. Establish development workflow before submitting changes *
•
2B. Auto verification of submission before humans look at it
•
2C. Humans Apply Code Review Criteria *
•
2D. Don’t merge unless you mean it *
•
2E. Merge kicks off an Integration Job
•
2F. Followed by a series of Promotion Jobs
•
2G. There is no spoon *

43. What We Got
•
•
Step 1: Develop a new change
Step 2:
(Pre-req) Test Driven Development
2A. Establish development workflow before submitting changes *
2B. Auto verification of submission before humans look at it
!
2C. Humans Apply Code Review Criteria *
2D. Don’t merge unless you mean it *
2E. Merge kicks off an Integration Job
!
2F. Followed by a series of Promotion Jobs
2G. There is no spoon *
•
Step 3: Production!
•
Step 4: Level Up. This is great!
•
Step 5: MOAR THINGS! Wait. This is hard!
•
Go to Step 1

44. Key Chef Ecosystem Tools
•
Test Kitchen — http://kitchen.ci/
•
Guard Plugin for Test Kitchen —
https://github.com/test-kitchen/guard-kitchen
•
Foodcritic — http://acrmp.github.io/foodcritic/
•
Berkshelf — http://berkshelf.com/

45. Helpful Jenkins Plugins
•
git
•
github
•
build-pipeline-plugin
•
ghprb
•
warnings
•
mailer

46. I want to hear from you!
!
@gmiranda23
gmiranda@getchef.com