An Open-Source
Chef Cookbook CI/CD Implementation
Using Jenkins Pipelines
Steffen Gebert (@StGebert)
Config Management Camp, Gent, 06.02.2017

2
Agenda
• Context
• Motivation
• Jenkins Pipelines (in general)
• Chef CI/CD using Jenkins Pipelines

About Me
Researcher / PhD Student
(Software-based Networks)
2011 - 2016
Core Team Member
2010 - 2013
Server Admin Team Member
since 2011
3
Co-Organizer DevOps Meetup Würzburg
since 2016

4
PHP-based Content Management System
Picture by Benjamin Kott

5
TYPO3 Open Source Project
• Established open-source
community since 1997
• TYPO3 Association as non-profit
organization
• Server Admin Team responsible
for project infrastructure *
Thanks for partially
funding my work and
covering travel costs!
* listed on opensourceinfra.org
Picture by Daniel Pötzinger: http://www.typo3-media.com/blog/article/typo3-wallpaper-for-wide-screen.html
Based on image by Mike Swanson: http://interfacelift.com/wallpaper/details.php?id=1402
PicturebyStefanBusemann

6
Chef at TYPO3
• Started with Chef in 2011
• Started with librarian-chef (monolithic repo)
• Took years to get rid of monolithic repo
• Currently operating 45 Debian nodes
• Usual tooling: berkshelf, test-kitchen
• berkshelf-api-based /universe (connecting to Chef Server 11*)
• Small team of volunteers, varying degree of Chef knowledge
* because of laziness

7
Demo Time!
Picture by annca/ pixabay:
https://pixabay.com/en/popcorn-cinema-ticket-film-1433327/

Working with Chef
(and knife, and berks, and..)

9
We have
workflow problems!
Picture by tpsdave / pixabay:
https://pixabay.com/en/firefighters-fire-flames-outside-115800/

10

11

12
https://blog.twitter.com/2017/the-infrastructure-behind-twitter-scale

13
Related Work
• zts/cooking-with-jenkins
• Provides resource to create single Jenkins freestyle jobs per cookbook
• Executes pre-defined rake tasks
• chef-solutions/pipeline
• Reads cookbooks from a Berksfile enclosed in main chef-repo
• Creates single Jenkins freestyle job per cookbook
• Chef Automate
• Commercial
• Little known to me, what's going on under the hood

14
Survey
• Who’s Chef cookbooks (Puppet modules/etc) are tested in CI?
• Who has an automated release process?

15
Jenkins Pipelines

16
Jenkins Pipelines

17

18

Jenkins Pipelines
Introduction to

20
Jenkins Pipeline DSL
• Groovy DSL
• Provides steps, like sh
• Defined in the Jenkinsfile
• Can use Groovy/Java logic (Chef users like that idea, right?)
• Pipeline plugin (workflow-aggregator)
• Formerly called "Workflow"
• Open-sourced one year ago
• Whole bunch of plugins (10+)
sh "make"
sh "make install"

21
Jenkins Jobs as Code?
• Jenkins Job Builder
• Python / YAML based
• From OpenStack
• Job DSL plugin (job-dsl)
• Also Groovy DSL
• Supports many, many plugins
• Creates only single jobs, not pipelines
# Job DSL example
job('my-project-main') {
scm {
git('https://github.com/...')
}
triggers {
scm('H/15 * * * *')
}
publishers {
downstream('my-project-unit')
}
}

22
Cookbook Pipeline?
sh 'berks install'
sh 'foodcritic'
sh 'kitchen test'
sh 'berks upload'
Picture by BarnImages / pixabay:
https://pixabay.com/en/sushi-food-japanese-fish-seafood-789820/

23
Stages
• Allow visual grouping
stage('lint') {
sh 'foodcritic'
sh 'cookstyle'
}
stage('resolve dependencies') {
sh 'berks install'
}
stage('test-kitchen') {
sh 'kitchen test'
}
stage('upload') {
sh 'berks upload'
}

24
Nodes
• Allocates a Jenkins executor (master/slave)
• Optionally with label
node {
stage('lint') {
sh '..'
}
stage('resolve') {
sh '..'
}
}
node('chefdk') {
stage('lint') {..}
stage('resolve') {..}
}
Master Agent 2
Agent 1
Executors (node):
Pipeline
Stage
Step
Step
Stage
Step
Step
Stage
Step
Step

25
Ready, Set, Go!

26
Multibranch Jobs
• Scans repo for branches containing
Jenkinsfile
• Automatically creates (deletes) jobs
• Works with plain Git
• Works better™ with GitHub / Bitbucket (via API calls)

27
More Steps to Come..
• Jenkins Plugins can contribute DSL steps

28
Global Variables
• Environment variables:
• Information about current build:
• Parameterized build*:
env.BRANCH_NAME
env.BUILD_URL
params.myparam
params.deployEnv
* well-hidden feature
see https://st-g.de/2016/12/parametrized-jenkins-pipelines
currentBuild.displayName = "v1.2.3"
CurrentBuild.rawBuild.getCause()
currentBuild.result = 'FAILED'

29
Now Copy & Paste?
Picture by aitroff / pixabay:
https://pixabay.com/en/stormtrooper-star-wars-lego-storm-1343772/

30
Pipeline Shared Libraries
• Additional code to be used by Jenkinsfiles
• Loaded from SCM repo
• Inclusion via
• @Library annotation in Jenkinsfile
• Configuration of enclosing folder
• Jenkins global configuration*
* then library code is "trusted"

31
Configuring Global Pipeline Library
• Jenkinsfile:
• Load implicitly would load it
without @Library
• Use non-default branch/tag
@Library('chefci')
org.example.Pipeline ...
@Library('chefci@testing')
Don't play
well together

32
Pipeline Shared Libraries
(root)
+- src # Groovy source files
| +- org
| +- foo
| +- Bar.groovy # for org.foo.Bar class
|
+- vars
| +- foo.groovy # for global 'foo' variable
| +- foo.txt # help for 'foo' variable
|
+- resources # resource files
| +- org
| +- foo
| +- bar.json # static helper data for org.foo.Bar
https://jenkins.io/doc/book/pipeline/shared-libraries/
static files
feel like
functions
the magic –
actually code

33
Global Variables
• Can store global state (for the current build)
• Can behave like steps
• Useful to simplify pipeline code
• Hide implementation details from users
# vars/deployTo.groovy
def call(def environment) {
echo "Starting deployment to ${environment}"
withCredentials(..) {
sh "rsync –avzh . ${environment}.example.com"
}
}
# Jenkinsfile
node {
sh "make test"
deployTo "production"
}

34
More Magic: Global Library Classes (src/)
• Groovy classes implementing arbitrary logic
• Make use of pipeline DSL steps
• Use other Jenkins functions
• Import and use Java libraries
à How we implement our pipeline

35
# Declarative Jenkinsfile
pipeline {
agent label:'has-docker', dockerfile: tru
environment {
GIT_COMMITTER_NAME = "jenkins"
}
stages {
stage("Build") {
steps { sh 'mvn clean install' }
}
stage("Archive"){
// ..
}
}
post {
always {
deleteDir()
}
success {
mail to:"me@example.com", subject:"SUC
Scripted vs. Declarative Pipelines
• Scripted pipelines
• Just (imperative) Groovy code
• The original implementation
• The approach used here
• Declarative pipelines
• Hit the 1.0 release last Friday
• Can be validated prior to execution
• Ease some tasks, i.e., failure handling
• Visual Pipeline Editor plugin

36
Jenkins Pipeline Summary
• Groovy DSL for specifying pipelines as code
• Ideally stored within the tested repo
• Extract shared functionality to library repos
• Pipeline execution triggered on push event

jenkins-chefci
An Open-Source Chef Cookbook CI/CD Implementation
Using Jenkins Pipelines

38
Forbidden Commands
$ berks upload
$ knife cookbook upload
$ knife data bag from file
$ knife data bag delete
$ knife environment from file
$ knife environment delete
$ knife role from file
$ knife role delete
Allowed Commands
$ git pull
$ git commit
$ git push
Pictures by PublicDomainPictures & serrano / pixabay:
https://pixabay.com/en/costume-demon-devil-board-female-15747/
https://pixabay.com/en/bebe-girl-child-child-portrait-1237704/

39
Meet the jenkins-chefci cookbook
• Sets up a Chef CI/CD infrastructure using Jenkins
• Sets up Jenkins master
• Installs ChefDK, configures credentials
• Configures job that scans a GitHub organization
• Configures our shared pipeline library
https://github.com/TYPO3-infrastructure/jenkins-pipeline-global-library-chefci/
• Add this Jenkinsfile to your cookbooks:
# Jenkinsfile
@Library('chefci') _
org.typo3.chefci.v2.cookbook.CookbookPipeline.builder(this, steps)
.buildDefaultPipeline().execute()

40
Meet the Shared Pipeline Library
• Implements pipelines for
• Cookbook testing, versioning & upload
• [WIP] Chef-repo's data bags, environments, roles
• Contains couple of Groovy classes
• Some helper classes
• *Pipeline as entry points from Jenkinsfile
• AbstractStage as base class for.. Stages
• Yes, we're at v2 already J

41
Executed Stages & Steps
• Linting
• Foodcritic
• Cookstyle
• Build
• Berkshelf install (endpoints for Supermarket & Chef Server)
Berksfile.lock in Git for top-level cookbooks
• Acceptance
• Test-Kitchen (using kitchen-docker)
• Publish
• Version Bump (thor-scmversion, based on user feedback)
• Berkshelf upload (to Chef Server)
Picture by Kaz/ pixabay:
https://pixabay.com/en/hands-hand-raised-hands-raised-220163/

42

43
node {
createKitchenYaml()
stash "cookbook"
}
parallel(
"essentials-debian-86": { node {
unstash "cookbook"
sh "kitchen test --destroy always essentials-debian-86" }},
"essentials-ubuntu-1604": { node { .. } },
"full-debian-86": { node { .. } },
"full-ubuntu-1604": { node { .. } }
)
Test-Kitchen
• If there is no .kitchen.docker.yml, put default one
• Use Jenkins' parallel step
expected result,
not hard-coded

44
Test-Kitchen (2)
• Parallel instances automatically derived from kitchen status
• Log files of failed instances are archived
• Less chaos than complete Jenkins log
• Traditional UI does not clearly show failed parallel branch
• Not limited to kitchen-docker

45* cookbook configures this via API

46

47

48

49
The Art of Cookbook Versioning
• Let's agree on SemVer
• I do not agree with manual changes to metadata.rb

50
Demo Time!
Picture by annca/ pixabay:
https://pixabay.com/en/popcorn-cinema-ticket-film-1433327/

51
Versioning
• Based on thor-scmversion
• Made by RiotGames for their cookbooks
• Uses Git tags to derive current version
• Version treated as "label", not part of the source code
• Jenkins pushes only Git tags, no code changes
# metadata.rb
name "example-cookbook"
version IO.read(File.join(File.dirname(__FILE__), 'VERSION')) rescue '0.0.1'

52
Versioning (2)
• Publish stage notifies via Slack
• Except commit message subject
indicates version increase
• Do this #patch
• Add this #minor
• Break this #major

53
"Main Chef-Repo"
• Not covered here, but in site-chefcitypo3org cookbook
• Freestyle job, defined via JobDSL
• Uses script to parse git diff and
upload changes using knife

54
Additional Notes
• Manually configured J organization-level webhook triggers
immediate builds
• Everything else is done automatically, no "manual handholding"

Using jenkins-chefci

56
Ease of Use
• Checkout cookbook from github.com/TYPO3-cookbooks/jenkins-chefci
• Increase GitHub API rate limit
• Allow to steal your Chef credentials
• Run test-kitchen
export JENKINS_GITHUB_LOGIN=johndoe
export JENKINS_GITHUB_TOKEN=123456supersecure
export JENKINS_COPY_CHEF_CREDENTIALS=1
kitchen converge full-ubuntu-1604
WARN: allows commit
status update
WARN: allows knife/berks
to access Chef Server

57
(Potential) TODOs for you
• Write your wrapper cookbook around jenkins-chefci
• Point it to your organization
• Add authentication, e.g., GitHub OAuth?
• Fork the global library
• Adjust pipeline to your needs
• You don't have to agree with our (current) implementation

58
Outlook / TODOs
• Documentation / blog posts
• Move more stuff from site-chefcitypo3org to jenkins-chefci
• chefdk() global function to run in chef/chefdk Docker container
• Store Chef private key as Jenkins credential
• Test using multiple chef-client version
• Use JobDSL for organization-folder setup (#966)
• Trigger downstream cookbooks
• Read out dependencies and subscribe to successful pipeline runs
• Paves the road to Policyfiles
• Use Jenkins slaves
• Collect chef-client deprecation warnings

59
Conclusion
• Jenkins Pipelines allow to define pipelines as code
• Groovy-based DSL allows programming of pipelines
• Can definitively become complex, i.e., debugging
• Jenkins Pipelines for Chef cookbook CI/CD
• Running at TYPO3 since May 2016 (site-chefcitypo3org cookbook)
• Public instance at https://chef-ci.typo3.org, open source from day 1
• Warning: Many cookbooks still use v1 pipeline (git-flow)
• jenkins-chefci cookbook as reusable implementation
• Makes setup accessible to broader audience
• No dependencies to other TYPO3 cookbooks

60
Further Reading
• TYPO3's Chef CI:
https://chef-ci.typo3.org
• TYPO3-cookbooks on GitHub:
https://github.com/TYPO3-cookbooks
• TYPO3's Shared Global Library:
https://github.com/TYPO3-infrastructure/jenkins-pipeline-global-library-chefci/
• Pipeline Tutorial:
https://github.com/jenkinsci/pipeline-plugin/blob/master/TUTORIAL.md
• Getting started with pipelines:
https://jenkins.io/pipeline/getting-started-pipelines/
• Step documentation:
https://jenkins.io/doc/pipeline/steps/
• Pipeline shared libraries:
https://jenkins.io/doc/book/pipeline/shared-libraries/
• Notifications (Mail, Slack, etc.) in Scripted Pipelines:
https://jenkins.io/blog/2016/07/18/pipline-notifications/
• Declarative Pipelines
https://jenkins.io/blog/2016/12/19/declarative-pipeline-beta/
https://jenkins.io/blog/2017/02/03/declarative-pipeline-ga/