Continuous Development with Jenkins - Stephen Connolly at PuppetCamp Dublin '12

Continuous Deployment with Jenkins
Stephen Connolly
Architect, CloudBees

TL;DR http://goo.gl/kNgyu

©2012 CloudBees, Inc.
All Rights Reserved

Who am I?
• One of the first non-Sun contributors to
Jenkins.
– Creator of the Weather Icons!
• Architect @ CloudBees
• Member & Maven PMC @ Apache Software
Foundation
• In my previous lives:
– Software Engineer @ Avaya
– IT Project Manager @ Elan Pharmaceuticals
– PhD development chemist @ Bristol Myer Squibb
– Freelance software developer since 1989

©2012 CloudBees, Inc. 2
All Rights Reserved

Jenkins

What the feck is this Jenkins thing anyway?

All Rights Reserved

Introducing Jenkins
• What is Jenkins?
– A relatively recent product, but growing in popularity
– Currently used by*:
• Yahoo
• NASA
• JBoss / RedHat
• Amazon
• HP
• And more!

*Source: Kohsuke Kawaguchi, lead developer of the Jenkins project

4

Introducing Jenkins
• What makes Jenkins so great?

Distributed Builds

Reporting
Plugins
Ease of use

5

jenkins-ci.org
• OSS CI server
– Written in Java
• About 7 years old
• Easy to install/use
• Extensible via 530+ plugins
• Widely adopted
– 40K+ installations

All Rights Reserved

The Deed Poll sed -e ‘s/hudson/jenkins/g’
• You might be more 14000
Total created tickets
familiar with the IP 12000 Jenkins
encumbered old name Hudson
10000
of the project.
8000
• The project was
renamed to Jenkins in 6000
January 2011. 4000
• Nearly 100 releases of 2000
Jenkins since the
0 Total resolved
rebrand tickets
• About 8 releases of
Hudson since the fork.

All Rights Reserved

Just for Continuous Integration?

Yeah! Look mate, this is a PUPPET conference not a Continuous
Integration conference… show me that you’re relevant before I start
lobbing soft fruits!

All Rights Reserved

Puppet manifests are code too
TL;DR
puppet parser validate
• Syntax Check
– Use puppet parser validate, e.g.

for file in $(find . -iname '*.pp’)
do
--render-as s
--modulepath=modules
"$file" || exit 1;
done

All Rights Reserved

• Syntax Check ✔
• Static Analysis
– Checks for anti-patterns
– Checks conformance to the style guide
– Run puppet-lint http://puppet-lint.com/

TL;DR
puppet-lint

All Rights Reserved

• Syntax Check ✔ TL;DR
RSpec-puppet
• Static Analysis ✔
• Automated Tests
– Use RSpec-puppet http://rspec-puppet.com
There are a lot of people confused by the purpose of
these tests as they can’t test the result of the
manifest on a live system. That is not the point of
rspec-puppet.

Rspec-puppet tests are there to test the behaviour of
Puppet when it compiles your manifests into a catalogue
of Puppet resources.
Source: http://rspec-puppet.com/tutorial/

All Rights Reserved

• Syntax Check ✔
• Static Analysis ✔
• Automated Tests ✔
• Setup Jenkins to run these on every change to the
Puppet manifests
✍ You are storing your scripts in Version Control?
• Now you know your Puppet scripts are doing what they
should do…
TL;DR
Jenkins can save my
ass!

All Rights Reserved

Continuous Integration for Puppet
• i.e. regularly run the puppet scripts against a test
environment and verify that the result is a valid
deployment
• Should be just part of the End-to-end Continuous
Integration testing.
• Use parameterized builds / build promotion so that QA
can re-use to deploy to their test environment.

TL;DR
I still need tests on
metal

All Rights Reserved

What tests should we run against metal?

Nagios health checks Real tests
• Pro • Pro
– This is what we will be – Verifies that the installed
using in production application works
– Checks that the – High Confidence
application is installed • Con
and alive
– Hard to automate
– Should be fast
– Long duration
• Con
– Does not verify that the TL;DR
installed application Nagios vs Real: use
works both

All Rights Reserved

The test mix lifecycle
80 Devs manually check
they implemented the QA gets
70 feature “blessed”
builds
60

50 Manual
Devs are
lazy and Automated
40
don’t run the Functional
full30
suite Unit
20 Nagios

10

0 Ops have
final
Dev CI Test Stage Prod sanity
check

All Rights Reserved

The cost of running tests
Lost Users
14 Staging env cost $$$
costs $$$
12

10
Keep Devs Manual =
8 in “The $$$
Flow”
6

4

2 Users pay
us to test
0 Robot
Dev CI Test Stage Prod

I can keep costs down and productivity up if
I have lots of tests for Jenkins to run for me
All Rights Reserved
16

Continuous Deployment is just one step more
• Let Jenkins manage the push to production servers
too.
TL;DR
Automation = Jenkins
• Risks?
– Accidental deployment
• Solve with build promotion plugin.
– Breaking production
• Revert the change and redeploy.
• Jenkins is actually a solution to this problem, not a cause
– Security
• Jenkins has strong security model.
• Use a second Jenkins instance (if truly paranoid)

All Rights Reserved

Setting up a Jenkins Server

Tips that will make your life easier down the road

All Rights Reserved

Invest in good URL

• If your users can’t see Jenkins, much of the benefit is
lost

– Make the URL easier to remember

http://sca14-3530.sca.cloudbees.com:8080/jenkins/

http://jenkins.cloudbees.com/

19

Share port 80 with other apps

• Apache reverse proxy
– Let you run Jenkins in non-root, too

browser Apache Jenkins

• For Windows
– IIS7+URL Rewrite+Application Request Routing

20

No “/jenkins”
• Use virtual host to distinguish multiple apps, not
context path

Jenkins

browser Apache / IIS

http://jenkins.cloudbees.com/ Redmine
http://redmine.cloudbees.com/

21

Jenkins Home Directory
• Prepare for disk usage growth
– Especially when you start to host jobs from people who aren’t
close to you
– Make sure you can throw more disks at the problem later

• No need to waste money on 15000rpm SCSI disks
– But bigger disk is nice

22

Plugins for Puppet

There’s 530+ plugins for Jenkins which ones do I actually
need?

All Rights Reserved

Jenkins Plugins of relevance to Puppet
• Here is a list of some of the plugins you may/will want:
– RVM Plugin
• Runs your entire build (from SCM check out to post-build actions)
within the context of an RVM managed environment of your
choice

All Rights Reserved

– RVM Plugin
– Warnings Plugin
• For integration with puppet-lint

All Rights Reserved

– RVM Plugin
– Warnings Plugin
– Promoted Builds Plugin
• Allows you to set up promotion pipelines that are necessary for
getting the control you need with continuous deployment

All Rights Reserved

– RVM Plugin
– Warnings Plugin
– CloudSmith’s StackHammer Plugin
• Makes testing and deploying “stacks” of puppet modules much
easier and more tractable
– Removes the left-over failed deployment mess hell

All Rights Reserved

– RVM Plugin
– Warnings Plugin
– CloudSmith’s StackHammer Plugin
– HTML Publisher Plugin
• Handy for publishing your puppet RDoc

TL;DR
Lots of plugins for
puppet

All Rights Reserved

Show me how

Ok, so quit showing me fancy slides, how the feck do I do all this

All Rights Reserved

Setup Jenkins
puppet module install rtyler-jenkins
puppet apply -v -e "include jenkins"

Quit complaining… this is a
Puppet conference!

Oh yeah, rtyler runs on Ubuntu
10.04
does not always work quite so
well on other OS…
may need tweaks to work on
Ubuntu 12.04

All Rights Reserved

Setup Jenkins (if you don’t want to use Puppet)
• Download http://mirrors.jenkins-
ci.org/war/latest/jenkins.war

• Either
– Deploy to your favorite Java Servlet Container
(Tomcat/Jetty/etc)
– Use built in
java -jar jenkins.war

• Goto http://localhost:8080/ or wherever your servlet
container deployed it

All Rights Reserved

Issues with testing Puppet on metal
• Puppet needs to run as root.
• There can be only one version
– Of Puppet on a machine
– Of RVM that Puppet will use on a machine
• Running tests from a clean baseline
– Side-effects from previous test runs can affect test results
– Solve with Virtualization
– Martyrs can use LXC & Snapshotted VolGroups

All Rights Reserved

Setup Ruby Version Manager (RVM)
• Requires RVM Jenkins plugin
• Enable “Run the build in a RVM-managed
environment”
• Best practice is to give each puppet module their own
Gemset
– A good idea is to use the build job’s name

All Rights Reserved

Ensure RVM environment has required Gems
One solution
• Stick a Gemfile file in the
root of the SCM checkout
source "http://rubygems.org"
gem "puppet", "=2.7.17”
gem "ci_reporter", "=1.7.0”
gem "rspec-puppet", "=0.1.3”
gem "puppet-lint", "=0.1.13"

• Add a shell buildstep
• Run “bundle install”
These fixed versions
work for me with Ruby
More than one way to skin 1.8.7
this cat, use whatever works
for you
All Rights Reserved

Syntax check
• Add a shell build step with the following:
for file in $(find . -iname '*.pp')
do
--render-as s
--modulepath=modules
"$file" || exit 1;
done

• Or do it your own
way! (Think of the
cats though)

All Rights Reserved

Static Analysis
find .
-iname *.pp
-exec
puppet-lint
--log-format
"%{path}:%{linenumber}:%{check}:%{KIND}:%{message}"
{}
;

• Add a “Scan for compiler
warnings” Post-Build Action
– Add Puppet-lint parser to
“Scan console log”

All Rights Reserved

RSpec-puppet
• Configure Rakefile for ci_reporter
require 'ci/reporter/rake/rspec'

• Add a shell Build Step with the following:
export CI_REPORTS=results
rake ci:setup:rspec spec

• Add a “Publish JUnit test result report” Post-Build
Action
**/results/SPEC-*.xml

All Rights Reserved

Documentation
## Cleanup old docs.
[ -d doc/ ] && rm -rf doc/
## Dummy manifests folder.
! [ -d manifests/ ] && mkdir manifests/
## Generate docs
puppet doc --mode rdoc --manifestdir manifests/ --modulepath ./modules/ --outputdir doc

## Fix docs to remove the complete workspace from all file paths.
if [ -d ${WORKSPACE}/doc/files/${WORKSPACE}/modules ]; then
mv -v "${WORKSPACE}/doc/files/${WORKSPACE}/modules" "${WORKSPACE}/doc/files/modules"
fi;
grep -l -R ${WORKSPACE} * | while read fname; do perl -pi -e "s@${WORKSPACE}/@/@g" $fname; done;

• Add a “Publish HTML Reports” Post-Build Action
– doc, index.html, Puppet Docs

All Rights Reserved

Promotion process ideas
• Use “Promote immediately once the
build is complete” to deploy into the
developer’s sandbox environment
• Use “Only when manually approved”
to allow QA to select which builds to
test
– Or Dev sign-off on being feature
complete
• Use “Only when manually approved”
(Manual tests) with “When the
following downstream projects build
successfully” (Automated tests) to
promote to ops
All Rights Reserved

Case study: Lookout

• Jenkins is internal, off the production network 100%
– Use Capistrano for deploying to production

• “Shipping code”
– git push production-puppet-master:/srv/git/puppet.git
– ssh production-puppet-master "(cd /etc/puppet/blah &&
git fetch origin && git checkout ${TAGNAME})"
– ssh production-puppet-master "run-puppet.sh"

All Rights Reserved

Useful links
• All the code snippets from this deck

http://goo.gl/kNgyu
• Others links
– http://cloudbees.com/ – http://jenkins-ci.org/
– http://rvm.io/ – http://puppet-lint.com/
– http://rspec-puppet.com/ – http://gembundler.com/
– https://wiki.jenkins-ci.org/display/JENKINS/RVM+Plugin
– https://wiki.jenkins-ci.org/display/JENKINS/Warnings+Plugin
– https://wiki.jenkins-ci.org/display/JENKINS/HTML+Publisher+Plugin
– https://wiki.jenkins-ci.org/display/JENKINS/Promoted+Builds+Plugin

All Rights Reserved

Continuous Development with Jenkins - Stephen Connolly at PuppetCamp Dublin '12

More Related Content

What's hot

Viewers also liked

Similar to Continuous Development with Jenkins - Stephen Connolly at PuppetCamp Dublin '12

More from Puppet

Recently uploaded

Continuous Development with Jenkins - Stephen Connolly at PuppetCamp Dublin '12

Editor's Notes