Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Continuous Development with Jenkins - Stephen Connolly at PuppetCamp Dublin '12


Published on

Stephen Connolly, Elite Developer and Architect at CloudBees, speaks at PuppetCamp Dublin '12 on
Continuous Deployment with Jenkins.


  • Be the first to comment

Continuous Development with Jenkins - Stephen Connolly at PuppetCamp Dublin '12

  1. 1. Continuous Deployment with JenkinsStephen ConnollyArchitect, CloudBeesTL;DR ©2012 CloudBees, Inc. All Rights Reserved
  2. 2. Who am I?• One of the first non-Sun contributors to Jenkins. – Creator of the Weather Icons!• Architect @ CloudBees• Member & Maven PMC @ Apache Software Foundation• In my previous lives: – Software Engineer @ Avaya – IT Project Manager @ Elan Pharmaceuticals – PhD development chemist @ Bristol Myer Squibb – Freelance software developer since 1989 ©2012 CloudBees, Inc. 2 All Rights Reserved
  3. 3. JenkinsWhat the feck is this Jenkins thing anyway? ©2012 CloudBees, Inc. All Rights Reserved
  4. 4. Introducing Jenkins • What is Jenkins? – A relatively recent product, but growing in popularity – Currently used by*: • Yahoo • NASA • JBoss / RedHat • Amazon • HP • And more!*Source: Kohsuke Kawaguchi, lead developer of the Jenkins project 4
  5. 5. Introducing Jenkins• What makes Jenkins so great? Distributed Builds Reporting Plugins Ease of use 5
  6. 6.• OSS CI server – Written in Java• About 7 years old• Easy to install/use• Extensible via 530+ plugins• Widely adopted – 40K+ installations ©2012 CloudBees, Inc. 6 All Rights Reserved
  7. 7. The Deed Poll sed -e ‘s/hudson/jenkins/g’• You might be more 14000 Total created tickets familiar with the IP 12000 Jenkins encumbered old name Hudson 10000 of the project. 8000• The project was renamed to Jenkins in 6000 January 2011. 4000• Nearly 100 releases of 2000 Jenkins since the 0 Total resolved rebrand tickets• About 8 releases of Hudson since the fork. ©2012 CloudBees, Inc. 7 All Rights Reserved
  8. 8. Just for Continuous Integration?Yeah! Look mate, this is a PUPPET conference not a ContinuousIntegration conference… show me that you’re relevant before I startlobbing soft fruits! ©2012 CloudBees, Inc. All Rights Reserved
  9. 9. Puppet manifests are code too TL;DR puppet parser validate• Syntax Check – Use puppet parser validate, e.g. for file in $(find . -iname *.pp’) do puppet parser validate --render-as s --modulepath=modules "$file" || exit 1; done ©2012 CloudBees, Inc. 9 All Rights Reserved
  10. 10. Puppet manifests are code too• Syntax Check ✔• Static Analysis – Checks for anti-patterns – Checks conformance to the style guide – Run puppet-lint TL;DR puppet-lint ©2012 CloudBees, Inc. 10 All Rights Reserved
  11. 11. Puppet manifests are code too• Syntax Check ✔ TL;DR RSpec-puppet• Static Analysis ✔• Automated Tests – Use RSpec-puppet There are a lot of people confused by the purpose of these tests as they can’t test the result of the manifest on a live system. That is not the point of rspec-puppet. Rspec-puppet tests are there to test the behaviour of Puppet when it compiles your manifests into a catalogue of Puppet resources. Source: ©2012 CloudBees, Inc. 11 All Rights Reserved
  12. 12. Puppet manifests are code too• Syntax Check ✔• Static Analysis ✔• Automated Tests ✔• Setup Jenkins to run these on every change to the Puppet manifests ✍ You are storing your scripts in Version Control?• Now you know your Puppet scripts are doing what they should do… TL;DR Jenkins can save my ass! ©2012 CloudBees, Inc. 12 All Rights Reserved
  13. 13. Continuous Integration for Puppet• i.e. regularly run the puppet scripts against a test environment and verify that the result is a valid deployment• Should be just part of the End-to-end Continuous Integration testing.• Use parameterized builds / build promotion so that QA can re-use to deploy to their test environment. TL;DR I still need tests on metal ©2012 CloudBees, Inc. 13 All Rights Reserved
  14. 14. What tests should we run against metal?Nagios health checks Real tests• Pro • Pro – This is what we will be – Verifies that the installed using in production application works – Checks that the – High Confidence application is installed • Con and alive – Hard to automate – Should be fast – Long duration• Con – Does not verify that the TL;DR installed application Nagios vs Real: use works both ©2012 CloudBees, Inc. 14 All Rights Reserved
  15. 15. The test mix lifecycle 80 Devs manually check they implemented the QA gets 70 feature “blessed” builds 60 50 ManualDevs arelazy and Automated 40don’t run the Functionalfull30 suite Unit 20 Nagios 10 0 Ops have final Dev CI Test Stage Prod sanity check ©2012 CloudBees, Inc. 15 All Rights Reserved
  16. 16. The cost of running tests Lost Users14 Staging env cost $$$ costs $$$1210 Keep Devs Manual = 8 in “The $$$ Flow” 6 4 2 Users pay us to test 0 Robot Dev CI Test Stage Prod I can keep costs down and productivity up if I have lots of tests for Jenkins to run for me ©2012 CloudBees, Inc. All Rights Reserved 16
  17. 17. Continuous Deployment is just one step more• Let Jenkins manage the push to production servers too. TL;DR Automation = Jenkins• Risks? – Accidental deployment • Solve with build promotion plugin. – Breaking production • Revert the change and redeploy. • Jenkins is actually a solution to this problem, not a cause – Security • Jenkins has strong security model. • Use a second Jenkins instance (if truly paranoid) ©2012 CloudBees, Inc. 17 All Rights Reserved
  18. 18. Setting up a Jenkins ServerTips that will make your life easier down the road ©2012 CloudBees, Inc. All Rights Reserved
  19. 19. Invest in good URL• If your users can’t see Jenkins, much of the benefit is lost – Make the URL easier to remember 19
  20. 20. Share port 80 with other apps• Apache reverse proxy – Let you run Jenkins in non-root, too browser Apache Jenkins• For Windows – IIS7+URL Rewrite+Application Request Routing 20
  21. 21. No “/jenkins”• Use virtual host to distinguish multiple apps, not context path Jenkins browser Apache / IIS Redmine 21
  22. 22. Jenkins Home Directory• Prepare for disk usage growth – Especially when you start to host jobs from people who aren’t close to you – Make sure you can throw more disks at the problem later• No need to waste money on 15000rpm SCSI disks – But bigger disk is nice 22
  23. 23. Plugins for PuppetThere’s 530+ plugins for Jenkins which ones do I actuallyneed? ©2012 CloudBees, Inc. All Rights Reserved
  24. 24. Jenkins Plugins of relevance to Puppet• Here is a list of some of the plugins you may/will want: – RVM Plugin • Runs your entire build (from SCM check out to post-build actions) within the context of an RVM managed environment of your choice ©2012 CloudBees, Inc. 24 All Rights Reserved
  25. 25. Jenkins Plugins of relevance to Puppet• Here is a list of some of the plugins you may/will want: – RVM Plugin – Warnings Plugin • For integration with puppet-lint ©2012 CloudBees, Inc. 25 All Rights Reserved
  26. 26. Jenkins Plugins of relevance to Puppet• Here is a list of some of the plugins you may/will want: – RVM Plugin – Warnings Plugin – Promoted Builds Plugin • Allows you to set up promotion pipelines that are necessary for getting the control you need with continuous deployment ©2012 CloudBees, Inc. 26 All Rights Reserved
  27. 27. Jenkins Plugins of relevance to Puppet• Here is a list of some of the plugins you may/will want: – RVM Plugin – Warnings Plugin – Promoted Builds Plugin – CloudSmith’s StackHammer Plugin • Makes testing and deploying “stacks” of puppet modules much easier and more tractable – Removes the left-over failed deployment mess hell ©2012 CloudBees, Inc. 27 All Rights Reserved
  28. 28. Jenkins Plugins of relevance to Puppet• Here is a list of some of the plugins you may/will want: – RVM Plugin – Warnings Plugin – Promoted Builds Plugin – CloudSmith’s StackHammer Plugin – HTML Publisher Plugin • Handy for publishing your puppet RDoc TL;DR Lots of plugins for puppet ©2012 CloudBees, Inc. 28 All Rights Reserved
  29. 29. Show me howOk, so quit showing me fancy slides, how the feck do I do all this ©2012 CloudBees, Inc. All Rights Reserved
  30. 30. Setup Jenkinspuppet module install rtyler-jenkinspuppet apply -v -e "include jenkins" Quit complaining… this is a Puppet conference! Oh yeah, rtyler runs on Ubuntu 10.04 does not always work quite so well on other OS… may need tweaks to work on Ubuntu 12.04 ©2012 CloudBees, Inc. 30 All Rights Reserved
  31. 31. ©2012 CloudBees, Inc. 31 All Rights Reserved
  32. 32. Setup Jenkins (if you don’t want to use Puppet)• Download http://mirrors.jenkins-• Either – Deploy to your favorite Java Servlet Container (Tomcat/Jetty/etc) – Use built in java -jar jenkins.war• Goto http://localhost:8080/ or wherever your servlet container deployed it ©2012 CloudBees, Inc. 32 All Rights Reserved
  33. 33. Or Don’t Setup Jenkins (the shameless plug) Just use CloudBees’ DEV@cloud (Jenkins as a Service) ©2012 CloudBees, Inc. 33 All Rights Reserved
  34. 34. Welcome to Jenkins ©2012 CloudBees, Inc. 34 All Rights Reserved
  35. 35. Testing Puppet with JenkinsIf you are reading these tags do a Mexican Wave on three… 1…2… 3! ©2012 CloudBees, Inc. All Rights Reserved
  36. 36. ©2012 CloudBees, Inc. 36 All Rights Reserved
  37. 37. Issues with testing Puppet on metal• Puppet needs to run as root.• There can be only one version – Of Puppet on a machine – Of RVM that Puppet will use on a machine• Running tests from a clean baseline – Side-effects from previous test runs can affect test results – Solve with Virtualization – Martyrs can use LXC & Snapshotted VolGroups ©2012 CloudBees, Inc. 37 All Rights Reserved
  38. 38. Setup Ruby Version Manager (RVM)• Requires RVM Jenkins plugin• Enable “Run the build in a RVM-managed environment”• Best practice is to give each puppet module their own Gemset – A good idea is to use the build job’s name ©2012 CloudBees, Inc. 38 All Rights Reserved
  39. 39. Ensure RVM environment has required GemsOne solution• Stick a Gemfile file in the root of the SCM checkout source "" gem "puppet", "=2.7.17” gem "ci_reporter", "=1.7.0” gem "rspec-puppet", "=0.1.3” gem "puppet-lint", "=0.1.13"• Add a shell buildstep• Run “bundle install” These fixed versions work for me with RubyMore than one way to skin 1.8.7this cat, use whatever worksfor you ©2012 CloudBees, Inc. 39 All Rights Reserved
  40. 40. Syntax check• Add a shell build step with the following: for file in $(find . -iname *.pp) do puppet parser validate --render-as s --modulepath=modules "$file" || exit 1; done• Or do it your own way! (Think of the cats though) ©2012 CloudBees, Inc. 40 All Rights Reserved
  41. 41. Static Analysis• Add a shell build step with the following: find . -iname *.pp -exec puppet-lint --log-format "%{path}:%{linenumber}:%{check}:%{KIND}:%{message}" {} ;• Add a “Scan for compiler warnings” Post-Build Action – Add Puppet-lint parser to “Scan console log” ©2012 CloudBees, Inc. 41 All Rights Reserved
  42. 42. RSpec-puppet• Configure Rakefile for ci_reporter require ci/reporter/rake/rspec• Add a shell Build Step with the following: export CI_REPORTS=results rake ci:setup:rspec spec• Add a “Publish JUnit test result report” Post-Build Action **/results/SPEC-*.xml ©2012 CloudBees, Inc. 42 All Rights Reserved
  43. 43. Documentation• Add a shell build step with the following: ## Cleanup old docs. [ -d doc/ ] && rm -rf doc/ ## Dummy manifests folder. ! [ -d manifests/ ] && mkdir manifests/ ## Generate docs puppet doc --mode rdoc --manifestdir manifests/ --modulepath ./modules/ --outputdir doc ## Fix docs to remove the complete workspace from all file paths. if [ -d ${WORKSPACE}/doc/files/${WORKSPACE}/modules ]; then mv -v "${WORKSPACE}/doc/files/${WORKSPACE}/modules" "${WORKSPACE}/doc/files/modules" fi; grep -l -R ${WORKSPACE} * | while read fname; do perl -pi -e "s@${WORKSPACE}/@/@g" $fname; done;• Add a “Publish HTML Reports” Post-Build Action – doc, index.html, Puppet Docs ©2012 CloudBees, Inc. 43 All Rights Reserved
  44. 44. Deploying to metal• Set up a build promotion process ©2012 CloudBees, Inc. 44 All Rights Reserved
  45. 45. Promotion process ideas• Use “Promote immediately once the build is complete” to deploy into the developer’s sandbox environment• Use “Only when manually approved” to allow QA to select which builds to test – Or Dev sign-off on being feature complete• Use “Only when manually approved” (Manual tests) with “When the following downstream projects build successfully” (Automated tests) to promote to ops ©2012 CloudBees, Inc. 45 All Rights Reserved
  46. 46. Case study: Lookout• Jenkins is internal, off the production network 100% – Use Capistrano for deploying to production• “Shipping code” – git push production-puppet-master:/srv/git/puppet.git – ssh production-puppet-master "(cd /etc/puppet/blah && git fetch origin && git checkout ${TAGNAME})" – ssh production-puppet-master "" ©2012 CloudBees, Inc. 46 All Rights Reserved
  47. 47. CloudSmith can help with on metal testing ©2012 CloudBees, Inc. 47 All Rights Reserved
  48. 48. Useful links• All the code snippets from this deck• Others links – – – – – – – – – – ©2012 CloudBees, Inc. 48 All Rights Reserved
  49. 49. ©2012 CloudBees, Inc. All Rights Reserved