Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

An Open-Source Chef Cookbook CI/CD Implementation Using Jenkins Pipelines

2,763 views

Published on

Talk at the Config Management Camp 2017
Gent, 06.02.2017

Published in: Technology

An Open-Source Chef Cookbook CI/CD Implementation Using Jenkins Pipelines

  1. 1. An Open-Source Chef Cookbook CI/CD Implementation Using Jenkins Pipelines Steffen Gebert (@StGebert) Config Management Camp, Gent, 06.02.2017
  2. 2. 2 Agenda • Context • Motivation • Jenkins Pipelines (in general) • Chef CI/CD using Jenkins Pipelines
  3. 3. About Me Researcher / PhD Student (Software-based Networks) 2011 - 2016 Core Team Member 2010 - 2013 Server Admin Team Member since 2011 3 Co-Organizer DevOps Meetup Würzburg since 2016
  4. 4. 4 PHP-based Content Management System Picture by Benjamin Kott
  5. 5. 5 TYPO3 Open Source Project • Established open-source community since 1997 • TYPO3 Association as non-profit organization • Server Admin Team responsible for project infrastructure * Thanks for partially funding my work and covering travel costs! * listed on opensourceinfra.org Picture by Daniel Pötzinger: http://www.typo3-media.com/blog/article/typo3-wallpaper-for-wide-screen.html Based on image by Mike Swanson: http://interfacelift.com/wallpaper/details.php?id=1402 PicturebyStefanBusemann
  6. 6. 6 Chef at TYPO3 • Started with Chef in 2011 • Started with librarian-chef (monolithic repo) • Took years to get rid of monolithic repo • Currently operating 45 Debian nodes • Usual tooling: berkshelf, test-kitchen • berkshelf-api-based /universe (connecting to Chef Server 11*) • Small team of volunteers, varying degree of Chef knowledge * because of laziness
  7. 7. 7 Demo Time! Picture by annca/ pixabay: https://pixabay.com/en/popcorn-cinema-ticket-film-1433327/
  8. 8. Working with Chef (and knife, and berks, and..)
  9. 9. 9 We have workflow problems! Picture by tpsdave / pixabay: https://pixabay.com/en/firefighters-fire-flames-outside-115800/
  10. 10. 10
  11. 11. 11
  12. 12. 12 https://blog.twitter.com/2017/the-infrastructure-behind-twitter-scale
  13. 13. 13 Related Work • zts/cooking-with-jenkins • Provides resource to create single Jenkins freestyle jobs per cookbook • Executes pre-defined rake tasks • chef-solutions/pipeline • Reads cookbooks from a Berksfile enclosed in main chef-repo • Creates single Jenkins freestyle job per cookbook • Chef Automate • Commercial • Little known to me, what's going on under the hood
  14. 14. 14 Survey • Who’s Chef cookbooks (Puppet modules/etc) are tested in CI? • Who has an automated release process?
  15. 15. 15 Jenkins Pipelines
  16. 16. 16 Jenkins Pipelines
  17. 17. 17
  18. 18. 18
  19. 19. Jenkins Pipelines Introduction to
  20. 20. 20 Jenkins Pipeline DSL • Groovy DSL • Provides steps, like sh • Defined in the Jenkinsfile • Can use Groovy/Java logic (Chef users like that idea, right?) • Pipeline plugin (workflow-aggregator) • Formerly called "Workflow" • Open-sourced one year ago • Whole bunch of plugins (10+) sh "make" sh "make install"
  21. 21. 21 Jenkins Jobs as Code? • Jenkins Job Builder • Python / YAML based • From OpenStack • Job DSL plugin (job-dsl) • Also Groovy DSL • Supports many, many plugins • Creates only single jobs, not pipelines # Job DSL example job('my-project-main') { scm { git('https://github.com/...') } triggers { scm('H/15 * * * *') } publishers { downstream('my-project-unit') } }
  22. 22. 22 Cookbook Pipeline? sh 'berks install' sh 'foodcritic' sh 'kitchen test' sh 'berks upload' Picture by BarnImages / pixabay: https://pixabay.com/en/sushi-food-japanese-fish-seafood-789820/
  23. 23. 23 Stages • Allow visual grouping stage('lint') { sh 'foodcritic' sh 'cookstyle' } stage('resolve dependencies') { sh 'berks install' } stage('test-kitchen') { sh 'kitchen test' } stage('upload') { sh 'berks upload' }
  24. 24. 24 Nodes • Allocates a Jenkins executor (master/slave) • Optionally with label node { stage('lint') { sh '..' } stage('resolve') { sh '..' } } node('chefdk') { stage('lint') {..} stage('resolve') {..} } Master Agent 2 Agent 1 Executors (node): Pipeline Stage Step Step Stage Step Step Stage Step Step
  25. 25. 25 Ready, Set, Go!
  26. 26. 26 Multibranch Jobs • Scans repo for branches containing Jenkinsfile • Automatically creates (deletes) jobs • Works with plain Git • Works better™ with GitHub / Bitbucket (via API calls)
  27. 27. 27 More Steps to Come.. • Jenkins Plugins can contribute DSL steps
  28. 28. 28 Global Variables • Environment variables: • Information about current build: • Parameterized build*: env.BRANCH_NAME env.BUILD_URL params.myparam params.deployEnv * well-hidden feature see https://st-g.de/2016/12/parametrized-jenkins-pipelines currentBuild.displayName = "v1.2.3" CurrentBuild.rawBuild.getCause() currentBuild.result = 'FAILED'
  29. 29. 29 Now Copy & Paste? Picture by aitroff / pixabay: https://pixabay.com/en/stormtrooper-star-wars-lego-storm-1343772/
  30. 30. 30 Pipeline Shared Libraries • Additional code to be used by Jenkinsfiles • Loaded from SCM repo • Inclusion via • @Library annotation in Jenkinsfile • Configuration of enclosing folder • Jenkins global configuration* * then library code is "trusted"
  31. 31. 31 Configuring Global Pipeline Library • Jenkinsfile: • Load implicitly would load it without @Library • Use non-default branch/tag @Library('chefci') org.example.Pipeline ... @Library('chefci@testing') Don't play well together
  32. 32. 32 Pipeline Shared Libraries (root) +- src # Groovy source files | +- org | +- foo | +- Bar.groovy # for org.foo.Bar class | +- vars | +- foo.groovy # for global 'foo' variable | +- foo.txt # help for 'foo' variable | +- resources # resource files | +- org | +- foo | +- bar.json # static helper data for org.foo.Bar https://jenkins.io/doc/book/pipeline/shared-libraries/ static files feel like functions the magic – actually code
  33. 33. 33 Global Variables • Can store global state (for the current build) • Can behave like steps • Useful to simplify pipeline code • Hide implementation details from users # vars/deployTo.groovy def call(def environment) { echo "Starting deployment to ${environment}" withCredentials(..) { sh "rsync –avzh . ${environment}.example.com" } } # Jenkinsfile node { sh "make test" deployTo "production" }
  34. 34. 34 More Magic: Global Library Classes (src/) • Groovy classes implementing arbitrary logic • Make use of pipeline DSL steps • Use other Jenkins functions • Import and use Java libraries à How we implement our pipeline
  35. 35. 35 # Declarative Jenkinsfile pipeline { agent label:'has-docker', dockerfile: tru environment { GIT_COMMITTER_NAME = "jenkins" } stages { stage("Build") { steps { sh 'mvn clean install' } } stage("Archive"){ // .. } } post { always { deleteDir() } success { mail to:"me@example.com", subject:"SUC Scripted vs. Declarative Pipelines • Scripted pipelines • Just (imperative) Groovy code • The original implementation • The approach used here • Declarative pipelines • Hit the 1.0 release last Friday • Can be validated prior to execution • Ease some tasks, i.e., failure handling • Visual Pipeline Editor plugin
  36. 36. 36 Jenkins Pipeline Summary • Groovy DSL for specifying pipelines as code • Ideally stored within the tested repo • Extract shared functionality to library repos • Pipeline execution triggered on push event
  37. 37. jenkins-chefci An Open-Source Chef Cookbook CI/CD Implementation Using Jenkins Pipelines
  38. 38. 38 Forbidden Commands $ berks upload $ knife cookbook upload $ knife data bag from file $ knife data bag delete $ knife environment from file $ knife environment delete $ knife role from file $ knife role delete Allowed Commands $ git pull $ git commit $ git push Pictures by PublicDomainPictures & serrano / pixabay: https://pixabay.com/en/costume-demon-devil-board-female-15747/ https://pixabay.com/en/bebe-girl-child-child-portrait-1237704/
  39. 39. 39 Meet the jenkins-chefci cookbook • Sets up a Chef CI/CD infrastructure using Jenkins • Sets up Jenkins master • Installs ChefDK, configures credentials • Configures job that scans a GitHub organization • Configures our shared pipeline library https://github.com/TYPO3-infrastructure/jenkins-pipeline-global-library-chefci/ • Add this Jenkinsfile to your cookbooks: # Jenkinsfile @Library('chefci') _ org.typo3.chefci.v2.cookbook.CookbookPipeline.builder(this, steps) .buildDefaultPipeline().execute()
  40. 40. 40 Meet the Shared Pipeline Library • Implements pipelines for • Cookbook testing, versioning & upload • [WIP] Chef-repo's data bags, environments, roles • Contains couple of Groovy classes • Some helper classes • *Pipeline as entry points from Jenkinsfile • AbstractStage as base class for.. Stages • Yes, we're at v2 already J
  41. 41. 41 Executed Stages & Steps • Linting • Foodcritic • Cookstyle • Build • Berkshelf install (endpoints for Supermarket & Chef Server) Berksfile.lock in Git for top-level cookbooks • Acceptance • Test-Kitchen (using kitchen-docker) • Publish • Version Bump (thor-scmversion, based on user feedback) • Berkshelf upload (to Chef Server) Picture by Kaz/ pixabay: https://pixabay.com/en/hands-hand-raised-hands-raised-220163/
  42. 42. 42
  43. 43. 43 node { createKitchenYaml() stash "cookbook" } parallel( "essentials-debian-86": { node { unstash "cookbook" sh "kitchen test --destroy always essentials-debian-86" }}, "essentials-ubuntu-1604": { node { .. } }, "full-debian-86": { node { .. } }, "full-ubuntu-1604": { node { .. } } ) Test-Kitchen • If there is no .kitchen.docker.yml, put default one • Use Jenkins' parallel step expected result, not hard-coded
  44. 44. 44 Test-Kitchen (2) • Parallel instances automatically derived from kitchen status • Log files of failed instances are archived • Less chaos than complete Jenkins log • Traditional UI does not clearly show failed parallel branch • Not limited to kitchen-docker
  45. 45. 45* cookbook configures this via API
  46. 46. 46
  47. 47. 47
  48. 48. 48
  49. 49. 49 The Art of Cookbook Versioning • Let's agree on SemVer • I do not agree with manual changes to metadata.rb
  50. 50. 50 Demo Time! Picture by annca/ pixabay: https://pixabay.com/en/popcorn-cinema-ticket-film-1433327/
  51. 51. 51 Versioning • Based on thor-scmversion • Made by RiotGames for their cookbooks • Uses Git tags to derive current version • Version treated as "label", not part of the source code • Jenkins pushes only Git tags, no code changes # metadata.rb name "example-cookbook" version IO.read(File.join(File.dirname(__FILE__), 'VERSION')) rescue '0.0.1'
  52. 52. 52 Versioning (2) • Publish stage notifies via Slack • Except commit message subject indicates version increase • Do this #patch • Add this #minor • Break this #major
  53. 53. 53 "Main Chef-Repo" • Not covered here, but in site-chefcitypo3org cookbook • Freestyle job, defined via JobDSL • Uses script to parse git diff and upload changes using knife
  54. 54. 54 Additional Notes • Manually configured J organization-level webhook triggers immediate builds • Everything else is done automatically, no "manual handholding"
  55. 55. Using jenkins-chefci
  56. 56. 56 Ease of Use • Checkout cookbook from github.com/TYPO3-cookbooks/jenkins-chefci • Increase GitHub API rate limit • Allow to steal your Chef credentials • Run test-kitchen export JENKINS_GITHUB_LOGIN=johndoe export JENKINS_GITHUB_TOKEN=123456supersecure export JENKINS_COPY_CHEF_CREDENTIALS=1 kitchen converge full-ubuntu-1604 WARN: allows commit status update WARN: allows knife/berks to access Chef Server
  57. 57. 57 (Potential) TODOs for you • Write your wrapper cookbook around jenkins-chefci • Point it to your organization • Add authentication, e.g., GitHub OAuth? • Fork the global library • Adjust pipeline to your needs • You don't have to agree with our (current) implementation
  58. 58. 58 Outlook / TODOs • Documentation / blog posts • Move more stuff from site-chefcitypo3org to jenkins-chefci • chefdk() global function to run in chef/chefdk Docker container • Store Chef private key as Jenkins credential • Test using multiple chef-client version • Use JobDSL for organization-folder setup (#966) • Trigger downstream cookbooks • Read out dependencies and subscribe to successful pipeline runs • Paves the road to Policyfiles • Use Jenkins slaves • Collect chef-client deprecation warnings
  59. 59. 59 Conclusion • Jenkins Pipelines allow to define pipelines as code • Groovy-based DSL allows programming of pipelines • Can definitively become complex, i.e., debugging • Jenkins Pipelines for Chef cookbook CI/CD • Running at TYPO3 since May 2016 (site-chefcitypo3org cookbook) • Public instance at https://chef-ci.typo3.org, open source from day 1 • Warning: Many cookbooks still use v1 pipeline (git-flow) • jenkins-chefci cookbook as reusable implementation • Makes setup accessible to broader audience • No dependencies to other TYPO3 cookbooks
  60. 60. 60 Further Reading • TYPO3's Chef CI: https://chef-ci.typo3.org • TYPO3-cookbooks on GitHub: https://github.com/TYPO3-cookbooks • TYPO3's Shared Global Library: https://github.com/TYPO3-infrastructure/jenkins-pipeline-global-library-chefci/ • Pipeline Tutorial: https://github.com/jenkinsci/pipeline-plugin/blob/master/TUTORIAL.md • Getting started with pipelines: https://jenkins.io/pipeline/getting-started-pipelines/ • Step documentation: https://jenkins.io/doc/pipeline/steps/ • Pipeline shared libraries: https://jenkins.io/doc/book/pipeline/shared-libraries/ • Notifications (Mail, Slack, etc.) in Scripted Pipelines: https://jenkins.io/blog/2016/07/18/pipline-notifications/ • Declarative Pipelines https://jenkins.io/blog/2016/12/19/declarative-pipeline-beta/ https://jenkins.io/blog/2017/02/03/declarative-pipeline-ga/

×