Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

An Open-Source Chef Cookbook CI/CD Implementation Using Jenkins Pipelines

4,667 views

Published on

Talk at the Config Management Camp 2017
Gent, 06.02.2017

Published in: Technology
  • Don't forget another good way of simplifying your writing is using external resources (such as ⇒ www.HelpWriting.net ⇐ ). This will definitely make your life more easier
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • ⇒ www.WritePaper.info ⇐ is a good website if you’re looking to get your essay written for you. You can also request things like research papers or dissertations. It’s really convenient and helpful.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • I just discovered this and suggest you see this immediately. There are over 16,000 woodworking plans that comes with step-by-step instructions and detailed photos ☺☺☺ https://url.cn/ktFCrsHZ
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THE BOOK INTO AVAILABLE FORMAT (New Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://urlzs.com/UABbn } ......................................................................................................................... Download Full EPUB Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... Download Full doc Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... Download PDF EBOOK here { https://urlzs.com/UABbn } ......................................................................................................................... Download EPUB Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... Download doc Ebook here { https://urlzs.com/UABbn } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book THE can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer THE is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBOOK .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, CookBOOK, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, EBOOK, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story THE Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money THE the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths THE Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Dating direct: ❶❶❶ http://bit.ly/2F90ZZC ❶❶❶
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

An Open-Source Chef Cookbook CI/CD Implementation Using Jenkins Pipelines

  1. 1. An Open-Source Chef Cookbook CI/CD Implementation Using Jenkins Pipelines Steffen Gebert (@StGebert) Config Management Camp, Gent, 06.02.2017
  2. 2. 2 Agenda • Context • Motivation • Jenkins Pipelines (in general) • Chef CI/CD using Jenkins Pipelines
  3. 3. About Me Researcher / PhD Student (Software-based Networks) 2011 - 2016 Core Team Member 2010 - 2013 Server Admin Team Member since 2011 3 Co-Organizer DevOps Meetup Würzburg since 2016
  4. 4. 4 PHP-based Content Management System Picture by Benjamin Kott
  5. 5. 5 TYPO3 Open Source Project • Established open-source community since 1997 • TYPO3 Association as non-profit organization • Server Admin Team responsible for project infrastructure * Thanks for partially funding my work and covering travel costs! * listed on opensourceinfra.org Picture by Daniel Pötzinger: http://www.typo3-media.com/blog/article/typo3-wallpaper-for-wide-screen.html Based on image by Mike Swanson: http://interfacelift.com/wallpaper/details.php?id=1402 PicturebyStefanBusemann
  6. 6. 6 Chef at TYPO3 • Started with Chef in 2011 • Started with librarian-chef (monolithic repo) • Took years to get rid of monolithic repo • Currently operating 45 Debian nodes • Usual tooling: berkshelf, test-kitchen • berkshelf-api-based /universe (connecting to Chef Server 11*) • Small team of volunteers, varying degree of Chef knowledge * because of laziness
  7. 7. 7 Demo Time! Picture by annca/ pixabay: https://pixabay.com/en/popcorn-cinema-ticket-film-1433327/
  8. 8. Working with Chef (and knife, and berks, and..)
  9. 9. 9 We have workflow problems! Picture by tpsdave / pixabay: https://pixabay.com/en/firefighters-fire-flames-outside-115800/
  10. 10. 10
  11. 11. 11
  12. 12. 12 https://blog.twitter.com/2017/the-infrastructure-behind-twitter-scale
  13. 13. 13 Related Work • zts/cooking-with-jenkins • Provides resource to create single Jenkins freestyle jobs per cookbook • Executes pre-defined rake tasks • chef-solutions/pipeline • Reads cookbooks from a Berksfile enclosed in main chef-repo • Creates single Jenkins freestyle job per cookbook • Chef Automate • Commercial • Little known to me, what's going on under the hood
  14. 14. 14 Survey • Who’s Chef cookbooks (Puppet modules/etc) are tested in CI? • Who has an automated release process?
  15. 15. 15 Jenkins Pipelines
  16. 16. 16 Jenkins Pipelines
  17. 17. 17
  18. 18. 18
  19. 19. Jenkins Pipelines Introduction to
  20. 20. 20 Jenkins Pipeline DSL • Groovy DSL • Provides steps, like sh • Defined in the Jenkinsfile • Can use Groovy/Java logic (Chef users like that idea, right?) • Pipeline plugin (workflow-aggregator) • Formerly called "Workflow" • Open-sourced one year ago • Whole bunch of plugins (10+) sh "make" sh "make install"
  21. 21. 21 Jenkins Jobs as Code? • Jenkins Job Builder • Python / YAML based • From OpenStack • Job DSL plugin (job-dsl) • Also Groovy DSL • Supports many, many plugins • Creates only single jobs, not pipelines # Job DSL example job('my-project-main') { scm { git('https://github.com/...') } triggers { scm('H/15 * * * *') } publishers { downstream('my-project-unit') } }
  22. 22. 22 Cookbook Pipeline? sh 'berks install' sh 'foodcritic' sh 'kitchen test' sh 'berks upload' Picture by BarnImages / pixabay: https://pixabay.com/en/sushi-food-japanese-fish-seafood-789820/
  23. 23. 23 Stages • Allow visual grouping stage('lint') { sh 'foodcritic' sh 'cookstyle' } stage('resolve dependencies') { sh 'berks install' } stage('test-kitchen') { sh 'kitchen test' } stage('upload') { sh 'berks upload' }
  24. 24. 24 Nodes • Allocates a Jenkins executor (master/slave) • Optionally with label node { stage('lint') { sh '..' } stage('resolve') { sh '..' } } node('chefdk') { stage('lint') {..} stage('resolve') {..} } Master Agent 2 Agent 1 Executors (node): Pipeline Stage Step Step Stage Step Step Stage Step Step
  25. 25. 25 Ready, Set, Go!
  26. 26. 26 Multibranch Jobs • Scans repo for branches containing Jenkinsfile • Automatically creates (deletes) jobs • Works with plain Git • Works better™ with GitHub / Bitbucket (via API calls)
  27. 27. 27 More Steps to Come.. • Jenkins Plugins can contribute DSL steps
  28. 28. 28 Global Variables • Environment variables: • Information about current build: • Parameterized build*: env.BRANCH_NAME env.BUILD_URL params.myparam params.deployEnv * well-hidden feature see https://st-g.de/2016/12/parametrized-jenkins-pipelines currentBuild.displayName = "v1.2.3" CurrentBuild.rawBuild.getCause() currentBuild.result = 'FAILED'
  29. 29. 29 Now Copy & Paste? Picture by aitroff / pixabay: https://pixabay.com/en/stormtrooper-star-wars-lego-storm-1343772/
  30. 30. 30 Pipeline Shared Libraries • Additional code to be used by Jenkinsfiles • Loaded from SCM repo • Inclusion via • @Library annotation in Jenkinsfile • Configuration of enclosing folder • Jenkins global configuration* * then library code is "trusted"
  31. 31. 31 Configuring Global Pipeline Library • Jenkinsfile: • Load implicitly would load it without @Library • Use non-default branch/tag @Library('chefci') org.example.Pipeline ... @Library('chefci@testing') Don't play well together
  32. 32. 32 Pipeline Shared Libraries (root) +- src # Groovy source files | +- org | +- foo | +- Bar.groovy # for org.foo.Bar class | +- vars | +- foo.groovy # for global 'foo' variable | +- foo.txt # help for 'foo' variable | +- resources # resource files | +- org | +- foo | +- bar.json # static helper data for org.foo.Bar https://jenkins.io/doc/book/pipeline/shared-libraries/ static files feel like functions the magic – actually code
  33. 33. 33 Global Variables • Can store global state (for the current build) • Can behave like steps • Useful to simplify pipeline code • Hide implementation details from users # vars/deployTo.groovy def call(def environment) { echo "Starting deployment to ${environment}" withCredentials(..) { sh "rsync –avzh . ${environment}.example.com" } } # Jenkinsfile node { sh "make test" deployTo "production" }
  34. 34. 34 More Magic: Global Library Classes (src/) • Groovy classes implementing arbitrary logic • Make use of pipeline DSL steps • Use other Jenkins functions • Import and use Java libraries à How we implement our pipeline
  35. 35. 35 # Declarative Jenkinsfile pipeline { agent label:'has-docker', dockerfile: tru environment { GIT_COMMITTER_NAME = "jenkins" } stages { stage("Build") { steps { sh 'mvn clean install' } } stage("Archive"){ // .. } } post { always { deleteDir() } success { mail to:"me@example.com", subject:"SUC Scripted vs. Declarative Pipelines • Scripted pipelines • Just (imperative) Groovy code • The original implementation • The approach used here • Declarative pipelines • Hit the 1.0 release last Friday • Can be validated prior to execution • Ease some tasks, i.e., failure handling • Visual Pipeline Editor plugin
  36. 36. 36 Jenkins Pipeline Summary • Groovy DSL for specifying pipelines as code • Ideally stored within the tested repo • Extract shared functionality to library repos • Pipeline execution triggered on push event
  37. 37. jenkins-chefci An Open-Source Chef Cookbook CI/CD Implementation Using Jenkins Pipelines
  38. 38. 38 Forbidden Commands $ berks upload $ knife cookbook upload $ knife data bag from file $ knife data bag delete $ knife environment from file $ knife environment delete $ knife role from file $ knife role delete Allowed Commands $ git pull $ git commit $ git push Pictures by PublicDomainPictures & serrano / pixabay: https://pixabay.com/en/costume-demon-devil-board-female-15747/ https://pixabay.com/en/bebe-girl-child-child-portrait-1237704/
  39. 39. 39 Meet the jenkins-chefci cookbook • Sets up a Chef CI/CD infrastructure using Jenkins • Sets up Jenkins master • Installs ChefDK, configures credentials • Configures job that scans a GitHub organization • Configures our shared pipeline library https://github.com/TYPO3-infrastructure/jenkins-pipeline-global-library-chefci/ • Add this Jenkinsfile to your cookbooks: # Jenkinsfile @Library('chefci') _ org.typo3.chefci.v2.cookbook.CookbookPipeline.builder(this, steps) .buildDefaultPipeline().execute()
  40. 40. 40 Meet the Shared Pipeline Library • Implements pipelines for • Cookbook testing, versioning & upload • [WIP] Chef-repo's data bags, environments, roles • Contains couple of Groovy classes • Some helper classes • *Pipeline as entry points from Jenkinsfile • AbstractStage as base class for.. Stages • Yes, we're at v2 already J
  41. 41. 41 Executed Stages & Steps • Linting • Foodcritic • Cookstyle • Build • Berkshelf install (endpoints for Supermarket & Chef Server) Berksfile.lock in Git for top-level cookbooks • Acceptance • Test-Kitchen (using kitchen-docker) • Publish • Version Bump (thor-scmversion, based on user feedback) • Berkshelf upload (to Chef Server) Picture by Kaz/ pixabay: https://pixabay.com/en/hands-hand-raised-hands-raised-220163/
  42. 42. 42
  43. 43. 43 node { createKitchenYaml() stash "cookbook" } parallel( "essentials-debian-86": { node { unstash "cookbook" sh "kitchen test --destroy always essentials-debian-86" }}, "essentials-ubuntu-1604": { node { .. } }, "full-debian-86": { node { .. } }, "full-ubuntu-1604": { node { .. } } ) Test-Kitchen • If there is no .kitchen.docker.yml, put default one • Use Jenkins' parallel step expected result, not hard-coded
  44. 44. 44 Test-Kitchen (2) • Parallel instances automatically derived from kitchen status • Log files of failed instances are archived • Less chaos than complete Jenkins log • Traditional UI does not clearly show failed parallel branch • Not limited to kitchen-docker
  45. 45. 45* cookbook configures this via API
  46. 46. 46
  47. 47. 47
  48. 48. 48
  49. 49. 49 The Art of Cookbook Versioning • Let's agree on SemVer • I do not agree with manual changes to metadata.rb
  50. 50. 50 Demo Time! Picture by annca/ pixabay: https://pixabay.com/en/popcorn-cinema-ticket-film-1433327/
  51. 51. 51 Versioning • Based on thor-scmversion • Made by RiotGames for their cookbooks • Uses Git tags to derive current version • Version treated as "label", not part of the source code • Jenkins pushes only Git tags, no code changes # metadata.rb name "example-cookbook" version IO.read(File.join(File.dirname(__FILE__), 'VERSION')) rescue '0.0.1'
  52. 52. 52 Versioning (2) • Publish stage notifies via Slack • Except commit message subject indicates version increase • Do this #patch • Add this #minor • Break this #major
  53. 53. 53 "Main Chef-Repo" • Not covered here, but in site-chefcitypo3org cookbook • Freestyle job, defined via JobDSL • Uses script to parse git diff and upload changes using knife
  54. 54. 54 Additional Notes • Manually configured J organization-level webhook triggers immediate builds • Everything else is done automatically, no "manual handholding"
  55. 55. Using jenkins-chefci
  56. 56. 56 Ease of Use • Checkout cookbook from github.com/TYPO3-cookbooks/jenkins-chefci • Increase GitHub API rate limit • Allow to steal your Chef credentials • Run test-kitchen export JENKINS_GITHUB_LOGIN=johndoe export JENKINS_GITHUB_TOKEN=123456supersecure export JENKINS_COPY_CHEF_CREDENTIALS=1 kitchen converge full-ubuntu-1604 WARN: allows commit status update WARN: allows knife/berks to access Chef Server
  57. 57. 57 (Potential) TODOs for you • Write your wrapper cookbook around jenkins-chefci • Point it to your organization • Add authentication, e.g., GitHub OAuth? • Fork the global library • Adjust pipeline to your needs • You don't have to agree with our (current) implementation
  58. 58. 58 Outlook / TODOs • Documentation / blog posts • Move more stuff from site-chefcitypo3org to jenkins-chefci • chefdk() global function to run in chef/chefdk Docker container • Store Chef private key as Jenkins credential • Test using multiple chef-client version • Use JobDSL for organization-folder setup (#966) • Trigger downstream cookbooks • Read out dependencies and subscribe to successful pipeline runs • Paves the road to Policyfiles • Use Jenkins slaves • Collect chef-client deprecation warnings
  59. 59. 59 Conclusion • Jenkins Pipelines allow to define pipelines as code • Groovy-based DSL allows programming of pipelines • Can definitively become complex, i.e., debugging • Jenkins Pipelines for Chef cookbook CI/CD • Running at TYPO3 since May 2016 (site-chefcitypo3org cookbook) • Public instance at https://chef-ci.typo3.org, open source from day 1 • Warning: Many cookbooks still use v1 pipeline (git-flow) • jenkins-chefci cookbook as reusable implementation • Makes setup accessible to broader audience • No dependencies to other TYPO3 cookbooks
  60. 60. 60 Further Reading • TYPO3's Chef CI: https://chef-ci.typo3.org • TYPO3-cookbooks on GitHub: https://github.com/TYPO3-cookbooks • TYPO3's Shared Global Library: https://github.com/TYPO3-infrastructure/jenkins-pipeline-global-library-chefci/ • Pipeline Tutorial: https://github.com/jenkinsci/pipeline-plugin/blob/master/TUTORIAL.md • Getting started with pipelines: https://jenkins.io/pipeline/getting-started-pipelines/ • Step documentation: https://jenkins.io/doc/pipeline/steps/ • Pipeline shared libraries: https://jenkins.io/doc/book/pipeline/shared-libraries/ • Notifications (Mail, Slack, etc.) in Scripted Pipelines: https://jenkins.io/blog/2016/07/18/pipline-notifications/ • Declarative Pipelines https://jenkins.io/blog/2016/12/19/declarative-pipeline-beta/ https://jenkins.io/blog/2017/02/03/declarative-pipeline-ga/

×