An Open-Source Chef Cookbook CI/CD Implementation Using Jenkins Pipelines

1. An Open-Source Chef Cookbook CI/CD Implementation Using Jenkins Pipelines Steffen Gebert (@StGebert) Config Management Camp, Gent, 06.02.2017

2. 2 Agenda • Context • Motivation • Jenkins Pipelines (in general) • Chef CI/CD using Jenkins Pipelines

3. About Me Researcher / PhD Student (Software-based Networks) 2011 - 2016 Core Team Member 2010 - 2013 Server Admin Team Member since 2011 3 Co-Organizer DevOps Meetup Würzburg since 2016

4. 4 PHP-based Content Management System Picture by Benjamin Kott

5. 5 TYPO3 Open Source Project • Established open-source community since 1997 • TYPO3 Association as non-profit organization • Server Admin Team responsible for project infrastructure * Thanks for partially funding my work and covering travel costs! * listed on opensourceinfra.org Picture by Daniel Pötzinger: http://www.typo3-media.com/blog/article/typo3-wallpaper-for-wide-screen.html Based on image by Mike Swanson: http://interfacelift.com/wallpaper/details.php?id=1402 PicturebyStefanBusemann

6. 6 Chef at TYPO3 • Started with Chef in 2011 • Started with librarian-chef (monolithic repo) • Took years to get rid of monolithic repo • Currently operating 45 Debian nodes • Usual tooling: berkshelf, test-kitchen • berkshelf-api-based /universe (connecting to Chef Server 11*) • Small team of volunteers, varying degree of Chef knowledge * because of laziness

7. 7 Demo Time! Picture by annca/ pixabay: https://pixabay.com/en/popcorn-cinema-ticket-film-1433327/

8. Working with Chef (and knife, and berks, and..)

9. 9 We have workflow problems! Picture by tpsdave / pixabay: https://pixabay.com/en/firefighters-fire-flames-outside-115800/

10. 10

11. 11

12. 12 https://blog.twitter.com/2017/the-infrastructure-behind-twitter-scale

13. 13 Related Work • zts/cooking-with-jenkins • Provides resource to create single Jenkins freestyle jobs per cookbook • Executes pre-defined rake tasks • chef-solutions/pipeline • Reads cookbooks from a Berksfile enclosed in main chef-repo • Creates single Jenkins freestyle job per cookbook • Chef Automate • Commercial • Little known to me, what's going on under the hood

14. 14 Survey • Who’s Chef cookbooks (Puppet modules/etc) are tested in CI? • Who has an automated release process?

15. 15 Jenkins Pipelines

16. 16 Jenkins Pipelines

17. 17

18. 18

19. Jenkins Pipelines Introduction to

20. 20 Jenkins Pipeline DSL • Groovy DSL • Provides steps, like sh • Defined in the Jenkinsfile • Can use Groovy/Java logic (Chef users like that idea, right?) • Pipeline plugin (workflow-aggregator) • Formerly called "Workflow" • Open-sourced one year ago • Whole bunch of plugins (10+) sh "make" sh "make install"

21. 21 Jenkins Jobs as Code? • Jenkins Job Builder • Python / YAML based • From OpenStack • Job DSL plugin (job-dsl) • Also Groovy DSL • Supports many, many plugins • Creates only single jobs, not pipelines # Job DSL example job('my-project-main') { scm { git('https://github.com/...') } triggers { scm('H/15 * * * *') } publishers { downstream('my-project-unit') } }

22. 22 Cookbook Pipeline? sh 'berks install' sh 'foodcritic' sh 'kitchen test' sh 'berks upload' Picture by BarnImages / pixabay: https://pixabay.com/en/sushi-food-japanese-fish-seafood-789820/

23. 23 Stages • Allow visual grouping stage('lint') { sh 'foodcritic' sh 'cookstyle' } stage('resolve dependencies') { sh 'berks install' } stage('test-kitchen') { sh 'kitchen test' } stage('upload') { sh 'berks upload' }

24. 24 Nodes • Allocates a Jenkins executor (master/slave) • Optionally with label node { stage('lint') { sh '..' } stage('resolve') { sh '..' } } node('chefdk') { stage('lint') {..} stage('resolve') {..} } Master Agent 2 Agent 1 Executors (node): Pipeline Stage Step Step Stage Step Step Stage Step Step

25. 25 Ready, Set, Go!

26. 26 Multibranch Jobs • Scans repo for branches containing Jenkinsfile • Automatically creates (deletes) jobs • Works with plain Git • Works better™ with GitHub / Bitbucket (via API calls)

27. 27 More Steps to Come.. • Jenkins Plugins can contribute DSL steps

28. 28 Global Variables • Environment variables: • Information about current build: • Parameterized build*: env.BRANCH_NAME env.BUILD_URL params.myparam params.deployEnv * well-hidden feature see https://st-g.de/2016/12/parametrized-jenkins-pipelines currentBuild.displayName = "v1.2.3" CurrentBuild.rawBuild.getCause() currentBuild.result = 'FAILED'

29. 29 Now Copy & Paste? Picture by aitroff / pixabay: https://pixabay.com/en/stormtrooper-star-wars-lego-storm-1343772/

30. 30 Pipeline Shared Libraries • Additional code to be used by Jenkinsfiles • Loaded from SCM repo • Inclusion via • @Library annotation in Jenkinsfile • Configuration of enclosing folder • Jenkins global configuration* * then library code is "trusted"

31. 31 Configuring Global Pipeline Library • Jenkinsfile: • Load implicitly would load it without @Library • Use non-default branch/tag @Library('chefci') org.example.Pipeline ... @Library('chefci@testing') Don't play well together

33. 33 Global Variables • Can store global state (for the current build) • Can behave like steps • Useful to simplify pipeline code • Hide implementation details from users # vars/deployTo.groovy def call(def environment) { echo "Starting deployment to ${environment}" withCredentials(..) { sh "rsync –avzh . ${environment}.example.com" } } # Jenkinsfile node { sh "make test" deployTo "production" }

34. 34 More Magic: Global Library Classes (src/) • Groovy classes implementing arbitrary logic • Make use of pipeline DSL steps • Use other Jenkins functions • Import and use Java libraries à How we implement our pipeline

35. 35 # Declarative Jenkinsfile pipeline { agent label:'has-docker', dockerfile: tru environment { GIT_COMMITTER_NAME = "jenkins" } stages { stage("Build") { steps { sh 'mvn clean install' } } stage("Archive"){ // .. } } post { always { deleteDir() } success { mail to:"me@example.com", subject:"SUC Scripted vs. Declarative Pipelines • Scripted pipelines • Just (imperative) Groovy code • The original implementation • The approach used here • Declarative pipelines • Hit the 1.0 release last Friday • Can be validated prior to execution • Ease some tasks, i.e., failure handling • Visual Pipeline Editor plugin

36. 36 Jenkins Pipeline Summary • Groovy DSL for specifying pipelines as code • Ideally stored within the tested repo • Extract shared functionality to library repos • Pipeline execution triggered on push event

37. jenkins-chefci An Open-Source Chef Cookbook CI/CD Implementation Using Jenkins Pipelines

38. 38 Forbidden Commands $ berks upload $ knife cookbook upload $ knife data bag from file $ knife data bag delete $ knife environment from file $ knife environment delete $ knife role from file $ knife role delete Allowed Commands $ git pull $ git commit $ git push Pictures by PublicDomainPictures & serrano / pixabay: https://pixabay.com/en/costume-demon-devil-board-female-15747/ https://pixabay.com/en/bebe-girl-child-child-portrait-1237704/

39. 39 Meet the jenkins-chefci cookbook • Sets up a Chef CI/CD infrastructure using Jenkins • Sets up Jenkins master • Installs ChefDK, configures credentials • Configures job that scans a GitHub organization • Configures our shared pipeline library https://github.com/TYPO3-infrastructure/jenkins-pipeline-global-library-chefci/ • Add this Jenkinsfile to your cookbooks: # Jenkinsfile @Library('chefci') _ org.typo3.chefci.v2.cookbook.CookbookPipeline.builder(this, steps) .buildDefaultPipeline().execute()

40. 40 Meet the Shared Pipeline Library • Implements pipelines for • Cookbook testing, versioning & upload • [WIP] Chef-repo's data bags, environments, roles • Contains couple of Groovy classes • Some helper classes • *Pipeline as entry points from Jenkinsfile • AbstractStage as base class for.. Stages • Yes, we're at v2 already J

41. 41 Executed Stages & Steps • Linting • Foodcritic • Cookstyle • Build • Berkshelf install (endpoints for Supermarket & Chef Server) Berksfile.lock in Git for top-level cookbooks • Acceptance • Test-Kitchen (using kitchen-docker) • Publish • Version Bump (thor-scmversion, based on user feedback) • Berkshelf upload (to Chef Server) Picture by Kaz/ pixabay: https://pixabay.com/en/hands-hand-raised-hands-raised-220163/

42. 42

43. 43 node { createKitchenYaml() stash "cookbook" } parallel( "essentials-debian-86": { node { unstash "cookbook" sh "kitchen test --destroy always essentials-debian-86" }}, "essentials-ubuntu-1604": { node { .. } }, "full-debian-86": { node { .. } }, "full-ubuntu-1604": { node { .. } } ) Test-Kitchen • If there is no .kitchen.docker.yml, put default one • Use Jenkins' parallel step expected result, not hard-coded

44. 44 Test-Kitchen (2) • Parallel instances automatically derived from kitchen status • Log files of failed instances are archived • Less chaos than complete Jenkins log • Traditional UI does not clearly show failed parallel branch • Not limited to kitchen-docker

45. 45* cookbook configures this via API

46. 46

47. 47

48. 48

49. 49 The Art of Cookbook Versioning • Let's agree on SemVer • I do not agree with manual changes to metadata.rb

50. 50 Demo Time! Picture by annca/ pixabay: https://pixabay.com/en/popcorn-cinema-ticket-film-1433327/

51. 51 Versioning • Based on thor-scmversion • Made by RiotGames for their cookbooks • Uses Git tags to derive current version • Version treated as "label", not part of the source code • Jenkins pushes only Git tags, no code changes # metadata.rb name "example-cookbook" version IO.read(File.join(File.dirname(__FILE__), 'VERSION')) rescue '0.0.1'

52. 52 Versioning (2) • Publish stage notifies via Slack • Except commit message subject indicates version increase • Do this #patch • Add this #minor • Break this #major

53. 53 "Main Chef-Repo" • Not covered here, but in site-chefcitypo3org cookbook • Freestyle job, defined via JobDSL • Uses script to parse git diff and upload changes using knife

54. 54 Additional Notes • Manually configured J organization-level webhook triggers immediate builds • Everything else is done automatically, no "manual handholding"

55. Using jenkins-chefci

56. 56 Ease of Use • Checkout cookbook from github.com/TYPO3-cookbooks/jenkins-chefci • Increase GitHub API rate limit • Allow to steal your Chef credentials • Run test-kitchen export JENKINS_GITHUB_LOGIN=johndoe export JENKINS_GITHUB_TOKEN=123456supersecure export JENKINS_COPY_CHEF_CREDENTIALS=1 kitchen converge full-ubuntu-1604 WARN: allows commit status update WARN: allows knife/berks to access Chef Server

57. 57 (Potential) TODOs for you • Write your wrapper cookbook around jenkins-chefci • Point it to your organization • Add authentication, e.g., GitHub OAuth? • Fork the global library • Adjust pipeline to your needs • You don't have to agree with our (current) implementation

58. 58 Outlook / TODOs • Documentation / blog posts • Move more stuff from site-chefcitypo3org to jenkins-chefci • chefdk() global function to run in chef/chefdk Docker container • Store Chef private key as Jenkins credential • Test using multiple chef-client version • Use JobDSL for organization-folder setup (#966) • Trigger downstream cookbooks • Read out dependencies and subscribe to successful pipeline runs • Paves the road to Policyfiles • Use Jenkins slaves • Collect chef-client deprecation warnings

59. 59 Conclusion • Jenkins Pipelines allow to define pipelines as code • Groovy-based DSL allows programming of pipelines • Can definitively become complex, i.e., debugging • Jenkins Pipelines for Chef cookbook CI/CD • Running at TYPO3 since May 2016 (site-chefcitypo3org cookbook) • Public instance at https://chef-ci.typo3.org, open source from day 1 • Warning: Many cookbooks still use v1 pipeline (git-flow) • jenkins-chefci cookbook as reusable implementation • Makes setup accessible to broader audience • No dependencies to other TYPO3 cookbooks

60. 60 Further Reading • TYPO3's Chef CI: https://chef-ci.typo3.org • TYPO3-cookbooks on GitHub: https://github.com/TYPO3-cookbooks • TYPO3's Shared Global Library: https://github.com/TYPO3-infrastructure/jenkins-pipeline-global-library-chefci/ • Pipeline Tutorial: https://github.com/jenkinsci/pipeline-plugin/blob/master/TUTORIAL.md • Getting started with pipelines: https://jenkins.io/pipeline/getting-started-pipelines/ • Step documentation: https://jenkins.io/doc/pipeline/steps/ • Pipeline shared libraries: https://jenkins.io/doc/book/pipeline/shared-libraries/ • Notifications (Mail, Slack, etc.) in Scripted Pipelines: https://jenkins.io/blog/2016/07/18/pipline-notifications/ • Declarative Pipelines https://jenkins.io/blog/2016/12/19/declarative-pipeline-beta/ https://jenkins.io/blog/2017/02/03/declarative-pipeline-ga/