Jenkins Pipelines

Steffen Gebert
Steffen GebertInfrastructure dude at EMnify
Jenkins Pipelines
Steffen Gebert (@StGebert)
DevOps Meetup Frankfurt, 20.10.2016
About Me
Researcher / PhD Student
(Software-based Networks)
2011 - 2016
Core Team Member
2010 - 2013
Server Admin Team Member
since 2011
2
Co-Organizer DevOps Meetup Würzburg
since last week
Continuous Delivery
4
Agile Development
• Sprints of ~1-4 weeks duration
• Result: Product that can be
shown/given to customer
• Return: Feedback
Image	credits:	Marekventur,
Wikimedia, CC-BY-SA-3.0
5
Continuous Delivery
• Reduce cycle times (even more)
• cf. “DevOps is not enough” talk
• “How long does it take to release a one-line change?”
• Potentially deliverable code with every commit
• Automated tests decide about acceptance
• You don’t have to release/deploy it, but you could
à Continuous Deployment: Deploy every successfully
tested commit automatically
6
Pipelines
• Every check-in triggers
pipeline execution
• Feedback to the team in
every stage
• “Bring the pain forward”
• “Fail fast, fail often”
• Minimize execution time
• Always aware of
latest stable release
CI & CD Tools
8
CI/CD Tools
• On-premise
• Jenkins
• Thoughtworks Go
• Gitlab CI
• SaaS
• TravisCI
• CircleCI
• AppVeyor
• Codeship
• Visual Studio Team Services
9
Why (I like) Jenkins
• Established open source project
• On premise installation
• Thousands of plugins
• Integrates with many tools/services
• Tailored to CI/CD
10
History of CI/CD with Jenkins
• Downstream Jobs
• Job A triggers job B triggers job C triggers job D trig...
• If one job fails, fail all
• Build Pipeline View
But that’s awkward
… and what I want instead
12
Configuration as Code
• Define jobs/pipelines as code
• Avoid point & click
• In version control
• Can live with the application
• Scales better
• Example: .travis.yml (from TravisCI)
• Similar to GitlabCI
language: php
services:
- redis-server
before_script:
- composer install
script:
- ./bin/phpunit -c …
notifications:
slack:
…
13
Code-driven Approaches in Jenkins
• Jenkins Job Builder
• Python / YAML based, from the OpenStack project
• Job DSL plugin
• Groovy DSL!
(e.g. query Github API and create
jobs for all branches)
• Great thing!
• But creates single jobs
job('my-project-main') {
scm {
git('https://github.com/...')
}
triggers {
scm('H/15 * * * *')
}
publishers {
downstream('my-project-unit')
}
}
Jenkins Pipeline Plugins
15
Jenkins Pipeline Plugins
• Whole suite of plugins (10+), open-sourced earlier this year
• Shipped with Jenkins 2.0
• Formerly commercially available by CloudBees, called Workflow
• Define pipeline as code (again Groovy DSL)
stage("Hello") {
echo "*Hello*"
}
stage("World") {
echo "*World*"
}
16
Pipeline Example
node {
stage("Build") {
sh "echo Could run 'composer install' now"
}
stage("Unit") {
sh "echo Could run 'phpunit' now"
}
// ...
}
17
Pipeline Execution
• node step allocates executor slot (“heavyweight executor”)
• As other Jenkins jobs
• Filter node by labels (i.e. node('php7'), node('master'))
• Required for heavy lifting
• Avoid many sequential allocations (slows down pipeline progress)
• Code outside of node
• Flyweight executor
• Running on master, "for free”
• Pipeline execution survives Jenkins restarts (CPS)
18
Pipeline DSL Steps
• Shellout
• *nix systems: sh
• sh("make"), sh("rm -rf /")
• Windows systems: bat
• bat("C:Program Files...")
• SCM
• checkout("https://github.com/..git")
• File handling
• readFile("my.config")
• writeFile(file: "README.md", text: "Hello World")
• fileExists
19
Pipeline DSL Steps (2)
• Build another job:
• Will not go further into detail J
build("jobname")
build("jobname", wait: false, propagate: false)
Build(job: 'jobname', parameters: [
[$class: 'StringParameterValue', name: 'target', value: target]
[$class: 'ListSubversionTagsParameterValue', name: 'release', tag: release],
[$class: 'BooleanParameterValue', name: 'update_composer', value:
update_composer.to_boolean()])
20
Pipeline DSL Steps (3)
• Copy workspace to next executor
• Will not go further into detail J
stage("build") {
node {
sh("make")
stash("my-workspace")
}
}
stage("release") {
node('magic-release-tool') {
unstash("my-workspace")
sh("release")
}
}
21
Pipeline DSL Steps (4)
• Specify custom environment variables
• Use variables provided by Jenkins
• Global Groovy variables
withEnv(['MYTOOL_HOME=/usr/local/mytool']) {
sh '$MYTOOL_HOME/bin/start‘
}
env.FOO = "bar"
sh 'echo $BAR'
sh 'echo $BUILD_NUMBER > version.txt'
currentBuild.result = 'FAILED'
22
Even More Pipeline Steps
• Plugins contribute additional steps
• Steps available in this Jenkins instance via
• Online reference: https://jenkins.io/doc/pipeline/steps/
23
Example: Manually Confirm Deployment
stage("Signoff") {
input("Deploy that to prod?")
milestone()
}
stage(name: "Deploy", concurrency: 1) { node {..} }
24
Docker
• Run build jobs within Docker containers
• No need to install software on Jenkins master/slave
• Use multiple versions of the same tool
• Containers can be existing ones
or built on demand
• .. and Kubernetes
stage("run in docker") {
node {
withDockerContainer("php:7-fpm") {
sh "php -v”
}
}
}
25
Snipped Editor & Docs
• Because first steps are hard..
• Auto-generated DSL documentation
(Pipeline Syntax → Step Reference)
Where to put that config?
27
Pipeline Configuration
• Paste code into job config (fine for testing)
• Create pipelines via JobDSL
• Commit it to your repo
• File called Jenkinsfile
• It evolves with the application
• It is versioned
• Everybody can read (and modify) it
• You can throw away your Jenkins master at any time
28
Multibranch & Organization Folder Plugins
• Scans a complete GitHub/Bitbucket organisation for Jenkinsfile
• Triggered by Webhook and/or runs periodically
• Automatically adds pipeline jobs per repo/branch/PR
29
DRY: Jenkins Global Library
• Provides shared functionality available for all jobs
• Stored on Jenkins master, available to all slaves
• Available via Jenkins-integrated Git server
• Can be loaded from remote Git repos, specified in global configuration
and Jenkinsfile
@Library("https://github.com/..")
node { deployToProd() }
Jenkins Pipelines for
Chef Cookbooks
Real-World Example
31
Chef CI/CD at TYPO3.org
• Code that runs the *.typo3.org infrastructure, chef-ci.typo3.org
• Objective: Chef cookbooks
• Server provisioning (installs packages, configures services)
• Code: github.com/TYPO3-cookbooks
32
Many Cookbooks, Many Pipelines
• Scans our GitHub organization TYPO3-cookbooks
• Automatically adds/removes pipelines for branches and pull requests*
• Triggered via Webhooks
• Contents of Jenkinsfile: def pipe = new org.typo3.chefci.v1.Pipeline()
pipe.execute()
* Currently suspicious to arbitrary code execution
33
Jenkins Global Library
• Pipelines implemented in Global Library
TYPO3-infrastructure/jenkins-pipeline-global-library-chefci
34
Parallelize Integration Tests
• Run Test-Kitchen (integration test for Chef cookbooks)
• Run all instances in parallel (by Jenkins)
$ kitchen status
Instance Driver Provisioner [..] Last Action
default-debian-78 Docker ChefZero <Not Created>
default-debian-82 Docker ChefZero <Not Created>
physical-debian-78 Docker ChefZero <Not Created>
physical-debian-82 Docker ChefZero <Not Created>
production-debian-78 Docker ChefZero <Not Created>
production-debian-82 Docker ChefZero <Not Created>
35
Parallelize Integration Tests (2)
• Goal: Extract instance list, run kitchen commands in parallel
• Expected result:
parallel(
'default-debian-82': {
node {
unstash('cookbook-tk')
sh('kitchen test --destroy always default-debian-82')
}
},
'physical-debian-82': {
node {
unstash('cookbook-tk')
sh('kitchen test --destroy always physical-debian-82')
}...
36
Parallelize Integration Tests (3)
• Grab list of instance names
def ArrayList<String> getInstances(){
def instanceNames = []
node {
def lines = sh(script: 'kitchen status', returnStdout: true).split('n')
for (int i = 1; i < lines.size(); i++) {
instanceNames << lines[i].tokenize(' ')[0]
}
}
return instanceNames
}
* Closures don’t always work well within Pipeline code (cf. @NonCPS)
37
Parallelize Integration Tests (4)
def Closure getNodeForInstance(String instanceName) {
return {
// this node (one per instance) is later executed in parallel
node {
// restore workspace
unstash('cookbook-tk')
sh('kitchen test --destroy always ' + instanceName)
}}}
for (int i = 0; i < instanceNames.size(); i++) {
def instanceName = instanceNames.get(i)
plNodes[instanceName] = this.getNodeForInstance(instanceName)
}
parallel plNodes
38
Failure Notification
• Pipeline stops, when any step fails
• But.. I want that info in Slack!
def run(Object step){
try {
step.execute()
} catch (err) {
this.postBuildNotify
failTheBuild("Build failed")
}
}
def execute() {
this.prepare()
this.run(new Lint())
this.run(new BerkshelfInstall())
this.run(new TestKitchen())
this.run(new ArchiveArtifacts())
// erm… git flow is a bad idea
if (env.BRANCH_NAME == "master") {
this.run(new BerkshelfUpload())
}
}
Blue Ocean & Other News
A new Jenkins UI
40
Blue Ocean
• Tailored to the pipeline plugins
41
Blue Ocean
• Tailored to the pipeline plugins
42
Pipeline Editor*
* Recently announced, not yet available
43
Declarative Pipelines
• Sorry, the code I’ve
shown might be already
outdated.. J
• Declarative approach
solves some issues
(failure handling, post-
build steps)
pipeline {
agent docker:'node:6.3'
stages {
stage('build') {
sh '..'
}
stage ('test') {
sh 'npm test'
}
}
postBuild {
always {
sh 'echo "This will always run"'
}
failure {
sh 'echo "This will run only if failed"'
}}}
https://jenkins.io/blog/2016/09/19/blueocean
-beta-declarative-pipeline-pipeline-editor/
44
Summary
• Jenkins pipeline suite modernizes its CI/CD capabilities
• CloudBees, Inc. is very actively pushing development
• Many Jenkins plugins already compatible
• Pipeline defined as code
• Versioned
• Doesn't mess up Jenkins jobs
• Code sharing
• Automated pipeline creation based on GitHub/Bitbucket APIs
• Blue Ocean refreshes Jenkins' UI
• Still couple of rough edges (failure handling, frequent changes)
45
Further Reading
• Pipeline Tutorial:
https://github.com/jenkinsci/pipeline-plugin/blob/master/TUTORIAL.md
• Getting started with pipelines:
https://jenkins.io/pipeline/getting-started-pipelines/
• Jenkins World 2016 Wrap Up - Pipelines:
https://jenkins.io/blog/2016/09/24/jenkins-world-2016-wrap-up-pipeline/
• Step documentation:
https://jenkins.io/doc/pipeline/steps/
• Pipeline global library:
https://github.com/jenkinsci/workflow-cps-global-lib-plugin/blob/master/README.md
• Docker in Jenkins pipelines:
https://jenkins.io/blog/2016/08/08/docker-pipeline-environments/
• Notifications (Mail, Slack, etc.):
https://jenkins.io/blog/2016/07/18/pipline-notifications/
• Parallel execution:
https://jenkins.io/blog/2016/06/16/parallel-test-executor-plugin/
• Extending pipeline DSL:
https://jenkins.io/blog/2016/04/21/dsl-plugins/
• Controlling the Flow with Stage, Lock, and Milestone:
https://jenkins.io/blog/2016/10/16/stage-lock-milestone/
• TYPO3's Chef CI:
https://chef-ci.typo3.org
46
1 of 46

Recommended

Jenkins by
JenkinsJenkins
JenkinsMohanRaviRohitth
1K views25 slides
Jenkins presentation by
Jenkins presentationJenkins presentation
Jenkins presentationValentin Buryakov
1.4K views19 slides
Jenkins by
JenkinsJenkins
JenkinsRoger Xia
3.9K views16 slides
Jenkins tutorial for beginners by
Jenkins tutorial for beginnersJenkins tutorial for beginners
Jenkins tutorial for beginnersBugRaptors
583 views11 slides
Jenkins CI presentation by
Jenkins CI presentationJenkins CI presentation
Jenkins CI presentationJonathan Holloway
13.8K views20 slides
Jenkins CI by
Jenkins CIJenkins CI
Jenkins CIViyaan Jhiingade
1.1K views76 slides

More Related Content

What's hot

What is Jenkins | Jenkins Tutorial for Beginners | Edureka by
What is Jenkins | Jenkins Tutorial for Beginners | EdurekaWhat is Jenkins | Jenkins Tutorial for Beginners | Edureka
What is Jenkins | Jenkins Tutorial for Beginners | EdurekaEdureka!
1.8K views30 slides
Jenkins-CI by
Jenkins-CIJenkins-CI
Jenkins-CIGong Haibing
750 views39 slides
Introduction to jenkins by
Introduction to jenkinsIntroduction to jenkins
Introduction to jenkinsAbe Diaz
12.9K views27 slides
Jenkins Introduction by
Jenkins IntroductionJenkins Introduction
Jenkins IntroductionPavan Gupta
1.2K views26 slides
Jenkins tutorial by
Jenkins tutorialJenkins tutorial
Jenkins tutorialMamun Rashid, CCDH
1.2K views32 slides
Jenkins Automation by
Jenkins AutomationJenkins Automation
Jenkins AutomationJulien Pivotto
5.2K views104 slides

What's hot(20)

What is Jenkins | Jenkins Tutorial for Beginners | Edureka by Edureka!
What is Jenkins | Jenkins Tutorial for Beginners | EdurekaWhat is Jenkins | Jenkins Tutorial for Beginners | Edureka
What is Jenkins | Jenkins Tutorial for Beginners | Edureka
Edureka!1.8K views
Introduction to jenkins by Abe Diaz
Introduction to jenkinsIntroduction to jenkins
Introduction to jenkins
Abe Diaz12.9K views
Jenkins Introduction by Pavan Gupta
Jenkins IntroductionJenkins Introduction
Jenkins Introduction
Pavan Gupta1.2K views
CI/CD with Jenkins and Docker - DevOps Meetup Day Thailand by Troublemaker Khunpech
CI/CD with Jenkins and Docker - DevOps Meetup Day ThailandCI/CD with Jenkins and Docker - DevOps Meetup Day Thailand
CI/CD with Jenkins and Docker - DevOps Meetup Day Thailand
Continuous integration by amscanne
Continuous integrationContinuous integration
Continuous integration
amscanne15.6K views
Seven Habits of Highly Effective Jenkins Users (2014 edition!) by Andrew Bayer
Seven Habits of Highly Effective Jenkins Users (2014 edition!)Seven Habits of Highly Effective Jenkins Users (2014 edition!)
Seven Habits of Highly Effective Jenkins Users (2014 edition!)
Andrew Bayer106.5K views
An Introduction To Jenkins by Knoldus Inc.
An Introduction To JenkinsAn Introduction To Jenkins
An Introduction To Jenkins
Knoldus Inc.23.2K views
CI CD Pipeline Using Jenkins | Continuous Integration and Deployment | DevOps... by Edureka!
CI CD Pipeline Using Jenkins | Continuous Integration and Deployment | DevOps...CI CD Pipeline Using Jenkins | Continuous Integration and Deployment | DevOps...
CI CD Pipeline Using Jenkins | Continuous Integration and Deployment | DevOps...
Edureka!5K views
Introduction to CI/CD by Hoang Le
Introduction to CI/CDIntroduction to CI/CD
Introduction to CI/CD
Hoang Le1K views
Introduction à l’intégration continue avec Jenkins by Eric Hogue
Introduction à l’intégration continue avec JenkinsIntroduction à l’intégration continue avec Jenkins
Introduction à l’intégration continue avec Jenkins
Eric Hogue9.1K views
Maven Basics - Explained by Smita Prasad
Maven Basics - ExplainedMaven Basics - Explained
Maven Basics - Explained
Smita Prasad1.1K views
Jenkins Pipeline Tutorial | Continuous Delivery Pipeline Using Jenkins | DevO... by Edureka!
Jenkins Pipeline Tutorial | Continuous Delivery Pipeline Using Jenkins | DevO...Jenkins Pipeline Tutorial | Continuous Delivery Pipeline Using Jenkins | DevO...
Jenkins Pipeline Tutorial | Continuous Delivery Pipeline Using Jenkins | DevO...
Edureka!2K views
CI and CD with Jenkins by Martin Málek
CI and CD with JenkinsCI and CD with Jenkins
CI and CD with Jenkins
Martin Málek93.4K views
Continuous Integration, Build Pipelines and Continuous Deployment by Christopher Read
Continuous Integration, Build Pipelines and Continuous DeploymentContinuous Integration, Build Pipelines and Continuous Deployment
Continuous Integration, Build Pipelines and Continuous Deployment
Christopher Read26.8K views

Similar to Jenkins Pipelines

(Declarative) Jenkins Pipelines by
(Declarative) Jenkins Pipelines(Declarative) Jenkins Pipelines
(Declarative) Jenkins PipelinesSteffen Gebert
21.7K views52 slides
An Open-Source Chef Cookbook CI/CD Implementation Using Jenkins Pipelines by
An Open-Source Chef Cookbook CI/CD Implementation Using Jenkins PipelinesAn Open-Source Chef Cookbook CI/CD Implementation Using Jenkins Pipelines
An Open-Source Chef Cookbook CI/CD Implementation Using Jenkins PipelinesSteffen Gebert
5.3K views60 slides
Atlanta Jenkins Area Meetup October 22nd 2015 by
Atlanta Jenkins Area Meetup October 22nd 2015Atlanta Jenkins Area Meetup October 22nd 2015
Atlanta Jenkins Area Meetup October 22nd 2015Kurt Madel
667 views25 slides
DevOps Odessa #TechTalks 21.01.2020 by
DevOps Odessa #TechTalks 21.01.2020DevOps Odessa #TechTalks 21.01.2020
DevOps Odessa #TechTalks 21.01.2020Lohika_Odessa_TechTalks
252 views28 slides
Docker and Puppet for Continuous Integration by
Docker and Puppet for Continuous IntegrationDocker and Puppet for Continuous Integration
Docker and Puppet for Continuous IntegrationGiacomo Vacca
5.8K views39 slides
Jenkins days workshop pipelines - Eric Long by
Jenkins days workshop  pipelines - Eric LongJenkins days workshop  pipelines - Eric Long
Jenkins days workshop pipelines - Eric Longericlongtx
1.8K views44 slides

Similar to Jenkins Pipelines(20)

(Declarative) Jenkins Pipelines by Steffen Gebert
(Declarative) Jenkins Pipelines(Declarative) Jenkins Pipelines
(Declarative) Jenkins Pipelines
Steffen Gebert21.7K views
An Open-Source Chef Cookbook CI/CD Implementation Using Jenkins Pipelines by Steffen Gebert
An Open-Source Chef Cookbook CI/CD Implementation Using Jenkins PipelinesAn Open-Source Chef Cookbook CI/CD Implementation Using Jenkins Pipelines
An Open-Source Chef Cookbook CI/CD Implementation Using Jenkins Pipelines
Steffen Gebert5.3K views
Atlanta Jenkins Area Meetup October 22nd 2015 by Kurt Madel
Atlanta Jenkins Area Meetup October 22nd 2015Atlanta Jenkins Area Meetup October 22nd 2015
Atlanta Jenkins Area Meetup October 22nd 2015
Kurt Madel667 views
Docker and Puppet for Continuous Integration by Giacomo Vacca
Docker and Puppet for Continuous IntegrationDocker and Puppet for Continuous Integration
Docker and Puppet for Continuous Integration
Giacomo Vacca5.8K views
Jenkins days workshop pipelines - Eric Long by ericlongtx
Jenkins days workshop  pipelines - Eric LongJenkins days workshop  pipelines - Eric Long
Jenkins days workshop pipelines - Eric Long
ericlongtx1.8K views
CIbox - OpenSource solution for making your #devops better by Andrii Podanenko
CIbox - OpenSource solution for making your #devops betterCIbox - OpenSource solution for making your #devops better
CIbox - OpenSource solution for making your #devops better
Andrii Podanenko4.3K views
OpenShift Build Pipelines @ Lightweight Java User Group Meetup by Tobias Schneck
OpenShift Build Pipelines @ Lightweight Java User Group MeetupOpenShift Build Pipelines @ Lightweight Java User Group Meetup
OpenShift Build Pipelines @ Lightweight Java User Group Meetup
Tobias Schneck578 views
Moving from Jenkins 1 to 2 declarative pipeline adventures by Frits Van Der Holst
Moving from Jenkins 1 to 2 declarative pipeline adventuresMoving from Jenkins 1 to 2 declarative pipeline adventures
Moving from Jenkins 1 to 2 declarative pipeline adventures
Frits Van Der Holst1.3K views
Pipeline as code - new feature in Jenkins 2 by Michal Ziarnik
Pipeline as code - new feature in Jenkins 2Pipeline as code - new feature in Jenkins 2
Pipeline as code - new feature in Jenkins 2
Michal Ziarnik2.3K views
Jenkins Days - Workshop - Let's Build a Pipeline - Los Angeles by Andy Pemberton
Jenkins Days - Workshop - Let's Build a Pipeline - Los AngelesJenkins Days - Workshop - Let's Build a Pipeline - Los Angeles
Jenkins Days - Workshop - Let's Build a Pipeline - Los Angeles
Andy Pemberton16.9K views
MoldCamp - multidimentional testing workflow. CIBox. by Andrii Podanenko
MoldCamp  - multidimentional testing workflow. CIBox.MoldCamp  - multidimentional testing workflow. CIBox.
MoldCamp - multidimentional testing workflow. CIBox.
Andrii Podanenko1.2K views
Building an Extensible, Resumable DSL on Top of Apache Groovy by jgcloudbees
Building an Extensible, Resumable DSL on Top of Apache GroovyBuilding an Extensible, Resumable DSL on Top of Apache Groovy
Building an Extensible, Resumable DSL on Top of Apache Groovy
jgcloudbees842 views
Continuous Integration With Jenkins Docker SQL Server by Chris Adkin
Continuous Integration With Jenkins Docker SQL ServerContinuous Integration With Jenkins Docker SQL Server
Continuous Integration With Jenkins Docker SQL Server
Chris Adkin925 views
Continuous Deployment with Kubernetes, Docker and GitLab CI by alexanderkiel
Continuous Deployment with Kubernetes, Docker and GitLab CIContinuous Deployment with Kubernetes, Docker and GitLab CI
Continuous Deployment with Kubernetes, Docker and GitLab CI
alexanderkiel6.9K views
Continuous Integration & Development with Gitlab by Ayush Sharma
Continuous Integration & Development with GitlabContinuous Integration & Development with Gitlab
Continuous Integration & Development with Gitlab
Ayush Sharma228 views
Practical introduction to dev ops with chef by LeanDog
Practical introduction to dev ops with chefPractical introduction to dev ops with chef
Practical introduction to dev ops with chef
LeanDog10.2K views
What to expect from Java 9 by Ivan Krylov
What to expect from Java 9What to expect from Java 9
What to expect from Java 9
Ivan Krylov717 views

More from Steffen Gebert

Wenn selbst ‘erlaube allen Verkehr von 0.0.0.0/0’ nicht hilft - Verbindungspr... by
Wenn selbst ‘erlaube allen Verkehr von 0.0.0.0/0’ nicht hilft - Verbindungspr...Wenn selbst ‘erlaube allen Verkehr von 0.0.0.0/0’ nicht hilft - Verbindungspr...
Wenn selbst ‘erlaube allen Verkehr von 0.0.0.0/0’ nicht hilft - Verbindungspr...Steffen Gebert
53 views71 slides
Feature Management Platforms by
Feature Management PlatformsFeature Management Platforms
Feature Management PlatformsSteffen Gebert
44 views34 slides
Serverless Networking - How We Provide Cloud-Native Connectivity for IoT Devices by
Serverless Networking - How We Provide Cloud-Native Connectivity for IoT DevicesServerless Networking - How We Provide Cloud-Native Connectivity for IoT Devices
Serverless Networking - How We Provide Cloud-Native Connectivity for IoT DevicesSteffen Gebert
252 views51 slides
How our Cloudy Mindsets Approached Physical Routers by
How our Cloudy Mindsets Approached Physical RoutersHow our Cloudy Mindsets Approached Physical Routers
How our Cloudy Mindsets Approached Physical RoutersSteffen Gebert
137 views63 slides
Jenkins vs. AWS CodePipeline (AWS User Group Berlin) by
Jenkins vs. AWS CodePipeline (AWS User Group Berlin)Jenkins vs. AWS CodePipeline (AWS User Group Berlin)
Jenkins vs. AWS CodePipeline (AWS User Group Berlin)Steffen Gebert
454 views64 slides
Jenkins vs. AWS CodePipeline by
Jenkins vs. AWS CodePipelineJenkins vs. AWS CodePipeline
Jenkins vs. AWS CodePipelineSteffen Gebert
2.6K views68 slides

More from Steffen Gebert(20)

Wenn selbst ‘erlaube allen Verkehr von 0.0.0.0/0’ nicht hilft - Verbindungspr... by Steffen Gebert
Wenn selbst ‘erlaube allen Verkehr von 0.0.0.0/0’ nicht hilft - Verbindungspr...Wenn selbst ‘erlaube allen Verkehr von 0.0.0.0/0’ nicht hilft - Verbindungspr...
Wenn selbst ‘erlaube allen Verkehr von 0.0.0.0/0’ nicht hilft - Verbindungspr...
Steffen Gebert53 views
Serverless Networking - How We Provide Cloud-Native Connectivity for IoT Devices by Steffen Gebert
Serverless Networking - How We Provide Cloud-Native Connectivity for IoT DevicesServerless Networking - How We Provide Cloud-Native Connectivity for IoT Devices
Serverless Networking - How We Provide Cloud-Native Connectivity for IoT Devices
Steffen Gebert252 views
How our Cloudy Mindsets Approached Physical Routers by Steffen Gebert
How our Cloudy Mindsets Approached Physical RoutersHow our Cloudy Mindsets Approached Physical Routers
How our Cloudy Mindsets Approached Physical Routers
Steffen Gebert137 views
Jenkins vs. AWS CodePipeline (AWS User Group Berlin) by Steffen Gebert
Jenkins vs. AWS CodePipeline (AWS User Group Berlin)Jenkins vs. AWS CodePipeline (AWS User Group Berlin)
Jenkins vs. AWS CodePipeline (AWS User Group Berlin)
Steffen Gebert454 views
Jenkins vs. AWS CodePipeline by Steffen Gebert
Jenkins vs. AWS CodePipelineJenkins vs. AWS CodePipeline
Jenkins vs. AWS CodePipeline
Steffen Gebert2.6K views
Monitoring Akka with Kamon 1.0 by Steffen Gebert
Monitoring Akka with Kamon 1.0Monitoring Akka with Kamon 1.0
Monitoring Akka with Kamon 1.0
Steffen Gebert2.4K views
Let's go HTTPS-only! - More Than Buying a Certificate by Steffen Gebert
Let's go HTTPS-only! - More Than Buying a CertificateLet's go HTTPS-only! - More Than Buying a Certificate
Let's go HTTPS-only! - More Than Buying a Certificate
Steffen Gebert1.1K views
Cleaning Up the Dirt of the Nineties - How New Protocols are Modernizing the Web by Steffen Gebert
Cleaning Up the Dirt of the Nineties - How New Protocols are Modernizing the WebCleaning Up the Dirt of the Nineties - How New Protocols are Modernizing the Web
Cleaning Up the Dirt of the Nineties - How New Protocols are Modernizing the Web
Steffen Gebert1.5K views
Investigating the Impact of Network Topology on the Processing Times of SDN C... by Steffen Gebert
Investigating the Impact of Network Topology on the Processing Times of SDN C...Investigating the Impact of Network Topology on the Processing Times of SDN C...
Investigating the Impact of Network Topology on the Processing Times of SDN C...
Steffen Gebert1.4K views
SDN interfaces and performance analysis of SDN components by Steffen Gebert
SDN interfaces and performance analysis of SDN componentsSDN interfaces and performance analysis of SDN components
SDN interfaces and performance analysis of SDN components
Steffen Gebert2.3K views
The Development Infrastructure of the TYPO3 Project by Steffen Gebert
The Development Infrastructure of the TYPO3 ProjectThe Development Infrastructure of the TYPO3 Project
The Development Infrastructure of the TYPO3 Project
Steffen Gebert881 views
Der Weg zu TYPO3 CMS 6.0 und Einblicke in die TYPO3-Entwicklung by Steffen Gebert
Der Weg zu TYPO3 CMS 6.0 und Einblicke in die TYPO3-EntwicklungDer Weg zu TYPO3 CMS 6.0 und Einblicke in die TYPO3-Entwicklung
Der Weg zu TYPO3 CMS 6.0 und Einblicke in die TYPO3-Entwicklung
Steffen Gebert6.7K views
Official typo3.org infrastructure &
the TYPO3 Server Admin Team by Steffen Gebert
Official typo3.org infrastructure &
the TYPO3 Server Admin TeamOfficial typo3.org infrastructure &
the TYPO3 Server Admin Team
Official typo3.org infrastructure &
the TYPO3 Server Admin Team
Steffen Gebert1.4K views
Neuigkeiten aus dem TYPO3-Projekt by Steffen Gebert
Neuigkeiten aus dem TYPO3-ProjektNeuigkeiten aus dem TYPO3-Projekt
Neuigkeiten aus dem TYPO3-Projekt
Steffen Gebert1.4K views

Recently uploaded

The Importance of Cybersecurity for Digital Transformation by
The Importance of Cybersecurity for Digital TransformationThe Importance of Cybersecurity for Digital Transformation
The Importance of Cybersecurity for Digital TransformationNUS-ISS
25 views26 slides
Upskilling the Evolving Workforce with Digital Fluency for Tomorrow's Challen... by
Upskilling the Evolving Workforce with Digital Fluency for Tomorrow's Challen...Upskilling the Evolving Workforce with Digital Fluency for Tomorrow's Challen...
Upskilling the Evolving Workforce with Digital Fluency for Tomorrow's Challen...NUS-ISS
23 views70 slides
Web Dev - 1 PPT.pdf by
Web Dev - 1 PPT.pdfWeb Dev - 1 PPT.pdf
Web Dev - 1 PPT.pdfgdsczhcet
52 views45 slides
Data-centric AI and the convergence of data and model engineering: opportunit... by
Data-centric AI and the convergence of data and model engineering:opportunit...Data-centric AI and the convergence of data and model engineering:opportunit...
Data-centric AI and the convergence of data and model engineering: opportunit...Paolo Missier
29 views40 slides
Photowave Presentation Slides - 11.8.23.pptx by
Photowave Presentation Slides - 11.8.23.pptxPhotowave Presentation Slides - 11.8.23.pptx
Photowave Presentation Slides - 11.8.23.pptxCXL Forum
126 views16 slides
AI: mind, matter, meaning, metaphors, being, becoming, life values by
AI: mind, matter, meaning, metaphors, being, becoming, life valuesAI: mind, matter, meaning, metaphors, being, becoming, life values
AI: mind, matter, meaning, metaphors, being, becoming, life valuesTwain Liu 刘秋艳
34 views16 slides

Recently uploaded(20)

The Importance of Cybersecurity for Digital Transformation by NUS-ISS
The Importance of Cybersecurity for Digital TransformationThe Importance of Cybersecurity for Digital Transformation
The Importance of Cybersecurity for Digital Transformation
NUS-ISS25 views
Upskilling the Evolving Workforce with Digital Fluency for Tomorrow's Challen... by NUS-ISS
Upskilling the Evolving Workforce with Digital Fluency for Tomorrow's Challen...Upskilling the Evolving Workforce with Digital Fluency for Tomorrow's Challen...
Upskilling the Evolving Workforce with Digital Fluency for Tomorrow's Challen...
NUS-ISS23 views
Web Dev - 1 PPT.pdf by gdsczhcet
Web Dev - 1 PPT.pdfWeb Dev - 1 PPT.pdf
Web Dev - 1 PPT.pdf
gdsczhcet52 views
Data-centric AI and the convergence of data and model engineering: opportunit... by Paolo Missier
Data-centric AI and the convergence of data and model engineering:opportunit...Data-centric AI and the convergence of data and model engineering:opportunit...
Data-centric AI and the convergence of data and model engineering: opportunit...
Paolo Missier29 views
Photowave Presentation Slides - 11.8.23.pptx by CXL Forum
Photowave Presentation Slides - 11.8.23.pptxPhotowave Presentation Slides - 11.8.23.pptx
Photowave Presentation Slides - 11.8.23.pptx
CXL Forum126 views
AI: mind, matter, meaning, metaphors, being, becoming, life values by Twain Liu 刘秋艳
AI: mind, matter, meaning, metaphors, being, becoming, life valuesAI: mind, matter, meaning, metaphors, being, becoming, life values
AI: mind, matter, meaning, metaphors, being, becoming, life values
"How we switched to Kanban and how it integrates with product planning", Vady... by Fwdays
"How we switched to Kanban and how it integrates with product planning", Vady..."How we switched to Kanban and how it integrates with product planning", Vady...
"How we switched to Kanban and how it integrates with product planning", Vady...
Fwdays61 views
.conf Go 2023 - How KPN drives Customer Satisfaction on IPTV by Splunk
.conf Go 2023 - How KPN drives Customer Satisfaction on IPTV.conf Go 2023 - How KPN drives Customer Satisfaction on IPTV
.conf Go 2023 - How KPN drives Customer Satisfaction on IPTV
Splunk86 views
Transcript: The Details of Description Techniques tips and tangents on altern... by BookNet Canada
Transcript: The Details of Description Techniques tips and tangents on altern...Transcript: The Details of Description Techniques tips and tangents on altern...
Transcript: The Details of Description Techniques tips and tangents on altern...
BookNet Canada119 views
The details of description: Techniques, tips, and tangents on alternative tex... by BookNet Canada
The details of description: Techniques, tips, and tangents on alternative tex...The details of description: Techniques, tips, and tangents on alternative tex...
The details of description: Techniques, tips, and tangents on alternative tex...
BookNet Canada110 views
"Fast Start to Building on AWS", Igor Ivaniuk by Fwdays
"Fast Start to Building on AWS", Igor Ivaniuk"Fast Start to Building on AWS", Igor Ivaniuk
"Fast Start to Building on AWS", Igor Ivaniuk
Fwdays36 views
Combining Orchestration and Choreography for a Clean Architecture by ThomasHeinrichs1
Combining Orchestration and Choreography for a Clean ArchitectureCombining Orchestration and Choreography for a Clean Architecture
Combining Orchestration and Choreography for a Clean Architecture
ThomasHeinrichs168 views
.conf Go 2023 - Data analysis as a routine by Splunk
.conf Go 2023 - Data analysis as a routine.conf Go 2023 - Data analysis as a routine
.conf Go 2023 - Data analysis as a routine
Splunk90 views
MemVerge: Memory Viewer Software by CXL Forum
MemVerge: Memory Viewer SoftwareMemVerge: Memory Viewer Software
MemVerge: Memory Viewer Software
CXL Forum118 views
TE Connectivity: Card Edge Interconnects by CXL Forum
TE Connectivity: Card Edge InterconnectsTE Connectivity: Card Edge Interconnects
TE Connectivity: Card Edge Interconnects
CXL Forum96 views
"Ukrainian Mobile Banking Scaling in Practice. From 0 to 100 and beyond", Vad... by Fwdays
"Ukrainian Mobile Banking Scaling in Practice. From 0 to 100 and beyond", Vad..."Ukrainian Mobile Banking Scaling in Practice. From 0 to 100 and beyond", Vad...
"Ukrainian Mobile Banking Scaling in Practice. From 0 to 100 and beyond", Vad...
Fwdays40 views
Business Analyst Series 2023 - Week 3 Session 5 by DianaGray10
Business Analyst Series 2023 -  Week 3 Session 5Business Analyst Series 2023 -  Week 3 Session 5
Business Analyst Series 2023 - Week 3 Session 5
DianaGray10165 views
Samsung: CMM-H Tiered Memory Solution with Built-in DRAM by CXL Forum
Samsung: CMM-H Tiered Memory Solution with Built-in DRAMSamsung: CMM-H Tiered Memory Solution with Built-in DRAM
Samsung: CMM-H Tiered Memory Solution with Built-in DRAM
CXL Forum105 views
Micron CXL product and architecture update by CXL Forum
Micron CXL product and architecture updateMicron CXL product and architecture update
Micron CXL product and architecture update
CXL Forum27 views

Jenkins Pipelines

  • 1. Jenkins Pipelines Steffen Gebert (@StGebert) DevOps Meetup Frankfurt, 20.10.2016
  • 2. About Me Researcher / PhD Student (Software-based Networks) 2011 - 2016 Core Team Member 2010 - 2013 Server Admin Team Member since 2011 2 Co-Organizer DevOps Meetup Würzburg since last week
  • 4. 4 Agile Development • Sprints of ~1-4 weeks duration • Result: Product that can be shown/given to customer • Return: Feedback Image credits: Marekventur, Wikimedia, CC-BY-SA-3.0
  • 5. 5 Continuous Delivery • Reduce cycle times (even more) • cf. “DevOps is not enough” talk • “How long does it take to release a one-line change?” • Potentially deliverable code with every commit • Automated tests decide about acceptance • You don’t have to release/deploy it, but you could à Continuous Deployment: Deploy every successfully tested commit automatically
  • 6. 6 Pipelines • Every check-in triggers pipeline execution • Feedback to the team in every stage • “Bring the pain forward” • “Fail fast, fail often” • Minimize execution time • Always aware of latest stable release
  • 7. CI & CD Tools
  • 8. 8 CI/CD Tools • On-premise • Jenkins • Thoughtworks Go • Gitlab CI • SaaS • TravisCI • CircleCI • AppVeyor • Codeship • Visual Studio Team Services
  • 9. 9 Why (I like) Jenkins • Established open source project • On premise installation • Thousands of plugins • Integrates with many tools/services • Tailored to CI/CD
  • 10. 10 History of CI/CD with Jenkins • Downstream Jobs • Job A triggers job B triggers job C triggers job D trig... • If one job fails, fail all • Build Pipeline View
  • 11. But that’s awkward … and what I want instead
  • 12. 12 Configuration as Code • Define jobs/pipelines as code • Avoid point & click • In version control • Can live with the application • Scales better • Example: .travis.yml (from TravisCI) • Similar to GitlabCI language: php services: - redis-server before_script: - composer install script: - ./bin/phpunit -c … notifications: slack: …
  • 13. 13 Code-driven Approaches in Jenkins • Jenkins Job Builder • Python / YAML based, from the OpenStack project • Job DSL plugin • Groovy DSL! (e.g. query Github API and create jobs for all branches) • Great thing! • But creates single jobs job('my-project-main') { scm { git('https://github.com/...') } triggers { scm('H/15 * * * *') } publishers { downstream('my-project-unit') } }
  • 15. 15 Jenkins Pipeline Plugins • Whole suite of plugins (10+), open-sourced earlier this year • Shipped with Jenkins 2.0 • Formerly commercially available by CloudBees, called Workflow • Define pipeline as code (again Groovy DSL) stage("Hello") { echo "*Hello*" } stage("World") { echo "*World*" }
  • 16. 16 Pipeline Example node { stage("Build") { sh "echo Could run 'composer install' now" } stage("Unit") { sh "echo Could run 'phpunit' now" } // ... }
  • 17. 17 Pipeline Execution • node step allocates executor slot (“heavyweight executor”) • As other Jenkins jobs • Filter node by labels (i.e. node('php7'), node('master')) • Required for heavy lifting • Avoid many sequential allocations (slows down pipeline progress) • Code outside of node • Flyweight executor • Running on master, "for free” • Pipeline execution survives Jenkins restarts (CPS)
  • 18. 18 Pipeline DSL Steps • Shellout • *nix systems: sh • sh("make"), sh("rm -rf /") • Windows systems: bat • bat("C:Program Files...") • SCM • checkout("https://github.com/..git") • File handling • readFile("my.config") • writeFile(file: "README.md", text: "Hello World") • fileExists
  • 19. 19 Pipeline DSL Steps (2) • Build another job: • Will not go further into detail J build("jobname") build("jobname", wait: false, propagate: false) Build(job: 'jobname', parameters: [ [$class: 'StringParameterValue', name: 'target', value: target] [$class: 'ListSubversionTagsParameterValue', name: 'release', tag: release], [$class: 'BooleanParameterValue', name: 'update_composer', value: update_composer.to_boolean()])
  • 20. 20 Pipeline DSL Steps (3) • Copy workspace to next executor • Will not go further into detail J stage("build") { node { sh("make") stash("my-workspace") } } stage("release") { node('magic-release-tool') { unstash("my-workspace") sh("release") } }
  • 21. 21 Pipeline DSL Steps (4) • Specify custom environment variables • Use variables provided by Jenkins • Global Groovy variables withEnv(['MYTOOL_HOME=/usr/local/mytool']) { sh '$MYTOOL_HOME/bin/start‘ } env.FOO = "bar" sh 'echo $BAR' sh 'echo $BUILD_NUMBER > version.txt' currentBuild.result = 'FAILED'
  • 22. 22 Even More Pipeline Steps • Plugins contribute additional steps • Steps available in this Jenkins instance via • Online reference: https://jenkins.io/doc/pipeline/steps/
  • 23. 23 Example: Manually Confirm Deployment stage("Signoff") { input("Deploy that to prod?") milestone() } stage(name: "Deploy", concurrency: 1) { node {..} }
  • 24. 24 Docker • Run build jobs within Docker containers • No need to install software on Jenkins master/slave • Use multiple versions of the same tool • Containers can be existing ones or built on demand • .. and Kubernetes stage("run in docker") { node { withDockerContainer("php:7-fpm") { sh "php -v” } } }
  • 25. 25 Snipped Editor & Docs • Because first steps are hard.. • Auto-generated DSL documentation (Pipeline Syntax → Step Reference)
  • 26. Where to put that config?
  • 27. 27 Pipeline Configuration • Paste code into job config (fine for testing) • Create pipelines via JobDSL • Commit it to your repo • File called Jenkinsfile • It evolves with the application • It is versioned • Everybody can read (and modify) it • You can throw away your Jenkins master at any time
  • 28. 28 Multibranch & Organization Folder Plugins • Scans a complete GitHub/Bitbucket organisation for Jenkinsfile • Triggered by Webhook and/or runs periodically • Automatically adds pipeline jobs per repo/branch/PR
  • 29. 29 DRY: Jenkins Global Library • Provides shared functionality available for all jobs • Stored on Jenkins master, available to all slaves • Available via Jenkins-integrated Git server • Can be loaded from remote Git repos, specified in global configuration and Jenkinsfile @Library("https://github.com/..") node { deployToProd() }
  • 30. Jenkins Pipelines for Chef Cookbooks Real-World Example
  • 31. 31 Chef CI/CD at TYPO3.org • Code that runs the *.typo3.org infrastructure, chef-ci.typo3.org • Objective: Chef cookbooks • Server provisioning (installs packages, configures services) • Code: github.com/TYPO3-cookbooks
  • 32. 32 Many Cookbooks, Many Pipelines • Scans our GitHub organization TYPO3-cookbooks • Automatically adds/removes pipelines for branches and pull requests* • Triggered via Webhooks • Contents of Jenkinsfile: def pipe = new org.typo3.chefci.v1.Pipeline() pipe.execute() * Currently suspicious to arbitrary code execution
  • 33. 33 Jenkins Global Library • Pipelines implemented in Global Library TYPO3-infrastructure/jenkins-pipeline-global-library-chefci
  • 34. 34 Parallelize Integration Tests • Run Test-Kitchen (integration test for Chef cookbooks) • Run all instances in parallel (by Jenkins) $ kitchen status Instance Driver Provisioner [..] Last Action default-debian-78 Docker ChefZero <Not Created> default-debian-82 Docker ChefZero <Not Created> physical-debian-78 Docker ChefZero <Not Created> physical-debian-82 Docker ChefZero <Not Created> production-debian-78 Docker ChefZero <Not Created> production-debian-82 Docker ChefZero <Not Created>
  • 35. 35 Parallelize Integration Tests (2) • Goal: Extract instance list, run kitchen commands in parallel • Expected result: parallel( 'default-debian-82': { node { unstash('cookbook-tk') sh('kitchen test --destroy always default-debian-82') } }, 'physical-debian-82': { node { unstash('cookbook-tk') sh('kitchen test --destroy always physical-debian-82') }...
  • 36. 36 Parallelize Integration Tests (3) • Grab list of instance names def ArrayList<String> getInstances(){ def instanceNames = [] node { def lines = sh(script: 'kitchen status', returnStdout: true).split('n') for (int i = 1; i < lines.size(); i++) { instanceNames << lines[i].tokenize(' ')[0] } } return instanceNames } * Closures don’t always work well within Pipeline code (cf. @NonCPS)
  • 37. 37 Parallelize Integration Tests (4) def Closure getNodeForInstance(String instanceName) { return { // this node (one per instance) is later executed in parallel node { // restore workspace unstash('cookbook-tk') sh('kitchen test --destroy always ' + instanceName) }}} for (int i = 0; i < instanceNames.size(); i++) { def instanceName = instanceNames.get(i) plNodes[instanceName] = this.getNodeForInstance(instanceName) } parallel plNodes
  • 38. 38 Failure Notification • Pipeline stops, when any step fails • But.. I want that info in Slack! def run(Object step){ try { step.execute() } catch (err) { this.postBuildNotify failTheBuild("Build failed") } } def execute() { this.prepare() this.run(new Lint()) this.run(new BerkshelfInstall()) this.run(new TestKitchen()) this.run(new ArchiveArtifacts()) // erm… git flow is a bad idea if (env.BRANCH_NAME == "master") { this.run(new BerkshelfUpload()) } }
  • 39. Blue Ocean & Other News A new Jenkins UI
  • 40. 40 Blue Ocean • Tailored to the pipeline plugins
  • 41. 41 Blue Ocean • Tailored to the pipeline plugins
  • 42. 42 Pipeline Editor* * Recently announced, not yet available
  • 43. 43 Declarative Pipelines • Sorry, the code I’ve shown might be already outdated.. J • Declarative approach solves some issues (failure handling, post- build steps) pipeline { agent docker:'node:6.3' stages { stage('build') { sh '..' } stage ('test') { sh 'npm test' } } postBuild { always { sh 'echo "This will always run"' } failure { sh 'echo "This will run only if failed"' }}} https://jenkins.io/blog/2016/09/19/blueocean -beta-declarative-pipeline-pipeline-editor/
  • 44. 44 Summary • Jenkins pipeline suite modernizes its CI/CD capabilities • CloudBees, Inc. is very actively pushing development • Many Jenkins plugins already compatible • Pipeline defined as code • Versioned • Doesn't mess up Jenkins jobs • Code sharing • Automated pipeline creation based on GitHub/Bitbucket APIs • Blue Ocean refreshes Jenkins' UI • Still couple of rough edges (failure handling, frequent changes)
  • 45. 45 Further Reading • Pipeline Tutorial: https://github.com/jenkinsci/pipeline-plugin/blob/master/TUTORIAL.md • Getting started with pipelines: https://jenkins.io/pipeline/getting-started-pipelines/ • Jenkins World 2016 Wrap Up - Pipelines: https://jenkins.io/blog/2016/09/24/jenkins-world-2016-wrap-up-pipeline/ • Step documentation: https://jenkins.io/doc/pipeline/steps/ • Pipeline global library: https://github.com/jenkinsci/workflow-cps-global-lib-plugin/blob/master/README.md • Docker in Jenkins pipelines: https://jenkins.io/blog/2016/08/08/docker-pipeline-environments/ • Notifications (Mail, Slack, etc.): https://jenkins.io/blog/2016/07/18/pipline-notifications/ • Parallel execution: https://jenkins.io/blog/2016/06/16/parallel-test-executor-plugin/ • Extending pipeline DSL: https://jenkins.io/blog/2016/04/21/dsl-plugins/ • Controlling the Flow with Stage, Lock, and Milestone: https://jenkins.io/blog/2016/10/16/stage-lock-milestone/ • TYPO3's Chef CI: https://chef-ci.typo3.org
  • 46. 46