Delivery Pipeline as Code: using Jenkins 2.0 Pipeline

RADIOLOGY WORKFLOW SOLUTIONS
© 2015 medavis GmbH. All rights reserved.
DevOps Karlsruhe
Slawa Giterman - July 2016
Delivery Pipeline as Code
using Jenkins 2.0 Pipeline

 Slawa Giterman
 Java Developer (10+ years)
 A lot of experience with DevOps:
 Continuous Integration & Build Tools
 Delivery Pipeline
 Automated acceptance testing
About myself

 medavis GmbH - Software solutions for
radiological workflows (MRT, CT, nuclear
medicine, etc.)
 Browser clients (Java, JEE, Vaadin)
 Fat clients (Windows)
 Integration with multiple 3rd party medical
services (HIS, HL7, DICOM, etc.)
About my company

 Life before Jenkins Pipeline
 Pipeline as Code
 Jenkins Pipeline DSL
 How to use existing plug-ins in pipeline
 Lessons learned
Agenda

"Old" Jenkins Pipeline Plug-Ins
 Since 2011 there have been multiple Jenkins plug-ins
which supported pipelines, e.g. Build Pipeline Plugin
or Build Pipeline Plugin (pictured below)

"Old" Jenkins Pipeline Plug-Ins
 They presented multiple Jenkins jobs as if they were
part of the same pipeline
 From Jenkins point of view these pipelines were just
collections of jobs which triggered each other

Decision Points and Fork / Join Logic
 It was necessary to use plug-ins to describe decision points
or fork / join workflows
 Maintenance cost for complex pipelines was too high and
rose exponentially with the number of steps

Jenkins Job DSL Plug-In
 A partial solution is to use Jenkins Job DSL plug-in
 It allows to create jobs using Groovy scripts instead
of complex manual configurations (see an example
on the next slide)
 But Groovy scripts can only be used during job
creation not during job executions

Jenkins Job DSL Plug-In Example
void createMavenJob(String jobName, String svnUrl) {
job(jobName) {
svn {
location(svnUrl) {
credentials(SVN_CREDENTIALS_ID)
}
}
...
steps {
maven {
goals('clean deploy')
mavenOpts('-Dserver.username=$username')
}
}

GoCD, Travis CI, Shippable
 Absence of the “full-fledged” pipeline support in
Jenkins led multiple teams to switch to alternative
products like GoCD, Travis CI, Shippable, etc.

Jenkins Pipeline
 This led Cloudbees to start working on Jenkins
Pipeline Project (Originally called Jenkins Workflow)
 Development started – May 2014
 Release 1.0 – November 2014
 Renamed to Jenkins Pipeline – January 2016
 Jenkins 2 (weekly: 2.0) – April 2016
 Jenkins 2 (LTS: 2.7.1) – July 2016

Jenkins Pipeline – New Project Dialog
 To create a new Pipeline using Jenkins 2 select
New Job -> Type: Pipeline

Jenkins Pipeline Project Configuration
 Then instead of configuring dozens of combo- and
checkboxes just configure your project SCM (Git,
SVN, etc.)

Pipeline as Code
node('java-build') {
checkout scm
sh 'mvn -B clean deploy'
}
 Definition of your build / pipeline will be stored
together with your code as a Jenkinsfile

Simple Maven Project
stage 'Compile'
checkout scm
sh 'mvn -B clean deploy'
junit testResults: 'build/test-results/*.xml'
}
stage 'Automated tests'
node('qa-environment') {
…
See step definitions
on the next slide

Jenkins Job DLS Steps
 Jenkinsfile describe pipeline using Jenkins Pipeline DSL
 Jenkins Pipeline DSL consist of steps like:
 stage: starts a new pipeline stage (see next slide)
 node: executes steps on an agent (node) with the
corresponding name / label
 checkout: checkouts code from SCM (e.g. Git or SVN)
 sh: executes command line or shell script
 bat: executes Windows command line or script
 junit: publishes JUnit test results

Stage View UI
 Each pipeline consists of stages, e.g. Compile Stage, QA
Stage, Release Stage, etc.
 Each stage has one or multiple steps

Pipeline DSL: Working with Files
//Write string to file
writeFile file: 'status.log', text: 'Status: OK',
encoding: 'UTF-8'
//Example: read XML file to string and then
//get artifact version using regex
pom = readFile('pom.xml')
matcher = (pom =~ '<version>(.+)</version>')
version = matcher ? matcher[0][1] : null
//Read file to string
results = readFile file: 'results.log‚
encoding: 'UTF-8

Pipeline DSL: Pass files between stages
//"Archive" an artifact file to be used later
stash name: 'artifact', includes: 'package-a.zip'
//Stash multiple files:
stash name: 'testResults',
includes: '**/test-results/*.xml'
//Get the artifact file e.g. on another node
unstash 'artifact'

Parallel Execution of Steps
parallel(
//Both deployments are executed in parallel
deployToEnv1: {
node('environment-1') {
deploy(…)
}
},
deployToEnv2: {
node(environment-2') {
deploy(…)
}
}
)
//Execution continues after the longer step finishes

Use existing plugins in pipeline
 It is possible to use existing Jenkins Plug-ins if they added
Pipeline support
 Majority of popular Jenkins plug-ins support Pipeline
already (see list here)
 E.g. Email-Ext plug-in:
emailext(
subject: "Build #$buildNumber, $status",
recipientProviders:'CulpritsRecipientProvider'
…
)

Docker Support
docker.image('.../java8-maven').inside {
checkout svn
sh 'mvn -B clean install'
}
 Jenkins Pipeline also provides Docker support
 See Orchestrating Workflows with Jenkins and Docker

Pipeline DSL API Reference
 See Pipeline DSL API Reference
 Or just use Snippet Generator in Jenkins

Manual steps in Pipeline
 Jenkins Pipeline supports manual steps, i.e. steps which
require user interaction, e.g. tester should choose which
environment to use for testing:

Manual steps – Confirmation dialog
//Clicking on Release will start the next phase and
//on Abort will abort the pipeline execution
input message: 'Release to production?', ok: 'Release'
 Confirmation dialogs are also supported

Multibranch Pipeline
 Jenkins can auto discover branches (Job type: Multibranch
Pipeline)
 If a new branch is created, a corresponding job will be
created for it. As soon as the branch is merged (deleted)
the job will be deleted as well.

 Reuse pipeline modules
 Use Scripts in other languages (e.g. Schell
script, Python, Ruby, etc.) if necessary
 How to test Pipeline scripts?
 New Jenkins GUI: Project Blue Ocean
Part 2: Lessons learned

 How it worked before pipeline
Our First Pipeline

 Delivery pipeline - an automated implementation of build,
deploy, test, and release process
 Automate the full process from Code Commit to Product
Release
Our First Pipeline

Pipeline Code Reuse
stage 'Compile'
checkout scm
bat 'mvn -B clean deploy‘
}
… 200 more lines …
 If the whole pipeline will be described in a single
Jenkinsfile script, it will be too long and its maintenance
will be too expensive

Pipeline Code Reuse
 To solve the problem use full Groovy potential (see the
following slides for examples) :
1. Extract methods to avoid copy-pasting and achieve code
reuse
2. Organize code in modules by extracting methods into
separate files
3. Reuse pipeline modules between multiple pipelines by
storing them into a separate VCS repository (e.g. in Git
or Subversion)

1. Extract Methods
void runPhase(String phase){ //extract code as method
checkout scm
bat "mvn -B $phase" //phase is a variable
}
}
…
stage 'Compile'
runPhase('deploy') //just call the extracted method

2. Extract Modules
In Jenkinsfile load and reuse the extracted modules:
def maven = load 'maven.groovy'
stage 'Compile'
maven.runPhase('deploy') //method runPhase() from file
//maven.groovy will be called
Copy methods to separate files, e.g. maven.groovy:
void runPhase(String phase){
…
}

3. Reuse Shared Modules
Jenkinsfile:
//checkout shared modules from VCS first
checkout 'http://svn/main/pipeline-framework/trunk'
//then just load modules as before
def maven = load 'maven.groovy'
stage 'Compile'
maven.runPhase('deploy')
Check-in all modules (like maven.groovy,
deployment.groovy, etc.) to VCS and then
reuse them in all pipelines:

VCS vs. Jenkins Global Library
1. You can specify which version of shared modules to use,
e.g. trunk/master, or stable branch, etc.:
checkout 'http://svn/main/pipeline/trunk'
checkout 'http://svn/main/pipeline/tags/1.0.0.2'
checkout 'http://svn/main/pipeline/branches/stable'
2. As an alternative to VCS, add modules to the
Jenkins Global Library (currently does not support versioning):
ssh://user@jenkins:3421/workflowLibs.git

Reuse Existing non-Groovy scripts
 If you prefer to use a language other than Groovy (e.g.
Python, Ruby, Shell script, etc.) or already have multiple
non-Groovy scripts which implement pipeline step:
 Implement in Groovy just the Pipeline orchestration
logic (e.g. decisions, parallel execution, etc.)
 Implement steps using your preferred language
 Call these scripts from Pipeline using sh (Linux) or bat
(Windows) DSL steps (see the next slide for an
example)

Example: Reusing PowerShell Scripts
2. Call the script from Jenkins Pipeline:
bat 'copy2share.ps source destination username password'
1. PowerSchell script copy2share.ps (or shell, python, ruby, etc.):
Map-Networkdrive $drive $destination $username $password
Copy-Item $source $destination
 We decided to reuse a PowerShell script which copies
artifacts to shared folders

How to Test Pipeline
 If Pipeline becomes complex and has multiple steps, then
testing it become time consuming as well
 Currently Jenkins Pipeline does not have any support for
testing or Dry-Run mode (see feature request)
 To solve the problem we implemented a simple unit-testing
module (testing.groovy) for the pipeline using xUnit
principles
 Methods like assertEquals() or assertTrue() were added. If
assertion fails, an exception is thrown (see the next slide)

Assert Methods in testing.groovy
void assertEquals(expected, actual, msg = null){
if (actual == expected) {
return
} else {
throwException(expected, actual, msg)
}
}
void assertTrue(actual, msg = null)
void assertContains(subString, string, msg = null)
…

Test Example
testing.test("Get Maven artifact coordinates") {
given: //just a label to structure code (see also Spock)
def pomXml = '''
<groupId>de.medavis</groupId>
<artifactId>project-a</artifactId>
<version>1.0.0.1</version>
'''
when:
def coordinates = maven.getCoordinates(pomXml)
then:
testing.assertEquals('de.medavis', coordinates.groupId)
testing.assertEquals('project-a', coordinates.artifactId)
testing.assertEquals('1.0.0.1', coordinates.version)
}
 Sample test of maven.getCoordinates() function:

How to Test Pipeline
 We also implemented a simple test reporting: if an
exception was thrown in test, it is marked as failed:

New UI –Blue Ocean
 Jenkins will soon get a new UI optimized for pipelines:
Blue Ocean – see the following slides
 It can be installed from the experimental update center:
http://updates.jenkins-ci.org/experimental/update-
center.json

Blue Ocean: Parallel Execution

Blue Ocean: Console Output

Delivery Pipeline as Code: using Jenkins 2.0 Pipeline

More Related Content

What's hot

Similar to Delivery Pipeline as Code: using Jenkins 2.0 Pipeline

Recently uploaded

Delivery Pipeline as Code: using Jenkins 2.0 Pipeline