State of the Jenkins automation ; presented at Open Source Datacenter Conference in Berlin in 2017.

1. Jenkins Automation
Julien Pivotto (@roidelapluie)
Open Source Data Center Conference
Berlin

2. whoami
Julien "roidelapluie" Pivotto
@roidelapluie
Sysadmin at inuits
Automation, monitoring, HA
Jenkins user for 5+ years

3. inuits

4. Jenkins in 2 lines
Open Source
Pluggable

5. The Jenkins Project
Java
Hudson (2005)
Jenkins (2011)

6. The evolution of Jenkins
Jenkins is moving fast (there is competition:
travis-ci, gitlab-runners)
Far away from just the java world (e.g. mvn
specifics are gone in Pipeline world)

7. What can you do with Jenkins?
Testing, building, deploying
Software
Services
Infrastructure

8. Mission Critical
Public Domain https://commons.wikimedia.org/wiki/File:Roaddamage59quake.JPG

9. Without Jenkins...
How do I build this?
Where do I deploy this?
Who knows where to upload this?
When did we deploy this?

10. Automating Jenkins
Creative Commons Attribution-ShareAlike 2.0 https://www.flickr.com/photos/machu/3131057286

11. Automating Jenkins
Why is it needed?
Make build processes transparent
Make build processes improvable
Make build process reproducible
Lower the barrier to update Jenkins
security - audit - bugfixes - plugins

12. Automating Jenkins
Why is it hard?
Building Software is hard
For long, Jenkins has been strongly UI-Driven
It is "easy" to deploy (.war file)
Hey I've deployed Jenkins on my Laptop!
It has lots of plugins (and you need them)

13. XML everywhere!
Creative Commons Attribution 2.0 https://www.flickr.com/photos/generated/6035308729

14. What can you automate in
Jenkins?

15. 1. The Operating System Jenkins is
running on
The platform/node/host/.. which runs your
Jenkins damon

16. 2. The Jenkins Service
The start/stop/reload of Jenkins
JAVA_OPTIONS

17. 3. Plugins
Which plugins to install
Which version
Which configuration

18. 4. Jenkins Global Configuration
SMTP Server
Email address
Authentication/Authorization

19. 5. Jobs - Views - Folder
Views - That really help people
Jobs and folders - Creation AND deletion,

20. 6. Jobs definition
Jobs content
Build steps
Deployments configuration

21. 7. Operating system you build your
software on
Jenkins nodes
Where your jobs run
(Linux, Android, ...)

22. Automating the OS
Creative Commons Attribution-Share Alike 2.0 https://www.flickr.com/photos/timdorr/168925884

23. Jenkins deserves attention
Jenkins Server should be automated
Receive OS updates
Be monitored like any other server

24. Automating the service
CC0 Public Domain https://pixabay.com/en/butler-tray-beverages-wine-964007/

25. Installing Jenkins
Do NOT use war file (yes I know it's easy)
Use Jenkins LTS (3 months lifecycle vs 1
week)
Think about: user, directories, backups,...
Packages provides: upgrade path, downgrade
path, control, checksums, signatures, ...

26. Automating the Jenkins Service
Chef: Jenkins in the supermarket
Puppet: module rtyler/jenkins
Playbooks/etc for other tools as well
Docker

27. Jenkins Plugins
You NEED them (never seen a Jenkins setup
without 20-100 plugins)
Fetched from https://updates.jenkins-ci.org
Can be installed from the UI :(
Can be installed from the CLI

28. Packaging Jenkins Plugins
Plugins have dependencies (against plugins &
Jenkins core)
They have a fixed download path
They are listed in updates.jenkins.io/update-
center.json
https://github.com/roidelapluie/Jenkins-
Plugins-RPM

29. Mirroring Jenkins Plugins
Mirror http://updates.jenkins-ci.org
By default, "latest" will be fetched
Don't cache too much
github.com/jenkinsci/docker install-
plugins.sh

30. Global configuration
CC0 Public Domain https://commons.wikimedia.org/wiki/File:Waiting_room_bell.jpg

31. Modifying XML files
DON'T

32. system-config-dsl-plugin
Like Job DSL, but for the system
https://github.com/jenkinsci/system-config-
dsl-plugin
JENKINS-31094
Downside: it does not exist yet

33. Creative Commons Attribution-Share Alike 3.0
https://commons.wikimedia.org/wiki/File:Groovy-logo.svg

34. Groovy
Yet another language to learn? yes.
Programming language for the Java platform
Scripting language
Fully integrated in Jenkins
Used by automation tools (chef cookbook,
puppet module...)

35. The Jenkins Script Console
A groovy console is available at /script
http://jenkins.example.com/script
also with curl
Requires "Overall/Run Scripts" permission

36. Sample Groovy scripts
Creative Commons Attribution-Share Alike 2.0
https://www.flickr.com/photos/bjornb/88101376

37. Set number of executors
import jenkins.model.Jenkins;
Jenkins.instance.setNumExecutors(0)

38. Set HTML formatter
import jenkins.model.Jenkins;
import hudson.markup.RawHtmlMarkupFormatter;
Jenkins.instance.setMarkupFormatter(
new RawHtmlMarkupFormatter(false));

39. Set system message
import jenkins.model.Jenkins;
Jenkins.instance.setSystemMessage("""
Welcome to <em>Jenkins</em>.
""");

40. Configure simple-theme
import jenkins.model.Jenkins;
def simpleThemePlugin =
Jenkins.instance.getDescriptor(
"org.codefirst.SimpleThemeDecorator")
simpleThemePlugin.cssUrl =
"/userContent/example.css"
simpleThemePlugin.save()
Everything in $JENKINS_HOME/userContent
is served under /userContent

41. Email
import jenkins.model.Jenkins
def desc = Jenkins.instance.getDescriptor(
"hudson.tasks.Mailer")
desc.setSmtpHost("172.17.0.1")
desc.save()

42. Disable usage statistics
import hudson.model.UsageStatistics;
hudson.model.UsageStatistics.DISABLED = true;

43. But also...
Setup Authorization/Authentication
Setup prefix url
Setup slaves, clouds
Setup global libraries
Setup credentials
Setup first jobs

44. Bonus
import org.jenkinsci.plugins.pipeline.utility
.steps.shaded.org.yaml.snakeyaml.Yaml
Yaml reader = new Yaml()
Map config = reader.load(text)
Thanks to pipeline utility step (readYaml()
step), you can easily read yaml files

45. Scripts sources
https://wiki.jenkins-ci.org/display/JENKINS/Jenkins+Script+Console
https://github.com/jenkinsci/jenkins-scripts/tree/master/scriptler
https://github.com/infOpen/ansible-role-jenkins/tree/master/files/groovy_scripts
https://github.com/samrocketman/jenkins-bootstrap-jervis/tree/master/scripts
https://github.com/jenkinsci/puppet-
jenkins/blob/master/files/puppet_helper.groovy
http://pghalliday.com/jenkins/groovy/sonar/chef/configuration/management/2014
/09/21/some-useful-jenkins-groovy-scripts.html

46. Jenkins Javadocs
https://jenkins.io/doc/developer/guides/
http://javadoc.jenkins.io/archive/jenkins-
2.32/
http://javadoc.jenkins.io/
http://javadoc.jenkins.io/plugin/gerrit-trigger/

47. Now what?
Creative Commons Attribution 2.0 https://www.flickr.com/photos/51029297@N00/5275403364

48. Jenkins init.groovy.d
Upon startup, Jenkins will run
$JENKINS_HOME/init.groovy.d/*.groovy
scripts
Meanwhile, "Jenkins is getting ready..."
message is displayed
Drop files, restart Jenkins
Allows you to preconfigure everything --
without the GUI

49. init.groovy.d directory
behaviour
Scripts executed sequentally (alphanum
order)
Scripts that throws exceptions make startup
fail

50. Inside Jenkins
Creative Commons Attribution-ShareAlike 4.0 International
https://commons.wikimedia.org/wiki/File:Mystery_Cave_passage.jpg

51. The Multiple Approaches
GUI .. but this talk is about automation, right?
init.groovy.d: to create your seed job
Jenkins Job Builder: declarative, python, yaml
Jenkins Job DSL: imperative, groovy

52. Jenkins Job Builder
An Openstack Project
Python (not a Jenkins Plugin!)
Support templates
Extensible
Can do raw xml
Limited support for plugins and pipeline
Put Jobs config under SCM

53. Jenkins Job DSL
A Jenkins Plugin
2012
Groovy DSL to create views & jobs
Put Jobs config under SCM

54. init.groovy.d
def jobManagement = new JenkinsJobManagement(
System.out, [:], new File('.'))
new DslScriptLoader(jobManagement).runScript("""
folder('jenkins') {
displayName('Jenkins')
}
pipelineJob("jenkins/seed"){
definition {
cpsScm {
scm {
git {
remote {
credentials('jenkins­git­na')
url('git.example.com/seed.git')
}
branches('master')
}
}
scriptPath('Jenkinsfile')
}
}
}
""")

55. Creative Commons Attribution-Share Alike 3.0
https://commons.wikimedia.org/wiki/File:Groovy-logo.svg

56. Job DSL support
Large community of users
Lots of plugins supported

57. Create a job
job('test') {
scm {
git('git://example.com/foo.git' 'master')
}
steps {
shell('make test')
}
}

58. Create a Pipeline job
pipelineJob('test'){
definition {
cps {
script(buildScript)
}
}
}

59. Set job properties
pipelineJob('test'){
description('Test Build')
parameters {
booleanParam('verbose', false, 'Be Verbose')
}
logRotator {
numToKeep(10)
}
}

60. Loops
config.each(){ jobConfig ­>
pipelineJob(jobConfig.name) {
triggers {
if (jobConfig.triggers?.scm) {
scm(jobConfig.triggers.scm)
}
}
}
}

61. Create a view
listView('Project A') {
recurse()
jobs {
regex('myprj/[^/]+')
}
columns {
status()
weather()
name()
lastSuccess()
lastFailure()
lastDuration()
lastBuildConsole()
buildButton()
progressBar()
}
}

62. Work with plugins
buildMonitorView("OnScreen Status") {
jobs {
name('WatchA')
}
}
${JENKINS_URL}/plugin/job-dsl/api-viewer/

63. In Pipeline
jobDsl removedJobAction: 'DELETE',
removedViewAction: 'DELETE',
targets: 'seeds/*.groovy'

64. Unleash the power of groovy
import jenkins.model.Jenkins;
dslCfg = Jenkins.instance.getDescriptor(
"javaposse.jobdsl.plugin.
GlobalJobDslSecurityConfiguration")
dslCfg.useScriptSecurity = false
==
(sec issue with old versions of the plugin)
(or power user trick)

65. Load extra libraries
jobDsl additionalClasspath: 'src/yaml.jar',
removedJobAction: 'DELETE',
removedViewAction: 'DELETE',
targets: 'seeds/*.groovy'

66. Read yaml
import org.yaml.snakeyaml.Yaml
Yaml reader = new Yaml()
Map config = reader.load(
readFileFromWorkspace('cfg.yaml')
Drop the snakeyaml jar file and add it to
classpath

67. Jenkins Pipeline
Creative Commons Attribution 2.0 https://www.flickr.com/photos/amerune/9294639633

68. Jenkins Pipeline
Jenkins Jobs as Code
"Jenkins file"
Imperative (aka scripted) Pipeline
Declarative Pipeline (2017)

69. What is a Jenkinsfile (aka
Pipeline)
A file that contains the definition of a job
No need of Gui
Defines Steps, Reports, Environments,
Nodes,...
Plugins can provide steps
Generic "step"

70. How to write Pipelines?
Visual Pipeline editor (WIP)
"Pipeline Syntax" link in jobs

71. Scriped pipeline
node ("Ubuntu && amd64") {
stage('checkout') {
checkout scm
}
stage('build') {
sh 'make'
}
stage('test') {
sh 'make test'
}
}

72. Real World Scripted Pipeline
node ("Ubuntu && amd64") {
checkout scm
stage('build') {
try {
sh 'make'
} catch(err) {
currentBuild.result = 'UNSTABLE'
publishArtifacts('err.log')
throw err
}
}
stage('test') {
sh 'make test'
}
step([$class: 'JavadocArchiver',
javadocDir: "target/site", keepAll: true])
}

73. Declarative Pipeline (1/2)
pipeline {
agent {
label("Ubuntu && amd64")
}
options {
timeout(time: 235, unit: 'MINUTES')
timestamps()
ansiColor('xterm')
}
stages {
stage('build') {
steps {
sh('make')
}
}
}

74. Declarative Pipeline (2/2)
pipeline {
post {
always {
junit params.jUnitReports
}
}
}

75. Declarative vs scripted
Declarative checkout code by default
Declarative get a workspace "OOTB"
Declarative enforces everything in stages
Declarative allow Pipeline-wide wrappers (e.g
ansiColor, timestamps)

76. Going further with Pipeline
Global Libraries Plugins
Job DSL Plugin

77. Jenkins Nodes
CC0 Public Domain https://www.flickr.com/photos/133259498@N05/27077266322/

78. Docker Docker Docker
Run jobs inside containers
Clean, short lived containers
Easy to update
Docker Plugin

79. Docker nodes pattern
Build Container
Tag it with a tag "candidate"
Push it to your registry
Run normal testing
Run actual builds with that "candidate"
If success -> tag with "release" && push

80. In practice
Docker Plugin config automated with groovy
Candidate and Release tags are setup as
slaves
They get two labels: "image" "tag" (e.g. "build-
centos-7" "candidate")
Jobs get a parameter "tag"

81. In the build Jenkinsfile
pipeline {
agent {
label("build­centos­7 && ${params.tag}")
}
}

82. In the docker-build Jenkinsfile
dockerImage = docker.build("build­centos­7",
"­­no­cache")
dockerImage.tag('candidate')
docker.withRegistry(url, credentials) {
dockerImage.push('candidate')
}
node("build­centos­7 && candidate") {
sh('test­script')
}
build job: 'myjob', parameters:
[string(name:'tag', value:'candidate')]
dockerImage.tag('release')
docker.withRegistry(url, credentials) {
dockerImage.push('release')
}

83. Pros/Cons
Updated containers won't block the builds (e.g
on packages updates)
Containers stay up to date
Don't forget that builds release artifacts
(sometime you don't want that in nodes tests)

84. Docker Moby Docker

85. Jenkins Master in Docker
WHY WOULD YOU DO THAT?
Atomic upgrade of all plugins at the same
time
Easy rollback / update
Easy to test
Reduce the cost to upgrade

86. Jenkins master in Docker
At which price?????
Do not run ANYTHING on master; slave
everything
Keep a Docker Registry with history
Assume Docker instability
Think docker deployment etc... (A pipeline for
your Jenkins master - how cool is this?)

87. Docker and Jenkins
Jenkins upstream images: alpine/ubuntu
LTS and weekly

88. Dockerfile
FROM jenkins:2.46.2­alpine
MAINTAINER dev­team@example.com
USER root
COPY jenkins/userContent/
/var/lib/jenkins/userContent
RUN mkdir /var/log/jenkins &&
mkdir /var/lib/jenkins &&
mkdir /var/cache/jenkins
chown ­R jenkins: /var/log/jenkins &&
chown ­R jenkins: /var/lib/jenkins &&
chown ­R jenkins: /var/cache/jenkins

89. Timezone (if needed)
yay alpine
RUN apk add ­­no­cache tzdata &&
cp /usr/share/zoneinfo/Europe/Brussels
/etc/localtime &&
echo "Europe/Brussels" > /etc/timezone &&
apk del tzdata

90. Install plugins
USER jenkins
COPY jenkins/plugins.txt /tmp/plugins.txt
RUN cat /tmp/plugins.txt | xargs
/usr/local/bin/install­plugins.sh
/usr/local/bin/install-plugins.sh is provided in the
upstream docker image
plugins.txt
cloudbees­folder
cucumber­reports:4.3.2
docker

91. init.groovy.d
COPY jenkins/init.groovy.d/
/usr/share/jenkins/ref/init.groovy.d/
Note: if you keep Jenkins config in a volume, you
need to suffix your files .override to update them

92. JAVA_OPTS
ENV JAVA_OPTS="­Xmx8192m
­Djenkins.install.runSetupWizard=false"
ENV JENKINS_OPTS="­­handlerCountMax=300
­­logfile=/var/log/jenkins/jenkins.log
­­webroot=/var/cache/jenkins/war"

93. Testing the image
Jenkins Pipeline
init.groovy.d switch:
if (System.getenv()['DO_NOT_BUILD']){
Jenkins.instance.doQuietDown()
}

94. Block until started
That script will fail until Jenkins is started:
#!/bin/bash ­xe
curl 127.0.0.1:8080/jenkins2/ |
grep 'All [Jenkins]' ­q ||
grep "WARNING: Failed to run script file"
/var/log/jenkins/jenkins.log ||
grep "SEVERE: Failed GroovyInitScript.init"
/var/log/jenkins/jenkins.log
retry(30) {
sleep 10
sh("""docker exec ${dockerContainer.id}
bash ­xe /tmp/block­until­started.sh""")
}

95. Testing init.groovy.d
set ­xe
if grep "SEVERE: Failed GroovyInitScript"
/var/log/jenkins/jenkins.log ||
grep "WARNING: Failed to run script file"
/var/log/jenkins/jenkins.log
then
sleep 600 # Let some time to investigate
echo "FOUNDÂ ERRORSÂ INÂ THEÂ LOGS"
false
fi

96. Seed jobs
Somewhere in init.groovy.d
cause = new RemoteCause('localhost',
'Build triggered by init.groovy.d')
job = Jenkins.instance.getItem("seed")
cAction = new CauseAction(cause)
build = job.scheduleBuild2(0, cAction)
res = build.get().getResult().toString()
assert res == 'SUCCESS'
Last lines check for completion of seed job

97. Benefits
A Pipeline to Deploy Jenkins
Automated checks of init.groovy.d etc
Check that the seed job works

98. Going further
Will the fun ever stop?

99. Trash the Jenkins config
Only keep build history
Do notput config in persistent volume
Let init.groovy.d/job dsl build your config only
from code
Still, keep job history

100. The glue
import jenkins.model.Jenkins;
import static groovy.io.FileType.FILES
def rawBuildsBaseDir = '/srv/jenkins/state'
Jenkins.instance.setRawBuildsDir(
"${rawBuildsBaseDir}/${ITEM_FULL_NAME}")
/srv/jenkins/state is a persistent volume
it will keep build history

102. Conclusion
Public Domain https://commons.wikimedia.org/wiki/File:YellowOnions.jpg

103. Jenkins can be FULLY
automated
My recommendations:
init.groovy.d
Jenkins Job DSL
Pipeline
Try dockerized Jenkins ; might work for you
(spoiler: works for us)

104. Julien Pivotto
roidelapluie
roidelapluie@inuits.eu
Inuits
https://inuits.eu
info@inuits.eu
Contact