Christian Lipphardt and Sebastian Menski talk about do and don't and giving good advices when rebuilding your CI infrastructure.

1. www.camunda.comtwitter.com/camundaBPM
18.09.15
Christian Lipphardt
Sebastian Menski
BerlinExpertDays2015
From Virtual Machines to Containers

2. www.camunda.com
Agenda
• Introduction
• The Dark Age
• The Promising Present
• Lessons Learned
• The Bright Future
-2-

3. www.camunda.com
Introduction - About Camunda
-3-
2014201320122011201020092008
Incorporation
Camunda Services GmbH
Berlin, Germany
Incorporation
Camunda Inc.
San Francisco, US
BPM Consulting BPM Software Vendor
● 30 Fulltime Employees ● No external funding ● Avg. 50% Turnover Growth per Year
Introduced
Hudson
Adapted
Jenkins

4. www.camunda.com
Introduction - Camunda BPM
-4-

5. www.camunda.com
Introduction - What is Camunda BPM
• Camunda BPM is an open source platform
for workflow and business process automation
• Integrates with:
• 7 Application Server (11 different versions)
• 6 Databases (17 different versions)
• 1 Development & 4 Maintained Versions
-5-

6. www.camunda.com
Introduction - Why CI is Important
• Every Camunda BPM version is tested against:
• 187 combinations of DBs and App Servers
• 11 JDKs
• ~ 400 Jobs per version
• Bi-Annual release of a new Camunda BPM version
• Support for Enterprise Customer (24/7 + Fix Time)
-6-

7. www.camunda.com
Why our CI had to change
The Dark Age
-7-

8. www.camunda.com
The Dark Age - The Numbers
• 1 Jenkins Master with lots of plugins
• 8 Jenkins Slaves VMs
• ~1000 Jobs total configured /
manually managed:
• 4 Camunda Versions
• Community Projects
• Websites
• Maintenance
-8-

9. www.camunda.com
The Dark Age - Isolation Problem
• Unit and Integration Tests need a database/application
server
• Only 1 Instance per Database
• All Jobs use the same Databases
• Every half-year a new Version (~400 Jobs) using same
Databases
-9-

10. www.camunda.com
The Dark Age - No Scalability
• Total Executors: 12
• 1 Jenkins -> 4 Executors
• 8 static heterogenous slave VMs, each with 1 Executor
• Jobs tied to slaves through labels
• Slaves “restrict” database access by allowing no other build
to run
-10-

11. www.camunda.com
The Dark Age - Maintenance Problems
• Upgrading Jenkins or any plugin
• Supporting a new Database vendor / version
• Supporting a new App Server version
• Creating jobs for new Camunda BPM version
• Disaster recovery
-11-

12. www.camunda.com
The Dark Age - The Other Problems
• Slow feedback cycle for developers
• Developers cannot reproduce CI environments
• QA engineers use and maintain separate bloated test build
setup
-12-

13. www.camunda.com
So …
-13-

14. www.camunda.com
How we solved our Problem
The Promising Present
-14-

15. www.camunda.com
The Present - What we achieved
• Configuration & Infrastructure as Code
• Isolated and Reproducible Jobs
• Scalable CI Infrastructure
-15-

16. www.camunda.com
The Present - Infrastructure as Code
1. Every configuration is checked into SCM
2. Every application/test runs in a Docker Container
3. Every Docker image is build automatically
-16-

17. www.camunda.com
The Present - Infrastructure as Code
1. Every Configuration is checked into SCM
• Docker for
• Applications
• Test Environments
• JobDSL for
• Jenkins Jobs
-17-

18. www.camunda.com
The Present - Example - JobDSL
package ci.community
import generator.JobGenerator
import util.*
Server.COMMUNITY().each { server ->
new JobGenerator(
name: "${Version.CURRENT}-webapp-IT-${server}",
pollScmSchedule: null,
pollRemoteJob: RemoteJobHelper.WEBAPP_DISTRO,
rootPom: 'qa/pom.xml',
goals: "clean verify -P${server},h2,webapps-integration",
envVariables: ['DISPLAY': ':0'],
jdkVersion: Tool.JDK_FOR_SERVER(server),
label: Label.CHROME
).build(this)
}
-18-

19. www.camunda.com
The Present - Infrastructure as Code
2. Every application/test runs in a Docker Container
Images:
• Application (Jenkins, Nexus …)
• Test Env. Images (DB + SSH)
• Build Env. Images
• DIND, QEMU + Packer.io
-19-

20. www.camunda.com
The Present - Example - DB2
FROM ci1.camunda.loc:5000/camunda-ci-base-centos
ENV DB2_VERSION=10.5 DB_USERNAME=camunda DB_PASSWORD=camunda DB_NAME=engine DB2_HOME=/opt/ibm/db2/V10.5
DB_DIR=/db2_databases
RUN save-env.sh DB2_VERSION DB_USERNAME DB_PASSWORD DB_NAME DB2_HOME
RUN install-packages.sh libaio
ADD etc/db2/db2expc.rsp /tmp/db2expc.rsp
RUN curl https://nginx.service.consul/ci/binaries/ibm/db2/v10.5_linuxx64_expc.tar.gz > /tmp/db2.tar.gz &&
mkdir -p /tmp/db2 && tar xzf /tmp/db2.tar.gz -C /tmp/db2 --strip 1 &&
cd /tmp/db2 && ./db2setup -r /tmp/db2expc.rsp && rm -rf /tmp/db2.tar.gz /tmp/db2 &&
mkdir ${DB_DIR} && chown db2inst1:db2iadm1 ${DB_DIR} && chmod 775 /${DB_DIR} &&
su -l db2inst1 -c "mkdir ${DB_DIR}/engine"
ADD bin/* /usr/local/bin/
ADD etc/supervisor.d/db2.conf.ini /etc/supervisord.d/
EXPOSE 50000
-20-

21. www.camunda.com
The Present - Infrastructure as Code
3. Every Docker Container is build automatically
• Own Jenkins for Docker/KVM Images
• KVM Images build in Docker Container
with Packer + QEMU
• KVM Images bundled in Docker Image
-21-

22. www.camunda.com
The Present - Example - SQL Server 2012
FROM ci1.camunda.loc:5000/ubuntu:14.04.2
ENV IMAGE_NAME=sqlserver-2012.qcow2 IMAGE_DIR=/qemu/ RUN_DIR=/qemu/run
RUN mkdir -p $IMAGE_DIR $RUN_DIR
VOLUME $RUN_DIR
ADD bin/* /usr/local/bin/
RUN apt-get update &&
apt-get -y install --no-install-recommends curl qemu-system-x86 qemu-utils &&
apt-get clean && rm -rf /var/cache/* /var/lib/apt/lists/* /tmp/* /var/tmp/*
RUN curl https://nginx.service.consul/ci/binaries/microsoft/${IMAGE_NAME} > ${IMAGE_DIR}
/${IMAGE_NAME}
EXPOSE 22 1433 5900
CMD ["/usr/local/bin/start-qemu.sh"]
-22-

23. www.camunda.com
The Present - The Current Flow
-23-
camunda-ci
camunda
Camunda BPM
Platform
Infrastructure
Jenkins
CI Jenkins

24. www.camunda.com
The Present - Isolation
One Jenkins per Concern:
• CI
• Release
• Infrastructure
• Community and other Projects
• Marketing
-24-

25. www.camunda.com
The Present - Isolation & Reproducibility
• Every Jobs runs in an One-Shot Docker Container
• No Interference between Jobs
• The Database Settings are well documented
• Every Docker Image is stored in a private registry
• Developers/QA can use the Docker Images for local testing
-25-

26. www.camunda.com
The Present - Scalability
• Jenkins uses Docker-Plugin with one Docker Cloud running
on Docker Swarm
• Docker images are added through Groovy scripting
• Running on Commodity Hardware
• 3 Infrastructure Hosts (Jenkins, Nexus, …)
• 4 Docker Hosts as 1 Swarm
-26-

27. www.camunda.com
The Present - Example - Jenkins Docker Cloud
images.each { imageName, imageValues ->
imageValues.versions.each { version ->
def templateBase = createLinuxTemplateBase(image, privileged)
def image = "${registryUrl}/${imageName}:${version}".toString()
def label = getLabel(imageValues.label, version)
def privileged = imageValues.privileged? imageValues.privileged : false
def remoteFs = '/home/camunda'
def instanceCap = linuxInstanceCap
def template = new DockerTemplate(templateBase, label, remoteFs, remoteFs,
instanceCap, Node.Mode.EXCLUSIVE, 1,
createDockerSshLauncher(),
createRetentionStrategy(), true,
DockerImagePullStrategy.PULL_ALWAYS)
dockerTemplates << template
}
}
-27-
/var/lib/jenkins/init.groovy.d/dockercloud.groovy

28. www.camunda.com
The Present - Scalability
-28-

29. www.camunda.com
The Present - CI Cluster
-29-
Docker
Consul
Swarm
Nexus
Registry
ci1
Docker
Docker
Docker
Consul
Swarm
4x
Jenkins
ci2
Docker
Docker
Consul
Swarm
Nginx
ci3
Docker
ci4-7
BIND
Infrastructure Build Slaves
C-Agent
S-Agent

30. www.camunda.com-30-
Jenkins Demo

31. www.camunda.com
The Present - Advantages
• Easy to add new Databases/Test Environments
• New Release = New Branch of JobDSL Repository
• Fully parallelized Job Execution
• Accountable Configuration History
• Testable Infrastructure
• Minimize Administration Overhead
-31-

32. www.camunda.com
The Present - Conclusion
• 2 People + 3 Months of Work
• A fully scalable, isolated and reproducible CI Infrastructure
• Faster Feedback
• Happy Developers and Product Owner
-32-

33. www.camunda.com
Some advice from us to you
Lessons learned
-33-

34. www.camunda.com
Lessons learned - Architecture
• Automate as much as you can
• Jenkins config
• Jobs config
• Environment creation
• Design to scale to support the business agility
-34-

35. www.camunda.com
Lessons learned - Testing
Test everything:
• Jenkins configuration
• required plugin features
• Job generation
• Docker images
• Scalability
• Disaster recovery
-35-

36. www.camunda.com
Lessons learned - Job DSL
• Unit-test the job generation
• Write JobGenerator classes to abstract the common build
logic of most jobs out of the box
• Use XML diffing to compare previously generated jobs with
new ones
-36-

37. www.camunda.com
Lessons learned - Docker
-37-
https://github.com/rancherio/vm/

38. www.camunda.com
Lessons learned - Plugins
• Pin your plugin versions
• Be prepared to contribute to plugin development or
maintain a branch yourself
• Choose the right plugin for the job
Our Top 3 plugins:
JobDSL, Docker-Plugin, Build-Failure-Analyzer
-38-

39. www.camunda.com
Lessons learned - Control
• Control as much as possible
• Third party binaries vs package manager
• explicit versions
• own mirrors for important packages
-39-

40. www.camunda.com
What comes next
The Bright Future
-40-

41. www.camunda.com
The Future
-41-
• Public Community Jenkins
• Internal Webapp for Developers and QA to start
Environments (Dattln)
• Continuous Deployment for Infrastructure Container
• Back to the Datacenter
• Centralized Logging and Monitoring (ELK)
• Secret management

42. www.camunda.com-42-
Logstash Plugin

43. www.camunda.com-43-

44. www.camunda.com-44-

45. www.camunda.com
Resources
• Camunda CI repository
https://github.com/camunda-ci
• Job DSL gradle example
https://github.com/sheehan/job-dsl-gradle-example
• Camunda BPM Docker image
https://registry.hub.docker.com/u/camunda/camunda-bpm-platform/
-45-

46. www.camunda.com
We Are Hiring
Come Join Us
@camundaBPM
http://camunda.com/jobs
-46-