How to plan and define your CI-CD pipeline

How to plan and define
your CI/CD pipelines
Francisco (aka Patxi) Gortázar
francisco.gortazar@urjc.es
@fgortazar
Funded by the
European Union

@fgortazar
Bio
Project Coordinator at @elastestio EU project
Devops @ Kurento/OpenVidu
TeachingTesting & Distributed Systems @ URJC
@fgortazar
https://es.linkedin.com/in/franciscogortazar

@fgortazar
How to plan and design your pipelines
● Some preliminary concepts
● Background
● Planning
● CI/CD environment
● Git workflow
● Pipelines
● Code review
● Merging
● Releasing
● Deploying

@fgortazar
Some preliminary concepts
● Software is usually built by teams
● People within the team share the code as they build it
by means of source control management tools
SCM
Tools / Services offering git repositories

@fgortazar
● Each time someone shares its changes through the
repository, we should check that:
● The project is still compiling
● All the tests still pass (they’re green)
● Compiling and running tests on a trusted
environment is Continuous Integration’s task
● When a problem arises, it is notified immediately
to the team

● Integration must happen continuously (at least once
per day)
● Fast feedback is key, so that we know our changes
are working when integrated with the rest of the
system
● Problems detected in our CI environment must be
solved immediately
● Status of the project in terms of builds, test, and
deployments is visible at any moment

Background
● Some numbers
● 30 code repositories
● ~400 Jenkins jobs
● +1,000 tests
● +20 different environments to test
● +80 artifacts to be deployed at release time

Planning
● What’s your Git workflow
● Branches are to be integrated as well?
● Depend-on components are to be considered?
● Infrastructure planning
● Snapshot versions
● Release versions
● Deployments

@fgortazar
CI/CD environment
● Feasible infrastructures and their consequences
● Public cloud vs private cloud (on premises)
● Controlling public cloud costs (e.g. spot instances, stopping
instances not in use...)
● Virtual Machines vs Docker
● 20 environments = 20 differentVMs
● Cost increases with each new environment!!
● Effort increases with each new environment to configure
(ops)!!
● Time-to-market increases also!!

@fgortazar
CI/CD environment
● KurentoVMs for building
clients
● 1 node mvn & jdk 7 &
node 4
node 6
node 4
node 6
● KurentoVMs for building
servers
● 2 UbuntuTrustyVMs
● 4 Ubuntu XenialVMs
● 2 Ubuntu latestVMs

@fgortazar
CI/CD environment
● Developers are pushing hard towards devops to
include changes in infrastructure
● Changes can hardly be reverted (possible, but…)
● Hard to test locally
● Works in my machine effect
● Wasted resources & insufficient resources at the same
time
● Can we do better?

CI/CD environment
● Developers own the infrastructure on top of which
their tests will run
● Updates can be done safely
● CI environment much more stable
● Docker images can be updated quickly and easily
● Infrastructure testing much easier

@fgortazar
Git workflow
● When working together in a shared repository,
some rules apply as to how changes from the
different people involved in the team are
integrated, and which is the source of truth when
releasing software versions
● We’ll briefly see some common models of git
workflows that people is using
● When necessary, I’ll point you towards specificities
for some models regading the definition of your CI/
CD pipeline

@fgortazar
Git workflows
● Basic / Master development /Trunk development
● Everything happens in the master branch
● Each feature is developed in a local repository
● When features are not small enough some problems arise
● Members of the team might push commits to master for
uncompleted features
● Impossible to release a new version before all features has been
completed and integrated
● Most of the time master is broken, and unplanned releases are a
risk

@fgortazar
Git workflows
● Branch per feature
● Each features is developed in its own branch (branch’ed
from master)
● The feature is integrated only when complete
● Master is stable, and can be released at any moment
● The branch might be local or it can be shared thus
enabling CI for the features under development
● If so you need a multi-branch pipeline job in Jenkins jargon

@fgortazar
Git workflows
● Branch per feature with merge requests
● Similar to branch per feature, however there’s a
hierarchical permission model
● Some people can integrate directly in master
● Others must request a merge, that will be reviewed by
some members of the time prior to merge
● The CI server might help in integrating changes

@fgortazar
Git workflows
● Gitflow
● Probably the most complex one
● Useful in big projects, with many features and several
release versions to maintain
● There are two branches
● Master → production branch. Integration into master only
happens if acceptance tests pass. Ready to be deployed.
● Develop → integration branch. Features, developed in branches,
are integrated here first, then tests are run.
http://nvie.com/posts/a-successful-git-branching-model/

@fgortazar
Git workflows
● Fork workflow
● One of the most used in GitHub
● Developers fork the repository into their accounts
● This is called “forking” the repo
● Then they clone their own copy into their machines
● Changes are integrated into the developer’s repo first
● Changes are requested to be integrated into the main repo by
means of “pull-requests” (PRs)
● Someone with the necessary permissions accepts or rejects
the PR, probably with some aid from the CI server
https://github.com/elastest/elastest-platform-manager/pulls?q=is%3Apr+is%3Aclosed

@fgortazar
Pipelines
● What is a pipeline?
● Two meanings
● Set of jobs that are run in sequence
● In Jenkins a Pipeline job is a job that leverages the
Pipeline syntax based on the Groovy language
● There are two different syntax for Pipeline jobs
● Declarative Pipeline (DSL)
● Scripted Pipeline (Groovy)

@fgortazar
Code review jobs
● Needed only when doing code review
● Pull Requests / Merge Requests
● Run per change request before the commit is push to
master
● Fast: unit tests mainly
● Fix the commit before getting too deep into any
other task
● CI votes the change
● +1 can be accepted & integrated→
● -1 cannot be integrated, amendment needed→

@fgortazar
Committing
node {
try {
stage("Test") {
docker.image('maven').inside('-v $HOME/.m2:/root/.m2') {
git https://github.com/codeurjc/testing.git
sh "cd tema1_4_ejem4; mvn test"
}
}
} catch(e) {
throw e
} finally {
junit "tema1_4_ejem4/**/target/surefire-reports/TEST-*.xml"
}
}

@fgortazar
Merge jobs
● Run once a something has been integrated into the
master branch
● Usually integration tests
● This might take longer
● Development versions are usually deployed to the
artifact repository if tests pass
● Living testing env updated if tests pass

@fgortazar
Merge jobs
pipeline {
agent any
stages {
stage("Preparation") {
steps {
git 'https://github.com/codeurjc/testing.git'
}
}
stage("Create app") {
steps {
sh "cd tema1_4_ejem4; docker-compose build"
}
}
stage("Start app") {
steps {
sh "cd tema1_4_ejem4; docker-compose up -d"
}
}
stage("Test") {
steps {
sleep 20
sh "cd tema1_4_ejem4; mvn test"
}
}
}
post {
always {
sh "cd tema1_4_ejem4; docker-compose logs"
sh "cd tema1_4_ejem4; docker-compose logs > all-logs.txt"
archive "tema1_4_ejem4/all-logs.txt"
sh "cd tema1_4_ejem4; docker-compose logs web > web-logs.txt"
archive "tema1_4_ejem4/web-logs.txt"
sh "cd tema1_4_ejem4; docker-compose logs db > db-logs.txt"
archive "tema1_4_ejem4/db-logs.txt"
sh "cd tema1_4_ejem4; docker-compose down"
junit "tema1_4_ejem4/**/target/surefire-reports/TEST-*.xml"
}
Clone repository
Create docker images
Start docker-compose
Execute Test
Archive logs
Shutdown SUT

@fgortazar
Merge jobs
●
SUT Logs are available as a text file
●
One file for all logs
●
One file per component

@fgortazar
Merge jobs
●
We need tools to manage logs
●
Put all logs in a single place
●
Test logs and SUT logs
●
Remote service to be used from SUT deployed in
any place
●
Advanced Log analysis capabilities
●
Filtering
●
Searching
●
Comparing

@fgortazar
Merge jobs
Elasticsearch Kibana Logstash Beats
https://www.elastic.co/

@fgortazar
Merge jobs
node{
elastest(tss: ['EUS']) {
stage("Preparation") {
git "https://github.com/elastest/full-teaching-experiment.git"
}
stage("Start SUT") {
sh "cd docker-compose/full-teaching-env; docker-compose up -d"
sh "sleep 20"
}
stage("Test") {
def sutIp = containerIp("fullteachingenv_full-teaching_1")
try {
sh "mvn -Dapp.url=https://" + sutIp +":5001/ test"
} finally {
sh "cd docker-compose/full-teaching-env; docker-compose down"
}
}
}
}
Use ElasTest Plugin with Browsers
Start SUT
Get web container IP
Teardown SUT
Execute Tests

@fgortazar
Release jobs
● Building binaries
● Tagging the repository
● Publishing binaries in the artifact repository
● Smoke tests in staging environment
● Performance tests
● These can be integrated into the merge jobs
● Based on the version number the development/release
behavior is triggered

@fgortazar
Deployment jobs
● Shipping binaries to production environment
● Migrating the database if necessary
● Deploy based on the chosen strategy
● Blue/green
● Canary
● ...
● Run smoke tests to assess the critical paths

@fgortazar
Nightly jobs
● Run during the night or any moment where the CI
infra is not under heavy use
● Some of these tests need to be run in isolation
● End-to-end tests
● Performance tests
● Might take several hours to complete
● Verify the critical paths
● New software version needs to be deployed first

@fgortazar
Final thoughts
● There might be dependencies…
● Consider if you’d like to run other projects that depend on this
one at every change /every release
● Increases the usage of the CI environment plan in advance→
● Which information is more important to you?
● Gather that information in your job
● Archive it for later inspection
● Additional tools might be needed
● ElasticSearch
● Kibana
● ElasTest

@fgortazar
Final thoughts
● Jobs can get really big…
● Separate them into different jobs
● Add a new job to orchestrate these new jobs
● Jobs can be called from other jobs
● Better visibility
stage('Update ci environments') {
steps {
build job: 'Development/run_ansible',
propagate: false
build job: 'Development/kurento_ci_build',
propagate: false,
parameters: [string(name: 'GERRIT_REFSPEC', value: 'master')]
}
}

@fgortazar
stage('Testing new environment') {
steps {
parallel (
"kurento_api_audit" : {
build job: 'Development/kurento_api_audit', propagate: false,
},
"kurento_app_audit" : {
build job: 'Development/kurento_app_audit', propagate: false,
},
"capability_functional_audit" : {
build job: 'Development/capability_functional_audit', propagate: false,
},
"capability_stability_audit" : {
build job: 'Development/capability_stability_audit', propagate: false,
},
"webrtc_audit" : {
build job: 'Development/webrtc_audit', propagate: false,
}
)
}
}
Final thoughts
● They can be run in parallel

@fgortazar
Final thoughts
● Don’t repeat yourself
● Shared libraries is a great way of factoring out
common parts in your pipelines into a library that can
be reused across jobs
● Groovy libraries in a code repository
● Need to be added into Jenkins global configuration
● Just import them and enjoy!

How to plan and define
your CI/CD pipelines
Francisco (aka Patxi) Gortázar
francisco.gortazar@urjc.es
@fgortazar
Funded by the
European Union
Thanks!

How to plan and define your CI-CD pipeline

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to How to plan and define your CI-CD pipeline

Similar to How to plan and define your CI-CD pipeline (20)

More from ElasTest Project

More from ElasTest Project (13)

Recently uploaded

Recently uploaded (20)

How to plan and define your CI-CD pipeline