We discuss things to be taken into account when deciding on a policy for your CI/CD pipelines. This might include Git workflows, testing approaches, and shipping strategies.
1. How to plan and define
your CI/CD pipelines
Francisco (aka Patxi) Gortรกzar
francisco.gortazar@urjc.es
@fgortazar
Funded by the
European Union
2. @fgortazar
Bio
Project Coordinator at @elastestio EU project
Devops @ Kurento/OpenVidu
TeachingTesting & Distributed Systems @ URJC
@fgortazar
https://es.linkedin.com/in/franciscogortazar
3. @fgortazar
Bio
Project Coordinator at @elastestio EU project
Devops @ Kurento/OpenVidu
TeachingTesting & Distributed Systems @ URJC
@fgortazar
https://es.linkedin.com/in/franciscogortazar
4. @fgortazar
How to plan and design your pipelines
โ Some preliminary concepts
โ Background
โ Planning
โ CI/CD environment
โ Git workflow
โ Pipelines
โ Code review
โ Merging
โ Releasing
โ Deploying
5. @fgortazar
Some preliminary concepts
โ Software is usually built by teams
โ People within the team share the code as they build it
by means of source control management tools
SCM
Tools / Services offering git repositories
6. @fgortazar
Some preliminary concepts
โ Each time someone shares its changes through the
repository, we should check that:
โ The project is still compiling
โ All the tests still pass (theyโre green)
โ Compiling and running tests on a trusted
environment is Continuous Integrationโs task
โ When a problem arises, it is notified immediately
to the team
7. Some preliminary concepts
โ Integration must happen continuously (at least once
per day)
โ Fast feedback is key, so that we know our changes
are working when integrated with the rest of the
system
โ Problems detected in our CI environment must be
solved immediately
โ Status of the project in terms of builds, test, and
deployments is visible at any moment
13. Background
โ Some numbers
โ 30 code repositories
โ ~400 Jenkins jobs
โ +1,000 tests
โ +20 different environments to test
โ +80 artifacts to be deployed at release time
14. @fgortazar
How to plan and design your pipelines
โ Some preliminary concepts
โ Background
โ Planning
โ CI/CD environment
โ Git workflow
โ Pipelines
โ Code review
โ Merging
โ Releasing
โ Deploying
15. Planning
โ Whatโs your Git workflow
โ Branches are to be integrated as well?
โ Depend-on components are to be considered?
โ Infrastructure planning
โ Snapshot versions
โ Release versions
โ Deployments
16. @fgortazar
CI/CD environment
โ Feasible infrastructures and their consequences
โ Public cloud vs private cloud (on premises)
โ Controlling public cloud costs (e.g. spot instances, stopping
instances not in use...)
โ Virtual Machines vs Docker
โ 20 environments = 20 differentVMs
โ Cost increases with each new environment!!
โ Effort increases with each new environment to configure
(ops)!!
โ Time-to-market increases also!!
18. @fgortazar
CI/CD environment
โ Developers are pushing hard towards devops to
include changes in infrastructure
โ Changes can hardly be reverted (possible, butโฆ)
โ Hard to test locally
โ Works in my machine effect
โ Wasted resources & insufficient resources at the same
time
โ Can we do better?
22. CI/CD environment
โ Developers own the infrastructure on top of which
their tests will run
โ Updates can be done safely
โ CI environment much more stable
โ Docker images can be updated quickly and easily
โ Infrastructure testing much easier
23. @fgortazar
Git workflow
โ When working together in a shared repository,
some rules apply as to how changes from the
different people involved in the team are
integrated, and which is the source of truth when
releasing software versions
โ Weโll briefly see some common models of git
workflows that people is using
โ When necessary, Iโll point you towards specificities
for some models regading the definition of your CI/
CD pipeline
24. @fgortazar
Git workflows
โ Basic / Master development /Trunk development
โ Everything happens in the master branch
โ Each feature is developed in a local repository
โ When features are not small enough some problems arise
โ Members of the team might push commits to master for
uncompleted features
โ Impossible to release a new version before all features has been
completed and integrated
โ Most of the time master is broken, and unplanned releases are a
risk
25. @fgortazar
Git workflows
โ Branch per feature
โ Each features is developed in its own branch (branchโed
from master)
โ The feature is integrated only when complete
โ Master is stable, and can be released at any moment
โ The branch might be local or it can be shared thus
enabling CI for the features under development
โ If so you need a multi-branch pipeline job in Jenkins jargon
26. @fgortazar
Git workflows
โ Branch per feature with merge requests
โ Similar to branch per feature, however thereโs a
hierarchical permission model
โ Some people can integrate directly in master
โ Others must request a merge, that will be reviewed by
some members of the time prior to merge
โ The CI server might help in integrating changes
27. @fgortazar
Git workflows
โ Gitflow
โ Probably the most complex one
โ Useful in big projects, with many features and several
release versions to maintain
โ There are two branches
โ Master โ production branch. Integration into master only
happens if acceptance tests pass. Ready to be deployed.
โ Develop โ integration branch. Features, developed in branches,
are integrated here first, then tests are run.
http://nvie.com/posts/a-successful-git-branching-model/
28. @fgortazar
Git workflows
โ Fork workflow
โ One of the most used in GitHub
โ Developers fork the repository into their accounts
โ This is called โforkingโ the repo
โ Then they clone their own copy into their machines
โ Changes are integrated into the developerโs repo first
โ Changes are requested to be integrated into the main repo by
means of โpull-requestsโ (PRs)
โ Someone with the necessary permissions accepts or rejects
the PR, probably with some aid from the CI server
https://github.com/elastest/elastest-platform-manager/pulls?q=is%3Apr+is%3Aclosed
29. @fgortazar
How to plan and design your pipelines
โ Some preliminary concepts
โ Background
โ Planning
โ CI/CD environment
โ Git workflow
โ Pipelines
โ Code review
โ Merging
โ Releasing
โ Deploying
30. @fgortazar
Pipelines
โ What is a pipeline?
โ Two meanings
โ Set of jobs that are run in sequence
โ In Jenkins a Pipeline job is a job that leverages the
Pipeline syntax based on the Groovy language
โ There are two different syntax for Pipeline jobs
โ Declarative Pipeline (DSL)
โ Scripted Pipeline (Groovy)
31. @fgortazar
Code review jobs
โ Needed only when doing code review
โ Pull Requests / Merge Requests
โ Run per change request before the commit is push to
master
โ Fast: unit tests mainly
โ Fix the commit before getting too deep into any
other task
โ CI votes the change
โ +1 can be accepted & integratedโ
โ -1 cannot be integrated, amendment neededโ
34. @fgortazar
Merge jobs
โ Run once a something has been integrated into the
master branch
โ Usually integration tests
โ This might take longer
โ Development versions are usually deployed to the
artifact repository if tests pass
โ Living testing env updated if tests pass
38. @fgortazar
Merge jobs
โ
We need tools to manage logs
โ
Put all logs in a single place
โ
Test logs and SUT logs
โ
Remote service to be used from SUT deployed in
any place
โ
Advanced Log analysis capabilities
โ
Filtering
โ
Searching
โ
Comparing
46. @fgortazar
Release jobs
โ Building binaries
โ Tagging the repository
โ Publishing binaries in the artifact repository
โ Smoke tests in staging environment
โ Performance tests
โ These can be integrated into the merge jobs
โ Based on the version number the development/release
behavior is triggered
47. @fgortazar
Deployment jobs
โ Shipping binaries to production environment
โ Migrating the database if necessary
โ Deploy based on the chosen strategy
โ Blue/green
โ Canary
โ ...
โ Run smoke tests to assess the critical paths
48. @fgortazar
Nightly jobs
โ Run during the night or any moment where the CI
infra is not under heavy use
โ Some of these tests need to be run in isolation
โ End-to-end tests
โ Performance tests
โ Might take several hours to complete
โ Verify the critical paths
โ New software version needs to be deployed first
50. @fgortazar
Final thoughts
โ There might be dependenciesโฆ
โ Consider if youโd like to run other projects that depend on this
one at every change /every release
โ Increases the usage of the CI environment plan in advanceโ
โ Which information is more important to you?
โ Gather that information in your job
โ Archive it for later inspection
โ Additional tools might be needed
โ ElasticSearch
โ Kibana
โ ElasTest
51. @fgortazar
Final thoughts
โ Jobs can get really bigโฆ
โ Separate them into different jobs
โ Add a new job to orchestrate these new jobs
โ Jobs can be called from other jobs
โ Better visibility
stage('Update ci environments') {
steps {
build job: 'Development/run_ansible',
propagate: false
build job: 'Development/kurento_ci_build',
propagate: false,
parameters: [string(name: 'GERRIT_REFSPEC', value: 'master')]
}
}
53. @fgortazar
Final thoughts
โ Donโt repeat yourself
โ Shared libraries is a great way of factoring out
common parts in your pipelines into a library that can
be reused across jobs
โ Groovy libraries in a code repository
โ Need to be added into Jenkins global configuration
โ Just import them and enjoy!
54. How to plan and define
your CI/CD pipelines
Francisco (aka Patxi) Gortรกzar
francisco.gortazar@urjc.es
@fgortazar
Funded by the
European Union
Thanks!