1.
©2019 Collibra Inc
Keeping track of your CI/CD mess
@DamienCoraboeuf (Damien Coraboeuf / CICD lead at Collibra)
Wrocław Java User Group - April, 25th, 2019

2.
A few words about myself…

3.
WHO AM I?
▸ Damien Coraboeuf (@DamienCoraboeuf)
▸ Family, hiking, good food, good beer, role playing, …
▸ Born to Java in 1996, developer, architect
▸ Working in Continuous Delivery since 2008
▸ Automating pipelines at scale

4.
What are we talking about?

5.
CI/CD IN A NUTSHELL
Ideas, features, fixes Happy customer
Fast feedback
Security
Quality at all times
Automation
Traceability / Accountability
Ready to release at all times
I work (mostly) here

6.
CI/CD ECOSYSTEM
It starts with some code…
Linked with some issues…
We build it….
We package and publish…
We run some tests…
We perform quality scans….
We need some infra…

7.
IT’S POURING INFORMATION ALL DAY LONG
A
rtefacts
Depen
d
en
cies
Triggering other
pipelinesRuns
tests
Collects
results
Scans
co
d
e
M
easures
perform
ances
Tags
/
com
m
its
Tickets
Deploying

8.
IT’S POURING INFORMATION ALL DAY LONG

9.
ALL DAY LONG…

10.
PUTTING SOME ORDER INTO THIS DATA
▸ Projects
▸ Branches
▸ Builds
▸ Validations
▸ Promotions
Metadata!!

11.
SEEING IS BELIEVING
▸ One tool to rule them all
▸ http://nemerosa.github.io/ontrack/
Builds
Promotions
Validations
Validation status
System of records for anything related to CI/CD

12.
SEEING IS BELIEVING
▸ Filtering
Promoted … despite anomalies in
deployments
▸ Status of your pipeline in one glance

13.
CONTRIBUTIONS

14.
BUILD INFORMATION

15.
SEEING IS BELIEVING
▸ Overview of all your branches

16.
SEEING IS BELIEVING
▸ Change logs between builds, between versions
▸ Permalink for automated release notes
▸ Supports GitHub, JIRA, Subversion, BitBucket…

17.
SEEING IS BELIEVING
▸ Using cross referenced information for searches:
▸ Where is my ticket?
▸ Where is my commit?
Example: where is my #600 ticket?
Here you go!

18.
SEARCH ON BUILDS
Change log

19.
MANY MORE FEATURES…
Auto promotions…
Filtered & custom views…
… and many more…
+ + + =

20.
Integration is everything

21.
SINGLE POINT OF KNOWLEDGE
Ontrack
Artifactory
GitLab
InfluxDB
Jenkins
Vault
JIRA
BitBucket
Subversion
GitHub
Git

22.
Ontrack
INTEGRATION WITH THE REST OF THE ECOSYSTEM
Jenkins plug-inGUI
REST API
GraphQL
InfluxDB metrics
Groovy DSL

23.
JENKINS PLUG-IN
Post build actions
and
steps

24.
JENKINS PLUG-IN
Integration with the Ontrack DSL

25.
ONTRACK DSL
▸ Groovy based DSL
Automation of
Change log

26.
JENKINS PIPELINE INTEGRATION

27.
FEEDING INFORMATION INTO ONTRACK
‣ Ontrack has a rich API
‣ … but better use the Ontrack Jenkins plug-in
‣ 100% integration with the Jenkins pipeline DSL
‣ … but @ Collibra, we provide integrated wrappers
Typical workflow:

28.
GETTING INFORMATION FROM ONTRACK
‣ Ontrack DSL can be used
‣ … wrappers provided @ Collibra

29.
AUTOMATED FEEDBACK
▸ How to get the best out of your information

30.
AUTOMATED FEEDBACK
▸ The collected information can be used in the ecosystem
▸ Based on Ontrack DSL
or GraphQL
▸ Example: promotions used
as inputs for on-demand deployments jobs

31.
META INFORMATION AT ALL LEVELS
▸ Attaching built-in or
custom properties to
all levels - projects,
branches, builds, etc.
▸ Used for reporting or
to drive processes
▸ Can be entered
manually or
automatically

32.
And this is NEVER enough…

33.
COLLECT YOUR OWN CUSTOM INFORMATION
Ontrack
▸ Extension system
▸ New connectors
▸ New services, screens and interfaces

34.
CUSTOMISING THE SOLUTION
Ontrack Core
Extension
Ontrack
Java API
Extension
code

35.
CUSTOMISING THE SOLUTION
▸ TV wallboards for development teams
▸ Version auto upgrade with PR on promotion
▸ Dependency graph
▸ …
▸ Imagination is the limit

36.
CUSTOMISING THE SOLUTION
▸ TV wallboards for development teams
GraphQL / REST

37.
PR CREATION ON PROMOTION FOR AUTO VERSION UPGRADES
A B
depends on
Manual update of version of A in B
Little automation
Loose coupling
Little control of quality

38.
PR CREATION ON PROMOTION FOR AUTO VERSION UPGRADES
A B
depends on
Automated PR for update of version of A in B
Control of quality
Strong coupling
Automation

39.
PR CREATION ON PROMOTION FOR AUTO VERSION UPGRADES
A B
depends on
Promotion
B A dependency
configuration
A
Subscription
Queue —> Listener Creates a PR
for update of A
in B
PR build for
control
Automated merge
after controls OK
Listens to
Control of quality Loose coupling Automation

40.
DEPENDENCY GRAPHS
Ontrack has built-in dependencies…
… which can be used to build a graph (in Neo4j)
… and execute complex queries
… to detect dependencies anomalies

41.
Getting Ontrack on track…

42.
ONTRACK MANAGEMENT
▸ Jobs
▸ Metrics
▸ Deployment
▸ Security model

43.
SYSTEM JOBS
▸ ~ 1000 jobs running in the background
▸ Critical system
▸ Collecting information all day long
▸ System administration interface

44.
METRICS
▸ Ontrack can export lot of information to InfluxDB
▸ Additional metrics can be created through extensions

45.
SECURITY
▸ Authentication
▸ Built-in users
▸ LDAP integration
▸ Extensible
▸ Authorisations
▸ Built-in ACL (roles & functions)
▸ Extensible
▸ Strong isolation between projects
▸ … but cross-projects roles

46.
Getting started with Ontrack…

47.
ONTRACK
▸ http://nemerosa.github.io/ontrack/
▸ Use it, fill tickets, extend, contribute

48.
THANK YOU!
Thank you: Contact / Info:
▸ @DamienCoraboeuf
▸ http://nemerosa.com
Marcin Stachniuk