Ignite @ CfgMgmtCamp 2019

1. Using Prometheus to monitor your build pipelines
CfgMgmtCamp 2019
Lander Van den Bulcke – Open Source Consultant @ Inuits
landervdb @ {twitter,github,freenode}

2. Pipelines...
1

3. Problem
We do promotions from DEV → UAT → PROD manually.
Problem: people seem to forget to click this button...
2

4. Solution?
Let’s monitor this stuff!
Let’s use Prometheus, there’s an exporter for that!
https://github.com/lovoo/jenkins_exporter
3

5. Problem (bis)
Uses the Jenkins JSON API
Does at least 1 HTTP request for every job
We have around 2500 jobs in or main Jenkins master
Great if you want to DoS your Jenkins master!
4

6. Solution (bis)!
I’ll just write my own!
Parse Jenkins config directly
Concurrent
Add custom metrics!
5

7. Assumptions
Build Environment plugin installed
Your pipelines all follow the same patterns
6

8. Assumptions: folder structure
/
├── folder
│ ├── subfolder
│ │ ├── checkout
│ │ ├── validate
│ │ ├── ...
│ │ ├── deploy-dev
│ │ └── deploy-prod
│ └── ...
└── ...
7

9. Frustration :(
Sooo... Jenkins decided to switch to XML 1.1
Go XML parser doesn’t support this...
Dirty hacking required for now :(
8

10. Results?
So what does this give us?
# HELP jenkins_last_build_duration_seconds Duration of the last build
# TYPE jenkins_last_build_duration_seconds gauge
jenkins_last_build_duration_seconds{folder="pipelines/puppet",
jenkins_job="checkout", result="stable"} 0.029→
jenkins_last_build_duration_seconds{folder="pipelines/puppet",
jenkins_job="checkout", result="successful"} 0.029→
...
# HELP jenkins_last_build_number Build number of the last build
# TYPE jenkins_last_build_number gauge
jenkins_last_build_number{folder="pipelines/puppet", jenkins_job="checkout",
result="stable"} 4→
jenkins_last_build_number{folder="pipelines/puppet", jenkins_job="checkout",
result="successful"} 4→
...
9

11. But what can I do with it?
changes(jenkins_last_build_number
{result="successful",jenkins_job="deploy-prod"}[1w])→
changes(jenkins_last_build_number
{result="successful",jenkins_job="deploy-dev"}[1w])
- ignoring(jenkins_job) changes(jenkins_last_build_number
{result="successful",jenkins_job="deploy-prod"}[1w])
→
→
→
10

12. Something more useful?
- record: jenkins:deploytimedelta:devprod
expr: jenkins_last_build_timestamp_seconds
{result="successful",jenkins_job="deploy-dev"} - ignoring(jenkins_job)
jenkins_last_build_timestamp_seconds
{result="successful",jenkins_job="deploy-prod"}
→
→
→
- alert: LongTimeSinceDeployToPROD
expr: jenkins:deploytimedelta:devprod > 300
labels:
severity: warning
annotations:
summary: "The build for {{ $labels.folder }} on PROD is at least 5 minutes
older than the one on DEV"→
11

13. Problems (again?!)
What if we trigger a promote on an older version?
12

14. Solution!
Custom metrics!
CHECKOUT_BUILD_NUMBER
⇓
jenkins_custom_last_checkout_build_number
13

15. Dashboards!
14

16. Dashboards!
15

17. Finally something we can work with!
- record: jenkins:deploydelta:devprod
expr: jenkins_custom_last_checkout_build_number
{result="successful",jenkins_job="deploy-dev"} - ignoring(jenkins_job)
jenkins_custom_last_checkout_build_number
{result="successful",jenkins_job="deploy-prod"}
→
→
→
- alert: BigVersionDifferencePROD
expr: jenkins:deploydelta:devprod > 3
labels:
severity: warning
annotations:
summary: "The build for {{ $labels.folder }} on PROD is at least 3 versions
older than the one on DEV"→
16

18. Alerts!
17

19. Find out more
Use it! Fork it! Send PR’s!
https://github.com/landervdb/jenkins_exporter
Configuration used in this presentation:
https://git.io/fhSUV
18

20. Bye!
18