The document discusses the management of multiple Jenkins servers and numerous Jenkins jobs within a DevOps framework, emphasizing the use of Jenkins Job Builder (JJB) for automation. It covers Jenkins architecture, job configuration, and integration with various source control systems. Additionally, it highlights the importance of using YAML files for job definitions and managing CI/CD pipelines in a cloud and on-premises environment.

1. Jenkins & Infrastructure as Code
HungWei Chiu

2. Why
Maintain and manage three Jenkins servers
Almost 1000 Jenkins Jobs
What I learned

3. Bio
HungWei Chiu(Hwchiu)
MTS @ Open Networking Foundation (ONF)
Kubernetes/Container/Linux/Network...etc
Blog: https://hwchiu.com
Facebook: 矽⾕⽜的耕⽥筆記

4. Agenda
Introduction to Jenkins
Infrastructure as Code and Jenkins Job Builder (JJB)
Demo
Global JJB

5. Jenkins
Self-containerd, open source automation server
Can be used to automate all sorts of taks
Java-based application.
Plugin ecosystem to enhance its functionality.

6. Supports
VSC system
Git/GitHub/GitLab/Gerrit
Bitbucket/SVC...etc
Authentication system
Internal DB, LDAP, OAuth, Azure AD, Crowd, OpenID..etc
Authorization
Matrix-based security (by user and groups)
Project-based
Almost all functions are powered by plugin.

7. Use Cases
Source Code
(GitHub/Git)
Developer Jenkins
Unit Test
Integration Test
Tasks
Build container
Push container
Deploy application
Run your scripts
Push Trigger
Feedback
Execute

8. Use Cases
Others
Developer Jenkins
Unit Test
Integration Test
Tasks
Build container
Push container
Deploy application
Run your scripts
Web hook
Feedback
Execute

9. Scheduled Job
Source Code
(GitHub/Git)
Developer Jenkins
Unit Test
Integration Test
Tasks
Build container
Push container
Deploy application
Run your scripts
Feedback
Pull
Execute

10. Architecture
Jenkins Server
Self-hosted
Jenkins Job
Freestyle
Pipeline
GitHub Organization
Multi-branch Pipeline
Executor

11. Architecture
Jenkins
Job A
Con
fi
guration
Tasks
Job B
Con
fi
guration
Tasks
Job C
Con
fi
guration
Tasks
Job D
Con
fi
guration
Tasks
GitHub Trigger Scheduled
Trigger by JobA WebHook Trigger
Test
CD
CI
Test

12. Architecture
Jenkins-
Job A
Con
fi
guration
Tasks
Job B
Con
fi
guration
Tasks
Virtual Machine
Jenkins Executors
Bare Metal Server
Container
Dispatch Jobs

13. Installation
System packages
Ansible playbook
Container
Docker command
Kubernetes
Helm
Jenkins X
CD foudnation

14. Installation
Con
fi
guration
General
Source Code Control
Build Trigger
...
Tasks
Pipeline scripts
Shell scripts
....

15. Demo
https://jenkins.hwchiu.com
curl -X POST "https://hwchiu:xxxx@jenkins.hwchiu.com/job/
manually-pipeline-2/build?token=haha"

16. Common Issues
Server Con
fi
gurations
Job Con
fi
gurations
Manual
Tasks
Manual
VCS (Pipeline scripts)

17. Common Issues
How to audit ?
who made what change
How to manage the Jenkins ?
Similar jobs

18. JJB
Jenkins Job Builder
Takes simple descriptions of Jenkins jobs in YAML format.
Use them to con
fi
gure the Jenkins jobs
Of
fi
cial site
https://jenkins-job-builder.readthedocs.io/en/latest/
index.html

19. JJB
All jobs are de
fi
ned by YAML
fi
les.
How it works
Prepare an authentication
fi
le with Jenkins credentials.
Install JJB tool
De
fi
ne your jobs in YAML
Execute the JJB to update your jobs in Jenkins server

20. Architecture
Jenkins
Job A
Con
fi
guration
Tasks
Job B
Con
fi
guration
Tasks
YAML
fi
les
JJB tool
Update
Read and to XML

21. Authentication FIle
Jenkins.ini

22. Tools
pip install --user jenkins-job-builder

23. List Jobs
jenkins-jobs --conf jenkins.ini list

24. De
fi
ne Job
jenkins-jobs --conf jenkins.ini update job.yaml

25. Job-Template
Need several jobs de
fi
ned that are nearly identical
Use the job-template to de
fi
ne the skeleton of a job
Use the project to realize the job with appropriate variable
substitution.

26. Specify the job-template by id

27. Pipeline job
De
fi
ne the pipeline script in the repo.
Include the pipeline in Job's de
fi
nition.
Use one VCS for both JJB and pipeline scripts

29. Real Case
A large scale open source project
Involved industries
Software, Hardware, System Integrator
A single Jenkins system for all purposes
How to con
fi
gure the Jenkins?

30. Software
Would like to developing the software
Unit test
Integration test
Compatibility
Jenkins executors
Cloud
On-premises

31. Hardware
Would like to provide the hardware to support the software
Integration test
Compatibility
Jenkins executors
On-premises

32. System Integrator
Develop their own software to connect the open source and hardware
Integration test
Compatibility
Jenkins executors
Cloud
On-premises

33. Software
Jenkins Server
Jenkins Executor
Software
Company
Jenkins Executor
USA
Cloud(VM/Container)
Job A
Con
fi
guration
Tasks
De
fi
ne
Require variety software

34. Hardware
Jenkins Server
Jenkins Executor
Hardware
Company
Europe
Job A
Con
fi
guration
Tasks
De
fi
ne

35. SI
Jenkins Server
Jenkins Executor
SI
Jenkins Executor
India
Cloud(VM/Container)
Job A
Con
fi
guration
Tasks
De
fi
ne
Require variety software

36. Architecture
Jenkins Server
Jenkins Executor
Jenkins Executor
Jenkins Executor
Jenkins Executor
USA India Europe
Cloud
Jenkins Executor
Cloud

37. Issues
All people want to de
fi
ne their own jobs.
Including con
fi
guration, tasks and executor environment.
For con
fi
guration and task
A single repo with JJB
fi
les
For executor environment
Packer

38. JJB
We need a Jenkins Job to execute the JJB to update all Jenkins
Job
Chicken or the egg
Manually create a Jenkins Job for updating JJB
Then, ask that Jenkins Job to update Jenkins by JJB YAML.

39. Possible Scenario
Jenkins Server
Operator
Git Repo
Job A
Con
fi
guration
Tasks
JJB Yaml
JJB Yaml
JJB Yaml
JJB Yaml
Read
Create
fi
rst Job

40. Possible Scenario
Jenkins Server
Git Repo
Job A
Con
fi
guration
Tasks
JJB Yaml
JJB Yaml
JJB Yaml
JJB Yaml
Jenkins Executor
Cloud(VM/Container)
Dispatch
JJB Update
Pull Code
Trigger Jenkins
2
1
3
4

41. Packer
HashiCorp Packer automates the creation of any type of machine image
It embraces modern con
fi
guration management
Automated scripts to install and con
fi
gure the software within your Packer-made images.
AWS/Azure/Docker...etc
AMI (build, push)
Docker (build + tag + push)
Prepare a Jenkins Job to perform the Packer to create the new image for Jenkins executor.

42. Possible Scenario
Jenkins Server
Git Repo
Job A
Con
fi
guration
Tasks
JJB Yaml
JJB Yaml
JJB Yaml
JJB Yaml
Jenkins Executor
Cloud(VM/Container)
Dispatch
Pull Code
Trigger Jenkins
2
1
3
Cloud Provider
4
Build and Push

43. Global JJB
Provided by Linux Foundation
Lots of pre-de
fi
ned JJB
fi
les
Jenkins jobs to perform the JJB to update Jenkins
Jenkins jobs to perform Packer to create image
ci-management