Brujug Jenkins pipeline scalability

Damien Coraboeuf
Damien CoraboeufContinuous Delivery Expert at Clear2Pay
JOB SCALABILITY IN JENKINS
GETTING OUT OF THE JOB JUNGLE
BruJUG - March 9th, 2016 - @DamienCoraboeuf
WHO AM I?
▸ Damien Coraboeuf (@DamienCoraboeuf)
▸ Born to Java in 1996
▸ “Brol” engineer since 2008
▸ Automating pipelines @ Clear2Pay / FIS & Multipharma
I’ll be your guide on this journey
THE MAP Short introduction to Jenkins
Birth of a pipeline
Manual mode
The Job DSL
Branching and its consequences
Pipeline as code
Pipeline as not code
Automation of automation
We want to get there
Look, there is another

beach here!
Do not go there!
💀
DO YOU KNOW WHO THIS IS?
JENKINS
▸ http://jenkins-ci.org/
▸ Very versatile Continuous Integration engine
▸ ~ 70% of the CI market (2012 figures)
▸ Open source - strong community
▸ Extensible
▸ more than 1000 plugins
▸ easy to develop new ones
▸ Job scheduler
▸ Integrates with basically everything
JENKINS - JOBS & FOLDERS
JENKINS - JOBS IN FOLDER
JENKINS - PIPELINE VIEWS
JENKINS - JOB
JENKINS - JOB CONFIGURATION
Let’s get some sources…
JENKINS - JOB CONFIGURATION
Let’s run some Bash…
JENKINS - JOB CONFIGURATION
Let’s launch some Gradle…
JENKINS - JOB CONFIGURATION
Let’s trigger another job…
PIPELINES
▸ Defining jobs
▸ Linking them together
▸ Running them in parallel or in sequence
▸ Some triggers are automatic, other ones are manual
BUILD
DEPLOY ENV 1
DEPLOY ENV 2
DEPLOY ENV 3
PUBLICATION RELEASE
A PIPELINE IS BORN
BUILD
DEPLOY ENV 1
DEPLOY ENV 2
DEPLOY ENV 3
PUBLICATION RELEASE
BUILD
DEPLOY ENV 1
DEPLOY ENV 2
DEPLOY ENV 3
PUBLICATION RELEASE
BUILD
DEPLOY ENV 1
DEPLOY ENV 2
DEPLOY ENV 3
PUBLICATION RELEASE
Commit 1
Commit 2
Commit 3
SO FAR SO GOOD
“Look Ma, I’ve built a pipeline!”
“Lovely. Now, build one for your sister.”
THE STORY
▸ 40+ projects…
▸ …several maintenance & feature branches per project
▸ … complex validation pipelines (10 - 20 jobs)
▸ … thousands of jobs
▸ … only a small team for Jenkins
Jenkins team
> 2700 jobs and growing…
Brujug Jenkins pipeline scalability
In order to scale, we want:
✔ Self service
✔ Security
✔ Simplicity
✔ Extensibility
“What are we?”
“Developers!”
“What do we do?”
“Code!”
“Why do we do it?”
“Automation!”
PIPELINE AS CODE
▸ The Job DSL
▸ Pipeline as code
▸ Pipeline libraries
▸ Pipeline as properties
▸ Pipeline automation
JOB DSL PLUGIN - START OF THE JOURNEY OUT OF THE JUNGLE
▸ Define a job using a Groovy based DSL!
JOB DSL PLUGIN
▸ Very well supported
▸ Very good documentation
▸ Supports most of the plugins
▸ Built-in extensibility: inline or DSL extensions
JOB DSL PLUGIN
▸ Folders & views
▸ … automatic triggers upon generation
▸ … using files in workspace
JOB DSL PLUG-IN
▸ This remains code…
JOB DSL PLUGIN
▸ DSL based generation done from a job
▸ Can generate and/or update other jobs
▸ Can even run while jobs are running :)
SEED JOB GENERATED JOB(S)
DSL SCRIPT
Generates
Accesses (SCM or inline)
GENERATED JOB(S)
GENERATED JOB(S)
BRANCHING
▸ One Seed DSL job is all very good but…
▸ Pipeline evolves with the code
▸ Different pipelines for different branches
develop
release/2.0
Needs a new job
Pipelines for releases

might be more complex
PIPELINE AS CODE
▸ Your pipeline is linked to the code it builds
▸ Define your pipeline in your code
Project
src
pom.xml
job-dsl-script.groovy
PIPELINE CODE DUPLICATION
▸ We can now generate a pipeline for any branch, any commit







▸ It evolves with your branches and is merged like any other piece of code
▸ That’s good enough for 1 project
▸ With several (many) projects, the level of DSL code duplication explodes!
DEVELOP FEATURE/BIG RELEASE/1.0
DSL SCRIPT DSL SCRIPT DSL SCRIPT
DEVELOP FEATURE/BIG RELEASE/1.0
DSL SCRIPT DSL SCRIPT DSL SCRIPT
DEVELOP FEATURE/BIG RELEASE/1.0
DSL SCRIPT DSL SCRIPT DSL SCRIPT
DEVELOP FEATURE/BIG RELEASE/1.0
DSL SCRIPT DSL SCRIPT DSL SCRIPT
DEVELOP FEATURE/BIG RELEASE/1.0
DSL SCRIPT DSL SCRIPT DSL SCRIPT
PIPELINE CODE DUPLICATION
▸ Job DSL code duplication has the same issues than
production code duplication
DEVELOP FEATURE/BIG RELEASE/1.0
DSL SCRIPT DSL SCRIPT DSL SCRIPT
DEVELOP FEATURE/BIG RELEASE/1.0
DSL SCRIPT DSL SCRIPT DSL SCRIPT
DEVELOP FEATURE/BIG RELEASE/1.0
DSL SCRIPT DSL SCRIPT DSL SCRIPT
DEVELOP FEATURE/BIG RELEASE/1.0
DSL SCRIPT DSL SCRIPT DSL SCRIPT
✘ ✘ ✘ ✘ ✘ ✘
✘✘✘✘✘✘
DEVELOP FEATURE/BIG RELEASE/1.0
DSL SCRIPT DSL SCRIPT DSL SCRIPT
DEVELOP FEATURE/BIG RELEASE/1.0
DSL SCRIPT DSL SCRIPT DSL SCRIPT
✘✘✘✘✘✘
DEVELOP FEATURE/BIG RELEASE/1.0
DSL SCRIPT DSL SCRIPT DSL SCRIPT
DEVELOP FEATURE/BIG RELEASE/1.0
DSL SCRIPT DSL SCRIPT DSL SCRIPT
✘✘✘✘✘✘
DEVELOP
DSL SCRIPT
DEVELOP
DSL SCRIPT
✘
✘
DEVELOP
DSL SCRIPT
✘
DEVELOP
DSL SCRIPT
✘
LEASE/1.0
L SCRIPT
LEASE/1.0
L SCRIPT
LEASE/1.0
L SCRIPT
LEASE/1.0
L SCRIPT
DEVELOP FEATURE/BIG RELEASE/1.0 DEVELOP FEATURE/BIG RELEASE/1.0 DEVELOPLEASE/1.0
Brujug Jenkins pipeline scalability
PIPELINE LIBRARIES TO THE RESCUE
▸ Pipeline is code
▸ Reuse of code through versioned libraries
DEVELOP FEATURE/BIG RELEASE/1.0
DSL SCRIPT DSL SCRIPT DSL SCRIPT
PIPELINE DSL LIBRARY
1.1 1.0
Normal project

Can be tested

Can be released
PIPELINE LIBRARIES TO THE RESCUE
▸ DSL libraries as code libraries
DEVELOP FEATURE/BIG RELEASE/1.0
DSL SCRIPT DSL SCRIPT DSL SCRIPT
PIPELINE DSL LIBRARY
1.1 1.0
COMMON DSL LIBRARY
1.0
Dependencies
ARE WE DONE?
▸ We can describe a pipeline using a DSL
▸ The pipeline is defined together with the code it builds
▸ The DSL can use libraries to reduce code duplication
?
NOT QUITE…
▸ This is not enough to really scale
▸ We still have to write some DSL
▸ Self service ✔
▸ Security ✘
▸ Simplicity ✘
▸ Extensibility ✔
PIPELINE AS… PROPERTIES
▸ Let’s describe the pipeline using a properties file
▸ Which pipeline library & which version
▸ Configuration properties - specific to the library
▸ Property file format ubiquitous
▸ No code running on the master
▸ Can be used for reporting on all the pipelines!
PIPELINE AS PROPERTIES
seed.properties
SEED PLUGIN
▸ https://github.com/jenkinsci/seed-plugin
▸ Allows to generate:
▸ projects folders
▸ branches pipelines
▸ based on seed files (properties and/or DSL)
▸ integration with hooks
SEVERAL INTEGRATION MODES
SEED.PROPERTIES
SEED PLUGIN
SEED.PROPERTIES
SEED.GROOVYSEED.GROOVY
PIPELINE LIBRARY PIPELINE LIBRARY
BRANCH PIPELINE
95% 4%
1%
GENERATION STRUCTURE
SEED
PROJECT GENERATOR
PROJECT FOLDER
BOOTSTRAPPING
BRANCH GENERATOR
BRANCH FOLDER
BRANCH JOB 1
BRANCH JOB 2
BRANCH JOB 3
BRANCH JOB 4
GENERATION
Project teamJenkins team
Project only visible
by project team
SEED JOBS
BOOTSTRAPPING
GENERATION
GENERATION
AUTOMATION
HOOKS
▸ Generations can be automated using hooks at SCM level
▸ Support for GitHub, BitBucket, generic HTTP calls (for SVN)
▸ Configurable. For example:
BRANCH CREATION
SCM EVENT
Seed plug-in
COMMIT
SEED CHANGED
BRANCH DELETION
PIPELINE EVENT
Generation
Triggers the pipeline
Regenerates the pipeline
Deletes the pipeline
RESPONSIBILITIES
Jenkins team
Pipeline libraries
Develops and
maintains
Development team
seed.properties
Defines
Seed plug-in
SCM
Triggers
WHAT DID WE JUST ACHIEVE?
▸ Automation of automation
▸ Self service ✔
▸ Pipeline automation from SCM
▸ Security ✔
▸ Project level authorisations
▸ No code on the master
▸ Simplicity ✔
▸ Property files
▸ Extensibility ✔
▸ Pipeline libraries
▸ Direct job DSL still possible
YOU CAN NOW RELAX…
DEMO TIME
▸ Bootstrapping a project
▸ Configuring the project for GitHub
▸ Creating a branch —> Branch folder generated
▸ Committing on the branch —> Pipeline started
▸ Updating the pipeline branch —> Pipeline regenerated
▸ Deleting the branch —> Branch folder gone
DEMO - PIPELINE AS CODE
seed/seed.groovy
DEMO - GITHUB SETUP
DEMO - JENKINS SETUP
DEMO - BOOTSTRAPPING
A GLIMPSE OF THE FUTURE
▸ Jenkins 2.0
▸ Pipeline plug-in
▸ “Pipeline as Code” in the core
▸ How does the current solution fit?
SEED & PIPELINE PLUGIN
▸ The Seed plugin will keep working
▸ It can already generate pipeline jobs (Job DSL)
▸ But overlap of functionalities
Branch pipelines
Pipeline as code
SEED
PIPELINE
Hook integration
Pipeline as properties
Community & support
Pipeline libraries
Pipeline script library
SEED IN PIPELINE PLUGIN
▸ The Seed plugin as extension of the Pipeline plugin
Branch pipelines
Pipeline as code
SEED
PIPELINE
Hook integrationPipeline as properties
Community & support
Pipeline libraries
Pipeline script library
THANKS YOU!
Thanks to:
‣ BruJUG
‣ Olivier Hubaut
‣ ESI
Contact:
▸ http://nemerosa.com
▸ @DamienCoraboeuf
1 of 55

Recommended

7 Habits of Highly Effective Jenkins Users by
7 Habits of Highly Effective Jenkins Users7 Habits of Highly Effective Jenkins Users
7 Habits of Highly Effective Jenkins UsersJules Pierre-Louis
88 views36 slides
Jenkins days workshop pipelines - Eric Long by
Jenkins days workshop  pipelines - Eric LongJenkins days workshop  pipelines - Eric Long
Jenkins days workshop pipelines - Eric Longericlongtx
1.8K views44 slides
7 Habits of Highly Effective Jenkins Users by
7 Habits of Highly Effective Jenkins Users7 Habits of Highly Effective Jenkins Users
7 Habits of Highly Effective Jenkins UsersJules Pierre-Louis
337 views50 slides
Pipeline as code - new feature in Jenkins 2 by
Pipeline as code - new feature in Jenkins 2Pipeline as code - new feature in Jenkins 2
Pipeline as code - new feature in Jenkins 2Michal Ziarnik
2.3K views27 slides
Jenkins Pipeline @ Scale. Building Automation Frameworks for Systems Integration by
Jenkins Pipeline @ Scale. Building Automation Frameworks for Systems IntegrationJenkins Pipeline @ Scale. Building Automation Frameworks for Systems Integration
Jenkins Pipeline @ Scale. Building Automation Frameworks for Systems IntegrationOleg Nenashev
1.4K views66 slides
Building Jenkins Pipelines at Scale by
Building Jenkins Pipelines at ScaleBuilding Jenkins Pipelines at Scale
Building Jenkins Pipelines at ScaleJulien Pivotto
7.6K views73 slides

More Related Content

What's hot

Delivery Pipeline as Code: using Jenkins 2.0 Pipeline by
Delivery Pipeline as Code: using Jenkins 2.0 PipelineDelivery Pipeline as Code: using Jenkins 2.0 Pipeline
Delivery Pipeline as Code: using Jenkins 2.0 PipelineSlawa Giterman
9K views45 slides
JUC Europe 2015: Scaling Your Jenkins Master with Docker by
JUC Europe 2015: Scaling Your Jenkins Master with DockerJUC Europe 2015: Scaling Your Jenkins Master with Docker
JUC Europe 2015: Scaling Your Jenkins Master with DockerCloudBees
843 views24 slides
Jenkins, pipeline and docker by
Jenkins, pipeline and docker Jenkins, pipeline and docker
Jenkins, pipeline and docker AgileDenver
3K views48 slides
Voxxed Luxembourd 2016 Jenkins 2.0 et Pipeline as code by
Voxxed Luxembourd 2016 Jenkins 2.0 et Pipeline as codeVoxxed Luxembourd 2016 Jenkins 2.0 et Pipeline as code
Voxxed Luxembourd 2016 Jenkins 2.0 et Pipeline as codeDamien Duportal
655 views28 slides
JavaOne 2016 - Pipeline as code by
JavaOne 2016 - Pipeline as codeJavaOne 2016 - Pipeline as code
JavaOne 2016 - Pipeline as codeBert Jan Schrijver
750 views27 slides
JUC Europe 2015: Scaling of Jenkins Pipeline Creation and Maintenance by
JUC Europe 2015: Scaling of Jenkins Pipeline Creation and MaintenanceJUC Europe 2015: Scaling of Jenkins Pipeline Creation and Maintenance
JUC Europe 2015: Scaling of Jenkins Pipeline Creation and MaintenanceCloudBees
284 views17 slides

What's hot(20)

Delivery Pipeline as Code: using Jenkins 2.0 Pipeline by Slawa Giterman
Delivery Pipeline as Code: using Jenkins 2.0 PipelineDelivery Pipeline as Code: using Jenkins 2.0 Pipeline
Delivery Pipeline as Code: using Jenkins 2.0 Pipeline
Slawa Giterman9K views
JUC Europe 2015: Scaling Your Jenkins Master with Docker by CloudBees
JUC Europe 2015: Scaling Your Jenkins Master with DockerJUC Europe 2015: Scaling Your Jenkins Master with Docker
JUC Europe 2015: Scaling Your Jenkins Master with Docker
CloudBees843 views
Jenkins, pipeline and docker by AgileDenver
Jenkins, pipeline and docker Jenkins, pipeline and docker
Jenkins, pipeline and docker
AgileDenver3K views
Voxxed Luxembourd 2016 Jenkins 2.0 et Pipeline as code by Damien Duportal
Voxxed Luxembourd 2016 Jenkins 2.0 et Pipeline as codeVoxxed Luxembourd 2016 Jenkins 2.0 et Pipeline as code
Voxxed Luxembourd 2016 Jenkins 2.0 et Pipeline as code
Damien Duportal655 views
JUC Europe 2015: Scaling of Jenkins Pipeline Creation and Maintenance by CloudBees
JUC Europe 2015: Scaling of Jenkins Pipeline Creation and MaintenanceJUC Europe 2015: Scaling of Jenkins Pipeline Creation and Maintenance
JUC Europe 2015: Scaling of Jenkins Pipeline Creation and Maintenance
CloudBees284 views
Building an Extensible, Resumable DSL on Top of Apache Groovy by jgcloudbees
Building an Extensible, Resumable DSL on Top of Apache GroovyBuilding an Extensible, Resumable DSL on Top of Apache Groovy
Building an Extensible, Resumable DSL on Top of Apache Groovy
jgcloudbees842 views
CI/CD Pipeline as a Code using Jenkins 2 by Mayank Patel
CI/CD Pipeline as a Code using Jenkins 2CI/CD Pipeline as a Code using Jenkins 2
CI/CD Pipeline as a Code using Jenkins 2
Mayank Patel381 views
(Declarative) Jenkins Pipelines by Steffen Gebert
(Declarative) Jenkins Pipelines(Declarative) Jenkins Pipelines
(Declarative) Jenkins Pipelines
Steffen Gebert21.7K views
Pimp your Continuous Delivery Pipeline with Jenkins workflow (W-JAX 14) by CloudBees
Pimp your Continuous Delivery Pipeline with Jenkins workflow (W-JAX 14)Pimp your Continuous Delivery Pipeline with Jenkins workflow (W-JAX 14)
Pimp your Continuous Delivery Pipeline with Jenkins workflow (W-JAX 14)
CloudBees35.7K views
Automated Deployment Pipeline using Jenkins, Puppet, Mcollective and AWS by Bamdad Dashtban
Automated Deployment Pipeline using Jenkins, Puppet, Mcollective and AWSAutomated Deployment Pipeline using Jenkins, Puppet, Mcollective and AWS
Automated Deployment Pipeline using Jenkins, Puppet, Mcollective and AWS
Bamdad Dashtban29.9K views
SD DevOps Meet-up - Jenkins 2.0 and Pipeline-as-Code by Brian Dawson
SD DevOps Meet-up - Jenkins 2.0 and Pipeline-as-CodeSD DevOps Meet-up - Jenkins 2.0 and Pipeline-as-Code
SD DevOps Meet-up - Jenkins 2.0 and Pipeline-as-Code
Brian Dawson3.1K views
Javaone 2014 - Git & Docker with Jenkins by Andy Pemberton
Javaone 2014 - Git & Docker with JenkinsJavaone 2014 - Git & Docker with Jenkins
Javaone 2014 - Git & Docker with Jenkins
Andy Pemberton2.4K views
Testing with Docker by toffermann
Testing with DockerTesting with Docker
Testing with Docker
toffermann5.9K views
Jenkins vs. AWS CodePipeline (AWS User Group Berlin) by Steffen Gebert
Jenkins vs. AWS CodePipeline (AWS User Group Berlin)Jenkins vs. AWS CodePipeline (AWS User Group Berlin)
Jenkins vs. AWS CodePipeline (AWS User Group Berlin)
Steffen Gebert454 views
Codecoon - A technical Case Study by Michael Lihs
Codecoon - A technical Case StudyCodecoon - A technical Case Study
Codecoon - A technical Case Study
Michael Lihs3.7K views
Continuous Delivery with Jenkins Workflow by Udaypal Aarkoti
Continuous Delivery with Jenkins WorkflowContinuous Delivery with Jenkins Workflow
Continuous Delivery with Jenkins Workflow
Udaypal Aarkoti4.2K views
JUC Europe 2015: Plugin Development with Gradle and Groovy by CloudBees
JUC Europe 2015: Plugin Development with Gradle and GroovyJUC Europe 2015: Plugin Development with Gradle and Groovy
JUC Europe 2015: Plugin Development with Gradle and Groovy
CloudBees433 views

Viewers also liked

Utilizing SVN Jenkins to Manage Multi-line Development to Deployments by
Utilizing SVN  Jenkins to Manage Multi-line Development to DeploymentsUtilizing SVN  Jenkins to Manage Multi-line Development to Deployments
Utilizing SVN Jenkins to Manage Multi-line Development to DeploymentsTeresa Garcia-Bovenmyer ☁
402 views41 slides
Introduction to continuous integration by
Introduction to continuous integrationIntroduction to continuous integration
Introduction to continuous integrationPhilip Johnson
1.3K views13 slides
An Integrated Pipeline for Private and Public Clouds with Jenkins, Artifactor... by
An Integrated Pipeline for Private and Public Clouds with Jenkins, Artifactor...An Integrated Pipeline for Private and Public Clouds with Jenkins, Artifactor...
An Integrated Pipeline for Private and Public Clouds with Jenkins, Artifactor...VMware Tanzu
1.9K views15 slides
The DevOps Hero Toolkit: Nexus, Jenkins and Docker by
The DevOps Hero Toolkit: Nexus, Jenkins and DockerThe DevOps Hero Toolkit: Nexus, Jenkins and Docker
The DevOps Hero Toolkit: Nexus, Jenkins and DockerSeniorStoryteller
975 views6 slides
Pipeline: Continuous Delivery as Code in Jenkins 2.0 by
Pipeline: Continuous Delivery as Code in Jenkins 2.0Pipeline: Continuous Delivery as Code in Jenkins 2.0
Pipeline: Continuous Delivery as Code in Jenkins 2.0Jules Pierre-Louis
2.4K views36 slides
Rundeck + Nexus (from Nexus Live on June 5, 2014) by
Rundeck + Nexus (from Nexus Live on June 5, 2014)Rundeck + Nexus (from Nexus Live on June 5, 2014)
Rundeck + Nexus (from Nexus Live on June 5, 2014)dev2ops
20.9K views21 slides

Viewers also liked(18)

Introduction to continuous integration by Philip Johnson
Introduction to continuous integrationIntroduction to continuous integration
Introduction to continuous integration
Philip Johnson1.3K views
An Integrated Pipeline for Private and Public Clouds with Jenkins, Artifactor... by VMware Tanzu
An Integrated Pipeline for Private and Public Clouds with Jenkins, Artifactor...An Integrated Pipeline for Private and Public Clouds with Jenkins, Artifactor...
An Integrated Pipeline for Private and Public Clouds with Jenkins, Artifactor...
VMware Tanzu1.9K views
The DevOps Hero Toolkit: Nexus, Jenkins and Docker by SeniorStoryteller
The DevOps Hero Toolkit: Nexus, Jenkins and DockerThe DevOps Hero Toolkit: Nexus, Jenkins and Docker
The DevOps Hero Toolkit: Nexus, Jenkins and Docker
SeniorStoryteller975 views
Pipeline: Continuous Delivery as Code in Jenkins 2.0 by Jules Pierre-Louis
Pipeline: Continuous Delivery as Code in Jenkins 2.0Pipeline: Continuous Delivery as Code in Jenkins 2.0
Pipeline: Continuous Delivery as Code in Jenkins 2.0
Jules Pierre-Louis2.4K views
Rundeck + Nexus (from Nexus Live on June 5, 2014) by dev2ops
Rundeck + Nexus (from Nexus Live on June 5, 2014)Rundeck + Nexus (from Nexus Live on June 5, 2014)
Rundeck + Nexus (from Nexus Live on June 5, 2014)
dev2ops20.9K views
The Leadership Disconnect infograph by Todd Wheatland
The Leadership Disconnect infographThe Leadership Disconnect infograph
The Leadership Disconnect infograph
Todd Wheatland2.4K views
Educational Collisions: How Social Media Can Foster Professional Growth by Chris Wejr
Educational Collisions: How Social Media Can Foster Professional GrowthEducational Collisions: How Social Media Can Foster Professional Growth
Educational Collisions: How Social Media Can Foster Professional Growth
Chris Wejr3.3K views
Salesforce and Origamo by Capgemini
Salesforce and OrigamoSalesforce and Origamo
Salesforce and Origamo
Capgemini835 views
Dominant Transportation Privilege for Senior and Disabled people by DD Med Trans Inc.
Dominant Transportation Privilege for Senior and Disabled peopleDominant Transportation Privilege for Senior and Disabled people
Dominant Transportation Privilege for Senior and Disabled people
DD Med Trans Inc.1.5K views
Employee Health & Financial Wellness approach by Warren Handsor
Employee Health & Financial Wellness approachEmployee Health & Financial Wellness approach
Employee Health & Financial Wellness approach
Warren Handsor331 views
1. What is the structure of the gmat exam? GMAT FAQ by 4gmatprep
1. What is the structure of the gmat exam? GMAT FAQ1. What is the structure of the gmat exam? GMAT FAQ
1. What is the structure of the gmat exam? GMAT FAQ
4gmatprep818 views
Moving Beyond the Newsletter: Communicating Through Social Media by Chris Wejr
Moving Beyond the Newsletter: Communicating Through Social MediaMoving Beyond the Newsletter: Communicating Through Social Media
Moving Beyond the Newsletter: Communicating Through Social Media
Chris Wejr4.8K views
Rain water harvesting and geostatistical modelling of ground water in and aro... by DHIRENDRA PRATAP SINGH
Rain water harvesting and geostatistical modelling of ground water in and aro...Rain water harvesting and geostatistical modelling of ground water in and aro...
Rain water harvesting and geostatistical modelling of ground water in and aro...
Epic Content Marketing Australia with Marketo and Joe Pulizzi by Joe Pulizzi
Epic Content Marketing Australia with Marketo and Joe PulizziEpic Content Marketing Australia with Marketo and Joe Pulizzi
Epic Content Marketing Australia with Marketo and Joe Pulizzi
Joe Pulizzi16.9K views
내 인생의 영화 by guest69b42c
내 인생의 영화내 인생의 영화
내 인생의 영화
guest69b42c9.7K views

Similar to Brujug Jenkins pipeline scalability

Getting out of the Job Jungle with Jenkins by
Getting out of the Job Jungle with JenkinsGetting out of the Job Jungle with Jenkins
Getting out of the Job Jungle with JenkinsSonatype
548 views51 slides
Belgium jenkins-meetup-job-jungle-0.1 by
Belgium jenkins-meetup-job-jungle-0.1Belgium jenkins-meetup-job-jungle-0.1
Belgium jenkins-meetup-job-jungle-0.1Damien Coraboeuf
593 views51 slides
Ontrack - Keeping track of your CI/CD mess by
Ontrack - Keeping track of your CI/CD messOntrack - Keeping track of your CI/CD mess
Ontrack - Keeping track of your CI/CD messDamien Coraboeuf
646 views49 slides
Collibra wrojug-ontrack-20100424 by
Collibra wrojug-ontrack-20100424Collibra wrojug-ontrack-20100424
Collibra wrojug-ontrack-20100424Damien Coraboeuf
247 views50 slides
Chef vs Puppet vs Ansible vs SaltStack | Configuration Management Tools Compa... by
Chef vs Puppet vs Ansible vs SaltStack | Configuration Management Tools Compa...Chef vs Puppet vs Ansible vs SaltStack | Configuration Management Tools Compa...
Chef vs Puppet vs Ansible vs SaltStack | Configuration Management Tools Compa...Edureka!
5.7K views29 slides
Full stack development best practice and toolset by
Full stack development best practice and toolsetFull stack development best practice and toolset
Full stack development best practice and toolsetReid Lai
173 views77 slides

Similar to Brujug Jenkins pipeline scalability(20)

Getting out of the Job Jungle with Jenkins by Sonatype
Getting out of the Job Jungle with JenkinsGetting out of the Job Jungle with Jenkins
Getting out of the Job Jungle with Jenkins
Sonatype 548 views
Belgium jenkins-meetup-job-jungle-0.1 by Damien Coraboeuf
Belgium jenkins-meetup-job-jungle-0.1Belgium jenkins-meetup-job-jungle-0.1
Belgium jenkins-meetup-job-jungle-0.1
Damien Coraboeuf593 views
Ontrack - Keeping track of your CI/CD mess by Damien Coraboeuf
Ontrack - Keeping track of your CI/CD messOntrack - Keeping track of your CI/CD mess
Ontrack - Keeping track of your CI/CD mess
Damien Coraboeuf646 views
Chef vs Puppet vs Ansible vs SaltStack | Configuration Management Tools Compa... by Edureka!
Chef vs Puppet vs Ansible vs SaltStack | Configuration Management Tools Compa...Chef vs Puppet vs Ansible vs SaltStack | Configuration Management Tools Compa...
Chef vs Puppet vs Ansible vs SaltStack | Configuration Management Tools Compa...
Edureka!5.7K views
Full stack development best practice and toolset by Reid Lai
Full stack development best practice and toolsetFull stack development best practice and toolset
Full stack development best practice and toolset
Reid Lai173 views
Javaone - Gradle: Harder, Better, Stronger, Faster by Andres Almiray
Javaone - Gradle: Harder, Better, Stronger, Faster Javaone - Gradle: Harder, Better, Stronger, Faster
Javaone - Gradle: Harder, Better, Stronger, Faster
Andres Almiray2.2K views
Why we don’t use the Term DevOps: the Journey to a Product Mindset - DevOpsCo... by Henning Jacobs
Why we don’t use the Term DevOps: the Journey to a Product Mindset - DevOpsCo...Why we don’t use the Term DevOps: the Journey to a Product Mindset - DevOpsCo...
Why we don’t use the Term DevOps: the Journey to a Product Mindset - DevOpsCo...
Henning Jacobs1.4K views
VCS for Teamwork - GIT Workshop by Anis Ahmad
VCS for Teamwork - GIT WorkshopVCS for Teamwork - GIT Workshop
VCS for Teamwork - GIT Workshop
Anis Ahmad176 views
SH 1 - SES 4 - Microservices - Andrew Morgan TLV.pptx by MongoDB
SH 1 - SES 4 - Microservices - Andrew Morgan TLV.pptxSH 1 - SES 4 - Microservices - Andrew Morgan TLV.pptx
SH 1 - SES 4 - Microservices - Andrew Morgan TLV.pptx
MongoDB296 views
How to successfully migrate to Bazel from Maven or Gradle - Riga Dev Days by Natan Silnitsky
How to successfully migrate to Bazel from Maven or Gradle - Riga Dev DaysHow to successfully migrate to Bazel from Maven or Gradle - Riga Dev Days
How to successfully migrate to Bazel from Maven or Gradle - Riga Dev Days
Natan Silnitsky629 views
Zero to Continuous Delivery on Google Cloud by James Heggs
Zero to Continuous Delivery on Google CloudZero to Continuous Delivery on Google Cloud
Zero to Continuous Delivery on Google Cloud
James Heggs1.4K views
Grunt, Gulp & fabs: Build Systems and Development-Workflow for Modern Web-App... by Philipp Burgmer
Grunt, Gulp & fabs: Build Systems and Development-Workflow for Modern Web-App...Grunt, Gulp & fabs: Build Systems and Development-Workflow for Modern Web-App...
Grunt, Gulp & fabs: Build Systems and Development-Workflow for Modern Web-App...
Philipp Burgmer1.3K views
Powering Microservices with Docker, Kubernetes, Kafka, and MongoDB by MongoDB
Powering Microservices with Docker, Kubernetes, Kafka, and MongoDBPowering Microservices with Docker, Kubernetes, Kafka, and MongoDB
Powering Microservices with Docker, Kubernetes, Kafka, and MongoDB
MongoDB826 views
Continuous Integration with Open Source Tools - PHPUgFfm 2014-11-20 by Michael Lihs
Continuous Integration with Open Source Tools - PHPUgFfm 2014-11-20Continuous Integration with Open Source Tools - PHPUgFfm 2014-11-20
Continuous Integration with Open Source Tools - PHPUgFfm 2014-11-20
Michael Lihs4.5K views
Future Proof Development by Jeff Segars
Future Proof DevelopmentFuture Proof Development
Future Proof Development
Jeff Segars982 views
Real-World DevOps — 20 Practical Developers Tips for Tightening Your Operatio... by VictorSzoltysek
Real-World DevOps — 20 Practical Developers Tips for Tightening Your Operatio...Real-World DevOps — 20 Practical Developers Tips for Tightening Your Operatio...
Real-World DevOps — 20 Practical Developers Tips for Tightening Your Operatio...
VictorSzoltysek80 views
Developer Experience at Zalando - Handelsblatt Strategisches IT-Management 2019 by Henning Jacobs
Developer Experience at Zalando - Handelsblatt Strategisches IT-Management 2019Developer Experience at Zalando - Handelsblatt Strategisches IT-Management 2019
Developer Experience at Zalando - Handelsblatt Strategisches IT-Management 2019
Henning Jacobs2.1K views

Recently uploaded

REACTJS.pdf by
REACTJS.pdfREACTJS.pdf
REACTJS.pdfArthyR3
39 views16 slides
Solution Challenge Introduction.pptx by
Solution Challenge Introduction.pptxSolution Challenge Introduction.pptx
Solution Challenge Introduction.pptxGDSCCEC
13 views16 slides
Here comes the Loom - Ya!vaConf.pdf by
Here comes the Loom - Ya!vaConf.pdfHere comes the Loom - Ya!vaConf.pdf
Here comes the Loom - Ya!vaConf.pdfKrystian Zybała
7 views134 slides
Basic Design Flow for Field Programmable Gate Arrays by
Basic Design Flow for Field Programmable Gate ArraysBasic Design Flow for Field Programmable Gate Arrays
Basic Design Flow for Field Programmable Gate ArraysUsha Mehta
24 views21 slides
Unlocking Research Visibility.pdf by
Unlocking Research Visibility.pdfUnlocking Research Visibility.pdf
Unlocking Research Visibility.pdfKhatirNaima
11 views19 slides
Field Programmable Gate Arrays : Architecture by
Field Programmable Gate Arrays : ArchitectureField Programmable Gate Arrays : Architecture
Field Programmable Gate Arrays : ArchitectureUsha Mehta
33 views74 slides

Recently uploaded(20)

REACTJS.pdf by ArthyR3
REACTJS.pdfREACTJS.pdf
REACTJS.pdf
ArthyR339 views
Solution Challenge Introduction.pptx by GDSCCEC
Solution Challenge Introduction.pptxSolution Challenge Introduction.pptx
Solution Challenge Introduction.pptx
GDSCCEC13 views
Basic Design Flow for Field Programmable Gate Arrays by Usha Mehta
Basic Design Flow for Field Programmable Gate ArraysBasic Design Flow for Field Programmable Gate Arrays
Basic Design Flow for Field Programmable Gate Arrays
Usha Mehta24 views
Unlocking Research Visibility.pdf by KhatirNaima
Unlocking Research Visibility.pdfUnlocking Research Visibility.pdf
Unlocking Research Visibility.pdf
KhatirNaima11 views
Field Programmable Gate Arrays : Architecture by Usha Mehta
Field Programmable Gate Arrays : ArchitectureField Programmable Gate Arrays : Architecture
Field Programmable Gate Arrays : Architecture
Usha Mehta33 views
Ansari: Practical experiences with an LLM-based Islamic Assistant by M Waleed Kadous
Ansari: Practical experiences with an LLM-based Islamic AssistantAnsari: Practical experiences with an LLM-based Islamic Assistant
Ansari: Practical experiences with an LLM-based Islamic Assistant
M Waleed Kadous13 views
REPORT Data Science EXPERT LECTURE.doc by Parulkhatri11
REPORT Data Science EXPERT LECTURE.docREPORT Data Science EXPERT LECTURE.doc
REPORT Data Science EXPERT LECTURE.doc
Parulkhatri117 views
AWS Certified Solutions Architect Associate Exam Guide_published .pdf by Kiran Kumar Malik
AWS Certified Solutions Architect Associate Exam Guide_published .pdfAWS Certified Solutions Architect Associate Exam Guide_published .pdf
AWS Certified Solutions Architect Associate Exam Guide_published .pdf
ASSIGNMENTS ON FUZZY LOGIC IN TRAFFIC FLOW.pdf by AlhamduKure
ASSIGNMENTS ON FUZZY LOGIC IN TRAFFIC FLOW.pdfASSIGNMENTS ON FUZZY LOGIC IN TRAFFIC FLOW.pdf
ASSIGNMENTS ON FUZZY LOGIC IN TRAFFIC FLOW.pdf
AlhamduKure11 views
2023-12 Emarei MRI Tool Set E2I0501ST (TQ).pdf by Philipp Daum
2023-12 Emarei MRI Tool Set E2I0501ST (TQ).pdf2023-12 Emarei MRI Tool Set E2I0501ST (TQ).pdf
2023-12 Emarei MRI Tool Set E2I0501ST (TQ).pdf
Philipp Daum6 views
taylor-2005-classical-mechanics.pdf by ArturoArreola10
taylor-2005-classical-mechanics.pdftaylor-2005-classical-mechanics.pdf
taylor-2005-classical-mechanics.pdf
ArturoArreola1040 views
Different type of computer networks .pptx by nazmul1514788
Different  type of computer networks .pptxDifferent  type of computer networks .pptx
Different type of computer networks .pptx
nazmul151478820 views
MODULE-1 CHAPTER 3- Operators - Object Oriented Programming with JAVA by Demian Antony D'Mello
MODULE-1 CHAPTER 3- Operators - Object Oriented Programming with JAVAMODULE-1 CHAPTER 3- Operators - Object Oriented Programming with JAVA
MODULE-1 CHAPTER 3- Operators - Object Oriented Programming with JAVA
Design of Structures and Foundations for Vibrating Machines, Arya-ONeill-Pinc... by csegroupvn
Design of Structures and Foundations for Vibrating Machines, Arya-ONeill-Pinc...Design of Structures and Foundations for Vibrating Machines, Arya-ONeill-Pinc...
Design of Structures and Foundations for Vibrating Machines, Arya-ONeill-Pinc...
csegroupvn17 views

Brujug Jenkins pipeline scalability

  • 1. JOB SCALABILITY IN JENKINS GETTING OUT OF THE JOB JUNGLE BruJUG - March 9th, 2016 - @DamienCoraboeuf
  • 2. WHO AM I? ▸ Damien Coraboeuf (@DamienCoraboeuf) ▸ Born to Java in 1996 ▸ “Brol” engineer since 2008 ▸ Automating pipelines @ Clear2Pay / FIS & Multipharma I’ll be your guide on this journey
  • 3. THE MAP Short introduction to Jenkins Birth of a pipeline Manual mode The Job DSL Branching and its consequences Pipeline as code Pipeline as not code Automation of automation We want to get there Look, there is another
 beach here! Do not go there! 💀
  • 4. DO YOU KNOW WHO THIS IS?
  • 5. JENKINS ▸ http://jenkins-ci.org/ ▸ Very versatile Continuous Integration engine ▸ ~ 70% of the CI market (2012 figures) ▸ Open source - strong community ▸ Extensible ▸ more than 1000 plugins ▸ easy to develop new ones ▸ Job scheduler ▸ Integrates with basically everything
  • 6. JENKINS - JOBS & FOLDERS
  • 7. JENKINS - JOBS IN FOLDER
  • 10. JENKINS - JOB CONFIGURATION Let’s get some sources…
  • 11. JENKINS - JOB CONFIGURATION Let’s run some Bash…
  • 12. JENKINS - JOB CONFIGURATION Let’s launch some Gradle…
  • 13. JENKINS - JOB CONFIGURATION Let’s trigger another job…
  • 14. PIPELINES ▸ Defining jobs ▸ Linking them together ▸ Running them in parallel or in sequence ▸ Some triggers are automatic, other ones are manual BUILD DEPLOY ENV 1 DEPLOY ENV 2 DEPLOY ENV 3 PUBLICATION RELEASE
  • 15. A PIPELINE IS BORN BUILD DEPLOY ENV 1 DEPLOY ENV 2 DEPLOY ENV 3 PUBLICATION RELEASE BUILD DEPLOY ENV 1 DEPLOY ENV 2 DEPLOY ENV 3 PUBLICATION RELEASE BUILD DEPLOY ENV 1 DEPLOY ENV 2 DEPLOY ENV 3 PUBLICATION RELEASE Commit 1 Commit 2 Commit 3
  • 16. SO FAR SO GOOD “Look Ma, I’ve built a pipeline!” “Lovely. Now, build one for your sister.”
  • 17. THE STORY ▸ 40+ projects… ▸ …several maintenance & feature branches per project ▸ … complex validation pipelines (10 - 20 jobs) ▸ … thousands of jobs ▸ … only a small team for Jenkins Jenkins team > 2700 jobs and growing…
  • 19. In order to scale, we want: ✔ Self service ✔ Security ✔ Simplicity ✔ Extensibility
  • 20. “What are we?” “Developers!” “What do we do?” “Code!” “Why do we do it?” “Automation!”
  • 21. PIPELINE AS CODE ▸ The Job DSL ▸ Pipeline as code ▸ Pipeline libraries ▸ Pipeline as properties ▸ Pipeline automation
  • 22. JOB DSL PLUGIN - START OF THE JOURNEY OUT OF THE JUNGLE ▸ Define a job using a Groovy based DSL!
  • 23. JOB DSL PLUGIN ▸ Very well supported ▸ Very good documentation ▸ Supports most of the plugins ▸ Built-in extensibility: inline or DSL extensions
  • 24. JOB DSL PLUGIN ▸ Folders & views ▸ … automatic triggers upon generation ▸ … using files in workspace
  • 25. JOB DSL PLUG-IN ▸ This remains code…
  • 26. JOB DSL PLUGIN ▸ DSL based generation done from a job ▸ Can generate and/or update other jobs ▸ Can even run while jobs are running :) SEED JOB GENERATED JOB(S) DSL SCRIPT Generates Accesses (SCM or inline) GENERATED JOB(S) GENERATED JOB(S)
  • 27. BRANCHING ▸ One Seed DSL job is all very good but… ▸ Pipeline evolves with the code ▸ Different pipelines for different branches develop release/2.0 Needs a new job Pipelines for releases
 might be more complex
  • 28. PIPELINE AS CODE ▸ Your pipeline is linked to the code it builds ▸ Define your pipeline in your code Project src pom.xml job-dsl-script.groovy
  • 29. PIPELINE CODE DUPLICATION ▸ We can now generate a pipeline for any branch, any commit
 
 
 
 ▸ It evolves with your branches and is merged like any other piece of code ▸ That’s good enough for 1 project ▸ With several (many) projects, the level of DSL code duplication explodes! DEVELOP FEATURE/BIG RELEASE/1.0 DSL SCRIPT DSL SCRIPT DSL SCRIPT DEVELOP FEATURE/BIG RELEASE/1.0 DSL SCRIPT DSL SCRIPT DSL SCRIPT DEVELOP FEATURE/BIG RELEASE/1.0 DSL SCRIPT DSL SCRIPT DSL SCRIPT DEVELOP FEATURE/BIG RELEASE/1.0 DSL SCRIPT DSL SCRIPT DSL SCRIPT DEVELOP FEATURE/BIG RELEASE/1.0 DSL SCRIPT DSL SCRIPT DSL SCRIPT
  • 30. PIPELINE CODE DUPLICATION ▸ Job DSL code duplication has the same issues than production code duplication DEVELOP FEATURE/BIG RELEASE/1.0 DSL SCRIPT DSL SCRIPT DSL SCRIPT DEVELOP FEATURE/BIG RELEASE/1.0 DSL SCRIPT DSL SCRIPT DSL SCRIPT DEVELOP FEATURE/BIG RELEASE/1.0 DSL SCRIPT DSL SCRIPT DSL SCRIPT DEVELOP FEATURE/BIG RELEASE/1.0 DSL SCRIPT DSL SCRIPT DSL SCRIPT ✘ ✘ ✘ ✘ ✘ ✘ ✘✘✘✘✘✘ DEVELOP FEATURE/BIG RELEASE/1.0 DSL SCRIPT DSL SCRIPT DSL SCRIPT DEVELOP FEATURE/BIG RELEASE/1.0 DSL SCRIPT DSL SCRIPT DSL SCRIPT ✘✘✘✘✘✘ DEVELOP FEATURE/BIG RELEASE/1.0 DSL SCRIPT DSL SCRIPT DSL SCRIPT DEVELOP FEATURE/BIG RELEASE/1.0 DSL SCRIPT DSL SCRIPT DSL SCRIPT ✘✘✘✘✘✘ DEVELOP DSL SCRIPT DEVELOP DSL SCRIPT ✘ ✘ DEVELOP DSL SCRIPT ✘ DEVELOP DSL SCRIPT ✘ LEASE/1.0 L SCRIPT LEASE/1.0 L SCRIPT LEASE/1.0 L SCRIPT LEASE/1.0 L SCRIPT DEVELOP FEATURE/BIG RELEASE/1.0 DEVELOP FEATURE/BIG RELEASE/1.0 DEVELOPLEASE/1.0
  • 32. PIPELINE LIBRARIES TO THE RESCUE ▸ Pipeline is code ▸ Reuse of code through versioned libraries DEVELOP FEATURE/BIG RELEASE/1.0 DSL SCRIPT DSL SCRIPT DSL SCRIPT PIPELINE DSL LIBRARY 1.1 1.0 Normal project
 Can be tested
 Can be released
  • 33. PIPELINE LIBRARIES TO THE RESCUE ▸ DSL libraries as code libraries DEVELOP FEATURE/BIG RELEASE/1.0 DSL SCRIPT DSL SCRIPT DSL SCRIPT PIPELINE DSL LIBRARY 1.1 1.0 COMMON DSL LIBRARY 1.0 Dependencies
  • 34. ARE WE DONE? ▸ We can describe a pipeline using a DSL ▸ The pipeline is defined together with the code it builds ▸ The DSL can use libraries to reduce code duplication ?
  • 35. NOT QUITE… ▸ This is not enough to really scale ▸ We still have to write some DSL ▸ Self service ✔ ▸ Security ✘ ▸ Simplicity ✘ ▸ Extensibility ✔
  • 36. PIPELINE AS… PROPERTIES ▸ Let’s describe the pipeline using a properties file ▸ Which pipeline library & which version ▸ Configuration properties - specific to the library ▸ Property file format ubiquitous ▸ No code running on the master ▸ Can be used for reporting on all the pipelines!
  • 38. SEED PLUGIN ▸ https://github.com/jenkinsci/seed-plugin ▸ Allows to generate: ▸ projects folders ▸ branches pipelines ▸ based on seed files (properties and/or DSL) ▸ integration with hooks
  • 39. SEVERAL INTEGRATION MODES SEED.PROPERTIES SEED PLUGIN SEED.PROPERTIES SEED.GROOVYSEED.GROOVY PIPELINE LIBRARY PIPELINE LIBRARY BRANCH PIPELINE 95% 4% 1%
  • 40. GENERATION STRUCTURE SEED PROJECT GENERATOR PROJECT FOLDER BOOTSTRAPPING BRANCH GENERATOR BRANCH FOLDER BRANCH JOB 1 BRANCH JOB 2 BRANCH JOB 3 BRANCH JOB 4 GENERATION Project teamJenkins team Project only visible by project team
  • 43. HOOKS ▸ Generations can be automated using hooks at SCM level ▸ Support for GitHub, BitBucket, generic HTTP calls (for SVN) ▸ Configurable. For example: BRANCH CREATION SCM EVENT Seed plug-in COMMIT SEED CHANGED BRANCH DELETION PIPELINE EVENT Generation Triggers the pipeline Regenerates the pipeline Deletes the pipeline
  • 44. RESPONSIBILITIES Jenkins team Pipeline libraries Develops and maintains Development team seed.properties Defines Seed plug-in SCM Triggers
  • 45. WHAT DID WE JUST ACHIEVE? ▸ Automation of automation ▸ Self service ✔ ▸ Pipeline automation from SCM ▸ Security ✔ ▸ Project level authorisations ▸ No code on the master ▸ Simplicity ✔ ▸ Property files ▸ Extensibility ✔ ▸ Pipeline libraries ▸ Direct job DSL still possible
  • 46. YOU CAN NOW RELAX…
  • 47. DEMO TIME ▸ Bootstrapping a project ▸ Configuring the project for GitHub ▸ Creating a branch —> Branch folder generated ▸ Committing on the branch —> Pipeline started ▸ Updating the pipeline branch —> Pipeline regenerated ▸ Deleting the branch —> Branch folder gone
  • 48. DEMO - PIPELINE AS CODE seed/seed.groovy
  • 49. DEMO - GITHUB SETUP
  • 50. DEMO - JENKINS SETUP
  • 52. A GLIMPSE OF THE FUTURE ▸ Jenkins 2.0 ▸ Pipeline plug-in ▸ “Pipeline as Code” in the core ▸ How does the current solution fit?
  • 53. SEED & PIPELINE PLUGIN ▸ The Seed plugin will keep working ▸ It can already generate pipeline jobs (Job DSL) ▸ But overlap of functionalities Branch pipelines Pipeline as code SEED PIPELINE Hook integration Pipeline as properties Community & support Pipeline libraries Pipeline script library
  • 54. SEED IN PIPELINE PLUGIN ▸ The Seed plugin as extension of the Pipeline plugin Branch pipelines Pipeline as code SEED PIPELINE Hook integrationPipeline as properties Community & support Pipeline libraries Pipeline script library
  • 55. THANKS YOU! Thanks to: ‣ BruJUG ‣ Olivier Hubaut ‣ ESI Contact: ▸ http://nemerosa.com ▸ @DamienCoraboeuf