SlideShare a Scribd company logo
Jenkins User Conference Israel #jenkinsconf
Presenter Name
Ohad Basan
Presenter Company
Www.redhat.com
July 16, 2014
Managing jenkins with multiple
components project
#jenkinsconf
Who am I
● Ohad Basan
● CI engineer at Red Hat
● Member of the RHEV (Red Hat Enterprise
Virtualization) CI team
● Full time geek
Agenda
● Testing multiple components w/o loosing your
mind AKA 'Factory Flow'
● “Puppet for jenkins jobs”, possible?
● Giving developers access to automation infra
● Automate cool tasks in jenkins, foreman and
other devops tools via fabric
What is Factory flow?
Problems with testing multiple
components project at once
● Project is made out of multiple components that
are constantly changing and depend on each
other
● If two or more components change before the
tests being – you have no idea which
component broke the functionality
● We had to isolate the environment of each
component while keep the other component
stable.
What we had
●
Every job builds all the components
– Time consuming.
– Load on git.
– Error prone.
● In order to isolate the environments we created distinct jenkins
slaves for each component. Each job compiles its own product.
– Time Consuming
– Error prone.
– Increase load of git.
– Resource heavy. (many slaves)
Solution - FACTORY
Listener job > Polls for SCM changes.
Create custom rpms > Builds the relevant component
Add missing from stable – Links the compiled component with the rest of the missing stable
component and
Publish - publishes the files on a repository that contains all the needed components. (full project)
Automation jobs – Receive the repository link from the publisher, deploy it on the jenkins slaves
and run the tests.
How did we solve it? Multijob!
Multijob Jenkins Plugin
● Pros
– Really easy to configure
– Nice and comfortable view
● Cons
– Huge mess when running the the same job concurrently.
– Not flexible – can't dynamically chose which jobs to run.
– Can't pass dynamic parameters from upstream jobs.
Evolution – Build Flow Plugin
● Pros
– Very dynamic and flexible
– Highly maintained
● Cons
– View is not always comfortable (We customized it to
make it more readable)
– Graphs not always working
How do we configure so many jobs?
● We test a huge matrix
– Different operating systems versions
– Different product components
– Different product versions
– Backward compatible restrictions
– Layered products consumption (jboss, kvm)
Layered product
● Ovirt (Backend) > Application Server (JBOSS)
> Operating system (Red Hat Linux).
● VDSM (Hypervisor) > Libvirt > KVM (Kernel
Virtual Machine) > Operating system
● Red Hat does ALL the components and
EVERYTHING is changing all the time.
“Puppet for jenkins jobs”, possible?
Meet jenkins-job builder
● http://ci.openstack.org/jenkins_jobs.html
● This tool uses YAML files that act as template
file to generate jenkins jobs with different
values and configuration
Example of YAML file
Giving developers access to
automation infra
How can a dev test a patch?
● Unit tests
● Lint
● Very basic sanity
But for system tests, they need massive infra to
run it, almost impossible for a dev
Mission: Patches verification
● Every patch has to be verified before merging
● Some patches are really hard to verify and
require a complicated setup (several bare metal
machines, storage servers, complicated
configuration and preparation)
● The Solution...
Developer job
Developer job
● We give developers a special job to test their
patches with
● It will checkout their branch, build the
component that they chose and will run an
automation job that they can choose.
● Build flow to rescue.. (again!)
● Using the factory flow infrastructure.
Managing the system
● Keeping track on problems can get complicated
● Problems can originate in infrastructure
problems, automation problems, or product
problems.
● We are using python fabric task to maintain and
keep track on our environment.
What is fabric?
● Fabric is a python library and cli for easy
deployments through ssh
● http://www.fabfile.org/
● We use it as a wrapper for performing all sorts
of commands on our ci environment.
● We use Foreman api to use Fabric for foreman
tasks.
JENKINS TASKS
Foreman Tasks
one can perform a command on a
foreman query with fabric.
What is fabric?
● Fabric is a python library and cli for easy
deployments through ssh
● http://www.fabfile.org/
● We use it as a wrapper for performing all sorts
of commands on our ci environment.
● We use Foreman api to use Fabric for foreman
tasks.
Fabric view update tasks
● We keep track on automation problems with
JIRA ticket tracking system. For bugs we use
bugzilla.
● We have fabric task that keeps the jobs
description updated for tracking the bugs'
statuses. (using jira and bugzilla api)
● All you have to do is write BZ:XX in the
description and a cronjob with fabric will update
the status
The Future
● Provisioner – provisioning slaves dynamically at the
beginning of each test and then tearing them down at
the end. Provisioning from foreman hostgroup using an
openstack setup
● Nested virtualization – Run the hypervisor tests on a vm
using nested kvm.
● Adding more flow and components to the factory flow
● Adopting these solutions to other Red Hat products
Links
● Fabric - http://www.fabfile.org/
● Build flow - https://wiki.jenkins-
ci.org/display/JENKINS/Build+Flow+Plugin
● Multi job - https://wiki.jenkins-
ci.org/display/JENKINS/Multijob+Plugin
● Job builder - http://ci.openstack.org/jenkins-job-
builder/
● Foreman api - https://pypi.python.org/pypi/python-
foreman
Open Source
● Red Hat Enterprise Virtualization is using code
from the upstream project “oVirt” - an open
source virtual data center manager.
● http://www.ovirt.org
● Public gerrit is on gerrit.ovirt.org
● FEEL FREE TO CONTRIBUTE CODE!
● My E-mail: obasan@redhat.com
Jenkins User Conference Israel #jenkinsconf
Thank You To Our Sponsors
Platinum
Gold
Silver

More Related Content

What's hot

Large scale automation with jenkins
Large scale automation with jenkinsLarge scale automation with jenkins
Large scale automation with jenkins
Kohsuke Kawaguchi
 
Puppet & Jenkins
Puppet & JenkinsPuppet & Jenkins
Puppet & Jenkins
Matthew Barr
 
Pimp your Continuous Delivery Pipeline with Jenkins workflow (W-JAX 14)
Pimp your Continuous Delivery Pipeline with Jenkins workflow (W-JAX 14)Pimp your Continuous Delivery Pipeline with Jenkins workflow (W-JAX 14)
Pimp your Continuous Delivery Pipeline with Jenkins workflow (W-JAX 14)
CloudBees
 
Master Continuous Delivery with CloudBees Jenkins Platform
Master Continuous Delivery with CloudBees Jenkins PlatformMaster Continuous Delivery with CloudBees Jenkins Platform
Master Continuous Delivery with CloudBees Jenkins Platformdcjuengst
 
Automated Deployment Pipeline using Jenkins, Puppet, Mcollective and AWS
Automated Deployment Pipeline using Jenkins, Puppet, Mcollective and AWSAutomated Deployment Pipeline using Jenkins, Puppet, Mcollective and AWS
Automated Deployment Pipeline using Jenkins, Puppet, Mcollective and AWSBamdad Dashtban
 
Jenkins Workflow Webinar - Dec 10, 2014
Jenkins Workflow Webinar - Dec 10, 2014Jenkins Workflow Webinar - Dec 10, 2014
Jenkins Workflow Webinar - Dec 10, 2014CloudBees
 
Securing jenkins
Securing jenkinsSecuring jenkins
Securing jenkins
CloudBees
 
Javaone 2014 - Git & Docker with Jenkins
Javaone 2014 - Git & Docker with JenkinsJavaone 2014 - Git & Docker with Jenkins
Javaone 2014 - Git & Docker with Jenkins
Andy Pemberton
 
SD DevOps Meet-up - Jenkins 2.0 and Pipeline-as-Code
SD DevOps Meet-up - Jenkins 2.0 and Pipeline-as-CodeSD DevOps Meet-up - Jenkins 2.0 and Pipeline-as-Code
SD DevOps Meet-up - Jenkins 2.0 and Pipeline-as-Code
Brian Dawson
 
Continuous Delivery with Jenkins Workflow
Continuous Delivery with Jenkins WorkflowContinuous Delivery with Jenkins Workflow
Continuous Delivery with Jenkins Workflow
Udaypal Aarkoti
 
Rise of the Machines - Automate your Development
Rise of the Machines - Automate your DevelopmentRise of the Machines - Automate your Development
Rise of the Machines - Automate your Development
Sven Peters
 
Rundeck + Nexus (from Nexus Live on June 5, 2014)
Rundeck + Nexus (from Nexus Live on June 5, 2014)Rundeck + Nexus (from Nexus Live on June 5, 2014)
Rundeck + Nexus (from Nexus Live on June 5, 2014)
dev2ops
 
Using Docker for Testing
Using Docker for TestingUsing Docker for Testing
Using Docker for Testing
Carlos Sanchez
 
Yale Jenkins Show and Tell
Yale Jenkins Show and TellYale Jenkins Show and Tell
Yale Jenkins Show and TellE. Camden Fisher
 
Jenkins CI
Jenkins CIJenkins CI
Jenkins CI
Viyaan Jhiingade
 
Getting started with Jenkins
Getting started with JenkinsGetting started with Jenkins
Getting started with Jenkins
Edureka!
 
Let’s start Continuous Integration with jenkins
Let’s start Continuous Integration with jenkinsLet’s start Continuous Integration with jenkins
Let’s start Continuous Integration with jenkins
Tomohide Kakeya
 
Journée DevOps : De l'intégration continue au déploiement continu avec Jenkins
Journée DevOps : De l'intégration continue au déploiement continu avec JenkinsJournée DevOps : De l'intégration continue au déploiement continu avec Jenkins
Journée DevOps : De l'intégration continue au déploiement continu avec Jenkins
Publicis Sapient Engineering
 
Introduction to jenkins
Introduction to jenkinsIntroduction to jenkins
Introduction to jenkins
Abe Diaz
 
Jenkins Introduction
Jenkins IntroductionJenkins Introduction
Jenkins Introduction
Pavan Gupta
 

What's hot (20)

Large scale automation with jenkins
Large scale automation with jenkinsLarge scale automation with jenkins
Large scale automation with jenkins
 
Puppet & Jenkins
Puppet & JenkinsPuppet & Jenkins
Puppet & Jenkins
 
Pimp your Continuous Delivery Pipeline with Jenkins workflow (W-JAX 14)
Pimp your Continuous Delivery Pipeline with Jenkins workflow (W-JAX 14)Pimp your Continuous Delivery Pipeline with Jenkins workflow (W-JAX 14)
Pimp your Continuous Delivery Pipeline with Jenkins workflow (W-JAX 14)
 
Master Continuous Delivery with CloudBees Jenkins Platform
Master Continuous Delivery with CloudBees Jenkins PlatformMaster Continuous Delivery with CloudBees Jenkins Platform
Master Continuous Delivery with CloudBees Jenkins Platform
 
Automated Deployment Pipeline using Jenkins, Puppet, Mcollective and AWS
Automated Deployment Pipeline using Jenkins, Puppet, Mcollective and AWSAutomated Deployment Pipeline using Jenkins, Puppet, Mcollective and AWS
Automated Deployment Pipeline using Jenkins, Puppet, Mcollective and AWS
 
Jenkins Workflow Webinar - Dec 10, 2014
Jenkins Workflow Webinar - Dec 10, 2014Jenkins Workflow Webinar - Dec 10, 2014
Jenkins Workflow Webinar - Dec 10, 2014
 
Securing jenkins
Securing jenkinsSecuring jenkins
Securing jenkins
 
Javaone 2014 - Git & Docker with Jenkins
Javaone 2014 - Git & Docker with JenkinsJavaone 2014 - Git & Docker with Jenkins
Javaone 2014 - Git & Docker with Jenkins
 
SD DevOps Meet-up - Jenkins 2.0 and Pipeline-as-Code
SD DevOps Meet-up - Jenkins 2.0 and Pipeline-as-CodeSD DevOps Meet-up - Jenkins 2.0 and Pipeline-as-Code
SD DevOps Meet-up - Jenkins 2.0 and Pipeline-as-Code
 
Continuous Delivery with Jenkins Workflow
Continuous Delivery with Jenkins WorkflowContinuous Delivery with Jenkins Workflow
Continuous Delivery with Jenkins Workflow
 
Rise of the Machines - Automate your Development
Rise of the Machines - Automate your DevelopmentRise of the Machines - Automate your Development
Rise of the Machines - Automate your Development
 
Rundeck + Nexus (from Nexus Live on June 5, 2014)
Rundeck + Nexus (from Nexus Live on June 5, 2014)Rundeck + Nexus (from Nexus Live on June 5, 2014)
Rundeck + Nexus (from Nexus Live on June 5, 2014)
 
Using Docker for Testing
Using Docker for TestingUsing Docker for Testing
Using Docker for Testing
 
Yale Jenkins Show and Tell
Yale Jenkins Show and TellYale Jenkins Show and Tell
Yale Jenkins Show and Tell
 
Jenkins CI
Jenkins CIJenkins CI
Jenkins CI
 
Getting started with Jenkins
Getting started with JenkinsGetting started with Jenkins
Getting started with Jenkins
 
Let’s start Continuous Integration with jenkins
Let’s start Continuous Integration with jenkinsLet’s start Continuous Integration with jenkins
Let’s start Continuous Integration with jenkins
 
Journée DevOps : De l'intégration continue au déploiement continu avec Jenkins
Journée DevOps : De l'intégration continue au déploiement continu avec JenkinsJournée DevOps : De l'intégration continue au déploiement continu avec Jenkins
Journée DevOps : De l'intégration continue au déploiement continu avec Jenkins
 
Introduction to jenkins
Introduction to jenkinsIntroduction to jenkins
Introduction to jenkins
 
Jenkins Introduction
Jenkins IntroductionJenkins Introduction
Jenkins Introduction
 

Similar to Jenkinsconf Presentation - Advance jenkins management with multiple projects.

DrupalCon Los Angeles - Continuous Integration Toolbox
DrupalCon Los Angeles - Continuous Integration ToolboxDrupalCon Los Angeles - Continuous Integration Toolbox
DrupalCon Los Angeles - Continuous Integration Toolbox
Andrii Podanenko
 
CIbox - OpenSource solution for making your #devops better
CIbox - OpenSource solution for making your #devops betterCIbox - OpenSource solution for making your #devops better
CIbox - OpenSource solution for making your #devops better
Andrii Podanenko
 
Devops with Python by Yaniv Cohen DevopShift
Devops with Python by Yaniv Cohen DevopShiftDevops with Python by Yaniv Cohen DevopShift
Devops with Python by Yaniv Cohen DevopShift
Yaniv cohen
 
Continuous integration for open source distros v 3.0
Continuous integration for open source distros v 3.0Continuous integration for open source distros v 3.0
Continuous integration for open source distros v 3.0
Sriram Narayanan
 
MoldCamp - multidimentional testing workflow. CIBox.
MoldCamp  - multidimentional testing workflow. CIBox.MoldCamp  - multidimentional testing workflow. CIBox.
MoldCamp - multidimentional testing workflow. CIBox.
Andrii Podanenko
 
Jenkins 101: Continuos Integration with Jenkins
Jenkins 101: Continuos Integration with JenkinsJenkins 101: Continuos Integration with Jenkins
Jenkins 101: Continuos Integration with Jenkins
All Things Open
 
Deploying software at Scale
Deploying software at ScaleDeploying software at Scale
Deploying software at Scale
Kris Buytaert
 
Steamlining your puppet development workflow
Steamlining your puppet development workflowSteamlining your puppet development workflow
Steamlining your puppet development workflow
Tomas Doran
 
Puppet Camp New York 2014: Streamlining Puppet Development Workflow
Puppet Camp New York 2014: Streamlining Puppet Development Workflow Puppet Camp New York 2014: Streamlining Puppet Development Workflow
Puppet Camp New York 2014: Streamlining Puppet Development Workflow
Puppet
 
My "Perfect" Toolchain Setup for Grails Projects
My "Perfect" Toolchain Setup for Grails ProjectsMy "Perfect" Toolchain Setup for Grails Projects
My "Perfect" Toolchain Setup for Grails Projects
GR8Conf
 
CI from scratch with Jenkins (EN)
CI from scratch with Jenkins (EN)CI from scratch with Jenkins (EN)
CI from scratch with Jenkins (EN)
Borislav Traykov
 
Continuous Integration
Continuous IntegrationContinuous Integration
Continuous Integration
Joerg Henning
 
Puppet Camp Denver 2015: Developing and Testing with Enhanced Oscar
Puppet Camp Denver 2015: Developing and Testing with Enhanced OscarPuppet Camp Denver 2015: Developing and Testing with Enhanced Oscar
Puppet Camp Denver 2015: Developing and Testing with Enhanced Oscar
Puppet
 
Developing and Testing with Enhanced Oscar
Developing and Testing with Enhanced OscarDeveloping and Testing with Enhanced Oscar
Developing and Testing with Enhanced Oscar
Jeff Scelza
 
Road to sbt 1.0 paved with server
Road to sbt 1.0   paved with serverRoad to sbt 1.0   paved with server
Road to sbt 1.0 paved with server
Eugene Yokota
 
How to build a tool for operating Flink on Kubernetes
How to build a tool for operating Flink on KubernetesHow to build a tool for operating Flink on Kubernetes
How to build a tool for operating Flink on Kubernetes
AndreaMedeghini
 
Agile Software Development & Tools
Agile Software Development & ToolsAgile Software Development & Tools
Agile Software Development & Tools
Luismi Amorós Martínez
 
Automation - fabric, django and more
Automation - fabric, django and moreAutomation - fabric, django and more
Automation - fabric, django and more
Ilian Iliev
 
SCM Puppet: from an intro to the scaling
SCM Puppet: from an intro to the scalingSCM Puppet: from an intro to the scaling
SCM Puppet: from an intro to the scaling
Stanislav Osipov
 
A Fabric/Puppet Build/Deploy System
A Fabric/Puppet Build/Deploy SystemA Fabric/Puppet Build/Deploy System
A Fabric/Puppet Build/Deploy System
adrian_nye
 

Similar to Jenkinsconf Presentation - Advance jenkins management with multiple projects. (20)

DrupalCon Los Angeles - Continuous Integration Toolbox
DrupalCon Los Angeles - Continuous Integration ToolboxDrupalCon Los Angeles - Continuous Integration Toolbox
DrupalCon Los Angeles - Continuous Integration Toolbox
 
CIbox - OpenSource solution for making your #devops better
CIbox - OpenSource solution for making your #devops betterCIbox - OpenSource solution for making your #devops better
CIbox - OpenSource solution for making your #devops better
 
Devops with Python by Yaniv Cohen DevopShift
Devops with Python by Yaniv Cohen DevopShiftDevops with Python by Yaniv Cohen DevopShift
Devops with Python by Yaniv Cohen DevopShift
 
Continuous integration for open source distros v 3.0
Continuous integration for open source distros v 3.0Continuous integration for open source distros v 3.0
Continuous integration for open source distros v 3.0
 
MoldCamp - multidimentional testing workflow. CIBox.
MoldCamp  - multidimentional testing workflow. CIBox.MoldCamp  - multidimentional testing workflow. CIBox.
MoldCamp - multidimentional testing workflow. CIBox.
 
Jenkins 101: Continuos Integration with Jenkins
Jenkins 101: Continuos Integration with JenkinsJenkins 101: Continuos Integration with Jenkins
Jenkins 101: Continuos Integration with Jenkins
 
Deploying software at Scale
Deploying software at ScaleDeploying software at Scale
Deploying software at Scale
 
Steamlining your puppet development workflow
Steamlining your puppet development workflowSteamlining your puppet development workflow
Steamlining your puppet development workflow
 
Puppet Camp New York 2014: Streamlining Puppet Development Workflow
Puppet Camp New York 2014: Streamlining Puppet Development Workflow Puppet Camp New York 2014: Streamlining Puppet Development Workflow
Puppet Camp New York 2014: Streamlining Puppet Development Workflow
 
My "Perfect" Toolchain Setup for Grails Projects
My "Perfect" Toolchain Setup for Grails ProjectsMy "Perfect" Toolchain Setup for Grails Projects
My "Perfect" Toolchain Setup for Grails Projects
 
CI from scratch with Jenkins (EN)
CI from scratch with Jenkins (EN)CI from scratch with Jenkins (EN)
CI from scratch with Jenkins (EN)
 
Continuous Integration
Continuous IntegrationContinuous Integration
Continuous Integration
 
Puppet Camp Denver 2015: Developing and Testing with Enhanced Oscar
Puppet Camp Denver 2015: Developing and Testing with Enhanced OscarPuppet Camp Denver 2015: Developing and Testing with Enhanced Oscar
Puppet Camp Denver 2015: Developing and Testing with Enhanced Oscar
 
Developing and Testing with Enhanced Oscar
Developing and Testing with Enhanced OscarDeveloping and Testing with Enhanced Oscar
Developing and Testing with Enhanced Oscar
 
Road to sbt 1.0 paved with server
Road to sbt 1.0   paved with serverRoad to sbt 1.0   paved with server
Road to sbt 1.0 paved with server
 
How to build a tool for operating Flink on Kubernetes
How to build a tool for operating Flink on KubernetesHow to build a tool for operating Flink on Kubernetes
How to build a tool for operating Flink on Kubernetes
 
Agile Software Development & Tools
Agile Software Development & ToolsAgile Software Development & Tools
Agile Software Development & Tools
 
Automation - fabric, django and more
Automation - fabric, django and moreAutomation - fabric, django and more
Automation - fabric, django and more
 
SCM Puppet: from an intro to the scaling
SCM Puppet: from an intro to the scalingSCM Puppet: from an intro to the scaling
SCM Puppet: from an intro to the scaling
 
A Fabric/Puppet Build/Deploy System
A Fabric/Puppet Build/Deploy SystemA Fabric/Puppet Build/Deploy System
A Fabric/Puppet Build/Deploy System
 

Recently uploaded

Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...
Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...
Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...
Anthony Dahanne
 
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...
Globus
 
Understanding Globus Data Transfers with NetSage
Understanding Globus Data Transfers with NetSageUnderstanding Globus Data Transfers with NetSage
Understanding Globus Data Transfers with NetSage
Globus
 
May Marketo Masterclass, London MUG May 22 2024.pdf
May Marketo Masterclass, London MUG May 22 2024.pdfMay Marketo Masterclass, London MUG May 22 2024.pdf
May Marketo Masterclass, London MUG May 22 2024.pdf
Adele Miller
 
Vitthal Shirke Microservices Resume Montevideo
Vitthal Shirke Microservices Resume MontevideoVitthal Shirke Microservices Resume Montevideo
Vitthal Shirke Microservices Resume Montevideo
Vitthal Shirke
 
Dominate Social Media with TubeTrivia AI’s Addictive Quiz Videos.pdf
Dominate Social Media with TubeTrivia AI’s Addictive Quiz Videos.pdfDominate Social Media with TubeTrivia AI’s Addictive Quiz Videos.pdf
Dominate Social Media with TubeTrivia AI’s Addictive Quiz Videos.pdf
AMB-Review
 
Lecture 1 Introduction to games development
Lecture 1 Introduction to games developmentLecture 1 Introduction to games development
Lecture 1 Introduction to games development
abdulrafaychaudhry
 
First Steps with Globus Compute Multi-User Endpoints
First Steps with Globus Compute Multi-User EndpointsFirst Steps with Globus Compute Multi-User Endpoints
First Steps with Globus Compute Multi-User Endpoints
Globus
 
A Sighting of filterA in Typelevel Rite of Passage
A Sighting of filterA in Typelevel Rite of PassageA Sighting of filterA in Typelevel Rite of Passage
A Sighting of filterA in Typelevel Rite of Passage
Philip Schwarz
 
AI Pilot Review: The World’s First Virtual Assistant Marketing Suite
AI Pilot Review: The World’s First Virtual Assistant Marketing SuiteAI Pilot Review: The World’s First Virtual Assistant Marketing Suite
AI Pilot Review: The World’s First Virtual Assistant Marketing Suite
Google
 
RISE with SAP and Journey to the Intelligent Enterprise
RISE with SAP and Journey to the Intelligent EnterpriseRISE with SAP and Journey to the Intelligent Enterprise
RISE with SAP and Journey to the Intelligent Enterprise
Srikant77
 
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERROR
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERRORTROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERROR
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERROR
Tier1 app
 
Corporate Management | Session 3 of 3 | Tendenci AMS
Corporate Management | Session 3 of 3 | Tendenci AMSCorporate Management | Session 3 of 3 | Tendenci AMS
Corporate Management | Session 3 of 3 | Tendenci AMS
Tendenci - The Open Source AMS (Association Management Software)
 
Webinar: Salesforce Document Management 2.0 - Smarter, Faster, Better
Webinar: Salesforce Document Management 2.0 - Smarter, Faster, BetterWebinar: Salesforce Document Management 2.0 - Smarter, Faster, Better
Webinar: Salesforce Document Management 2.0 - Smarter, Faster, Better
XfilesPro
 
GlobusWorld 2024 Opening Keynote session
GlobusWorld 2024 Opening Keynote sessionGlobusWorld 2024 Opening Keynote session
GlobusWorld 2024 Opening Keynote session
Globus
 
Graphic Design Crash Course for beginners
Graphic Design Crash Course for beginnersGraphic Design Crash Course for beginners
Graphic Design Crash Course for beginners
e20449
 
Enhancing Project Management Efficiency_ Leveraging AI Tools like ChatGPT.pdf
Enhancing Project Management Efficiency_ Leveraging AI Tools like ChatGPT.pdfEnhancing Project Management Efficiency_ Leveraging AI Tools like ChatGPT.pdf
Enhancing Project Management Efficiency_ Leveraging AI Tools like ChatGPT.pdf
Jay Das
 
Providing Globus Services to Users of JASMIN for Environmental Data Analysis
Providing Globus Services to Users of JASMIN for Environmental Data AnalysisProviding Globus Services to Users of JASMIN for Environmental Data Analysis
Providing Globus Services to Users of JASMIN for Environmental Data Analysis
Globus
 
BoxLang: Review our Visionary Licenses of 2024
BoxLang: Review our Visionary Licenses of 2024BoxLang: Review our Visionary Licenses of 2024
BoxLang: Review our Visionary Licenses of 2024
Ortus Solutions, Corp
 
Cracking the code review at SpringIO 2024
Cracking the code review at SpringIO 2024Cracking the code review at SpringIO 2024
Cracking the code review at SpringIO 2024
Paco van Beckhoven
 

Recently uploaded (20)

Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...
Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...
Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...
 
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...
 
Understanding Globus Data Transfers with NetSage
Understanding Globus Data Transfers with NetSageUnderstanding Globus Data Transfers with NetSage
Understanding Globus Data Transfers with NetSage
 
May Marketo Masterclass, London MUG May 22 2024.pdf
May Marketo Masterclass, London MUG May 22 2024.pdfMay Marketo Masterclass, London MUG May 22 2024.pdf
May Marketo Masterclass, London MUG May 22 2024.pdf
 
Vitthal Shirke Microservices Resume Montevideo
Vitthal Shirke Microservices Resume MontevideoVitthal Shirke Microservices Resume Montevideo
Vitthal Shirke Microservices Resume Montevideo
 
Dominate Social Media with TubeTrivia AI’s Addictive Quiz Videos.pdf
Dominate Social Media with TubeTrivia AI’s Addictive Quiz Videos.pdfDominate Social Media with TubeTrivia AI’s Addictive Quiz Videos.pdf
Dominate Social Media with TubeTrivia AI’s Addictive Quiz Videos.pdf
 
Lecture 1 Introduction to games development
Lecture 1 Introduction to games developmentLecture 1 Introduction to games development
Lecture 1 Introduction to games development
 
First Steps with Globus Compute Multi-User Endpoints
First Steps with Globus Compute Multi-User EndpointsFirst Steps with Globus Compute Multi-User Endpoints
First Steps with Globus Compute Multi-User Endpoints
 
A Sighting of filterA in Typelevel Rite of Passage
A Sighting of filterA in Typelevel Rite of PassageA Sighting of filterA in Typelevel Rite of Passage
A Sighting of filterA in Typelevel Rite of Passage
 
AI Pilot Review: The World’s First Virtual Assistant Marketing Suite
AI Pilot Review: The World’s First Virtual Assistant Marketing SuiteAI Pilot Review: The World’s First Virtual Assistant Marketing Suite
AI Pilot Review: The World’s First Virtual Assistant Marketing Suite
 
RISE with SAP and Journey to the Intelligent Enterprise
RISE with SAP and Journey to the Intelligent EnterpriseRISE with SAP and Journey to the Intelligent Enterprise
RISE with SAP and Journey to the Intelligent Enterprise
 
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERROR
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERRORTROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERROR
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERROR
 
Corporate Management | Session 3 of 3 | Tendenci AMS
Corporate Management | Session 3 of 3 | Tendenci AMSCorporate Management | Session 3 of 3 | Tendenci AMS
Corporate Management | Session 3 of 3 | Tendenci AMS
 
Webinar: Salesforce Document Management 2.0 - Smarter, Faster, Better
Webinar: Salesforce Document Management 2.0 - Smarter, Faster, BetterWebinar: Salesforce Document Management 2.0 - Smarter, Faster, Better
Webinar: Salesforce Document Management 2.0 - Smarter, Faster, Better
 
GlobusWorld 2024 Opening Keynote session
GlobusWorld 2024 Opening Keynote sessionGlobusWorld 2024 Opening Keynote session
GlobusWorld 2024 Opening Keynote session
 
Graphic Design Crash Course for beginners
Graphic Design Crash Course for beginnersGraphic Design Crash Course for beginners
Graphic Design Crash Course for beginners
 
Enhancing Project Management Efficiency_ Leveraging AI Tools like ChatGPT.pdf
Enhancing Project Management Efficiency_ Leveraging AI Tools like ChatGPT.pdfEnhancing Project Management Efficiency_ Leveraging AI Tools like ChatGPT.pdf
Enhancing Project Management Efficiency_ Leveraging AI Tools like ChatGPT.pdf
 
Providing Globus Services to Users of JASMIN for Environmental Data Analysis
Providing Globus Services to Users of JASMIN for Environmental Data AnalysisProviding Globus Services to Users of JASMIN for Environmental Data Analysis
Providing Globus Services to Users of JASMIN for Environmental Data Analysis
 
BoxLang: Review our Visionary Licenses of 2024
BoxLang: Review our Visionary Licenses of 2024BoxLang: Review our Visionary Licenses of 2024
BoxLang: Review our Visionary Licenses of 2024
 
Cracking the code review at SpringIO 2024
Cracking the code review at SpringIO 2024Cracking the code review at SpringIO 2024
Cracking the code review at SpringIO 2024
 

Jenkinsconf Presentation - Advance jenkins management with multiple projects.

  • 1. Jenkins User Conference Israel #jenkinsconf Presenter Name Ohad Basan Presenter Company Www.redhat.com July 16, 2014 Managing jenkins with multiple components project #jenkinsconf
  • 2. Who am I ● Ohad Basan ● CI engineer at Red Hat ● Member of the RHEV (Red Hat Enterprise Virtualization) CI team ● Full time geek
  • 3. Agenda ● Testing multiple components w/o loosing your mind AKA 'Factory Flow' ● “Puppet for jenkins jobs”, possible? ● Giving developers access to automation infra ● Automate cool tasks in jenkins, foreman and other devops tools via fabric
  • 5.
  • 6. Problems with testing multiple components project at once ● Project is made out of multiple components that are constantly changing and depend on each other ● If two or more components change before the tests being – you have no idea which component broke the functionality ● We had to isolate the environment of each component while keep the other component stable.
  • 7. What we had ● Every job builds all the components – Time consuming. – Load on git. – Error prone. ● In order to isolate the environments we created distinct jenkins slaves for each component. Each job compiles its own product. – Time Consuming – Error prone. – Increase load of git. – Resource heavy. (many slaves)
  • 8. Solution - FACTORY Listener job > Polls for SCM changes. Create custom rpms > Builds the relevant component Add missing from stable – Links the compiled component with the rest of the missing stable component and Publish - publishes the files on a repository that contains all the needed components. (full project) Automation jobs – Receive the repository link from the publisher, deploy it on the jenkins slaves and run the tests.
  • 9. How did we solve it? Multijob!
  • 10. Multijob Jenkins Plugin ● Pros – Really easy to configure – Nice and comfortable view ● Cons – Huge mess when running the the same job concurrently. – Not flexible – can't dynamically chose which jobs to run. – Can't pass dynamic parameters from upstream jobs.
  • 11. Evolution – Build Flow Plugin ● Pros – Very dynamic and flexible – Highly maintained ● Cons – View is not always comfortable (We customized it to make it more readable) – Graphs not always working
  • 12.
  • 13.
  • 14. How do we configure so many jobs? ● We test a huge matrix – Different operating systems versions – Different product components – Different product versions – Backward compatible restrictions – Layered products consumption (jboss, kvm)
  • 15. Layered product ● Ovirt (Backend) > Application Server (JBOSS) > Operating system (Red Hat Linux). ● VDSM (Hypervisor) > Libvirt > KVM (Kernel Virtual Machine) > Operating system ● Red Hat does ALL the components and EVERYTHING is changing all the time.
  • 16.
  • 17. “Puppet for jenkins jobs”, possible? Meet jenkins-job builder ● http://ci.openstack.org/jenkins_jobs.html ● This tool uses YAML files that act as template file to generate jenkins jobs with different values and configuration
  • 19.
  • 20. Giving developers access to automation infra How can a dev test a patch? ● Unit tests ● Lint ● Very basic sanity But for system tests, they need massive infra to run it, almost impossible for a dev
  • 21.
  • 22. Mission: Patches verification ● Every patch has to be verified before merging ● Some patches are really hard to verify and require a complicated setup (several bare metal machines, storage servers, complicated configuration and preparation) ● The Solution...
  • 24. Developer job ● We give developers a special job to test their patches with ● It will checkout their branch, build the component that they chose and will run an automation job that they can choose. ● Build flow to rescue.. (again!) ● Using the factory flow infrastructure.
  • 25.
  • 26. Managing the system ● Keeping track on problems can get complicated ● Problems can originate in infrastructure problems, automation problems, or product problems. ● We are using python fabric task to maintain and keep track on our environment.
  • 27. What is fabric? ● Fabric is a python library and cli for easy deployments through ssh ● http://www.fabfile.org/ ● We use it as a wrapper for performing all sorts of commands on our ci environment. ● We use Foreman api to use Fabric for foreman tasks.
  • 29.
  • 30. one can perform a command on a foreman query with fabric.
  • 31. What is fabric? ● Fabric is a python library and cli for easy deployments through ssh ● http://www.fabfile.org/ ● We use it as a wrapper for performing all sorts of commands on our ci environment. ● We use Foreman api to use Fabric for foreman tasks.
  • 32. Fabric view update tasks ● We keep track on automation problems with JIRA ticket tracking system. For bugs we use bugzilla. ● We have fabric task that keeps the jobs description updated for tracking the bugs' statuses. (using jira and bugzilla api) ● All you have to do is write BZ:XX in the description and a cronjob with fabric will update the status
  • 33.
  • 34. The Future ● Provisioner – provisioning slaves dynamically at the beginning of each test and then tearing them down at the end. Provisioning from foreman hostgroup using an openstack setup ● Nested virtualization – Run the hypervisor tests on a vm using nested kvm. ● Adding more flow and components to the factory flow ● Adopting these solutions to other Red Hat products
  • 35. Links ● Fabric - http://www.fabfile.org/ ● Build flow - https://wiki.jenkins- ci.org/display/JENKINS/Build+Flow+Plugin ● Multi job - https://wiki.jenkins- ci.org/display/JENKINS/Multijob+Plugin ● Job builder - http://ci.openstack.org/jenkins-job- builder/ ● Foreman api - https://pypi.python.org/pypi/python- foreman
  • 36. Open Source ● Red Hat Enterprise Virtualization is using code from the upstream project “oVirt” - an open source virtual data center manager. ● http://www.ovirt.org ● Public gerrit is on gerrit.ovirt.org ● FEEL FREE TO CONTRIBUTE CODE! ● My E-mail: obasan@redhat.com
  • 37.
  • 38. Jenkins User Conference Israel #jenkinsconf Thank You To Our Sponsors Platinum Gold Silver