SlideShare a Scribd company logo
1 of 42
Download to read offline
Running a production Jenkins instance
Harpreet Singh,
Senior Director, Product Management

Kohsuke Kawaguchi
Jenkins founder


                 ©2012 CloudBees, Inc. All Rights Reserved
Agenda
• Failures – a fact of life
  – Getting ready for failures
  – Preventing failures
  – Debugging failures
• Run an efficient Jenkins installation




               ©2012 CloudBees, Inc. All Rights Reserved   2
Day: A period of 24 hours, mostly
misspent…
              ©2011 CloudBees, Inc. All Rights   3
                        Reserved
©2011 CloudBees, Inc. All Rights   4
          Reserved
CloudBees – Who are we?
•   Jenkins founder on-board
•   Key Jenkins contributors on-board
•   Built Jenkins as a Service
•   Run the biggest Jenkins installation
    anywhere (2k+) masters




                ©2011 CloudBees, Inc. All Rights Reserved   5
CloudBees’ Mission - Eliminate
Downtime
• Eliminate time wasted due to
  – Jenkins issues
  – User issues
  – Lack of right tools…
• Improve efficiency for administrators and
  developers
• Rely on Jenkins…



               ©2011 CloudBees, Inc. All Rights Reserved   6
Good Management of Jenkins
•   Organize jobs better
•   Secure your jobs
•   Replicate good practices
•   Respond quicker to requests
•   Ensure compliance
•   Bounce back from failures
•   Prevent failures
•   Everything should be as fast as possible…if
    not faster

                 ©2011 CloudBees, Inc. All Rights Reserved   7
Recovering from failures
High Availability, Backing up




                    ©2011 CloudBees, Inc. All Rights
                              Reserved
Backing up Jenkins
                                          Jenkins Enterprise
Problem: Disk Failures                    Solution
• JENKINS_HOME                            • Backup plugin
   – Plugins, users,
     jobs…everything
                                          • Backup-to-cloud

Solution: Back it up
• Push HOME to a repo
   – HOME tends to be large
   – Commit only vital info
   – Run nightly
• Push to S3

                       ©2011 CloudBees, Inc. All Rights        9
                                 Reserved
JE Backup Plugin
• Backup as a Jenkins                    • Where to backup
  job                                           – Local Directory
• What to backup                                – Sftp server
  – Job configuration                           – WebDav
  – Build records                        • Retention Policy
  – System                                      – All
    Configuration                               – Last N
     • Plugin binaries, plugin
                                                – Exponential decay
       configs etc
     • Everything except job



                      ©2011 CloudBees, Inc. All Rights                10
                                Reserved
Demo




       ©2011 CloudBees, Inc. All Rights   11
                 Reserved
Making Jenkins Highly Available
                                                       Jenkins Enterprise
Problem: Jenkins failures                              Solution
•   Machine/Jenkins failure has                        •    Highly Available
    high cost to productivity                                – Setup multiple Jenkins
                                                               masters
Solution: Notified by unhappy                                – Uses jgroups to elect a
customers ;-)                                                  primary master
•   Issues:                                                  – Promotes a backup master
     –   Receive emails from unhappy                           as primary
         customers and log in and fix it
•   You do have JENKINS_HOME
    backed up else where – don’t
    you?




                                    ©2011 CloudBees, Inc. All Rights                      12
                                              Reserved
Bounce Back Faster: High Availability

       Reverse Proxy                                      Reverse Proxy




                                                                     Jenkins
  Jenkins           Jenkins                                          Master
  Master            Master

                                                                           MT
                           MT                           Jenkins Cluster
     Jenkins Cluster



            JENKINS_HOME                                   JENKINS_HOME
                                                                NFS




                                ©2011 CloudBees, Inc. All Rights                13
                                          Reserved
Demo




       ©2011 CloudBees, Inc. All Rights   14
                 Reserved
Miscellaneous
• Jenkins is not just JENKINS_HOME…think about the slaves
   – Offload builds onto slaves
   – Other executables on the system: git, ruby, java etc as well
   – Preferably use Chef/Puppet to replicate installations
• What about geo redundancy?
   – Technically you can use HA but network latency comes in play
   – Ideally, use HA in a localized data center and a manual failover
     to a different geo
• What HA is not?
   – Does not load balance between instances




                      ©2011 CloudBees, Inc. All Rights Reserved     15
Preventing failures
Git Validated Merges plugin




                   ©2011 CloudBees, Inc. All Rights
                             Reserved
How can you delegate more to Jenkins?
• Does your CI server shift work from
  laptops to servers?
  – You need to commit to have Jenkins test it
  – But if your commit is bad, it blocks others
  – You end up testing locally before committing
  – FAIL




                                                   17
Motivation
• We want to make changes safely
  – Your mistake shouldn’t block others
  – Only push after changes are validated
• We want to run tests asynchronously
  – Your brain has more important things to do
  – Make change and move on
  – Even with TDD!
• We want to run tests on the server
  – Your laptop has more important things to do

                                                  18
Solution: Jenkins should be Git server
•   I push to Jenkins
•   Jenkins merges it with upstream
•   Jenkins tests it
•   If good, Jenkins pushes it upstream


                                    upstream
                                      repo
                        gate
                        repo

                                               19
Another way to look at it

                  Tip of master in upstream
                          Tip of master in upstream




                   My changes




                                                      20
Implementation
• Transport
  – HTTP
  – SSH
• JGit embedded in Jenkins for git server
  functionality
  – A bit of magic like Gerrit to make it seamless
• Additional tags to let you pull submitted
  changes


                                                     21
Demo




       ©2011 CloudBees, Inc. All Rights   22
                 Reserved
Running an efficient production
system




           ©2011 CloudBees, Inc. All Rights
                     Reserved
Test Instance
• Run mini 2nd instance
  – Test new core version before putting it to
    prod
  – Test new versions of plugins
  – Play with new plugins
• Copy over some jobs from prod

• Bootstrap dry-run
  – -Djenkins.model.Jenkins.killAfterLoad=true
               ©2011 CloudBees, Inc. All Rights Reserved   24
Configuring Jenkins for efficiency
• Fast archiver plugin
  – Conserve network bandwidth
• No build on master
  – Also good for security




               ©2011 CloudBees, Inc. All Rights Reserved   25
Managing and Pruning Plugins
                                                  Jenkins Enterprise
Problem: Discovering what
plugins are used in an                            Solution
installation                                      •    Plugin Usage Plugin
•   No visibility if a particular plugin                – Tabular view of Plugin
    is used or how many jobs use it                        name, # of jobs and the job
                                                           names using the plugin




                               ©2011 CloudBees, Inc. All Rights                          26
                                         Reserved
Demo




       ©2011 CloudBees, Inc. All Rights   27
                 Reserved
Monitoring Jenkins




          ©2012 CloudBees, Inc. All Rights
                    Reserved
Why?




       ©2011 CloudBees, Inc. All Rights Reserved   29
What?
• What the user sees
    – GUI (load time)
• JVM memory size
    – Beware of several independent pieces
•   System load
•   Free space on $JENKINS_HOME
•   Slave availability
•   Queue length

                 ©2011 CloudBees, Inc. All Rights Reserved   30
Groovy Console
$ cat queue.groovy
j=Jenkins.instances
println j.queue.items.length

$ curl –u "user:apiToken“ 
  –data-urlencode script@queue.groovy 
  http://jenkins/scriptText
13




                                          31
Remote API
$ curl http://jenkins/computer/api/json?pretty=true
{
  busyExecutors: 0,
  totalExecutors: 2,
  ...
}




                                                  32
Jenkins Monitoring plugin
• JavaMelody in Jenkins




             ©2011 CloudBees, Inc. All Rights Reserved   33
Nagios (or others like it)
• Server app for monitoring
  stuff
  – Extensible, allowing all sorts
    of things to be monitored
• Used in jenkins-ci.org/DEV@cloud




                ©2011 CloudBees, Inc. All Rights Reserved   34
©2011 CloudBees, Inc. All Rights Reserved   35
Thread dump
• Tells us where Jenkins is stuck
• When?
  – Hang or slowness
• Look for threads that’s stuck
  – HTTP request threads
  – Executor threads




              ©2011 CloudBees, Inc. All Rights Reserved   36
How to get a thread dump
• http://jenkins/threadDump
• kill -3 <PID>




             ©2011 CloudBees, Inc. All Rights Reserved   37
Heap dump
• Tells us what’s eating memory
• When?
  – OutOfMemoryError
  – Monitoring shows abnormal growth
• Look for objects that are big
  – Sessions
  – Classes from plugins



              ©2011 CloudBees, Inc. All Rights Reserved   38
How to get a memory dump
• curl –L http://jenkins/heapDump >
  dump.hprof
• jmap -dump:format=b,file=dump.hprof
  PID
• -XX:+HeapDumpOnOutOfMemoryError




            ©2011 CloudBees, Inc. All Rights Reserved   39
Wrapping up

                             Thank You!

 More Info       http://www.cloudbees.com/jenkins-enterprise-by-cloudbees-overview.cb

  Free Trial    http://www.cloudbees.com/jenkins-enterprise-by-cloudbees-download.cb

  Wiki Page        https://wiki.cloudbees.com/bin/view/Jenkins+Enterprise/WebHome

 User Guide    http://jenkins-enterprise.cloudbees.com/docs/user-guide-bundle/index.html#




                          ©2011 CloudBees, Inc. All Rights                                  40
                                    Reserved
Day: A period of 24 hours, mostly
misspent…      ©2011 CloudBees, Inc. All Rights   41
                         Reserved
©2012 CloudBees, Inc. All Rights
          Reserved

More Related Content

What's hot

Novell Success Stories: Endpoint Management in Government
Novell Success Stories: Endpoint Management in GovernmentNovell Success Stories: Endpoint Management in Government
Novell Success Stories: Endpoint Management in GovernmentNovell
 
Accelerating Server Hardware Upgrades with PlateSpin Migrate P2P
Accelerating Server Hardware Upgrades with PlateSpin Migrate P2PAccelerating Server Hardware Upgrades with PlateSpin Migrate P2P
Accelerating Server Hardware Upgrades with PlateSpin Migrate P2PNovell
 
Beyond Disaster Recovery: Restoring Production Workloads with PlateSpin Forge
Beyond Disaster Recovery: Restoring Production Workloads with PlateSpin ForgeBeyond Disaster Recovery: Restoring Production Workloads with PlateSpin Forge
Beyond Disaster Recovery: Restoring Production Workloads with PlateSpin ForgeNovell
 
Application Live Migration in LAN/WAN Environment
Application Live Migration in LAN/WAN EnvironmentApplication Live Migration in LAN/WAN Environment
Application Live Migration in LAN/WAN EnvironmentMahendra Kutare
 
Kernel Recipes 2014 - Xen as a foundation for cloud infrastructure
Kernel Recipes 2014 - Xen as a foundation for cloud infrastructureKernel Recipes 2014 - Xen as a foundation for cloud infrastructure
Kernel Recipes 2014 - Xen as a foundation for cloud infrastructureAnne Nicolas
 
Cloud Computing Conf 1209
Cloud Computing Conf 1209Cloud Computing Conf 1209
Cloud Computing Conf 1209mandeepdhami
 
Migrating P2V: SUSE Linux Enterprise Server with Xen
Migrating P2V: SUSE Linux Enterprise Server with XenMigrating P2V: SUSE Linux Enterprise Server with Xen
Migrating P2V: SUSE Linux Enterprise Server with XenNovell
 
Deploying large payloads at scale
Deploying large payloads at scaleDeploying large payloads at scale
Deploying large payloads at scaleramonvanalteren
 
Xen PV Performance Status and Optimization Opportunities
Xen PV Performance Status and Optimization OpportunitiesXen PV Performance Status and Optimization Opportunities
Xen PV Performance Status and Optimization OpportunitiesThe Linux Foundation
 
Xen Summit Asia2009 Gplhost Dtc Xen En
Xen Summit Asia2009 Gplhost Dtc Xen EnXen Summit Asia2009 Gplhost Dtc Xen En
Xen Summit Asia2009 Gplhost Dtc Xen EnThe Linux Foundation
 
Leveraging CentOS and Xen for the Go Daddy Private Cloud
Leveraging CentOS and Xen for the Go Daddy Private CloudLeveraging CentOS and Xen for the Go Daddy Private Cloud
Leveraging CentOS and Xen for the Go Daddy Private CloudThe Linux Foundation
 
Sergey Dzyuban "To Build My Own Cloud with Blackjack…"
Sergey Dzyuban "To Build My Own Cloud with Blackjack…"Sergey Dzyuban "To Build My Own Cloud with Blackjack…"
Sergey Dzyuban "To Build My Own Cloud with Blackjack…"Fwdays
 
ICONUK 2013 - An XPager's Guide to Process Server-Side Jobs on IBM® Domino®
ICONUK 2013 - An XPager's Guide to Process Server-Side Jobs on IBM® Domino®ICONUK 2013 - An XPager's Guide to Process Server-Side Jobs on IBM® Domino®
ICONUK 2013 - An XPager's Guide to Process Server-Side Jobs on IBM® Domino®Serdar Basegmez
 
Linaro Connect Asia 13 : Citrix - Xen on ARM plenary session
Linaro Connect Asia 13 : Citrix - Xen on ARM plenary sessionLinaro Connect Asia 13 : Citrix - Xen on ARM plenary session
Linaro Connect Asia 13 : Citrix - Xen on ARM plenary sessionThe Linux Foundation
 
What’s New in vCloud Director 1.5
What’s New in vCloud Director 1.5What’s New in vCloud Director 1.5
What’s New in vCloud Director 1.5Eric Sloof
 
Continuous Delivery Overview
Continuous Delivery OverviewContinuous Delivery Overview
Continuous Delivery OverviewWill Iverson
 
Xen, XenServer, and XAPI: What’s the Difference?-XPUS13 Bulpin,Pavlicek
Xen, XenServer, and XAPI: What’s the Difference?-XPUS13 Bulpin,PavlicekXen, XenServer, and XAPI: What’s the Difference?-XPUS13 Bulpin,Pavlicek
Xen, XenServer, and XAPI: What’s the Difference?-XPUS13 Bulpin,PavlicekThe Linux Foundation
 

What's hot (20)

Novell Success Stories: Endpoint Management in Government
Novell Success Stories: Endpoint Management in GovernmentNovell Success Stories: Endpoint Management in Government
Novell Success Stories: Endpoint Management in Government
 
Accelerating Server Hardware Upgrades with PlateSpin Migrate P2P
Accelerating Server Hardware Upgrades with PlateSpin Migrate P2PAccelerating Server Hardware Upgrades with PlateSpin Migrate P2P
Accelerating Server Hardware Upgrades with PlateSpin Migrate P2P
 
Beyond Disaster Recovery: Restoring Production Workloads with PlateSpin Forge
Beyond Disaster Recovery: Restoring Production Workloads with PlateSpin ForgeBeyond Disaster Recovery: Restoring Production Workloads with PlateSpin Forge
Beyond Disaster Recovery: Restoring Production Workloads with PlateSpin Forge
 
Application Live Migration in LAN/WAN Environment
Application Live Migration in LAN/WAN EnvironmentApplication Live Migration in LAN/WAN Environment
Application Live Migration in LAN/WAN Environment
 
Kernel Recipes 2014 - Xen as a foundation for cloud infrastructure
Kernel Recipes 2014 - Xen as a foundation for cloud infrastructureKernel Recipes 2014 - Xen as a foundation for cloud infrastructure
Kernel Recipes 2014 - Xen as a foundation for cloud infrastructure
 
Cloud Computing Conf 1209
Cloud Computing Conf 1209Cloud Computing Conf 1209
Cloud Computing Conf 1209
 
VM Live Migration Speedup in Xen
VM Live Migration Speedup in XenVM Live Migration Speedup in Xen
VM Live Migration Speedup in Xen
 
Migrating P2V: SUSE Linux Enterprise Server with Xen
Migrating P2V: SUSE Linux Enterprise Server with XenMigrating P2V: SUSE Linux Enterprise Server with Xen
Migrating P2V: SUSE Linux Enterprise Server with Xen
 
Deploying large payloads at scale
Deploying large payloads at scaleDeploying large payloads at scale
Deploying large payloads at scale
 
Xen PV Performance Status and Optimization Opportunities
Xen PV Performance Status and Optimization OpportunitiesXen PV Performance Status and Optimization Opportunities
Xen PV Performance Status and Optimization Opportunities
 
Xen Summit Asia2009 Gplhost Dtc Xen En
Xen Summit Asia2009 Gplhost Dtc Xen EnXen Summit Asia2009 Gplhost Dtc Xen En
Xen Summit Asia2009 Gplhost Dtc Xen En
 
Skylark: Easy Cloud Computing
Skylark: Easy Cloud ComputingSkylark: Easy Cloud Computing
Skylark: Easy Cloud Computing
 
Leveraging CentOS and Xen for the Go Daddy Private Cloud
Leveraging CentOS and Xen for the Go Daddy Private CloudLeveraging CentOS and Xen for the Go Daddy Private Cloud
Leveraging CentOS and Xen for the Go Daddy Private Cloud
 
Sergey Dzyuban "To Build My Own Cloud with Blackjack…"
Sergey Dzyuban "To Build My Own Cloud with Blackjack…"Sergey Dzyuban "To Build My Own Cloud with Blackjack…"
Sergey Dzyuban "To Build My Own Cloud with Blackjack…"
 
ICONUK 2013 - An XPager's Guide to Process Server-Side Jobs on IBM® Domino®
ICONUK 2013 - An XPager's Guide to Process Server-Side Jobs on IBM® Domino®ICONUK 2013 - An XPager's Guide to Process Server-Side Jobs on IBM® Domino®
ICONUK 2013 - An XPager's Guide to Process Server-Side Jobs on IBM® Domino®
 
Linaro Connect Asia 13 : Citrix - Xen on ARM plenary session
Linaro Connect Asia 13 : Citrix - Xen on ARM plenary sessionLinaro Connect Asia 13 : Citrix - Xen on ARM plenary session
Linaro Connect Asia 13 : Citrix - Xen on ARM plenary session
 
What’s New in vCloud Director 1.5
What’s New in vCloud Director 1.5What’s New in vCloud Director 1.5
What’s New in vCloud Director 1.5
 
Xen @ Google, 2011
Xen @ Google, 2011Xen @ Google, 2011
Xen @ Google, 2011
 
Continuous Delivery Overview
Continuous Delivery OverviewContinuous Delivery Overview
Continuous Delivery Overview
 
Xen, XenServer, and XAPI: What’s the Difference?-XPUS13 Bulpin,Pavlicek
Xen, XenServer, and XAPI: What’s the Difference?-XPUS13 Bulpin,PavlicekXen, XenServer, and XAPI: What’s the Difference?-XPUS13 Bulpin,Pavlicek
Xen, XenServer, and XAPI: What’s the Difference?-XPUS13 Bulpin,Pavlicek
 

Viewers also liked

JUC Europe 2015: Continuous Integration and Distribution in the Cloud with DE...
JUC Europe 2015: Continuous Integration and Distribution in the Cloud with DE...JUC Europe 2015: Continuous Integration and Distribution in the Cloud with DE...
JUC Europe 2015: Continuous Integration and Distribution in the Cloud with DE...CloudBees
 
Accelerate your PaaS to the Mobile World: Silicon Valley Code Camp 2012
Accelerate your PaaS to the Mobile World: Silicon Valley Code Camp 2012Accelerate your PaaS to the Mobile World: Silicon Valley Code Camp 2012
Accelerate your PaaS to the Mobile World: Silicon Valley Code Camp 2012CloudBees
 
Successful PaaS and CI in the Cloud - EclipseCon 2012
Successful PaaS and CI in the Cloud - EclipseCon 2012Successful PaaS and CI in the Cloud - EclipseCon 2012
Successful PaaS and CI in the Cloud - EclipseCon 2012CloudBees
 
How can Liferay Developers, Deployers move to the Cloud
How can Liferay Developers, Deployers move to the CloudHow can Liferay Developers, Deployers move to the Cloud
How can Liferay Developers, Deployers move to the CloudCloudBees
 
Pimp your jenkins platform with docker - Devops.com 2015/11
Pimp your jenkins platform with docker - Devops.com 2015/11Pimp your jenkins platform with docker - Devops.com 2015/11
Pimp your jenkins platform with docker - Devops.com 2015/11CloudBees
 
CI and CD at Scale: Scaling Jenkins with Docker and Apache Mesos
CI and CD at Scale: Scaling Jenkins with Docker and Apache MesosCI and CD at Scale: Scaling Jenkins with Docker and Apache Mesos
CI and CD at Scale: Scaling Jenkins with Docker and Apache MesosCarlos Sanchez
 
Continuous integration
Continuous integrationContinuous integration
Continuous integrationamscanne
 
Yale Jenkins Show and Tell
Yale Jenkins Show and TellYale Jenkins Show and Tell
Yale Jenkins Show and TellE. Camden Fisher
 
Taking Control of your Data with GraphQL
Taking Control of your Data with GraphQLTaking Control of your Data with GraphQL
Taking Control of your Data with GraphQLVinci Rufus
 
CI and CD with Jenkins
CI and CD with JenkinsCI and CD with Jenkins
CI and CD with JenkinsMartin Málek
 

Viewers also liked (11)

JUC Europe 2015: Continuous Integration and Distribution in the Cloud with DE...
JUC Europe 2015: Continuous Integration and Distribution in the Cloud with DE...JUC Europe 2015: Continuous Integration and Distribution in the Cloud with DE...
JUC Europe 2015: Continuous Integration and Distribution in the Cloud with DE...
 
Accelerate your PaaS to the Mobile World: Silicon Valley Code Camp 2012
Accelerate your PaaS to the Mobile World: Silicon Valley Code Camp 2012Accelerate your PaaS to the Mobile World: Silicon Valley Code Camp 2012
Accelerate your PaaS to the Mobile World: Silicon Valley Code Camp 2012
 
Successful PaaS and CI in the Cloud - EclipseCon 2012
Successful PaaS and CI in the Cloud - EclipseCon 2012Successful PaaS and CI in the Cloud - EclipseCon 2012
Successful PaaS and CI in the Cloud - EclipseCon 2012
 
How can Liferay Developers, Deployers move to the Cloud
How can Liferay Developers, Deployers move to the CloudHow can Liferay Developers, Deployers move to the Cloud
How can Liferay Developers, Deployers move to the Cloud
 
Pimp your jenkins platform with docker - Devops.com 2015/11
Pimp your jenkins platform with docker - Devops.com 2015/11Pimp your jenkins platform with docker - Devops.com 2015/11
Pimp your jenkins platform with docker - Devops.com 2015/11
 
CI and CD at Scale: Scaling Jenkins with Docker and Apache Mesos
CI and CD at Scale: Scaling Jenkins with Docker and Apache MesosCI and CD at Scale: Scaling Jenkins with Docker and Apache Mesos
CI and CD at Scale: Scaling Jenkins with Docker and Apache Mesos
 
Continuous integration
Continuous integrationContinuous integration
Continuous integration
 
Yale Jenkins Show and Tell
Yale Jenkins Show and TellYale Jenkins Show and Tell
Yale Jenkins Show and Tell
 
Jenkins CI
Jenkins CIJenkins CI
Jenkins CI
 
Taking Control of your Data with GraphQL
Taking Control of your Data with GraphQLTaking Control of your Data with GraphQL
Taking Control of your Data with GraphQL
 
CI and CD with Jenkins
CI and CD with JenkinsCI and CD with Jenkins
CI and CD with Jenkins
 

Similar to Running productioninstance 1-localcopy

CloudBees Toronto Presentation
CloudBees Toronto PresentationCloudBees Toronto Presentation
CloudBees Toronto PresentationXebiaLabs
 
Continuous Development with Jenkins - Stephen Connolly at PuppetCamp Dublin '12
Continuous Development with Jenkins - Stephen Connolly at PuppetCamp Dublin '12Continuous Development with Jenkins - Stephen Connolly at PuppetCamp Dublin '12
Continuous Development with Jenkins - Stephen Connolly at PuppetCamp Dublin '12Puppet
 
Jumping from Continuous Integration to Continuous Delivery with Jenkins Enter...
Jumping from Continuous Integration to Continuous Delivery with Jenkins Enter...Jumping from Continuous Integration to Continuous Delivery with Jenkins Enter...
Jumping from Continuous Integration to Continuous Delivery with Jenkins Enter...CloudBees
 
Securing jenkins
Securing jenkinsSecuring jenkins
Securing jenkinsCloudBees
 
Jenkins scaling best practices
Jenkins scaling best practicesJenkins scaling best practices
Jenkins scaling best practicesHenry McBride
 
Java / Opening Open Source the Jenkins Way - Nicolas de Loof, CloudBees
Java / Opening Open Source the Jenkins Way - Nicolas de Loof, CloudBeesJava / Opening Open Source the Jenkins Way - Nicolas de Loof, CloudBees
Java / Opening Open Source the Jenkins Way - Nicolas de Loof, CloudBeesParis Open Source Summit
 
Jenkins Performance - Avoiding Pitfalls, Diagnosing Issues, and Scaling for G...
Jenkins Performance - Avoiding Pitfalls, Diagnosing Issues, and Scaling for G...Jenkins Performance - Avoiding Pitfalls, Diagnosing Issues, and Scaling for G...
Jenkins Performance - Avoiding Pitfalls, Diagnosing Issues, and Scaling for G...Owen Mehegan
 
Continuous delivery with Jenkins Enterprise and Deployit
Continuous delivery with Jenkins Enterprise and DeployitContinuous delivery with Jenkins Enterprise and Deployit
Continuous delivery with Jenkins Enterprise and DeployitXebiaLabs
 
Jenkins Enterprise by CloudBees Webinar
Jenkins Enterprise by CloudBees WebinarJenkins Enterprise by CloudBees Webinar
Jenkins Enterprise by CloudBees WebinarCloudBees
 
Lessons Learned: Novell Open Enterprise Server Upgrades Made Easy
Lessons Learned: Novell Open Enterprise Server Upgrades Made EasyLessons Learned: Novell Open Enterprise Server Upgrades Made Easy
Lessons Learned: Novell Open Enterprise Server Upgrades Made EasyNovell
 
DevOps World | Jenkins World 2018 and The Future of Jenkins
DevOps World | Jenkins World 2018 and The Future of JenkinsDevOps World | Jenkins World 2018 and The Future of Jenkins
DevOps World | Jenkins World 2018 and The Future of JenkinsNigel Charman
 
Simplified, Robust and Speedy Novell Identity Manager Implementation with Des...
Simplified, Robust and Speedy Novell Identity Manager Implementation with Des...Simplified, Robust and Speedy Novell Identity Manager Implementation with Des...
Simplified, Robust and Speedy Novell Identity Manager Implementation with Des...Novell
 
Deploying your SaaS stack OnPrem
Deploying your SaaS stack OnPremDeploying your SaaS stack OnPrem
Deploying your SaaS stack OnPremKris Buytaert
 
Continuous Delivery to Kubernetes with Jenkins and Helm
Continuous Delivery to Kubernetes with Jenkins and HelmContinuous Delivery to Kubernetes with Jenkins and Helm
Continuous Delivery to Kubernetes with Jenkins and HelmDavid Currie
 
Developing ZenPacks the Right Way: Introducing the SDK
Developing ZenPacks the Right Way: Introducing the SDK Developing ZenPacks the Right Way: Introducing the SDK
Developing ZenPacks the Right Way: Introducing the SDK Zenoss
 
Jenkins Meetup Pune
Jenkins Meetup PuneJenkins Meetup Pune
Jenkins Meetup PuneUmesh Kumhar
 
Docker + jenkins in the enterprise (3)
Docker + jenkins in the enterprise (3)Docker + jenkins in the enterprise (3)
Docker + jenkins in the enterprise (3)Kurt Madel
 
From Continuous Integration to Continuous Delivery with Jenkins - javaland.de...
From Continuous Integration to Continuous Delivery with Jenkins - javaland.de...From Continuous Integration to Continuous Delivery with Jenkins - javaland.de...
From Continuous Integration to Continuous Delivery with Jenkins - javaland.de...CloudBees
 
7 Habits of Highly Effective Jenkins Users
7 Habits of Highly Effective Jenkins Users7 Habits of Highly Effective Jenkins Users
7 Habits of Highly Effective Jenkins UsersJules Pierre-Louis
 
Seven Habits of Highly Effective Jenkins Users (2014 edition!)
Seven Habits of Highly Effective Jenkins Users (2014 edition!)Seven Habits of Highly Effective Jenkins Users (2014 edition!)
Seven Habits of Highly Effective Jenkins Users (2014 edition!)Andrew Bayer
 

Similar to Running productioninstance 1-localcopy (20)

CloudBees Toronto Presentation
CloudBees Toronto PresentationCloudBees Toronto Presentation
CloudBees Toronto Presentation
 
Continuous Development with Jenkins - Stephen Connolly at PuppetCamp Dublin '12
Continuous Development with Jenkins - Stephen Connolly at PuppetCamp Dublin '12Continuous Development with Jenkins - Stephen Connolly at PuppetCamp Dublin '12
Continuous Development with Jenkins - Stephen Connolly at PuppetCamp Dublin '12
 
Jumping from Continuous Integration to Continuous Delivery with Jenkins Enter...
Jumping from Continuous Integration to Continuous Delivery with Jenkins Enter...Jumping from Continuous Integration to Continuous Delivery with Jenkins Enter...
Jumping from Continuous Integration to Continuous Delivery with Jenkins Enter...
 
Securing jenkins
Securing jenkinsSecuring jenkins
Securing jenkins
 
Jenkins scaling best practices
Jenkins scaling best practicesJenkins scaling best practices
Jenkins scaling best practices
 
Java / Opening Open Source the Jenkins Way - Nicolas de Loof, CloudBees
Java / Opening Open Source the Jenkins Way - Nicolas de Loof, CloudBeesJava / Opening Open Source the Jenkins Way - Nicolas de Loof, CloudBees
Java / Opening Open Source the Jenkins Way - Nicolas de Loof, CloudBees
 
Jenkins Performance - Avoiding Pitfalls, Diagnosing Issues, and Scaling for G...
Jenkins Performance - Avoiding Pitfalls, Diagnosing Issues, and Scaling for G...Jenkins Performance - Avoiding Pitfalls, Diagnosing Issues, and Scaling for G...
Jenkins Performance - Avoiding Pitfalls, Diagnosing Issues, and Scaling for G...
 
Continuous delivery with Jenkins Enterprise and Deployit
Continuous delivery with Jenkins Enterprise and DeployitContinuous delivery with Jenkins Enterprise and Deployit
Continuous delivery with Jenkins Enterprise and Deployit
 
Jenkins Enterprise by CloudBees Webinar
Jenkins Enterprise by CloudBees WebinarJenkins Enterprise by CloudBees Webinar
Jenkins Enterprise by CloudBees Webinar
 
Lessons Learned: Novell Open Enterprise Server Upgrades Made Easy
Lessons Learned: Novell Open Enterprise Server Upgrades Made EasyLessons Learned: Novell Open Enterprise Server Upgrades Made Easy
Lessons Learned: Novell Open Enterprise Server Upgrades Made Easy
 
DevOps World | Jenkins World 2018 and The Future of Jenkins
DevOps World | Jenkins World 2018 and The Future of JenkinsDevOps World | Jenkins World 2018 and The Future of Jenkins
DevOps World | Jenkins World 2018 and The Future of Jenkins
 
Simplified, Robust and Speedy Novell Identity Manager Implementation with Des...
Simplified, Robust and Speedy Novell Identity Manager Implementation with Des...Simplified, Robust and Speedy Novell Identity Manager Implementation with Des...
Simplified, Robust and Speedy Novell Identity Manager Implementation with Des...
 
Deploying your SaaS stack OnPrem
Deploying your SaaS stack OnPremDeploying your SaaS stack OnPrem
Deploying your SaaS stack OnPrem
 
Continuous Delivery to Kubernetes with Jenkins and Helm
Continuous Delivery to Kubernetes with Jenkins and HelmContinuous Delivery to Kubernetes with Jenkins and Helm
Continuous Delivery to Kubernetes with Jenkins and Helm
 
Developing ZenPacks the Right Way: Introducing the SDK
Developing ZenPacks the Right Way: Introducing the SDK Developing ZenPacks the Right Way: Introducing the SDK
Developing ZenPacks the Right Way: Introducing the SDK
 
Jenkins Meetup Pune
Jenkins Meetup PuneJenkins Meetup Pune
Jenkins Meetup Pune
 
Docker + jenkins in the enterprise (3)
Docker + jenkins in the enterprise (3)Docker + jenkins in the enterprise (3)
Docker + jenkins in the enterprise (3)
 
From Continuous Integration to Continuous Delivery with Jenkins - javaland.de...
From Continuous Integration to Continuous Delivery with Jenkins - javaland.de...From Continuous Integration to Continuous Delivery with Jenkins - javaland.de...
From Continuous Integration to Continuous Delivery with Jenkins - javaland.de...
 
7 Habits of Highly Effective Jenkins Users
7 Habits of Highly Effective Jenkins Users7 Habits of Highly Effective Jenkins Users
7 Habits of Highly Effective Jenkins Users
 
Seven Habits of Highly Effective Jenkins Users (2014 edition!)
Seven Habits of Highly Effective Jenkins Users (2014 edition!)Seven Habits of Highly Effective Jenkins Users (2014 edition!)
Seven Habits of Highly Effective Jenkins Users (2014 edition!)
 

More from CloudBees

JUC Europe 2015: Scaling Your Jenkins Master with Docker
JUC Europe 2015: Scaling Your Jenkins Master with DockerJUC Europe 2015: Scaling Your Jenkins Master with Docker
JUC Europe 2015: Scaling Your Jenkins Master with DockerCloudBees
 
JUC Europe 2015: Plugin Development with Gradle and Groovy
JUC Europe 2015: Plugin Development with Gradle and GroovyJUC Europe 2015: Plugin Development with Gradle and Groovy
JUC Europe 2015: Plugin Development with Gradle and GroovyCloudBees
 
JUC Europe 2015: Multi-Node Environment as a Jenkins Slave (Compound-Slave)
JUC Europe 2015: Multi-Node Environment as a Jenkins Slave (Compound-Slave)JUC Europe 2015: Multi-Node Environment as a Jenkins Slave (Compound-Slave)
JUC Europe 2015: Multi-Node Environment as a Jenkins Slave (Compound-Slave)CloudBees
 
JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Je...
JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Je...JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Je...
JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Je...CloudBees
 
JUC Europe 2015: Jenkins Made Easy
JUC Europe 2015: Jenkins Made EasyJUC Europe 2015: Jenkins Made Easy
JUC Europe 2015: Jenkins Made EasyCloudBees
 
JUC Europe 2015: Scaling of Jenkins Pipeline Creation and Maintenance
JUC Europe 2015: Scaling of Jenkins Pipeline Creation and MaintenanceJUC Europe 2015: Scaling of Jenkins Pipeline Creation and Maintenance
JUC Europe 2015: Scaling of Jenkins Pipeline Creation and MaintenanceCloudBees
 
JUC Europe 2015: From Virtual Machines to Containers: Achieving Continuous In...
JUC Europe 2015: From Virtual Machines to Containers: Achieving Continuous In...JUC Europe 2015: From Virtual Machines to Containers: Achieving Continuous In...
JUC Europe 2015: From Virtual Machines to Containers: Achieving Continuous In...CloudBees
 
JUC Europe 2015: Hey! What Did We Just Release?
JUC Europe 2015: Hey! What Did We Just Release?JUC Europe 2015: Hey! What Did We Just Release?
JUC Europe 2015: Hey! What Did We Just Release?CloudBees
 
JUC Europe 2015: Orchestrating Your Pipelines with Jenkins, Python and the Je...
JUC Europe 2015: Orchestrating Your Pipelines with Jenkins, Python and the Je...JUC Europe 2015: Orchestrating Your Pipelines with Jenkins, Python and the Je...
JUC Europe 2015: Orchestrating Your Pipelines with Jenkins, Python and the Je...CloudBees
 
JUC Europe 2015: Jenkins Pipeline for Continuous Delivery of Big Data Projects
JUC Europe 2015: Jenkins Pipeline for Continuous Delivery of Big Data ProjectsJUC Europe 2015: Jenkins Pipeline for Continuous Delivery of Big Data Projects
JUC Europe 2015: Jenkins Pipeline for Continuous Delivery of Big Data ProjectsCloudBees
 
JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardw...
JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardw...JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardw...
JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardw...CloudBees
 
JUC Europe 2015: Optimizing Your CI: Lessons Learned from a Successful Jenkin...
JUC Europe 2015: Optimizing Your CI: Lessons Learned from a Successful Jenkin...JUC Europe 2015: Optimizing Your CI: Lessons Learned from a Successful Jenkin...
JUC Europe 2015: Optimizing Your CI: Lessons Learned from a Successful Jenkin...CloudBees
 
JUC Europe 2015: Enabling Continuous Delivery for Major Retailers
JUC Europe 2015: Enabling Continuous Delivery for Major RetailersJUC Europe 2015: Enabling Continuous Delivery for Major Retailers
JUC Europe 2015: Enabling Continuous Delivery for Major RetailersCloudBees
 
JUC Europe 2015: Bringing CD at Cloud-Scale with Jenkins, Docker and "Tiger"
JUC Europe 2015: Bringing CD at Cloud-Scale with Jenkins, Docker and "Tiger"JUC Europe 2015: Bringing CD at Cloud-Scale with Jenkins, Docker and "Tiger"
JUC Europe 2015: Bringing CD at Cloud-Scale with Jenkins, Docker and "Tiger"CloudBees
 
JUC Europe 2015: Making Strides towards Enterprise-Scale DevOps...with Jenkin...
JUC Europe 2015: Making Strides towards Enterprise-Scale DevOps...with Jenkin...JUC Europe 2015: Making Strides towards Enterprise-Scale DevOps...with Jenkin...
JUC Europe 2015: Making Strides towards Enterprise-Scale DevOps...with Jenkin...CloudBees
 
JUC Europe 2015: Evolving the Jenkins UI
JUC Europe 2015: Evolving the Jenkins UIJUC Europe 2015: Evolving the Jenkins UI
JUC Europe 2015: Evolving the Jenkins UICloudBees
 
JUC Europe 2015: Using Infrastructure Nodes Wisely With Jenkins and Apache Mesos
JUC Europe 2015: Using Infrastructure Nodes Wisely With Jenkins and Apache MesosJUC Europe 2015: Using Infrastructure Nodes Wisely With Jenkins and Apache Mesos
JUC Europe 2015: Using Infrastructure Nodes Wisely With Jenkins and Apache MesosCloudBees
 
JUC Europe 2015: How to Optimize Automated Testing with Everyone's Favorite B...
JUC Europe 2015: How to Optimize Automated Testing with Everyone's Favorite B...JUC Europe 2015: How to Optimize Automated Testing with Everyone's Favorite B...
JUC Europe 2015: How to Optimize Automated Testing with Everyone's Favorite B...CloudBees
 
JUC 2015 - Keynote Address and Opening Remarks by Kohsuke Kawaguchi, Founder,...
JUC 2015 - Keynote Address and Opening Remarks by Kohsuke Kawaguchi, Founder,...JUC 2015 - Keynote Address and Opening Remarks by Kohsuke Kawaguchi, Founder,...
JUC 2015 - Keynote Address and Opening Remarks by Kohsuke Kawaguchi, Founder,...CloudBees
 
JUC Europe 2015: Configuration as Code: The Job DSL Plugin
JUC Europe 2015: Configuration as Code: The Job DSL PluginJUC Europe 2015: Configuration as Code: The Job DSL Plugin
JUC Europe 2015: Configuration as Code: The Job DSL PluginCloudBees
 

More from CloudBees (20)

JUC Europe 2015: Scaling Your Jenkins Master with Docker
JUC Europe 2015: Scaling Your Jenkins Master with DockerJUC Europe 2015: Scaling Your Jenkins Master with Docker
JUC Europe 2015: Scaling Your Jenkins Master with Docker
 
JUC Europe 2015: Plugin Development with Gradle and Groovy
JUC Europe 2015: Plugin Development with Gradle and GroovyJUC Europe 2015: Plugin Development with Gradle and Groovy
JUC Europe 2015: Plugin Development with Gradle and Groovy
 
JUC Europe 2015: Multi-Node Environment as a Jenkins Slave (Compound-Slave)
JUC Europe 2015: Multi-Node Environment as a Jenkins Slave (Compound-Slave)JUC Europe 2015: Multi-Node Environment as a Jenkins Slave (Compound-Slave)
JUC Europe 2015: Multi-Node Environment as a Jenkins Slave (Compound-Slave)
 
JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Je...
JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Je...JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Je...
JUC Europe 2015: The Famous Cows of Cambridge: A Non-Standard Use Case for Je...
 
JUC Europe 2015: Jenkins Made Easy
JUC Europe 2015: Jenkins Made EasyJUC Europe 2015: Jenkins Made Easy
JUC Europe 2015: Jenkins Made Easy
 
JUC Europe 2015: Scaling of Jenkins Pipeline Creation and Maintenance
JUC Europe 2015: Scaling of Jenkins Pipeline Creation and MaintenanceJUC Europe 2015: Scaling of Jenkins Pipeline Creation and Maintenance
JUC Europe 2015: Scaling of Jenkins Pipeline Creation and Maintenance
 
JUC Europe 2015: From Virtual Machines to Containers: Achieving Continuous In...
JUC Europe 2015: From Virtual Machines to Containers: Achieving Continuous In...JUC Europe 2015: From Virtual Machines to Containers: Achieving Continuous In...
JUC Europe 2015: From Virtual Machines to Containers: Achieving Continuous In...
 
JUC Europe 2015: Hey! What Did We Just Release?
JUC Europe 2015: Hey! What Did We Just Release?JUC Europe 2015: Hey! What Did We Just Release?
JUC Europe 2015: Hey! What Did We Just Release?
 
JUC Europe 2015: Orchestrating Your Pipelines with Jenkins, Python and the Je...
JUC Europe 2015: Orchestrating Your Pipelines with Jenkins, Python and the Je...JUC Europe 2015: Orchestrating Your Pipelines with Jenkins, Python and the Je...
JUC Europe 2015: Orchestrating Your Pipelines with Jenkins, Python and the Je...
 
JUC Europe 2015: Jenkins Pipeline for Continuous Delivery of Big Data Projects
JUC Europe 2015: Jenkins Pipeline for Continuous Delivery of Big Data ProjectsJUC Europe 2015: Jenkins Pipeline for Continuous Delivery of Big Data Projects
JUC Europe 2015: Jenkins Pipeline for Continuous Delivery of Big Data Projects
 
JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardw...
JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardw...JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardw...
JUC Europe 2015: Jenkins-Based Continuous Integration for Heterogeneous Hardw...
 
JUC Europe 2015: Optimizing Your CI: Lessons Learned from a Successful Jenkin...
JUC Europe 2015: Optimizing Your CI: Lessons Learned from a Successful Jenkin...JUC Europe 2015: Optimizing Your CI: Lessons Learned from a Successful Jenkin...
JUC Europe 2015: Optimizing Your CI: Lessons Learned from a Successful Jenkin...
 
JUC Europe 2015: Enabling Continuous Delivery for Major Retailers
JUC Europe 2015: Enabling Continuous Delivery for Major RetailersJUC Europe 2015: Enabling Continuous Delivery for Major Retailers
JUC Europe 2015: Enabling Continuous Delivery for Major Retailers
 
JUC Europe 2015: Bringing CD at Cloud-Scale with Jenkins, Docker and "Tiger"
JUC Europe 2015: Bringing CD at Cloud-Scale with Jenkins, Docker and "Tiger"JUC Europe 2015: Bringing CD at Cloud-Scale with Jenkins, Docker and "Tiger"
JUC Europe 2015: Bringing CD at Cloud-Scale with Jenkins, Docker and "Tiger"
 
JUC Europe 2015: Making Strides towards Enterprise-Scale DevOps...with Jenkin...
JUC Europe 2015: Making Strides towards Enterprise-Scale DevOps...with Jenkin...JUC Europe 2015: Making Strides towards Enterprise-Scale DevOps...with Jenkin...
JUC Europe 2015: Making Strides towards Enterprise-Scale DevOps...with Jenkin...
 
JUC Europe 2015: Evolving the Jenkins UI
JUC Europe 2015: Evolving the Jenkins UIJUC Europe 2015: Evolving the Jenkins UI
JUC Europe 2015: Evolving the Jenkins UI
 
JUC Europe 2015: Using Infrastructure Nodes Wisely With Jenkins and Apache Mesos
JUC Europe 2015: Using Infrastructure Nodes Wisely With Jenkins and Apache MesosJUC Europe 2015: Using Infrastructure Nodes Wisely With Jenkins and Apache Mesos
JUC Europe 2015: Using Infrastructure Nodes Wisely With Jenkins and Apache Mesos
 
JUC Europe 2015: How to Optimize Automated Testing with Everyone's Favorite B...
JUC Europe 2015: How to Optimize Automated Testing with Everyone's Favorite B...JUC Europe 2015: How to Optimize Automated Testing with Everyone's Favorite B...
JUC Europe 2015: How to Optimize Automated Testing with Everyone's Favorite B...
 
JUC 2015 - Keynote Address and Opening Remarks by Kohsuke Kawaguchi, Founder,...
JUC 2015 - Keynote Address and Opening Remarks by Kohsuke Kawaguchi, Founder,...JUC 2015 - Keynote Address and Opening Remarks by Kohsuke Kawaguchi, Founder,...
JUC 2015 - Keynote Address and Opening Remarks by Kohsuke Kawaguchi, Founder,...
 
JUC Europe 2015: Configuration as Code: The Job DSL Plugin
JUC Europe 2015: Configuration as Code: The Job DSL PluginJUC Europe 2015: Configuration as Code: The Job DSL Plugin
JUC Europe 2015: Configuration as Code: The Job DSL Plugin
 

Running productioninstance 1-localcopy

  • 1. Running a production Jenkins instance Harpreet Singh, Senior Director, Product Management Kohsuke Kawaguchi Jenkins founder ©2012 CloudBees, Inc. All Rights Reserved
  • 2. Agenda • Failures – a fact of life – Getting ready for failures – Preventing failures – Debugging failures • Run an efficient Jenkins installation ©2012 CloudBees, Inc. All Rights Reserved 2
  • 3. Day: A period of 24 hours, mostly misspent… ©2011 CloudBees, Inc. All Rights 3 Reserved
  • 4. ©2011 CloudBees, Inc. All Rights 4 Reserved
  • 5. CloudBees – Who are we? • Jenkins founder on-board • Key Jenkins contributors on-board • Built Jenkins as a Service • Run the biggest Jenkins installation anywhere (2k+) masters ©2011 CloudBees, Inc. All Rights Reserved 5
  • 6. CloudBees’ Mission - Eliminate Downtime • Eliminate time wasted due to – Jenkins issues – User issues – Lack of right tools… • Improve efficiency for administrators and developers • Rely on Jenkins… ©2011 CloudBees, Inc. All Rights Reserved 6
  • 7. Good Management of Jenkins • Organize jobs better • Secure your jobs • Replicate good practices • Respond quicker to requests • Ensure compliance • Bounce back from failures • Prevent failures • Everything should be as fast as possible…if not faster ©2011 CloudBees, Inc. All Rights Reserved 7
  • 8. Recovering from failures High Availability, Backing up ©2011 CloudBees, Inc. All Rights Reserved
  • 9. Backing up Jenkins Jenkins Enterprise Problem: Disk Failures Solution • JENKINS_HOME • Backup plugin – Plugins, users, jobs…everything • Backup-to-cloud Solution: Back it up • Push HOME to a repo – HOME tends to be large – Commit only vital info – Run nightly • Push to S3 ©2011 CloudBees, Inc. All Rights 9 Reserved
  • 10. JE Backup Plugin • Backup as a Jenkins • Where to backup job – Local Directory • What to backup – Sftp server – Job configuration – WebDav – Build records • Retention Policy – System – All Configuration – Last N • Plugin binaries, plugin – Exponential decay configs etc • Everything except job ©2011 CloudBees, Inc. All Rights 10 Reserved
  • 11. Demo ©2011 CloudBees, Inc. All Rights 11 Reserved
  • 12. Making Jenkins Highly Available Jenkins Enterprise Problem: Jenkins failures Solution • Machine/Jenkins failure has • Highly Available high cost to productivity – Setup multiple Jenkins masters Solution: Notified by unhappy – Uses jgroups to elect a customers ;-) primary master • Issues: – Promotes a backup master – Receive emails from unhappy as primary customers and log in and fix it • You do have JENKINS_HOME backed up else where – don’t you? ©2011 CloudBees, Inc. All Rights 12 Reserved
  • 13. Bounce Back Faster: High Availability Reverse Proxy Reverse Proxy Jenkins Jenkins Jenkins Master Master Master MT MT Jenkins Cluster Jenkins Cluster JENKINS_HOME JENKINS_HOME NFS ©2011 CloudBees, Inc. All Rights 13 Reserved
  • 14. Demo ©2011 CloudBees, Inc. All Rights 14 Reserved
  • 15. Miscellaneous • Jenkins is not just JENKINS_HOME…think about the slaves – Offload builds onto slaves – Other executables on the system: git, ruby, java etc as well – Preferably use Chef/Puppet to replicate installations • What about geo redundancy? – Technically you can use HA but network latency comes in play – Ideally, use HA in a localized data center and a manual failover to a different geo • What HA is not? – Does not load balance between instances ©2011 CloudBees, Inc. All Rights Reserved 15
  • 16. Preventing failures Git Validated Merges plugin ©2011 CloudBees, Inc. All Rights Reserved
  • 17. How can you delegate more to Jenkins? • Does your CI server shift work from laptops to servers? – You need to commit to have Jenkins test it – But if your commit is bad, it blocks others – You end up testing locally before committing – FAIL 17
  • 18. Motivation • We want to make changes safely – Your mistake shouldn’t block others – Only push after changes are validated • We want to run tests asynchronously – Your brain has more important things to do – Make change and move on – Even with TDD! • We want to run tests on the server – Your laptop has more important things to do 18
  • 19. Solution: Jenkins should be Git server • I push to Jenkins • Jenkins merges it with upstream • Jenkins tests it • If good, Jenkins pushes it upstream upstream repo gate repo 19
  • 20. Another way to look at it Tip of master in upstream Tip of master in upstream My changes 20
  • 21. Implementation • Transport – HTTP – SSH • JGit embedded in Jenkins for git server functionality – A bit of magic like Gerrit to make it seamless • Additional tags to let you pull submitted changes 21
  • 22. Demo ©2011 CloudBees, Inc. All Rights 22 Reserved
  • 23. Running an efficient production system ©2011 CloudBees, Inc. All Rights Reserved
  • 24. Test Instance • Run mini 2nd instance – Test new core version before putting it to prod – Test new versions of plugins – Play with new plugins • Copy over some jobs from prod • Bootstrap dry-run – -Djenkins.model.Jenkins.killAfterLoad=true ©2011 CloudBees, Inc. All Rights Reserved 24
  • 25. Configuring Jenkins for efficiency • Fast archiver plugin – Conserve network bandwidth • No build on master – Also good for security ©2011 CloudBees, Inc. All Rights Reserved 25
  • 26. Managing and Pruning Plugins Jenkins Enterprise Problem: Discovering what plugins are used in an Solution installation • Plugin Usage Plugin • No visibility if a particular plugin – Tabular view of Plugin is used or how many jobs use it name, # of jobs and the job names using the plugin ©2011 CloudBees, Inc. All Rights 26 Reserved
  • 27. Demo ©2011 CloudBees, Inc. All Rights 27 Reserved
  • 28. Monitoring Jenkins ©2012 CloudBees, Inc. All Rights Reserved
  • 29. Why? ©2011 CloudBees, Inc. All Rights Reserved 29
  • 30. What? • What the user sees – GUI (load time) • JVM memory size – Beware of several independent pieces • System load • Free space on $JENKINS_HOME • Slave availability • Queue length ©2011 CloudBees, Inc. All Rights Reserved 30
  • 31. Groovy Console $ cat queue.groovy j=Jenkins.instances println j.queue.items.length $ curl –u "user:apiToken“ –data-urlencode script@queue.groovy http://jenkins/scriptText 13 31
  • 32. Remote API $ curl http://jenkins/computer/api/json?pretty=true { busyExecutors: 0, totalExecutors: 2, ... } 32
  • 33. Jenkins Monitoring plugin • JavaMelody in Jenkins ©2011 CloudBees, Inc. All Rights Reserved 33
  • 34. Nagios (or others like it) • Server app for monitoring stuff – Extensible, allowing all sorts of things to be monitored • Used in jenkins-ci.org/DEV@cloud ©2011 CloudBees, Inc. All Rights Reserved 34
  • 35. ©2011 CloudBees, Inc. All Rights Reserved 35
  • 36. Thread dump • Tells us where Jenkins is stuck • When? – Hang or slowness • Look for threads that’s stuck – HTTP request threads – Executor threads ©2011 CloudBees, Inc. All Rights Reserved 36
  • 37. How to get a thread dump • http://jenkins/threadDump • kill -3 <PID> ©2011 CloudBees, Inc. All Rights Reserved 37
  • 38. Heap dump • Tells us what’s eating memory • When? – OutOfMemoryError – Monitoring shows abnormal growth • Look for objects that are big – Sessions – Classes from plugins ©2011 CloudBees, Inc. All Rights Reserved 38
  • 39. How to get a memory dump • curl –L http://jenkins/heapDump > dump.hprof • jmap -dump:format=b,file=dump.hprof PID • -XX:+HeapDumpOnOutOfMemoryError ©2011 CloudBees, Inc. All Rights Reserved 39
  • 40. Wrapping up Thank You! More Info http://www.cloudbees.com/jenkins-enterprise-by-cloudbees-overview.cb Free Trial http://www.cloudbees.com/jenkins-enterprise-by-cloudbees-download.cb Wiki Page https://wiki.cloudbees.com/bin/view/Jenkins+Enterprise/WebHome User Guide http://jenkins-enterprise.cloudbees.com/docs/user-guide-bundle/index.html# ©2011 CloudBees, Inc. All Rights 40 Reserved
  • 41. Day: A period of 24 hours, mostly misspent… ©2011 CloudBees, Inc. All Rights 41 Reserved
  • 42. ©2012 CloudBees, Inc. All Rights Reserved

Editor's Notes

  1. さっき散々サーバを使いこなすことが重要だという話をした。&lt;&lt;スライド&gt;&gt;それはCIが無価値だということではないが、もっと活用できるポテンシャルがあるのにいかせていない