This document discusses Jenkins performance tuning and scaling. It begins with an introduction to Jenkins and common performance issues like poor disk performance. It then provides best practices for initial setup and configuration. Methods for diagnosing performance problems are outlined, including analyzing slow request logs, GC logs, and thread dumps. The document concludes with recommendations for horizontally scaling Jenkins by adding additional master instances and load balancing jobs across masters.

1. © 2019 CloudBees, Inc. All Rights Reserved.© 2019 CloudBees, Inc. All Rights Reserved.
Jenkins Performance
Avoiding Pitfalls, Diagnosing Issues, and Scaling for Growth

2. © 2019 CloudBees, Inc. All Rights Reserved.
About Owen
● Currently a Developer Support Engineer at CloudBees
● Previously, various roles in Ops, QA, Dev Tools and Productivity
● Involved with the Jenkins open source project since 2010
● 20 years as a Linux sysadmin

3. © 2019 CloudBees, Inc. All Rights Reserved.
About Jenkins
● Created at Sun in 2005; originally named Hudson, forked/renamed to Jenkins in 2010
● Has grown to over 250,000 installations and 1,700 plugins
● Monolithic Java application
● Master/agent architecture
● Plugin-focused architecture

4. © 2019 CloudBees, Inc. All Rights Reserved.
Jenkins Admin Challenges
● Java performance tuning skills are not universal
● Usage grows organically and load comes from multiple areas
● Jenkins architecture has some quirks

5. © 2019 CloudBees, Inc. All Rights Reserved.
Common Performance Issues
● Poor disk performance
● Unoptimized memory/GC settings
● REST API abuse
● Non-performant plugins

6. © 2019 CloudBees, Inc. All Rights Reserved.
Initial Setup Best Practices

7. © 2019 CloudBees, Inc. All Rights Reserved.
Initial Configuration
● Rotate build history
● Use a JDK install of Java, not a JRE
● Use SSD storage
● Use NFS v4.1
● Use the G1 garbage collector
● Enable GC logging
● Set heap size explicitly; 2-4GB starting point
● Monitor CPU, IO performance, and JVM-specific memory metrics
● Install the Support Core plugin
● https://tinyurl.com/jenkins-jvm-args - Recommended JVM settings for Jenkins

8. © 2019 CloudBees, Inc. All Rights Reserved.
Diagnosing Performance Issues

9. © 2019 CloudBees, Inc. All Rights Reserved.
Initial Diagnosis
● Check JVM memory metrics - max heap size, heap in use vs. free heap
● Check storage performance/availability
● Check dmesg log
● CPU usage is often a distraction from the real issue
● Main Jenkins service log
● “Slow requests” logs (in slow-requests dir)

10. © 2019 CloudBees, Inc. All Rights Reserved.
Slow Request Log
87350msec elapsed in Handling GET / from 172.26.22.250 : qtp739498517-1292250 View/index.jelly
UptimeColumn/column.jelly
jenkins.model.lazy.AbstractLazyLoadRunMap.getByNumber(AbstractLazyLoadRunMap.java:370)
jenkins.model.lazy.AbstractLazyLoadRunMap.search(AbstractLazyLoadRunMap.java:346)
jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:74)
jenkins.model.lazy.LazyLoadRunMapEntrySet$1.next(LazyLoadRunMapEntrySet.java:63)
java.util.AbstractMap$2$1.next(AbstractMap.java:418)
org.jenkinsci.plugins.uptime.DefaultUptimeService.getUptimePercentage(DefaultUptimeService.java:62)
org.jenkinsci.plugins.uptime.UptimeColumn.getShortName(UptimeColumn.java:53)
sun.reflect.GeneratedMethodAccessor425.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
...

11. © 2019 CloudBees, Inc. All Rights Reserved.
Slow Request Log
This plugin adds an "Uptime" column in the list view that shows the percentage of time that a job has been building successfully since its first build.

12. © 2019 CloudBees, Inc. All Rights Reserved.
Slow Request Log
This plugin adds an "Uptime" column in the list view that shows the percentage of time that a job has been building successfully since its first build.

13. © 2019 CloudBees, Inc. All Rights Reserved.
GC and Thread Dump Analysis
● https://gceasy.io / https://fastthread.io
● Upload a thread dump or GC log for fast analysis
● Free basic version, paid full-featured version
● Paid can be run on-premise if desired

14. © 2019 CloudBees, Inc. All Rights Reserved.
GC Analysis

15. © 2019 CloudBees, Inc. All Rights Reserved.
Thread Dump Analysis

16. © 2019 CloudBees, Inc. All Rights Reserved.
Thread Dump Analysis

17. © 2019 CloudBees, Inc. All Rights Reserved.
Scaling Horizontally

18. © 2019 CloudBees, Inc. All Rights Reserved.
When To Scale Out
● “It depends.”
● Jenkins is memory- and IO-bound; monitor them
● Avoid huge masters with thousands of jobs, etc.
● Segment masters logically: by team, business area, etc.
● Rule of thumb: no more than 16GB heap

19. © 2019 CloudBees, Inc. All Rights Reserved.
How To Scale Out
● OSS users: DIY
○ Spin up new master
○ Migrate config, validate
○ Migrate jobs, validate
○ Configuration as Code makes this easier now
● CloudBees customers: magic
○ CloudBees JOC for master management
○ One-click to spin up new masters
○ Enforce common policy and configuration
○ Centralize auth
○ Easily move jobs between masters
○ Easily trigger jobs across masters

20. © 2019 CloudBees, Inc. All Rights Reserved.
Resources
● http://www.cloudbees.com - Enterprise DevOps products including Jenkins, OSS Jenkins support, etc.
● http://gceasy.io - Online analysis of Java GC logs
● http://fastthread.io - Online analysis of Java thread dumps
● https://tinyurl.com/jenkins-jvm-args - Recommended JVM settings for Jenkins
● https://tinyurl.com/jenkins-thread-dump - How to get a thread dump from Jenkins
● @literatesavant - Tweet me if you have questions or need help!

21. © 2019 CloudBees, Inc. All Rights Reserved.
Thanks!