Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Why does my jenkins freeze sometimes and what can I do about it?

1,131 views

Published on

After getting frustrated with Jenkins getting slow or even stuck occasionally, I decided to investigate the root cause for that bad performance. In this talk I will show my findings about that issue (hint: GC), how Jenkins performance can be easily improved by tuning the JVM GC and a few exciting tools I've discovered along the way.
Originally presented at Jenkins TLV meetup @ Taboola

Published in: Software
  • Be the first to comment

Why does my jenkins freeze sometimes and what can I do about it?

  1. 1. Why does my Jenkins freeze sometimes and what can I do about it? Jenkins session you may like
  2. 2. Taboola - Numbers On Average Every American Sees Taboola 70 Times a Month (comScore) 500K+ Requests/sec 15B recommendation/day 1B Monthly unique users 17TB+ incoming raw data/day 8 Data-Centers globally, with over ~2500 Production servers
  3. 3. Taboola - Jenkins 4 Jenkins servers 48 slaves Hundreds of builds / day ~ 5 releases per day
  4. 4. Taboola - Technologies 6
  5. 5. Agenda ● The problem ● JVM architecture ● Garbage collectors ● The solution
  6. 6. The problem Problem: Jenkins pages get load very slowly
  7. 7. JVM architecture http://www.oracle.com/technetwork/tutorials/tutorials-1876574.html
  8. 8. JVM architecture http://coding-geek.com/jvm-memory-model/
  9. 9. JVM architecture http://javatutorialhere.blogspot.co.il/2014/08/how-objects-are-stored-in-heap-and.html
  10. 10. JVM architecture Choose >, < or = 32GB Heap 35GB Heap ?
  11. 11. JVM architecture -Xmx - heap maximum size -Xms - initial heap size jinfo -flag UseCompressedOops <pid>
  12. 12. JVM architecture http://java-latte.blogspot.co.il/2014/03/metaspace-in-java-8.html
  13. 13. Garbage Collector
  14. 14. Garbage Collector
  15. 15. Garbage Collector https://medium.com/@petloverooka/heres-how-garbage-collection-in-java-really-work-2acacc171c1
  16. 16. Garbage Collector http://www.oracle.com/technetwork/tutorials/tutorials-1876574.html
  17. 17. The solution - G1GC and logs Enable G1Gc: -XX:+UseG1GC Logs: -Xloggc:/myJenkinsDir/gc-%t.log -XX:NumberOfGCLogFiles=5 -XX:+UseGCLogFileRotation - XX:GCLogFileSize=20m -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps - XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintGCCause - XX:+PrintTenuringDistribution -XX:+PrintReferenceGC -XX:+PrintAdaptiveSizePolicy - verbose:gc -XX:+UnlockDiagnosticVMOptions -XX:+PrintFlagsFinal
  18. 18. The solution - GC logs
  19. 19. The solution - Tools http://gceasy.io/
  20. 20. The solution - Tools
  21. 21. The solution - More GC Flags -XX:MaxGCPauseMillis=<n> (default 200ms) -XX:InitiatingHeapOccupancyPercent=45 -XX:ParallelGCThreads=15 -XX:ConcGCThreads=4 -XX:+PrintStringDeduplicationStatistics -XX:+UseStringDeduplication -XX:+DisableExplicitGC G1GC Tuning
  22. 22. The solution - Jconsole
  23. 23. The solution - Jconsole -Djava.rmi.server.hostname=<your Jenkins host> - Dcom.sun.management.jmxremote.port=<choose a port> - Dcom.sun.management.jmxremote.authenticate=false - Dcom.sun.management.jmxremote.ssl=false
  24. 24. The solution - Jenkins Old data Jenkins Old data: http://yourJenkinsHost/administrativeMonitor/OldData/manage
  25. 25. The solution - Summary ● Switch to G1GC ● Enable and analyze logs ● Add relevant flags ● Analyze logs again ● Remove old Jenkins data ● Monitor
  26. 26. Thank You! Tidhar Klein Orbach @tizkiko tidharko Tidhar Klein Orbach
  27. 27. Resources ● Joining the Big Leagues: Tuning Jenkins GC For Responsiveness and Stability ● JVM memory model ● Getting Started with the G1 Garbage Collector ● Everything I Ever Learned about JVM Performance Tuning @twitter ● Here’s How Garbage Collection in JAVA Really Work ● Java HotSpot VM Options ● Why 35GB Heap is Less Than 32GB – Java JVM Memory Oddities

×