1. Bill Au
CBS Interactive
http://www.apache.org/~billa/oscon2012/
Troubleshooting Slow or Hung Java Applications

2. 2
Agenda
» Tools of the Trade
» Case Studies
2

3. 3
Tools – Out Of The Box
» jinfo, jmap, jstack, jstat
» JConsole, VisualVM
– Monitor CPU usage by using JTop plug-in ($JAVA_HOME/
demo/management/JTop)
» java.lang.management
– Platform MXBeans: runtime, operating system, class loading,
compilation, garbage collection, memory, thread
– Sample code in $JAVA_HOME/demo/management
– JDK demo and sample was part of JDK download for
jdk6u31 or older, separate download for u32+
3

4. 4
Tools – java.lang.management
» Sample code:
import java.lang.management.*;
ThreadMXBean tb=ManagementFactory.getThreadMXBean();
if (tb.isThreadCpuTimeSupported()) {
long[] tids=tb.getAllThreadIds();
for (long tid : tids) {
System.out.println(“cputime:”+tb.getThreadCpuTime(tid));
System.out.println(“usertime:”+tb.getThreadUserTime(tid));
}
}
4

5. 5
Tools
» Garbage collection log
– -Xloggc:<filename>
– Timestamp and duration of GC events
– Sizes before and after GC events
– More options to explore:
− -XX:+PrintGCDateStamps
− -XX:+PrintGCDetails
− -XX:+PrintTenuringDistribution
− -XX:+PrintGCApplicationsStoppedTime
– Some GC flags can be set dynamically by jinfo/JConsole/
VisualVM:
PrintGC, PrintGCDetails, PrintGCTimeStamps
» visualgc – GUI for garbage collection monitoring
5

6. 6
Tools
» Thread dump
– kill -3 (or kill –SIGHUP)
– JConsole, VisualVM
– ThreadMXBean
– jstack (-F)
» Consecutive thread dumps
– Stop motion animation
6

7. 7
Hung or Slow App
» Long GC pauses and/or high GC overhead
– Basic GC tuning
− max heap size: -Xmx
− type of collector used
− -XX:+UserSerialGC
− -XX:+UseParallelGC/-XX:+UseParallelOldGC
− -XX:+UseConcMarkSweepGC
− -XX:+UnlockExperimentalVMOptions -XX:+UseG1GC (jdk6u14+)
– Advanced GC tuning
− young generation size: -Xmn
– Baseline, tune, measure one change at a time
» Demo: HPjmeter – garbage collection analysis (and
more…)
7

8. 8
Hung or Slow App
» Deadlock
– Take thread dump to run the JVM’s deadlock detector
» Looping Threads
– Monitor CPU times of running threads
− ThreadMXBean
− JTop plug-in for JConsole/VisualVM
– Inspect stack trace of suspicious threads
8

9. 9
Slow or Hung App
» Blocked threads
– Look for blocked threads waiting on the same condition
– Identify bottleneck for resource contention
– Inspect code for incautious and/or improper synchronization
» Demo: samurai – thread dump analysis tool
» Stuck threads
– Look for runnable threads with the same stack trace across
consecutive thread dumps
– Some common causes:
− Blocking for a network call without specifying connect and/or
read timeout (default may be no timeout)
− Long execution time for calls to backend servers
9

10. 10
Links
» Presentation
http://www.apache.org/~billa/oscon2012
» HPjmeter
https://h20392.www2.hp.com/portal/swdepot/
displayProductInfo.do?productNumber=HPJMETER
» samurai
http://yusuke.homeip.net/samurai/en/index.html
https://github.com/yusuke/samurai
10