The slides cover an introduction to debugging techniques for performance bottlenecks in a J2EE app with a real life example. There is an additional pdf document uploaded covering thread dumps at http://www.slideshare.net/AjitBhingarkar/thread-dumps which needs to be referred to.

1. Debugging performance bottlenecks
in a Java Enterprise Application
March 2012

2. Diagnostic parameters to monitor
• CPU usage
• Java heap usage
• Thread dumps
• Heap dumps
• Database – ADDM report for Oracle
• File system (read/write of files)

3. CPU Usage
• Monitor using JVisualVM or JConsole or command
‘top’ or ‘vmstat’ on UNIX
• High CPU usage shows high activity
• For higher loads this may be normal/expected
• For not so high loads, we need to debug why CPU
usage is high.
• For low CPU usages, when performance is not as
expected further analysis is required to bring it up.

5. CPU usage cont…
• Reasons:
– High number of threads are spawned without throttling (e.g. blocking
queue) in place.
– Too much of resource contention (lot of synchronized blocks in code)
– Threads stuck doing not so optimized DB reads
– Too much of XML validation/parsing or XPath or translation using XSLT
• How to debug High/Low CPU usage or
low throughput?
– Thread dump gets us a snapshot of the system to see how different
threads are executing
– You can get thread dumps by using jvisualvm or jconsole or using
command <JAVA_HOME>/bin/jstack –l <pid>.

8. How to analyze thread dumps:
Thread states: A thread typically shows following states of
importance in a thread dump
1. Runnable – thread is executing/running in JVM
• Locked ownable synchronizers: <objid> -> A ReentratLock help by
current thread on which other threads may be waiting for.
2. Waiting – thread is waiting for another thread to perform some
action
• waiting on <0x4a09ff12> : Indicates inter thread
communication through WAIT/NOTIFY
• parking to wait for <objid> : We are waiting for condition
object <objid> or on blocking queues
3. Blocked – thread is blocked waiting for a monitor (indicates
resource contention like DB locks)
• waiting to lock <objid>

9. Heap dumps
Why do we need heap dumps (.hprof files) ?
1. To analyze OutOfMemory (OOM) issues
• Always use -XX:+ HeapDumpOnOutOfMemoryError JVM
flag so that we get the heap dump captured on every OOM
automatically. Dumps captured before or later do not
provide any diagnostic information.
E.g. java -XX:+HeapDumpOnOutOfMemory -
XX:HeapDumpPath=D:myApphprof-dumps myApp
2. To analyze memory leaks – heap keeps growing without any
GC activity, slowing down the performance after certain
amount of time.

10. Heap dumps
How do we capture heap dumps (.hprof
files) ?
• JVisualVM and JConsole both have an option to
capture heap dump at any specific point.
• Explicitly: kill –QUIT (UNIX) and CTRL<Break>
(Windows)

12. Heap dumps
How do we analyze heap dumps (.hprof files) ?
1. JVisualVM allows you to run profiler and find out
memory footprint for Java objects.
2. Java Heap Analysis Tool (jhat) allows us to browse
a heap dump file. Usage: jhat <heap dump file>
Profilers:
Often information from heap browsing may not be
conclusive, and we may have to profile the application
using profilers like YourKit or JProfiler.

14. Real life use case
Symptoms:
1. Java application deployed in JBoss app
server used to go down with OOM or JVM
crash in about an hour’s time after restart.
2. Heap usage would never come down, with no
GC happening.
3. Slow performance of Web-Service requests
executed on the application.

15. Real life use case: CPU and Heap
Usage Pattern

16. Real life use case
• Analysis of thread dumps shows that out of 8 asynchronous processing
threads, 7 were busy on the database operations, mostly reads.
• Analyzing database ops indicated missing index, and further query tuning
resolved the issue. Check out the subsequent CPU and Heap usage pattern
on the next slide.
Pls refer to the companion document (thread-dumps.
pdf) also uploaded with this slide deck. Some
parts of the thread stack traces were removed to
avoid showing customer or product information.
Important segments are highlighted.
(Note: A single thread dump is never enough, you have to take a few every 30 seconds or every
minute to find any meaningful information and any conclusion.)

18. Database performance diagnostics
- Oracle
The Automatic Database Diagnostic Monitor (ADDM)
analyses data in the Automatic Workload Repository (AWR)
to identify potential performance bottlenecks.
• How to generate ADDM report?
1. Run Oracle enterprise manager e.g. http://<IP
Address>:1158/em
Go to “Advisor Central” -> ADDM.
2. Run script from SQLPlus e.g.
@C:oracleproduct10.2.0db_1rdbmsadminaddmrpt.
sql

19. Thank You!
A_BHINGARKAR@YAHOO.COM