Overview
Real life situations
Thread dump format
Taking thread dumps
TDA tools
Reading material
Questions
Real life situations
Problems with I/O
Threads maybe blocked or perform slowly during access to I/O resources:
    File system
    Network
    Database

Single or multiple thread dumps may help understading the problem
Problems with
      algorithms
Algorithm complexity is not suited well for the data set size
Algorithm is overusing expensive resources or memory
Algorithm is overusing locks, while it can be implemented in a lock-free manner
Single or multiple thread dumps may help noticing the problem
Problems with
      locks and
    syncronization
No locking (not thread-safe implementations)
Deadlocks (overly syncronised)
Lock contention (overly syncronised)
Lock overhead (too many locks)
Single or multiple thread dumps may explain the situation
Example:
 HashMap gives
 100% CPU load
Usage of not syncronised HashMap in a multi-threaded application
In certain situations it goes into infinite loop
This causes 100% CPU load
The probabilty of entering infinite loop depends on the size of the HashMap
Single thread dump reveals the problem
From Wikipedia:
                   Deadlocks
  “A deadlock is a situation in which two or more competing actions are each waiting
  for the other to finish, and thus neither ever does.
Deadlocks
In real life, deadlocks are pretty rare
CPU usage is usally close to 0
Single thread dump can reveal a deadlock situation
Lock contention
From Wikipedia:

  “This occurs whenever one process or thread attempts to acquire a lock held by
  another process or thread. The more granular the available locks, the less likely one
  process/thread will request a lock held by the other. (For example, locking a row
  rather than the entire table, or locking a cell rather than the entire row.)
Lock contention
The most common problem with multi-threaded applications related to lock
handling
Sometimes easy, sometimes very hard to detect
You can only see one layer at a time
Multiple thread dumps may help to detect lock contention situations
Sporadic
             application
             behaviour
May be related to not thread safe code
May be related to load patterns
May be related to specific data
Threads dumps may or may not help
Thread dump
   format
Formats
Formats differ for various JVM and application server implementations
There is no tool that supports them all
Though format is usually very readble and follows same principles
HotSpot format
Thread states I
http://docs.oracle.com/javase/1.5.0/docs/api/java/lang/Thread.State.html

  “A thread can be in one of the following states:

       NEW - A thread that has not yet started is in this state.

       RUNNABLE - A thread executing in the Java virtual machine is in this state.

       BLOCKED - A thread that is blocked waiting for a monitor lock is in this state.
Thread states II
“
WAITING - A thread that is waiting indefinitely for another thread to perform a
particular action is in this state.
TIMED_WAITING - A thread that is waiting for another thread to perform an
action for up to a specified waiting time is in this state.
TERMINATED - A thread that has exited is in this state.
Taking thread
   dumps
How?
jstack
jconsole
jvisualvm
write your own JMX client
use your favorite profiler
Demo
Thread dump
analysis tools
TDA tools
IBM Thread and Monitor Dump analysis
Java Thread Dump Analyzer
Samurai
Thread Dump Analyzer
Thread Dump Viewer
Demo
Book
Questions?
JVM Internals
           Training
Aestas/IT organizes "JVM Internals" training on March 25/26, 2013.

This 2–day course takes the developer on an in-depth tour of the Java Virtual Machine.
Intended for experienced JVM-based programmers, who want to get deeper with the
platform, this course will introduce the major subsystems of the JVM and practical ways
to apply this knowledge in real applications.
Topics I
History of Java and JVM
Specifications
Bytecode
Base components
Detailed architecture
JIT compilation
Garbage collection
Threads
Locks
Classloading
Topics II
Security
Monitoring tools
Java agents
Debugging
Memory analysis
Tuning principles
GC tuning
Responsiveness and throughput
Tuning Options
Registration
Training registration at:

http://aestasit.com/jvm-training-riga-2013

Or by sending e-mail to:

training@aestasit.com
Thanks!

Thread Dump Analysis

  • 1.
    Overview Real life situations Threaddump format Taking thread dumps TDA tools Reading material Questions
  • 2.
  • 3.
    Problems with I/O Threadsmaybe blocked or perform slowly during access to I/O resources: File system Network Database Single or multiple thread dumps may help understading the problem
  • 4.
    Problems with algorithms Algorithm complexity is not suited well for the data set size Algorithm is overusing expensive resources or memory Algorithm is overusing locks, while it can be implemented in a lock-free manner Single or multiple thread dumps may help noticing the problem
  • 5.
    Problems with locks and syncronization No locking (not thread-safe implementations) Deadlocks (overly syncronised) Lock contention (overly syncronised) Lock overhead (too many locks) Single or multiple thread dumps may explain the situation
  • 6.
    Example: HashMap gives 100% CPU load Usage of not syncronised HashMap in a multi-threaded application In certain situations it goes into infinite loop This causes 100% CPU load The probabilty of entering infinite loop depends on the size of the HashMap Single thread dump reveals the problem
  • 7.
    From Wikipedia: Deadlocks “A deadlock is a situation in which two or more competing actions are each waiting for the other to finish, and thus neither ever does.
  • 8.
    Deadlocks In real life,deadlocks are pretty rare CPU usage is usally close to 0 Single thread dump can reveal a deadlock situation
  • 9.
    Lock contention From Wikipedia: “This occurs whenever one process or thread attempts to acquire a lock held by another process or thread. The more granular the available locks, the less likely one process/thread will request a lock held by the other. (For example, locking a row rather than the entire table, or locking a cell rather than the entire row.)
  • 10.
    Lock contention The mostcommon problem with multi-threaded applications related to lock handling Sometimes easy, sometimes very hard to detect You can only see one layer at a time Multiple thread dumps may help to detect lock contention situations
  • 11.
    Sporadic application behaviour May be related to not thread safe code May be related to load patterns May be related to specific data Threads dumps may or may not help
  • 12.
  • 13.
    Formats Formats differ forvarious JVM and application server implementations There is no tool that supports them all Though format is usually very readble and follows same principles
  • 14.
  • 15.
    Thread states I http://docs.oracle.com/javase/1.5.0/docs/api/java/lang/Thread.State.html “A thread can be in one of the following states: NEW - A thread that has not yet started is in this state. RUNNABLE - A thread executing in the Java virtual machine is in this state. BLOCKED - A thread that is blocked waiting for a monitor lock is in this state.
  • 16.
    Thread states II “ WAITING- A thread that is waiting indefinitely for another thread to perform a particular action is in this state. TIMED_WAITING - A thread that is waiting for another thread to perform an action for up to a specified waiting time is in this state. TERMINATED - A thread that has exited is in this state.
  • 17.
  • 18.
    How? jstack jconsole jvisualvm write your ownJMX client use your favorite profiler
  • 19.
  • 20.
  • 21.
    TDA tools IBM Threadand Monitor Dump analysis Java Thread Dump Analyzer Samurai Thread Dump Analyzer Thread Dump Viewer
  • 22.
  • 23.
  • 24.
  • 25.
    JVM Internals Training Aestas/IT organizes "JVM Internals" training on March 25/26, 2013. This 2–day course takes the developer on an in-depth tour of the Java Virtual Machine. Intended for experienced JVM-based programmers, who want to get deeper with the platform, this course will introduce the major subsystems of the JVM and practical ways to apply this knowledge in real applications.
  • 26.
    Topics I History ofJava and JVM Specifications Bytecode Base components Detailed architecture JIT compilation Garbage collection Threads Locks Classloading
  • 27.
    Topics II Security Monitoring tools Javaagents Debugging Memory analysis Tuning principles GC tuning Responsiveness and throughput Tuning Options
  • 28.
  • 29.