Thread dump troubleshooting
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Thread dump troubleshooting

on

  • 17,955 views

 

Statistics

Views

Total Views
17,955
Views on SlideShare
15,171
Embed Views
2,784

Actions

Likes
7
Downloads
399
Comments
0

47 Embeds 2,784

http://www.wikiconsole.com 1035
http://wlatricksntips.blogspot.in 725
http://wlatricksntips.blogspot.com 508
http://www.bernie-eng.com 81
http://wlatricksntips.blogspot.co.uk 71
http://wlatricksntips.blogspot.sg 40
http://wlatricksntips.blogspot.ca 29
http://wlatricksntips.blogspot.com.au 25
http://wlatricksntips.blogspot.de 24
http://wlatricksntips.blogspot.ru 19
http://www.scoop.it 19
http://wlatricksntips.blogspot.fr 18
http://wlatricksntips.blogspot.hk 17
http://wlatricksntips.blogspot.cz 15
http://wlatricksntips.blogspot.it 14
http://wlatricksntips.blogspot.com.br 14
http://wlatricksntips.blogspot.com.es 13
http://wlatricksntips.blogspot.nl 12
http://wlatricksntips.blogspot.mx 10
http://wlatricksntips.blogspot.com.tr 10
http://wlatricksntips.blogspot.be 9
http://wlatricksntips.blogspot.gr 8
http://wlatricksntips.blogspot.kr 6
http://wlatricksntips.blogspot.jp 6
http://wlatricksntips.blogspot.ie 6
http://wlatricksntips.blogspot.com.ar 5
http://www.slideshare.net 4
http://wlatricksntips.blogspot.ae 4
http://wlatricksntips.blogspot.fi 4
http://wlatricksntips.blogspot.ch 4
http://wlatricksntips.blogspot.pt 3
http://wlatricksntips.blogspot.tw 3
http://wlatricksntips.blogspot.no 3
http://wlatricksntips.blogspot.se 3
http://wlatricksntips.blogspot.co.il 2
http://feeds2.feedburner.com 2
http://wlatricksntips.blogspot.hu 2
http://wlatricksntips.blogspot.ro 2
http://webcache.googleusercontent.com 1
http://prlog.ru 1
http://news.google.com 1
http://wlatricksntips.blogspot.dk 1
http://wildfire.gigya.com 1
http://static.slidesharecdn.com 1
http://feeds.feedburner.com 1
http://plus.url.google.com 1
http://www.google.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Thread dump troubleshooting Presentation Transcript

  • 1. Thread Dump Troubleshooting
    Jie Chen
    March 2010
  • 2. ©2010 Oracle Corporation
    Agenda
    Why & When to take Thread Dump
    How to take Thread Dump
    Thread State
    Analyzing
    Case Study
  • 3. ©2010 Oracle Corporation
    Why & When to take Thread Dump
    What / Why
    A snapshot of all threads in Java application
    Showing each thread’s state / health
    When
    Bad performance
    System hangs
    CPU High Usage
  • 4. How to take Thread Dump
    Weblogic
    Windows: (Note 659452.1)
    Edit *.cmd to output the dump
    Press Ctrl+Break
    Unix/Linux: (Note 659452.1)
    Edit *.sh
    Get PID by “ps -ef|grep java”
    “Kill -3 <PID>”
    ©2010 Oracle Corporation
    E.g, startAgile.cmd
    %JAVA_HOME%injava %JAVA_VM% %MEM_ARGS% %JAVA_OPTIONS% -Dweblogic.Name=%SERVER_NAME% … weblogic.Server>c:/temp/mydump.log
  • 5. How to take Thread Dump
    Oracle AS
    Windows: (Note 437464.1)
    Get PID (Process ID) by “opmnctl status –l”
    Run thdump utility by “thdump <PID>”
    Unix/Linux (Note 361729.1)
    Get PID by “opmnctl status -l”
    Run “kill -3 <PID>”
    Output written to OC4J~home~default-island~1 (default)
    ©2010 Oracle Corporation
  • 6. Thread State
    Running
    Blocked / Waiting for Monitor Entry
    Waiting / Timed Waiting
    Sleeping / Waiting on Condition
    ©2010 Oracle Corporation
  • 7. Thread State
    Running
    Blocked / Waiting for Monitor Entry
    Waiting / Timed Waiting
    Sleeping / Waiting on Condition
    ©2010 Oracle Corporation
  • 8. Analyzing
    Understand Application
    Take more Dumps in a Short Duration
    Eliminate Running / Idle Threads
    Analyze Blocked / Waiting Threads
    ©2010 Oracle Corporation
  • 9. Case Study 1
    CPU High Usage
    Get Thread Dump first
    Get TID (Thread ID)
    Process Explorer (Windows)
    “ps –eL|grep java” (Unix/Linux)
    3. Convert TID to hexadecimal
    4. Find out the problem stack
    ©2010 Oracle Corporation
  • 10. Case Study 2
    Classic Deadlock
    One thread is blocked to acquire resource which is hold by other blocked thread(s)
    ©2010 Oracle Corporation
    ===================================================
    "Thread-1":
    at zigzag.threaddump.demo.TestDeadLock.run(TestDeadLock.java:27)
    - waiting to lock <0x22a15c98> (a java.lang.Object)
    - locked <0x22a15ca0> (a java.lang.Object)
    at java.lang.Thread.run(Unknown Source)
    "Thread-0":
    at zigzag.threaddump.demo.TestDeadLock.run(TestDeadLock.java:16)
    - waiting to lock <0x22a15ca0> (a java.lang.Object)
    - locked <0x22a15c98> (a java.lang.Object)
    at java.lang.Thread.run(Unknown Source)
    Found 1 deadlock.
    System Hangs
  • 11. Case Study 3
    Resource Contention
    Two or more threads are fighting for the same resource(s)
    ©2010 Oracle Corporation
    "Thread-2" prio=6 tid=0x02b48000 nid=0x120c waiting for monitor entry [0x02f1f000]
    java.lang.Thread.State: BLOCKED (on object monitor)
    at zigzag.threaddump.demo.ResourceContention.run(ResourceContention.java:11)
    - waiting to lock <0x22a15e30> (a java.lang.Object)
    at java.lang.Thread.run(Unknown Source)
    "Thread-1" prio=6 tid=0x02b46c00 nid=0xc3c waiting for monitor entry [0x02ecf000]
    java.lang.Thread.State: BLOCKED (on object monitor)
    at zigzag.threaddump.demo.ResourceContention.run(ResourceContention.java:11)
    - waiting to lock <0x22a15e30> (a java.lang.Object)
    at java.lang.Thread.run(Unknown Source)
    "Thread-0" prio=6 tid=0x02b45800 nid=0x49c waiting on condition [0x02e7f000]
    java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at zigzag.threaddump.demo.ResourceContention.run(ResourceContention.java:12)
    - locked <0x22a15e30> (a java.lang.Object)
    at java.lang.Thread.run(Unknown Source)
    Bad Performance
  • 12. Case Study 4
    Out of Threads
    No more room for additional threads running
    ©2010 Oracle Corporation
    E.g. 5 tasks need to run but only 2 threads are allowed concurrently.
    Task 2,3 and 4 have to wait until Task 0 and 1 finish.
    "Zigzag_ThreadPool-1" prio=6 tid=0x02b47000 nid=0x133c runnable [0x02ecf000]
    java.lang.Thread.State: RUNNABLE
    at zigzag.threaddump.demo.ThreadPoolTest$1.run(ThreadPoolTest.java:44)
    at zigzag.threaddump.demo.ThreadPool$PooledThread.run(ThreadPoolTest.java:185)
    "Zigzag_ThreadPool-0" prio=6 tid=0x02b46400 nid=0x11d0 runnable [0x02e7f000]
    java.lang.Thread.State: RUNNABLE
    at zigzag.threaddump.demo.ThreadPoolTest$1.run(ThreadPoolTest.java:44)
    at zigzag.threaddump.demo.ThreadPool$PooledThread.run(ThreadPoolTest.java:185)
    Bad Performance
  • 13. Case Script
    CPU High Usage
    Classic Deadlock
    Resource Contention
    Out of Threads
    ©2010 Oracle Corporation
  • 14. Third Part Utility
    Process Explorer
    http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx
    TDA (Thread Dump Analyzer)
    https://tda.dev.java.net
    ©2010 Oracle Corporation
  • 15. ©2010 Oracle Corporation
    Q/A