スローダウン、ハングを一発解決 スレッドダンプはトラブルシューティングの味方 #wlstudy

  • 3,635 views
Uploaded on

 

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
3,635
On Slideshare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
55
Comments
0
Likes
13

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. スローダウン、ハングを一発解決スレッドダンプはトラブルシューティングの味方 山本 裕介 @yusuke Yusuke Yamamoto @yusuke #wlstudy
  • 2. スローダウン、ハングYusuke Yamamoto @yusuke #wlstudy
  • 3. スローダウン• アプリケーションは動いている• レスポンスがなかなか帰ってこないYusuke Yamamoto @yusuke #wlstudy
  • 4. ハング• アプリケーションが固まる• 応答がない• 503が帰ってくるYusuke Yamamoto @yusuke #wlstudy
  • 5. スローダウン、ハングの原因• マシンスペック不足 • キャパシティプランニングに間違い ‣ キャパシティプランニングをしていない ‣ 台数不足• 割り当てCPU/メモリ不足 • 仮想化時のリソース配分Yusuke Yamamoto @yusuke #wlstudy
  • 6. スローダウン、ハングの原因• チューニング不足 • JVMオプション • カーネルオプション• アプリケーションサーバ設定Yusuke Yamamoto @yusuke #wlstudy
  • 7. スローダウン、ハングの原因• アプリケーション • 非効率なロジック • リソースの奪い合い • I/O待ちYusuke Yamamoto @yusuke #wlstudy
  • 8. 切り分け• GCログの取得• GC時の挙動を確認• スレッドダンプの取得、解析Yusuke Yamamoto @yusuke #wlstudy
  • 9. スレッドダンプとは• JVMのスレッド状態のスナップショット• わかること • ある時点で • それぞれのスレッドが • 何をしているかYusuke Yamamoto @yusuke #wlstudy
  • 10. スレッドダンプでわからないこと• クラッシュ / ストールの原因• プロセスが落ちたらスレッドダンプは 取れないYusuke Yamamoto @yusuke #wlstudy
  • 11. スレッドダンプの利点• 追加ツール不要 • JProfiler • JProbe • YourKit Java Profiler• 取得時に負荷がかからない • 運用環境でも手軽に使える• オフラインで解析できるYusuke Yamamoto @yusuke #wlstudy
  • 12. スレッドダンプの見た目Yusuke Yamamoto @yusuke #wlstudy
  • 13. 2013-01-23 16:32:13Full thread dump Java HotSpot(TM) 64-Bit Server VM (23.6-b04 mixed mode): at java.lang.Object.wait(Native Method)"Attach Listener" daemon prio=5 tid=0x00007f8dbd9dd000 nid=0x9407 waiting on condition [0x0000000000000000] at java.lang.Object.wait(Object.java:503) - locked <0x000000011ab16fa8> (a weblogic.work.ExecuteThread) java.lang.Thread.State: RUNNABLE at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) - locked <0x000000011ab16fa8> (a weblogic.work.ExecuteThread)"[STANDBY] ExecuteThread: 40 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbcd3c000 nid=0x9303 in Object.wait() [0x0000000144526000] at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: 18 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dba9a3000 nid=0x7d03 in Object.wait() [0x0000000142d1b000] java.lang.Thread.State: WAITING (on object monitor) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) "[ACTIVE] ExecuteThread: 18 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dba9a3000 nid=0x7d03 in Object.wait() [0x0000000142d1b000] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.lang.Object.wait(Native Method) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011c40fc88> (a weblogic.work.ExecuteThread) at java.lang.Object.wait(Object.java:503) - locked <0x000000011ab18448> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) - locked <0x000000011ab18448> (a weblogic.work.ExecuteThread)"[STANDBY] ExecuteThread: 39 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbdab9800 nid=0x9203 in Object.wait() [0x000000014437b000] at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[STANDBY] ExecuteThread: 17 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbae29800 nid=0x7c03 in Object.wait() [0x0000000142c18000] java.lang.Thread.State: WAITING (on object monitor) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) "[STANDBY] ExecuteThread: 17 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbae29800 nid=0x7c03 in Object.wait() [0x0000000142c18000] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.lang.Object.wait(Native Method) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab2f568> (a weblogic.work.ExecuteThread) at java.lang.Object.wait(Object.java:503) - locked <0x000000011ab15510> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) - locked <0x000000011ab15510> (a weblogic.work.ExecuteThread)"[ACTIVE] ExecuteThread: 38 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbd1bf000 nid=0x9103 in Object.wait() [0x0000000144278000] at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: 16 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbd0b7000 nid=0x7b03 in Object.wait() [0x0000000142b15000] java.lang.Thread.State: WAITING (on object monitor) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) "[ACTIVE] ExecuteThread: 16 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbd0b7000 nid=0x7b03 in Object.wait() [0x0000000142b15000] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.lang.Object.wait(Native Method) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab2e4a0> (a weblogic.work.ExecuteThread) at java.lang.Object.wait(Object.java:503) - locked <0x000000011ab17fc8> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) - locked <0x000000011ab17fc8> (a weblogic.work.ExecuteThread)"[ACTIVE] ExecuteThread: 37 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dba80b000 nid=0x9003 in Object.wait() [0x0000000144175000] at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: 15 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbcaa6800 nid=0x7a03 in Object.wait() [0x0000000142a12000] java.lang.Thread.State: WAITING (on object monitor) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) "[ACTIVE] ExecuteThread: 15 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbcaa6800 nid=0x7a03 in Object.wait() [0x0000000142a12000] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.lang.Object.wait(Native Method) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab2d1b0> (a weblogic.work.ExecuteThread) at java.lang.Object.wait(Object.java:503) - locked <0x000000011ab1aeb8> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) - locked <0x000000011ab1aeb8> (a weblogic.work.ExecuteThread)"[ACTIVE] ExecuteThread: 36 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dba479800 nid=0x8f03 in Object.wait() [0x0000000144072000] at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: 14 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbbc5c800 nid=0x7903 in Object.wait() [0x0000000142856000] java.lang.Thread.State: WAITING (on object monitor) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) "[ACTIVE] ExecuteThread: 14 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbbc5c800 nid=0x7903 in Object.wait() [0x0000000142856000] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.lang.Object.wait(Native Method) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab27da0> (a weblogic.work.ExecuteThread) at java.lang.Object.wait(Object.java:503) - locked <0x000000011ab1bed8> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) - locked <0x000000011ab1bed8> (a weblogic.work.ExecuteThread)"[STANDBY] ExecuteThread: 35 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8db959f000 nid=0x8e03 in Object.wait() [0x0000000143f6f000] at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: 13 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbb83c000 nid=0x7803 in Object.wait() [0x0000000142650000] java.lang.Thread.State: WAITING (on object monitor) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) "[ACTIVE] ExecuteThread: 13 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbb83c000 nid=0x7803 in Object.wait() [0x0000000142650000] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.lang.Object.wait(Native Method) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab26c90> (a weblogic.work.ExecuteThread) at java.lang.Object.wait(Object.java:503) - locked <0x000000011ab144c8> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) - locked <0x000000011ab144c8> (a weblogic.work.ExecuteThread)"[ACTIVE] ExecuteThread: 34 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dba9a4000 nid=0x8d03 in Object.wait() [0x0000000143e6c000] at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: 12 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbd9c0000 nid=0x7703 in Object.wait() [0x0000000140a29000] java.lang.Thread.State: WAITING (on object monitor) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) "[ACTIVE] ExecuteThread: 12 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbd9c0000 nid=0x7703 in Object.wait() [0x0000000140a29000] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.lang.Object.wait(Native Method) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab29978> (a weblogic.work.ExecuteThread) at java.lang.Object.wait(Object.java:503) - locked <0x000000011ab19548> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) - locked <0x000000011ab19548> (a weblogic.work.ExecuteThread)"[STANDBY] ExecuteThread: 33 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbbc5b800 nid=0x8c03 in Object.wait() [0x0000000143d69000] at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: 11 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dba94d800 nid=0x7603 in Object.wait() [0x0000000142347000] java.lang.Thread.State: WAITING (on object monitor) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) "[ACTIVE] ExecuteThread: 11 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dba94d800 nid=0x7603 in Object.wait() [0x0000000142347000] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.lang.Object.wait(Native Method) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab29a28> (a weblogic.work.ExecuteThread) at java.lang.Object.wait(Object.java:503) - locked <0x000000011ab0d3f8> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) - locked <0x000000011ab0d3f8> (a weblogic.work.ExecuteThread)"[ACTIVE] ExecuteThread: 32 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbbc5b000 nid=0x8b03 in Object.wait() [0x0000000143c66000] at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[STANDBY] ExecuteThread: 10 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbe196000 nid=0x7503 in Object.wait() [0x0000000140c23000] java.lang.Thread.State: WAITING (on object monitor) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) "[STANDBY] ExecuteThread: 10 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbe196000 nid=0x7503 in Object.wait() [0x0000000140c23000] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.lang.Object.wait(Native Method) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab25c70> (a weblogic.work.ExecuteThread) at java.lang.Object.wait(Object.java:503) - locked <0x000000011ab0e418> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) - locked <0x000000011ab0e418> (a weblogic.work.ExecuteThread)"[ACTIVE] ExecuteThread: 31 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbbd54800 nid=0x8a03 in Object.wait() [0x00000001435d3000] at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: 9 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbe195800 nid=0x7403 in Object.wait() [0x00000001408be000] java.lang.Thread.State: WAITING (on object monitor) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) "[ACTIVE] ExecuteThread: 9 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbe195800 nid=0x7403 in Object.wait() [0x00000001408be000] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.lang.Object.wait(Native Method) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab28da8> (a weblogic.work.ExecuteThread) at java.lang.Object.wait(Object.java:503) - locked <0x000000011ab10b38> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) - locked <0x000000011ab10b38> (a weblogic.work.ExecuteThread)"[ACTIVE] ExecuteThread: 30 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbbd8f800 nid=0x8903 in Object.wait() [0x00000001434d0000] at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: 8 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbba6d800 nid=0x7303 in Object.wait() [0x000000013f0a9000] java.lang.Thread.State: WAITING (on object monitor) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) "[ACTIVE] ExecuteThread: 8 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbba6d800 nid=0x7303 in Object.wait() [0x000000013f0a9000] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.lang.Object.wait(Native Method) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab2a9a0> (a weblogic.work.ExecuteThread) at java.lang.Object.wait(Object.java:503) - locked <0x000000011ab0c2d0> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) - locked <0x000000011ab0c2d0> (a weblogic.work.ExecuteThread)"[STANDBY] ExecuteThread: 29 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbbd8f000 nid=0x8803 in Object.wait() [0x0000000143b63000] at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: 7 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbc812800 nid=0x7203 in Object.wait() [0x00000001407bb000] java.lang.Thread.State: WAITING (on object monitor) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) "[ACTIVE] ExecuteThread: 7 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbc812800 nid=0x7203 in Object.wait() [0x00000001407bb000] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.lang.Object.wait(Native Method) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab229f0> (a weblogic.work.ExecuteThread) at java.lang.Object.wait(Object.java:503) - locked <0x000000011ab01498> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) - locked <0x000000011ab01498> (a weblogic.work.ExecuteThread)"[ACTIVE] ExecuteThread: 28 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbcb4e000 nid=0x8703 in Object.wait() [0x0000000143a60000] at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: 6 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8db96b5800 nid=0x7103 in Object.wait() [0x00000001406b8000] java.lang.Thread.State: WAITING (on object monitor) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) "[ACTIVE] ExecuteThread: 6 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8db96b5800 nid=0x7103 in Object.wait() [0x00000001406b8000] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.lang.Object.wait(Native Method) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab23a10> (a weblogic.work.ExecuteThread) at java.lang.Object.wait(Object.java:503) - locked <0x000000011ab024b8> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) - locked <0x000000011ab024b8> (a weblogic.work.ExecuteThread)"[ACTIVE] ExecuteThread: 27 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbbc91800 nid=0x8603 in Object.wait() [0x000000014395d000] at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[STANDBY] ExecuteThread: 5 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8db971b000 nid=0x7003 in Object.wait() [0x00000001403de000] java.lang.Thread.State: WAITING (on object monitor) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) "[STANDBY] ExecuteThread: 5 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8db971b000 nid=0x7003 in Object.wait() [0x00000001403de000] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.lang.Object.wait(Native Method) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab24a30> (a weblogic.work.ExecuteThread) at java.lang.Object.wait(Object.java:503) - locked <0x000000011ab06208> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) - locked <0x000000011ab06208> (a weblogic.work.ExecuteThread)"[ACTIVE] ExecuteThread: 26 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbb015800 nid=0x8503 in Object.wait() [0x000000014385a000] at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: 4 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbbd5d000 nid=0x6f03 in Object.wait() [0x00000001401d1000] java.lang.Thread.State: WAITING (on object monitor) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) "[ACTIVE] ExecuteThread: 4 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbbd5d000 nid=0x6f03 in Object.wait() [0x00000001401d1000] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.lang.Object.wait(Native Method) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab1f7b0> (a weblogic.work.ExecuteThread) at java.lang.Object.wait(Object.java:503) - locked <0x000000011ab07228> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) - locked <0x000000011ab07228> (a weblogic.work.ExecuteThread)"[STANDBY] ExecuteThread: 25 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dba709800 nid=0x8403 in Object.wait() [0x0000000143757000] at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "DynamicListenThread[Default]" daemon prio=5 tid=0x00007f8dbd82b000 nid=0x6e03 runnable [0x0000000142753000] java.lang.Thread.State: WAITING (on object monitor) java.lang.Thread.State: RUNNABLE at java.lang.Object.wait(Native Method) "DynamicListenThread[Default]" daemon prio=5 tid=0x00007f8dbd82b000 nid=0x6e03 runnable [0x0000000142753000] at java.net.PlainSocketImpl.socketAccept(Native Method) at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: RUNNABLE at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.ServerSocket.implAccept(ServerSocket.java:522) - locked <0x000000011ab207d0> (a weblogic.work.ExecuteThread) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at java.net.ServerSocket.accept(ServerSocket.java:490) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at java.net.ServerSocket.implAccept(ServerSocket.java:522) at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:38) at java.net.ServerSocket.accept(ServerSocket.java:490) at weblogic.server.channels.DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:535)"[ACTIVE] ExecuteThread: 24 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbcb54800 nid=0x8303 in Object.wait() [0x000000013f1cd000] at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:38) at weblogic.server.channels.DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:417) java.lang.Thread.State: WAITING (on object monitor) at weblogic.server.channels.DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:535) at weblogic.server.channels.DynamicListenThread.run(DynamicListenThread.java:173) at java.lang.Object.wait(Native Method) at weblogic.server.channels.DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:417) at java.lang.Thread.run(Thread.java:722) at java.lang.Object.wait(Object.java:503) at weblogic.server.channels.DynamicListenThread.run(DynamicListenThread.java:173) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.lang.Thread.run(Thread.java:722) "DynamicListenThread[Default[1]]" daemon prio=5 tid=0x00007f8db97d7800 nid=0x6c03 runnable [0x000000014254d000] - locked <0x000000011ab21860> (a weblogic.work.ExecuteThread) java.lang.Thread.State: RUNNABLE at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "DynamicListenThread[Default[1]]" daemon prio=5 tid=0x00007f8db97d7800 nid=0x6c03 runnable [0x000000014254d000] at java.net.PlainSocketImpl.socketAccept(Native Method) java.lang.Thread.State: RUNNABLE at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)"[ACTIVE] ExecuteThread: 23 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbc945800 nid=0x8203 in Object.wait() [0x000000014322a000] at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.ServerSocket.implAccept(ServerSocket.java:522) java.lang.Thread.State: WAITING (on object monitor) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at java.net.ServerSocket.accept(ServerSocket.java:490) at java.lang.Object.wait(Native Method) at java.net.ServerSocket.implAccept(ServerSocket.java:522) at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:38) at java.lang.Object.wait(Object.java:503) at java.net.ServerSocket.accept(ServerSocket.java:490) at weblogic.server.channels.DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:535) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:38) at weblogic.server.channels.DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:417) - locked <0x000000011ab1d8c8> (a weblogic.work.ExecuteThread) at weblogic.server.channels.DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:535) at weblogic.server.channels.DynamicListenThread.run(DynamicListenThread.java:173) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at weblogic.server.channels.DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:417) at java.lang.Thread.run(Thread.java:722) at weblogic.server.channels.DynamicListenThread.run(DynamicListenThread.java:173)"[STANDBY] ExecuteThread: 22 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8db959d800 nid=0x8103 in Object.wait() [0x0000000143127000] at java.lang.Thread.run(Thread.java:722) "DynamicListenThread[Default[7]]" daemon prio=5 tid=0x00007f8dbe02c800 nid=0x6b03 runnable [0x000000014244a000] java.lang.Thread.State: WAITING (on object monitor) java.lang.Thread.State: RUNNABLE at java.lang.Object.wait(Native Method) "DynamicListenThread[Default[7]]" daemon prio=5 tid=0x00007f8dbe02c800 nid=0x6b03 runnable [0x000000014244a000] at java.net.PlainSocketImpl.socketAccept(Native Method) at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: RUNNABLE at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.ServerSocket.implAccept(ServerSocket.java:522) - locked <0x000000011ab12600> (a weblogic.work.ExecuteThread) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at java.net.ServerSocket.accept(ServerSocket.java:490) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at java.net.ServerSocket.implAccept(ServerSocket.java:522) at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:38) at java.net.ServerSocket.accept(ServerSocket.java:490) at weblogic.server.channels.DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:535)"[ACTIVE] ExecuteThread: 21 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbcaa7000 nid=0x8003 in Object.wait() [0x0000000143024000] at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:38) at weblogic.server.channels.DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:417) java.lang.Thread.State: WAITING (on object monitor) at weblogic.server.channels.DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:535) at weblogic.server.channels.DynamicListenThread.run(DynamicListenThread.java:173) at java.lang.Object.wait(Native Method) at weblogic.server.channels.DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:417) at java.lang.Thread.run(Thread.java:722) at java.lang.Object.wait(Object.java:503) at weblogic.server.channels.DynamicListenThread.run(DynamicListenThread.java:173) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.lang.Thread.run(Thread.java:722) "DynamicListenThread[Default[ - locked <0x000000011ab13620> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "DynamicListenThread[Default[4]]" daemon prio=5 tid=0x00007f8dbc8c7800 nid=0x6903 runnable [0x0000000142244000] java.lang.Thread.State: RUNNABLE"[STANDBY] ExecuteThread: 20 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbb832800 nid=0x7f03 in Object.wait() [0x0000000142f21000] at java.net.PlainSocketImpl.socketAccept(Native Method) java.lang.Thread.State: WAITING (on object monitor) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at java.lang.Object.wait(Native Method) at java.net.ServerSocket.implAccept(ServerSocket.java:522) at java.lang.Object.wait(Object.java:503) at java.net.ServerSocket.accept(ServerSocket.java:490) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:38) - locked <0x000000011ab164e0> (a weblogic.work.ExecuteThread) at weblogic.server.channels.DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:535) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at weblogic.server.channels.DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:417) at weblogic.server.channels.DynamicListenThread.run(DynamicListenThread.java:173)"[ACTIVE] ExecuteThread: 19 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbbc5d800 nid=0x7e03 in Object.wait() [0x0000000142e1e000] at java.lang.Thread.run(Thread.java:722) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) "DynamicListenThread[Default[8]]" daemon prio=5 tid=0x00007f8dba13f800 nid=0x6803 runnable [0x0000000142141000] at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: RUNNABLE at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.net.PlainSocketImpl.socketAccept(Native Method) - locked <0x000000011ab16fa8> (a weblogic.work.ExecuteThread) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at java.net.ServerSocket.implAccept(ServerSocket.java:522) at java.net.ServerSocket.accept(ServerSocket.java:490)"[ACTIVE] ExecuteThread: 18 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dba9a3000 nid=0x7d03 in Object.wait() [0x0000000142d1b000] at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:38) java.lang.Thread.State: WAITING (on object monitor) at weblogic.server.channels.DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:535) at java.lang.Object.wait(Native Method) at weblogic.server.channels.DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:417) at java.lang.Object.wait(Object.java:503) at weblogic.server.channels.DynamicListenThread.run(DynamicListenThread.java:173) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.lang.Thread.run(Thread.java:722) - locked <0x000000011ab18448> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "DynamicListenThread[Default[2]]" daemon prio=5 tid=0x00007f8dbbd06000 nid=0x6703 runnable [0x000000014203e000] java.lang.Thread.State: RUNNABLE"[STANDBY] ExecuteThread: 17 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbae29800 nid=0x7c03 in Object.wait() [0x0000000142c18000] at java.net.PlainSocketImpl.socketAccept(Native Method) java.lang.Thread.State: WAITING (on object monitor) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at java.lang.Object.wait(Native Method) at java.net.ServerSocket.implAccept(ServerSocket.java:522) at java.lang.Object.wait(Object.java:503) at java.net.ServerSocket.accept(ServerSocket.java:490) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:38) - locked <0x000000011ab15510> (a weblogic.work.ExecuteThread) at weblogic.server.channels.DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:535) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at weblogic.server.channels.DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:417) at weblogic.server.channels.DynamicListenThread.run(DynamicListenThread.java:173)"[ACTIVE] ExecuteThread: 16 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbd0b7000 nid=0x7b03 in Object.wait() [0x0000000142b15000] at java.lang.Thread.run(Thread.java:722) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) "weblogic.GCMonitor" daemon prio=5 tid=0x00007f8dbe0c0800 nid=0x4507 in Object.wait() [0x0000000140eed000] at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: TIMED_WAITING (on object monitor) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.lang.Object.wait(Native Method) - locked <0x000000011ab17fc8> (a weblogic.work.ExecuteThread) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) - locked <0x000000011a6d83f0> (a java.lang.ref.ReferenceQueue$Lock) at weblogic.platform.GCMonitorThread.waitForNotification(GCMonitorThread.java:88)"[ACTIVE] ExecuteThread: 15 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbcaa6800 nid=0x7a03 in Object.wait() [0x0000000142a12000] at weblogic.platform.GCMonitorThread.run(GCMonitorThread.java:64) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) "Thread-11" prio=5 tid=0x00007f8dbc884000 nid=0x4007 waiting on condition [0x0000000140dea000] at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: WAITING (parking) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at sun.misc.Unsafe.park(Native Method) - locked <0x000000011ab1aeb8> (a weblogic.work.ExecuteThread) - parking to wait for <0x000000011a50cfa8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)"[ACTIVE] ExecuteThread: 14 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbbc5c800 nid=0x7903 in Object.wait() [0x0000000142856000] at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.lang.Thread.State: WAITING (on object monitor) at weblogic.utils.concurrent.JDK15ConcurrentBlockingQueue.take(JDK15ConcurrentBlockingQueue.java:89) at java.lang.Object.wait(Native Method) at weblogic.store.internal.PersistentStoreImpl.getOutstandingWork(PersistentStoreImpl.java:676) at java.lang.Object.wait(Object.java:503) at weblogic.store.internal.PersistentStoreImpl.run(PersistentStoreImpl.java:751) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.lang.Thread.run(Thread.java:722) - locked <0x000000011ab1bed8> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: 3 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dba33e800 nid=0x430f in Object.wait() [0x000000013f4ca000] java.lang.Thread.State: WAITING (on object monitor)"[ACTIVE] ExecuteThread: 13 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbb83c000 nid=0x7803 in Object.wait() [0x0000000142650000] at java.lang.Object.wait(Native Method) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Object.java:503) at java.lang.Object.wait(Native Method) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.lang.Object.wait(Object.java:503) - locked <0x000000011a4b00e8> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) - locked <0x000000011ab144c8> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: 2 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbad9f800 nid=0x4207 in Object.wait() [0x00000001404e7000] java.lang.Thread.State: WAITING (on object monitor)"[ACTIVE] ExecuteThread: 12 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbd9c0000 nid=0x7703 in Object.wait() [0x0000000140a29000] at java.lang.Object.wait(Native Method) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Object.java:503) at java.lang.Object.wait(Native Method) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.lang.Object.wait(Object.java:503) - locked <0x000000011a4b0d30> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) - locked <0x000000011ab19548> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "Thread-10" prio=5 tid=0x00007f8dbad89000 nid=0x6603 waiting on condition [0x00000001402db000] java.lang.Thread.State: WAITING (parking)"[ACTIVE] ExecuteThread: 11 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dba94d800 nid=0x7603 in Object.wait() [0x0000000142347000] at sun.misc.Unsafe.park(Native Method) java.lang.Thread.State: WAITING (on object monitor) - parking to wait for <0x000000011a208ea0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.lang.Object.wait(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.lang.Object.wait(Object.java:503) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) - locked <0x000000011ab0d3f8> (a weblogic.work.ExecuteThread) at weblogic.utils.concurrent.JDK15ConcurrentBlockingQueue.take(JDK15ConcurrentBlockingQueue.java:89) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at weblogic.store.internal.PersistentStoreImpl.getOutstandingWork(PersistentStoreImpl.java:676) at weblogic.store.internal.PersistentStoreImpl.run(PersistentStoreImpl.java:751)"[STANDBY] ExecuteThread: 10 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbe196000 nid=0x7503 in Object.wait() [0x0000000140c23000] at java.lang.Thread.run(Thread.java:722) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) "DoSManager" daemon prio=5 tid=0x00007f8dbd84c800 nid=0x6503 waiting on condition [0x00000001400ce000] at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: TIMED_WAITING (sleeping) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.lang.Thread.sleep(Native Method) - locked <0x000000011ab0e418> (a weblogic.work.ExecuteThread) at com.octetstring.vde.DoSManager.run(DoSManager.java:433) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "VDE Transaction Processor Thread" daemon prio=5 tid=0x00007f8dbc82b000 nid=0x6303 in Object.wait() [0x000000013ffcb000]"[ACTIVE] ExecuteThread: 9 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbe195800 nid=0x7403 in Object.wait() [0x00000001408be000] java.lang.Thread.State: WAITING (on object monitor) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at java.lang.Object.wait(Object.java:503) at com.octetstring.vde.backend.standard.TransactionProcessor.waitChange(TransactionProcessor.java:367) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x0000000119edae88> (a com.octetstring.vde.backend.standard.TransactionProcessor) - locked <0x000000011ab10b38> (a weblogic.work.ExecuteThread) at com.octetstring.vde.backend.standard.TransactionProcessor.run(TransactionProcessor.java:212) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "ExecuteThread: 3 for queue: weblogic.socket.Muxer" daemon prio=5 tid=0x00007f8db9712800 nid=0x6203 waiting for monitor entry [0x000000013fec8000]"[ACTIVE] ExecuteThread: 8 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbba6d800 nid=0x7303 in Object.wait() [0x000000013f0a9000] java.lang.Thread.State: BLOCKED (on object monitor) java.lang.Thread.State: WAITING (on object monitor) at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:93) at java.lang.Object.wait(Native Method) - waiting to lock <0x0000000119f22540> (a weblogic.socket.PosixSocketMuxer$1) at java.lang.Object.wait(Object.java:503) at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42) - locked <0x000000011ab0c2d0> (a weblogic.work.ExecuteThread) at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:147) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:119)"[ACTIVE] ExecuteThread: 7 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbc812800 nid=0x7203 in Object.wait() [0x00000001407bb000] "ExecuteThread: 2 for queue: weblogic.socket.Muxer" daemon prio=5 tid=0x00007f8dbac3a000 nid=0x6103 runnable [0x000000013fdc5000] java.lang.Thread.State: WAITING (on object monitor) java.lang.Thread.State: RUNNABLE at java.lang.Object.wait(Native Method) at weblogic.socket.PosixSocketMuxer.poll(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:102) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x0000000119f22540> (a weblogic.socket.PosixSocketMuxer$1) - locked <0x000000011ab01498> (a weblogic.work.ExecuteThread) at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42) at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:147)"[ACTIVE] ExecuteThread: 6 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8db96b5800 nid=0x7103 in Object.wait() [0x00000001406b8000] at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:119) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) "ExecuteThread: 1 for queue: weblogic.socket.Muxer" daemon prio=5 tid=0x00007f8db973b000 nid=0x6003 waiting for monitor entry [0x000000013fcc2000] at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: BLOCKED (on object monitor) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:93) - locked <0x000000011ab024b8> (a weblogic.work.ExecuteThread) - waiting to lock <0x0000000119f22540> (a weblogic.socket.PosixSocketMuxer$1) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29) at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)"[STANDBY] ExecuteThread: 5 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8db971b000 nid=0x7003 in Object.wait() [0x00000001403de000] at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:147) java.lang.Thread.State: WAITING (on object monitor) at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:119) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) "ExecuteThread: 0 for queue: weblogic.socket.Muxer" daemon prio=5 tid=0x00007f8dbaa1b800 nid=0x5f03 waiting for monitor entry [0x000000013fbbf000] at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) java.lang.Thread.State: BLOCKED (on object monitor) - locked <0x000000011ab06208> (a weblogic.work.ExecuteThread) at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:93) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) - waiting to lock <0x0000000119f22540> (a weblogic.socket.PosixSocketMuxer$1) at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)"[ACTIVE] ExecuteThread: 4 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbbd5d000 nid=0x6f03 in Object.wait() [0x00000001401d1000] at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42) java.lang.Thread.State: WAITING (on object monitor) at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:147) at java.lang.Object.wait(Native Method) at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:119) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) "[ACTIVE] ExecuteThread: 1 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dba4cc800 nid=0x5e03 in Object.wait() [0x000000013fabc000] - locked <0x000000011ab07228> (a weblogic.work.ExecuteThread) java.lang.Thread.State: WAITING (on object monitor) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503)"DynamicListenThread[Default]" daemon prio=5 tid=0x00007f8dbd82b000 nid=0x6e03 runnable [0x0000000142753000] at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) java.lang.Thread.State: RUNNABLE - locked <0x0000000119fa2d20> (a weblogic.work.ExecuteThread) at java.net.PlainSocketImpl.socketAccept(Native Method) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at java.net.ServerSocket.implAccept(ServerSocket.java:522) "weblogic.timers.TimerThread" daemon prio=5 tid=0x00007f8db9738000 nid=0x5d03 in Object.wait() [0x000000013f9b9000] at java.net.ServerSocket.accept(ServerSocket.java:490) java.lang.Thread.State: TIMED_WAITING (on object monitor) at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:38) at java.lang.Object.wait(Native Method) at weblogic.server.channels.DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:535) at weblogic.timers.internal.TimerThread$Thread.run(TimerThread.java:262) at weblogic.server.channels.DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:417) - locked <0x0000000119e7f158> (a weblogic.timers.internal.TimerThread) "weblogic.time.TimeEventGenerator" daemon prio=5 tid=0x00007f8db9724000 nid=0x5c03 in Object.wait() [0x000000013f8b6000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at weblogic.time.common.internal.TimeTable.snooze(TimeTable.java:286) - locked <0x0000000119dfc2e0> (a weblogic.time.common.internal.TimeTable) at weblogic.time.common.internal.TimeEventGenerator.run(TimeEventGenerator.java:117) at java.lang.Thread.run(Thread.java:722) "[ACTIVE] ExecuteThread: 0 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbb232000 nid=0x5b03 in Object.wait() [0x000000013f7b3000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x0000000119fa37e0> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) Yusuke Yamamoto @yusuke #wlstudy
  • 14. 2013-01-23 16:32:13Full thread dump Java HotSpot(TM) 64-Bit Server VM (23.6-b04 mixed mode): at java.lang.Object.wait(Native Method)"Attach Listener" daemon prio=5 tid=0x00007f8dbd9dd000 nid=0x9407 waiting on condition [0x0000000000000000] at java.lang.Object.wait(Object.java:503) - locked <0x000000011ab16fa8> (a weblogic.work.ExecuteThread) java.lang.Thread.State: RUNNABLE at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) - locked <0x000000011ab16fa8> (a weblogic.work.ExecuteThread)"[STANDBY] ExecuteThread: 40 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbcd3c000 nid=0x9303 in Object.wait() [0x0000000144526000] at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: 18 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dba9a3000 nid=0x7d03 in Object.wait() [0x0000000142d1b000] java.lang.Thread.State: WAITING (on object monitor) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) "[ACTIVE] ExecuteThread: 18 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dba9a3000 nid=0x7d03 in Object.wait() [0x0000000142d1b000] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.lang.Object.wait(Native Method) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011c40fc88> (a weblogic.work.ExecuteThread) at java.lang.Object.wait(Object.java:503) - locked <0x000000011ab18448> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) - locked <0x000000011ab18448> (a weblogic.work.ExecuteThread)"[STANDBY] ExecuteThread: 39 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbdab9800 nid=0x9203 in Object.wait() [0x000000014437b000] at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[STANDBY] ExecuteThread: 17 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbae29800 nid=0x7c03 in Object.wait() [0x0000000142c18000] java.lang.Thread.State: WAITING (on object monitor) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) "[STANDBY] ExecuteThread: 17 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbae29800 nid=0x7c03 in Object.wait() [0x0000000142c18000] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.lang.Object.wait(Native Method) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab2f568> (a weblogic.work.ExecuteThread) at java.lang.Object.wait(Object.java:503) - locked <0x000000011ab15510> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) - locked <0x000000011ab15510> (a weblogic.work.ExecuteThread)"[ACTIVE] ExecuteThread: 38 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbd1bf000 nid=0x9103 in Object.wait() [0x0000000144278000] at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: 16 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbd0b7000 nid=0x7b03 in Object.wait() [0x0000000142b15000] java.lang.Thread.State: WAITING (on object monitor) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) "[ACTIVE] ExecuteThread: 16 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbd0b7000 nid=0x7b03 in Object.wait() [0x0000000142b15000] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.lang.Object.wait(Native Method) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab2e4a0> (a weblogic.work.ExecuteThread) at java.lang.Object.wait(Object.java:503) - locked <0x000000011ab17fc8> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) - locked <0x000000011ab17fc8> (a weblogic.work.ExecuteThread)"[ACTIVE] ExecuteThread: 37 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dba80b000 nid=0x9003 in Object.wait() [0x0000000144175000] at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: 15 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbcaa6800 nid=0x7a03 in Object.wait() [0x0000000142a12000] java.lang.Thread.State: WAITING (on object monitor) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) "[ACTIVE] ExecuteThread: 15 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbcaa6800 nid=0x7a03 in Object.wait() [0x0000000142a12000] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.lang.Object.wait(Native Method) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab2d1b0> (a weblogic.work.ExecuteThread) at java.lang.Object.wait(Object.java:503) - locked <0x000000011ab1aeb8> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) - locked <0x000000011ab1aeb8> (a weblogic.work.ExecuteThread)"[ACTIVE] ExecuteThread: 36 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dba479800 nid=0x8f03 in Object.wait() [0x0000000144072000] at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: 14 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbbc5c800 nid=0x7903 in Object.wait() [0x0000000142856000] java.lang.Thread.State: WAITING (on object monitor) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) "[ACTIVE] ExecuteThread: 14 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbbc5c800 nid=0x7903 in Object.wait() [0x0000000142856000] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.lang.Object.wait(Native Method) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab27da0> (a weblogic.work.ExecuteThread) at java.lang.Object.wait(Object.java:503) - locked <0x000000011ab1bed8> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) - locked <0x000000011ab1bed8> (a weblogic.work.ExecuteThread)"[STANDBY] ExecuteThread: 35 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8db959f000 nid=0x8e03 in Object.wait() [0x0000000143f6f000] at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: 13 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbb83c000 nid=0x7803 in Object.wait() [0x0000000142650000] java.lang.Thread.State: WAITING (on object monitor) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) "[ACTIVE] ExecuteThread: 13 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbb83c000 nid=0x7803 in Object.wait() [0x0000000142650000] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.lang.Object.wait(Native Method) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab26c90> (a weblogic.work.ExecuteThread) at java.lang.Object.wait(Object.java:503) - locked <0x000000011ab144c8> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) - locked <0x000000011ab144c8> (a weblogic.work.ExecuteThread)"[ACTIVE] ExecuteThread: 34 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dba9a4000 nid=0x8d03 in Object.wait() [0x0000000143e6c000] at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: 12 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbd9c0000 nid=0x7703 in Object.wait() [0x0000000140a29000] java.lang.Thread.State: WAITING (on object monitor) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) "[ACTIVE] ExecuteThread: 12 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbd9c0000 nid=0x7703 in Object.wait() [0x0000000140a29000] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.lang.Object.wait(Native Method) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab29978> (a weblogic.work.ExecuteThread) at java.lang.Object.wait(Object.java:503) - locked <0x000000011ab19548> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) - locked <0x000000011ab19548> (a weblogic.work.ExecuteThread)"[STANDBY] ExecuteThread: 33 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbbc5b800 nid=0x8c03 in Object.wait() [0x0000000143d69000] at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: 11 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dba94d800 nid=0x7603 in Object.wait() [0x0000000142347000] java.lang.Thread.State: WAITING (on object monitor) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) "[ACTIVE] ExecuteThread: 11 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dba94d800 nid=0x7603 in Object.wait() [0x0000000142347000] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.lang.Object.wait(Native Method) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab29a28> (a weblogic.work.ExecuteThread) at java.lang.Object.wait(Object.java:503) - locked <0x000000011ab0d3f8> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) - locked <0x000000011ab0d3f8> (a weblogic.work.ExecuteThread)"[ACTIVE] ExecuteThread: 32 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbbc5b000 nid=0x8b03 in Object.wait() [0x0000000143c66000] at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[STANDBY] ExecuteThread: 10 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbe196000 nid=0x7503 in Object.wait() [0x0000000140c23000] java.lang.Thread.State: WAITING (on object monitor) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) "[STANDBY] ExecuteThread: 10 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbe196000 nid=0x7503 in Object.wait() [0x0000000140c23000] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.lang.Object.wait(Native Method) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab25c70> (a weblogic.work.ExecuteThread) at java.lang.Object.wait(Object.java:503) - locked <0x000000011ab0e418> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) - locked <0x000000011ab0e418> (a weblogic.work.ExecuteThread)"[ACTIVE] ExecuteThread: 31 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbbd54800 nid=0x8a03 in Object.wait() [0x00000001435d3000] at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: 9 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbe195800 nid=0x7403 in Object.wait() [0x00000001408be000] java.lang.Thread.State: WAITING (on object monitor) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) "[ACTIVE] ExecuteThread: 9 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbe195800 nid=0x7403 in Object.wait() [0x00000001408be000] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.lang.Object.wait(Native Method) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab28da8> (a weblogic.work.ExecuteThread) at java.lang.Object.wait(Object.java:503) - locked <0x000000011ab10b38> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) - locked <0x000000011ab10b38> (a weblogic.work.ExecuteThread)"[ACTIVE] ExecuteThread: 30 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbbd8f800 nid=0x8903 in Object.wait() [0x00000001434d0000] at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: 8 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbba6d800 nid=0x7303 in Object.wait() [0x000000013f0a9000] java.lang.Thread.State: WAITING (on object monitor) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) "[ACTIVE] ExecuteThread: 8 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbba6d800 nid=0x7303 in Object.wait() [0x000000013f0a9000] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.lang.Object.wait(Native Method) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab2a9a0> (a weblogic.work.ExecuteThread) at java.lang.Object.wait(Object.java:503) - locked <0x000000011ab0c2d0> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) - locked <0x000000011ab0c2d0> (a weblogic.work.ExecuteThread)"[STANDBY] ExecuteThread: 29 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbbd8f000 nid=0x8803 in Object.wait() [0x0000000143b63000] at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: 7 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbc812800 nid=0x7203 in Object.wait() [0x00000001407bb000] java.lang.Thread.State: WAITING (on object monitor) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) "[ACTIVE] ExecuteThread: 7 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbc812800 nid=0x7203 in Object.wait() [0x00000001407bb000] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.lang.Object.wait(Native Method) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab229f0> (a weblogic.work.ExecuteThread) at java.lang.Object.wait(Object.java:503) - locked <0x000000011ab01498> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) - locked <0x000000011ab01498> (a weblogic.work.ExecuteThread)"[ACTIVE] ExecuteThread: 28 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbcb4e000 nid=0x8703 in Object.wait() [0x0000000143a60000] at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: 6 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8db96b5800 nid=0x7103 in Object.wait() [0x00000001406b8000] java.lang.Thread.State: WAITING (on object monitor) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) "[ACTIVE] ExecuteThread: 6 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8db96b5800 nid=0x7103 in Object.wait() [0x00000001406b8000] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.lang.Object.wait(Native Method) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab23a10> (a weblogic.work.ExecuteThread) at java.lang.Object.wait(Object.java:503) - locked <0x000000011ab024b8> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) - locked <0x000000011ab024b8> (a weblogic.work.ExecuteThread)"[ACTIVE] ExecuteThread: 27 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbbc91800 nid=0x8603 in Object.wait() [0x000000014395d000] at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[STANDBY] ExecuteThread: 5 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8db971b000 nid=0x7003 in Object.wait() [0x00000001403de000] java.lang.Thread.State: WAITING (on object monitor) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) "[STANDBY] ExecuteThread: 5 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8db971b000 nid=0x7003 in Object.wait() [0x00000001403de000] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.lang.Object.wait(Native Method) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab24a30> (a weblogic.work.ExecuteThread) at java.lang.Object.wait(Object.java:503) - locked <0x000000011ab06208> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) - locked <0x000000011ab06208> (a weblogic.work.ExecuteThread)"[ACTIVE] ExecuteThread: 26 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbb015800 nid=0x8503 in Object.wait() [0x000000014385a000] at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: 4 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbbd5d000 nid=0x6f03 in Object.wait() [0x00000001401d1000] java.lang.Thread.State: WAITING (on object monitor) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) "[ACTIVE] ExecuteThread: 4 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbbd5d000 nid=0x6f03 in Object.wait() [0x00000001401d1000] at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.lang.Object.wait(Native Method) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011ab1f7b0> (a weblogic.work.ExecuteThread) at java.lang.Object.wait(Object.java:503) - locked <0x000000011ab07228> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) - locked <0x000000011ab07228> (a weblogic.work.ExecuteThread)"[STANDBY] ExecuteThread: 25 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dba709800 nid=0x8403 in Object.wait() [0x0000000143757000] at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "DynamicListenThread[Default]" daemon prio=5 tid=0x00007f8dbd82b000 nid=0x6e03 runnable [0x0000000142753000] java.lang.Thread.State: WAITING (on object monitor) java.lang.Thread.State: RUNNABLE at java.lang.Object.wait(Native Method) "DynamicListenThread[Default]" daemon prio=5 tid=0x00007f8dbd82b000 nid=0x6e03 runnable [0x0000000142753000] at java.net.PlainSocketImpl.socketAccept(Native Method) at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: RUNNABLE at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.ServerSocket.implAccept(ServerSocket.java:522) - locked <0x000000011ab207d0> (a weblogic.work.ExecuteThread) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at java.net.ServerSocket.accept(ServerSocket.java:490) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at java.net.ServerSocket.implAccept(ServerSocket.java:522) at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:38) at java.net.ServerSocket.accept(ServerSocket.java:490) at weblogic.server.channels.DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:535)"[ACTIVE] ExecuteThread: 24 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbcb54800 nid=0x8303 in Object.wait() [0x000000013f1cd000] at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:38) at weblogic.server.channels.DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:417) java.lang.Thread.State: WAITING (on object monitor) at weblogic.server.channels.DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:535) at weblogic.server.channels.DynamicListenThread.run(DynamicListenThread.java:173) at java.lang.Object.wait(Native Method) at weblogic.server.channels.DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:417) at java.lang.Thread.run(Thread.java:722) at java.lang.Object.wait(Object.java:503) at weblogic.server.channels.DynamicListenThread.run(DynamicListenThread.java:173) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.lang.Thread.run(Thread.java:722) "DynamicListenThread[Default[1]]" daemon prio=5 tid=0x00007f8db97d7800 nid=0x6c03 runnable [0x000000014254d000] - locked <0x000000011ab21860> (a weblogic.work.ExecuteThread) java.lang.Thread.State: RUNNABLE at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "DynamicListenThread[Default[1]]" daemon prio=5 tid=0x00007f8db97d7800 nid=0x6c03 runnable [0x000000014254d000] at java.net.PlainSocketImpl.socketAccept(Native Method) java.lang.Thread.State: RUNNABLE at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)"[ACTIVE] ExecuteThread: 23 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbc945800 nid=0x8203 in Object.wait() [0x000000014322a000] at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.ServerSocket.implAccept(ServerSocket.java:522) java.lang.Thread.State: WAITING (on object monitor) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at java.net.ServerSocket.accept(ServerSocket.java:490) at java.lang.Object.wait(Native Method) at java.net.ServerSocket.implAccept(ServerSocket.java:522) at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:38) at java.lang.Object.wait(Object.java:503) at java.net.ServerSocket.accept(ServerSocket.java:490) at weblogic.server.channels.DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:535) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:38) at weblogic.server.channels.DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:417) 2013-01-23 16:32:13 - locked <0x000000011ab1d8c8> (a weblogic.work.ExecuteThread) at weblogic.server.channels.DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:535) at weblogic.server.channels.DynamicListenThread.run(DynamicListenThread.java:173) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at weblogic.server.channels.DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:417) at java.lang.Thread.run(Thread.java:722) at weblogic.server.channels.DynamicListenThread.run(DynamicListenThread.java:173)"[STANDBY] ExecuteThread: 22 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8db959d800 nid=0x8103 in Object.wait() [0x0000000143127000] at java.lang.Thread.run(Thread.java:722) "DynamicListenThread[Default[7]]" daemon prio=5 tid=0x00007f8dbe02c800 nid=0x6b03 runnable [0x000000014244a000] java.lang.Thread.State: WAITING (on object monitor) java.lang.Thread.State: RUNNABLE at java.lang.Object.wait(Native Method) "DynamicListenThread[Default[7]]" daemon prio=5 tid=0x00007f8dbe02c800 nid=0x6b03 runnable [0x000000014244a000] at java.net.PlainSocketImpl.socketAccept(Native Method) at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: RUNNABLE at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.net.PlainSocketImpl.socketAccept(Native Method) Full thread dump Java HotSpot(TM) 64-Bit Server VM (23.6-b04 mixed mode): at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at java.net.ServerSocket.implAccept(ServerSocket.java:522) - locked <0x000000011ab12600> (a weblogic.work.ExecuteThread) at java.net.ServerSocket.accept(ServerSocket.java:490) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at java.net.ServerSocket.implAccept(ServerSocket.java:522) at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:38) at java.net.ServerSocket.accept(ServerSocket.java:490) at weblogic.server.channels.DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:535)"[ACTIVE] ExecuteThread: 21 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbcaa7000 nid=0x8003 in Object.wait() [0x0000000143024000] at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:38) at weblogic.server.channels.DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:417) java.lang.Thread.State: WAITING (on object monitor) at weblogic.server.channels.DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:535) at weblogic.server.channels.DynamicListenThread.run(DynamicListenThread.java:173) at java.lang.Object.wait(Native Method) at weblogic.server.channels.DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:417) at java.lang.Thread.run(Thread.java:722) at java.lang.Object.wait(Object.java:503) at weblogic.server.channels.DynamicListenThread.run(DynamicListenThread.java:173) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.lang.Thread.run(Thread.java:722) "DynamicListenThread[Default[ - locked <0x000000011ab13620> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "DynamicListenThread[Default[4]]" daemon prio=5 tid=0x00007f8dbc8c7800 nid=0x6903 runnable [0x0000000142244000] java.lang.Thread.State: RUNNABLE"[STANDBY] ExecuteThread: 20 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbb832800 nid=0x7f03 in Object.wait() [0x0000000142f21000] at java.net.PlainSocketImpl.socketAccept(Native Method) java.lang.Thread.State: WAITING (on object monitor) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at java.lang.Object.wait(Native Method) at java.net.ServerSocket.implAccept(ServerSocket.java:522) at java.lang.Object.wait(Object.java:503) at java.net.ServerSocket.accept(ServerSocket.java:490) "Attach Listener" daemon prio=5 tid=0x00007f8dbd9dd000 nid=0x9407 waiting on condition at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:38) - locked <0x000000011ab164e0> (a weblogic.work.ExecuteThread) at weblogic.server.channels.DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:535) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at weblogic.server.channels.DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:417) at weblogic.server.channels.DynamicListenThread.run(DynamicListenThread.java:173)"[ACTIVE] ExecuteThread: 19 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbbc5d800 nid=0x7e03 in Object.wait() [0x0000000142e1e000] at java.lang.Thread.run(Thread.java:722) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) "DynamicListenThread[Default[8]]" daemon prio=5 tid=0x00007f8dba13f800 nid=0x6803 runnable [0x0000000142141000] [0x0000000000000000] at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: RUNNABLE at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.net.PlainSocketImpl.socketAccept(Native Method) - locked <0x000000011ab16fa8> (a weblogic.work.ExecuteThread) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at java.net.ServerSocket.implAccept(ServerSocket.java:522) at java.net.ServerSocket.accept(ServerSocket.java:490)"[ACTIVE] ExecuteThread: 18 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dba9a3000 nid=0x7d03 in Object.wait() [0x0000000142d1b000] at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:38) java.lang.Thread.State: WAITING (on object monitor) at weblogic.server.channels.DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:535) at java.lang.Object.wait(Native Method) at weblogic.server.channels.DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:417) java.lang.Thread.State: RUNNABLE at java.lang.Object.wait(Object.java:503) at weblogic.server.channels.DynamicListenThread.run(DynamicListenThread.java:173) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.lang.Thread.run(Thread.java:722) - locked <0x000000011ab18448> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "DynamicListenThread[Default[2]]" daemon prio=5 tid=0x00007f8dbbd06000 nid=0x6703 runnable [0x000000014203e000] java.lang.Thread.State: RUNNABLE"[STANDBY] ExecuteThread: 17 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbae29800 nid=0x7c03 in Object.wait() [0x0000000142c18000] at java.net.PlainSocketImpl.socketAccept(Native Method) java.lang.Thread.State: WAITING (on object monitor) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at java.lang.Object.wait(Native Method) at java.net.ServerSocket.implAccept(ServerSocket.java:522) at java.lang.Object.wait(Object.java:503) at java.net.ServerSocket.accept(ServerSocket.java:490) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:38) - locked <0x000000011ab15510> (a weblogic.work.ExecuteThread) at weblogic.server.channels.DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:535) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at weblogic.server.channels.DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:417) at weblogic.server.channels.DynamicListenThread.run(DynamicListenThread.java:173)"[ACTIVE] ExecuteThread: 16 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbd0b7000 nid=0x7b03 in Object.wait() [0x0000000142b15000] at java.lang.Thread.run(Thread.java:722) java.lang.Thread.State: WAITING (on object monitor) "[STANDBY] ExecuteThread: 40 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 at java.lang.Object.wait(Native Method) "weblogic.GCMonitor" daemon prio=5 tid=0x00007f8dbe0c0800 nid=0x4507 in Object.wait() [0x0000000140eed000] at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: TIMED_WAITING (on object monitor) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.lang.Object.wait(Native Method) - locked <0x000000011ab17fc8> (a weblogic.work.ExecuteThread) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) - locked <0x000000011a6d83f0> (a java.lang.ref.ReferenceQueue$Lock) at weblogic.platform.GCMonitorThread.waitForNotification(GCMonitorThread.java:88)"[ACTIVE] ExecuteThread: 15 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbcaa6800 nid=0x7a03 in Object.wait() [0x0000000142a12000] at weblogic.platform.GCMonitorThread.run(GCMonitorThread.java:64) tid=0x00007f8dbcd3c000 nid=0x9303 in Object.wait() [0x0000000144526000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) "Thread-11" prio=5 tid=0x00007f8dbc884000 nid=0x4007 waiting on condition [0x0000000140dea000] at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: WAITING (parking) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at sun.misc.Unsafe.park(Native Method) - locked <0x000000011ab1aeb8> (a weblogic.work.ExecuteThread) - parking to wait for <0x000000011a50cfa8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)"[ACTIVE] ExecuteThread: 14 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbbc5c800 nid=0x7903 in Object.wait() [0x0000000142856000] at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.lang.Thread.State: WAITING (on object monitor) java.lang.Thread.State: WAITING (on object monitor) at weblogic.utils.concurrent.JDK15ConcurrentBlockingQueue.take(JDK15ConcurrentBlockingQueue.java:89) at java.lang.Object.wait(Native Method) at weblogic.store.internal.PersistentStoreImpl.getOutstandingWork(PersistentStoreImpl.java:676) at java.lang.Object.wait(Object.java:503) at weblogic.store.internal.PersistentStoreImpl.run(PersistentStoreImpl.java:751) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.lang.Thread.run(Thread.java:722) - locked <0x000000011ab1bed8> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: 3 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dba33e800 nid=0x430f in Object.wait() [0x000000013f4ca000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method)"[ACTIVE] ExecuteThread: 13 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbb83c000 nid=0x7803 in Object.wait() [0x0000000142650000] at java.lang.Object.wait(Native Method) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Object.java:503) at java.lang.Object.wait(Native Method) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.lang.Object.wait(Object.java:503) - locked <0x000000011a4b00e8> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) - locked <0x000000011ab144c8> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: 2 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbad9f800 nid=0x4207 in Object.wait() [0x00000001404e7000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Object.java:503)"[ACTIVE] ExecuteThread: 12 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbd9c0000 nid=0x7703 in Object.wait() [0x0000000140a29000] at java.lang.Object.wait(Native Method) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Object.java:503) at java.lang.Object.wait(Native Method) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.lang.Object.wait(Object.java:503) - locked <0x000000011a4b0d30> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) - locked <0x000000011ab19548> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "Thread-10" prio=5 tid=0x00007f8dbad89000 nid=0x6603 waiting on condition [0x00000001402db000] at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) java.lang.Thread.State: WAITING (parking)"[ACTIVE] ExecuteThread: 11 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dba94d800 nid=0x7603 in Object.wait() [0x0000000142347000] at sun.misc.Unsafe.park(Native Method) java.lang.Thread.State: WAITING (on object monitor) - parking to wait for <0x000000011a208ea0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.lang.Object.wait(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186) at java.lang.Object.wait(Object.java:503) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) - locked <0x000000011ab0d3f8> (a weblogic.work.ExecuteThread) at weblogic.utils.concurrent.JDK15ConcurrentBlockingQueue.take(JDK15ConcurrentBlockingQueue.java:89) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at weblogic.store.internal.PersistentStoreImpl.getOutstandingWork(PersistentStoreImpl.java:676) - locked <0x000000011c40fc88> (a weblogic.work.ExecuteThread) at weblogic.store.internal.PersistentStoreImpl.run(PersistentStoreImpl.java:751)"[STANDBY] ExecuteThread: 10 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbe196000 nid=0x7503 in Object.wait() [0x0000000140c23000] at java.lang.Thread.run(Thread.java:722) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) "DoSManager" daemon prio=5 tid=0x00007f8dbd84c800 nid=0x6503 waiting on condition [0x00000001400ce000] at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: TIMED_WAITING (sleeping) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.lang.Thread.sleep(Native Method) - locked <0x000000011ab0e418> (a weblogic.work.ExecuteThread) at com.octetstring.vde.DoSManager.run(DoSManager.java:433) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "VDE Transaction Processor Thread" daemon prio=5 tid=0x00007f8dbc82b000 nid=0x6303 in Object.wait() [0x000000013ffcb000]"[ACTIVE] ExecuteThread: 9 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbe195800 nid=0x7403 in Object.wait() [0x00000001408be000] java.lang.Thread.State: WAITING (on object monitor) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at java.lang.Object.wait(Object.java:503) at com.octetstring.vde.backend.standard.TransactionProcessor.waitChange(TransactionProcessor.java:367) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x0000000119edae88> (a com.octetstring.vde.backend.standard.TransactionProcessor) - locked <0x000000011ab10b38> (a weblogic.work.ExecuteThread) at com.octetstring.vde.backend.standard.TransactionProcessor.run(TransactionProcessor.java:212) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "ExecuteThread: 3 for queue: weblogic.socket.Muxer" daemon prio=5 tid=0x00007f8db9712800 nid=0x6203 waiting for monitor entry [0x000000013fec8000]"[ACTIVE] ExecuteThread: 8 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbba6d800 nid=0x7303 in Object.wait() [0x000000013f0a9000] java.lang.Thread.State: BLOCKED (on object monitor) java.lang.Thread.State: WAITING (on object monitor) at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:93) at java.lang.Object.wait(Native Method) - waiting to lock <0x0000000119f22540> (a weblogic.socket.PosixSocketMuxer$1) at java.lang.Object.wait(Object.java:503) at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42) "[STANDBY] ExecuteThread: 39 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 - locked <0x000000011ab0c2d0> (a weblogic.work.ExecuteThread) at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:147) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:119)"[ACTIVE] ExecuteThread: 7 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbc812800 nid=0x7203 in Object.wait() [0x00000001407bb000] "ExecuteThread: 2 for queue: weblogic.socket.Muxer" daemon prio=5 tid=0x00007f8dbac3a000 nid=0x6103 runnable [0x000000013fdc5000] java.lang.Thread.State: WAITING (on object monitor) java.lang.Thread.State: RUNNABLE at java.lang.Object.wait(Native Method) at weblogic.socket.PosixSocketMuxer.poll(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:102) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x0000000119f22540> (a weblogic.socket.PosixSocketMuxer$1) tid=0x00007f8dbdab9800 nid=0x9203 in Object.wait() [0x000000014437b000] - locked <0x000000011ab01498> (a weblogic.work.ExecuteThread) at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42) at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:147)"[ACTIVE] ExecuteThread: 6 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8db96b5800 nid=0x7103 in Object.wait() [0x00000001406b8000] at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:119) java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) "ExecuteThread: 1 for queue: weblogic.socket.Muxer" daemon prio=5 tid=0x00007f8db973b000 nid=0x6003 waiting for monitor entry [0x000000013fcc2000] at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: BLOCKED (on object monitor) java.lang.Thread.State: WAITING (on object monitor) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:93) - locked <0x000000011ab024b8> (a weblogic.work.ExecuteThread) - waiting to lock <0x0000000119f22540> (a weblogic.socket.PosixSocketMuxer$1) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29) at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)"[STANDBY] ExecuteThread: 5 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8db971b000 nid=0x7003 in Object.wait() [0x00000001403de000] at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:147) java.lang.Thread.State: WAITING (on object monitor) at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:119) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) "ExecuteThread: 0 for queue: weblogic.socket.Muxer" daemon prio=5 tid=0x00007f8dbaa1b800 nid=0x5f03 waiting for monitor entry [0x000000013fbbf000] at java.lang.Object.wait(Native Method) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) java.lang.Thread.State: BLOCKED (on object monitor) - locked <0x000000011ab06208> (a weblogic.work.ExecuteThread) at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:93) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) - waiting to lock <0x0000000119f22540> (a weblogic.socket.PosixSocketMuxer$1) at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)"[ACTIVE] ExecuteThread: 4 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbbd5d000 nid=0x6f03 in Object.wait() [0x00000001401d1000] at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42) java.lang.Thread.State: WAITING (on object monitor) at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:147) at java.lang.Object.wait(Native Method) at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:119) at java.lang.Object.wait(Object.java:503) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) "[ACTIVE] ExecuteThread: 1 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dba4cc800 nid=0x5e03 in Object.wait() [0x000000013fabc000] - locked <0x000000011ab07228> (a weblogic.work.ExecuteThread) java.lang.Thread.State: WAITING (on object monitor) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503)"DynamicListenThread[Default]" daemon prio=5 tid=0x00007f8dbd82b000 nid=0x6e03 runnable [0x0000000142753000] at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) java.lang.Thread.State: RUNNABLE - locked <0x0000000119fa2d20> (a weblogic.work.ExecuteThread) at java.net.PlainSocketImpl.socketAccept(Native Method) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at java.net.ServerSocket.implAccept(ServerSocket.java:522) "weblogic.timers.TimerThread" daemon prio=5 tid=0x00007f8db9738000 nid=0x5d03 in Object.wait() [0x000000013f9b9000] at java.net.ServerSocket.accept(ServerSocket.java:490) java.lang.Thread.State: TIMED_WAITING (on object monitor) at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:38) at java.lang.Object.wait(Native Method) at weblogic.server.channels.DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:535) at weblogic.timers.internal.TimerThread$Thread.run(TimerThread.java:262) at weblogic.server.channels.DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:417) - locked <0x0000000119e7f158> (a weblogic.timers.internal.TimerThread) - locked <0x000000011ab2f568> (a weblogic.work.ExecuteThread) "weblogic.time.TimeEventGenerator" daemon prio=5 tid=0x00007f8db9724000 nid=0x5c03 in Object.wait() [0x000000013f8b6000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at weblogic.time.common.internal.TimeTable.snooze(TimeTable.java:286) - locked <0x0000000119dfc2e0> (a weblogic.time.common.internal.TimeTable) at weblogic.time.common.internal.TimeEventGenerator.run(TimeEventGenerator.java:117) at java.lang.Thread.run(Thread.java:722) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: 0 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5 tid=0x00007f8dbb232000 nid=0x5b03 in Object.wait() [0x000000013f7b3000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x0000000119fa37e0> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) Yusuke Yamamoto @yusuke #wlstudy
  • 15. スレッドダンプ取得方法• jps / jstack• kill -3 $PID (Unix / Linux)• ctrl-Break (Windows)• VisualVMYusuke Yamamoto @yusuke #wlstudy
  • 16. スレッドダンプ取得方法• jps / jstack• kill -3 $PID (Unix / Linux)• ctrl-Break (Windows)• VisualVMYusuke Yamamoto @yusuke #wlstudy
  • 17. スレッドダンプ取得方法• jps JVMプロセスID,mainクラス名一覧Yusuke Yamamoto @yusuke #wlstudy
  • 18. スレッドダンプ取得方法• jps JVMプロセスID,mainクラス名一覧 “Server”がWebLogicのプロセス mainクラスはweblogic.ServerなのでYusuke Yamamoto @yusuke #wlstudy
  • 19. スレッドダンプ取得方法• jstack [pid]Yusuke Yamamoto @yusuke #wlstudy
  • 20. スレッドダンプ取得のコツ• 時間を空けて複数回取得• スレッドの動きを見るため• 取得した複数のスレッドダンプを比較Yusuke Yamamoto @yusuke #wlstudy
  • 21. スレッドダンプ取得方法• jstack [pid] >> dump.txt 同じファイルに追記するのが現実的Yusuke Yamamoto @yusuke #wlstudy
  • 22. WebLogic独自のスレッドダンプ取得方法 • WLSTconnect(weblogic,weblogic ,t3://localhost:7001)cd (Servers)cd (AdminServer)threadDump() Yusuke Yamamoto @yusuke #wlstudy
  • 23. WebLogic独自のスレッドダンプ取得方法• 管理コンソールYusuke Yamamoto @yusuke #wlstudy
  • 24. スレッドダンプ解析は・・・• 結構難しい• トラブルが起きる前に練習をYusuke Yamamoto @yusuke #wlstudy
  • 25. スレッドヘッダとスタックトレース • スタックトレースの見た目"ExecuteThread: 3 for queue: weblogic.socket.Muxer" daemon prio=5tid=0x00007f8db9712800 nid=0x6203 waiting for monitor entry [0x000000013fec8000]      java.lang.Thread.State: BLOCKED (on object monitor)      at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:93)      - waiting to lock <0x000000011c797450> (a weblogic.socket.PosixSocketMuxer$1)      at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)      at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)      at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:147)      at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:119) Yusuke Yamamoto @yusuke #wlstudy
  • 26. スレッドヘッダ • スレッドの先頭行 スレッドヘッダ"ExecuteThread: 3 for queue: weblogic.socket.Muxer" daemon prio=5tid=0x00007f8db9712800 nid=0x6203 waiting for monitor entry [0x000000013fec8000]      java.lang.Thread.State: BLOCKED (on object monitor)      at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:93)      - waiting to lock <0x000000011c797450> (a weblogic.socket.PosixSocketMuxer$1)      at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)      at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)      at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:147)      at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:119) Yusuke Yamamoto @yusuke #wlstudy
  • 27. スレッド名 • スレッドに付けられた区別用の名前 • weblogic.socket.Muxer[n] : ソケットマクサー • weblogic.kernel.Default[n] : 実行スレッド ‣ 通常weblogic.kernel.Default[n]を解析"ExecuteThread: 3 for queue: weblogic.socket.Muxer" daemon prio=5tid=0x00007f8db9712800 nid=0x6203 waiting for monitor entry [0x000000013fec8000]      java.lang.Thread.State: BLOCKED (on object monitor) スレッド名      at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:93)      - waiting to lock <0x000000011c797450> (a weblogic.socket.PosixSocketMuxer$1)      at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)      at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)      at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:147)      at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:119) Yusuke Yamamoto @yusuke #wlstudy
  • 28. スレッドの状態 • スレッドがしていることの概要 • runnable • waiting for monitor entry • in Object.wait()"ExecuteThread: 3 for queue: weblogic.socket.Muxer" daemon prio=5tid=0x00007f8db9712800 nid=0x6203 waiting for monitor entry [0x000000013fec8000]      java.lang.Thread.State: BLOCKED (on object monitor)      at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:93) スレッドの状態      - waiting to lock <0x000000011c797450> (a weblogic.socket.PosixSocketMuxer$1)      at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)      at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)      at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:147)      at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:119) Yusuke Yamamoto @yusuke #wlstudy
  • 29. スレッドの状態 • runnable • Javaコードを実行中 • ネイティブコードを実行中 • 基本的に正常 • データース待ちも runnable"DynamicListenThread[Default]" daemon prio=5 tid=0x00007f8dbd82b000 nid=0x6e03 runnable java.lang.Thread.State: RUNNABLE at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398) at java.net.ServerSocket.implAccept(ServerSocket.java:522) at java.net.ServerSocket.accept(ServerSocket.java:490) Yusuke Yamamoto @yusuke #wlstudy
  • 30. スレッドの状態 • 待ち状態 • in Object.wait() • 他のスレッドから叩かれるのを待っ ている"[ACTIVE] ExecuteThread: 3 for queue: weblogic.kernel.Default (self-tuning)" daemon prio=5tid=0x00007f8dba33e800 nid=0x430f in Object.wait() [0x000000013f4ca000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:503) at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) - locked <0x000000011a4b00e8> (a weblogic.work.ExecuteThread) at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) Yusuke Yamamoto @yusuke #wlstudy
  • 31. スレッドの状態 • 待ち状態 • waiting for monitor entry"ExecuteThread: 1 for queue: weblogic.socket.Muxer" daemon prio=5 tid=0x00007f8db973b000nid=0x6003 waiting for monitor entry [0x000000013fcc2000] java.lang.Thread.State: BLOCKED (on object monitor) at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:93) - waiting to lock <0x0000000119f22540> (a weblogic.socket.PosixSocketMuxer$1) at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:119)"ExecuteThread: 1 for queue: weblogic.socket.Muxer" daemon prio=5 tid=0x00007f8db973b000nid=0x6003 runnable [0x000000013fcc2000] java.lang.Thread.State: RUNNABLE at weblogic.socket.PosixSocketMuxer.poll(Native Method) at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:102) - locked <0x0000000119f22540> (a weblogic.socket.PosixSocketMuxer$1) at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29) at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:119) Yusuke Yamamoto @yusuke #wlstudy
  • 32. スレッドダンプ解析の肝• スレッドの状態を解析• waiting for monitor entryのスレッドが多 い場合は注目• ロックしているスレッドで握っている リソースは? • 外部リソース • 大きなファイルの読み込み Yusuke Yamamoto @yusuke #wlstudy
  • 33. スレッドダンプ解析の肝• waiting for monitor entryがほとんどない• runnableなスレッドは何をしている?• 何度とっても同じ、似たスタックト レースのスレッドは注意 Yusuke Yamamoto @yusuke #wlstudy
  • 34. スレッドダンプ解析は面倒 • ツールで解析 •侍http://samuraism.jp/samurai/ja/index.html • Java Thread Dump Analyserhttp://mchr3k.github.com/javathreaddumpanalyser/ Yusuke Yamamoto @yusuke #wlstudy
  • 35. 侍 http://samuraism.jp/samurai/Yusuke Yamamoto @yusuke #wlstudy
  • 36. 侍の起動• Java Web Startで起動 クリックYusuke Yamamoto @yusuke #wlstudy
  • 37. 侍の起動• アプリケーションをダウンロード(Mac) http://samuraism.jp/samurai/ja/samurai.zip ダブルクリックYusuke Yamamoto @yusuke #wlstudy
  • 38. スレッドダンプを読み込ませるファイルをドラッグ&ドロップするだけYusuke Yamamoto @yusuke #wlstudy
  • 39. スレッドダンプをビジュアルに表現Yusuke Yamamoto @yusuke #wlstudy
  • 40. スレッドダンプをビジュアルに表現 列: スレッドダンプ1回分Yusuke Yamamoto @yusuke #wlstudy
  • 41. スレッドダンプをビジュアルに表現 行: 同じスレッドの状態遷移Yusuke Yamamoto @yusuke #wlstudy
  • 42. スレッドの状態で色分け表示 色: スレッドの状態Yusuke Yamamoto @yusuke #wlstudy
  • 43. 1つのスレッドを時系列で確認Yusuke Yamamoto @yusuke #wlstudy
  • 44. 状態に変化のないスレッド検出Yusuke Yamamoto @yusuke #wlstudy
  • 45. 状態に変化のないスレッド検出• テーブルビューの場合Yusuke Yamamoto @yusuke #wlstudy
  • 46. 状態に変化のないスレッド検出• 時系列ビューの場合Yusuke Yamamoto @yusuke #wlstudy
  • 47. スレッドダンプパターンYusuke Yamamoto @yusuke #wlstudy
  • 48. アイドル状態• ほとんどのスレッドが灰色Yusuke Yamamoto @yusuke #wlstudy
  • 49. がんばっている状態• ほとんどのスレッドが緑色Yusuke Yamamoto @yusuke #wlstudy
  • 50. 詰まっている状態• ほとんどのスレッドが赤、一部オレンジYusuke Yamamoto @yusuke #wlstudy
  • 51. デッドロックYusuke Yamamoto @yusuke #wlstudy
  • 52. WebLogicのデッドロック自動検出• BEA-000394Yusuke Yamamoto @yusuke #wlstudy
  • 53. WebLogicのデッドロック自動検出Yusuke Yamamoto @yusuke #wlstudy