スローダウン、ハングを一発解決スレッドダンプはトラブルシューティングの味方 #wlstudy

スローダウン、ハングを一発解決

スレッドダンプはトラブルシューティングの味方
山本裕介 @yusuke

Yusuke Yamamoto @yusuke #wlstudy

スローダウン、ハング


スローダウン
• アプリケーションは動いている
• レスポンスがなかなか帰ってこない


ハング
• アプリケーションが固まる
• 応答がない
• 503が帰ってくる


スローダウン、ハングの原因
• マシンスペック不足
• キャパシティプランニングに間違い
‣ キャパシティプランニングをしていない
‣ 台数不足
• 割り当てCPU/メモリ不足
• 仮想化時のリソース配分

• チューニング不足
• JVMオプション
• カーネルオプション
• アプリケーションサーバ設定


• アプリケーション
• 非効率なロジック
• リソースの奪い合い
• I/O待ち


切り分け
• GCログの取得
• GC時の挙動を確認
• スレッドダンプの取得、解析


スレッドダンプとは
• JVMのスレッド状態のスナップショット

• わかること
• ある時点で
• それぞれのスレッドが
• 何をしているか

スレッドダンプでわからないこと

• クラッシュ / ストールの原因
• プロセスが落ちたらスレッドダンプは
取れない


スレッドダンプの利点
• 追加ツール不要
• JProﬁler
• JProbe
• YourKit Java Proﬁler
• 取得時に負荷がかからない
• 運用環境でも手軽に使える
• オフラインで解析できる

スレッドダンプの見た目


2013-01-23 16:32:13
Full thread dump Java HotSpot(TM) 64-Bit Server VM (23.6-b04 mixed mode):

at java.lang.Object.wait(Native Method)
"Attach Listener" daemon prio=5 tid=0x00007f8dbd9dd000 nid=0x9407 waiting on condition [0x0000000000000000]

at java.lang.Object.wait(Object.java:503)

- locked <0x000000011ab16fa8> (a weblogic.work.ExecuteThread)
java.lang.Thread.State: RUNNABLE

at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205)

at weblogic.work.ExecuteThread.run(ExecuteThread.java:226)

"[STANDBY] ExecuteThread: '40' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbcd3c000 nid=0x9303 in Object.wait() [0x0000000144526000]

at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dba9a3000 nid=0x7d03 in Object.wait() [0x0000000142d1b000]
java.lang.Thread.State: WAITING (on object monitor) java.lang.Thread.State: WAITING (on object monitor)

at java.lang.Object.wait(Native Method) "[ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dba9a3000 nid=0x7d03 in Object.wait() [0x0000000142d1b000]


at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: WAITING (on object monitor)





- locked <0x000000011c40fc88> (a weblogic.work.ExecuteThread)


- locked <0x000000011ab18448> (a weblogic.work.ExecuteThread)




"[STANDBY] ExecuteThread: '39' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbdab9800 nid=0x9203 in Object.wait() [0x000000014437b000]

at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[STANDBY] ExecuteThread: '17' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbae29800 nid=0x7c03 in Object.wait() [0x0000000142c18000]

at java.lang.Object.wait(Native Method) "[STANDBY] ExecuteThread: '17' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbae29800 nid=0x7c03 in Object.wait() [0x0000000142c18000]







- locked <0x000000011ab2f568> (a weblogic.work.ExecuteThread)






"[ACTIVE] ExecuteThread: '38' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbd1bf000 nid=0x9103 in Object.wait() [0x0000000144278000]

at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: '16' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbd0b7000 nid=0x7b03 in Object.wait() [0x0000000142b15000]

at java.lang.Object.wait(Native Method) "[ACTIVE] ExecuteThread: '16' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbd0b7000 nid=0x7b03 in Object.wait() [0x0000000142b15000]







- locked <0x000000011ab2e4a0> (a weblogic.work.ExecuteThread)


- locked <0x000000011ab17fc8> (a weblogic.work.ExecuteThread)




"[ACTIVE] ExecuteThread: '37' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dba80b000 nid=0x9003 in Object.wait() [0x0000000144175000]

at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: '15' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbcaa6800 nid=0x7a03 in Object.wait() [0x0000000142a12000]

at java.lang.Object.wait(Native Method) "[ACTIVE] ExecuteThread: '15' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbcaa6800 nid=0x7a03 in Object.wait() [0x0000000142a12000]







- locked <0x000000011ab2d1b0> (a weblogic.work.ExecuteThread)


- locked <0x000000011ab1aeb8> (a weblogic.work.ExecuteThread)




"[ACTIVE] ExecuteThread: '36' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dba479800 nid=0x8f03 in Object.wait() [0x0000000144072000]

at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: '14' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbbc5c800 nid=0x7903 in Object.wait() [0x0000000142856000]

at java.lang.Object.wait(Native Method) "[ACTIVE] ExecuteThread: '14' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbbc5c800 nid=0x7903 in Object.wait() [0x0000000142856000]







- locked <0x000000011ab27da0> (a weblogic.work.ExecuteThread)


- locked <0x000000011ab1bed8> (a weblogic.work.ExecuteThread)




"[STANDBY] ExecuteThread: '35' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8db959f000 nid=0x8e03 in Object.wait() [0x0000000143f6f000]

at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbb83c000 nid=0x7803 in Object.wait() [0x0000000142650000]

at java.lang.Object.wait(Native Method) "[ACTIVE] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbb83c000 nid=0x7803 in Object.wait() [0x0000000142650000]







- locked <0x000000011ab26c90> (a weblogic.work.ExecuteThread)






"[ACTIVE] ExecuteThread: '34' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dba9a4000 nid=0x8d03 in Object.wait() [0x0000000143e6c000]

at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: '12' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbd9c0000 nid=0x7703 in Object.wait() [0x0000000140a29000]

at java.lang.Object.wait(Native Method) "[ACTIVE] ExecuteThread: '12' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbd9c0000 nid=0x7703 in Object.wait() [0x0000000140a29000]













"[STANDBY] ExecuteThread: '33' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbbc5b800 nid=0x8c03 in Object.wait() [0x0000000143d69000]

at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dba94d800 nid=0x7603 in Object.wait() [0x0000000142347000]

at java.lang.Object.wait(Native Method) "[ACTIVE] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dba94d800 nid=0x7603 in Object.wait() [0x0000000142347000]







- locked <0x000000011ab29a28> (a weblogic.work.ExecuteThread)


- locked <0x000000011ab0d3f8> (a weblogic.work.ExecuteThread)




"[ACTIVE] ExecuteThread: '32' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbbc5b000 nid=0x8b03 in Object.wait() [0x0000000143c66000]

at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[STANDBY] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbe196000 nid=0x7503 in Object.wait() [0x0000000140c23000]

at java.lang.Object.wait(Native Method) "[STANDBY] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbe196000 nid=0x7503 in Object.wait() [0x0000000140c23000]









- locked <0x000000011ab0e418> (a weblogic.work.ExecuteThread)




"[ACTIVE] ExecuteThread: '31' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbbd54800 nid=0x8a03 in Object.wait() [0x00000001435d3000]

at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbe195800 nid=0x7403 in Object.wait() [0x00000001408be000]

at java.lang.Object.wait(Native Method) "[ACTIVE] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbe195800 nid=0x7403 in Object.wait() [0x00000001408be000]









- locked <0x000000011ab10b38> (a weblogic.work.ExecuteThread)




"[ACTIVE] ExecuteThread: '30' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbbd8f800 nid=0x8903 in Object.wait() [0x00000001434d0000]

at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbba6d800 nid=0x7303 in Object.wait() [0x000000013f0a9000]

at java.lang.Object.wait(Native Method) "[ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbba6d800 nid=0x7303 in Object.wait() [0x000000013f0a9000]







- locked <0x000000011ab2a9a0> (a weblogic.work.ExecuteThread)


- locked <0x000000011ab0c2d0> (a weblogic.work.ExecuteThread)




"[STANDBY] ExecuteThread: '29' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbbd8f000 nid=0x8803 in Object.wait() [0x0000000143b63000]

at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbc812800 nid=0x7203 in Object.wait() [0x00000001407bb000]

at java.lang.Object.wait(Native Method) "[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbc812800 nid=0x7203 in Object.wait() [0x00000001407bb000]













"[ACTIVE] ExecuteThread: '28' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbcb4e000 nid=0x8703 in Object.wait() [0x0000000143a60000]

at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8db96b5800 nid=0x7103 in Object.wait() [0x00000001406b8000]

at java.lang.Object.wait(Native Method) "[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8db96b5800 nid=0x7103 in Object.wait() [0x00000001406b8000]













"[ACTIVE] ExecuteThread: '27' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbbc91800 nid=0x8603 in Object.wait() [0x000000014395d000]

at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[STANDBY] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8db971b000 nid=0x7003 in Object.wait() [0x00000001403de000]

at java.lang.Object.wait(Native Method) "[STANDBY] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8db971b000 nid=0x7003 in Object.wait() [0x00000001403de000]













"[ACTIVE] ExecuteThread: '26' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbb015800 nid=0x8503 in Object.wait() [0x000000014385a000]

at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbbd5d000 nid=0x6f03 in Object.wait() [0x00000001401d1000]

at java.lang.Object.wait(Native Method) "[ACTIVE] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbbd5d000 nid=0x6f03 in Object.wait() [0x00000001401d1000]







- locked <0x000000011ab1f7b0> (a weblogic.work.ExecuteThread)






"[STANDBY] ExecuteThread: '25' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dba709800 nid=0x8403 in Object.wait() [0x0000000143757000]

at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "DynamicListenThread[Default]" daemon prio=5 tid=0x00007f8dbd82b000 nid=0x6e03 runnable [0x0000000142753000]
java.lang.Thread.State: WAITING (on object monitor) java.lang.Thread.State: RUNNABLE

at java.lang.Object.wait(Native Method) "DynamicListenThread[Default]" daemon prio=5 tid=0x00007f8dbd82b000 nid=0x6e03 runnable [0x0000000142753000]

at java.net.PlainSocketImpl.socketAccept(Native Method)

at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: RUNNABLE

at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)



at java.net.ServerSocket.implAccept(ServerSocket.java:522)

- locked <0x000000011ab207d0> (a weblogic.work.ExecuteThread)


at java.net.ServerSocket.accept(ServerSocket.java:490)



at weblogic.socket.WeblogicServerSocket.accept(WeblogicServerSocket.java:38)


at weblogic.server.channels.DynamicListenThread$SocketAccepter.accept(DynamicListenThread.java:535)
"[ACTIVE] ExecuteThread: '24' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbcb54800 nid=0x8303 in Object.wait() [0x000000013f1cd000]


at weblogic.server.channels.DynamicListenThread$SocketAccepter.access$200(DynamicListenThread.java:417)
java.lang.Thread.State: WAITING (on object monitor)


at weblogic.server.channels.DynamicListenThread.run(DynamicListenThread.java:173)



at java.lang.Thread.run(Thread.java:722)




at java.lang.Thread.run(Thread.java:722) "DynamicListenThread[Default[1]]" daemon prio=5 tid=0x00007f8db97d7800 nid=0x6c03 runnable [0x000000014254d000]

- locked <0x000000011ab21860> (a weblogic.work.ExecuteThread) java.lang.Thread.State: RUNNABLE

at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "DynamicListenThread[Default[1]]" daemon prio=5 tid=0x00007f8db97d7800 nid=0x6c03 runnable [0x000000014254d000]


"[ACTIVE] ExecuteThread: '23' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbc945800 nid=0x8203 in Object.wait() [0x000000014322a000]














- locked <0x000000011ab1d8c8> (a weblogic.work.ExecuteThread)






"[STANDBY] ExecuteThread: '22' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8db959d800 nid=0x8103 in Object.wait() [0x0000000143127000]

at java.lang.Thread.run(Thread.java:722) "DynamicListenThread[Default[7]]" daemon prio=5 tid=0x00007f8dbe02c800 nid=0x6b03 runnable [0x000000014244a000]

at java.lang.Object.wait(Native Method) "DynamicListenThread[Default[7]]" daemon prio=5 tid=0x00007f8dbe02c800 nid=0x6b03 runnable [0x000000014244a000]














"[ACTIVE] ExecuteThread: '21' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbcaa7000 nid=0x8003 in Object.wait() [0x0000000143024000]











at java.lang.Thread.run(Thread.java:722) "DynamicListenThread[Default[


at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "DynamicListenThread[Default[4]]" daemon prio=5 tid=0x00007f8dbc8c7800 nid=0x6903 runnable [0x0000000142244000]
"[STANDBY] ExecuteThread: '20' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbb832800 nid=0x7f03 in Object.wait() [0x0000000142f21000]













"[ACTIVE] ExecuteThread: '19' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbbc5d800 nid=0x7e03 in Object.wait() [0x0000000142e1e000]


at java.lang.Object.wait(Native Method) "DynamicListenThread[Default[8]]" daemon prio=5 tid=0x00007f8dba13f800 nid=0x6803 runnable [0x0000000142141000]








"[ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dba9a3000 nid=0x7d03 in Object.wait() [0x0000000142d1b000]










at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "DynamicListenThread[Default[2]]" daemon prio=5 tid=0x00007f8dbbd06000 nid=0x6703 runnable [0x000000014203e000]
"[STANDBY] ExecuteThread: '17' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbae29800 nid=0x7c03 in Object.wait() [0x0000000142c18000]













"[ACTIVE] ExecuteThread: '16' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbd0b7000 nid=0x7b03 in Object.wait() [0x0000000142b15000]


at java.lang.Object.wait(Native Method) "weblogic.GCMonitor" daemon prio=5 tid=0x00007f8dbe0c0800 nid=0x4507 in Object.wait() [0x0000000140eed000]

at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: TIMED_WAITING (on object monitor)




at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)


- locked <0x000000011a6d83f0> (a java.lang.ref.ReferenceQueue$Lock)

at weblogic.platform.GCMonitorThread.waitForNotiﬁcation(GCMonitorThread.java:88)
"[ACTIVE] ExecuteThread: '15' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbcaa6800 nid=0x7a03 in Object.wait() [0x0000000142a12000]

at weblogic.platform.GCMonitorThread.run(GCMonitorThread.java:64)

at java.lang.Object.wait(Native Method) "Thread-11" prio=5 tid=0x00007f8dbc884000 nid=0x4007 waiting on condition [0x0000000140dea000]

at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: WAITING (parking)


at sun.misc.Unsafe.park(Native Method)


- parking to wait for <0x000000011a50cfa8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)


at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
"[ACTIVE] ExecuteThread: '14' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbbc5c800 nid=0x7903 in Object.wait() [0x0000000142856000]

at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)

at weblogic.utils.concurrent.JDK15ConcurrentBlockingQueue.take(JDK15ConcurrentBlockingQueue.java:89)


at weblogic.store.internal.PersistentStoreImpl.getOutstandingWork(PersistentStoreImpl.java:676)


at weblogic.store.internal.PersistentStoreImpl.run(PersistentStoreImpl.java:751)




at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dba33e800 nid=0x430f in Object.wait() [0x000000013f4ca000]
"[ACTIVE] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbb83c000 nid=0x7803 in Object.wait() [0x0000000142650000]






- locked <0x000000011a4b00e8> (a weblogic.work.ExecuteThread)




at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbad9f800 nid=0x4207 in Object.wait() [0x00000001404e7000]
"[ACTIVE] ExecuteThread: '12' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbd9c0000 nid=0x7703 in Object.wait() [0x0000000140a29000]






- locked <0x000000011a4b0d30> (a weblogic.work.ExecuteThread)




at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "Thread-10" prio=5 tid=0x00007f8dbad89000 nid=0x6603 waiting on condition [0x00000001402db000]
java.lang.Thread.State: WAITING (parking)
"[ACTIVE] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dba94d800 nid=0x7603 in Object.wait() [0x0000000142347000]


- parking to wait for <0x000000011a208ea0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)











"[STANDBY] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbe196000 nid=0x7503 in Object.wait() [0x0000000140c23000]


at java.lang.Object.wait(Native Method) "DoSManager" daemon prio=5 tid=0x00007f8dbd84c800 nid=0x6503 waiting on condition [0x00000001400ce000]

at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: TIMED_WAITING (sleeping)


at java.lang.Thread.sleep(Native Method)


at com.octetstring.vde.DoSManager.run(DoSManager.java:433)

"VDE Transaction Processor Thread" daemon prio=5 tid=0x00007f8dbc82b000 nid=0x6303 in Object.wait() [0x000000013ffcb000]
"[ACTIVE] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbe195800 nid=0x7403 in Object.wait() [0x00000001408be000] java.lang.Thread.State: WAITING (on object monitor)





at com.octetstring.vde.backend.standard.TransactionProcessor.waitChange(TransactionProcessor.java:367)


- locked <0x0000000119edae88> (a com.octetstring.vde.backend.standard.TransactionProcessor)


at com.octetstring.vde.backend.standard.TransactionProcessor.run(TransactionProcessor.java:212)

"ExecuteThread: '3' for queue: 'weblogic.socket.Muxer'" daemon prio=5 tid=0x00007f8db9712800 nid=0x6203 waiting for monitor entry [0x000000013fec8000]
"[ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbba6d800 nid=0x7303 in Object.wait() [0x000000013f0a9000] java.lang.Thread.State: BLOCKED (on object monitor)

at weblogic.socket.PosixSocketMuxer.processSockets(PosixSocketMuxer.java:93)


- waiting to lock <0x0000000119f22540> (a weblogic.socket.PosixSocketMuxer$1)


at weblogic.socket.SocketReaderRequest.run(SocketReaderRequest.java:29)


at weblogic.socket.SocketReaderRequest.execute(SocketReaderRequest.java:42)


at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:147)


at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:119)

"[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbc812800 nid=0x7203 in Object.wait() [0x00000001407bb000] "ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" daemon prio=5 tid=0x00007f8dbac3a000 nid=0x6103 runnable [0x000000013fdc5000]


at weblogic.socket.PosixSocketMuxer.poll(Native Method)




- locked <0x0000000119f22540> (a weblogic.socket.PosixSocketMuxer$1)





"[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8db96b5800 nid=0x7103 in Object.wait() [0x00000001406b8000]


at java.lang.Object.wait(Native Method) "ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" daemon prio=5 tid=0x00007f8db973b000 nid=0x6003 waiting for monitor entry [0x000000013fcc2000]

at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: BLOCKED (on object monitor)







"[STANDBY] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8db971b000 nid=0x7003 in Object.wait() [0x00000001403de000]




at java.lang.Object.wait(Object.java:503) "ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" daemon prio=5 tid=0x00007f8dbaa1b800 nid=0x5f03 waiting for monitor entry [0x000000013fbbf000]

at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) java.lang.Thread.State: BLOCKED (on object monitor)





"[ACTIVE] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbbd5d000 nid=0x6f03 in Object.wait() [0x00000001401d1000]






at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) "[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dba4cc800 nid=0x5e03 in Object.wait() [0x000000013fabc000]

- locked <0x000000011ab07228> (a weblogic.work.ExecuteThread) java.lang.Thread.State: WAITING (on object monitor)



"DynamicListenThread[Default]" daemon prio=5 tid=0x00007f8dbd82b000 nid=0x6e03 runnable [0x0000000142753000]


- locked <0x0000000119fa2d20> (a weblogic.work.ExecuteThread)




at java.net.ServerSocket.implAccept(ServerSocket.java:522) "weblogic.timers.TimerThread" daemon prio=5 tid=0x00007f8db9738000 nid=0x5d03 in Object.wait() [0x000000013f9b9000]

at java.net.ServerSocket.accept(ServerSocket.java:490) java.lang.Thread.State: TIMED_WAITING (on object monitor)




at weblogic.timers.internal.TimerThread$Thread.run(TimerThread.java:262)


- locked <0x0000000119e7f158> (a weblogic.timers.internal.TimerThread)
"weblogic.time.TimeEventGenerator" daemon prio=5 tid=0x00007f8db9724000 nid=0x5c03 in Object.wait() [0x000000013f8b6000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)


at weblogic.time.common.internal.TimeTable.snooze(TimeTable.java:286)

- locked <0x0000000119dfc2e0> (a weblogic.time.common.internal.TimeTable)

at weblogic.time.common.internal.TimeEventGenerator.run(TimeEventGenerator.java:117)

"[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbb232000 nid=0x5b03 in Object.wait() [0x000000013f7b3000]




- locked <0x0000000119fa37e0> (a weblogic.work.ExecuteThread)



2013-01-23 16:32:13

"Attach Listener" daemon prio=5 tid=0x00007f8dbd9dd000 nid=0x9407 waiting on condition [0x0000000000000000]





"[STANDBY] ExecuteThread: '40' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbcd3c000 nid=0x9303 in Object.wait() [0x0000000144526000]

at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dba9a3000 nid=0x7d03 in Object.wait() [0x0000000142d1b000]

at java.lang.Object.wait(Native Method) "[ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dba9a3000 nid=0x7d03 in Object.wait() [0x0000000142d1b000]













"[STANDBY] ExecuteThread: '39' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbdab9800 nid=0x9203 in Object.wait() [0x000000014437b000]

at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[STANDBY] ExecuteThread: '17' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbae29800 nid=0x7c03 in Object.wait() [0x0000000142c18000]

at java.lang.Object.wait(Native Method) "[STANDBY] ExecuteThread: '17' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbae29800 nid=0x7c03 in Object.wait() [0x0000000142c18000]













"[ACTIVE] ExecuteThread: '38' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbd1bf000 nid=0x9103 in Object.wait() [0x0000000144278000]

at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: '16' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbd0b7000 nid=0x7b03 in Object.wait() [0x0000000142b15000]

at java.lang.Object.wait(Native Method) "[ACTIVE] ExecuteThread: '16' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbd0b7000 nid=0x7b03 in Object.wait() [0x0000000142b15000]







- locked <0x000000011ab2e4a0> (a weblogic.work.ExecuteThread)






"[ACTIVE] ExecuteThread: '37' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dba80b000 nid=0x9003 in Object.wait() [0x0000000144175000]

at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: '15' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbcaa6800 nid=0x7a03 in Object.wait() [0x0000000142a12000]

at java.lang.Object.wait(Native Method) "[ACTIVE] ExecuteThread: '15' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbcaa6800 nid=0x7a03 in Object.wait() [0x0000000142a12000]







- locked <0x000000011ab2d1b0> (a weblogic.work.ExecuteThread)






"[ACTIVE] ExecuteThread: '36' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dba479800 nid=0x8f03 in Object.wait() [0x0000000144072000]

at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: '14' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbbc5c800 nid=0x7903 in Object.wait() [0x0000000142856000]

at java.lang.Object.wait(Native Method) "[ACTIVE] ExecuteThread: '14' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbbc5c800 nid=0x7903 in Object.wait() [0x0000000142856000]













"[STANDBY] ExecuteThread: '35' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8db959f000 nid=0x8e03 in Object.wait() [0x0000000143f6f000]

at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbb83c000 nid=0x7803 in Object.wait() [0x0000000142650000]

at java.lang.Object.wait(Native Method) "[ACTIVE] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbb83c000 nid=0x7803 in Object.wait() [0x0000000142650000]













"[ACTIVE] ExecuteThread: '34' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dba9a4000 nid=0x8d03 in Object.wait() [0x0000000143e6c000]

at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: '12' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbd9c0000 nid=0x7703 in Object.wait() [0x0000000140a29000]

at java.lang.Object.wait(Native Method) "[ACTIVE] ExecuteThread: '12' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbd9c0000 nid=0x7703 in Object.wait() [0x0000000140a29000]













"[STANDBY] ExecuteThread: '33' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbbc5b800 nid=0x8c03 in Object.wait() [0x0000000143d69000]

at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dba94d800 nid=0x7603 in Object.wait() [0x0000000142347000]

at java.lang.Object.wait(Native Method) "[ACTIVE] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dba94d800 nid=0x7603 in Object.wait() [0x0000000142347000]













"[ACTIVE] ExecuteThread: '32' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbbc5b000 nid=0x8b03 in Object.wait() [0x0000000143c66000]

at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[STANDBY] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbe196000 nid=0x7503 in Object.wait() [0x0000000140c23000]

at java.lang.Object.wait(Native Method) "[STANDBY] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbe196000 nid=0x7503 in Object.wait() [0x0000000140c23000]













"[ACTIVE] ExecuteThread: '31' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbbd54800 nid=0x8a03 in Object.wait() [0x00000001435d3000]

at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbe195800 nid=0x7403 in Object.wait() [0x00000001408be000]

at java.lang.Object.wait(Native Method) "[ACTIVE] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbe195800 nid=0x7403 in Object.wait() [0x00000001408be000]













"[ACTIVE] ExecuteThread: '30' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbbd8f800 nid=0x8903 in Object.wait() [0x00000001434d0000]

at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbba6d800 nid=0x7303 in Object.wait() [0x000000013f0a9000]

at java.lang.Object.wait(Native Method) "[ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbba6d800 nid=0x7303 in Object.wait() [0x000000013f0a9000]







- locked <0x000000011ab2a9a0> (a weblogic.work.ExecuteThread)






"[STANDBY] ExecuteThread: '29' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbbd8f000 nid=0x8803 in Object.wait() [0x0000000143b63000]

at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbc812800 nid=0x7203 in Object.wait() [0x00000001407bb000]

at java.lang.Object.wait(Native Method) "[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbc812800 nid=0x7203 in Object.wait() [0x00000001407bb000]













"[ACTIVE] ExecuteThread: '28' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbcb4e000 nid=0x8703 in Object.wait() [0x0000000143a60000]

at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8db96b5800 nid=0x7103 in Object.wait() [0x00000001406b8000]

at java.lang.Object.wait(Native Method) "[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8db96b5800 nid=0x7103 in Object.wait() [0x00000001406b8000]













"[ACTIVE] ExecuteThread: '27' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbbc91800 nid=0x8603 in Object.wait() [0x000000014395d000]

at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[STANDBY] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8db971b000 nid=0x7003 in Object.wait() [0x00000001403de000]

at java.lang.Object.wait(Native Method) "[STANDBY] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8db971b000 nid=0x7003 in Object.wait() [0x00000001403de000]













"[ACTIVE] ExecuteThread: '26' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbb015800 nid=0x8503 in Object.wait() [0x000000014385a000]

at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbbd5d000 nid=0x6f03 in Object.wait() [0x00000001401d1000]

at java.lang.Object.wait(Native Method) "[ACTIVE] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbbd5d000 nid=0x6f03 in Object.wait() [0x00000001401d1000]







- locked <0x000000011ab1f7b0> (a weblogic.work.ExecuteThread)






"[STANDBY] ExecuteThread: '25' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dba709800 nid=0x8403 in Object.wait() [0x0000000143757000]

at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "DynamicListenThread[Default]" daemon prio=5 tid=0x00007f8dbd82b000 nid=0x6e03 runnable [0x0000000142753000]

at java.lang.Object.wait(Native Method) "DynamicListenThread[Default]" daemon prio=5 tid=0x00007f8dbd82b000 nid=0x6e03 runnable [0x0000000142753000]







- locked <0x000000011ab207d0> (a weblogic.work.ExecuteThread)







"[ACTIVE] ExecuteThread: '24' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbcb54800 nid=0x8303 in Object.wait() [0x000000013f1cd000]











at java.lang.Thread.run(Thread.java:722) "DynamicListenThread[Default[1]]" daemon prio=5 tid=0x00007f8db97d7800 nid=0x6c03 runnable [0x000000014254d000]

- locked <0x000000011ab21860> (a weblogic.work.ExecuteThread) java.lang.Thread.State: RUNNABLE

at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "DynamicListenThread[Default[1]]" daemon prio=5 tid=0x00007f8db97d7800 nid=0x6c03 runnable [0x000000014254d000]


"[ACTIVE] ExecuteThread: '23' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbc945800 nid=0x8203 in Object.wait() [0x000000014322a000]














2013-01-23 16:32:13

- locked <0x000000011ab1d8c8> (a weblogic.work.ExecuteThread)






"[STANDBY] ExecuteThread: '22' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8db959d800 nid=0x8103 in Object.wait() [0x0000000143127000]

at java.lang.Thread.run(Thread.java:722) "DynamicListenThread[Default[7]]" daemon prio=5 tid=0x00007f8dbe02c800 nid=0x6b03 runnable [0x000000014244a000]

at java.lang.Object.wait(Native Method) "DynamicListenThread[Default[7]]" daemon prio=5 tid=0x00007f8dbe02c800 nid=0x6b03 runnable [0x000000014244a000]















"[ACTIVE] ExecuteThread: '21' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbcaa7000 nid=0x8003 in Object.wait() [0x0000000143024000]











at java.lang.Thread.run(Thread.java:722) "DynamicListenThread[Default[


at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "DynamicListenThread[Default[4]]" daemon prio=5 tid=0x00007f8dbc8c7800 nid=0x6903 runnable [0x0000000142244000]
"[STANDBY] ExecuteThread: '20' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbb832800 nid=0x7f03 in Object.wait() [0x0000000142f21000]







"Attach Listener" daemon prio=5 tid=0x00007f8dbd9dd000 nid=0x9407 waiting on condition







"[ACTIVE] ExecuteThread: '19' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbbc5d800 nid=0x7e03 in Object.wait() [0x0000000142e1e000]


at java.lang.Object.wait(Native Method) "DynamicListenThread[Default[8]]" daemon prio=5 tid=0x00007f8dba13f800 nid=0x6803 runnable [0x0000000142141000]

[0x0000000000000000]








"[ACTIVE] ExecuteThread: '18' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dba9a3000 nid=0x7d03 in Object.wait() [0x0000000142d1b000]











at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "DynamicListenThread[Default[2]]" daemon prio=5 tid=0x00007f8dbbd06000 nid=0x6703 runnable [0x000000014203e000]
"[STANDBY] ExecuteThread: '17' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbae29800 nid=0x7c03 in Object.wait() [0x0000000142c18000]













"[ACTIVE] ExecuteThread: '16' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbd0b7000 nid=0x7b03 in Object.wait() [0x0000000142b15000]


"[STANDBY] ExecuteThread: '40' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5

at java.lang.Object.wait(Native Method) "weblogic.GCMonitor" daemon prio=5 tid=0x00007f8dbe0c0800 nid=0x4507 in Object.wait() [0x0000000140eed000]

at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: TIMED_WAITING (on object monitor)




at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)


- locked <0x000000011a6d83f0> (a java.lang.ref.ReferenceQueue$Lock)

at weblogic.platform.GCMonitorThread.waitForNotiﬁcation(GCMonitorThread.java:88)
"[ACTIVE] ExecuteThread: '15' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbcaa6800 nid=0x7a03 in Object.wait() [0x0000000142a12000]

at weblogic.platform.GCMonitorThread.run(GCMonitorThread.java:64)

tid=0x00007f8dbcd3c000 nid=0x9303 in Object.wait() [0x0000000144526000]

at java.lang.Object.wait(Native Method) "Thread-11" prio=5 tid=0x00007f8dbc884000 nid=0x4007 waiting on condition [0x0000000140dea000]

at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: WAITING (parking)




- parking to wait for <0x000000011a50cfa8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)



"[ACTIVE] ExecuteThread: '14' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbbc5c800 nid=0x7903 in Object.wait() [0x0000000142856000]











at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dba33e800 nid=0x430f in Object.wait() [0x000000013f4ca000]

"[ACTIVE] ExecuteThread: '13' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbb83c000 nid=0x7803 in Object.wait() [0x0000000142650000]










at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbad9f800 nid=0x4207 in Object.wait() [0x00000001404e7000]

"[ACTIVE] ExecuteThread: '12' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbd9c0000 nid=0x7703 in Object.wait() [0x0000000140a29000]






- locked <0x000000011a4b0d30> (a weblogic.work.ExecuteThread)




at weblogic.work.ExecuteThread.run(ExecuteThread.java:226) "Thread-10" prio=5 tid=0x00007f8dbad89000 nid=0x6603 waiting on condition [0x00000001402db000]

java.lang.Thread.State: WAITING (parking)
"[ACTIVE] ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dba94d800 nid=0x7603 in Object.wait() [0x0000000142347000]


- parking to wait for <0x000000011a208ea0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)












"[STANDBY] ExecuteThread: '10' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbe196000 nid=0x7503 in Object.wait() [0x0000000140c23000]


at java.lang.Object.wait(Native Method) "DoSManager" daemon prio=5 tid=0x00007f8dbd84c800 nid=0x6503 waiting on condition [0x00000001400ce000]

at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: TIMED_WAITING (sleeping)


at java.lang.Thread.sleep(Native Method)


at com.octetstring.vde.DoSManager.run(DoSManager.java:433)


"VDE Transaction Processor Thread" daemon prio=5 tid=0x00007f8dbc82b000 nid=0x6303 in Object.wait() [0x000000013ffcb000]
"[ACTIVE] ExecuteThread: '9' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbe195800 nid=0x7403 in Object.wait() [0x00000001408be000] java.lang.Thread.State: WAITING (on object monitor)





at com.octetstring.vde.backend.standard.TransactionProcessor.waitChange(TransactionProcessor.java:367)


- locked <0x0000000119edae88> (a com.octetstring.vde.backend.standard.TransactionProcessor)


at com.octetstring.vde.backend.standard.TransactionProcessor.run(TransactionProcessor.java:212)

"ExecuteThread: '3' for queue: 'weblogic.socket.Muxer'" daemon prio=5 tid=0x00007f8db9712800 nid=0x6203 waiting for monitor entry [0x000000013fec8000]
"[ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbba6d800 nid=0x7303 in Object.wait() [0x000000013f0a9000] java.lang.Thread.State: BLOCKED (on object monitor)








"[STANDBY] ExecuteThread: '39' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5





"[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbc812800 nid=0x7203 in Object.wait() [0x00000001407bb000] "ExecuteThread: '2' for queue: 'weblogic.socket.Muxer'" daemon prio=5 tid=0x00007f8dbac3a000 nid=0x6103 runnable [0x000000013fdc5000]







tid=0x00007f8dbdab9800 nid=0x9203 in Object.wait() [0x000000014437b000]





"[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8db96b5800 nid=0x7103 in Object.wait() [0x00000001406b8000]


at java.lang.Object.wait(Native Method) "ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" daemon prio=5 tid=0x00007f8db973b000 nid=0x6003 waiting for monitor entry [0x000000013fcc2000]

at java.lang.Object.wait(Object.java:503) java.lang.Thread.State: BLOCKED (on object monitor)








"[STANDBY] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8db971b000 nid=0x7003 in Object.wait() [0x00000001403de000]




at java.lang.Object.wait(Object.java:503) "ExecuteThread: '0' for queue: 'weblogic.socket.Muxer'" daemon prio=5 tid=0x00007f8dbaa1b800 nid=0x5f03 waiting for monitor entry [0x000000013fbbf000]


at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) java.lang.Thread.State: BLOCKED (on object monitor)





"[ACTIVE] ExecuteThread: '4' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbbd5d000 nid=0x6f03 in Object.wait() [0x00000001401d1000]







at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:205) "[ACTIVE] ExecuteThread: '1' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dba4cc800 nid=0x5e03 in Object.wait() [0x000000013fabc000]

- locked <0x000000011ab07228> (a weblogic.work.ExecuteThread) java.lang.Thread.State: WAITING (on object monitor)



"DynamicListenThread[Default]" daemon prio=5 tid=0x00007f8dbd82b000 nid=0x6e03 runnable [0x0000000142753000]


- locked <0x0000000119fa2d20> (a weblogic.work.ExecuteThread)





at java.net.ServerSocket.implAccept(ServerSocket.java:522) "weblogic.timers.TimerThread" daemon prio=5 tid=0x00007f8db9738000 nid=0x5d03 in Object.wait() [0x000000013f9b9000]

at java.net.ServerSocket.accept(ServerSocket.java:490) java.lang.Thread.State: TIMED_WAITING (on object monitor)




at weblogic.timers.internal.TimerThread$Thread.run(TimerThread.java:262)


- locked <0x0000000119e7f158> (a weblogic.timers.internal.TimerThread)

"weblogic.time.TimeEventGenerator" daemon prio=5 tid=0x00007f8db9724000 nid=0x5c03 in Object.wait() [0x000000013f8b6000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)


at weblogic.time.common.internal.TimeTable.snooze(TimeTable.java:286)

- locked <0x0000000119dfc2e0> (a weblogic.time.common.internal.TimeTable)

at weblogic.time.common.internal.TimeEventGenerator.run(TimeEventGenerator.java:117)


"[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5 tid=0x00007f8dbb232000 nid=0x5b03 in Object.wait() [0x000000013f7b3000]




- locked <0x0000000119fa37e0> (a weblogic.work.ExecuteThread)



スレッドダンプ取得方法
• jps / jstack
• kill -3 $PID (Unix / Linux)
• ctrl-Break (Windows)
• VisualVM


• jps

JVMプロセスID,mainクラス名一覧


• jps

JVMプロセスID,mainクラス名一覧

“Server”がWebLogicのプロセス
mainクラスはweblogic.Serverなので


• jstack [pid]


スレッドダンプ取得のコツ
• 時間を空けて複数回取得
• スレッドの動きを見るため
• 取得した複数のスレッドダンプを比較


• jstack [pid] >> dump.txt

同じファイルに追記するのが現実的


WebLogic独自のスレッドダンプ取得方法

• WLST
connect('weblogic','weblogic'
,'t3://localhost:7001')
cd ('Servers')
cd ('AdminServer')
threadDump()


WebLogic独自のスレッドダンプ取得方法

• 管理コンソール


スレッドダンプ解析は・・・
• 結構難しい
• トラブルが起きる前に練習を


スレッドヘッダとスタックトレース

• スタックトレースの見た目

"ExecuteThread: '3' for queue: 'weblogic.socket.Muxer'" daemon prio=5
tid=0x00007f8db9712800 nid=0x6203 waiting for monitor entry [0x000000013fec8000]
     java.lang.Thread.State: BLOCKED (on object monitor)
     - waiting to lock <0x000000011c797450> (a weblogic.socket.PosixSocketMuxer$1)


スレッドヘッダ
• スレッドの先頭行

スレッドヘッダ



スレッド名
• スレッドに付けられた区別用の名前
• weblogic.socket.Muxer[n] : ソケットマクサー
• weblogic.kernel.Default[n] : 実行スレッド
‣ 通常weblogic.kernel.Default[n]を解析
スレッド名


スレッドの状態
• スレッドがしていることの概要
• runnable
• waiting for monitor entry
• in Object.wait()


• runnable
• Javaコードを実行中
• ネイティブコードを実行中
• 基本的に正常
• データース待ちも runnable
"DynamicListenThread[Default]" daemon prio=5 tid=0x00007f8dbd82b000 nid=0x6e03 runnable






• 待ち状態
• in Object.wait()
• 他のスレッドから叩かれるのを待っ
ている

"[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=5
tid=0x00007f8dba33e800 nid=0x430f in Object.wait() [0x000000013f4ca000]






• 待ち状態
• waiting for monitor entry
"ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" daemon prio=5 tid=0x00007f8db973b000
nid=0x6003 waiting for monitor entry [0x000000013fcc2000]




"ExecuteThread: '1' for queue: 'weblogic.socket.Muxer'" daemon prio=5 tid=0x00007f8db973b000
nid=0x6003 runnable [0x000000013fcc2000]







スレッドダンプ解析の肝
• スレッドの状態を解析
• waiting for monitor entryのスレッドが多
い場合は注目

• ロックしているスレッドで握っている
リソースは？

• 外部リソース
• 大きなファイルの読み込み

スレッドダンプ解析の肝
• waiting for monitor entryがほとんどない
• runnableなスレッドは何をしている？
• 何度とっても同じ、似たスタックト
レースのスレッドは注意


スレッドダンプ解析は面倒
• ツールで解析
•侍
http://samuraism.jp/samurai/ja/index.html

• Java Thread Dump Analyser
http://mchr3k.github.com/javathreaddumpanalyser/


侍

http://samuraism.jp/samurai/

侍の起動
• Java Web Startで起動

クリック


侍の起動
• アプリケーションをダウンロード(Mac)
http://samuraism.jp/samurai/ja/samurai.zip

ダブルクリック


スレッドダンプを読み込ませる

ファイルをドラッグ&ドロップするだけ


スレッドダンプをビジュアルに表現


列: スレッドダンプ１回分


行: 同じスレッドの状態遷移


スレッドの状態で色分け表示
色: スレッドの状態


１つのスレッドを時系列で確認


状態に変化のないスレッド検出


• テーブルビューの場合


• 時系列ビューの場合


スレッドダンプパターン


アイドル状態
• ほとんどのスレッドが灰色


がんばっている状態
• ほとんどのスレッドが緑色


詰まっている状態
• ほとんどのスレッドが赤、一部オレンジ


デッドロック


WebLogicのデッドロック自動検出

• BEA-000394


WebLogicのデッドロック自動検出


スローダウン、ハングを一発解決 スレッドダンプはトラブルシューティングの味方 #wlstudy

More Related Content

What's hot

More from Yusuke Yamamoto

Recently uploaded