This document contains screenshots and data from JMX monitoring of a Camel server experiencing performance issues. It shows there are around 1000 Camel threads, most waiting on locks or parked. The configuration file contains 424 routes but the MBeans show 626 available routes. The screenshots also show details on specific stalled Camel and Quartz threads, as well as numerous thread pools and ProducerCache instances that could be related to the performance problems.

1. Jconsole JMX data capture
of a Camel server
This document contains screenshots taken on a production
server. Its solely purpose is to share the collected data with
people on the Camel list and help diagnose some performance
issues.

3. Memory

11. Threads

13. Camel threads
• Some 1000 of them, most of them waiting for some lock, or parked.
• 424 <route> tags in a 4200 lines config file, there are some 626 routes available in the
Mbeans :-? (several routes are FTP from one URI to many URI)
Name: Camel (myCamel) thread #10 - ftp://retalixcon@SRV0043/export
State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@16c83b9
Total blocked: 147 Total waited: 2
Stack trace:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
java.lang.Thread.run(Thread.java:662)
Name: Camel (myCamel) thread #192 - ftp://retalixcon@SRV0126/export
State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@e91163
Total blocked: 28 Total waited: 2
Stack trace:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
java.lang.Thread.run(Thread.java:662)

14. Camel threads (continued)
Name: Camel (myCamel) thread #463 - ftp://retalixcon@SRV0153/export
State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@1a1156f
Total blocked: 426 Total waited: 2
Stack trace:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
java.lang.Thread.run(Thread.java:662)
Name: Camel (myCamel) thread #942 - ftp://retalixcon@SRV1004/export
State: WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@195f6a7
Total blocked: 130 Total waited: 2
Stack trace:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.DelayQueue.take(DelayQueue.java:160)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
java.lang.Thread.run(Thread.java:662)

15. Quartz threads
• 10 of them, mostly seem to be waiting (which should be normal).
Name: DefaultQuartzScheduler_Worker-1
State: TIMED_WAITING on org.quartz.simpl.SimpleThreadPool$WorkerThread@1efb21d
Total blocked: 0 Total waited: 123.407
Stack trace:
java.lang.Object.wait(Native Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)
Name: DefaultQuartzScheduler_Worker-5
State: TIMED_WAITING on org.quartz.simpl.SimpleThreadPool$WorkerThread@188145f
Total blocked: 0 Total waited: 123.428
Stack trace:
java.lang.Object.wait(Native Method)
org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:543)

16. Classes

18. Summary

20. JMX Mbeans
• Many thread pools (some 100) of
FileConsumer and FtpConsumer type alike.
• Many ProducerCache