https://meetups.mulesoft.com/events/details/mulesoft-online-group-english-presents-advanced-troubleshooting-of-application-performance/#/

1. All contents © MuleSoft, LLC
Register at https://connect.mulesoft.com/ -
it’s free!

2. All contents © MuleSoft, LLC
Vitalii Mykytenko
April 2020
Advanced Troubleshooting of
Application Performance
MuleSoft Meetup

3. All contents © MuleSoft, LLC
Professional
About Myself
● Senior Customer Architect, JAPAC Support Team
● 20+ years working in IT
● 5+ years working with heavily loaded near real-time large scale integration platforms
processing billions of records daily
● Joined MuleSoft as a Customer Architect in November 2017
● Specialising in Performance and Reliability Troubleshooting
3

4. All contents © MuleSoft, LLC
What’s Covered
4
● Mule 4 on top of JVM:
○ memory distribution
○ execution engine
○ dumps
● Getting applications ready, i.e. proactive troubleshooting:
○ CloudHub
○ On-Premise
● Troubleshooting Toolset
● Demo

5. All contents © MuleSoft, LLC
Mule 4 Memory and Threading

6. All contents © MuleSoft, LLC
Mule 4. JVM MemoryJVMProcess
Heap: 1GB
New Generation: 512MB
Old Generation: 512MB
Native Memory: Variable
Metaspace: 256MB
Thread Stack: 1MB/thr,
160-180 threads
Code Cache: up to 240MB
by default. Usually
considerably less
Internal: used by NIO
buffers, streaming, etc. By
default can grow up to the
size of HEAP
Others: GC, Compiler,
Symbol, Arena Chunk...
Fig. Typical memory distribution for an application
deployed to 1GB Heap runtime
Recommendations
● Always reserve at least 50% of physical memory free
when calculating memory required for a runtime
server. Caveat: Maybe different on a large server
● Add -XX:NativeMemoryTracking=summary to
wrapper.conf for an enhanced monitoring as even
Java Profilers do not show all allocations. Caution:
might have a performance impact

7. All contents © MuleSoft, LLC
Scheduler Pools
Mule 4. Execution Engine
● Thread Types:
○ CPU Light, 2*cores, http:request
○ CPU Intensive, 2*cores, ee:transform
○ IO Blocking Threads, db:select
● Event processors and Mule SDK
● MULE_HOME/conf/scheduler-pools.conf
● CloudHub is not configurable
● Thread type are printed in logs and thread dumps
● No thread leaks unlike Mule 3
● MULE: FLOW_BACK_PRESSURE
7

8. All contents © MuleSoft, LLC
Thread Dump
Mule 4. Java Dumps
● Look for patterns
● Pay attention to locks
● Check the app source code
● Check which thread is consuming CPU
8
OS Thread ID

9. All contents © MuleSoft, LLC
Heap Dump
Mule 4. Java Dumps
● jmap - java memory map
○ generate HEAP
○ build histogram
● Eclipse MAT
○ Leak Suspect Report
○ OQL
9

10. All contents © MuleSoft, LLC
Getting Applications Ready
● CloudHub
● On-Premise

11. All contents © MuleSoft, LLC
CloudHub. Challenges
Getting Applications Ready
● App works perfect in Anypoint Studio yet unresponsive/restarts in CloudHub
● 1 vCore vs. 5x0.2 vCore
● Load testing and CloudHub fractional vCore workers
● “No space left on device”
● Logging and CloudHub
11

12. All contents © MuleSoft, LLC
CloudHub. Memory Management 1/2
Getting Applications Ready
12
● Processes never works with physical memory directly
● Own VIRTual address space for every process
● Process can allocate more than an actual physical
memory is available
● top command:
○ VIRT: Addressed virtual space
○ RES: Actual size in physical memory
● If I add swap does it mean I can process process
more?
Kernel Page
Table
JVM Page Table
swap
0x0000
0x4a00
...
0xXXXX
JVM Process
Virtual Memory
Virtual
Address
Physical
Address
0x0000 0x0000
0x4a00 0x5000
...
0xXXXX 0xZZZZ
0x0000
0x2000
...
Physical Memory
0x5000
...
0x7000
...
0xYYYY
0xZZZZ
0x0000
0x4a00
...
0xXXXX
Process X Virtual
Memory X Page Table
Virtual
Address
Physical
Address
0x0000 0x1000
0x4a00 0x6000
...
0xXXXX 0xYYYY

13. All contents © MuleSoft, LLC
CloudHub. Memory Management 2/2
Getting Applications Ready
13
Memory fragmentation
● external fragmentation
● internal fragmentation
● swap is for inactive pages
● intensive swapping = performance
degradation
payload1(3)
payload2(2)
free(payload2)
payload4(6) ?
payload3(4)
/proc/$(pidof
java)/status
Name: java
VmPeak: 3301880 kB
VmSize: 3167708 kB
VmHWM: 849720 kB
VmRSS: 692832 kB
VmData: 1263096 kB
VmSwap: 294144 kB
Threads: 123
Payload
Memory Page
Internal
fragmentation

14. All contents © MuleSoft, LLC
CloudHub. Native Memory
Getting Applications Ready
14

15. All contents © MuleSoft, LLC
CloudHub. Let’s Get the App Ready in Studio
Getting Applications Ready
● scheduler-pools.conf of Studio embedded runtime
● Run As -> Mule Application (configure)
● Not considered:
○ burstable resources
○ memory fragmentation
○ disk space
○ logging
15
0.1 vCore
* a laptop has 2 physical or 4
hyper threading (HT) CPU cores

16. All contents © MuleSoft, LLC
● Try simulating the CloudHub worker specs:
○ threading profile
○ HEAP allocated
○ limit native memory allocations
● Consider limited disk free space
● Logging. Production deployments should have it limited
● Capacity planning and impact assessment:
○ 1 vCore vs. 5x0.2 vCore:
■ payload size
■ processing intensity
○ Load testing and CloudHub fractional vCore workers?
CloudHub. To Consider
Getting Applications Ready
16

17. All contents © MuleSoft, LLC
Getting Applications Ready
● CloudHub
● On-Premise

18. All contents © MuleSoft, LLC
On-premise. Challenges
Getting Applications Ready
● Resource allocation
○ memory
○ CPU cores
● Operating system limits
○ processes
○ open file descriptors
● No. of apps to deploy
● Gradual performance degradation
○ resources allocated
○ No. of apps deployed is growing
● Supportability
○ JDK or JRE?
○ Additional tools
18

19. All contents © MuleSoft, LLC
1 conf/wrapper.conf: keep 50-50 proportion, i.e. 50% of
memory for Heap and 50% of Heap for New Generation
area. Leave the rest of native memory occupiers and OS
# Might differ on large machines
wrapper.java.initmemory=1024 (change defaults)
wrapper.java.maxmemory=1024 (change defaults)
2 conf/scheduler-pools.conf: 1CPU-core-32GB-RAM VM box:
FLOW OVERLOAD, MULE:FLOW_BACK_PRESSURE
Fine-tuning the CPU intensive and CPU Light threads is
required
3 System limits not configured
$ ulimit -a might misinform
$ cat /proc/$(pidof java)/limits
Max processes 65000 65000 processes
Max open file 65000 65000 files
4 swap / vm.swappiness Configure SWAP partition and fine-tune vm.swappiness to
reduce unnecessary swapping if needed
5 No. of applications per a runtime The high limit of apps per runtime is 200. However, it is
recommended to keep this limit ~20 apps per runtime. This
helps to improve the performance and supportability
6 HEAP OOME wrapper.java.additional.<n>=-XX:+HeapDumpOnOutOfMemoryErr
or
7 The level of logging Manage the volume of logging through implementing the
adaptive logging (links)
8 Support Information Collector All troubleshooting metrics in one place (links)
On-Premise. To Consider
Getting Applications Ready
19

20. All contents © MuleSoft, LLC
Troubleshooting Toolset
● JDK CLI
● Linux
● Others

21. All contents © MuleSoft, LLC
JDK CLU
Troubleshooting Tools
21
jstack -l $(pidof java) > thread.dump Generate a thread dump, the execution state snapshot with every thread stack traces.
Useful, when taken several ones with some interval. Allows to understand at class the
execution is blocked. Also useful in conjunction with other tools to identify, for
example, CPU consuming thread
jmap -dump:format=b,file=<APP_NAME>.hprof $(pidof java) Generate a heap dump, the current JVM HEAP memory snapshot.
jmap -histo $(pidof java) Generate heap memory histogram
-XX:+HeapDumpOnOutOfMemoryError Generated heap dump on out of memory error. On-prem installations
-XX:NativeMemoryTracking=summary
jcmd $(pidof java) VM.native_memory
Allows to track native memory occupiers. Helps to identify the real memory
requirement for the application. Caveat: has performance impact
jcmd $(pidof java) GC.run Perform full garbage collection
jstat -gccause -h 1000 -t $(pidof java) 10s > gc.log Generate GC log per an interval of time. E.g. 10s. Helps to identify if it’s the time to
resize JVM Heap
jps Java process list

22. All contents © MuleSoft, LLC
top -d1 -c update interval in seconds, -c – show full command, -d - update
interval, -H to show process threads, Shift+N to change sorting mode
watch Allows to execute commands in a “top”-monitoring way
sar 1 1000
vmstat 1 1000
sysstat is powerful package to monitor all OS resources, e.g. CPU
utilisation breakdown (user, system, io wait, etc.), io, swapping -W,
paging -B, memory -r
vmstat: virtual memory, io, swapping, CPU breakdown stats (similar to
sar)
lsof -p $(pidof java) check open file descriptors for a process
/proc/meminfo memory statistics at a moment of time (next slides)
/proc/cpuinfo a quick way to check number of CPU cores
/proc/$(pidof java)/limits a proper way of checking process limits, i.e. ulimit -a
/proc/$(pidof java)/status running process status, memory, threads, etc. (next slides)
netstat -tulpn check network ports open
kill -3 $(pidof java) generate native thread dump in mule_ee.log including memory report.
Useful, for example, to roughly estimate JVM memory occupied
Linux
Troubleshooting Tools
22
/proc/meminfo
MemFree: 82072 kB
MemAvailable: 234700 kB
Buffers: 100928 kB
Cached: 165108 kB
SwapCached: 0 kB
Active: 667976 kB
Inactive: 164084 kB
Active(file): 148080 kB
Inactive(file):117900 kB
SwapTotal: 262140 kB
SwapFree: 262140 kB
VmallocTotal: 34359738367 kB
/proc/$(pidof
java)/status
Name: java
VmPeak: 3301880 kB
VmSize: 3167708 kB
VmHWM: 849720 kB
VmRSS: 692832 kB
VmData: 1263096 kB
VmSwap: 294144 kB
Threads: 123

23. All contents © MuleSoft, LLC
Others. GUI and CLI
Troubleshooting Tools
● JConsole. Supplied with JDK
● Eclipse MAT. Heap dump analyser.
● VisualVM
● YourKit
● JMeter
● hey
● curl
23

24. All contents © MuleSoft, LLC
Troubleshooting
Exercise
Demo
24
Configuration
• 1 GB RAM
• 50% for HEAP
• Tiny swap partition
Demos
• Sanity checks using Linux commands
– memory occupied
– residential memory high watermark
– number of threads
• Identify CPU consuming thread
• Native memory tracking
• Quick HEAP issue estimate
• Take a HEAP dump, MAT

25. All contents © MuleSoft, LLC
Resources Used
Links
MuleSoft Documentation
● https://docs.mulesoft.com/runtime-manager/cloudhub-architecture
● https://docs.mulesoft.com/runtime-manager/deploying-to-cloudhub
● https://www.mulesoft.com/lp/whitepaper/api/reactive-programming
MuleSoft Help Centre
● https://help.mulesoft.com/s/article/Application-Deployed-to-a-Fractional-vCore-Worker-in-CloudHub-Throws-java-lang-OutOf
MemoryError-Direct-buffer-memory
● https://help.mulesoft.com/s/article/The-Impact-of-The-Excessive-Verbose-Logging-on-Application-Performance
● https://help.mulesoft.com/s/article/Support-information-collector
● https://help.mulesoft.com/s/article/Dealing-with-MULE-FLOW-BACK-PRESSURE-error
● https://help.mulesoft.com/s/article/scripts-for-memory-leaking-issues-linux
Other
● https://docs.oracle.com/javase/8/
25

26. All contents © MuleSoft, LLC
Thank you!
Q&A