Download to read offline
More Related Content
Similar to Java Native Memory Leaks: The Hidden Villain Behind JVM Performance Issues
Java Native Memory Leaks: The Hidden Villain Behind JVM Performance Issues
- 1.
- 2. Does Java processmemory utilization go beyond –Xmx? –XX:MaxMetaspaceSize -Xmx Young Old Metaspac e Threads JNI misc GC Direc t Buff Code Cach e Heap Memory Native Memory Java Process Memory -XX:MaxDirectMemorySize -XX:ReservedCodeCacheSize
- 3. What is ourTypical Response When there is OutOfMemoryError? Increase -Xmx Will it Work When There is a Native Memory Leak? NO. You will worsen the problem!
- 4. • Stores artifactsrequired to execute our application (Threads, GC, Metadata ...) • Managed by OS & JVM (except Metaspace) • -XX:MaxMetaspaceSize, - XX:ReservedCodeCacheSize, - XX:MaxDirectMemorySize. ... • Faster • Stores Application Objects i.e., Customer, Account, List… • Managed by Garbage Collector • -Xmx • Not as fast as Native Memory - involves traversing pointers and managing object references Heap Memory Native Memory
- 5.
- 6. 6 Heap Usage ofan Application Suffering from Heap Memory Leak - Full Garbage Collection Event
- 7. 7 Heap Usage ofan Application Suffering from Native Memory Leak - Full Garbage Collection Event
- 8. 8 Only Native MemoryRegion Grows! Since Native Memory Regions are quite small. Native Memory Leaks are not quite visible in the APM Tools.
- 9. 9 How To FindWhich Native Memory Region Is Leaking? Native Memory Tracking (NMT) OutOfMemoryError Type java.lang.OutOfMemoryError: <type> OutOfMemoryError Type Impacted Native Memory Region java.lang.OutOfMemoryError: Metaspace Metaspace java.lang.OutOfMemoryError: unable to create new native thread Threads java.lang.OutOfMemoryError: Direct buffer memory Direct Memory java.lang.OutOfMemoryError: Code Cache full Code Cache
- 10.
- 11. 11 Native Memory Tracking(NMT) Start your application with this JVM flag: java -XX:NativeMemoryTracking=summary -jar YourApplication.jar In Period Intervals (say every 10 minutes) for 24 hours, issue the command: jcmd <pid> VM.native_memory summary > nmt_report.txt Upload nmt_report.txt to GCeasy tool (https://gceasy.io)
- 12. 12 Common Native MemoryLeaks Causes, Artifacts, Tools & Solution Young Old Metaspac e Threads JNI misc GC Direc t Buff Code Cach e Heap Memory Native Memory Java Process Memory
- 13. 13 Threads Leak 1 Young Old Metaspac e ThreadsJNI misc GC Direc t Buff Code Cach e Heap Memory Native Memory Java Process Memory java.lang.OutOfMemoryError: Unable to create new native threads
- 14. 14 How to captureThread Dump? 9 options:https://blog.fastthread.io/how-to-take-thread-dumps-7-options/
- 15. Open-Source yc-360 Script: https://github.com/ycrash/yc-360-script 5.GC Log 10. netstat 12. vmstat 2. Thread Dump 9. dmesg 3. Heap Dump 13. ps 15. Disk Usage 7. top 6. Extended Data 11. ping 14. Kernel Params 1. App Logs 16. metadata 4. Heap Substitute 8. top -H 15 Effective Troubleshooting Strategy: 360° Artifacts
- 16. 16 Real Case Study Slowdownin a Major Financial Institution’s Middleware in USA http://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMTcvMDMvMTQvLS10aHJlYWREdW1wLTIudHh0LS0xMi0yOC0zMw==&s=t
- 17. 17 OutOfMemoryError: Unable tocreate new native threads Causes 1. Threads are leaking Solutions 1. Fix thread leak 2. Increase the Thread Limits Set at Operating System(ulimit –u) 3. Kill other processes 4. Increase RAM capacity 5. Reduce Java Heap Size 6. Reduce thread stack size (-Xss). Note: can cause Artifacts 1. Thread Dump Tools 1. FastThread 2. Text Editor
- 18. 18 Metaspace Memory Leak 2 YoungOld Metaspac e Threads JNI misc GC Direc t Buff Code Cach e Heap Memory Native Memory Java Process Memory java.lang.OutOfMemoryError: Metaspace
- 19. 19 Real Case Study MetaspaceMemory Leak Due to a Kotlin Library https://blog.gceasy.io/troubleshooting-microservices-outofmemoryerror-metaspace/
- 20. 20 GC Behavior DuringMetaspace Memory Leak - Full Garbage Collection Event
- 21.
- 22. 22 How to studyMetaspace Behavior? java {app_name} -verbose:class Till Java 8 java {app_name} - Xlog:class+load=info:<filename> From Java 9
- 23. 23 OutOfMemoryError: Metaspace Causes 1. Creatinglarge number of Dynamic classes (Java Reflection, Groovy type of scripting languages) 2. Loading large number of Classes (3rd party libraries/frameworks) 3. Loading large number of Classloaders Solutions 1. Fix Memory Leak code 2. Increase -XX:MetaspaceSize and -XX:MaxMetaspaceSize. Artifacts 1. Java 8 & below: java {app_name} -verbose:class 2. Java 9 & above: java {app_name} - Xlog:class+load=info:<filena me> 3. jcmd {pid} GC.class_histogram 4. Heap Dump Tools 1. GCeasy (GC log analysis) 2. HeapHero 3. Eclipse MAT
- 24. 24 DirectBuffer Memory Leak 3 YoungOld Metaspac e Threads JNI misc GC Direc t Buff Code Cach e Heap Memory Native Memory Java Process Memory java.lang.OutOfMemoryError: Direct buffer memory
- 25.
- 26. 26 OutOfMemoryError: Direct buffermemory Causes 1. Increase(or Leak) in direct buffers usage a. java.nio package b. Moving from Spring RestTemplate to WebClient c. Image Processing Libraries d. Networking Libraries e. Some JDBC Drivers Solutions 1. Fix Memory Leak code 2. Increase - XX:MaxDirectMemorySize 3. Upgrade to Java 17 (There some issues in Java 11) Artifacts 1. App Log or Std error 2. Native Memory Tracking Tools
- 27.
- 28. Thank you, myfriends! Ram Lakshmanan @ycrash_rca This deck will be published in: https://blog.ycrash.io https://www.linkedin.com/company/ycrash
Editor's Notes
- #1 HK: Title Slide
- #14 https://blog.fastthread.io/how-to-take-thread-dumps-7-options/
- #16 C:\workspace\speeches\java-native-memory-leaks\middleware-thread-leak.txt http://localhost:8080/yc-report.jsp?ou=SAP&de=10.139.1.87&app=yc&ts=2025-04-23T13-28-53
- #20 https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMjIvMDcvMTYvbWV0YXNwYWNlLW9vbWVycm9yLmdjLS0xOC0xNy0zMg==&s=t&channel=WEB
- #21 OOMMetaspace
- #23 OOMMetaspace_verbose
- #25 OOMDirectBuffer