SlideShare a Scribd company logo
1
Does Java process memory utilization go beyond –Xmx?
2
–XX:MaxMetaspaceSize
-Xmx
Young Old
Metaspac
e
Threads JNI misc
GC
Direct
Buff
Code
Cach
e
Heap Memory Native Memory
Java Process Memory
3
Demo
OutOfMemoryError
4
How to know the type of OutOfMemoryError?
java.lang.OutOfMemoryError: <type>
5
Java Heap Space
Metaspac
e
Threads JNI misc
GC
Direct
Buff
Code
Cach
e
Heap Memory Native Memory
JVM Memory
Most common type
6
GC Behavior of a Healthy Application
- Full Garbage Collection Event
7
GC Behavior of Acute Memory Leak
- Full Garbage Collection Event
8
GC Behavior when there is a Memory Leak
- Full Garbage Collection Event
9
GC Behavior when App needs more memory
- Full Garbage Collection Event
10
How to study GC Behavior?
-XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:<file-path>
Till Java 8
-Xlog:gc*:file=<file-path>
From Java 9
Enable GC logs (always)
Almost zero overhead
11
GCeasy
GC Log Analysis Demo
12
How to capture Heap Dumps?
8 options: https://blog.heaphero.io/2017/10/13/how-to-capture-java-heap-dumps-7-options/
1. GC Log
10. netstat
12. vmstat
2. Thread Dump
9. dmesg
3. Heap Dump
6. ps
8. Disk Usage
5. top 13. iostat
11. ping
14. Kernel Params
15. App Logs
16. metadata
4. Heap Substitute
7. top -H
13
Open-source script:
https://github.com/ycrash/yc-data-script
360° Troubleshooting artifacts
14
HeapHero
Heap Dump Analysis Demo
15
OutOfMemoryError: Java Heap Space
Causes
1. Increase in Traffic Volume
2. Memory leak due to
buggy code
Solutions
1. Fix Memory Leak in the
code
2. Increase heap size (-Xmx)
Artifacts
1. Garbage Collection Log
2. Heap Dump
Tools
1. GCeasy (GC log
analysis)
2. HeapHero
3. Eclipse MAT
4. JVisualVM
16
GC overhead limit exceeded
Metaspac
e
Threads JNI misc
GC
Direct
Buff
Code
Cach
e
Heap Memory Native Memory
JVM Memory
17
OutOfMemoryError: GC overhead limit exceeded
Diagnosis: Same as ‘Java Heap Space’ 
Java process is spending more than 98% of its time doing garbage collection and recovering
less than 2% of the heap and has been doing so far the last 5 consecutive garbage
collections
18
Requested array size exceeds VM limit
Young Old
Metaspac
e
Threads JNI misc
GC
Direct
Buff
Code
Cach
e
Heap Memory Native Memory
JVM Memory
19
Demo
OutOfMemoryError: Requested array size exceeds VM
limit
20
OutOfMemoryError: Requested array size exceeds VM limit
Causes
1. Allocated array larger than
heap size
Solutions
1. Fix code which allocates large
array size
2. Increase heap size (-Xmx)
Artifacts
1. Application Log or std error
Tools
21
Metaspace
Young Old Threads JNI misc
GC
Direct
Buff
Code
Cach
e
Heap Memory Native Memory
JVM Memory
22
Demo
OutOfMemoryError: Metaspace
23
Metaspace Memory Leak
- Full Garbage Collection Event
24
Metaspace Analysis
1. Java 8 & below: java {app_name} -verbose:class
2. Java 9 & above: java {app_name} -Xlog:class+load=info:<filename>
25
OutOfMemoryError: Metaspace
Causes
1. Creating large number of
Dynamic classes (Java
Reflection, Groovy type of
scripting languages)
2. Loading large number of Classes
(3rd party libraries/frameworks)
3. Loading large number of
Classloaders
Solutions
1. Fix Memory Leak code
2. Increase -XX:MetaspaceSize
and
-XX:MaxMetaspaceSize.
Artifacts
1. Java 8 & below: java
{app_name} -verbose:class
2. Java 9 & above: java
{app_name} -
Xlog:class+load=info:<filena
me>
3. jcmd {pid} GC.class_histogram
4. Heap Dump
Tools
1. GCeasy (GC log analysis)
2. HeapHero
3. Eclipse MAT
26
Permgen space
Young Old Threads JNI misc
GC
Direct
Buff
Code
Cach
e
Heap Memory Native Memory
JVM Memory
Happens only in Java 7 & below
27
OutOfMemoryError: Permgen Space
Diagnosis: Same as ‘Metaspace’ 
28
Unable to create new native threads
Young Old
Metaspac
e
JNI misc
GC
Direct
Buff
Code
Cach
e
Heap Memory Native Memory
JVM Memory
29
How to capture Thread Dump?
9 options:https://blog.fastthread.io/how-to-take-thread-dumps-7-options/
1. GC Log
10. netstat
12. vmstat
2. Thread Dump
9. dmesg
3. Heap Dump
6. ps
8. Disk Usage
5. top 13. iostat
11. ping
14. Kernel Params
15. App Logs
16. metadata
4. Heap Substitute
7. top -H
30
Open-source script:
https://github.com/ycrash/yc-data-script
360° Troubleshooting artifacts
31
Real Case Study
Slowdown in a Major Financial Institution’s Middleware
http://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMTcvMDMvMTQvLS10aHJlYWREdW1wLTIudHh0LS0xMi0yOC0zMw==&s=t
32
OutOfMemoryError: Unable to create new native
threads
Causes
1. Threads are leaking
Solutions
1. Fix thread leak
2. Increase the Thread Limits Set
at Operating System(ulimit –u)
3. Kill other processes
4. Increase RAM capacity
5. Reduce Java Heap Size
6. Reduce thread stack size (-Xss).
Note: can cause
Artifacts
1. Thread Dump
Tools
1. FastThread
2. Text Editor
33
Direct buffer memory
Young Old
Metaspac
e
Threads JNI misc
GC
Code
Cach
e
Heap Memory Native Memory
JVM Memory
Becoming Pervasive in modern Spring Boot Frameworks
34
Demo
OutOfMemoryError: Direct buffer Memory
35
OutOfMemoryError: Direct buffer memory
Causes
1. Increase(or Leak) in direct buffers
usage
a. java.nio package
b. Moving from Spring
RestTemplate to WebClient
c. Image Processing Libraries
d. Networking Libraries
e. Some JDBC Drivers
Solutions
1. Fix Memory Leak code
2. Increase -
XX:MaxDirectMemorySize
3. Upgrade to Java 17 (There some
issues in Java 11)
Artifacts
1. App Log or Std error
2. Native Memory Tracking
Tools
36
Kill process or sacrifice child
JVM Kernel
X
1. GC Log
10. netstat
12. vmstat
2. Thread Dump
9. dmesg
3. Heap Dump
6. ps
8. Disk Usage
5. top 13. iostat
11. ping
14. Kernel Params
15. App Logs
16. metadata
4. Heap Substitute
7. top -H
37
Open-source script:
https://github.com/ycrash/yc-data-script
360° Troubleshooting artifacts
38
39
Real Case Study
Intermittent HTTP 502 errors in AWS EBS Service
EBS Architecture
40
Clue: Nginx Error
41
42
43
OutOfMemoryError: Kill process or sacrifice child
Causes
1. Lack of RAM capacity
2. High Memory utilization by
other processes in the
environment
3. Memory Leak in the code
Solutions
1. Kill other processes
2. Increase RAM capacity
3. Fix Memory Leak in the code
Artifacts
1. dmesg
2. GC Log
3. Heap Dump
4. Native Memory Tracking
Tools
1. HeapHero
2. Eclipse MAT
3. yCrash
44
Reason stack_trace_with_native_method
Young Old
Metaspac
e
Threads misc
GC
Direct
Buff
Code
Cach
e
Heap Memory Native Memory
JVM Memory
45
OutOfMemoryError: Reason stack_trace_with_native_method
Don’t worry, if you are not using JNI 
46
OutOfMemoryError: Reason stack_trace_with_native_method
Causes
1. Heavy Usage of Native
Methods
2. Recursive Native Method
Calls:
Solutions
1. Fix the problem in the native
app
Tools
1. OS native tools
a. Dtrace
b. pmap
c. pstack
Artifacts
1. Application Log or std error
Ram Lakshmanan ram@tier1app.com
@tier1app https://www.linkedin.com/company/ycrash
This deck will be published in:
https://blog.heaphero.io
If you want to learn more …
47
THANK YOU
FRIENDS

More Related Content

Similar to TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERROR

Eclipse Memory Analyzer
Eclipse Memory AnalyzerEclipse Memory Analyzer
Eclipse Memory Analyzer
nayashkova
 
[BGOUG] Java GC - Friend or Foe
[BGOUG] Java GC - Friend or Foe[BGOUG] Java GC - Friend or Foe
[BGOUG] Java GC - Friend or Foe
SAP HANA Cloud Platform
 
Efficient Memory and Thread Management in Highly Parallel Java Applications
Efficient Memory and Thread Management in Highly Parallel Java ApplicationsEfficient Memory and Thread Management in Highly Parallel Java Applications
Efficient Memory and Thread Management in Highly Parallel Java Applications
Phillip Koza
 
Java performance tuning
Java performance tuningJava performance tuning
Java performance tuning
Mohammed Fazuluddin
 
Troubleshooting XLConnect - OutofMemoryError(java) GC overhead limit exceeded
Troubleshooting XLConnect - OutofMemoryError(java) GC overhead limit exceededTroubleshooting XLConnect - OutofMemoryError(java) GC overhead limit exceeded
Troubleshooting XLConnect - OutofMemoryError(java) GC overhead limit exceeded
Rupak Roy
 
Eclipse Memory Analyzer - More Than Just a Heap Walker
Eclipse Memory Analyzer - More Than Just a Heap WalkerEclipse Memory Analyzer - More Than Just a Heap Walker
Eclipse Memory Analyzer - More Than Just a Heap Walker
guest62fd60c
 
It's always sunny with OpenJ9
It's always sunny with OpenJ9It's always sunny with OpenJ9
It's always sunny with OpenJ9
DanHeidinga
 
Don't dump thread dumps
Don't dump thread dumpsDon't dump thread dumps
Don't dump thread dumps
Tier1 App
 
Performance Tuning - Memory leaks, Thread deadlocks, JDK tools
Performance Tuning -  Memory leaks, Thread deadlocks, JDK toolsPerformance Tuning -  Memory leaks, Thread deadlocks, JDK tools
Performance Tuning - Memory leaks, Thread deadlocks, JDK tools
Haribabu Nandyal Padmanaban
 
java-monitoring-troubleshooting
java-monitoring-troubleshootingjava-monitoring-troubleshooting
java-monitoring-troubleshooting
William Au
 
Devoxx Fr 2022 - Remèdes aux oomkill, warm-ups, et lenteurs pour des conteneu...
Devoxx Fr 2022 - Remèdes aux oomkill, warm-ups, et lenteurs pour des conteneu...Devoxx Fr 2022 - Remèdes aux oomkill, warm-ups, et lenteurs pour des conteneu...
Devoxx Fr 2022 - Remèdes aux oomkill, warm-ups, et lenteurs pour des conteneu...
Jean-Philippe BEMPEL
 
Java и Linux — особенности эксплуатации / Алексей Рагозин (Дойче Банк)
Java и Linux — особенности эксплуатации / Алексей Рагозин (Дойче Банк)Java и Linux — особенности эксплуатации / Алексей Рагозин (Дойче Банк)
Java и Linux — особенности эксплуатации / Алексей Рагозин (Дойче Банк)
Ontico
 
predicting-m3-devopsconMunich-2023-v2.pptx
predicting-m3-devopsconMunich-2023-v2.pptxpredicting-m3-devopsconMunich-2023-v2.pptx
predicting-m3-devopsconMunich-2023-v2.pptx
Tier1 app
 
Web Sphere Problem Determination Ext
Web Sphere Problem Determination ExtWeb Sphere Problem Determination Ext
Web Sphere Problem Determination Ext
Rohit Kelapure
 
Mastering java in containers - MadridJUG
Mastering java in containers - MadridJUGMastering java in containers - MadridJUG
Mastering java in containers - MadridJUG
Jorge Morales
 
Rapid Application Development with Cocoon
Rapid Application Development with CocoonRapid Application Development with Cocoon
Rapid Application Development with Cocoon
tcurdt
 
Java on Linux for devs and ops
Java on Linux for devs and opsJava on Linux for devs and ops
Java on Linux for devs and ops
aragozin
 
JBoss Enterprise Application Platform 6 Troubleshooting
JBoss Enterprise Application Platform 6 TroubleshootingJBoss Enterprise Application Platform 6 Troubleshooting
JBoss Enterprise Application Platform 6 Troubleshooting
Alexandre Cavalcanti
 
Cache is King
Cache is KingCache is King
Cache is King
Steve Souders
 
What to do if Your Kafka Streams App Gets OOMKilled? with Andrey Serebryanskiy
What to do if Your Kafka Streams App Gets OOMKilled? with Andrey SerebryanskiyWhat to do if Your Kafka Streams App Gets OOMKilled? with Andrey Serebryanskiy
What to do if Your Kafka Streams App Gets OOMKilled? with Andrey Serebryanskiy
HostedbyConfluent
 

Similar to TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERROR (20)

Eclipse Memory Analyzer
Eclipse Memory AnalyzerEclipse Memory Analyzer
Eclipse Memory Analyzer
 
[BGOUG] Java GC - Friend or Foe
[BGOUG] Java GC - Friend or Foe[BGOUG] Java GC - Friend or Foe
[BGOUG] Java GC - Friend or Foe
 
Efficient Memory and Thread Management in Highly Parallel Java Applications
Efficient Memory and Thread Management in Highly Parallel Java ApplicationsEfficient Memory and Thread Management in Highly Parallel Java Applications
Efficient Memory and Thread Management in Highly Parallel Java Applications
 
Java performance tuning
Java performance tuningJava performance tuning
Java performance tuning
 
Troubleshooting XLConnect - OutofMemoryError(java) GC overhead limit exceeded
Troubleshooting XLConnect - OutofMemoryError(java) GC overhead limit exceededTroubleshooting XLConnect - OutofMemoryError(java) GC overhead limit exceeded
Troubleshooting XLConnect - OutofMemoryError(java) GC overhead limit exceeded
 
Eclipse Memory Analyzer - More Than Just a Heap Walker
Eclipse Memory Analyzer - More Than Just a Heap WalkerEclipse Memory Analyzer - More Than Just a Heap Walker
Eclipse Memory Analyzer - More Than Just a Heap Walker
 
It's always sunny with OpenJ9
It's always sunny with OpenJ9It's always sunny with OpenJ9
It's always sunny with OpenJ9
 
Don't dump thread dumps
Don't dump thread dumpsDon't dump thread dumps
Don't dump thread dumps
 
Performance Tuning - Memory leaks, Thread deadlocks, JDK tools
Performance Tuning -  Memory leaks, Thread deadlocks, JDK toolsPerformance Tuning -  Memory leaks, Thread deadlocks, JDK tools
Performance Tuning - Memory leaks, Thread deadlocks, JDK tools
 
java-monitoring-troubleshooting
java-monitoring-troubleshootingjava-monitoring-troubleshooting
java-monitoring-troubleshooting
 
Devoxx Fr 2022 - Remèdes aux oomkill, warm-ups, et lenteurs pour des conteneu...
Devoxx Fr 2022 - Remèdes aux oomkill, warm-ups, et lenteurs pour des conteneu...Devoxx Fr 2022 - Remèdes aux oomkill, warm-ups, et lenteurs pour des conteneu...
Devoxx Fr 2022 - Remèdes aux oomkill, warm-ups, et lenteurs pour des conteneu...
 
Java и Linux — особенности эксплуатации / Алексей Рагозин (Дойче Банк)
Java и Linux — особенности эксплуатации / Алексей Рагозин (Дойче Банк)Java и Linux — особенности эксплуатации / Алексей Рагозин (Дойче Банк)
Java и Linux — особенности эксплуатации / Алексей Рагозин (Дойче Банк)
 
predicting-m3-devopsconMunich-2023-v2.pptx
predicting-m3-devopsconMunich-2023-v2.pptxpredicting-m3-devopsconMunich-2023-v2.pptx
predicting-m3-devopsconMunich-2023-v2.pptx
 
Web Sphere Problem Determination Ext
Web Sphere Problem Determination ExtWeb Sphere Problem Determination Ext
Web Sphere Problem Determination Ext
 
Mastering java in containers - MadridJUG
Mastering java in containers - MadridJUGMastering java in containers - MadridJUG
Mastering java in containers - MadridJUG
 
Rapid Application Development with Cocoon
Rapid Application Development with CocoonRapid Application Development with Cocoon
Rapid Application Development with Cocoon
 
Java on Linux for devs and ops
Java on Linux for devs and opsJava on Linux for devs and ops
Java on Linux for devs and ops
 
JBoss Enterprise Application Platform 6 Troubleshooting
JBoss Enterprise Application Platform 6 TroubleshootingJBoss Enterprise Application Platform 6 Troubleshooting
JBoss Enterprise Application Platform 6 Troubleshooting
 
Cache is King
Cache is KingCache is King
Cache is King
 
What to do if Your Kafka Streams App Gets OOMKilled? with Andrey Serebryanskiy
What to do if Your Kafka Streams App Gets OOMKilled? with Andrey SerebryanskiyWhat to do if Your Kafka Streams App Gets OOMKilled? with Andrey Serebryanskiy
What to do if Your Kafka Streams App Gets OOMKilled? with Andrey Serebryanskiy
 

More from Tier1 app

Top-5-java-perf-problems-jax_mainz_2024.pptx
Top-5-java-perf-problems-jax_mainz_2024.pptxTop-5-java-perf-problems-jax_mainz_2024.pptx
Top-5-java-perf-problems-jax_mainz_2024.pptx
Tier1 app
 
KnowAPIs-UnknownPerf-jaxMainz-2024 (1).pptx
KnowAPIs-UnknownPerf-jaxMainz-2024 (1).pptxKnowAPIs-UnknownPerf-jaxMainz-2024 (1).pptx
KnowAPIs-UnknownPerf-jaxMainz-2024 (1).pptx
Tier1 app
 
Top-5-production-devconMunich-2023-v2.pptx
Top-5-production-devconMunich-2023-v2.pptxTop-5-production-devconMunich-2023-v2.pptx
Top-5-production-devconMunich-2023-v2.pptx
Tier1 app
 
Top-5-production-devconMunich-2023.pptx
Top-5-production-devconMunich-2023.pptxTop-5-production-devconMunich-2023.pptx
Top-5-production-devconMunich-2023.pptx
Tier1 app
 
predicting-m3-devopsconMunich-2023.pptx
predicting-m3-devopsconMunich-2023.pptxpredicting-m3-devopsconMunich-2023.pptx
predicting-m3-devopsconMunich-2023.pptx
Tier1 app
 
Predicting Production Outages: Unleashing the Power of Micro-Metrics – ADDO C...
Predicting Production Outages: Unleashing the Power of Micro-Metrics – ADDO C...Predicting Production Outages: Unleashing the Power of Micro-Metrics – ADDO C...
Predicting Production Outages: Unleashing the Power of Micro-Metrics – ADDO C...
Tier1 app
 
7-JVM-arguments-JaxLondon-2023.pptx
7-JVM-arguments-JaxLondon-2023.pptx7-JVM-arguments-JaxLondon-2023.pptx
7-JVM-arguments-JaxLondon-2023.pptx
Tier1 app
 
Top-5-Performance-JaxLondon-2023.pptx
Top-5-Performance-JaxLondon-2023.pptxTop-5-Performance-JaxLondon-2023.pptx
Top-5-Performance-JaxLondon-2023.pptx
Tier1 app
 
16 ARTIFACTS TO CAPTURE WHEN YOUR CONTAINER APPLICATION IS IN TROUBLE
16 ARTIFACTS TO CAPTURE WHEN YOUR CONTAINER APPLICATION IS IN TROUBLE16 ARTIFACTS TO CAPTURE WHEN YOUR CONTAINER APPLICATION IS IN TROUBLE
16 ARTIFACTS TO CAPTURE WHEN YOUR CONTAINER APPLICATION IS IN TROUBLE
Tier1 app
 
MAJOR OUTAGES IN MAJOR ENTERPRISES
MAJOR OUTAGES IN MAJOR ENTERPRISESMAJOR OUTAGES IN MAJOR ENTERPRISES
MAJOR OUTAGES IN MAJOR ENTERPRISES
Tier1 app
 
KnowAPIs-UnknownPerf-confoo-2023 (1).pptx
KnowAPIs-UnknownPerf-confoo-2023 (1).pptxKnowAPIs-UnknownPerf-confoo-2023 (1).pptx
KnowAPIs-UnknownPerf-confoo-2023 (1).pptx
Tier1 app
 
memory-patterns-confoo-2023.pptx
memory-patterns-confoo-2023.pptxmemory-patterns-confoo-2023.pptx
memory-patterns-confoo-2023.pptx
Tier1 app
 
this-is-garbage-talk-2022.pptx
this-is-garbage-talk-2022.pptxthis-is-garbage-talk-2022.pptx
this-is-garbage-talk-2022.pptx
Tier1 app
 
millions-gc-jax-2022.pptx
millions-gc-jax-2022.pptxmillions-gc-jax-2022.pptx
millions-gc-jax-2022.pptx
Tier1 app
 
lets-crash-apps-jax-2022.pptx
lets-crash-apps-jax-2022.pptxlets-crash-apps-jax-2022.pptx
lets-crash-apps-jax-2022.pptx
Tier1 app
 
‘16 artifacts’ to capture when there is a production problem
‘16 artifacts’ to capture when there is a production problem‘16 artifacts’ to capture when there is a production problem
‘16 artifacts’ to capture when there is a production problem
Tier1 app
 
Lets crash-applications
Lets crash-applicationsLets crash-applications
Lets crash-applications
Tier1 app
 
16 artifacts to capture when there is a production problem
16 artifacts to capture when there is a production problem16 artifacts to capture when there is a production problem
16 artifacts to capture when there is a production problem
Tier1 app
 
Lets crash-applications
Lets crash-applicationsLets crash-applications
Lets crash-applications
Tier1 app
 
7 habits of highly effective Performance Troubleshooters
7 habits of highly effective Performance Troubleshooters7 habits of highly effective Performance Troubleshooters
7 habits of highly effective Performance Troubleshooters
Tier1 app
 

More from Tier1 app (20)

Top-5-java-perf-problems-jax_mainz_2024.pptx
Top-5-java-perf-problems-jax_mainz_2024.pptxTop-5-java-perf-problems-jax_mainz_2024.pptx
Top-5-java-perf-problems-jax_mainz_2024.pptx
 
KnowAPIs-UnknownPerf-jaxMainz-2024 (1).pptx
KnowAPIs-UnknownPerf-jaxMainz-2024 (1).pptxKnowAPIs-UnknownPerf-jaxMainz-2024 (1).pptx
KnowAPIs-UnknownPerf-jaxMainz-2024 (1).pptx
 
Top-5-production-devconMunich-2023-v2.pptx
Top-5-production-devconMunich-2023-v2.pptxTop-5-production-devconMunich-2023-v2.pptx
Top-5-production-devconMunich-2023-v2.pptx
 
Top-5-production-devconMunich-2023.pptx
Top-5-production-devconMunich-2023.pptxTop-5-production-devconMunich-2023.pptx
Top-5-production-devconMunich-2023.pptx
 
predicting-m3-devopsconMunich-2023.pptx
predicting-m3-devopsconMunich-2023.pptxpredicting-m3-devopsconMunich-2023.pptx
predicting-m3-devopsconMunich-2023.pptx
 
Predicting Production Outages: Unleashing the Power of Micro-Metrics – ADDO C...
Predicting Production Outages: Unleashing the Power of Micro-Metrics – ADDO C...Predicting Production Outages: Unleashing the Power of Micro-Metrics – ADDO C...
Predicting Production Outages: Unleashing the Power of Micro-Metrics – ADDO C...
 
7-JVM-arguments-JaxLondon-2023.pptx
7-JVM-arguments-JaxLondon-2023.pptx7-JVM-arguments-JaxLondon-2023.pptx
7-JVM-arguments-JaxLondon-2023.pptx
 
Top-5-Performance-JaxLondon-2023.pptx
Top-5-Performance-JaxLondon-2023.pptxTop-5-Performance-JaxLondon-2023.pptx
Top-5-Performance-JaxLondon-2023.pptx
 
16 ARTIFACTS TO CAPTURE WHEN YOUR CONTAINER APPLICATION IS IN TROUBLE
16 ARTIFACTS TO CAPTURE WHEN YOUR CONTAINER APPLICATION IS IN TROUBLE16 ARTIFACTS TO CAPTURE WHEN YOUR CONTAINER APPLICATION IS IN TROUBLE
16 ARTIFACTS TO CAPTURE WHEN YOUR CONTAINER APPLICATION IS IN TROUBLE
 
MAJOR OUTAGES IN MAJOR ENTERPRISES
MAJOR OUTAGES IN MAJOR ENTERPRISESMAJOR OUTAGES IN MAJOR ENTERPRISES
MAJOR OUTAGES IN MAJOR ENTERPRISES
 
KnowAPIs-UnknownPerf-confoo-2023 (1).pptx
KnowAPIs-UnknownPerf-confoo-2023 (1).pptxKnowAPIs-UnknownPerf-confoo-2023 (1).pptx
KnowAPIs-UnknownPerf-confoo-2023 (1).pptx
 
memory-patterns-confoo-2023.pptx
memory-patterns-confoo-2023.pptxmemory-patterns-confoo-2023.pptx
memory-patterns-confoo-2023.pptx
 
this-is-garbage-talk-2022.pptx
this-is-garbage-talk-2022.pptxthis-is-garbage-talk-2022.pptx
this-is-garbage-talk-2022.pptx
 
millions-gc-jax-2022.pptx
millions-gc-jax-2022.pptxmillions-gc-jax-2022.pptx
millions-gc-jax-2022.pptx
 
lets-crash-apps-jax-2022.pptx
lets-crash-apps-jax-2022.pptxlets-crash-apps-jax-2022.pptx
lets-crash-apps-jax-2022.pptx
 
‘16 artifacts’ to capture when there is a production problem
‘16 artifacts’ to capture when there is a production problem‘16 artifacts’ to capture when there is a production problem
‘16 artifacts’ to capture when there is a production problem
 
Lets crash-applications
Lets crash-applicationsLets crash-applications
Lets crash-applications
 
16 artifacts to capture when there is a production problem
16 artifacts to capture when there is a production problem16 artifacts to capture when there is a production problem
16 artifacts to capture when there is a production problem
 
Lets crash-applications
Lets crash-applicationsLets crash-applications
Lets crash-applications
 
7 habits of highly effective Performance Troubleshooters
7 habits of highly effective Performance Troubleshooters7 habits of highly effective Performance Troubleshooters
7 habits of highly effective Performance Troubleshooters
 

Recently uploaded

Fundamentals of Programming and Language Processors
Fundamentals of Programming and Language ProcessorsFundamentals of Programming and Language Processors
Fundamentals of Programming and Language Processors
Rakesh Kumar R
 
Oracle Database 19c New Features for DBAs and Developers.pptx
Oracle Database 19c New Features for DBAs and Developers.pptxOracle Database 19c New Features for DBAs and Developers.pptx
Oracle Database 19c New Features for DBAs and Developers.pptx
Remote DBA Services
 
openEuler Case Study - The Journey to Supply Chain Security
openEuler Case Study - The Journey to Supply Chain SecurityopenEuler Case Study - The Journey to Supply Chain Security
openEuler Case Study - The Journey to Supply Chain Security
Shane Coughlan
 
Neo4j - Product Vision and Knowledge Graphs - GraphSummit Paris
Neo4j - Product Vision and Knowledge Graphs - GraphSummit ParisNeo4j - Product Vision and Knowledge Graphs - GraphSummit Paris
Neo4j - Product Vision and Knowledge Graphs - GraphSummit Paris
Neo4j
 
DDS-Security 1.2 - What's New? Stronger security for long-running systems
DDS-Security 1.2 - What's New? Stronger security for long-running systemsDDS-Security 1.2 - What's New? Stronger security for long-running systems
DDS-Security 1.2 - What's New? Stronger security for long-running systems
Gerardo Pardo-Castellote
 
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian CompaniesE-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
Quickdice ERP
 
How to write a program in any programming language
How to write a program in any programming languageHow to write a program in any programming language
How to write a program in any programming language
Rakesh Kumar R
 
Why Choose Odoo 17 Community & How it differs from Odoo 17 Enterprise Edition
Why Choose Odoo 17 Community & How it differs from Odoo 17 Enterprise EditionWhy Choose Odoo 17 Community & How it differs from Odoo 17 Enterprise Edition
Why Choose Odoo 17 Community & How it differs from Odoo 17 Enterprise Edition
Envertis Software Solutions
 
Introducing Crescat - Event Management Software for Venues, Festivals and Eve...
Introducing Crescat - Event Management Software for Venues, Festivals and Eve...Introducing Crescat - Event Management Software for Venues, Festivals and Eve...
Introducing Crescat - Event Management Software for Venues, Festivals and Eve...
Crescat
 
Oracle 23c New Features For DBAs and Developers.pptx
Oracle 23c New Features For DBAs and Developers.pptxOracle 23c New Features For DBAs and Developers.pptx
Oracle 23c New Features For DBAs and Developers.pptx
Remote DBA Services
 
Empowering Growth with Best Software Development Company in Noida - Deuglo
Empowering Growth with Best Software  Development Company in Noida - DeugloEmpowering Growth with Best Software  Development Company in Noida - Deuglo
Empowering Growth with Best Software Development Company in Noida - Deuglo
Deuglo Infosystem Pvt Ltd
 
Unveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdfUnveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdf
brainerhub1
 
Microservice Teams - How the cloud changes the way we work
Microservice Teams - How the cloud changes the way we workMicroservice Teams - How the cloud changes the way we work
Microservice Teams - How the cloud changes the way we work
Sven Peters
 
原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样
原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样
原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样
mz5nrf0n
 
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI AppAI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App
Google
 
SMS API Integration in Saudi Arabia| Best SMS API Service
SMS API Integration in Saudi Arabia| Best SMS API ServiceSMS API Integration in Saudi Arabia| Best SMS API Service
SMS API Integration in Saudi Arabia| Best SMS API Service
Yara Milbes
 
E-commerce Application Development Company.pdf
E-commerce Application Development Company.pdfE-commerce Application Development Company.pdf
E-commerce Application Development Company.pdf
Hornet Dynamics
 
Need for Speed: Removing speed bumps from your Symfony projects ⚡️
Need for Speed: Removing speed bumps from your Symfony projects ⚡️Need for Speed: Removing speed bumps from your Symfony projects ⚡️
Need for Speed: Removing speed bumps from your Symfony projects ⚡️
Łukasz Chruściel
 
Artificia Intellicence and XPath Extension Functions
Artificia Intellicence and XPath Extension FunctionsArtificia Intellicence and XPath Extension Functions
Artificia Intellicence and XPath Extension Functions
Octavian Nadolu
 
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CDKuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
rodomar2
 

Recently uploaded (20)

Fundamentals of Programming and Language Processors
Fundamentals of Programming and Language ProcessorsFundamentals of Programming and Language Processors
Fundamentals of Programming and Language Processors
 
Oracle Database 19c New Features for DBAs and Developers.pptx
Oracle Database 19c New Features for DBAs and Developers.pptxOracle Database 19c New Features for DBAs and Developers.pptx
Oracle Database 19c New Features for DBAs and Developers.pptx
 
openEuler Case Study - The Journey to Supply Chain Security
openEuler Case Study - The Journey to Supply Chain SecurityopenEuler Case Study - The Journey to Supply Chain Security
openEuler Case Study - The Journey to Supply Chain Security
 
Neo4j - Product Vision and Knowledge Graphs - GraphSummit Paris
Neo4j - Product Vision and Knowledge Graphs - GraphSummit ParisNeo4j - Product Vision and Knowledge Graphs - GraphSummit Paris
Neo4j - Product Vision and Knowledge Graphs - GraphSummit Paris
 
DDS-Security 1.2 - What's New? Stronger security for long-running systems
DDS-Security 1.2 - What's New? Stronger security for long-running systemsDDS-Security 1.2 - What's New? Stronger security for long-running systems
DDS-Security 1.2 - What's New? Stronger security for long-running systems
 
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian CompaniesE-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
 
How to write a program in any programming language
How to write a program in any programming languageHow to write a program in any programming language
How to write a program in any programming language
 
Why Choose Odoo 17 Community & How it differs from Odoo 17 Enterprise Edition
Why Choose Odoo 17 Community & How it differs from Odoo 17 Enterprise EditionWhy Choose Odoo 17 Community & How it differs from Odoo 17 Enterprise Edition
Why Choose Odoo 17 Community & How it differs from Odoo 17 Enterprise Edition
 
Introducing Crescat - Event Management Software for Venues, Festivals and Eve...
Introducing Crescat - Event Management Software for Venues, Festivals and Eve...Introducing Crescat - Event Management Software for Venues, Festivals and Eve...
Introducing Crescat - Event Management Software for Venues, Festivals and Eve...
 
Oracle 23c New Features For DBAs and Developers.pptx
Oracle 23c New Features For DBAs and Developers.pptxOracle 23c New Features For DBAs and Developers.pptx
Oracle 23c New Features For DBAs and Developers.pptx
 
Empowering Growth with Best Software Development Company in Noida - Deuglo
Empowering Growth with Best Software  Development Company in Noida - DeugloEmpowering Growth with Best Software  Development Company in Noida - Deuglo
Empowering Growth with Best Software Development Company in Noida - Deuglo
 
Unveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdfUnveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdf
 
Microservice Teams - How the cloud changes the way we work
Microservice Teams - How the cloud changes the way we workMicroservice Teams - How the cloud changes the way we work
Microservice Teams - How the cloud changes the way we work
 
原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样
原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样
原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样
 
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI AppAI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App
AI Fusion Buddy Review: Brand New, Groundbreaking Gemini-Powered AI App
 
SMS API Integration in Saudi Arabia| Best SMS API Service
SMS API Integration in Saudi Arabia| Best SMS API ServiceSMS API Integration in Saudi Arabia| Best SMS API Service
SMS API Integration in Saudi Arabia| Best SMS API Service
 
E-commerce Application Development Company.pdf
E-commerce Application Development Company.pdfE-commerce Application Development Company.pdf
E-commerce Application Development Company.pdf
 
Need for Speed: Removing speed bumps from your Symfony projects ⚡️
Need for Speed: Removing speed bumps from your Symfony projects ⚡️Need for Speed: Removing speed bumps from your Symfony projects ⚡️
Need for Speed: Removing speed bumps from your Symfony projects ⚡️
 
Artificia Intellicence and XPath Extension Functions
Artificia Intellicence and XPath Extension FunctionsArtificia Intellicence and XPath Extension Functions
Artificia Intellicence and XPath Extension Functions
 
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CDKuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
 

TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERROR

  • 1. 1
  • 2. Does Java process memory utilization go beyond –Xmx? 2 –XX:MaxMetaspaceSize -Xmx Young Old Metaspac e Threads JNI misc GC Direct Buff Code Cach e Heap Memory Native Memory Java Process Memory
  • 4. 4 How to know the type of OutOfMemoryError? java.lang.OutOfMemoryError: <type>
  • 5. 5 Java Heap Space Metaspac e Threads JNI misc GC Direct Buff Code Cach e Heap Memory Native Memory JVM Memory Most common type
  • 6. 6 GC Behavior of a Healthy Application - Full Garbage Collection Event
  • 7. 7 GC Behavior of Acute Memory Leak - Full Garbage Collection Event
  • 8. 8 GC Behavior when there is a Memory Leak - Full Garbage Collection Event
  • 9. 9 GC Behavior when App needs more memory - Full Garbage Collection Event
  • 10. 10 How to study GC Behavior? -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:<file-path> Till Java 8 -Xlog:gc*:file=<file-path> From Java 9 Enable GC logs (always) Almost zero overhead
  • 12. 12 How to capture Heap Dumps? 8 options: https://blog.heaphero.io/2017/10/13/how-to-capture-java-heap-dumps-7-options/
  • 13. 1. GC Log 10. netstat 12. vmstat 2. Thread Dump 9. dmesg 3. Heap Dump 6. ps 8. Disk Usage 5. top 13. iostat 11. ping 14. Kernel Params 15. App Logs 16. metadata 4. Heap Substitute 7. top -H 13 Open-source script: https://github.com/ycrash/yc-data-script 360° Troubleshooting artifacts
  • 15. 15 OutOfMemoryError: Java Heap Space Causes 1. Increase in Traffic Volume 2. Memory leak due to buggy code Solutions 1. Fix Memory Leak in the code 2. Increase heap size (-Xmx) Artifacts 1. Garbage Collection Log 2. Heap Dump Tools 1. GCeasy (GC log analysis) 2. HeapHero 3. Eclipse MAT 4. JVisualVM
  • 16. 16 GC overhead limit exceeded Metaspac e Threads JNI misc GC Direct Buff Code Cach e Heap Memory Native Memory JVM Memory
  • 17. 17 OutOfMemoryError: GC overhead limit exceeded Diagnosis: Same as ‘Java Heap Space’  Java process is spending more than 98% of its time doing garbage collection and recovering less than 2% of the heap and has been doing so far the last 5 consecutive garbage collections
  • 18. 18 Requested array size exceeds VM limit Young Old Metaspac e Threads JNI misc GC Direct Buff Code Cach e Heap Memory Native Memory JVM Memory
  • 20. 20 OutOfMemoryError: Requested array size exceeds VM limit Causes 1. Allocated array larger than heap size Solutions 1. Fix code which allocates large array size 2. Increase heap size (-Xmx) Artifacts 1. Application Log or std error Tools
  • 21. 21 Metaspace Young Old Threads JNI misc GC Direct Buff Code Cach e Heap Memory Native Memory JVM Memory
  • 23. 23 Metaspace Memory Leak - Full Garbage Collection Event
  • 24. 24 Metaspace Analysis 1. Java 8 & below: java {app_name} -verbose:class 2. Java 9 & above: java {app_name} -Xlog:class+load=info:<filename>
  • 25. 25 OutOfMemoryError: Metaspace Causes 1. Creating large number of Dynamic classes (Java Reflection, Groovy type of scripting languages) 2. Loading large number of Classes (3rd party libraries/frameworks) 3. Loading large number of Classloaders Solutions 1. Fix Memory Leak code 2. Increase -XX:MetaspaceSize and -XX:MaxMetaspaceSize. Artifacts 1. Java 8 & below: java {app_name} -verbose:class 2. Java 9 & above: java {app_name} - Xlog:class+load=info:<filena me> 3. jcmd {pid} GC.class_histogram 4. Heap Dump Tools 1. GCeasy (GC log analysis) 2. HeapHero 3. Eclipse MAT
  • 26. 26 Permgen space Young Old Threads JNI misc GC Direct Buff Code Cach e Heap Memory Native Memory JVM Memory Happens only in Java 7 & below
  • 27. 27 OutOfMemoryError: Permgen Space Diagnosis: Same as ‘Metaspace’ 
  • 28. 28 Unable to create new native threads Young Old Metaspac e JNI misc GC Direct Buff Code Cach e Heap Memory Native Memory JVM Memory
  • 29. 29 How to capture Thread Dump? 9 options:https://blog.fastthread.io/how-to-take-thread-dumps-7-options/
  • 30. 1. GC Log 10. netstat 12. vmstat 2. Thread Dump 9. dmesg 3. Heap Dump 6. ps 8. Disk Usage 5. top 13. iostat 11. ping 14. Kernel Params 15. App Logs 16. metadata 4. Heap Substitute 7. top -H 30 Open-source script: https://github.com/ycrash/yc-data-script 360° Troubleshooting artifacts
  • 31. 31 Real Case Study Slowdown in a Major Financial Institution’s Middleware http://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMTcvMDMvMTQvLS10aHJlYWREdW1wLTIudHh0LS0xMi0yOC0zMw==&s=t
  • 32. 32 OutOfMemoryError: Unable to create new native threads Causes 1. Threads are leaking Solutions 1. Fix thread leak 2. Increase the Thread Limits Set at Operating System(ulimit –u) 3. Kill other processes 4. Increase RAM capacity 5. Reduce Java Heap Size 6. Reduce thread stack size (-Xss). Note: can cause Artifacts 1. Thread Dump Tools 1. FastThread 2. Text Editor
  • 33. 33 Direct buffer memory Young Old Metaspac e Threads JNI misc GC Code Cach e Heap Memory Native Memory JVM Memory Becoming Pervasive in modern Spring Boot Frameworks
  • 35. 35 OutOfMemoryError: Direct buffer memory Causes 1. Increase(or Leak) in direct buffers usage a. java.nio package b. Moving from Spring RestTemplate to WebClient c. Image Processing Libraries d. Networking Libraries e. Some JDBC Drivers Solutions 1. Fix Memory Leak code 2. Increase - XX:MaxDirectMemorySize 3. Upgrade to Java 17 (There some issues in Java 11) Artifacts 1. App Log or Std error 2. Native Memory Tracking Tools
  • 36. 36 Kill process or sacrifice child JVM Kernel X
  • 37. 1. GC Log 10. netstat 12. vmstat 2. Thread Dump 9. dmesg 3. Heap Dump 6. ps 8. Disk Usage 5. top 13. iostat 11. ping 14. Kernel Params 15. App Logs 16. metadata 4. Heap Substitute 7. top -H 37 Open-source script: https://github.com/ycrash/yc-data-script 360° Troubleshooting artifacts
  • 38. 38
  • 39. 39 Real Case Study Intermittent HTTP 502 errors in AWS EBS Service
  • 42. 42
  • 43. 43 OutOfMemoryError: Kill process or sacrifice child Causes 1. Lack of RAM capacity 2. High Memory utilization by other processes in the environment 3. Memory Leak in the code Solutions 1. Kill other processes 2. Increase RAM capacity 3. Fix Memory Leak in the code Artifacts 1. dmesg 2. GC Log 3. Heap Dump 4. Native Memory Tracking Tools 1. HeapHero 2. Eclipse MAT 3. yCrash
  • 44. 44 Reason stack_trace_with_native_method Young Old Metaspac e Threads misc GC Direct Buff Code Cach e Heap Memory Native Memory JVM Memory
  • 46. 46 OutOfMemoryError: Reason stack_trace_with_native_method Causes 1. Heavy Usage of Native Methods 2. Recursive Native Method Calls: Solutions 1. Fix the problem in the native app Tools 1. OS native tools a. Dtrace b. pmap c. pstack Artifacts 1. Application Log or std error
  • 47. Ram Lakshmanan ram@tier1app.com @tier1app https://www.linkedin.com/company/ycrash This deck will be published in: https://blog.heaphero.io If you want to learn more … 47 THANK YOU FRIENDS

Editor's Notes

  1. Eclipse: MapManager, LaunchPad_oom
  2. Gceasy - C:\workspace\ycrash\speeches\9-types-oom\gc-memory-leak.zip
  3. https://blog.heaphero.io/2017/10/13/how-to-capture-java-heap-dumps-7-options/
  4. http://localhost:8080/heap-index.jsp C:\workspace\tmp\dev-nexus\dump-2024-04-09.hprof http://localhost:8080/heap-report-wc.jsp?p=ZDI0ZHIxRGJtaUhHQXN3eUFWa3htQW9QeERicTZpZjNUczlXTFVVTE9jcWpMeHBsRkpSZEV4Vkk5cDZqZHJIK2NzditQeWx0UDZDbUhmNEIySTBZMkZ6NUhjMkh3SFdOSUdPZU5WTHZHZlk2dEdnQnNCMmp2MkltMEdoUnFxcUY=
  5. OOMRequestedArraySizeExceedsVMLimit
  6. OOMRequestedArraySizeExceedsVMLimit
  7. OOMMetaspace OOMMetaspace_verbose
  8. https://gceasy.io/my-gc-report.jsp?p=c2hhcmVkLzIwMjIvMDcvMTYvbWV0YXNwYWNlLW9vbWVycm9yLmdjLS0xOC0xNy0zMg==&s=t&channel=WEB
  9. OOMMetaspace_verbose
  10. OOMMetaspace_verbose
  11. https://blog.fastthread.io/how-to-take-thread-dumps-7-options/
  12. C:\workspace\ycrash\speeches\9-types-oom\thread-leak.txt Backup 1: http://localhost:8080/my-thread-report.jsp?p=Qzpcd29ya3NwYWNlXHRtcFx5Yy0yXzEyLWJlcmxpblxzaGFyZWRcMjAyNC01LTI1XHRocmVhZC1sZWFrLnR4dC0xNy0xMy01Mjs7& Backup 2: http://localhost:8080/yc-report.jsp?ou=SAP&de=127.0.0.1&app=yc&ts=2024-04-09T11-09-51
  13. OOMDirectBuffer
  14. https://test.ycrash.io/yc-report-kernel.jsp?ou=czlWbG0rUko0UXAxazlSbjZrSUIwUT09&de=172.31.7.106&app=yc&ts=2023-09-01T10-25-39
  15. https://test.ycrash.io/yc-report-kernel.jsp?ou=czlWbG0rUko0UXAxazlSbjZrSUIwUT09&de=172.31.7.106&app=yc&ts=2023-09-01T10-25-39