SlideShare a Scribd company logo
1 of 39
1
Does Java process memory utilization go beyond –Xmx?
2
–XX:MaxMetaspaceSize
-Xmx
Young Old
Metaspac
e
Threads JNI misc
GC
Direct
Buff
Code
Cach
e
Heap Memory Native Memory
Java Process Memory
3
Demo
OutOfMemoryError
4
How to know the type of OutOfMemoryError?
java.lang.OutOfMemoryError: <type>
5
Java Heap Space
Metaspac
e
Threads JNI misc
GC
Direct
Buff
Code
Cach
e
Heap Memory Native Memory
JVM Memory
Most common type
6
GC Behavior of a Healthy Application
- Full Garbage Collection Event
7
GC Behavior of Acute Memory Leak
- Full Garbage Collection Event
8
GC Behavior when there is a Memory Leak
- Full Garbage Collection Event
9
How to study GC Behavior?
-XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:<file-path>
Till Java 8
-Xlog:gc*:file=<file-path>
From Java 9
Enable GC logs (always)
Almost zero overhead
10
GCeasy
GC Log Analysis Demo
11
How to capture Heap Dumps?
8 options: https://blog.heaphero.io/2017/10/13/how-to-capture-java-heap-dumps-7-options/
1. GC Log
10. netstat
12. vmstat
2. Thread Dump
9. dmesg
3. Heap Dump
6. ps
8. Disk Usage
5. top 13. iostat
11. ping
14. Kernel Params
15. App Logs
16. metadata
4. Heap Substitute
7. top -H
12
Open-source script:
https://github.com/ycrash/yc-data-script
360° Troubleshooting artifacts
13
HeapHero
Heap Dump Analysis Demo
14
OutOfMemoryError: Java Heap Space
Causes
1. Increase in Traffic Volume
2. Memory leak due to
buggy code
Solutions
1. Fix Memory Leak in the
code
2. Increase heap size (-Xmx)
Artifacts
1. Garbage Collection Log
2. Heap Dump
Tools
1. GCeasy (GC log
analysis)
2. HeapHero
3. Eclipse MAT
4. JProfiler
15
GC overhead limit exceeded
Metaspac
e
Threads JNI misc
GC
Direct
Buff
Code
Cach
e
Heap Memory Native Memory
JVM Memory
16
OutOfMemoryError: GC overhead limit exceeded
Diagnosis: Same as ‘Java Heap Space’ 
17
Requested array size exceeds VM limit
Young Old
Metaspac
e
Threads JNI misc
GC
Direct
Buff
Code
Cach
e
Heap Memory Native Memory
JVM Memory
18
Demo
OutOfMemoryError: Requested array size exceeds VM
limit
19
OutOfMemoryError: Requested array size exceeds VM limit
Causes
1. Allocated array larger than
heap size
Solutions
1. Fix Memory Leak code
2. Increase heap size (-Xmx)
Artifacts
1. Application Log or std error
Tools
20
Metaspace
Young Old Threads JNI misc
GC
Direct
Buff
Code
Cach
e
Heap Memory Native Memory
JVM Memory
21
Demo
OutOfMemoryError: Metaspace
22
Metaspace Memory Leak
- Full Garbage Collection Event
23
OutOfMemoryError: Metaspace
Causes
1. Creating large number of
Dynamic classes (Java
Reflection, Groovy type of script
languages)
2. Loading large number of Classes
3. Loading large number of
Classloaders
Solutions
1. Fix Memory Leak code
2. Increase -XX:MetaspaceSize
and
-XX:MaxMetaspaceSize.
Artifacts
1. Java 8 & below: java
{app_name} -verbose:class
2. Java 9 & above: java
{app_name} -
Xlog:class+load=info:<filena
me>
3. jcmd {pid} GC.class_histogram
4. Heap Dump
Tools
1. Gceasy (GC log analysis)
2. HeapHero
3. Eclipse MAT
4. JProfiler
5. JHat
24
Permgen space
Young Old Threads JNI misc
GC
Direct
Buff
Code
Cach
e
Heap Memory Native Memory
JVM Memory
Happens only in Java 7 & below
25
OutOfMemoryError: Permgen Space
Diagnosis: Same as ‘Metaspace’ 
26
Unable to create new native threads
Young Old
Metaspac
e
JNI misc
GC
Direct
Buff
Code
Cach
e
Heap Memory Native Memory
JVM Memory
27
Real Case Study
Slowdown in a Major Financial Institution’s Middleware
http://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMTcvMDMvMTQvLS10aHJlYWREdW1wLTIudHh0LS0xMi0yOC0zMw==&s=t
28
OutOfMemoryError: Unable to create new native
threads
Causes
1. Threads are leaking
Solutions
1. Fix thread leak
2. Increase the Thread Limits Set
at Operating System(ulimit –u)
3. Reduce Java Heap Size
4. Kills other processes
5. Increase RAM capacity
6. Reduce thread stack size (-Xss).
Note: can cause
Artifacts
1. Thread Dump
Tools
1. FastThread
2. Text Editor
29
Direct buffer memory
Young Old
Metaspac
e
Threads JNI misc
GC
Code
Cach
e
Heap Memory Native Memory
JVM Memory
Becoming Prevasive in modern Spring Boot Frameworks
30
OutOfMemoryError: Direct buffer memory
Causes
1. Increase(or Leak) in direct buffers
usage
a. java.nio package
b. Moving from Spring
RestTemplate to WebClient
c. Image Processing Libraries
d. Networking Libraries
e. Some JDBC Drives
Solutions
1. Fix Memory Leak code
2. Increase -
XX:MaxDirectMemorySize
3. Upgrade to Java 17 (There some
issues in Java 11)
Artifacts
1. App Log or Std error
2. Native Memory Tracking
Tools
31
Kill process or sacrifice child
JVM Kernel
X
32
Real Case Study
Intermittent HTTP 502 errors in AWS EBS Service
EBS Architecture
33
Clue: Nginx Error
34
35
36
OutOfMemoryError: Kill process or sacrifice child
Causes
1. Lack of RAM capacity
2. High Memory utilization by
other processes
3. Memory Leak in the code
Solutions
1. Kill other processes
2. Increase RAM capacity
3. Fix Memory Leak in the code
Artifacts
1. Dmesg
2. GC Log
3. Heap Dump
4. Native Memory Tracking
Tools
1. HeapHero
2. Eclipse MAT
3. yCrash
37
Reason stack_trace_with_native_method
Young Old
Metaspac
e
Threads misc
GC
Direct
Buff
Code
Cach
e
Heap Memory Native Memory
JVM Memory
38
OutOfMemoryError: Reason stack_trace_with_native_method
Don’t worry, if you are not using JNI 
Ram Lakshmanan ram@tier1app.com
@tier1app https://www.linkedin.com/company/ycrash
This deck will be published in:
https://blog.heaphero.io
If you want to learn more …
39
THANK YOU
FRIENDS

More Related Content

Similar to Effectively Troubleshoot 9 Types of OutOfMemoryError

Efficient Memory and Thread Management in Highly Parallel Java Applications
Efficient Memory and Thread Management in Highly Parallel Java ApplicationsEfficient Memory and Thread Management in Highly Parallel Java Applications
Efficient Memory and Thread Management in Highly Parallel Java ApplicationsPhillip Koza
 
Devoxx Fr 2022 - Remèdes aux oomkill, warm-ups, et lenteurs pour des conteneu...
Devoxx Fr 2022 - Remèdes aux oomkill, warm-ups, et lenteurs pour des conteneu...Devoxx Fr 2022 - Remèdes aux oomkill, warm-ups, et lenteurs pour des conteneu...
Devoxx Fr 2022 - Remèdes aux oomkill, warm-ups, et lenteurs pour des conteneu...Jean-Philippe BEMPEL
 
Don't dump thread dumps
Don't dump thread dumpsDon't dump thread dumps
Don't dump thread dumpsTier1 App
 
Shooting the troubles: Crashes, Slowdowns, CPU Spikes
Shooting the troubles: Crashes, Slowdowns, CPU SpikesShooting the troubles: Crashes, Slowdowns, CPU Spikes
Shooting the troubles: Crashes, Slowdowns, CPU SpikesTier1 app
 
Mastering java in containers - MadridJUG
Mastering java in containers - MadridJUGMastering java in containers - MadridJUG
Mastering java in containers - MadridJUGJorge Morales
 
Java и Linux — особенности эксплуатации / Алексей Рагозин (Дойче Банк)
Java и Linux — особенности эксплуатации / Алексей Рагозин (Дойче Банк)Java и Linux — особенности эксплуатации / Алексей Рагозин (Дойче Банк)
Java и Linux — особенности эксплуатации / Алексей Рагозин (Дойче Банк)Ontico
 
java-monitoring-troubleshooting
java-monitoring-troubleshootingjava-monitoring-troubleshooting
java-monitoring-troubleshootingWilliam Au
 
What to do if Your Kafka Streams App Gets OOMKilled? with Andrey Serebryanskiy
What to do if Your Kafka Streams App Gets OOMKilled? with Andrey SerebryanskiyWhat to do if Your Kafka Streams App Gets OOMKilled? with Andrey Serebryanskiy
What to do if Your Kafka Streams App Gets OOMKilled? with Andrey SerebryanskiyHostedbyConfluent
 
JBoss Enterprise Application Platform 6 Troubleshooting
JBoss Enterprise Application Platform 6 TroubleshootingJBoss Enterprise Application Platform 6 Troubleshooting
JBoss Enterprise Application Platform 6 TroubleshootingAlexandre Cavalcanti
 
Introduction of Java GC Tuning and Java Java Mission Control
Introduction of Java GC Tuning and Java Java Mission ControlIntroduction of Java GC Tuning and Java Java Mission Control
Introduction of Java GC Tuning and Java Java Mission ControlLeon Chen
 
16 artifacts to capture when there is a production problem
16 artifacts to capture when there is a production problem16 artifacts to capture when there is a production problem
16 artifacts to capture when there is a production problemTier1 app
 
Troubleshooting real production problems
Troubleshooting real production problemsTroubleshooting real production problems
Troubleshooting real production problemsTier1 app
 
Java 어플리케이션 성능튜닝 Part1
Java 어플리케이션 성능튜닝 Part1Java 어플리케이션 성능튜닝 Part1
Java 어플리케이션 성능튜닝 Part1상욱 송
 
Java on Linux for devs and ops
Java on Linux for devs and opsJava on Linux for devs and ops
Java on Linux for devs and opsaragozin
 

Similar to Effectively Troubleshoot 9 Types of OutOfMemoryError (20)

Efficient Memory and Thread Management in Highly Parallel Java Applications
Efficient Memory and Thread Management in Highly Parallel Java ApplicationsEfficient Memory and Thread Management in Highly Parallel Java Applications
Efficient Memory and Thread Management in Highly Parallel Java Applications
 
Devoxx Fr 2022 - Remèdes aux oomkill, warm-ups, et lenteurs pour des conteneu...
Devoxx Fr 2022 - Remèdes aux oomkill, warm-ups, et lenteurs pour des conteneu...Devoxx Fr 2022 - Remèdes aux oomkill, warm-ups, et lenteurs pour des conteneu...
Devoxx Fr 2022 - Remèdes aux oomkill, warm-ups, et lenteurs pour des conteneu...
 
Java performance tuning
Java performance tuningJava performance tuning
Java performance tuning
 
Javasession10
Javasession10Javasession10
Javasession10
 
Don't dump thread dumps
Don't dump thread dumpsDon't dump thread dumps
Don't dump thread dumps
 
Shooting the troubles: Crashes, Slowdowns, CPU Spikes
Shooting the troubles: Crashes, Slowdowns, CPU SpikesShooting the troubles: Crashes, Slowdowns, CPU Spikes
Shooting the troubles: Crashes, Slowdowns, CPU Spikes
 
Mastering java in containers - MadridJUG
Mastering java in containers - MadridJUGMastering java in containers - MadridJUG
Mastering java in containers - MadridJUG
 
Java и Linux — особенности эксплуатации / Алексей Рагозин (Дойче Банк)
Java и Linux — особенности эксплуатации / Алексей Рагозин (Дойче Банк)Java и Linux — особенности эксплуатации / Алексей Рагозин (Дойче Банк)
Java и Linux — особенности эксплуатации / Алексей Рагозин (Дойче Банк)
 
[BGOUG] Java GC - Friend or Foe
[BGOUG] Java GC - Friend or Foe[BGOUG] Java GC - Friend or Foe
[BGOUG] Java GC - Friend or Foe
 
java-monitoring-troubleshooting
java-monitoring-troubleshootingjava-monitoring-troubleshooting
java-monitoring-troubleshooting
 
Cache is King
Cache is KingCache is King
Cache is King
 
What to do if Your Kafka Streams App Gets OOMKilled? with Andrey Serebryanskiy
What to do if Your Kafka Streams App Gets OOMKilled? with Andrey SerebryanskiyWhat to do if Your Kafka Streams App Gets OOMKilled? with Andrey Serebryanskiy
What to do if Your Kafka Streams App Gets OOMKilled? with Andrey Serebryanskiy
 
JBoss Enterprise Application Platform 6 Troubleshooting
JBoss Enterprise Application Platform 6 TroubleshootingJBoss Enterprise Application Platform 6 Troubleshooting
JBoss Enterprise Application Platform 6 Troubleshooting
 
Introduction of Java GC Tuning and Java Java Mission Control
Introduction of Java GC Tuning and Java Java Mission ControlIntroduction of Java GC Tuning and Java Java Mission Control
Introduction of Java GC Tuning and Java Java Mission Control
 
16 artifacts to capture when there is a production problem
16 artifacts to capture when there is a production problem16 artifacts to capture when there is a production problem
16 artifacts to capture when there is a production problem
 
Troubleshooting real production problems
Troubleshooting real production problemsTroubleshooting real production problems
Troubleshooting real production problems
 
Inside the jvm
Inside the jvmInside the jvm
Inside the jvm
 
Java 어플리케이션 성능튜닝 Part1
Java 어플리케이션 성능튜닝 Part1Java 어플리케이션 성능튜닝 Part1
Java 어플리케이션 성능튜닝 Part1
 
Heap & thread dump
Heap & thread dumpHeap & thread dump
Heap & thread dump
 
Java on Linux for devs and ops
Java on Linux for devs and opsJava on Linux for devs and ops
Java on Linux for devs and ops
 

More from Tier1 app

Top-5-java-perf-problems-jax_mainz_2024.pptx
Top-5-java-perf-problems-jax_mainz_2024.pptxTop-5-java-perf-problems-jax_mainz_2024.pptx
Top-5-java-perf-problems-jax_mainz_2024.pptxTier1 app
 
KnowAPIs-UnknownPerf-jaxMainz-2024 (1).pptx
KnowAPIs-UnknownPerf-jaxMainz-2024 (1).pptxKnowAPIs-UnknownPerf-jaxMainz-2024 (1).pptx
KnowAPIs-UnknownPerf-jaxMainz-2024 (1).pptxTier1 app
 
Top-5-production-devconMunich-2023-v2.pptx
Top-5-production-devconMunich-2023-v2.pptxTop-5-production-devconMunich-2023-v2.pptx
Top-5-production-devconMunich-2023-v2.pptxTier1 app
 
predicting-m3-devopsconMunich-2023-v2.pptx
predicting-m3-devopsconMunich-2023-v2.pptxpredicting-m3-devopsconMunich-2023-v2.pptx
predicting-m3-devopsconMunich-2023-v2.pptxTier1 app
 
Top-5-production-devconMunich-2023.pptx
Top-5-production-devconMunich-2023.pptxTop-5-production-devconMunich-2023.pptx
Top-5-production-devconMunich-2023.pptxTier1 app
 
predicting-m3-devopsconMunich-2023.pptx
predicting-m3-devopsconMunich-2023.pptxpredicting-m3-devopsconMunich-2023.pptx
predicting-m3-devopsconMunich-2023.pptxTier1 app
 
Predicting Production Outages: Unleashing the Power of Micro-Metrics – ADDO C...
Predicting Production Outages: Unleashing the Power of Micro-Metrics – ADDO C...Predicting Production Outages: Unleashing the Power of Micro-Metrics – ADDO C...
Predicting Production Outages: Unleashing the Power of Micro-Metrics – ADDO C...Tier1 app
 
7-JVM-arguments-JaxLondon-2023.pptx
7-JVM-arguments-JaxLondon-2023.pptx7-JVM-arguments-JaxLondon-2023.pptx
7-JVM-arguments-JaxLondon-2023.pptxTier1 app
 
Top-5-Performance-JaxLondon-2023.pptx
Top-5-Performance-JaxLondon-2023.pptxTop-5-Performance-JaxLondon-2023.pptx
Top-5-Performance-JaxLondon-2023.pptxTier1 app
 
16 ARTIFACTS TO CAPTURE WHEN YOUR CONTAINER APPLICATION IS IN TROUBLE
16 ARTIFACTS TO CAPTURE WHEN YOUR CONTAINER APPLICATION IS IN TROUBLE16 ARTIFACTS TO CAPTURE WHEN YOUR CONTAINER APPLICATION IS IN TROUBLE
16 ARTIFACTS TO CAPTURE WHEN YOUR CONTAINER APPLICATION IS IN TROUBLETier1 app
 
MAJOR OUTAGES IN MAJOR ENTERPRISES
MAJOR OUTAGES IN MAJOR ENTERPRISESMAJOR OUTAGES IN MAJOR ENTERPRISES
MAJOR OUTAGES IN MAJOR ENTERPRISESTier1 app
 
KnowAPIs-UnknownPerf-confoo-2023 (1).pptx
KnowAPIs-UnknownPerf-confoo-2023 (1).pptxKnowAPIs-UnknownPerf-confoo-2023 (1).pptx
KnowAPIs-UnknownPerf-confoo-2023 (1).pptxTier1 app
 
memory-patterns-confoo-2023.pptx
memory-patterns-confoo-2023.pptxmemory-patterns-confoo-2023.pptx
memory-patterns-confoo-2023.pptxTier1 app
 
this-is-garbage-talk-2022.pptx
this-is-garbage-talk-2022.pptxthis-is-garbage-talk-2022.pptx
this-is-garbage-talk-2022.pptxTier1 app
 
millions-gc-jax-2022.pptx
millions-gc-jax-2022.pptxmillions-gc-jax-2022.pptx
millions-gc-jax-2022.pptxTier1 app
 
lets-crash-apps-jax-2022.pptx
lets-crash-apps-jax-2022.pptxlets-crash-apps-jax-2022.pptx
lets-crash-apps-jax-2022.pptxTier1 app
 
‘16 artifacts’ to capture when there is a production problem
‘16 artifacts’ to capture when there is a production problem‘16 artifacts’ to capture when there is a production problem
‘16 artifacts’ to capture when there is a production problemTier1 app
 
Lets crash-applications
Lets crash-applicationsLets crash-applications
Lets crash-applicationsTier1 app
 
Lets crash-applications
Lets crash-applicationsLets crash-applications
Lets crash-applicationsTier1 app
 
7 habits of highly effective Performance Troubleshooters
7 habits of highly effective Performance Troubleshooters7 habits of highly effective Performance Troubleshooters
7 habits of highly effective Performance TroubleshootersTier1 app
 

More from Tier1 app (20)

Top-5-java-perf-problems-jax_mainz_2024.pptx
Top-5-java-perf-problems-jax_mainz_2024.pptxTop-5-java-perf-problems-jax_mainz_2024.pptx
Top-5-java-perf-problems-jax_mainz_2024.pptx
 
KnowAPIs-UnknownPerf-jaxMainz-2024 (1).pptx
KnowAPIs-UnknownPerf-jaxMainz-2024 (1).pptxKnowAPIs-UnknownPerf-jaxMainz-2024 (1).pptx
KnowAPIs-UnknownPerf-jaxMainz-2024 (1).pptx
 
Top-5-production-devconMunich-2023-v2.pptx
Top-5-production-devconMunich-2023-v2.pptxTop-5-production-devconMunich-2023-v2.pptx
Top-5-production-devconMunich-2023-v2.pptx
 
predicting-m3-devopsconMunich-2023-v2.pptx
predicting-m3-devopsconMunich-2023-v2.pptxpredicting-m3-devopsconMunich-2023-v2.pptx
predicting-m3-devopsconMunich-2023-v2.pptx
 
Top-5-production-devconMunich-2023.pptx
Top-5-production-devconMunich-2023.pptxTop-5-production-devconMunich-2023.pptx
Top-5-production-devconMunich-2023.pptx
 
predicting-m3-devopsconMunich-2023.pptx
predicting-m3-devopsconMunich-2023.pptxpredicting-m3-devopsconMunich-2023.pptx
predicting-m3-devopsconMunich-2023.pptx
 
Predicting Production Outages: Unleashing the Power of Micro-Metrics – ADDO C...
Predicting Production Outages: Unleashing the Power of Micro-Metrics – ADDO C...Predicting Production Outages: Unleashing the Power of Micro-Metrics – ADDO C...
Predicting Production Outages: Unleashing the Power of Micro-Metrics – ADDO C...
 
7-JVM-arguments-JaxLondon-2023.pptx
7-JVM-arguments-JaxLondon-2023.pptx7-JVM-arguments-JaxLondon-2023.pptx
7-JVM-arguments-JaxLondon-2023.pptx
 
Top-5-Performance-JaxLondon-2023.pptx
Top-5-Performance-JaxLondon-2023.pptxTop-5-Performance-JaxLondon-2023.pptx
Top-5-Performance-JaxLondon-2023.pptx
 
16 ARTIFACTS TO CAPTURE WHEN YOUR CONTAINER APPLICATION IS IN TROUBLE
16 ARTIFACTS TO CAPTURE WHEN YOUR CONTAINER APPLICATION IS IN TROUBLE16 ARTIFACTS TO CAPTURE WHEN YOUR CONTAINER APPLICATION IS IN TROUBLE
16 ARTIFACTS TO CAPTURE WHEN YOUR CONTAINER APPLICATION IS IN TROUBLE
 
MAJOR OUTAGES IN MAJOR ENTERPRISES
MAJOR OUTAGES IN MAJOR ENTERPRISESMAJOR OUTAGES IN MAJOR ENTERPRISES
MAJOR OUTAGES IN MAJOR ENTERPRISES
 
KnowAPIs-UnknownPerf-confoo-2023 (1).pptx
KnowAPIs-UnknownPerf-confoo-2023 (1).pptxKnowAPIs-UnknownPerf-confoo-2023 (1).pptx
KnowAPIs-UnknownPerf-confoo-2023 (1).pptx
 
memory-patterns-confoo-2023.pptx
memory-patterns-confoo-2023.pptxmemory-patterns-confoo-2023.pptx
memory-patterns-confoo-2023.pptx
 
this-is-garbage-talk-2022.pptx
this-is-garbage-talk-2022.pptxthis-is-garbage-talk-2022.pptx
this-is-garbage-talk-2022.pptx
 
millions-gc-jax-2022.pptx
millions-gc-jax-2022.pptxmillions-gc-jax-2022.pptx
millions-gc-jax-2022.pptx
 
lets-crash-apps-jax-2022.pptx
lets-crash-apps-jax-2022.pptxlets-crash-apps-jax-2022.pptx
lets-crash-apps-jax-2022.pptx
 
‘16 artifacts’ to capture when there is a production problem
‘16 artifacts’ to capture when there is a production problem‘16 artifacts’ to capture when there is a production problem
‘16 artifacts’ to capture when there is a production problem
 
Lets crash-applications
Lets crash-applicationsLets crash-applications
Lets crash-applications
 
Lets crash-applications
Lets crash-applicationsLets crash-applications
Lets crash-applications
 
7 habits of highly effective Performance Troubleshooters
7 habits of highly effective Performance Troubleshooters7 habits of highly effective Performance Troubleshooters
7 habits of highly effective Performance Troubleshooters
 

Recently uploaded

Community is Just as Important as Code by Andrea Goulet
Community is Just as Important as Code by Andrea GouletCommunity is Just as Important as Code by Andrea Goulet
Community is Just as Important as Code by Andrea GouletAndrea Goulet
 
Rapidoform for Modern Form Building and Insights
Rapidoform for Modern Form Building and InsightsRapidoform for Modern Form Building and Insights
Rapidoform for Modern Form Building and Insightsrapidoform
 
The Strategic Impact of Buying vs Building in Test Automation
The Strategic Impact of Buying vs Building in Test AutomationThe Strategic Impact of Buying vs Building in Test Automation
The Strategic Impact of Buying vs Building in Test AutomationElement34
 
architecting-ai-in-the-enterprise-apis-and-applications.pdf
architecting-ai-in-the-enterprise-apis-and-applications.pdfarchitecting-ai-in-the-enterprise-apis-and-applications.pdf
architecting-ai-in-the-enterprise-apis-and-applications.pdfWSO2
 
Software Engineering - Introduction + Process Models + Requirements Engineering
Software Engineering - Introduction + Process Models + Requirements EngineeringSoftware Engineering - Introduction + Process Models + Requirements Engineering
Software Engineering - Introduction + Process Models + Requirements EngineeringPrakhyath Rai
 
Jax, FL Admin Community Group 05.14.2024 Combined Deck
Jax, FL Admin Community Group 05.14.2024 Combined DeckJax, FL Admin Community Group 05.14.2024 Combined Deck
Jax, FL Admin Community Group 05.14.2024 Combined DeckMarc Lester
 
OpenChain Webinar: AboutCode and Beyond - End-to-End SCA
OpenChain Webinar: AboutCode and Beyond - End-to-End SCAOpenChain Webinar: AboutCode and Beyond - End-to-End SCA
OpenChain Webinar: AboutCode and Beyond - End-to-End SCAShane Coughlan
 
Prompt Engineering - an Art, a Science, or your next Job Title?
Prompt Engineering - an Art, a Science, or your next Job Title?Prompt Engineering - an Art, a Science, or your next Job Title?
Prompt Engineering - an Art, a Science, or your next Job Title?Maxim Salnikov
 
From Knowledge Graphs via Lego Bricks to scientific conversations.pptx
From Knowledge Graphs via Lego Bricks to scientific conversations.pptxFrom Knowledge Graphs via Lego Bricks to scientific conversations.pptx
From Knowledge Graphs via Lego Bricks to scientific conversations.pptxNeo4j
 
Evolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI EraEvolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI Eraconfluent
 
Microsoft365_Dev_Security_2024_05_16.pdf
Microsoft365_Dev_Security_2024_05_16.pdfMicrosoft365_Dev_Security_2024_05_16.pdf
Microsoft365_Dev_Security_2024_05_16.pdfMarkus Moeller
 
Workshop: Enabling GenAI Breakthroughs with Knowledge Graphs - GraphSummit Milan
Workshop: Enabling GenAI Breakthroughs with Knowledge Graphs - GraphSummit MilanWorkshop: Enabling GenAI Breakthroughs with Knowledge Graphs - GraphSummit Milan
Workshop: Enabling GenAI Breakthroughs with Knowledge Graphs - GraphSummit MilanNeo4j
 
CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...
CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...
CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...Neo4j
 
Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...
Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...
Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...drm1699
 
Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...
Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...
Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...Abortion Clinic
 
Effective Strategies for Wix's Scaling challenges - GeeCon
Effective Strategies for Wix's Scaling challenges - GeeConEffective Strategies for Wix's Scaling challenges - GeeCon
Effective Strategies for Wix's Scaling challenges - GeeConNatan Silnitsky
 
BusinessGPT - Security and Governance for Generative AI
BusinessGPT  - Security and Governance for Generative AIBusinessGPT  - Security and Governance for Generative AI
BusinessGPT - Security and Governance for Generative AIAGATSoftware
 

Recently uploaded (20)

Community is Just as Important as Code by Andrea Goulet
Community is Just as Important as Code by Andrea GouletCommunity is Just as Important as Code by Andrea Goulet
Community is Just as Important as Code by Andrea Goulet
 
Rapidoform for Modern Form Building and Insights
Rapidoform for Modern Form Building and InsightsRapidoform for Modern Form Building and Insights
Rapidoform for Modern Form Building and Insights
 
The Strategic Impact of Buying vs Building in Test Automation
The Strategic Impact of Buying vs Building in Test AutomationThe Strategic Impact of Buying vs Building in Test Automation
The Strategic Impact of Buying vs Building in Test Automation
 
architecting-ai-in-the-enterprise-apis-and-applications.pdf
architecting-ai-in-the-enterprise-apis-and-applications.pdfarchitecting-ai-in-the-enterprise-apis-and-applications.pdf
architecting-ai-in-the-enterprise-apis-and-applications.pdf
 
Software Engineering - Introduction + Process Models + Requirements Engineering
Software Engineering - Introduction + Process Models + Requirements EngineeringSoftware Engineering - Introduction + Process Models + Requirements Engineering
Software Engineering - Introduction + Process Models + Requirements Engineering
 
Abortion Pill Prices Germiston ](+27832195400*)[ 🏥 Women's Abortion Clinic in...
Abortion Pill Prices Germiston ](+27832195400*)[ 🏥 Women's Abortion Clinic in...Abortion Pill Prices Germiston ](+27832195400*)[ 🏥 Women's Abortion Clinic in...
Abortion Pill Prices Germiston ](+27832195400*)[ 🏥 Women's Abortion Clinic in...
 
Jax, FL Admin Community Group 05.14.2024 Combined Deck
Jax, FL Admin Community Group 05.14.2024 Combined DeckJax, FL Admin Community Group 05.14.2024 Combined Deck
Jax, FL Admin Community Group 05.14.2024 Combined Deck
 
OpenChain Webinar: AboutCode and Beyond - End-to-End SCA
OpenChain Webinar: AboutCode and Beyond - End-to-End SCAOpenChain Webinar: AboutCode and Beyond - End-to-End SCA
OpenChain Webinar: AboutCode and Beyond - End-to-End SCA
 
Prompt Engineering - an Art, a Science, or your next Job Title?
Prompt Engineering - an Art, a Science, or your next Job Title?Prompt Engineering - an Art, a Science, or your next Job Title?
Prompt Engineering - an Art, a Science, or your next Job Title?
 
From Knowledge Graphs via Lego Bricks to scientific conversations.pptx
From Knowledge Graphs via Lego Bricks to scientific conversations.pptxFrom Knowledge Graphs via Lego Bricks to scientific conversations.pptx
From Knowledge Graphs via Lego Bricks to scientific conversations.pptx
 
Abortion Pill Prices Turfloop ](+27832195400*)[ 🏥 Women's Abortion Clinic in ...
Abortion Pill Prices Turfloop ](+27832195400*)[ 🏥 Women's Abortion Clinic in ...Abortion Pill Prices Turfloop ](+27832195400*)[ 🏥 Women's Abortion Clinic in ...
Abortion Pill Prices Turfloop ](+27832195400*)[ 🏥 Women's Abortion Clinic in ...
 
Evolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI EraEvolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI Era
 
Microsoft365_Dev_Security_2024_05_16.pdf
Microsoft365_Dev_Security_2024_05_16.pdfMicrosoft365_Dev_Security_2024_05_16.pdf
Microsoft365_Dev_Security_2024_05_16.pdf
 
Workshop: Enabling GenAI Breakthroughs with Knowledge Graphs - GraphSummit Milan
Workshop: Enabling GenAI Breakthroughs with Knowledge Graphs - GraphSummit MilanWorkshop: Enabling GenAI Breakthroughs with Knowledge Graphs - GraphSummit Milan
Workshop: Enabling GenAI Breakthroughs with Knowledge Graphs - GraphSummit Milan
 
Abortion Clinic In Springs ](+27832195400*)[ 🏥 Safe Abortion Pills in Springs...
Abortion Clinic In Springs ](+27832195400*)[ 🏥 Safe Abortion Pills in Springs...Abortion Clinic In Springs ](+27832195400*)[ 🏥 Safe Abortion Pills in Springs...
Abortion Clinic In Springs ](+27832195400*)[ 🏥 Safe Abortion Pills in Springs...
 
CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...
CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...
CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...
 
Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...
Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...
Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...
 
Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...
Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...
Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...
 
Effective Strategies for Wix's Scaling challenges - GeeCon
Effective Strategies for Wix's Scaling challenges - GeeConEffective Strategies for Wix's Scaling challenges - GeeCon
Effective Strategies for Wix's Scaling challenges - GeeCon
 
BusinessGPT - Security and Governance for Generative AI
BusinessGPT  - Security and Governance for Generative AIBusinessGPT  - Security and Governance for Generative AI
BusinessGPT - Security and Governance for Generative AI
 

Effectively Troubleshoot 9 Types of OutOfMemoryError

  • 1. 1
  • 2. Does Java process memory utilization go beyond –Xmx? 2 –XX:MaxMetaspaceSize -Xmx Young Old Metaspac e Threads JNI misc GC Direct Buff Code Cach e Heap Memory Native Memory Java Process Memory
  • 4. 4 How to know the type of OutOfMemoryError? java.lang.OutOfMemoryError: <type>
  • 5. 5 Java Heap Space Metaspac e Threads JNI misc GC Direct Buff Code Cach e Heap Memory Native Memory JVM Memory Most common type
  • 6. 6 GC Behavior of a Healthy Application - Full Garbage Collection Event
  • 7. 7 GC Behavior of Acute Memory Leak - Full Garbage Collection Event
  • 8. 8 GC Behavior when there is a Memory Leak - Full Garbage Collection Event
  • 9. 9 How to study GC Behavior? -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:<file-path> Till Java 8 -Xlog:gc*:file=<file-path> From Java 9 Enable GC logs (always) Almost zero overhead
  • 11. 11 How to capture Heap Dumps? 8 options: https://blog.heaphero.io/2017/10/13/how-to-capture-java-heap-dumps-7-options/
  • 12. 1. GC Log 10. netstat 12. vmstat 2. Thread Dump 9. dmesg 3. Heap Dump 6. ps 8. Disk Usage 5. top 13. iostat 11. ping 14. Kernel Params 15. App Logs 16. metadata 4. Heap Substitute 7. top -H 12 Open-source script: https://github.com/ycrash/yc-data-script 360° Troubleshooting artifacts
  • 14. 14 OutOfMemoryError: Java Heap Space Causes 1. Increase in Traffic Volume 2. Memory leak due to buggy code Solutions 1. Fix Memory Leak in the code 2. Increase heap size (-Xmx) Artifacts 1. Garbage Collection Log 2. Heap Dump Tools 1. GCeasy (GC log analysis) 2. HeapHero 3. Eclipse MAT 4. JProfiler
  • 15. 15 GC overhead limit exceeded Metaspac e Threads JNI misc GC Direct Buff Code Cach e Heap Memory Native Memory JVM Memory
  • 16. 16 OutOfMemoryError: GC overhead limit exceeded Diagnosis: Same as ‘Java Heap Space’ 
  • 17. 17 Requested array size exceeds VM limit Young Old Metaspac e Threads JNI misc GC Direct Buff Code Cach e Heap Memory Native Memory JVM Memory
  • 19. 19 OutOfMemoryError: Requested array size exceeds VM limit Causes 1. Allocated array larger than heap size Solutions 1. Fix Memory Leak code 2. Increase heap size (-Xmx) Artifacts 1. Application Log or std error Tools
  • 20. 20 Metaspace Young Old Threads JNI misc GC Direct Buff Code Cach e Heap Memory Native Memory JVM Memory
  • 22. 22 Metaspace Memory Leak - Full Garbage Collection Event
  • 23. 23 OutOfMemoryError: Metaspace Causes 1. Creating large number of Dynamic classes (Java Reflection, Groovy type of script languages) 2. Loading large number of Classes 3. Loading large number of Classloaders Solutions 1. Fix Memory Leak code 2. Increase -XX:MetaspaceSize and -XX:MaxMetaspaceSize. Artifacts 1. Java 8 & below: java {app_name} -verbose:class 2. Java 9 & above: java {app_name} - Xlog:class+load=info:<filena me> 3. jcmd {pid} GC.class_histogram 4. Heap Dump Tools 1. Gceasy (GC log analysis) 2. HeapHero 3. Eclipse MAT 4. JProfiler 5. JHat
  • 24. 24 Permgen space Young Old Threads JNI misc GC Direct Buff Code Cach e Heap Memory Native Memory JVM Memory Happens only in Java 7 & below
  • 25. 25 OutOfMemoryError: Permgen Space Diagnosis: Same as ‘Metaspace’ 
  • 26. 26 Unable to create new native threads Young Old Metaspac e JNI misc GC Direct Buff Code Cach e Heap Memory Native Memory JVM Memory
  • 27. 27 Real Case Study Slowdown in a Major Financial Institution’s Middleware http://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMTcvMDMvMTQvLS10aHJlYWREdW1wLTIudHh0LS0xMi0yOC0zMw==&s=t
  • 28. 28 OutOfMemoryError: Unable to create new native threads Causes 1. Threads are leaking Solutions 1. Fix thread leak 2. Increase the Thread Limits Set at Operating System(ulimit –u) 3. Reduce Java Heap Size 4. Kills other processes 5. Increase RAM capacity 6. Reduce thread stack size (-Xss). Note: can cause Artifacts 1. Thread Dump Tools 1. FastThread 2. Text Editor
  • 29. 29 Direct buffer memory Young Old Metaspac e Threads JNI misc GC Code Cach e Heap Memory Native Memory JVM Memory Becoming Prevasive in modern Spring Boot Frameworks
  • 30. 30 OutOfMemoryError: Direct buffer memory Causes 1. Increase(or Leak) in direct buffers usage a. java.nio package b. Moving from Spring RestTemplate to WebClient c. Image Processing Libraries d. Networking Libraries e. Some JDBC Drives Solutions 1. Fix Memory Leak code 2. Increase - XX:MaxDirectMemorySize 3. Upgrade to Java 17 (There some issues in Java 11) Artifacts 1. App Log or Std error 2. Native Memory Tracking Tools
  • 31. 31 Kill process or sacrifice child JVM Kernel X
  • 32. 32 Real Case Study Intermittent HTTP 502 errors in AWS EBS Service
  • 35. 35
  • 36. 36 OutOfMemoryError: Kill process or sacrifice child Causes 1. Lack of RAM capacity 2. High Memory utilization by other processes 3. Memory Leak in the code Solutions 1. Kill other processes 2. Increase RAM capacity 3. Fix Memory Leak in the code Artifacts 1. Dmesg 2. GC Log 3. Heap Dump 4. Native Memory Tracking Tools 1. HeapHero 2. Eclipse MAT 3. yCrash
  • 37. 37 Reason stack_trace_with_native_method Young Old Metaspac e Threads misc GC Direct Buff Code Cach e Heap Memory Native Memory JVM Memory
  • 39. Ram Lakshmanan ram@tier1app.com @tier1app https://www.linkedin.com/company/ycrash This deck will be published in: https://blog.heaphero.io If you want to learn more … 39 THANK YOU FRIENDS

Editor's Notes

  1. Eclipse: LaunchPad_oom
  2. Gceasy - C:\workspace\ycrash\speeches\9-types-oom\memory-leak.log
  3. https://blog.heaphero.io/2017/10/13/how-to-capture-java-heap-dumps-7-options/
  4. http://localhost:8080/heap-index.jsp C:\workspace\tmp\dev-nexus\dump-2024-04-09.hprof http://localhost:8080/heap-report-wc.jsp?p=ZDI0ZHIxRGJtaUhHQXN3eUFWa3htQW9QeERicTZpZjNUczlXTFVVTE9jcWpMeHBsRkpSZEV4Vkk5cDZqZHJIK2NzditQeWx0UDZDbUhmNEIySTBZMkZ6NUhjMkh3SFdOSUdPZU5WTHZHZlk2dEdnQnNCMmp2MkltMEdoUnFxcUY=
  5. OOMRequestedArraySizeExceedsVMLimit
  6. com.buggyapp.oom.RequestedArraySizeExceedsVMLimit
  7. OOMMetaspace OOMMetaspace_verbose
  8. C:\workspace\ycrash\speeches\9-types-oom\thread-leak.txt Backup: http://localhost:8080/yc-report.jsp?ou=SAP&de=127.0.0.1&app=yc&ts=2024-04-09T11-09-51
  9. http://localhost:8080/yc-report.jsp?ou=SAP&de=10.160.1.221&app=yc&ts=2024-04-02T09-59-41
  10. https://test.ycrash.io/yc-report-kernel.jsp?ou=czlWbG0rUko0UXAxazlSbjZrSUIwUT09&de=172.31.7.106&app=yc&ts=2023-09-01T10-25-39