Heap dumps can look overwhelming, but with the right techniques, they become one of the most powerful tools for diagnosing Java memory issues. In this webinar deck, we cover: Querying heap dumps using OQL (Object Query Language) Automating investigations with Heap Dump Analysis APIs Handling remote heap dumps efficiently Tracing incoming and outgoing references to identify memory leaks Achieving a 360° JVM health view by correlating heap dumps with GC logs, thread dumps, NMT, kernel logs, and application logs These techniques will help you pinpoint root causes faster, troubleshoot OutOfMemoryErrors effectively, and build confidence in handling even the largest heap dumps. 📌 Explore the slides to sharpen your JVM troubleshooting skills and take your performance engineering to the next level.

2. Basic Heap Dump Analysis Techniques
https://www.youtube.com/watch?v=a_ojS6S86cE

3. Challenge: Transmitting Heap Dump From Production
Prod
Server
Shared
Drive
Your
Local
Machine
Co-Workers
Machines
Heap Dump Heap Dump
Heap Dump
Heap Dump

4. Challenge: Sensitive Heap Dump is now in Multiple Locations
Prod
Server
Shared
Drive
Your
Local
Machine
Co-Workers
Machines
Heap Dump Heap Dump
Heap Dump
Heap Dump
1 2
3
4

5. Challenge: Does Your Local Machine Have Sufficient Memory?

6. 1. REST API
Instantly Push & Analyze Heap Dumps From Production Servers
https://blog.heaphero.io/heap-dump-analysis-api/

7. Solution: Transmitting Heap Dump Directly to HeapHero Server
Prod
Server
HeapHero
Server
Heap Dump
You
Heap Dump in One Location. Safe &
Secure
Parse Large Heap Dump easily
REST API

8. 2. Analyzing From Remote Location
Eliminate the Hazle of Downloading Heap Dumps

9. Solution: Pull Heap Dump From Shared Drive/Remote Storage
Remote
Storage
HeapHero
Server
You
Pull
Model!
1
2
Heap Dump

10. Challenge: Sensitive Information Handled Carefully?

11. Heap Dump Report

12. 3. Heap Dump Sanitization
Remove Sensitive Information from the Heap Dump
https://docs.ycrash.io/ycrash-server/app-settings/sanitize-heap-dump.html

13. Heap Dump After Sanitization

14. OutOfMemoryError Required Artifacts
OutOfMemoryError: Java Heap Space GC Log, Heap Dump
OutOfMemoryError: GC overhead limit exceeded GC Log, Heap Dump
OutOfMemoryError: Unable to create native
threads
Thread Dump
OutOfMemoryError: Metaspace GC Log, Verbose Class Loading Log
OutOfMemoryError: PermGen Space GC Log, Verbose Class Loading Log
OutOfMemoryError: Kill process or sacrifice child dmesg
OutOfMemoryError: Requested array size
exceeds VM limit
Application Log
OutOfMemoryError: Direct buffer Memory Application Log, Native Memory Tracking
Challenge: Is Heap Dumps Sufficient To Troubleshoot
OutOfMemoryError?
How Many Types of OutOfMemoryError?9
https://blog.heaphero.io/types-of-outofmemoryerror/

15. 4. 360 ° Troubleshooting Artifacts
https://github.com/ycrash/yc-360-script

16. 16
yc-360° Open-Source script:
https://github.com/ycrash/yc-360-script
360° Artifacts
./yc –p <PROCESS_ID>
1. GC Log
10. netstat
12. vmstat
2. Thread Dump
9. dmesg
3. Heap Dump (optional)
6. ps
8. Disk Usage
5. top 13. iostat
11. ping
14. Kernel Params
15. App Logs
16. metadata
4. Heap Substitute
7. top -H

17. -XX:OnOutOfMemoryError
-XX:OnOutOfMemoryError="/usr/local/bin/yc-360-wrapper.sh"
Learn More: https://blog.heaphero.io/handling-outofmemoryerror-in-java-applications-with-the-yc-360-wrapper-script/

18. Synthetic Data != Production Data
Challenge: Why Memory Problems Not Caught In Performance Labs?
18
Performance Environment != Production Environment
Lack of Long Running Tests
Agreed!
Absence of Real-World Chaos

19. 19
However, at an acute scale. Not big enough to catch our attention.
Because we monitor only Macro-Metrics:
1. Response Time 2. Memory 3. CPU
But, Production Problems Do Happen In Our
Performance Labs

20. 5. Monitor Micro-Metrics In Performance Labs | CI/CD Pipeline
https://blog.ycrash.io/9-micro-metrics-that-forecast-production-outages-in-performance-labs/

21. 1. GC Behavior Pattern
2. Object Creation Rate
3. GC Throughput
4. GC Pause Time (Avg & Max)
5. Thread Patterns
6. Thread States
7. Thread activities within the Pool
8. TCP/IP Connection Count & States
9. Error Trends in Application Logs
Solution: Monitor these 9 Micro-Metrics In Your Performance Labs
https://www.youtube.com/watch?v=MQNDbIC2qrk

22. 22
GC Behavior of a Healthy Application
- Full Garbage Collection Event

23. 23
Application experiencing OutOfMemoryError in Production

24. 6. Object Query Language (OQL)
https://blog.heaphero.io/what-is-oql-and-how-it-helps-in-heap-dump-analysis/

25. OQL Examples
SELECT * FROM java.lang.String
Example 1: Find All String in the Application
Example 2: Find Strings which are greater than 5MB
SELECT * FROM java.lang.String s WHERE s.@retainedHeapSize > 5124
https://blog.heaphero.io/what-is-oql-and-how-it-helps-in-heap-dump-analysis/
OQL Syntax

26. 7. Sunburst Chart To Visualize Objects
https://blog.heaphero.io/worlds-first-interactive-memory-graph/

27. 27

28. Thank you my friends!
Ram Lakshmanan
@ycrash_rca
This deck will be published in: https://blog.ycrash.io
https://www.linkedin.com/company/ycrash