Exploring Java Heap Dumps (Oracle Code One 2018)
•Download as PPTX, PDF•
4 likes•2,025 views
Memory leaks are not always simple or easy to find. Heap dumps from production systems are often gigantic (4+ gigs) with millions of objects in memory. Simple spot checking with traditional tools is woefully inadequate in these situations, especially with real data. Leaks can be entire object graphs with enormous amounts of noise. This session will show you how to build custom tools using the Apache NetBeans Profiler/Heapwalker APIs. Using these APIs, you can read and analyze Java heaps programmatically to ask really hard questions. This gives you the power to analyze complex object graphs with tens of thousands of objects in seconds.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Exploring Java Heap Dumps (Oracle Code One 2018)
Similar to Exploring Java Heap Dumps (Oracle Code One 2018) (20)
More from Ryan Cuprak
More from Ryan Cuprak (20)
Recently uploaded
Recently uploaded (20)
Exploring Java Heap Dumps (Oracle Code One 2018)
- 1. Exploring Java Heap Dumps Ryan Cuprak
- 2. Java Heap Review • Java objects are stored in the heap • All objects are globally reachable in the heap • Heap is created when an application starts • Size of heap is configured using –Xmx and –Xmx • Garbage collection prunes the heap and removes objects no longer reachable • Stack memory - variable values are stored when their methods are invoked Heap Contains Everything and can be DUMPED to DISK
- 3. Why Analyze Heaps? • IT reports Java EE/Spring server memory footprint has grown to 9 gigs • Server app logs contain OutOfMemoryExceptions • Connections to queueing or database are exhausted • Serialized Java objects in queue are unreasonably large • Desktop application becomes unresponsive • Excessive amount of garbage collection
- 4. Java Heap Analysis MAT Profiler All you need is a profiler right?
- 5. Memory Leaks Textbook memory leaks - easy to find and fix. JConsole
- 6. Production Heap Dumps • 9 gigs of data • 13k classes loaded • ~ 136 million instances • ~6,000 GC roots
- 7. Production Heap Dumps Capture the Heap Dump and…
- 9. Heap Dump Panic Too much data! • Impossible to comprehend • No human way to explore the data • Application data model is too complicated
- 10. Real Memory Leaks Bank Account: 1231209 Owner Bob Owner JulieReport January 2018 Bank Account: 1231210 Bank Account: 1231209 Report January 2018 Owner Bob Challenge: Data looks good everywhere…
- 11. Real Memory Leaks Causes: • Faulty clone methods • Duplicate singletons • Accidently cached data • Cache logic bugs Complications • May NOT GROW over time (leaks gets cleaned-up) • More than one non-trivial memory leak
- 12. What about OQL? • OQL • Object Query Language – used for querying heaps • SQL-like language • Supports JavaScript expressions • Supported in NetBeans and VisualVM • Downside • Poorly documented and hard to use • Easy to create runaway queries
- 14. NetBeans Profiler • NetBeans is open source IDE/platform • Modular architecture • Clean code base Profiler GUI Profiler API
- 15. NetBeans Profiler API • Parses hprof files • Creates an object model representing the hprof file • Pages data in from disk • Simple API (master in about 10 minutes) • Independent of NetBeans • Can be extract and use in any IDE – Plain old Java Talk is really about how to build a custom heap analysis tool: • To answer specific data model questions • With custom logic for your data model
- 17. Generating Heap Dumps • Command line parameter: • -XX +HeapDumpOnOutOfMemoryError • Command line: • jmap –dump:format=b,file=dump.hprof <pid> • jhsdb jmap --binaryheap --pid <pid> • jcmd <pid> GC.heap_dump <file name> • Ctrl-Break Command Line
- 20. Heap Dump Warning Dumping the heap: • Takes time • Consumes diskspace • Negatively affects performance
- 21. Targeted Heap Dumps • Serialize object graphs from application to a file. • Read the serialized data into another tool and then programmatically create a heap dump.
- 23. Building Custom Profiler Create NetBeans Platform App Copy API src out of NetBeans
- 25. NetBeans Platform App Add dependency on “Java Profiler (JFluid)”
- 26. Profiler Sources Checkout source: https://github.com/apache/incubator-netbeans.git Profiler code: netbeans/profiler/lib.profiler/src/netbeans/lib.profiler/heap Copy heap directory
- 27. Which Approach? • Copying sources easiest • Most analysis apps are command line (one-offs) - Note - You don’t need the classpath of the application from which the heap was generated.
- 29. Opening a Heap That’s All!
- 30. Heap Object Methods getJavaClassByName(String fqn) : JavaClass getAllClasses() : List getBiggestObjectsByRetainedSize(int number) : List getGCRoots(): GCRoot getInstanceByID(long instanceId) : Instance getJavaClassByID(long javaclassId) : JavaClass getJavaClassesByRegExp(String regexp) : HeapSummary getSummary() : Properties
- 32. Heap Summary • getTotalLiveInstances() : long • getTime() : long • getTotalAllocatedBytes() : long • getTotalAllocatedInstances() : long • getTotalLiveBytes() : long
- 33. Exploration Starting Points • GCRoots • Threads (really GCRoots) • Class Types
- 34. GC Roots • Garbage Collection Root is an object that is accessible from outside the heap. • Objects that aren’t accessible from a GC Root are garbage collected • GC root categorization: • Class loaded by system class loader • Thread • Stack Local • Monitor • JNI Reference • Held by JVM
- 35. Garbage Collection Roots Java frame: 44 thread object: 5 JNI global: 29 sticky class: 1284
- 36. GCRoot Objects
- 37. GC Roots JNI_GLOBAL = "JNI global"; JNI_LOCAL = "JNI local"; JAVA_FRAME = "Java frame"; NATIVE_STACK = "native stack"; STICKY_CLASS = "sticky class"; THREAD_BLOCK = "thread block"; MONITOR_USED = "monitor used"; THREAD_OBJECT = "thread object"; UNKNOWN = "unknown"; INTERNED_STRING = "interned string"; FINALIZING = "finalizing"; DEBUGGER = "debugger"; REFERENCE_CLEANUP = "reference cleanup"; VM_INTERNAL = "VM internal"; JNI_MONITOR = "JNI monitor"; root.getKind() : String
- 38. Finding Classes • Can perform lookup using: • Fully qualified class name (ex. java.lang.String) • Class ID • Instance ID • IDs are unique to heap dump • Hash codes are not available!
- 39. Profiler Data Model JavaClass Instance B Value Value Instance A Value Value
- 41. Instances From an instance: • Who references the instance • Who does the instance reference Perform instanceof to find out: • ObjectArray • PrimitiveArray GCRoot can take forever…
- 42. Values If you ask an instance for its references, you get a list of Value objects.
- 43. Example: Member Variables Iterates over all Person objects and prints member variables.
- 47. String Extraction Strings are objects – array of characters
- 51. ArrayList
- 52. ArrayList
- 54. Noise: Ignore Internal Classes Ignore internal JVM classes
- 55. Puzzler Prints: Count: 231 • 230 entries have 1 are referenced by one other object • 1 entry is “owned” by 822 other objects
- 57. Demo App Note: Used HashSets, Arrays[][], Lists, and Vectors
- 58. Demo App
- 59. Demo App
- 60. Demo App
- 61. String Utilization • 5159 Strings in heap dump • 15 associated with data model
- 63. Data Model Leak Add logic to fire an employee:
- 64. Data Model Leak Fired Adam – shouldn’t be in the system!
- 66. Data Model Leak Leaking here!
- 67. Best Practices • Be mindful of your heap • Cache analysis on disk when processing large heaps • Heap processing is I/O bound • Not all profiler calls are the same • Look for Javadoc: Speed: normal • Maintain a list of processed objects • Easy to run in circles • Exclude JVM internal classes from analysis • Revisit graph algorithms!
- 68. Summary • Heap snapshot can be easily explored • Excellent way to verify application logic • Only way to identify deep data model/logic errors • Can be used to recover data • Generate a heap snapshot from a frozen/corrupted application and then mine
- 69. Q&A Twitter: @ctjava Email: rcuprak@gmail.com / r5k@3ds.com Blog: cuprak.info Linkedin: www.linkedin.com/in/rcuprak Slides: www.slideshare.net/rcuprak/presentations
Editor's Notes
- APM tools