Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Exploring Java Heap Dumps (Oracle Code One 2018)

389 views

Published on

Memory leaks are not always simple or easy to find. Heap dumps from production systems are often gigantic (4+ gigs) with millions of objects in memory. Simple spot checking with traditional tools is woefully inadequate in these situations, especially with real data. Leaks can be entire object graphs with enormous amounts of noise. This session will show you how to build custom tools using the Apache NetBeans Profiler/Heapwalker APIs. Using these APIs, you can read and analyze Java heaps programmatically to ask really hard questions. This gives you the power to analyze complex object graphs with tens of thousands of objects in seconds.

Published in: Software
  • Be the first to comment

Exploring Java Heap Dumps (Oracle Code One 2018)

  1. 1. Exploring Java Heap Dumps Ryan Cuprak
  2. 2. Java Heap Review • Java objects are stored in the heap • All objects are globally reachable in the heap • Heap is created when an application starts • Size of heap is configured using –Xmx and –Xmx • Garbage collection prunes the heap and removes objects no longer reachable • Stack memory - variable values are stored when their methods are invoked Heap Contains Everything and can be DUMPED to DISK
  3. 3. Why Analyze Heaps? • IT reports Java EE/Spring server memory footprint has grown to 9 gigs • Server app logs contain OutOfMemoryExceptions • Connections to queueing or database are exhausted • Serialized Java objects in queue are unreasonably large • Desktop application becomes unresponsive • Excessive amount of garbage collection
  4. 4. Java Heap Analysis MAT Profiler All you need is a profiler right?
  5. 5. Memory Leaks Textbook memory leaks - easy to find and fix. JConsole
  6. 6. Production Heap Dumps • 9 gigs of data • 13k classes loaded • ~ 136 million instances • ~6,000 GC roots
  7. 7. Production Heap Dumps Capture the Heap Dump and…
  8. 8. Production Heap Dumps
  9. 9. Heap Dump Panic Too much data! • Impossible to comprehend • No human way to explore the data • Application data model is too complicated
  10. 10. Real Memory Leaks Bank Account: 1231209 Owner Bob Owner JulieReport January 2018 Bank Account: 1231210 Bank Account: 1231209 Report January 2018 Owner Bob Challenge: Data looks good everywhere…
  11. 11. Real Memory Leaks Causes: • Faulty clone methods • Duplicate singletons • Accidently cached data • Cache logic bugs Complications • May NOT GROW over time (leaks gets cleaned-up) • More than one non-trivial memory leak
  12. 12. What about OQL? • OQL • Object Query Language – used for querying heaps • SQL-like language • Supports JavaScript expressions • Supported in NetBeans and VisualVM • Downside • Poorly documented and hard to use • Easy to create runaway queries
  13. 13. Heap Analysis Solution
  14. 14. NetBeans Profiler • NetBeans is open source IDE/platform • Modular architecture • Clean code base Profiler GUI Profiler API
  15. 15. NetBeans Profiler API • Parses hprof files • Creates an object model representing the hprof file • Pages data in from disk • Simple API (master in about 10 minutes) • Independent of NetBeans • Can be extract and use in any IDE – Plain old Java Talk is really about how to build a custom heap analysis tool: • To answer specific data model questions • With custom logic for your data model
  16. 16. Generating Heap Dumps
  17. 17. Generating Heap Dumps • Command line parameter: • -XX +HeapDumpOnOutOfMemoryError • Command line: • jmap –dump:format=b,file=dump.hprof <pid> • jhsdb jmap --binaryheap --pid <pid> • jcmd <pid> GC.heap_dump <file name> • Ctrl-Break Command Line
  18. 18. Generating Heap Dumps Programmatic
  19. 19. Generating Heap Dumps JMX
  20. 20. Heap Dump Warning Dumping the heap: • Takes time • Consumes diskspace • Negatively affects performance
  21. 21. Targeted Heap Dumps • Serialize object graphs from application to a file. • Read the serialized data into another tool and then programmatically create a heap dump.
  22. 22. Building a Profiler
  23. 23. Building Custom Profiler Create NetBeans Platform App Copy API src out of NetBeans
  24. 24. NetBeans Platform App
  25. 25. NetBeans Platform App Add dependency on “Java Profiler (JFluid)”
  26. 26. Profiler Sources Checkout source: https://github.com/apache/incubator-netbeans.git Profiler code: netbeans/profiler/lib.profiler/src/netbeans/lib.profiler/heap Copy heap directory
  27. 27. Which Approach? • Copying sources easiest • Most analysis apps are command line (one-offs) - Note - You don’t need the classpath of the application from which the heap was generated.
  28. 28. NetBeans Profiler API
  29. 29. Opening a Heap That’s All!
  30. 30. Heap Object Methods getJavaClassByName(String fqn) : JavaClass getAllClasses() : List getBiggestObjectsByRetainedSize(int number) : List getGCRoots(): GCRoot getInstanceByID(long instanceId) : Instance getJavaClassByID(long javaclassId) : JavaClass getJavaClassesByRegExp(String regexp) : HeapSummary getSummary() : Properties
  31. 31. System Properties
  32. 32. Heap Summary • getTotalLiveInstances() : long • getTime() : long • getTotalAllocatedBytes() : long • getTotalAllocatedInstances() : long • getTotalLiveBytes() : long
  33. 33. Exploration Starting Points • GCRoots • Threads (really GCRoots) • Class Types
  34. 34. GC Roots • Garbage Collection Root is an object that is accessible from outside the heap. • Objects that aren’t accessible from a GC Root are garbage collected • GC root categorization: • Class loaded by system class loader • Thread • Stack Local • Monitor • JNI Reference • Held by JVM
  35. 35. Garbage Collection Roots Java frame: 44 thread object: 5 JNI global: 29 sticky class: 1284
  36. 36. GCRoot Objects
  37. 37. GC Roots JNI_GLOBAL = "JNI global"; JNI_LOCAL = "JNI local"; JAVA_FRAME = "Java frame"; NATIVE_STACK = "native stack"; STICKY_CLASS = "sticky class"; THREAD_BLOCK = "thread block"; MONITOR_USED = "monitor used"; THREAD_OBJECT = "thread object"; UNKNOWN = "unknown"; INTERNED_STRING = "interned string"; FINALIZING = "finalizing"; DEBUGGER = "debugger"; REFERENCE_CLEANUP = "reference cleanup"; VM_INTERNAL = "VM internal"; JNI_MONITOR = "JNI monitor"; root.getKind() : String
  38. 38. Finding Classes • Can perform lookup using: • Fully qualified class name (ex. java.lang.String) • Class ID • Instance ID • IDs are unique to heap dump • Hash codes are not available!
  39. 39. Profiler Data Model JavaClass Instance B Value Value Instance A Value Value
  40. 40. Class Java.lang.String Java.util.List
  41. 41. Instances From an instance: • Who references the instance • Who does the instance reference Perform instanceof to find out: • ObjectArray • PrimitiveArray GCRoot can take forever…
  42. 42. Values If you ask an instance for its references, you get a list of Value objects.
  43. 43. Example: Member Variables Iterates over all Person objects and prints member variables.
  44. 44. Example: Static Variables
  45. 45. Example: References
  46. 46. String Implementation
  47. 47. String Extraction Strings are objects – array of characters
  48. 48. LinkedList Implementation
  49. 49. LinkedList Implementation
  50. 50. LinkedList Extract
  51. 51. ArrayList
  52. 52. ArrayList
  53. 53. Thread Extraction
  54. 54. Noise: Ignore Internal Classes Ignore internal JVM classes
  55. 55. Puzzler Prints: Count: 231 • 230 entries have 1 are referenced by one other object • 1 entry is “owned” by 822 other objects
  56. 56. Demo App Exploration
  57. 57. Demo App Note: Used HashSets, Arrays[][], Lists, and Vectors
  58. 58. Demo App
  59. 59. Demo App
  60. 60. Demo App
  61. 61. String Utilization • 5159 Strings in heap dump • 15 associated with data model
  62. 62. String Utilization Output
  63. 63. Data Model Leak Add logic to fire an employee:
  64. 64. Data Model Leak Fired Adam – shouldn’t be in the system!
  65. 65. Data Model Leak Found!
  66. 66. Data Model Leak Leaking here!
  67. 67. Best Practices • Be mindful of your heap • Cache analysis on disk when processing large heaps • Heap processing is I/O bound • Not all profiler calls are the same • Look for Javadoc: Speed: normal • Maintain a list of processed objects • Easy to run in circles • Exclude JVM internal classes from analysis • Revisit graph algorithms!
  68. 68. Summary • Heap snapshot can be easily explored • Excellent way to verify application logic • Only way to identify deep data model/logic errors • Can be used to recover data • Generate a heap snapshot from a frozen/corrupted application and then mine
  69. 69. Q&A Twitter: @ctjava Email: rcuprak@gmail.com / r5k@3ds.com Blog: cuprak.info Linkedin: www.linkedin.com/in/rcuprak Slides: www.slideshare.net/rcuprak/presentations

×