Debugging Native heap OOM - JavaOne 2013
Upcoming SlideShare
Loading in...5
×
 

Debugging Native heap OOM - JavaOne 2013

on

  • 1,287 views

Native out-of-memory errors happen when a Java application runs out of memory, not in the Java object heap but outside it. The cause may be memory use for native libraries, class loading, ...

Native out-of-memory errors happen when a Java application runs out of memory, not in the Java object heap but outside it. The cause may be memory use for native libraries, class loading, multithreading, working data for the Java VM, backing storage for Java objects, or other reasons. No single tool can give you all the answers, and we need to cross-reference information from multiple sources to isolate a problem. Operating system tools, Java dumps, logs, and debuggers all provide useful perspectives, and your challenge is to line them up to see the whole picture. This session works through the tools and data available on the main server platforms to give you a repeatable framework for native out-of-memory error debug.

Statistics

Views

Total Views
1,287
Views on SlideShare
1,287
Embed Views
0

Actions

Likes
5
Downloads
39
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as OpenOffice

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Debugging Native heap OOM - JavaOne 2013 Debugging Native heap OOM - JavaOne 2013 Presentation Transcript

  • Matthew Kilner – IBM Java L3 Service – Core team lead 23rd September 2013 Debugging Native Heap OOM - Tools & Techniques © 2013 IBM Corporation
  • Important Disclaimers THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY. WHILST EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. ALL PERFORMANCE DATA INCLUDED IN THIS PRESENTATION HAVE BEEN GATHERED IN A CONTROLLED ENVIRONMENT. YOUR OWN TEST RESULTS MAY VARY BASED ON HARDWARE, SOFTWARE OR INFRASTRUCTURE DIFFERENCES. ALL DATA INCLUDED IN THIS PRESENTATION ARE MEANT TO BE USED ONLY AS A GUIDE. IN ADDITION, THE INFORMATION CONTAINED IN THIS PRESENTATION IS BASED ON IBM’S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM, WITHOUT NOTICE. IBM AND ITS AFFILIATED COMPANIES SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION. NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF: - CREATING ANY WARRANT OR REPRESENTATION FROM IBM, ITS AFFILIATED COMPANIES OR ITS OR THEIR SUPPLIERS AND/OR LICENSORS 2 © 2013 IBM Corporation
  • About me  Matthew Kilner   Work for IBM – 13 years working on IBM Java • Memory Management • Class Sharing • RAS – Currently leading the Core customer support team.   Contact info – kilnerm@uk.ibm.com – Twitter: @IBMJTC – Youtube: IBM_JTC   Visit the IBM booth #5112 and meet other IBM developers at JavaOne 2013 3 © 2013 IBM Corporation View slide
  • What should you get from this talk?  An understanding of what we mean by the Native Heap.   A clear problem determination path for Native Heap OOM Errors: – How to determine you have a native heap OOM. – An outline process for determining what is causing it. © 2013 IBM Corporation View slide
  • The Java process  All applications run within the bounds of an operating system process – Java is no exception –  The JVM is subject to the same restrictions as any other application, the most pertinent being: • Addressing • OS Memory Model © 2013 IBM Corporation
  • What do we mean by addressing restrictions?  Every process has a finite address space which is dictated by the architecture it runs on. –  A 32bit architecture has an addressable range of: – 2^32 –0x00000000 – 0xFFFFFFFF –which is 4GB –  A 64bit architecture has an addressable range of: –2^64 –0x0000000000000000 – 0xFFFFFFFFFFFFFFFF –which is 16 EiB © 2013 IBM Corporation
  • What do we mean by Memory Model Restrictions?  Not all addressable memory is available to a process.  The operating system has its own requirements such as: –The kernel –The runtime support libraries  Requirements vary by Operating System both in terms of: – How much memory is needed, and – Where that memory is located  The addressable memory remaining is often referred to as “User Space” © 2013 IBM Corporation
  • A view by platform  The chart shows default maximum user space available on common 32-bit platforms 9 8 7 6 5 GiB 4 3 2 1 0 Windows 32 Windows 32 /3GB Linux 32 bit Linux 32 bit Hugemem Operating System zLinux 31 AIX 32 zOS Kernel Space User Space © 2013 IBM Corporation
  • What goes in the “User Space” Kernel Space  Java Heap  Just In Time (JIT) Data –Runtime data & executable code  Virtual Machine Resources –RAS engines & GC Infrastructure  Native & JNI Allocations  Resources to underpin Java Objects –Classes and ClassLoaders –Threads –Direct java.nio.ByteBuffers –Sockets Java Heap User Space JIT Data VM Resources Native & JNI Allocations Java Libraries © 2013 IBM Corporation
  • The Native heap  We define the native heap as: – – Native Heap = “User Space” - Maximum Heap Size –  It is the total “User Space” not reserved for backing the Java Heap.   © 2013 IBM Corporation
  • Not quite the whole story  “User Space” availability is not our only consideration when looking at native memory shortage.   Machines have to be able to back addressable memory with physical memory.   The total physical memory available on a machine is  Physical RAM + Swap Space   Problem symptoms vary based on which resource runs out. © 2013 IBM Corporation
  • Failure Symptoms  The chart shows some of the symptoms you see when a particular resource is exhausted Resource Address Space OutOfMemoryError Symptoms Physical RAM OutOfMemoryError Memory Pages Console Messages Crash/Other? Physical + Swap (Virtual Memory) Unresponsive apps Linux: OOM Killer Win/Sol: alloc's fail © 2013 IBM Corporation
  • How do I know if I have a problem?  Detecting a problem is easy if you hit one of the symptoms described previously. – OutOfMemoryError's should be fatal to your application. – Paging will cause obvious unresponsiveness or slowdown.   Early detection is possible if you monitor the size of your process. – Monitoring is also an important part of understanding any native memory issue. © 2013 IBM Corporation
  • Monitoring the size of your process  Process sizes are reported in two ways across all platforms: – Resident Size – Virtual Size  Each platform has its own methods for obtaining this information: – Windows: Performance Monitor – Linux & z/OS: ps – AIX: svmon  The IBM Garbage Collection & Memory Visualizer (GCMV) tool provides scripts and instructions in its help documentation for gathering the necessary data © 2013 IBM Corporation
  • Plotting the size of your process  Analysis of the process size is best done visually.   GCMV and Performance Monitor will plot the raw data for you.   A persistent growth in the virtual size of your process may indicate an issue.   © 2013 IBM Corporation
  • How do I identify a Native OOM from a javacore?  When any OOME occurs the IBM JVM's default configuration will write a javacore file.   This file provides several pointers to the fact you have an OOM related to the native heap: – A header that includes information on which resource cannot be allocated – Details of the current memory usage on the Java heap – A record of recent Garbage Collection activity – The stack of the thread encountering the problem    © 2013 IBM Corporation
  • The javacore header  At the top of each javacore is its header which tells you what event caused the file to be written.   Under certain conditions additional information is written at the head of the javacore when the OOME is triggered: 1TISIGINFO Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError" "Failed to create a thread: retVal -1073741830, errno 11" received   On a Java Heap OOM you will see: 1TISIGINFO Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError" "Java heap space" received © 2013 IBM Corporation
  • The javacore heap usage summary  Within the javacore you will find the MEMINFO section.  0SECTION MEMINFO subcomponent dump routine NULL ================================= 1STHEAPFREE Bytes of Heap Space Free: 3B3AE8 1STHEAPALLOC Bytes of Heap Space Allocated: 400000   If you see a large value for bytes free then it is a good indicator that you are experiencing a native heap OOME.   © 2013 IBM Corporation
  • The javacore GC history The javacore file also contains a snapshot of the most recent GC activity. Where an OOM is due to a java heap allocation failing you will see this in the data:  1STGCHTYPE 3STHSTTYPE GC History 14:12:41:476340000 GMT j9mm.101 - J9AllocateIndexableObject() returning NULL! 8000024 bytes requested for object of class 00007FD4801D6E10 from memory space 'Flat' id=00007FD480046EE8 If this entry is not present it is another good indicator you are experiencing a native heap OOME. © 2013 IBM Corporation
  • The javacore current thread The current thread within the javacore is the thread which triggered the OOME – The top stack frame contains the interesting data You can identify whether the frame is native: 3XMTHREADINFO ............... 3XMTHREADINFO3 4XESTACKTRACE 4XESTACKTRACE "main" J9VMThread:0xB8D1C600, j9thread_t:0xB8D019E4, java/lang/Thread:0x98B01960, state:R, prio=5 Java callstack: at java/lang/Thread.startImpl(Native Method) at java/lang/Thread.start(Thread.java:891) Or java: 3XMTHREADINFO ............... 3XMTHREADINFO3 4XESTACKTRACE "main" J9VMThread:0x00007FD480043D00, j9thread_t:0x00007FD4800079B0, Java callstack: at StringOOM.main(StringOOM.java:11) If it is native then this is another good indicator you are experiencing a native heap OOME. © 2013 IBM Corporation
  • How do I find out what is causing my native OOME?  It can be tricky to attribute a root cause to a Native OOME – Debug capabilities vary by platform.   The fundamental approach is the same irrespective of platform: • 1) Understand the rate of Native Memory Growth • 2) Capture multiple snapshots of data over time. • 3) Compare the snapshots and attribute growth to components. © 2013 IBM Corporation
  • Understanding the rate of memory growth The rate of memory growth can be determined from the size of your process.  Calculate the delta in virtual size of your process between data snapshots.  Other data snapshots will identify different areas of native memory growth, understanding the proportion each area contributes to the total growth is key to identifying a root cause. © 2013 IBM Corporation
  • What other data is needed? Some data is common across platforms, other data is platform specific  Common data: – Javacore files taken at regular intervals – Core files taken at regular intervals (optional)  Platform specific data: – Windows: UMDH tracing, Debug Diag tracing, VMMAP tracing – Linux: No recommended tools – AIX: Debug malloc tracing © 2013 IBM Corporation
  • The javacore NATIVEMEMINFO section From the J9 2.4 JVM the javacore file contains a NATIVEMEMINFO section. NATIVEMEMINFO subcomponent dump routine ======================================= JRE: 555,698,264 bytes / 1208 allocations +--VM: 552,977,664 bytes / 856 allocations | +--Classes: 1,949,664 bytes / 92 allocations | +--Memory Manager (GC): 547,705,848 bytes / 146 allocations | | +--Java Heap: 536,875,008 bytes / 1 allocation | | +--Other: 10,830,840 bytes / 145 allocations | +--Threads: 2,660,804 bytes / 104 allocations | | +--Java Stack: 64,944 bytes / 9 allocations | | +--Native Stack: 2,523,136 bytes / 11 allocations | | +--Other: 72,724 bytes / 84 allocations | +--Trace: 92,464 bytes / 208 allocations | +--JVMTI: 17,328 bytes / 13 allocations | +--JNI: 15,944 bytes / 32 allocations | +--Port Library: 6,824 bytes / 56 allocations | +--Other: 528,788 bytes / 205 allocations +--JIT: 1,748,808 bytes / 82 allocations | +--JIT Code Cache: 524,320 bytes / 1 allocation | +--JIT Data Cache: 524,336 bytes / 1 allocation | +--Other: 700,152 bytes / 80 allocations +--Class Libraries: 971,792 bytes / 270 allocations | +--Harmony Class Libraries: 1,024 bytes / 1 allocation | +--VM Class Libraries: 970,768 bytes / 276 allocations | | +--sun.misc.Unsafe: 69,688 bytes / 1 allocation | | +--Other: 901,080 bytes / 275 allocations Comparing the output from multiple javacores can identify areas of growth in the JVM If an identified area is a significant portion of the total memory growth it is likely the cause of the problem. © 2013 IBM Corporation
  • Javacores from earlier JDK versions If your JDK is based on a JVM earlier than the J9 2.4 JVM the javacore file doe not have a NATIVEMEMINFO section.  They do still contain valuable insight, although a little more work is required to obtain it. – The “MEMINFO subcomponent dump routine” lists a summary of memory blocks the JDK has allocated for various purposes – The “Classes subcomponent dump routine” lists a summary of classloaders and loaded classes.  By parsing and comparing this information across multiple javacore files you can determine you have any signs of a memory area growth or classloader leak. © 2013 IBM Corporation
  • What core files offer Binary core files provide the same view as the javacore but require processing with external tools: – Interactive Diagnostic Dump Explorer (IDDE). – On earlier JDK versions they provide a more accurate summary of JDK memory allocations the the javacore file.  They also provide a complete image of the process, which means: – You can inspect the content of memory – You can inspect free memory blocks (subject to platform) – You can inspect allocated memory blocks (subject to platform)  These advantages are offset by the size of the files and additional overhead of processing them. © 2013 IBM Corporation
  • Windows tooling Windows offers three excellent options for understanding your native memory growth: – UMDH – DebugDiag – VMMAP  Each has distinct usage characteristics: – UMDH is command line driven. – DebugDiag injects a tracking library into the process and parses core dumps. – VMMAP launches the application you wish to track and is GUI based. © 2013 IBM Corporation
  • Linux tooling Commercial tools are available but carry a license fee  Free tools also exist but carry a large performance overhead  We have custom built tooling that logs all calls to allocate and free memory – Building your own is possible. © 2013 IBM Corporation
  • AIX tooling  AIX provides a debug extension directly into the malloc subsystem – MALLOCDEBUG –  Enables tracing in the allocation subroutines   At termination of the process a report is generated detailing all allocations that were not freed – Some additional parsing is needed © 2013 IBM Corporation
  • What the platform tooling tells you While each platform has different tools, the end result from them is largely the same  The tools give you one or more stack traces that relate to memory allocations that have not been freed.  You are looking for the stacks that demonstrate the same or similar rates of growth as the total process size between snapshots of data. © 2013 IBM Corporation
  • What next? The next step depends on the stack that has been identified as the root cause. – – If it is native code you own: • Check to see you are releasing the memory your are allocating – –If it is native code relating to a java class: • Check that you don't have an on heap leak of the related object type • Check for known issues • Contact the JDK vendor for assistance – –If it is third party native code: • Check for any known issues • Contact the vendor for assistance © 2013 IBM Corporation
  • In Summary The process for diagnosis and root cause determination for a native OOME is as follows: 1) Understand the limitations of the platform 2) Monitor the size of the process to understand the rate of memory growth 3) Use a combination of JDK and platform diagnostics to determine the area or stack driving the growth © 2013 IBM Corporation
  • I would like to know more  Visit the IBM booth #5112   BOF 4159 - The Most Useful Tools for Debugging on Windows – Today: 9/23/13 (Monday) 7:30 PM - Hilton - Continental Ballroom 6   “Thanks for the memory” – http://www.ibm.com/developerworks/java/library/j-nativememory-linux/ – https://www.ibm.com/developerworks/java/library/j-nativememory-aix/ –  © 2013 IBM Corporation
  • Questions? –  © 2013 IBM Corporation
  • IBM@JavaOne http://ibm.co/JavaOne2013 35 IBM Confidential © 2013 IBM Corporation