JavaOne 2014: Java Debugging

Chris Bailey – IBM Runtime Monitoring and Diagnostics
30th September 2014
Java Debugging
Tools and Tricks for Troubleshooting
Document number
© 2014 IBM Corporation

Important Disclaimers
THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY.
WHILST EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION
CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED.
ALL PERFORMANCE DATA INCLUDED IN THIS PRESENTATION HAVE BEEN GATHERED IN A CONTROLLED
ENVIRONMENT. YOUR OWN TEST RESULTS MAY VARY BASED ON HARDWARE, SOFTWARE OR
INFRASTRUCTURE DIFFERENCES.
ALL DATA INCLUDED IN THIS PRESENTATION ARE MEANT TO BE USED ONLY AS A GUIDE.
IN ADDITION, THE INFORMATION CONTAINED IN THIS PRESENTATION IS BASED ON IBM’S CURRENT PRODUCT
PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM, WITHOUT NOTICE.
IBM AND ITS AFFILIATED COMPANIES SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE
USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION.
NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF:
- CREATING ANY WARRANT OR REPRESENTATION FROM IBM, ITS AFFILIATED COMPANIES OR ITS OR THEIR
SUPPLIERS AND/OR LICENSORS
2

Introduction to the speaker
 Chris Bailey
IBM Runtime Monitoring and Diagnostics Architect
- 14 years working with Java and JVM technologies
 Recent work focus:
- Java monitoring, diagnostics and troubleshooting
- Java integration into the cloud
- JavaScript monitoring, diagnostics and troubleshooting
 My contact information:
- baileyc@uk.ibm.com
- http://www.linkedin.com/in/chrisbaileyibm
- http://www.slideshare.net/cnbailey/
- @Chris__Bailey
© 2014 3 IBM Corporation

OutOfMemory

Java Memory Usage
 Maximum memory available to a process is determined by architecture
– 32bit process is limited to 2^32 bytes = 4GB
– 64bit process is limited to 2^64 bytes = 16 EiB
● Example for 32bit:
0 GB 4 GB
-Xmx
OS and C-Runtime Java Heap(s)
Native Heap
0x0 0x80000000 0xFFFFFFFF 0x40000000 0xC0000000
 Amount and location of memory used for “OS and C Runtime” determined by OS

Layout by OS
 Windows 32bit:
0 GB 4 GB
-Xmx
Java Heap(s) Native Heap
OS and C-Runtime
0x0 0x80000000 0xFFFFFFFF 0x40000000 0xC0000000
 Windows 32bit with /3GB:
Libraries
0 GB 4 GB
-Xmx
OS and C-Runtime
0x0 0x80000000 0xFFFFFFFF 0x40000000 0xC0000000
Libraries

Layout by OS
 Linux 32bit:
0 GB 4 GB
-Xmx
Kernel
0x0 0x80000000 0xFFFFFFFF 0x40000000 0xC0000000
 Linux with hugemem kernel:
0 GB 4 GB
-Xmx
K
0x0 0x80000000 0xFFFFFFFF 0x40000000 0xC0000000

Native Heap Usage
Java Heap Native Heap

OutOfMemoryErrors
 An OutOfMemoryError occurs if the Java heap or Native heap is exhausted
– Or the PermArea (for HotSpot before Java 8)
 Tools are needed to track the memory usage of each of the pools:
– Kernel/OS: OS (malloc) managed
– Native Heap: OS (malloc) managed
– Java Heap: Garbage Collection managed
 Malloc managed memory needs to be tracked at the OS level*
 Garbage Collection needs to be tracked at the Java runtime level
* The JVM can grab this information from the OS for you

OS Level Memory Monitoring
Linux: “ps” utility Windows: “perfmon” Virtual Bytes counters
#!/bin/sh
PID=$1
INTERVAL=3
# Echo time at start of monitoring.
echo timestamp = `date +%s`
# Echo the interval frequency.
echo "ps interval = $INTERVAL"
# Run the system command at intervals.
while ([ -d /proc/$PID ]) do
ps -p $PID -o pid,vsz,rss
sleep $INTERVAL
done
Can be collected to CSV file

Linux: “ps” utility Windows: “perfmon” Virtual Bytes counters
#!/bin/sh
PID=$1
INTERVAL=3
# Echo time at start of monitoring.
echo timestamp = `date +%s`
# Echo the interval frequency.
echo "ps interval = $INTERVAL"
# Run the system command at intervals.
while ([ -d /proc/$PID ]) do
ps -p $PID -o pid,vsz,rss
sleep $INTERVAL
done

Visualization of OS level memory monitoring
● Garbage Collection and Memory Visualizer (GCMV):
- Available from
● Tool to analyze Java Heap and “Native” Heap memory
● Provides Graphical Display of Data
- Allows zooming and cropping
- Allows change of axes value and units
- Allows comparison of multiple files
● Analysis and Recommendations
- Statistical summary and recommendations
- Flags errors and warnings
● Supports Linux, Windows, AIX, i5/OS and z/OS

Runtime Monitoring using JVM Tools
 IBM Health Center (IBM JVMs*)
– -Xhealthcenter
 Machine and process memory usage
 Native heap breakdown
* limited support exists for HotSpot
 Oracle Mission Control (HotSpot JVMs)
– -XX:+UnlockCommercialFeatures
– -XX:+FlightRecorder
 Machine and process memory usage

OS Level Memory Profiling
Windows: UMDH
 Takes snapshot of the current native heap
 Tracks allocating stack traces
 Second snapshot is taken and stack traces
for outstanding memory is produced
000000AD bytes in 0x1 allocations (@ 0x00000031 +
0x0000001F) by: BackTrace00468
ntdll!RtlpNtMakeTemporaryKey+000074D0
ntdll!RtlInitializeSListHead+00010D08
ntdll!wcsncat+00000224
leakyjniapp!
Java_com_ibm_jtc_demos_LeakyJNIApp_nativeMethod
 Detailed on Microsoft website:
http://support.microsoft.com/kb/268343
 Also check out “debugdiag”
Linux:Debug Malloc / Malloc Interception
 Approach 1:
– Intercept calls to malloc and free
– Create a stacks allocation table
– IBM provides a Linux Memory Tracker
 Approach 2: valgrind
valgrind --trace-children=yes --leak-check=full
LEAK SUMMARY:
definitely lost: 63,957 bytes in 69 blocks.
indirectly lost: 2,168 bytes in 12 blocks.
possibly lost: 8,600 bytes in 11 blocks.
still reachable: 5,156,340 bytes in 980 blocks.
suppressed: 0 bytes in 0 blocks.
Reachable blocks (those to which a pointer was
found) are not shown.

Garbage Collection Monitoring
 IBM JVMs:
– -verbose:gc
– -Xverbosegclog:<name>,X,Y
 Oracle JVMs:
– -XX:+PrintGCDateStamps
– -XX:+PrintGCDetails
– -XX:+PrintGCTimeStamps
– -Xloggc:<name>
 Visualization available in GCMV for both IBM and Oracle verbose:gc

Runtime GC Monitoring
– -Xhealthcenter
 Used Heap and Pause Times
 Allocation Profiling
* limited support exists for HotSpot
 Allocation profiling

Java Heap Profiling: Heap Dump Analysis
 IBM JVMs: System Dump of PHD
– -Xdump:system|heap:events=systhrow,
filter=java/lang/OutOfMemoryError
– -Xdump:system|heap:events=user
– -Xtrace:maximal=mt,trigger=
method{<name>,sysdump|heapdump}
– com.ibm.jvm.Dump.JavaDump()/HeapDump()
– Health Center
 Oracle JVMs: HPROF dump
 Visualization available in IBM Memory Analyzer
– -XX:+HeapDumpOnOutOfMemoryError
– jmap -dump:format=b
– JConsole
– Supports both IBM and HotSpot dumps
– Provides leak and footprint analysis
– Provides heap and application visualization

Java Heap Profiling: Heap Dump Analysis

Debugging from Dumps

Debugging from Dumps
● System dumps and HPROF dumps contain Object fields and values
● System dumps also have thread stacks and local variables
● Provides visibility into code flow
● Zero ongoing performance overhead
● Dump time is ~2s/GB
● Makes it possible to diagnose a much wider set of issues
● Exceptions: ability to look at data not exposed in the message
● Functional Issues: possible to look at the data being acted on

Exceptions
● java.net.ConnectException
java.net.ConnectException: Connection refused: connect
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:352)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:214)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:201)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:377)
● Stack trace in Memory Analyzer gives
access to local variables
- host Name = Bailey-W530
- port = 100

Functional Issues
● java.lang.OutOfMemoryError allocating a 1,129,665,376 byte object
at com/ibm/wsspi/security/token/WSOpaqueTokenHelper.readTokens(WSOpaqueTokenHelper.java:924)
at com/ibm/wsspi/security/token/WSOpaqueTokenHelper.createTokenHolderListFromOpaqueToken(WSOpaqueTokenHelper.java:844)
at com/ibm/ws/sib/security/impl/MessagingEngineIdentityImpl.fromBytesInDomain(MessagingEngineIdentityImpl.java:439)
at com/ibm/ws/sib/security/impl/MessagingEngineIdentityImpl.access$200(MessagingEngineIdentityImpl.java:91)
at com/ibm/ws/sib/security/impl/MessagingEngineIdentityImpl$2.run(MessagingEngineIdentityImpl.java:299)
at com/ibm/ws/sib/security/impl/MessagingEngineIdentityImpl$2.run(MessagingEngineIdentityImpl.java:293)
at com/ibm/ws/sib/security/impl/BusUtilities.doInBusDomain(BusUtilities.java:115)
● The readTokens method references a ByteArrayInputStream with a pos of 2369
● Content of ByteArrayStream around 2369:
● Which converts to 1,129,665,364

Performance

Performance Analysis
 Need to look at each of the layers of the application:
– Platform: Machine and Operating System resources
– Runtime: Java heap resources
– Application: Inefficient code, synchronization
 Recommended to look at Platform and Runtime first
– “Easy to resolve”
 Biggest gains are in the application and back end responsiveness
– OS and Java Runtime are already highly tuned and self adapting

Linux:
vmstat {interval} {samples} > vmstat.out
procs -----------memory---------- -swap- --io-- --system-- ---cpu---
r b swpd free buff cache si so bi bo in cs us sy id wa
0 0 17196 679860 1196656 2594884 0 0 1 4 0 0 0 0 100 0
0 0 17196 679868 1196656 2594884 0 0 0 40 1012 43 0 0 100 0
0 0 17196 679992 1196656 2594884 0 0 0 3 1004 43 0 0 100 0
top -H -b -c > top_threads.out
Windows: “perfmon”:
Process > Page Faults/sec
Process > %Processor Time
Network > Output Queue Length
Physical Disk > Current Disk Queue Length
top - 15:46:52 up 178 days, 4:53, 2 users, load average: 0.31,
0.08, 0.02
Tasks: 77 total, 2 running, 74 sleeping, 1 stopped, 0 zombie
Cpu(s): 24.6% us, 0.5% sy, 0.0% ni, 74.9% id, 0.0% wa, 0.0% hi,
0.0% si
Mem: 5591016k total, 5416896k used, 174120k free, 1196656k
buffers
Swap: 2104472k total, 17196k used, 2087276k free, 2594884k
cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
8502 user1 25 0 599m 466m 5212 R 99.9 8.5 0:23.92 java
7547 root 15 0 33472 2916 1144 S 0.3 0.1 778:17.98 X
1 root 16 0 656 228 192 S 0.0 0.0 0:07.43 init

Garbage Collection Monitoring
 IBM JVMs:
– -verbose:gc
– -Xverbosegclog:<name>,X,Y
 Oracle JVMs:
– -XX:+PrintGCDateStamps
– -XX:+PrintGCDetails
– -XX:+PrintGCTimeStamps
– -Xloggc:<name>
 Visualization available in GCMV for both IBM and Oracle verbose:gc

Runtime GC Monitoring
– -Xhealthcenter
 Allocation Profiling
 Allocation profiling

Application method CPU usage
– -Xhealthcenter
 CPU usage per method
 Method call graphs
 CPU usage per method
 Method call graphs

Application lock usage
– -Xhealthcenter
 Contention frequency and time
 Lock hold time
 Contention frequency and time
 Lock hold time

Questions?

IBM Developer Kits for Java
ibm.biz/javasdk
WebShere Liberty Profile
wasdev.net
IBM Bluemix
ibm.com/bluemix
IBM Developer Kits for Node.js
ibm.biz/nodesdk

www.ibm.com/developer
Discover
new technical resources.
ibm.biz/javaone2014
Visit Booth 5511 to learn about Cloud, DevOps and Mobile solutions
Develop
your coding skills.
Connect
with developers.

Copyright and Trademarks
© IBM Corporation 2013. All Rights Reserved.
IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International
Business Machines Corp., and registered in many jurisdictions worldwide.
Other product and service names might be trademarks of IBM or other companies.
A current list of IBM trademarks is available on the Web – see the IBM “Copyright and
trademark information” page at URL: www.ibm.com/legal/copytrade.shtml

JavaOne 2014: Java Debugging

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (16)

Similar to JavaOne 2014: Java Debugging

Similar to JavaOne 2014: Java Debugging (20)

More from Chris Bailey

More from Chris Bailey (20)

Recently uploaded

Recently uploaded (20)

JavaOne 2014: Java Debugging