Production Time Profiling and Diagnostics on the JVM

Copyright © 2018, Oracle and/or its affiliates. All rights reserved.
Production-Time Profiling
and Diagnostics on the JVM
Marcus Hirt
Consulting Member of Technical Staff
OMC
October, 2018

Agenda
• Introduction to various OpenJDK serviceability technologies
• Simple examples
• Pointers and hints
• Examples on using several technologies together
2

The JVM…
• Translating our byte code to machine code since 1995
– …is a highly performant piece of technology (JIT)
• Gives us the illusion of infinite memory
– …many GC algorithms to choose from (Parallel, CMS, G1, ZGC)
• But it is so much more!
– Plenty of serviceability technologies!
– Makes running programs on the JVM easier to:
• Monitor
• Manage
• Diagnose
• Profile
3

Serviceability Technologies available in OpenJDK 11
Disclaimers:
• Not going to talk about all of them
– There are even more of them available, SA etc
• They will all get a shallow treatment – not enough time!
• This talk will provide you with a starting point…
– …with a production time focus
– …with examples to get up and running
– …with a few examples on using them together
4

Attach
5

Dynamic Attach (com.sun.tools.attach)
• Attach to locally running JVMs
• Get the system properties
• Load agents (JVMTI, JPLIS)
• Load the (JMX) management agent
• If we downcast to HotSpotVirtualMachine…
– Dump hprof format heap dumps
– Execute Diagnostic Commands (JCMD)…
– …which allows us to control the flight recorder
6

Attach Example
7
Print system properties
public static void main(String[] args) throws Exception {
String pid = AttachUtils.checkPid(args);
VirtualMachine vm = VirtualMachine.attach(pid);
System.out.println(vm.getSystemProperties());
vm.detach();
}

Attach Example
8
Start Local Management Agent
System.out.println(vm.startLocalManagementAgent());
vm.detach();
}
JMX Service URL

Attach Example
9
Naughty little hobbitses edition
…
HotSpotVirtualMachine hsvm = (HotSpotVirtualMachine) vm;
String result = AttachUtils.readFromStream(hsvm.dumpHeap(outFileStr));
System.out.println(result);
vm.detach();
}
…
HotSpotVirtualMachine hsvm = (HotSpotVirtualMachine) vm;
String result = AttachUtils.readFromStream(hsvm.executeJCmd("Thread.print"));
System.out.println(result);
vm.detach();
}

JMX
10

JMX
• The platform MBean server / java.lang.management (JSR-174, Java 1.5)
• Access to key metrics in the Java platform
– Memory/GC metrics
– CPU load
• Access to third-party metrics
• Remote access through built in agent
• More info if being naughty little hobbitses
11

JMX Code Example
12
In Process
public static void main(String[] args) {
System.out.println(ManagementFactory.getRuntimeMXBean()
.getSystemProperties());
}

JMX Code Example
13
Other Process
…
JMXConnector connector = JMXConnectorFactory
.connect(jmxServiceUrl);
RuntimeMXBean runtimeMXBean = ManagementFactory
.newPlatformMXBeanProxy(connector.getMBeanServerConnection(),
ManagementFactory.RUNTIME_MXBEAN_NAME, RuntimeMXBean.class);
System.out.println(runtimeMXBean.getSystemProperties());
}

JMX Example
14
Nasty little hobbitses CPU Load
…
JMXConnector connector =
JMXConnectorFactory.connect(jmxServiceUrl);
OperatingSystemMXBean osMXBean =
ManagementFactory.newPlatformMXBeanProxy(
connector.getMBeanServerConnection(),
ManagementFactory.OPERATING_SYSTEM_MXBEAN_NAME,
com.sun.management.OperatingSystemMXBean.class);
System.out.println("JVM process load: " + osMXBean.getProcessCpuLoad());
System.out.println("System load: " + osMXBean.getSystemCpuLoad());
System.out.println("Process cpu time: " + osMXBean.getProcessCpuTime() /
1000_000.0d + " ms");
}

Copyright © 2018, Oracle and/or its affiliates. All rights reserved. 15
Invoking Diagnostic Commands
private final static String DC_OBJECT_NAME = "com.sun.management:type=DiagnosticCommand";
…
MBeanServerConnection connection = connector.getMBeanServerConnection();
String result = (String) connection.invoke(new ObjectName(DC_OBJECT_NAME),
"threadPrint",
new Object[] { new String[0] },
new String[] { String[].class.getName() });

JMX Concerns
• Content type for attributes not available
– Units descriptor use not standardized
– Some tooling (JMC) provides it own metadata client side
• Memory pressure/overhead
– Don’t get excessive amounts of data too often
– Marshalling/Unmarshalling to/from CompositeData
– Be wary of stack traces or similar data structures
(no compact representation e.g. a constant pool)
16

JMX Fun Facts
• JRockit had the world’s first JVM Management API (JMAPI)
• The funny looking long Base64 encoded JMXServiceURL used to connect to
a locally running JVM -> serialized connection stub
17

Perf Counters
18

The JVM Perf Counters
• Access internal instrumentation buffers maintained by the JVM
• Local access only
– Remote access example later
• Entire API internal and restricted
– Very naughty little hobbitses indeed…
19

Example Listing all of the Perf Counters
20
public static void main(String[] args) throws IOException {
Perf p = Perf.getPerf();
ByteBuffer buffer = p.attach(Integer.parseInt(pid), "r");
PerfInstrumentation perfInstrumentation =
new PerfInstrumentation(buffer);
for (Counter counter : perfInstrumentation.getAllCounters()) {
System.out.println(
String.format("%s = %s [Variability: %s, Units: %s]",
counter.getName(),
String.valueOf(counter.getValue()),
counter.getVariability(), counter.getUnits()));
}
}

Some counter examples
…
sun.cls.appClassBytes = 44119 [Variability: Monotonic, Units: Bytes]
sun.cls.appClassLoadCount = 51 [Variability: Monotonic, Units: Events]
sun.cls.appClassLoadTime = 28404804 [Variability: Monotonic, Units: Ticks]
…
sun.property.sun.boot.library.path = /Library/Java/JavaVirtualMachines/jdk-
11.jdk/Contents/Home/lib [Variability: Constant, Units: String]
…
21

JVMTI
22

JVMTI (The JVM Tool Interface)
• Is basically the old JVMDI + JVMPI
• Native library interface
– Import a header
– Compile a library/DLL for all supported platforms
– Package it all up (different distributable for different platforms)
23

Pros and Cons
• On the plus side:
– Agent has access to platform specific (native) APIs
– Powerful general API
• On the negative side:
– Agent needs to be compiled for the various platforms of interest
– User must use a library compiled for the specific platform he is using
– The general API must support all kinds of tooling
-> sometimes harder to find efficient solutions for very specific tasks
– Hard to write correct JVMTI code, JNI code for sure
24

JDI
25

JDI (Java Debug Interface)
• Java has a comprehensive debugger architecture (JPDA)
• JDI is a Java API for attaching to Java processes as a debugger client
– Socket communication (dt_socket, host & port, over JDWP)
– Local attach (process identrified by PID)
– Shared Memory (Win32 only, dt_shmem, identified by unique name)
• Powerful, for example
– Allows for setting breakpoints
– Allows for accessing contents on the stack (parameters, local variables)
– Allows for executing code in the remote JVM
26

JDI Examples
• Three examples provided by Oracle
• Trace
– Displays traces of program execution (< 600 lines)
– Uses most of basic JDI à good starting point
• Jdb
– Command line debugger
• Javadt
– The beginnings of a GUI debugger.
• Simple scaled down Hello World example at the java-svc GitHub repo
27

Production Time?
• No, not really
– People setting breakpoints in your production code?
– Executing arbitrary code?
– Performance penalties (severe, especially if evaluating expressions in breakpoints)
• Sometimes the one remaining option
– Start a node with dt_socket
– Assumes that you either have direct access (!), or access to establish an SSH bridge
– Attach using your favourite remote debugger in your IDE
28

Example
java -Xdebug
-Xrunjdwp:transport=dt_socket,
address=9876,
server=y,
suspend=y
MyClass
ssh -L 9876:localhost:9876 <user>@<host> -p <sshd port>
29
For completeness…

JPLIS
30

JPLIS
• Provides the concept of an Agent
– Agent can have a premain method
– An agent can also have an agentmain method (run when loaded dynamically)
– The agent is a jar, and the manifest specifies the Premain-/Agent-Class
• Provides an interface for manipulating the byte code of a class
– java.lang.Instrumentation
– ClassFileTransformer used when class is loaded/retransformed
• Can be loaded from the command line with –javaagent
• Can be loaded dynamically with attach
31
Java Programming Language Instrumentation Services

Transforming a Class
32
java HelloWorld
load HelloWorld.class
run HelloWorld.main(…)

Transforming a Class
33
java -javaagent:myagent.jar HelloWorld
Agent.premain registers Transformer
load HelloWorld.class
byte[] Transformer.transform(transform(ClassLoader loader,
String className,…, byte[] classfileBuffer))
run HelloWorld.main(…)

Transformation
• The transformation usually is done using a bytecode engineering library
• Multiple to chose from
– ASM
– BCEL
– javassist
– ByteBuddy
– …
• Simple example using ASM in the java-svc repo
– List methods to be instrumented in a text file
– Will time the time spent in the method body for the listed methods and print it out
34

Manifest decides what is allowed (capabilities)
• Premain-Class: se.hirt.examples.svc.jplis.ExampleAgent
• Agent-Class: se.hirt.examples.svc.jplis.ExampleAgent
• Can-Redefine-Classes: false
• Can-Retransform-Classes: true
• Can-Set-Native-Method-Prefix: false
35

Example - premain
36
public static void premain(String agentArguments, Instrumentation instrumentation) {
try {
List<TransformDescriptor> probes = readProbes(agentArguments);
instrumentation.addTransformer(new ExampleTransformer(probes));
} catch (IOException e) {
getLogger().log(Level.SEVERE,
"Could not initialize agent! Agent will not be active!", e);
}
}

Example - ASM
37
private void emitTimingPrologue() {
mv.visitMethodInsn(INVOKESTATIC, "java/lang/System", "nanoTime",
"()J", false);
localVariableIndexTime = newLocal(Type.LONG_TYPE);
mv.visitVarInsn(LSTORE, localVariableIndexTime);
}
private void emitTimingEpilogue() {
mv.visitMethodInsn(INVOKESTATIC, "java/lang/System", "nanoTime",
"()J", false);
mv.visitVarInsn(LLOAD, localVariableIndexTime);
mv.visitInsn(LSUB);
mv.visitVarInsn(LSTORE, localVariableIndexTime);
}

Be Careful
• Redefinition limitations
– Can’t add/remove methods
– No hierarchy changes
– Can’t change (method/class) modifiers
– Can’t add/remove fields
• You may want to generate classes on the fly
– Defining your generated classes? à sun.misc.Unsafe to the rescue
• Careful not to alter the execution of the original application
– Inadvertedly alter the execution path, e.g. by throwing exceptions
• Deopts!
38

JCMD
39

JCMD – Java Command
• Sends command to local Java Process
• Retrieves a String response
• Executes Diagnostic Commands (as mentioned in previous examples)
• Launcher usually both forward and backwards compatible
– Use JDK 11 to send commands to JDK 7
– Use JDK 7 to send commands to JDK 11
– Use the help command to find out what commands are available
• Commands available to control the JDK Flight Recorder
40

Example
$ jcmd
89840 org.codehaus.plexus.classworlds.launcher.Launcher jetty:run
97845 jdk.jcmd/sun.tools.jcmd.JCmd
$ jcmd 89840 help
89840:
The following commands are available:
JFR.stop
JFR.start
JFR.dump
JFR.check
…
41
List all processes

Example
$ jcmd 89840 help JFR.stop
89840:
JFR.stop
Stops a JFR recording
Impact: Low
Permission: java.lang.management.ManagementPermission(monitor)
Syntax : JFR.stop [options]
Options: (options must be specified using the <key> or <key>=<value>
syntax)
name : [optional] Recording name,.e.g "My Recording" (STRING, no
default value)
recording : [optional] Recording number, see JFR.check for a list of
available recordings (JLONG, -1)
42
Get help

Note
• Providing the PID 0 will execute the command in all JVMs
• As mentioned, remotable over JMX (DiagnosticCommandMBean)
• Often used to control the JDK Flight Recorder from scripts or from the
command line
• Only for processes with the same effective user as the one running jcmd
43

JFR
44

Production Time Profiling and Diagnostics
45
“The big challenge is no longer really performance. The big
challenge is profiling, and especially profiling in production.”
- Tony Printezis, JVM engineer, Twitter
(Devoxx 2015, ”Life of a Twitter JVM Engineer”, 49:49)

"Java Mission Control is the best
profiler by far.“
- T Jake Luciani, PMC Cassandra
“Java Mission Control is my current
favourite profiler for the Oracle JVM."
- Nitsan Wakart, Azul
“… Our real-time messaging products can
publish millions of messages a second to many
thousands of connections - only JMC can keep
up with this level of load."
- Phil Aston, Product Architect, Push Technology
"JMC not only saves time trying to resolve
performance issues and bugs, it can give you a
detailed view on your application you cannot get
with other commercial profilers"
- Peter Lawrey, CEO, Chronicle Software
“Once more @javamissionctrl is saving my day! "
- Michael Nitschinger, SDK Engineer, Couchbase
"For the record: Java Mission Control is the best
profiler ever, I use it daily, and so should you"
- Marcus Lagergren, Lead Architect, Nasdaq

"I am ACS engineer since 2008, delivering local Middleware support to several
customers. Since I started to work with Java/JRockit Mission Control, it
became a key tool for my work, helping me to troubleshooting, identifying root
causes and bottlenecks, and also for doing proactive follow up services to
customers. Without it, I would be blind."
- Iratxe Etxebarria, Oracle (ACS)
"In Fusion we create hundreds of thousands of Flight Recordings, and we can
figure out 95% of the issues using nothing but the recordings."
- Joe Albowicz, Oracle (Fusion Application Development)

Flight Recorder
• High Performance Event Recorder
• Built into the JVM
• Binary recordings
• Chunks
– Self contained
– Self describing
101

OpenJDK has Flight Recorder!
• OpenSourced, and available in JDK 11!
• Records information about the JVM and the application running
• Can be extended with custom (contextual) information (Event API)
• Can be controlled from Diagnostic Commands (jcmd)
• Can be controlled from JMX (diagnostic commands and dedicated API)
• Also POJO local Java API for controlling the recorder
49

Rich Data
• Method profiling (multiple variants)
– asyncGetCalltrace (no need to safepoint)
– New event in JDK 10 for calls out to native
• Allocation profiling
• Latency profiling
• GC information
• Method compilation information
• Old object sampling
– New event in JDK 10 for sorting out memory leaks
50

Low Overhead
• Access to data already collected in the runtime
• Thread local native buffers
• Invariant TSC for time stamping
• Method profiling data even from outside safe-points
• Faster and more accurate allocation profiling
(scalarization not undone by profiler)
• No overhead when event is turned off (and on the critical path)
51

Java API Events
JVM Events
Event
Event
Event
Event
ThreadBuffer
Global Buffer
Global Buffer Global Buffer Global Buffer
Global Buffer
Global Buffer Global Buffer Global Buffer
Copied into
JFR File

Example
53
Adding a custom event
public class HelloJfr {
@Label("Hello World!")
static class HelloWorldEvent extends Event {
@Label("Message")
String message;
}
public static void main(String... args) throws IOException {
HelloWorldEvent event = new HelloWorldEvent();
event.message = "Hello World!";
event.commit();
}
}

Another Custom Event
54
@Label("Fibonacci")
@Description("An example event")
@Category("Fibonacci")
@Enabled(true)
public final class FibonacciEvent extends Event {
@Label("Fibonacci Number")
@Description("The number in the sequence calculated, i.e. the n:th number")
public int number;
@Label("Fibonacci Sum")
@Description("The value of the n:th number")
public long value;
}

API for Parsing Flight Recorder Files
public class ParserExample {
long maxNanos = Long.MIN_VALUE;
RecordedEvent maxEvent = null;
for (RecordedEvent event : RecordingFile.readAllEvents(Paths.get(args[0]))) {
if (event.getEventType().getName().equals("se.hirt.jfr4junit.TestEvent")) {
long nanos = event.getDuration();
if (nanos > maxNanos) {
maxNanos = nanos;
maxEvent = event;
}
}
}
System.out.printf("Longest running test was: %s for %dsn",
maxEvent.getValue("displayName"), maxNanos / 1_000_000);
System.out.println("Event was:n" + maxEvent);
}
}

Control API and Parsing Example
public class RecordAndConsume {
Path path = Paths.get(args[0]);
try (Recording recording = new Recording()) {
recording.setName("Fibonacci Recording");
recording.start();
recording.enable(FibonacciEvent.class);
for (int n = 0; n < 50; n++) {
FibonacciEvent event = new FibonacciEvent();
event.number = n;
event.begin();
event.value = Fibonacci.fibonacciIterative(n);
event.commit();
}
recording.stop();
recording.dump(path);
for (RecordedEvent event : RecordingFile.readAllEvents(path)) {
int number = event.getValue("number");
long value = event.getValue("value");
System.out.printf("fibonacci(%d) = %d (time: %dns)n", number, value, event.getDuration());
}
}
}
}

JMC Core
57

JDK Mission Control
• Two parts – the application and the core libraries
• The core libraries contain
– A backwards compatible JFR parser capable of running on JDK 7 and above
– Can parse JFR recordings from Oracle JDK 7u40 and above
– An API for filtering and aggregating data (from, for example JFR)
– An API for working with quantities and units of measurement
– An API for analysing JFR recordings
• See my jmc-jshell repo for examples
58

Example
59
Calculating the Standard Deviation for Monitor Enter events
IItemCollection events = JfrLoaderToolkit.loadEvents(new File(args[0]));
IItemCollection monitorEnterEvents = events.apply(JdkFilters.MONITOR_ENTER);
IQuantity stddev = monitorEnterEvents.getAggregate(
Aggregators.stddev(JfrAttributes.DURATION));
System.out.println(stddev.displayUsing(IDisplayable.AUTO)));
}

Example
60
Getting an HTML report
String report = JfrHtmlRulesReport.createReport(
JfrLoaderToolkit.loadEvents(new File(args[0])));
System.out.println(report);
}

Putting it together
61

Some Already Mentioned Examples
• Using JCMD from scripts to control the JDK Flight Recorder
– Dumping already defined recordings at a regular interval, moving them somewhere
and pre-process them/run custom rules on them
– Do a m minute profiling recording every h hours
• Using JMX to remotely to access JCMD
62

Using JMX + PerfCounters
• Expose the perf counters through a Dynamic MBean
63
https://github.com/thegreystone/java-svc/tree/master/perfcounters

Using JMX + JPLIS + JFR
• Use JFR to record JMX data directly to JFR
– Stacktraces not interesting, so don’t record them
– Let the user configure what to record
– Use not only the data type, but let the user add content type information
• Build it as a JPLIS agent
– Not really using BCI at all…
– …but can now be loaded on the command line using –javaagent
– …and loaded dynamically into a running process (for example through attach)
• Now have access to e.g. third party information available in JMX, but in the
context of everything else recorded in JFR!
Prototype: https://github.com/thegreystone/jmx2jfr
64

Using JPLIS + JFR to Inject Events
• Building a JPLIS agent to inject JFR events around methods
– Dynamically adds JFR events to method entries/exits (either or both)
– Method timing using efficient JFR events!
– Selectively records parameters and/or return values
– Optionally records stack traces, again using efficient JFR events
– Ongoing project in the JMC project (jmc agent)
• First prototype available with JMC 7
• Uses ASM to add well defined byte code
• Generates support classes as needed, and defines them using sun.misc.Unsafe…
Prototype here: http://hg.openjdk.java.net/jmc/jmc
(under core/ org.openjdk.jmc.agent)
65

Summing it all up…
66

Summary
• Java has a rich and powerful set of serviceability features
• Some of them are rather suited for production use
• Using them together makes them even more useful
• They are rather easy to get started with
• One place to start playing with them is here:
https://github.com/thegreystone/java-svc
67

Safe Harbor Statement
The preceding is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole discretion of Oracle.

Other JMC-/JFR-related Presentations
Day Time Title Location
Monday 10:30 Contributing to the Mission Control OpenJDK Project MW 2004
Tuesday 11:30 Three Productive Ways to Use Open Source Java Flight Recorder and
Java Mission Control
MW 2024
Tuesday 12:30 Flight Recorder in OpenJDK MW 2018
Tuesday 14:30 Fast and Furious: Java Flight Recorder and Flame Graphs MW 2020
Wednesday 10:30 Production-Time Profiling and Diagnostics on the JVM MW 2004
Wednesday 12:30 OpenJDK Mission Control: The Hands-on-Lab MW 2001A
Wednesday 16:30 Diagnose Your Microservices: OpenTracing/Oracle APM Cloud MW 2011
Thursday 11:00 Getting Started with the (Open Source) JDK Mission Control MW 2014
69

Production Time Profiling and Diagnostics on the JVM

More Related Content

What's hot

Similar to Production Time Profiling and Diagnostics on the JVM

Recently uploaded

Production Time Profiling and Diagnostics on the JVM