How to monitor Java application and JVM performance with Flight Recorder and Mission Control. Starts with a discussion of general JVM performance considerations like GC, JIT and threads.

1. Java Performance Monitoring
With Flight Recorder
and Mission Control
Presented by Simon Ritter, Deputy CTO | Azul

2. 2
Agenda
• Understanding Java (JVM) Performance
• How Flight Recorder works
• Using the Flight Recorder API
• JVM analysis with Mission Control
• Performance monitoring demonstration

3. Understanding Java (JVM) Performance

4. 4
Statically Compiled Application
Application Executable
Operating System
Hardware (Physical Machine)

5. 5
Java Virtual Machine (JVM) Application
Application Executable
Operating System
Hardware (Physical Machine)
Java Virtual Machine

6. 6
Java Virtual Machine
• Managed runtime environment
• Several key features
o Bytecode interpretation
o Just-in-Time (JIT) compilation
o Automatic memory management with Garbage Collection (GC)
o Thread synchronisation

7. 7
JVM Performance Graph: Ideal

8. 8
JVM Performance Graph: Reality
Bytecodes
interpreted
C1 JIT plus
profiling C2 JIT with
Deoptimisations
Steady optimised
state
GC
Pauses

9. 9
JIT Compilation
• Code is compiled as the application is running
• Full profiling data of execution to date is available
• Accurate view of all loaded classes
o Any method to be called is unambiguous
o More aggressive method inlining
• Ability to use speculative optimisations
• Additional load on CPU/threads
o Can introduce stop-the-world pauses when code is compiled

10. 10
Speculative Optimisations
• Profiling data shows that value (so far) has never been greater than 9
10
int computeMagnitude(int value) {
if (value > 9)
bias = computeBias(value);
else
bias = 1;
return Math.log10(bias + 99);
}

11. 11
Speculative Optimisations
• Assume that, based on profiling, value will continue to be less than 10
11
int computeMagnitude(int value) {
if (value > 9)
uncommonTrap(); // Deoptimise
return 2; // Math.log10(100)
}

12. 12
Speculative Optimisations
• Our analysis shows that over half of performance benefits of JIT compilation in the JVM come from
speculative optimisations
• Therefore critical for good performance
• For best results the number of deoptimisations should be minimised
12

13. 13
Java Memory Management
• Java does not use explicit pointers
o No arithmetic manipulation: no buffer overruns
o Implicit pointers to objects
• The JVM handles all memory management
o Allocation of space for objects on the heap
o Reclaiming space (garbage collecting) when no longer required
• Sounds simple but the reality is a bit more complicated

14. 14
Garbage Collection Misconceptions
• It's better than you think
o GC is extremely efficient, much better than malloc/free
o Dead objects cost nothing to collect
o GC will find all dead objects (including cyclic references)
• It’s worse than you think
o Yes it does stop for ~1s per live Gb (except Azul Prime)
o GC does not prevent memory leaks
o Just because you eliminate pauses in a 20 minute run doesn’t mean they’ve gone
• Rule of thumb: don't try to be smarter than the JVM
o Object pooling is mostly pointless (except heavyweight objects like threads and JDBC connections)
o Put modest effort into eliminating object creation
o Zero garbage approach is (probably) wrong

15. 15
Garbage Collection Terminology
• Parallel: Work can be distributed amongst multiple threads
• Concurrent: GC work happens while application threads are running
• Stop-the-world: Application threads are paused while GC work is done (for safety)
• Safepoints: Point at which a thread can pause for GC work to happen
• Global safepoint: Point at which all threads have reached a safepoint

16. 16
Garbage Collection Basics
• There are multiple algorithms for performing GC
o Different approaches affect application performance in different ways
• For HotSpot (OpenJDK) most algorithms use a multi-generational heap
o This makes sense as the weak generational hypothesis holds for almost all Java applications
Eden Space
From
Space
To
Space
Tenured Space
Young Generation
Old Generation
Object
allocation
Survivor spaces
Object copying

17. 17
HotSpot Garbage Collectors
• Serial
• Parallel
• Concurrent Mark Sweep (CMS)
• G1
• Shenandoah
• ZGC

18. 18
Azul Prime JVM
• C4: Continuous Concurrent Compacting Collector
• Uses loaded value barrier (LVB)
o A read barrier to all object allocations
• Guarantees all objects used by application is marked
o No accidental garbage collection
• Guarantees application will always receive the correct location for the object in heap
o No danger of making changes that get lost during object relocation
• No stop-the-world full compacting GC
• Eliminates GC pauses

19. 19
Threads
• JVM specification defines a threading model
• Implementations can map to native threads, or implement their own model
o Early JVM had green threads for platforms that didn't have threads (Windows 95)
• JVM threads can run at one of 10 priorities (10 is highest)
• Threads are guaranteed to receive CPU time only when all higher priority threads are blocked
o This is important
o Lower priority threads might receive some CPU time (or might not)

20. 20
How Flight Recorder Works

21. 21
JDK Flight Recorder and Mission Control
• Originally developed by the JRockit team
o Flight Recorder intended to help development of the JVM
o Mission Control
• After Oracle acquired Sun Microsystems both were rebranded
o Java Flight Recorder
o Java Mission Control
• Contributed by Oracle to OpenJDK 11
o Part of convergence between Oracle JDK and OpenJDK
o Rebranded (again) to JDK Flight Recorder and JDK Mission Control
• Backported to OpenJDK 8 by Azul engineers

22. 22
Flight Recorder Basics
• A profiling and event collection framework
o Gathers low-level information in the JVM
o Assists with analysing application behaviour
• Very low-overhead
o Much of the information collected is used internally by the JVM
o Does not rely on the JVM TI
o Suitable for use in production
o Typically less than 1% overhead
o Default stack depth is 64. More than that will degrade performance.

23. 23
Flight Recorder Architecture
JVM
Flight Recorder
Operating System
Application
JFR
Recording
Mission Control

24. 24
Flight Recorder Control
• -XX:-FlightRecorder
o Turns off flight recorder, on by default
• -XX:FlightRecorderOptions=option1=value1,option2=value2
• -XX:StartFlightRecording=option1=value1,option2=value2
o Configuration of basic options
o Whether to record, how long to wait before recording, where to dump data, etc.
• Backport of JDK 11 JFR control
java -XX:StartFlightRecording=dumponexit=true,filename=/tmp/dump.jfr -jar LoadGenerator.jar

25. 25
Flight Recorder Remote Access
• Remote JVM needs to be configured for connection
-Djava.rmi.server.hostname=10.0.0.3
-Dcom.sun.management.jmxremote.port=7091
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
• For Linux system, hosts file must include IP address
Do not use these
in production!

26. 26
Continuous Recordings
• Data can be written to disk
• Or held in memory and dumped when requested
o Uses a global buffer
o Circular usage
o When buffer fills up, oldest data is discarded to store new data
o Default is 462848 bytes
o Configurable via globalbuffersize option
• Event recording configured via default.jfc file
o $JAVA_HOME/jre/lib/jfr (JDK 8)
o $JAVA_HOME/lib/jfr (JDK 11)
o Contains over 130 types of event that are recorded

27. 27
Flight Recorder and jcmd
• Use jcmd to control Flight Recorder on running JVM
• jcmd <jvm-pid> JFR.start
• jcmd <jvm-pid> JFR.start defaultrecording=true
• jcmd <jvm-pid> JFR.dump name=1 filename=/tmp/load.jfr
• jcmd <jvm-pid> JFR.stop name=1

28. 28
Flight Recorder Event Structure
• Event ID
• Timestamp (CPU ticks)
• Duration (CPU ticks)
• Thread ID
• StackTrace ID
• Event Specific Payload

29. 29
Events Generated By The Java Runtime
• Examples of the over 130 events that can be recorded
Category Event
Environment Command line arguments
JDK version information
OS
CPU
Java execution environment I/O: File and network
Thread sampling
JVM operations Class loading
GC
JIT compilation

30. 30
Event Data Flow
Application
Events
JVM
Events
Event
Event
Event
Event
Event
...
Thread buffer
Disk
Chunk
Repository
Global
Buffer
Global
Buffer
Global
Buffer
Copy when
buffer full
Copy once per second
or when full
* when streaming

31. 31
JFR File Format
• Compact (not compressed) data format
• Little-endian base (LEB) 128 format
• Self-describing
o Meta-data describes how to interpret events
o Links to preceding events as necessary
File
Header
User
Events
Constant Pool
Event
User
Events
Constant Pool
Event
User
Events
Constant Pool
Event
Metadata
Event

32. 32
Event Filtering
• Filter by
o Type
o Name
o Duration
• Powerful feature for highlighting relevant information

33. 33
Event Correlation
• Events from multiple levels are included in the event stream
o Application
o Java runtime libraries
o JVM
o Operating system
• Ideal for in-depth analysis
o Identify issue in application, progress to libraries, JVM effects and even OS/hardware

34. 34
Using the Flight Recorder API

35. 35
Creating An Event
import jdk.jfr.*;
public class AzulEvent extends Event {
}
public void importantMethod() {
// Critical code executed here
}

36. 36
Using An Event
import jdk.jfr.*;
public class AzulEvent extends Event {
}
public void importantMethod() {
AzulEvent event = new AzulEvent();
event.begin();
// Critical code executed here
event.end();
event.commit();
}

37. 37
Making An Event More Useful
import jdk.jfr.*;
@Name("com.azul.azulevent")
@Label("Azul Event")
public class AzulEvent extends Event {
@Label("Message")
private final String message;
@Label("Code")
private final int code;
public AzulEvent(String message, int code) {
this.message = message;
this.code = code;
}
}

38. 38
Looking At Events
public void importantMethod() {
AzulEvent event = new AzulEvent("Critical code event", 99);
...
}
$ java ... -XX:StartFlightRecording:filename=azul.jfr ...
$ jfr print --events com.azul.azulevent azul.jfr
com.azul.azulevent {
startTime = 14:36:32.103
duration = 169.619 us
message = "Critical code event"
code = 99
eventThread = "main" (javaThreadId = 1)
stackTrace = [
eventtest.EventTest.main(String[]) line: 21
]
}

39. 39
JFR Annotations

40. 40
Consuming Events
try (var rs = new RecordingStream()) {
rs.enable("jdk.JavaMonitorEnter").withThreshold(Duration.ofMillis(10));
rs.onEvent("jdk.JavaMonitorEnter", event -> {
System.out.println(event.getClass("monitorClass"));
});
rs.enable("jdk.CPULoad").withPeriod(Duration.ofSeconds(1));
rs.onEvent("jdk.CPULoad", event -> {
System.out.println(event.getFloat("machineTotal"));
});
rs.start();
}

41. 41
JVM Analysis With Mission Control

42. 42
Mission Control Basics
• Advanced set of tools for analysing data recorded by JDK Flight Recorder
• JVM browser to select application to interact with
o Also supports remote JVMs via JMX
• Options available after connecting to a JVM
o Use JMX console
o Initiate flight recording
- fixed time
- continuous

43. 43
JMX Console
• Connects to JVM
• Monitors in real time
• Uses Java Management Extensions (JMX)
• Rules for actions can be created
o e.g. Collect data if CPU use exceeds 90%

44. 44
JMX Live Monitoring

45. 45
Mission Control Rules
• Pre-defined rules or create your own
• Trigger action predicated on condition

46. 46
Comprehensive Data Analysis
• All aspects of application and JVM behaviour

47. Performance Monitoring Demo

48. Summary

49. 49
Conclusions
• The JVM can often deliver better performance than statically compiled code
• Understanding the way the JVM works is key to getting the best performance
• Flight Recorder is a powerful way to collect information on how an application is working
• Mission Control provides an excellent graphical interface for performance analysis

50. 50
Azul Platform Core / Azul Zulu Builds of OpenJDK
• Enhanced build of OpenJDK source code
o Fully TCK tested
o JDK 6, 7, 8, 11, 13 and 15
o TLS1.3, Flight Recorder backports
• Wide platform support:
o Intel 64-bit Windows, Mac, Linux
o Intel 32-bit Windows and Linux
• Real drop-in replacement for Oracle JDK
o Many enterprise customers
o No reports of any compatibility issues

51. 51
Azul Platform Core Extended Support
• Backporting of bug fixes and security patches from supported OpenJDK release
• Azul Zulu Builds of OpenJDK 8 supported until December 2030
• LTS releases have 9 years active + 2 years passive support
• JDK 15 is a Medium Term Support release
o Bridge to next LTS release (JDK 17)
o Supported until 18 months after JDK 17 release

52. Thank You.
Simon Ritter, Deputy CTO
sritter@azul.com
1.650.230.6600
@speakjava