Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.1
Java Flight Recorder
Behind the Scenes
Staffan Larsen
Java SE Serviceability Architect
staffan.larsen@oracle.com
@stalar
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.3
The following is intended to outline our general pro...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.4
Java Flight Recorder
§  Tracer and Profiler
§  Non...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.5
Tracer and Profiler
§  Captures both JVM and applic...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.6
Non-Intrusive
§  Typical overhead in benchmarks: 2-...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.7
Built into the JVM itself
§  Core of JFR is inside ...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.8
On-Demand Profiling
§  Start from Java Mission Cont...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.9
After-the-Fact Analysis
§  In its default mode, ver...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.10
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.11
Agenda
§  Overview of JFR
§  Demo!
§  Configurat...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.12
Configuration
§  Enable
-­‐XX:+UnlockCommercialFea...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.13
Advanced Configuration
Per Recording Session
Max ag...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.14
Recording Sessions
§  Recordings can specify exact...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.15
Many Simultaneous Recording Sessions
§  This works...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.16
JVM
How Is It Built?
§  Information gathering
–  I...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.17
“Everything Is an Event”
§  Header
§  Payload
–  ...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.18
Event Types
§  Instant
–  Single point in time
–  ...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.19
Event Meta Data
§  For every event
–  Name, Path, ...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.20
“Content Type”
§  Describes the semantics of a val...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.21
Event Definition in Hotspot
<event 	
  id="ThreadSl...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.22
Event Emission in Hotspot
JVM_Sleep(int	
  millis)	...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.23
Thread Park
<event	
  id="ThreadPark"	
  path="java...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.24
Thread Park
UnsafePark(jboolean	
  isAbsolute,	
  j...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.25
Event Graph
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.26
Event Details
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.27
Buffers
§  “Circular”
§  Designed for low content...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.28
Filtering Early
§  Enable/disable event
§  Thresh...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.29
File Format
§  Self-contained
–  Everything needed...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.30
Dynamic Runtime and Long-Running
Recordings
§  Can...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.31
Problem: Many Events Reference Classes
§  If every...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.32
Problem: When Do We Write the Class IDs?
§  IDs ne...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.33
Problem: Size of the Class List
§  Many classes ar...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.34
Problem: Leaking Memory
§  Over time many classes ...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.35
Constant Pools
§  The Class List is a special case...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.36
Checkpoints
§  At regular intervals, a “checkpoint...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.37
Optimizations
§  Fast Timestamps
–  Fast, high res...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.38
Differences vs. JRockit
§  I/O: File path, Socket ...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.39
More Information
§  Whitepaper
http://www.oracle.c...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.40
-­‐XX:+UnlockCommercialFeatures	
  	
  
-­‐XX:+Flig...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.41
@stalar
staffan.larsen@oracle.com
serviceability-de...
Copyright © 2013, Oracle and/or its affiliates. All rights reserved.42
Upcoming SlideShare
Loading in …5
×

Java Flight Recorder Behind the Scenes

3,114 views

Published on

Presentation from JavaOne 2013

Published in: Technology, News & Politics

Java Flight Recorder Behind the Scenes

  1. 1. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.1
  2. 2. Java Flight Recorder Behind the Scenes Staffan Larsen Java SE Serviceability Architect staffan.larsen@oracle.com @stalar
  3. 3. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.3 The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle s products remains at the sole discretion of Oracle.
  4. 4. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.4 Java Flight Recorder §  Tracer and Profiler §  Non-intrusive §  Built into the JVM itself §  On-demand profiling §  After-the-fact capture and analysis §  First released in 7u40 Photograph by Jeffrey Milstein http://butdoesitfloat.com/In-the-presence-of-massive-gravitational-fields-this-perfect
  5. 5. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.5 Tracer and Profiler §  Captures both JVM and application data –  Garbage Collections –  Synchronization –  Compiler –  CPU Usage –  Exceptions –  I/O §  Sampling-based profiler –  Very low overhead –  Accurate data
  6. 6. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.6 Non-Intrusive §  Typical overhead in benchmarks: 2-3% (!) §  Often not noticeable in typical production environments §  Turn on and off in runtime §  Information already available in the JVM –  Zero extra cost Photograph: Andrew Rivett http://www.flickr.com/photos/veggiefrog/3435380297/
  7. 7. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.7 Built into the JVM itself §  Core of JFR is inside the JVM §  Can easily interact with other JVM subsystems §  Optimized C++ code §  Supporting functionality written in Java
  8. 8. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.8 On-Demand Profiling §  Start from Java Mission Control –  Or from the command line §  Easily configure the amount of information to capture §  For a profile, a higher overhead can be acceptable §  When done, no overhead §  Powerful GUI for analysis
  9. 9. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.9 After-the-Fact Analysis §  In its default mode, very low overhead §  Designed to be always-on §  Uses circular buffers to store data –  In-memory or on-disk §  When an SLA breach is detected, dump the current buffers §  Dump will have information leading up to the problem n n+1 n+2n+3 n+4
  10. 10. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.10
  11. 11. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.11 Agenda §  Overview of JFR §  Demo! §  Configuration topics §  Implementation details
  12. 12. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.12 Configuration §  Enable -­‐XX:+UnlockCommercialFeatures  -­‐XX:+FlightRecorder   §  Start -­‐XX:StartFlightRecording=filename=<path>,duration=<time>   §  Or jcmd  <pid>  JFR.start  filename=<path>  duration=<time>  
  13. 13. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.13 Advanced Configuration Per Recording Session Max age of data maxage=<time> Max size to keep maxsize=<size> Global Settings (-XX:FlightRecorderOptions) Max stack trace depth stackdepth=<n> (default 64) Save recording on exit dumponexit=true Logging loglevel=[ERROR|WARN|INFO| DEBUG|TRACE] Repository path repository=<path>
  14. 14. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.14 Recording Sessions §  Recordings can specify exactly which information to capture –  ~80 events with 3 settings each §  But: two preconfigured settings –  “default”: provides as much information as possible while keeping overhead to a minimum –  “profile”: has more information, but also higher overhead §  You can configure your own favorites in Mission Control
  15. 15. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.15 Many Simultaneous Recording Sessions §  This works great §  Each session can have its own settings §  Caveat: If there are multiple sessions all of them get the union of the enabled events –  Ex: If event A is enabled in on recording, all recordings will see event A –  Ex: If event B has two different thresholds, the lower value will apply
  16. 16. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.16 JVM How Is It Built? §  Information gathering –  Instrumentation calls all over the JVM –  Application information via Java API §  Collected in Thread Local buffers ⇢ Global Buffers ⇢Disk §  Binary, proprietary file format §  Managed via JMX JFR Thread Buffers Disk JMX Java API Global Buffers GC RuntimeCompiler
  17. 17. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.17 “Everything Is an Event” §  Header §  Payload –  Event specific data Event ID End Time Start Time Thread ID StackTrace ID Payload Header Optional Size
  18. 18. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.18 Event Types §  Instant –  Single point in time –  Ex: Thread starts §  Duration –  Timing for something –  Ex: GC §  Requestable –  Happens with a specified frequency –  Ex: CPU Usage every second
  19. 19. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.19 Event Meta Data §  For every event –  Name, Path, Description §  For every payload item –  Name, Type, Description, Content Type
  20. 20. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.20 “Content Type” §  Describes the semantics of a value §  Used to correctly display the value in the UI Content Type Displayed as Bytes 4 MB Percentage 34 % Address 0x23CDA540 Millis 17 ms Nanos 4711 ns
  21. 21. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.21 Event Definition in Hotspot <event  id="ThreadSleep"        path="java/thread_sleep"    label="Java  Thread  Sleep"  ...>        <value  field="time"      type="MILLIS"      label="Sleep  Time"/>   </event>   §  XML definitions are processed into C++ classes
  22. 22. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.22 Event Emission in Hotspot JVM_Sleep(int  millis)  {        EventThreadSleep  event;          ...  //  actual  sleep  happens  here          event.set_time(millis);      event.commit();   }     §  Done! Data is now available in JFR
  23. 23. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.23 Thread Park <event  id="ThreadPark"  path="java/thread_park"              label="Java  Thread  Park"              has_thread="true"  has_stacktrace="true"  is_instant="false">        <value  type="CLASS"  field="klass"  label="Class  Parked  On"/>        <value  type="MILLIS"  field="timeout"  label="Park  Timeout"/>        <value  type="ADDRESS"  field="address"                        label="Address  of  Object  Parked"/>   </event>  
  24. 24. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.24 Thread Park UnsafePark(jboolean  isAbsolute,  jlong  time)  {      EventThreadPark  event;      JavaThreadParkedState  jtps(thread,  time  !=  0);      thread-­‐>parker()-­‐>park(isAbsolute  !=  0,  time);      if  (event.should_commit())  {          oop  obj  =  thread-­‐>current_park_blocker();          event.set_klass(obj  ?  obj-­‐>klass()  :  NULL);          event.set_timeout(time);          event.set_address(obj  ?  (TYPE_ADDRESS)obj  :  0);          event.commit();      }   }  
  25. 25. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.25 Event Graph
  26. 26. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.26 Event Details
  27. 27. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.27 Buffers §  “Circular” §  Designed for low contention Thread Local Buffers Global Buffers Disk Chunk Repository JVM Java
  28. 28. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.28 Filtering Early §  Enable/disable event §  Thresholds –  Only if duration is longer than X §  Enable/disable stack trace §  Frequency –  Sample every X
  29. 29. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.29 File Format §  Self-contained –  Everything needed to parse an event is included in the file –  New events instantly viewable in the UI §  Binary, proprietary §  Designed for fast writing §  Single file, no dependencies Event Records Event Definitions …Header
  30. 30. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.30 Dynamic Runtime and Long-Running Recordings §  Can’t leak memory –  Can’t aggregate information eternally –  Can’t keep references that prohibits class unloading §  Dynamic Runtime –  Classes can come and go –  Threads can come and go §  Solutions: Constant Pools, Checkpoints
  31. 31. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.31 Problem: Many Events Reference Classes §  If every event contained the class name as a string, we would waste lots of space §  Solution: Class IDs class  Klass  {        …        u8  _trace_id;        …   }   17event id class count java.lang.Integer 2 … 17event id class count 4711 2 …
  32. 32. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.32 Problem: When Do We Write the Class IDs? §  IDs need to be part of the file §  Classes can be unloaded at any time –  Class may not be around until end of recording §  Solution: write Class ID when classes are unloaded
  33. 33. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.33 Problem: Size of the Class List §  Many classes are loaded, not all are referenced in events, we want to save space §  Solution: when a class ID is referenced, the class is also “tagged” –  Write only tagged classes in the JFR file #define  CLASS_USED  1     void  use_class_id(Klass*  const  klass)  {      klass-­‐>_trace_id  |=  CLASS_USED;   }  
  34. 34. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.34 Problem: Leaking Memory §  Over time many classes will be tagged, the size of the class list will increase §  Solution: reset the tags each time a class list is written to disk §  We call this a “Checkpoint” §  A recording file may contain many class lists, each one is only valid for the data immediately preceding it
  35. 35. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.35 Constant Pools §  The Class List is a special case of a Constant Pool §  Classes §  Methods §  Threads §  Thread Groups §  Stack Traces §  Strings class_pool.lookup(4711)      ➞  java.lang.Integer     method_pool.lookup(1729)      ➞  java.lang.Math:pow()  
  36. 36. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.36 Checkpoints §  At regular intervals, a “checkpoint” is created in the recording §  Has everything needed to parse the recording since the last checkpoint checkpoint = events + constant pools + event meta-data events constant pools meta-data checkpoint JFR File
  37. 37. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.37 Optimizations §  Fast Timestamps –  Fast, high resolution CPU time where available –  Invariant TSC instructions §  Stack Traces –  Each event stores the thread’s stack trace –  Pool of stack traces
  38. 38. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.38 Differences vs. JRockit §  I/O: File path, Socket address §  Exceptions §  Reverse call trace view in Mission Control §  Easier configuration in Mission Control §  Deeper (configurable) stack traces §  Internal JVM differences: GC
  39. 39. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.39 More Information §  Whitepaper http://www.oracle.com/missioncontrol §  User Guide http://docs.oracle.com/javase/7/docs/technotes/guides/jfr/index.html §  Forum https://forums.oracle.com/community/developer/english/java/ java_hotspot_virtual_machine/java_mission_control
  40. 40. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.40 -­‐XX:+UnlockCommercialFeatures     -­‐XX:+FlightRecorder   Remember
  41. 41. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.41 @stalar staffan.larsen@oracle.com serviceability-dev@openjdk.java.net
  42. 42. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.42

×