Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Fire & Fury:
Java Flight Recorder &
Flamegraphs
©AmadeusITGroupanditsaffiliatesandsubsidiaries
_Java Flight Recorder
_Java Mission Control & jcmd
_Flamegraphs
_Flamegrapher
2
©AmadeusITGroupanditsaffiliatesandsubsidia...
A little bit of history
©AmadeusITGroupanditsaffiliatesandsubsidiaries
4
©AmadeusITGroupanditsaffiliatesandsubsidiaries
Have you heard about
Java Flight Recorder
©AmadeusITGroupanditsaffiliatesandsubsidiaries
_An “airplane flight recorder” for JVM
_Recording telemetry information while the JVM is running
_Has very low overhead (<...
_CPU & thread stack sampling
_Garbage collection information
_Exceptions created
_File and socket operations
_Locks that w...
_Use cases
• Profiling of running application
• Debugging or investigating issues
• Black box analysis & anomaly detection...
Java Mission Control
©AmadeusITGroupanditsaffiliatesandsubsidiaries
_Tool for managing, monitoring, profiling JVM processes
• $JAVA_HOME/bin/jmc
• JMX Console
• Java Flight Recorder
_Online ...
jcmd
©AmadeusITGroupanditsaffiliatesandsubsidiaries
_Tool to send diagnostic commands to JVM
_Replaces: jheap, jmap, jps, jstack, jstat
_Usage:
• jcmd <process id|main class>...
_jcmd VM.flags
_jcmd VM.command_line
19
©AmadeusITGroupanditsaffiliatesandsubsidiaries
Some examples
20
©AmadeusITGroupanditsaffiliatesandsubsidiaries
Oracle JDK and Open JDK, 8 vs 10
jcmd help
The following commands are av...
Flamegraphs 🔥
©AmadeusITGroupanditsaffiliatesandsubsidiaries
_Data visualization developed by Brendan Gregg
_Trying to make sense of the output provided by Linux perf
_Allows you to:
...
Stack
depth
Alphabet
SAMPLING
Main
Credits: https://speakerdeck.com/mrfoto/what-are-flame-graphs-and-how-to-read-them-rubyconfby-2017
SAMPLING
Main
Parse HTTP headers
SAMPLING
Main
Parse HTTP headers
Parse HTTP parameters
SAMPLING
Main
Call service
SAMPLING
Main
Call service
Call ORM
SAMPLING
Main
Call service
Call ORM
Call database driver
SAMPLING
Main
Call service
SAMPLING
Main
Call service
SAMPLING
Main
Build HTTP response body
SAMPLING
Main
Add HTTP response headers
SAMPLING
Main
GROUPING
GROUPING
GROUPING
Header
Driver
Body Header
Main
Request Service
ORM
Stack
depth
Alphabet
Credits: Brendan Gregg
a ( )
b ( )
c ( )
d ( )
e ( ) f ( )
g ( )
h ( )
i ( )
Credits: Brendan Gregg
a ( )
b ( )
c ( )
d ( )
e ( ) f ( )
g ( )
h ( )
i ( )
Top down shows the stack
a ( )
b ( )
c ( )
d ( )
e ( ) f ( )
g ( )
h ( )
i ( )
What’s running on CPU
Credits: Brendan Gregg
a ( )
b ( )
c ( )
d ( )
e ( ) f ( )
g ( )
h ( )
i ( )
How much (width)
Credits: Brendan Gregg
Credits: Brendan Gregg
a ( )
b ( )
c ( )
d ( )
e ( ) f ( )
g ( )
h ( )
i ( )
Widths are proportional to presence in samples
_Flamegraphs are great!
_We need a way to get them out of JFR recordings
47
©AmadeusITGroupanditsaffiliatesandsubsidiaries...
48
©AmadeusITGroupanditsaffiliatesandsubsidiaries
Java Flight Recorder & Flamegraphs
49
©AmadeusITGroupanditsaffiliatesandsubsidiaries
Java Flight Recorder & Flamegraphs
50
©AmadeusITGroupanditsaffiliatesandsubsidiaries
Java Flight Recorder & Flamegraphs
As off today, JFR is a commercial fea...
51
©AmadeusITGroupanditsaffiliatesandsubsidiaries
Java Flight Recorder & Flamegraphs
Start a new record
Using the profilin...
52
©AmadeusITGroupanditsaffiliatesandsubsidiaries
Java Flight Recorder & Flamegraphs
Dump the recording Specify location
53
©AmadeusITGroupanditsaffiliatesandsubsidiaries
Java Flight Recorder & Flamegraphs
Dumping a recording doesn’t stop it
W...
54
©AmadeusITGroupanditsaffiliatesandsubsidiaries
Java Flight Recorder & Flamegraphs
55
©AmadeusITGroupanditsaffiliatesandsubsidiaries
Java Flight Recorder & Flamegraphs
By default, creates a CPU flamegraph....
56
©AmadeusITGroupanditsaffiliatesandsubsidiaries
Java Flight Recorder & Flamegraphs
Flamegrapher 🔥
©AmadeusITGroupanditsaffiliatesandsubsidiaries
_ To generate a flamegraph:
• Run JCMD
• Start a recording
• Dump the record
• Stop the recording
• Run tool to generate S...
59
©AmadeusITGroupanditsaffiliatesandsubsidiaries
Wouldn’t it be nicer to simply
open the browser?
60
©AmadeusITGroupanditsaffiliatesandsubsidiaries
github.com/flamegrapher
61
©AmadeusITGroupanditsaffiliatesandsubsidiaries
Demo time
a ( )
b ( )
c ( )
d ( )
e ( ) f ( )
g ( )
h ( )
i ( )
For how long threads have been held waiting to acquire this lock?
Lo...
a ( )
b ( )
c ( )
d ( )
e ( ) f ( )
g ( )
h ( )
i ( )
Width is time in milliseconds
Locks flamegraphs
a ( )
b ( )
c ( )
d ( )
e ( ) f ( )
g ( )
h ( )
i ( )
Stack trace leads to an exception (or error) being created
Exception...
g ( )
a ( )
b ( )
c ( )
d ( )
e ( ) f ( )
h ( )
i ( )
Width is the number of instances of that exception (or error),
that ...
67
©AmadeusITGroupanditsaffiliatesandsubsidiaries
Demo time
68
©AmadeusITGroupanditsaffiliatesandsubsidiaries
jcmd
Mission Control
Flamegraph
JFR
Not only java
©AmadeusITGroupanditsaffiliatesandsubsidiaries
_https://github.com/google/pprof (golang)
_https://github.com/evanhempel/python-flamegraph (python)
_https://nodejs.org/en...
Questions?
@NenadBo
@lgomes
github.com/flamegrapher
Thank you!
©AmadeusITGroupanditsaffiliatesandsubsidiaries
You can follow us on:
AmadeusITgroup
amadeus.com
amadeus.com/blog
_https://www.flickr.com/photos/lsfbs/34770141071/
_https://www.flickr.com/photos/genista/20091294
_https://www.flickr.com/...
Java Flight Recorder & Flamegraphs
Java Flight Recorder & Flamegraphs
Java Flight Recorder & Flamegraphs
Java Flight Recorder & Flamegraphs
Upcoming SlideShare
Loading in …5
×

Java Flight Recorder & Flamegraphs

Presentation done at Riviera Dev 18 by Nenad Bogojevic & Leonardo F. Gomes

  • Be the first to comment

  • Be the first to like this

Java Flight Recorder & Flamegraphs

  1. 1. Fire & Fury: Java Flight Recorder & Flamegraphs ©AmadeusITGroupanditsaffiliatesandsubsidiaries
  2. 2. _Java Flight Recorder _Java Mission Control & jcmd _Flamegraphs _Flamegrapher 2 ©AmadeusITGroupanditsaffiliatesandsubsidiaries Agenda
  3. 3. A little bit of history ©AmadeusITGroupanditsaffiliatesandsubsidiaries
  4. 4. 4 ©AmadeusITGroupanditsaffiliatesandsubsidiaries Have you heard about
  5. 5. Java Flight Recorder ©AmadeusITGroupanditsaffiliatesandsubsidiaries
  6. 6. _An “airplane flight recorder” for JVM _Recording telemetry information while the JVM is running _Has very low overhead (< 1% for the long running application) _Bundled with JVM _Extensible _CPU profiling outside of safepoints 6 ©AmadeusITGroupanditsaffiliatesandsubsidiaries Java Flight Recorder
  7. 7. _CPU & thread stack sampling _Garbage collection information _Exceptions created _File and socket operations _Locks that were held for longer than 10ms _Allocations on Thread Local Allocation Buffer (TLAB) and outside TLABs. 7 ©AmadeusITGroupanditsaffiliatesandsubsidiaries Java Flight Recorder (A subset of) collected events
  8. 8. _Use cases • Profiling of running application • Debugging or investigating issues • Black box analysis & anomaly detection _Use • Can be started at launch • Or dynamically for the running project 8 ©AmadeusITGroupanditsaffiliatesandsubsidiaries Java Flight Recorder Use Cases
  9. 9. Java Mission Control ©AmadeusITGroupanditsaffiliatesandsubsidiaries
  10. 10. _Tool for managing, monitoring, profiling JVM processes • $JAVA_HOME/bin/jmc • JMX Console • Java Flight Recorder _Online from running program or offline from JFR dump 10 ©AmadeusITGroupanditsaffiliatesandsubsidiaries What is Mission Control
  11. 11. jcmd ©AmadeusITGroupanditsaffiliatesandsubsidiaries
  12. 12. _Tool to send diagnostic commands to JVM _Replaces: jheap, jmap, jps, jstack, jstat _Usage: • jcmd <process id|main class> <command> • jcmd • jcmd my.MainClass help • jcmd my.MainClass VM.command_line • jcmd 3871 Thread.print 18 ©AmadeusITGroupanditsaffiliatesandsubsidiaries jcmd jcmd
  13. 13. _jcmd VM.flags _jcmd VM.command_line 19 ©AmadeusITGroupanditsaffiliatesandsubsidiaries Some examples
  14. 14. 20 ©AmadeusITGroupanditsaffiliatesandsubsidiaries Oracle JDK and Open JDK, 8 vs 10 jcmd help The following commands are available: JFR.configure JFR.stop JFR.start JFR.dump JFR.check VM.log VM.native_memory VM.check_commercial_features VM.unlock_commercial_features ManagementAgent.status ManagementAgent.stop ManagementAgent.start_local ManagementAgent.start Compiler.directives_clear Compiler.directives_remove Compiler.directives_add Compiler.directives_print VM.print_touched_methods Compiler.codecache Compiler.codelist Compiler.queue VM.classloader_stats Thread.print JVMTI.data_dump JVMTI.agent_load VM.stringtable VM.symboltable VM.class_hierarchy VM.systemdictionary GC.class_stats GC.class_histogram GC.heap_dump GC.finalizer_info GC.heap_info GC.run_finalization GC.run VM.info VM.uptime VM.dynlibs VM.set_flag VM.flags VM.system_properties VM.command_line VM.version help The following commands are available: JFR.stop JFR.start JFR.dump JFR.check VM.native_memory VM.check_commercial_features VM.unlock_commercial_features ManagementAgent.stop ManagementAgent.start_local ManagementAgent.start GC.rotate_log Thread.print GC.class_stats GC.class_histogram GC.heap_dump GC.run_finalization GC.run VM.uptime VM.flags VM.system_properties VM.command_line VM.version help JDK 8 JDK 10 The following commands are available: GC.rotate_log Thread.print GC.class_stats GC.class_histogram GC.heap_dump GC.run_finalization GC.run VM.uptime VM.flags VM.system_properties VM.command_line VM.version help Open JDK
  15. 15. Flamegraphs 🔥 ©AmadeusITGroupanditsaffiliatesandsubsidiaries
  16. 16. _Data visualization developed by Brendan Gregg _Trying to make sense of the output provided by Linux perf _Allows you to: • Identify most frequent code-paths easily • Once you understand how it works :) 22 ©AmadeusITGroupanditsaffiliatesandsubsidiaries Flamegraphs?
  17. 17. Stack depth Alphabet
  18. 18. SAMPLING Main Credits: https://speakerdeck.com/mrfoto/what-are-flame-graphs-and-how-to-read-them-rubyconfby-2017
  19. 19. SAMPLING Main Parse HTTP headers
  20. 20. SAMPLING Main Parse HTTP headers Parse HTTP parameters
  21. 21. SAMPLING Main Call service
  22. 22. SAMPLING Main Call service Call ORM
  23. 23. SAMPLING Main Call service Call ORM Call database driver
  24. 24. SAMPLING Main Call service
  25. 25. SAMPLING Main Call service
  26. 26. SAMPLING Main Build HTTP response body
  27. 27. SAMPLING Main Add HTTP response headers
  28. 28. SAMPLING Main
  29. 29. GROUPING
  30. 30. GROUPING
  31. 31. GROUPING Header Driver Body Header Main Request Service ORM
  32. 32. Stack depth Alphabet
  33. 33. Credits: Brendan Gregg a ( ) b ( ) c ( ) d ( ) e ( ) f ( ) g ( ) h ( ) i ( )
  34. 34. Credits: Brendan Gregg a ( ) b ( ) c ( ) d ( ) e ( ) f ( ) g ( ) h ( ) i ( ) Top down shows the stack
  35. 35. a ( ) b ( ) c ( ) d ( ) e ( ) f ( ) g ( ) h ( ) i ( ) What’s running on CPU Credits: Brendan Gregg
  36. 36. a ( ) b ( ) c ( ) d ( ) e ( ) f ( ) g ( ) h ( ) i ( ) How much (width) Credits: Brendan Gregg
  37. 37. Credits: Brendan Gregg a ( ) b ( ) c ( ) d ( ) e ( ) f ( ) g ( ) h ( ) i ( ) Widths are proportional to presence in samples
  38. 38. _Flamegraphs are great! _We need a way to get them out of JFR recordings 47 ©AmadeusITGroupanditsaffiliatesandsubsidiaries Java Flight Recorder & Flamegraphs
  39. 39. 48 ©AmadeusITGroupanditsaffiliatesandsubsidiaries Java Flight Recorder & Flamegraphs
  40. 40. 49 ©AmadeusITGroupanditsaffiliatesandsubsidiaries Java Flight Recorder & Flamegraphs
  41. 41. 50 ©AmadeusITGroupanditsaffiliatesandsubsidiaries Java Flight Recorder & Flamegraphs As off today, JFR is a commercial feature of Oracle JDK Non-production usage is free
  42. 42. 51 ©AmadeusITGroupanditsaffiliatesandsubsidiaries Java Flight Recorder & Flamegraphs Start a new record Using the profiling settings
  43. 43. 52 ©AmadeusITGroupanditsaffiliatesandsubsidiaries Java Flight Recorder & Flamegraphs Dump the recording Specify location
  44. 44. 53 ©AmadeusITGroupanditsaffiliatesandsubsidiaries Java Flight Recorder & Flamegraphs Dumping a recording doesn’t stop it We need to explicitly stop it.
  45. 45. 54 ©AmadeusITGroupanditsaffiliatesandsubsidiaries Java Flight Recorder & Flamegraphs
  46. 46. 55 ©AmadeusITGroupanditsaffiliatesandsubsidiaries Java Flight Recorder & Flamegraphs By default, creates a CPU flamegraph. Needs to have Brendan Gregg’s repository cloned locally for SVG generation
  47. 47. 56 ©AmadeusITGroupanditsaffiliatesandsubsidiaries Java Flight Recorder & Flamegraphs
  48. 48. Flamegrapher 🔥 ©AmadeusITGroupanditsaffiliatesandsubsidiaries
  49. 49. _ To generate a flamegraph: • Run JCMD • Start a recording • Dump the record • Stop the recording • Run tool to generate SVG • Open in the your favorite browser 58 ©AmadeusITGroupanditsaffiliatesandsubsidiaries Why Flamegrapher?
  50. 50. 59 ©AmadeusITGroupanditsaffiliatesandsubsidiaries Wouldn’t it be nicer to simply open the browser?
  51. 51. 60 ©AmadeusITGroupanditsaffiliatesandsubsidiaries github.com/flamegrapher
  52. 52. 61 ©AmadeusITGroupanditsaffiliatesandsubsidiaries Demo time
  53. 53. a ( ) b ( ) c ( ) d ( ) e ( ) f ( ) g ( ) h ( ) i ( ) For how long threads have been held waiting to acquire this lock? Locks flamegraphs
  54. 54. a ( ) b ( ) c ( ) d ( ) e ( ) f ( ) g ( ) h ( ) i ( ) Width is time in milliseconds Locks flamegraphs
  55. 55. a ( ) b ( ) c ( ) d ( ) e ( ) f ( ) g ( ) h ( ) i ( ) Stack trace leads to an exception (or error) being created Exceptions flamegraphs
  56. 56. g ( ) a ( ) b ( ) c ( ) d ( ) e ( ) f ( ) h ( ) i ( ) Width is the number of instances of that exception (or error), that have been created Exceptions flamegraphs
  57. 57. 67 ©AmadeusITGroupanditsaffiliatesandsubsidiaries Demo time
  58. 58. 68 ©AmadeusITGroupanditsaffiliatesandsubsidiaries jcmd Mission Control Flamegraph JFR
  59. 59. Not only java ©AmadeusITGroupanditsaffiliatesandsubsidiaries
  60. 60. _https://github.com/google/pprof (golang) _https://github.com/evanhempel/python-flamegraph (python) _https://nodejs.org/en/blog/uncategorized/profiling-node-js/ (nodejs) _https://github.com/jstepien/flames/ (clojure) 70 ©AmadeusITGroupanditsaffiliatesandsubsidiaries Other languages
  61. 61. Questions?
  62. 62. @NenadBo @lgomes github.com/flamegrapher
  63. 63. Thank you! ©AmadeusITGroupanditsaffiliatesandsubsidiaries You can follow us on: AmadeusITgroup amadeus.com amadeus.com/blog
  64. 64. _https://www.flickr.com/photos/lsfbs/34770141071/ _https://www.flickr.com/photos/genista/20091294 _https://www.flickr.com/photos/neilmoralee/8457411309 _https://www.flickr.com/photos/andrewmalone/5162620357/ _https://www.flickr.com/photos/richardthomas78/59947625 74 ©AmadeusITGroupanditsaffiliatesandsubsidiaries Photo credits

×