Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Java Flight Recorder & Flamegraphs

45 views

Published on

Presentation done at Riviera Dev 18 by Nenad Bogojevic & Leonardo F. Gomes

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Java Flight Recorder & Flamegraphs

  1. 1. Fire & Fury: Java Flight Recorder & Flamegraphs ©AmadeusITGroupanditsaffiliatesandsubsidiaries
  2. 2. _Java Flight Recorder _Java Mission Control & jcmd _Flamegraphs _Flamegrapher 2 ©AmadeusITGroupanditsaffiliatesandsubsidiaries Agenda
  3. 3. A little bit of history ©AmadeusITGroupanditsaffiliatesandsubsidiaries
  4. 4. 4 ©AmadeusITGroupanditsaffiliatesandsubsidiaries Have you heard about
  5. 5. Java Flight Recorder ©AmadeusITGroupanditsaffiliatesandsubsidiaries
  6. 6. _An “airplane flight recorder” for JVM _Recording telemetry information while the JVM is running _Has very low overhead (< 1% for the long running application) _Bundled with JVM _Extensible _CPU profiling outside of safepoints 6 ©AmadeusITGroupanditsaffiliatesandsubsidiaries Java Flight Recorder
  7. 7. _CPU & thread stack sampling _Garbage collection information _Exceptions created _File and socket operations _Locks that were held for longer than 10ms _Allocations on Thread Local Allocation Buffer (TLAB) and outside TLABs. 7 ©AmadeusITGroupanditsaffiliatesandsubsidiaries Java Flight Recorder (A subset of) collected events
  8. 8. _Use cases • Profiling of running application • Debugging or investigating issues • Black box analysis & anomaly detection _Use • Can be started at launch • Or dynamically for the running project 8 ©AmadeusITGroupanditsaffiliatesandsubsidiaries Java Flight Recorder Use Cases
  9. 9. Java Mission Control ©AmadeusITGroupanditsaffiliatesandsubsidiaries
  10. 10. _Tool for managing, monitoring, profiling JVM processes • $JAVA_HOME/bin/jmc • JMX Console • Java Flight Recorder _Online from running program or offline from JFR dump 10 ©AmadeusITGroupanditsaffiliatesandsubsidiaries What is Mission Control
  11. 11. jcmd ©AmadeusITGroupanditsaffiliatesandsubsidiaries
  12. 12. _Tool to send diagnostic commands to JVM _Replaces: jheap, jmap, jps, jstack, jstat _Usage: • jcmd <process id|main class> <command> • jcmd • jcmd my.MainClass help • jcmd my.MainClass VM.command_line • jcmd 3871 Thread.print 18 ©AmadeusITGroupanditsaffiliatesandsubsidiaries jcmd jcmd
  13. 13. _jcmd VM.flags _jcmd VM.command_line 19 ©AmadeusITGroupanditsaffiliatesandsubsidiaries Some examples
  14. 14. 20 ©AmadeusITGroupanditsaffiliatesandsubsidiaries Oracle JDK and Open JDK, 8 vs 10 jcmd help The following commands are available: JFR.configure JFR.stop JFR.start JFR.dump JFR.check VM.log VM.native_memory VM.check_commercial_features VM.unlock_commercial_features ManagementAgent.status ManagementAgent.stop ManagementAgent.start_local ManagementAgent.start Compiler.directives_clear Compiler.directives_remove Compiler.directives_add Compiler.directives_print VM.print_touched_methods Compiler.codecache Compiler.codelist Compiler.queue VM.classloader_stats Thread.print JVMTI.data_dump JVMTI.agent_load VM.stringtable VM.symboltable VM.class_hierarchy VM.systemdictionary GC.class_stats GC.class_histogram GC.heap_dump GC.finalizer_info GC.heap_info GC.run_finalization GC.run VM.info VM.uptime VM.dynlibs VM.set_flag VM.flags VM.system_properties VM.command_line VM.version help The following commands are available: JFR.stop JFR.start JFR.dump JFR.check VM.native_memory VM.check_commercial_features VM.unlock_commercial_features ManagementAgent.stop ManagementAgent.start_local ManagementAgent.start GC.rotate_log Thread.print GC.class_stats GC.class_histogram GC.heap_dump GC.run_finalization GC.run VM.uptime VM.flags VM.system_properties VM.command_line VM.version help JDK 8 JDK 10 The following commands are available: GC.rotate_log Thread.print GC.class_stats GC.class_histogram GC.heap_dump GC.run_finalization GC.run VM.uptime VM.flags VM.system_properties VM.command_line VM.version help Open JDK
  15. 15. Flamegraphs 🔥 ©AmadeusITGroupanditsaffiliatesandsubsidiaries
  16. 16. _Data visualization developed by Brendan Gregg _Trying to make sense of the output provided by Linux perf _Allows you to: • Identify most frequent code-paths easily • Once you understand how it works :) 22 ©AmadeusITGroupanditsaffiliatesandsubsidiaries Flamegraphs?
  17. 17. Stack depth Alphabet
  18. 18. SAMPLING Main Credits: https://speakerdeck.com/mrfoto/what-are-flame-graphs-and-how-to-read-them-rubyconfby-2017
  19. 19. SAMPLING Main Parse HTTP headers
  20. 20. SAMPLING Main Parse HTTP headers Parse HTTP parameters
  21. 21. SAMPLING Main Call service
  22. 22. SAMPLING Main Call service Call ORM
  23. 23. SAMPLING Main Call service Call ORM Call database driver
  24. 24. SAMPLING Main Call service
  25. 25. SAMPLING Main Call service
  26. 26. SAMPLING Main Build HTTP response body
  27. 27. SAMPLING Main Add HTTP response headers
  28. 28. SAMPLING Main
  29. 29. GROUPING
  30. 30. GROUPING
  31. 31. GROUPING Header Driver Body Header Main Request Service ORM
  32. 32. Stack depth Alphabet
  33. 33. Credits: Brendan Gregg a ( ) b ( ) c ( ) d ( ) e ( ) f ( ) g ( ) h ( ) i ( )
  34. 34. Credits: Brendan Gregg a ( ) b ( ) c ( ) d ( ) e ( ) f ( ) g ( ) h ( ) i ( ) Top down shows the stack
  35. 35. a ( ) b ( ) c ( ) d ( ) e ( ) f ( ) g ( ) h ( ) i ( ) What’s running on CPU Credits: Brendan Gregg
  36. 36. a ( ) b ( ) c ( ) d ( ) e ( ) f ( ) g ( ) h ( ) i ( ) How much (width) Credits: Brendan Gregg
  37. 37. Credits: Brendan Gregg a ( ) b ( ) c ( ) d ( ) e ( ) f ( ) g ( ) h ( ) i ( ) Widths are proportional to presence in samples
  38. 38. _Flamegraphs are great! _We need a way to get them out of JFR recordings 47 ©AmadeusITGroupanditsaffiliatesandsubsidiaries Java Flight Recorder & Flamegraphs
  39. 39. 48 ©AmadeusITGroupanditsaffiliatesandsubsidiaries Java Flight Recorder & Flamegraphs
  40. 40. 49 ©AmadeusITGroupanditsaffiliatesandsubsidiaries Java Flight Recorder & Flamegraphs
  41. 41. 50 ©AmadeusITGroupanditsaffiliatesandsubsidiaries Java Flight Recorder & Flamegraphs As off today, JFR is a commercial feature of Oracle JDK Non-production usage is free
  42. 42. 51 ©AmadeusITGroupanditsaffiliatesandsubsidiaries Java Flight Recorder & Flamegraphs Start a new record Using the profiling settings
  43. 43. 52 ©AmadeusITGroupanditsaffiliatesandsubsidiaries Java Flight Recorder & Flamegraphs Dump the recording Specify location
  44. 44. 53 ©AmadeusITGroupanditsaffiliatesandsubsidiaries Java Flight Recorder & Flamegraphs Dumping a recording doesn’t stop it We need to explicitly stop it.
  45. 45. 54 ©AmadeusITGroupanditsaffiliatesandsubsidiaries Java Flight Recorder & Flamegraphs
  46. 46. 55 ©AmadeusITGroupanditsaffiliatesandsubsidiaries Java Flight Recorder & Flamegraphs By default, creates a CPU flamegraph. Needs to have Brendan Gregg’s repository cloned locally for SVG generation
  47. 47. 56 ©AmadeusITGroupanditsaffiliatesandsubsidiaries Java Flight Recorder & Flamegraphs
  48. 48. Flamegrapher 🔥 ©AmadeusITGroupanditsaffiliatesandsubsidiaries
  49. 49. _ To generate a flamegraph: • Run JCMD • Start a recording • Dump the record • Stop the recording • Run tool to generate SVG • Open in the your favorite browser 58 ©AmadeusITGroupanditsaffiliatesandsubsidiaries Why Flamegrapher?
  50. 50. 59 ©AmadeusITGroupanditsaffiliatesandsubsidiaries Wouldn’t it be nicer to simply open the browser?
  51. 51. 60 ©AmadeusITGroupanditsaffiliatesandsubsidiaries github.com/flamegrapher
  52. 52. 61 ©AmadeusITGroupanditsaffiliatesandsubsidiaries Demo time
  53. 53. a ( ) b ( ) c ( ) d ( ) e ( ) f ( ) g ( ) h ( ) i ( ) For how long threads have been held waiting to acquire this lock? Locks flamegraphs
  54. 54. a ( ) b ( ) c ( ) d ( ) e ( ) f ( ) g ( ) h ( ) i ( ) Width is time in milliseconds Locks flamegraphs
  55. 55. a ( ) b ( ) c ( ) d ( ) e ( ) f ( ) g ( ) h ( ) i ( ) Stack trace leads to an exception (or error) being created Exceptions flamegraphs
  56. 56. g ( ) a ( ) b ( ) c ( ) d ( ) e ( ) f ( ) h ( ) i ( ) Width is the number of instances of that exception (or error), that have been created Exceptions flamegraphs
  57. 57. 67 ©AmadeusITGroupanditsaffiliatesandsubsidiaries Demo time
  58. 58. 68 ©AmadeusITGroupanditsaffiliatesandsubsidiaries jcmd Mission Control Flamegraph JFR
  59. 59. Not only java ©AmadeusITGroupanditsaffiliatesandsubsidiaries
  60. 60. _https://github.com/google/pprof (golang) _https://github.com/evanhempel/python-flamegraph (python) _https://nodejs.org/en/blog/uncategorized/profiling-node-js/ (nodejs) _https://github.com/jstepien/flames/ (clojure) 70 ©AmadeusITGroupanditsaffiliatesandsubsidiaries Other languages
  61. 61. Questions?
  62. 62. @NenadBo @lgomes github.com/flamegrapher
  63. 63. Thank you! ©AmadeusITGroupanditsaffiliatesandsubsidiaries You can follow us on: AmadeusITgroup amadeus.com amadeus.com/blog
  64. 64. _https://www.flickr.com/photos/lsfbs/34770141071/ _https://www.flickr.com/photos/genista/20091294 _https://www.flickr.com/photos/neilmoralee/8457411309 _https://www.flickr.com/photos/andrewmalone/5162620357/ _https://www.flickr.com/photos/richardthomas78/59947625 74 ©AmadeusITGroupanditsaffiliatesandsubsidiaries Photo credits

×