Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

mjprof: Monadic approach for JVM profiling

1,119 views

Published on

A traditional Java profiler consists of two components. One collects profile data from the running application and the other is a visualization user interface to query the data. The profiler capabilities are limited by the data collected but also by the provided reports and functionality. This can be limiting when it comes to complex query of data. In this session we will introduce mjprof. It is an open source textual visualization profiler. It is extremely powerful as it enables you to compose a sequence of simple steps (monads) such as filters, transformations, group-by which let you slice and dice the data to pinpoint the problem. Working with mjprof resembles working with UNIX pipes. We will explain how to use this tool and present use cases and success stories, using this profiler in the last months. mjprof is written in Java and can be found on github as part of the AdoptOpenJDK project.

Published in: Software
  • Be the first to comment

mjprof: Monadic approach for JVM profiling

  1. 1. mjprof : A Monadic Approach for JVM Profiling Haim Yadid Performize-IT #Devoxx #Performance #AdoptOpenJDK @lifeyx
  2. 2. About Me: Haim Yadid • 21 Years of SW development experience • independent Performance Expert • Consulting R&D Groups • Training: Java Performance Optimization • Community leader : Java.IL #Devoxx #Performance #AdoptOpenJDK @lifeyx
  3. 3. Java.IL • Israeli Java Community and User Group • Meetup : www.meetup.com/JavaIL/ • Twitter: @java_il • ~ 400 members • Meeting every month #Devoxx #Performance #AdoptOpenJDK @lifeyx
  4. 4. Background & Motivation #Devoxx #Performance #AdoptOpenJDK @lifeyx
  5. 5. The Two Most Important Questions #Devoxx #Performance #AdoptOpenJDK @lifeyx
  6. 6. The Two Most Important Questions My wife asks me #Devoxx #Performance #AdoptOpenJDK @lifeyx
  7. 7. The Two Most Important Questions My wife asks me What are you doing? When will you be back? #Devoxx #Performance #AdoptOpenJDK @lifeyx
  8. 8. Profiler • A Performance analysis tool • Which answers questions about method calls: • E.g. : Number of invocation, Average duration,CPU consumptions Contention Memory allocation IO, Cache misses etc….. #Devoxx #Performance #AdoptOpenJDK @lifeyx
  9. 9. Statistical Profiler • A.k.a. as Sampling Profiler • Samples the treads activity every several milliseconds • Not very accurate for short tasks • Low/Medium overhead • Do not provide the number of invocations #Devoxx #Performance #AdoptOpenJDK @lifeyx
  10. 10. Instrumenting Profiler • A.k.a Tracing • Instruments the code • Knows when a method starts and when it ends • High overhead (depends on instrumentation) #Devoxx #Performance #AdoptOpenJDK @lifeyx
  11. 11. Instrumenting Profiler • A.k.a Tracing • Instruments the code • Knows when a method starts and when it ends • High overhead (depends on instrumentation) Public void foo(a) { Measure start time increase invocation count bla bla; Measure end time } Injected code #Devoxx #Performance #AdoptOpenJDK @lifeyx
  12. 12. jstack - A Poor Man’s Profiler • The most trivial sampling profiler • Part of the JDK • Connects to a running JVM and samples all the threads • Only once ! • Writes to standard output #Devoxx #Performance #AdoptOpenJDK @lifeyx
  13. 13. And it looks like this • For each thread we have the following block #Devoxx #Performance #AdoptOpenJDK @lifeyx
  14. 14. And it looks like this • For each thread we have the following block Thread name Priority Id and native id "pool-35-thread-1" prio=5 tid=10315b800 nid=0x12c1fe000 waiting on condition [12c1fd000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <107ac0140> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos ..... Locked ownable synchronizers: - None #Devoxx #Performance #AdoptOpenJDK @lifeyx
  15. 15. Now do this a Million Times! #Devoxx #Performance #AdoptOpenJDK @lifeyx
  16. 16. Now do this a Million Times! • Tons of threads • Deep stacks #Devoxx #Performance #AdoptOpenJDK @lifeyx
  17. 17. mjprof #Devoxx #Performance #AdoptOpenJDK @lifeyx
  18. 18. mjprof • A command line tool • Acts on jstack output • Extends it (enriched thread dumps) • Applies a sequence of operations (monads) to • Filter • Aggregate • Manipulate #Devoxx #Performance #AdoptOpenJDK @lifeyx
  19. 19. Prerequisites • Java installed • Set JAVA_HOME (JDK1.7 or later) • Add ${JAVA_HOME}/bin to path • Attaching to processes requires JDK and not JRE • Can run on a headless machine #Devoxx #Performance #AdoptOpenJDK @lifeyx
  20. 20. Installing mjprof • Git repo : https://github.com/AdoptOpenJDK/mjprof • Download and unzip unzip mjprof1.0-bin.zip • Or clone and build mvn install assembly:assembly • Add mjprof directory to your path #Devoxx #Performance #AdoptOpenJDK @lifeyx
  21. 21. Synopsis and Help (Smoke test) • Running mjprof without parameters will print help • It is a good way to see it is functioning #Devoxx #Performance #AdoptOpenJDK @lifeyx
  22. 22. Synopsis and Help (Smoke test) • Running mjprof without parameters will print help • It is a good way to see it is functioning ➜ mjprof git:(master) ✗ mjprof Synopsis A list of the following monads concatenated with . … Data sources: jstack/pid,[count],[sleep]/ -Generate dumps using stack path/path/ -Read thread dump from file stdin -Read thread dumps from standard input visualvm/path/ -Read profiling session from xml export of VisualVM …. #Devoxx #Performance #AdoptOpenJDK @lifeyx
  23. 23. Monads • Pipeline Building blocks • Monads can • Collect data • Filter out certain threads • Transform threads information • Aggregate threads information • Snapshot write state to a file or GUI • Concatenate with a . (period) #Devoxx #Performance #AdoptOpenJDK @lifeyx
  24. 24. Data Sources stdin file jstack jmx #Devoxx #Performance #AdoptOpenJDK @lifeyx
  25. 25. Data Sources stdin • cat mythreaddump.txt | mjprof … file jstack jmx #Devoxx #Performance #AdoptOpenJDK @lifeyx
  26. 26. Data Sources • cat mythreaddump.txt | mjprof … • mjprof path/mythreaddump.txt/ stdin file jstack jmx #Devoxx #Performance #AdoptOpenJDK @lifeyx
  27. 27. Data Sources • cat mythreaddump.txt | mjprof … • mjprof path/mythreaddump.txt/ • mjprof jstack/19873/ stdin file jstack jmx #Devoxx #Performance #AdoptOpenJDK @lifeyx
  28. 28. Data Sources • cat mythreaddump.txt | mjprof … • mjprof path/mythreaddump.txt/ • mjprof jstack/19873/ • mjprof jmx/myhost:65327/ stdin file jstack jmx #Devoxx #Performance #AdoptOpenJDK @lifeyx
  29. 29. Monads Filters Aggregations Stack frame level contains -contains group -prop -fn merge sort -sort frame waiting -locks count mergedcalles -frame top bottom trimbelow namesuf -fn -pkg jvm -jvm blocked locks running withstack output stdout gui snapshot #Devoxx #Performance #AdoptOpenJDK @lifeyx
  30. 30. Extending mjprof • mjprof is inherently extensible • Macros • A text file named macros.properties • Contains a sequence of macros • Plugins • Annotate your extension with @Plugin • Compile it and place your jar in the directory plugins under mjprof installation #Devoxx #Performance #AdoptOpenJDK @lifeyx
  31. 31. 10 mjprof Recipes #Devoxx #Performance #AdoptOpenJDK @lifeyx
  32. 32. #Devoxx #Performance #AdoptOpenJDK @lifeyx
  33. 33. Question: How many threads my process has? #Devoxx #Performance #AdoptOpenJDK @lifeyx
  34. 34. Question: How many threads my process has? path/stack.txt/ mjprof jstack/1971/ jstack/MainClass/ #Devoxx #Performance #AdoptOpenJDK @lifeyx
  35. 35. Question: How many threads my process has? path/stack.txt/ mjprof jstack/1971/ . count jstack/MainClass/ ➜ distro mjprof jstack/1971/.count 2014-11-01 10:34:10 Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.0-b70 mixed mode): Total number of threads is 1045 #Devoxx #Performance #AdoptOpenJDK @lifeyx
  36. 36. Question: How many jetty thread are working ATM? #Devoxx #Performance #AdoptOpenJDK @lifeyx
  37. 37. Question: How many jetty thread are working ATM? mjprof jstack/MyApp/ #Devoxx #Performance #AdoptOpenJDK @lifeyx
  38. 38. Question: How many jetty thread are working ATM? mjprof jstack/MyApp/ . contains/name,qtp/ #Devoxx #Performance #AdoptOpenJDK @lifeyx
  39. 39. Question: How many jetty thread are working ATM? mjprof jstack/MyApp/ . contains/name,qtp/ . contains/state,RUN/ ➜ 04112014 mjprof jstack/My4tApp/.contains/name,qtp/ 2014-11-04 20:05:03 Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.0-b70 mixed mode): "qtp817406040-17-selector-ServerConnectorManager@69b87e8d/3" prio=5 ….3 runnable [0x000000011bc5f000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:202) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x0000000780209080> (a sun.nio.ch.Util$2) - locked <0x0000000780209070> (a java.util.Collections$UnmodifiableSet) #Devoxx #Performance #AdoptOpenJDK @lifeyx
  40. 40. Question: How many jetty thread are working ATM? mjprof jstack/MyApp/ . contains/name,qtp/ . contains/state,RUN/ . count ➜ 04112014 mjprof jstack/My4tApp/.contains/name,qtp/ 2014-11-04 20:05:03 Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.0-b70 mixed mode): "qtp817406040-17-selector-ServerConnectorManager@69b87e8d/3" prio=5 ….3 runnable [0x000000011bc5f000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:202) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x0000000780209080> (a sun.nio.ch.Util$2) - locked <0x0000000780209070> (a java.util.Collections$UnmodifiableSet) #Devoxx #Performance #AdoptOpenJDK @lifeyx
  41. 41. Question: Which threads are running my logic ? #Devoxx #Performance #AdoptOpenJDK @lifeyx
  42. 42. Question: Which threads are running my logic ? mjprof jstack/MyApp/ . contains/stack,myapp/ 2014-11-04 20:11:55 Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.0-b70 mixed mode): "Indexing daemon" prio=5 tid=0x00007fb306bf4800 nid=0x6307 waiting on condition [0x000000011c171000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at com.myapp.IndexManagement$1.run(IndexManagement.java:148) "main" prio=5 tid=0x00007fb303809000 nid=0x1303 in Object.wait() [0x0000000105500000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at com.myapp.MyAppApp.main(MyAppApp.java:18) #Devoxx #Performance #AdoptOpenJDK @lifeyx
  43. 43. Question: Stack-trace is cluttered, how do I shorten it ? #Devoxx #Performance #AdoptOpenJDK @lifeyx
  44. 44. Question: Stack-trace is cluttered, how do I shorten it ? mjprof jmx/MyApp/ . #Devoxx #Performance #AdoptOpenJDK @lifeyx
  45. 45. Question: Stack-trace is cluttered, how do I shorten it ? mjprof jmx/MyApp/ . frame/mycompany/ #Devoxx #Performance #AdoptOpenJDK @lifeyx
  46. 46. Question: Stack-trace is cluttered, how do I shorten it ? mjprof jmx/MyApp/ . frame/mycompany/ -frame/grizzly/ #Devoxx #Performance #AdoptOpenJDK @lifeyx
  47. 47. Question: Stack-trace is cluttered, how do I shorten it ? mjprof jmx/MyApp/ . frame/mycompany/ -frame/grizzly/ top/3/ #Devoxx #Performance #AdoptOpenJDK @lifeyx
  48. 48. Question: How can I group threads by state ? #Devoxx #Performance #AdoptOpenJDK @lifeyx
  49. 49. Question: How can I group threads by state ? mjprof jmx/MyApp/ . group/state/ . gui #Devoxx #Performance #AdoptOpenJDK @lifeyx
  50. 50. Question: How do I aggregate several thread dumps #Devoxx #Performance #AdoptOpenJDK @lifeyx
  51. 51. Question: How do I aggregate several thread dumps mjprof jmx/MyApp,10,1000/ #Devoxx #Performance #AdoptOpenJDK @lifeyx
  52. 52. Question: How do I aggregate several thread dumps mjprof jmx/MyApp,10,1000/ . group/state/ merge #Devoxx #Performance #AdoptOpenJDK @lifeyx
  53. 53. Question: How do I aggregate several thread dumps mjprof jmx/MyApp,10,1000/ . group/state/ merge . #Devoxx #Performance #AdoptOpenJDK @lifeyx
  54. 54. Question: How can I combine thread dumps from several processes? #Devoxx #Performance #AdoptOpenJDK @lifeyx
  55. 55. Question: How can I combine thread dumps from several processes? mjprof jmx/myhost1.com:5467/ . jmx/myhost2.com:5467/ . group/state/ . gui #Devoxx #Performance #AdoptOpenJDK @lifeyx
  56. 56. #Devoxx #Performance #AdoptOpenJDK @lifeyx
  57. 57. How do I measure thread pool CPU consumption? #Devoxx #Performance #AdoptOpenJDK @lifeyx
  58. 58. How do I measure thread pool CPU consumption? mjprof jmxc/MyApp/ #Devoxx #Performance #AdoptOpenJDK @lifeyx
  59. 59. How do I measure thread pool CPU consumption? mjprof jmxc/MyApp/ . contains/name,COFFEE/ #Devoxx #Performance #AdoptOpenJDK @lifeyx
  60. 60. How do I measure thread pool CPU consumption? mjprof jmxc/MyApp/ . contains/name,COFFEE/ . -pkg . group "*" count=7 tid=* cpu_ns=1164536000 wall_ms=4804 %cpu=24.24 java.lang.Thread.State: TIMED_WAITING 100.00% [7/7] at Thread.run(Thread.java:744) 100.00% [7/7] at Sleeper.run(Sleeper.java:51) 100.00% [7/7] at Sleeper.loopForever(Sleeper.java:37) 100.00% [7/7] at Sleeper.longerStackTrace(Sleeper.java:32) 100.00% [7/7] at Sleeper.longerStackTrace(Sleeper.java:32) 100.00% [7/7] at Sleeper.longerStackTrace(Sleeper.java:32) 100.00% [7/7] at Sleeper.longerStackTrace(Sleeper.java:32) 100.00% [7/7] at Sleeper.longerStackTrace(Sleeper.java:32) 100.00% [7/7] at Sleeper.longerStackTrace(Sleeper.java:30) 100.00% [7/7] at Recurse.singleCycle(Recurse.java:16) 100.00% [7/7] at Recurse.doRecursion(Recurse.java:25) 100.00% [7/7] at Recurse.doRecursion(Recurse.java:25) #Devoxx #Performance #AdoptOpenJDK @lifeyx
  61. 61. Question: Which threads are blocked by the lock I hold #Devoxx #Performance #AdoptOpenJDK @lifeyx
  62. 62. Question: Which threads are blocked by the lock I hold mjprof path/threaddump.txt/ . blocked . contains/stack,0x00000007aab51b38/ #Devoxx #Performance #AdoptOpenJDK @lifeyx
  63. 63. Question: What is the overall locking state of my application #Devoxx #Performance #AdoptOpenJDK @lifeyx
  64. 64. Question: What is the overall locking state of my application mjprof path/threaddump.txt/ . contains/stack,myapp/ locks . "CPU3_5" prio=5 tid=0x00007fc4f4889800 nid=0x5703 waiting for monitor… java.lang.Thread.State: BLOCKED (on object monitor) - waiting to lock <0x00000007aad98988> (a java.lang.Object) "CPU3_4" prio=5 tid=0x00007fc4f404a800 nid=0x5503 waiting for monitor… java.lang.Thread.State: BLOCKED (on object monitor) - waiting to lock <0x00000007aad98988> (a java.lang.Object) "CPU3_3" prio=5 tid=0x00007fc4f4049800 nid=0x5303 waiting on … java.lang.Thread.State: TIMED_WAITING (sleeping) - locked <0x00000007aad98988> (a java.lang.Object) #Devoxx #Performance #AdoptOpenJDK @lifeyx
  65. 65. 3#! #Devoxx #Performance #AdoptOpenJDK @lifeyx
  66. 66. Thanks / Contact Me lifey@performize-it.com blog.performize-it.com www.performize-it.com https://github.com/lifey http://il.linkedin.com/in/haimyadid @lifeyx #Devoxx #Performance #AdoptOpenJDK @lifeyx

×