Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Need for speed: tools for JVM tuning for Geecon Prague 2016

0 views

Published on

Monitoring your production JVMs is extremely important. Though, not always you have a luxury of having direct access to production servers, which suffer from performance peaks, memory leaks or concurrency issues. Not always you are able to catch JVM issues without an ability to debug the code in-place. There are quite many tools that can help analyzing performance issues of your JVMs based on real-time monitoring data as well as on logs available only post-mortem. This presentation will give an overview of these tools.

Published in: Technology

Need for speed: tools for JVM tuning for Geecon Prague 2016

  1. 1. 01
  2. 2. About me 02
  3. 3. Andrey Adamovich Java/Groovy developer (Sometimes) application performance analyst Co­author of Groovy 2 Cookbook Co­organizer of @latcraft and @devternity • • • • 03
  4. 4. Contact details E­mail: andrey@aestasit.com Linkedin: http://www.linkedin.com/in/andreyadamovich Lanyrd: http://lanyrd.com/profile/andrey­adamovich GitHub: https://github.com/aadamovich SO: http://stackoverflow.com/users/162792/andrey­adamovich Twitter: @codingandrey, @aestasit • • • • • • 04
  5. 5. What's in this presentation? 05
  6. 6. The tools 06
  7. 7. The game 07
  8. 8. Rules Picture is shown You guess what the next topic is First one to guess gets a sticker! • • • 08
  9. 9. Let's try! 09
  10. 10. 10
  11. 11. Java's (Hash)Map 11
  12. 12. 12
  13. 13. BigData 13
  14. 14. Let the games begin! 14
  15. 15. 15
  16. 16. GC Pauses 16
  17. 17. What are the good parameters?17
  18. 18. So many options.... 18
  19. 19. So many options.... > java ‐XX:+PrintFlagsFinal ‐version | wc ‐l java version "1.8.0_45" Java(TM) SE Runtime Environment (build 1.8.0_45‐b15) Java HotSpot(TM) 64‐Bit Server VM (build 25.45‐b02, mixed mode) 714 01. 02. 03. 04. 05. 19
  20. 20. So many options.... > java ‐XX:+PrintFlagsFinal ‐version | wc ‐l java version "1.7.0_80" Java(TM) SE Runtime Environment (build 1.7.0_80‐b15) Java HotSpot(TM) 64‐Bit Server VM (build 24.80‐b11, mixed mode) 668 01. 02. 03. 04. 05. 20
  21. 21. So many options.... > java ‐XX:+PrintFlagsFinal ‐version | wc ‐l java version "1.6.0_45" Java(TM) SE Runtime Environment (build 1.6.0_45‐b06) Java HotSpot(TM) 64‐Bit Server VM (build 20.45‐b01, mixed mode) 670 01. 02. 03. 04. 05. 21
  22. 22. So many options.... > java ‐XX:+PrintFlagsFinal ‐version | grep GC | wc ‐l 65 01. 02. 22
  23. 23. Gazzillion permutations Imagine all those parameters are Boolean 2^65 combinations 2^65 ~ 10^21 = 1 sextillion • • • 23
  24. 24. Use cases Web application Background job Streaming/reactive application Low latency application • • • • 24
  25. 25. Memory tuning? ‐Xms<heap size>[g|m|k]  ‐Xmx<heap size>[g|m|k] ‐XX:PermSize=<perm gen size>[g|m|k]  ‐XX:MaxPermSize=<perm gen size>[g|m|k] ‐Xmn<young size>[g|m|k] ‐XX:SurvivorRatio=<ratio> 01. 02. 03. 04. 05. 06. 25
  26. 26. Leverage multi­core ‐XX:+UseConcMarkSweepGC  ‐XX:+UseCMSInitiatingOccupancyOnly  ‐XX:+ScavengeBeforeFullGC  ‐XX:+CMSScavengeBeforeRemark ‐XX:+CMSConcurrentMTEnabled ‐XX:CMSInitiatingOccupancyFraction=70 01. 02. 03. 04. 05. 06. 26
  27. 27. G1 ‐XX:MaxGCPauseMillis=200 ‐XX:G1HeapRegionSize=n ‐XX:G1NewSizePercent=5 ‐XX:G1MaxNewSizePercent=60 ‐XX:InitiatingHeapOccupancyPercent=45 01. 02. 03. 04. 05. 27
  28. 28. GC logging ‐XX:+PrintGCDetails  ‐XX:+PrintGCDateStamps  ‐Xloggc:<file‐path>  ‐XX:+PrintGCCause ‐XX:+PrintGCApplicationStoppedTime  ‐XX:+PrintGCApplicationConcurrentTime  ‐XX:+PrintSafepointStatistics  ‐XX:+PrintHeapAtGC 01. 02. 03. 04. 05. 06. 07. 08. 28
  29. 29. Log analysis 29
  30. 30. Open source: GCViewer 30
  31. 31. Comercial: Censum 31
  32. 32. Next riddle 32
  33. 33. 33
  34. 34. Memory dumps 34
  35. 35. How to take? On out of memory: ‐XX:+HeapDumpOnOutOfMemoryError  ‐XX:HeapDumpPath=<path to dump>`date`.hprof Or manually: jmap ‐dump:file=<path to dump> <java process id> 01. 02. 01. 35
  36. 36. jhat 36
  37. 37. Eclipse MAT 37
  38. 38. OQL 38
  39. 39. Useful characteristics Object Shallow Size Object Retained Size Stale Objects (not connected to GC roots) • • • 39
  40. 40. Take several dumps At application start At load time (several) At quiet time • • • 40
  41. 41. Comercial YourKit jProfiler • • 41
  42. 42. Next riddle 42
  43. 43. 43
  44. 44. Thread dumps 44
  45. 45. Situations Dead locks 100% CPU load Hanging I/O • • • 45
  46. 46. jstack jstack <java process pid>01. 46
  47. 47. JMX Use MBean with name  java.lang:type=Threading Call  getAllThreadIds Loop through all retrieved IDs and call  getThreadInfo • • • 47
  48. 48. The cheapest profiler for ((a=1; a <= 60; a++)) do    jstack 7789 > thread‐dump.$a   sleep 1   echo $a done 01. 02. 03. 04. 05. 06. 48
  49. 49. Thread dump analysis 49
  50. 50. Thread dump structure 50
  51. 51. Thread states I A thread can be in one of the following states: NEW  A thread that has not yet started is in this state. RUNNABLE  A thread executing in the Java virtual machine is in this state. BLOCKED  A thread that is blocked waiting for a monitor lock is in this state. • • • 51
  52. 52. Thread states II WAITING  A thread that is waiting indefinitely for another thread to perform a particular action is in this state. TIMED_WAITING  A thread that is waiting for another thread to perform an action for up to a specified waiting time is in this state. TERMINATED  A thread that has exited is in this state. • • • 52
  53. 53. TDA 53
  54. 54. Samurai TDA 54
  55. 55. IBM TMDA 55
  56. 56. Use custom scripts 56
  57. 57. Next riddle 57
  58. 58. 58
  59. 59. Debugging 59
  60. 60. We need it when we have... Non­reproducable issues Hard­to­understand code • • 60
  61. 61. Enable debugging ‐agentlib:jdwp=transport=dt_socket,server=y,address=133701. 61
  62. 62. Use your IDE (Eclipse) 62
  63. 63. Use your IDE (Intellij IDEA) 63
  64. 64. jdb 64
  65. 65. jdb 65
  66. 66. YouDebug 66
  67. 67. YouDebug YouDebug is a non­interactive debugger scripted by Groovy to assist remote troubleshooting and data collection to analyze failures. “ 67
  68. 68. YouDebug http://youdebug.kohsuke.org/ https://github.com/kohsuke/youdebug • • 68
  69. 69. YouDebug (buggy class) public class SubStringTest {   public static void main(String[] args) {     String s = someLengthComputationOfString();     System.out.println(s.substring(5));   }   private static String someLengthComputationOfString() {     ...;   } } 01. 02. 03. 04. 05. 06. 07. 08. 09. 69
  70. 70. YouDebug (SubStringMonitor.ydb) breakpoint("com.acme.SubStringTest",3) {   println "s="+s; } 01. 02. 03. 70
  71. 71. YouDebug (run debugger) > java ‐jar youdebug.jar ‐socket 1337 SubStringMonitor.ydb s=test 01. 02. 71
  72. 72. Next riddle 72
  73. 73. 73
  74. 74. Decompiling 74
  75. 75. Java Decompiler 75
  76. 76. Java Decompiler 76
  77. 77. Issues Lack of support for latest JVM features Lack of support for other JVM langauges Obfuscation is always an issue • • • 77
  78. 78. Next riddle 78
  79. 79. 79
  80. 80. Profiling 80
  81. 81. Types of profiling Allocation profiling CPU profiling Lock profiling • • • 81
  82. 82. Types of profiling Sampling Intrumented • • 82
  83. 83. VisualVM 83
  84. 84. Honest Profiler 84
  85. 85. Flight Recorder 85
  86. 86. Flight Recorder ‐XX:+UnlockCommercialFeatures ‐XX:+FlightRecorder ‐ XX:StartFlightRecording=duration=60s,filename=myrecording.jfr • • • 86
  87. 87. Comercial YourKit jProfiler • • 87
  88. 88. Next riddle 88
  89. 89. 89
  90. 90. Benchmarks 90
  91. 91. JMH MH is a Java harness for building, running, and analysing nano/micro/milli/macro benchmarks written in Java and other languages targetting the JVM. “ 91
  92. 92. JMH: example package org.openjdk.jmh.samples; import org.openjdk.jmh.annotations.Benchmark; import org.openjdk.jmh.runner.Runner; import org.openjdk.jmh.runner.RunnerException; import org.openjdk.jmh.runner.options.Options; import org.openjdk.jmh.runner.options.OptionsBuilder; 01. 02.03. 04. 05. 06. 07. 92
  93. 93. JHM: example public class JMHSample_01_HelloWorld {   @Benchmark   public void wellHelloThere() {     // this method was intentionally left blank.   } 01. 02. 03. 04. 05. 93
  94. 94. JMH: example   public static void main(String[] args) throws RunnerException {     Options opt = new OptionsBuilder()             .include(JMHSample_01_HelloWorld.class.getSimpleName()             .forks(1)             .build();     new Runner(opt).run();   } } 01. 02. 03. 04. 05. 06. 07. 08. 94
  95. 95. JMH: example > mvn clean install > java ‐jar target/benchmarks.jar JMHSample_01_HelloWorld 01. 02. 95
  96. 96. JMH: annotations @Benchmark @BenchmarkMode @CompilerControl  @Fork @Group @GroupThreads @Level @Measurement @Mode @OperationsPerInvocation @OutputTimeUnit @Param @Scope @Setup @State @TearDown @Threads @Timeout @Warmup 01. 02. 03. 04. 05. 06. 96
  97. 97. JMH: output 97
  98. 98. Load testing tools jMeter (http://jmeter.apache.org/) ab (http://httpd.apache.org/docs/2.4/programs/ab.html) gatling (http://gatling.io/) Faban (http://faban.org/) • • • • 98
  99. 99. Next riddle 99
  100. 100. 100
  101. 101. Leak detection 101
  102. 102. Open source Leak Prevention Library Leak Canary • • 102
  103. 103. Comercial Plumbr AppDynamics NewRelic • • • 103
  104. 104. Next riddle 104
  105. 105. 105
  106. 106. Monitoring 106
  107. 107. jstat Monitors Java Virtual Machine (JVM) statistics. This command is experimental and unsupported. “ 107
  108. 108. JMX is your friend! 108
  109. 109. VisualVM (again) 109
  110. 110. Mission Control 110
  111. 111. Metrics collection 111
  112. 112. JMXTrans This is effectively the missing connector between speaking to a JVM via JMX on one end and whatever logging / monitoring / graphing package that you can dream up on the other end. “ 112
  113. 113. jHiccup jHiccup is a non­intrusive instrumentation tool that logs and records platform "hiccups" ­ including the JVM stalls that often happen when Java applications are executed and/or any OS or hardware platform noise that may cause the running application to not be continuously runnable. “ 113
  114. 114. jHiccup > java ‐javaagent:jHiccup.jar MyProgram01. 114
  115. 115. jHiccup 115
  116. 116. Drop Wizard Metrics Metrics provides a powerful toolkit of ways to measure the behavior of critical components in your production environment. “ 116
  117. 117. Netflix Hysterix 117
  118. 118. Stagemonitor 118
  119. 119. 119
  120. 120. Single JVM? 120
  121. 121. Really? 121
  122. 122. 122
  123. 123. Dead JVM 123
  124. 124. 124
  125. 125. Many JVMs 125
  126. 126. Open­source tools Graphite Graphana Logstash Kibana Graylog2 Fluentd • • • • • • 126
  127. 127. Comercial tools NewRelic Plumbr AppDynamics Dynatrace Takipi Hyperic • • • • • • 127
  128. 128. Final riddle 128
  129. 129. 129
  130. 130. Links 130
  131. 131. Links: GC http://spin.atomicobject.com/2014/09/03/visualizing­garbage­ collection­algorithms/ http://techblog.netflix.com/2013/05/garbage­collection­ visualization.html http://blog.sokolenko.me/2014/11/javavm­options­production.html https://blog.codecentric.de/en/2013/10/useful­jvm­flags­part­7­cms­ collector/ • • • • 131
  132. 132. Links: GC http://blog.ragozin.info/2011/09/hotspot­jvm­garbage­collection­ options.html http://blog.tier1app.com/2014/11/19/understand­garbage­collection­ log/ http://www.tagtraum.com/gcviewer­vmflags.html • • • 132
  133. 133. Links: GC http://fasterj.com/tools/gcloganalysers.shtml https://github.com/chewiebug/GCViewer http://www.jclarity.com/censum/ • • • 133
  134. 134. Links: Thread dumps http://www.javacodegeeks.com/2012/03/jvm­how­to­analyze­thread­ dump.html http://www.slideshare.net/buzdin/thread­dump­analysis http://docs.oracle.com/javase/7/docs/api/java/lang/Thread.State.html • • • 134
  135. 135. Links: Tools https://github.com/akullpp/awesome­java https://github.com/aragozin/jvm­tools http://openjdk.java.net/projects/code­tools/jmh/ • • • 135
  136. 136. Links: Tools http://youdebug.kohsuke.org/ https://java.net/projects/tda http://samuraism.jp/samurai/en/index.html • • • 136
  137. 137. Links: Tools https://github.com/jmxtrans/jmxtrans http://www.stagemonitor.org/ http://jamonapi.sourceforge.net/ https://github.com/Netflix/Hystrix • • • • 137
  138. 138. Links: Tools http://www.eclipse.org/tptp/home/documents/tutorials/profilingtool/profilingexample https://github.com/mjiderhamn/classloader­leak­prevention • • 138
  139. 139. Links: Tools jMeter (http://jmeter.apache.org/) ab (http://httpd.apache.org/docs/2.4/programs/ab.html) gatling (http://gatling.io/) Faban (http://faban.org/) • • • • 139
  140. 140. Reading material 140
  141. 141. Java Performance 141
  142. 142. Java Performance Companion 142
  143. 143. Java Performance: The Defenetive Guide 143
  144. 144. Java Performance And Scalability 144
  145. 145. This is it! 145
  146. 146. Questions? 146
  147. 147. Thank you! 147
  148. 148. Devternity 1­2 of December, 2016, Riga Software Excellence Conference 4 tracks, 30+ speakers, 6 workshops http://devternity.com • • • • 148

×