Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Way Improved :) GC Tuning Confessions - presented at JavaOne2015

3,222 views

Published on

This is my latest presentation of the GC Tuning Confessions series. I presented this deck at JavaOne 2015

Published in: Software

Way Improved :) GC Tuning Confessions - presented at JavaOne2015

  1. 1. GC Tuning Confessions Of A Performance Engineer Monica Beckwith monica@codekaram.com @mon_beck www.linkedin.com/in/monicabeckwith JavaOne Conference Oct. 26th, 2015
  2. 2. ©2015 CodeKaram About Me • JVM/GC Performance Engineer/Consultant • Worked at AMD, Sun, Oracle… • Worked with HotSpot JVM • JVM heuristics, JIT compiler, GCs: Parallel(Old) GC, G1 GC, CMS GC 2
  3. 3. ©2015 CodeKaram About Today’s Talk • A little bit about Performance Engineering • Insight into Garbage Collectors • OpenJDK HotSpot GCs • The Tradeoffs • GC Algorithms • Key Topics • Summary • GC Tuneables 3
  4. 4. Performance Engineering 4
  5. 5. ©2015 CodeKaram Performance Engineering 5 Requirements Design Performance Requirements & Test Plan & Development Implementation Performance Analysis
  6. 6. ©2015 CodeKaram Performance Requirements • Service level agreements (SLAs) for: • throughput, • latency and other response time related metrics E.g. Response time (RT) metrics - Average (RT), max or worst-case RT, 99th percentile RT… 6
  7. 7. ©2015 CodeKaram Response Time Metrics Number of GC events Minimum (ms) System1 37353 7.622 System2 34920 7.258 System3 36270 6.432 System4 40636 7.353 7
  8. 8. ©2015 CodeKaram Number of GC events Minimum (ms) Average (ms) System1 37353 7.622 307.741 System2 34920 7.258 320.778 System3 36270 6.432 321.483 System4 40636 7.353 323.143 8 Response Time Metrics
  9. 9. ©2015 CodeKaram Number of GC events Minimum (ms) Average (ms) 99th Percentile (ms) System1 37353 7.622 307.741 940.901 System2 34920 7.258 320.778 1006.607 System3 36270 6.432 321.483 1004.018 System4 40636 7.353 323.143 1041.225 9 Response Time Metrics
  10. 10. ©2015 CodeKaram Number of GC events Minimum (ms) Average (ms) 99th Percentile (ms) Maximum (ms) System 1 37353 7.622 307.741 940.901 3131.331 System 2 34920 7.258 320.778 1006.607 2744.588 System 3 36270 6.432 321.483 1004.018 1681.308 System 4 40636 7.353 323.143 1041.225 20699.505 10 Response Time Metrics
  11. 11. ©2015 CodeKaram Average (ms) 99th Percentile (ms) Maximum (ms) System1 307.741 940.901 3131.331 System2 320.778 1006.607 2744.588 System3 321.483 1004.018 1681.308 System4 323.143 1041.225 20699.505 5 full GCs and 10 evacuation failures 11 Response Time Metrics
  12. 12. Insight into GCs 12
  13. 13. ©2015 CodeKaram Fun Facts! • GC can NOT eliminate your memory leaks! • GC (and heap dump) can provide an insight into your application. 13
  14. 14. ©2015 CodeKaram Ideal GC? Maximize Throughput 14
  15. 15. ©2015 CodeKaram Ideal GC? Maximize Throughput Minimize Latency 15
  16. 16. ©2015 CodeKaram Ideal GC? Maximize Throughput Minimize Latency Minimize Footprint 16
  17. 17. ©2015 CodeKaram The Reality! Pick Any Two! :( 17
  18. 18. OpenJDK HotSpot GCs: The TRADE/OFF 18
  19. 19. ©2015 CodeKaram Fun Fact! Most OpenJDK HotSpot users would like to increase their (Java) heap space but they fear full garbage collections. 19
  20. 20. ©2015 CodeKaram The Tradeoff - • Throughput and latency are the two main drivers towards refinement of GC algorithms. 20
  21. 21. ©2015 CodeKaram The Throughput Maximizer 21 Maximize Throughput Generational Old Parallel Work Young Concurrent Threads Stop-the- world Threads
  22. 22. ©2015 CodeKaram Fun Fact! All GCs in OpenJDK HotSpot are generational. 22
  23. 23. ©2015 CodeKaram Mr. Latency Sensitive 23 Latency Sensitive No/Partial Compaction Concurrency Garbage First Mostly Concurrent Mark and Sweep
  24. 24. ©2015 CodeKaram Fun Fact! All GCs in OpenJDK HotSpot fallback to a fully compacting stop-the-world garbage collection called the “full” GC. Tuning can help avoid or postpone full GCs in many cases. 24
  25. 25. OpenJDK HotSpot GCs: Algorithms 25
  26. 26. ©2015 CodeKaram The Generational Java Heap Eden S0 S1 Old Generation Young Generation Allocations Survivors Tenured 26
  27. 27. ©2015 CodeKaram OpenJDK HotSpot Collectors *Similar GC Algorithms Different GC Algorithms Always collected in its entirety 27
  28. 28. ©2015 CodeKaram The Serial Collector Single Threaded Single Threaded - Entire Heap Marked, Swept and Compacted in its entirety Always collected in its entirety 28
  29. 29. ©2015 CodeKaram Young GC Thread Java Application Threads Old GC Thread Java Application Threads Java Application Threads The Serial Collector 29
  30. 30. ©2015 CodeKaram The Throughput Collector Multi-Threaded Multi-Threaded - Entire Heap Marked, Swept and Compacted in its entirety Always collected in its entirety 30
  31. 31. ©2015 CodeKaram Young GC Threads Java Application Threads Old GC Threads Java Application Threads Java Application Threads The Throughput Collector 31
  32. 32. ©2015 CodeKaram The CMS Collector Multi-Threaded Multi-Threaded - Mostly Concurrent Marked and Swept in its entirety Always collected in its entirety 32
  33. 33. ©2015 CodeKaram The CMS Collector Young GC Threads CMS Initial Mark Threads Java Application Threads Young GC Threads CMS Remark Threads Concurrent CMS Threads 33 Concurrent CMS Threads Concurrent CMS Threads Java Application Threads Java Application Threads Java Application Threads Java Application Threads
  34. 34. GC Algorithms - Key Topics 34
  35. 35. 35 What Triggers Full (Fail-Safe) Collections?
  36. 36. Promotion Failures! 36
  37. 37. Promotion Failures In The Throughput Collector 37
  38. 38. ©2015 CodeKaram The Throughput Collector - Java Heap Eden S0 S1 Old Generation Young Generation Allocations Survivors Tenured 38
  39. 39. ©2015 CodeKaram Old Generation Free Space Occupied Space The Throughput Collector - Contiguous Old Generation 39
  40. 40. ©2015 CodeKaram Old Generation To-be Promoted Object 1 The Throughput Collector - Contiguous Old Generation 40
  41. 41. ©2015 CodeKaram The Throughput Collector - Contiguous Old Generation 41 Old Generation Free Space Occupied Space
  42. 42. ©2015 CodeKaram The Throughput Collector - Contiguous Old Generation 42 Old Generation To-be Promoted Object 2
  43. 43. ©2015 CodeKaram The Throughput Collector - Contiguous Old Generation 43 Old Generation Free Space Occupied Space
  44. 44. ©2015 CodeKaram The Throughput Collector - Contiguous Old Generation 44 Old Generation To-be Promoted Object 3
  45. 45. ©2015 CodeKaram The Throughput Collector - Contiguous Old Generation 45 Old Generation Free Space Occupied Space
  46. 46. Promotion Failures & Concurrent Mode Failures In The CMS Collector 46
  47. 47. ©2015 CodeKaram The CMS Collector - Old Generation Maintained By Free Lists 47 Free List Old Generation Free Space Occupied Space
  48. 48. ©2015 CodeKaram To-be Promoted Object 1 48 Old Generation The CMS Collector - Old Generation Maintained By Free Lists
  49. 49. ©2015 CodeKaram49 Free List Old Generation Free Space Occupied Space The CMS Collector - Old Generation Maintained By Free Lists
  50. 50. ©2015 CodeKaram50 To-be Promoted Object 2 Old Generation The CMS Collector - Old Generation Maintained By Free Lists
  51. 51. ©2015 CodeKaram51 Free List Old Generation Free Space Occupied Space The CMS Collector - Old Generation Maintained By Free Lists
  52. 52. ©2015 CodeKaram X 52 To-be Promoted Object 3 Old Generation The CMS Collector - Old Generation Maintained By Free Lists
  53. 53. ©2015 CodeKaram X 53 To-be Promoted Object 3 Old Generation The CMS Collector - Old Generation Maintained By Free Lists
  54. 54. ©2015 CodeKaram X 54 To-be Promoted Object 3 Old Generation The CMS Collector - Old Generation Maintained By Free Lists
  55. 55. ©2015 CodeKaram X 55 To-be Promoted Object 3 Old Generation The CMS Collector - Old Generation Maintained By Free Lists
  56. 56. ©2015 CodeKaram ?? 56 To-be Promoted Object 3 Old Generation The CMS Collector - Old Generation Maintained By Free Lists
  57. 57. ©2015 CodeKaram Promotion Failure!! 57 To-be Promoted Object 3 Old Generation The CMS Collector - Fragmented Old Generation
  58. 58. ©2015 CodeKaram • Your old generation is getting filled before a concurrent cycle can complete and free up space. • Fragmentation has crept in. • Causes - marking threshold is too high, heap too small, or high application mutation rate 58 The CMS Collector - Concurrent Mode Failures
  59. 59. Incremental Compaction In The G1 Collector 59
  60. 60. ©2015 CodeKaram The Garbage First Collector - Regionalized Heap Eden Old Old Eden Old Survivor Humongous 60
  61. 61. ©2015 CodeKaram The Garbage First Collector Eden Old Old Eden Old Surv ivor E.g.: Current heap configuration - 61
  62. 62. ©2015 CodeKaram The Garbage First Collector Eden Old Old Eden Old Surv ivor E.g.: During a young collection - 62
  63. 63. ©2015 CodeKaram The Garbage First Collector Old Old Old E.g.: After a young collection - Sur viv or Ol d 63
  64. 64. ©2015 CodeKaram The Garbage First Collector Old Old Old E.g.: Current heap configuration - Sur viv or Ol d Eden Eden Old 64
  65. 65. ©2015 CodeKaram Old Old Old E.g.: Reclamation of a garbage-filled region during the cleanup phase - Sur viv or Ol d Eden Eden Old The Garbage First Collector 65
  66. 66. ©2015 CodeKaram Old Old Old E.g.: Reclamation of a garbage-filled region during the cleanup phase - Sur viv or Ol d Eden Eden The Garbage First Collector 66
  67. 67. ©2015 CodeKaram Old Old Old E.g.: Current heap configuration - Sur viv or Ol d Eden Eden The Garbage First Collector 67
  68. 68. ©2015 CodeKaram Old Old Old E.g.: During a mixed collection - Sur viv or Ol d Eden Eden The Garbage First Collector 68
  69. 69. ©2015 CodeKaram Old E.g.: After a mixed collection - O l d Sur vivo r Old The Garbage First Collector 69
  70. 70. Promotion/Evacuation Failures In The G1 Collector 70
  71. 71. ©2015 CodeKaram • When there are no more regions available for survivors or tenured objects, G1 GC encounters an evacuation failure. • An evacuation failure is expensive and the usual pattern is that if you see a couple of evacuation failures; full GC could* soon follow. The Garbage First Collector - Evacuation Failures 71
  72. 72. ©2015 CodeKaram A heavily tuned JVM command line may be restricting the G1 GC ergonomics and adaptability. Start with just your heap sizes and a reasonable pause time goal The Garbage First Collector - Avoiding Evacuation Failures 72
  73. 73. ©2015 CodeKaram Your live data set + long live transient data may be too large for the old generation Check LDS+ and increase heap to accommodate everything in the old generation. The Garbage First Collector - Avoiding Evacuation Failures 73
  74. 74. ©2015 CodeKaram Initiating Heap Occupancy Threshold could be the issue. Check IHOP and make sure it accommodates the LDS+. IHOP threshold too high -> Delayed marking -> Delayed incremental compaction -> Evacuation Failures! The Garbage First Collector - Avoiding Evacuation Failures 74
  75. 75. ©2015 CodeKaram Marking Cycle could be taking too long to complete? Increase concurrent marking threads Reduce IHOP The Garbage First Collector - Avoiding Evacuation Failures 75
  76. 76. ©2015 CodeKaram to-space survivors are the problem? Increase the G1ReservePercent, if to-space survivors are triggering the evacuation failures! The Garbage First Collector - Avoiding Evacuation Failures 76
  77. 77. ©2015 CodeKaram fragmentation an issue? The Garbage First Collector - Avoiding Evacuation Failures 77
  78. 78. Fragmentation In The G1 Collector 78
  79. 79. ©2015 CodeKaram • G1 GC is designed to “absorb” some fragmentation. • Default is 5% of the total Java heap • Tradeoff so that expensive regions are left out. G1 Heap Waste Percentage 79
  80. 80. ©2015 CodeKaram G1 Mixed GC (Region) Liveness Threshold 80 • G1 GC’s old regions are also designed to “absorb” some fragmentation. • Default is 85% liveness in a G1 region. • Tradeoff so that expensive regions are left out.
  81. 81. ©2015 CodeKaram The Garbage First Collector - Humongous Objects 81 An old generation region A young generation region
  82. 82. ©2015 CodeKaram The Garbage First Collector - Humongous Objects 82 Object < 50% of G1 region size Object >= 50% of G1 region size Object > G1 region size ??
  83. 83. ©2015 CodeKaram The Garbage First Collector - Humongous Objects 83 Object < 50% of G1 region size
  84. 84. ©2015 CodeKaram The Garbage First Collector - Humongous Objects 84 Object >= 50% of G1 region size
  85. 85. ©2015 CodeKaram The Garbage First Collector - Humongous Objects 85 Object > G1 region size
  86. 86. ©2015 CodeKaram The Garbage First Collector - Humongous Objects 86 Object NOT Humongous Object Humongous Object Humongous -> Needs Contiguous Regions Wasted Space!
  87. 87. ©2015 CodeKaram Ideally, humongous objects are few in number and are short lived. A lot of long-lived humongous objects can cause evacuation failures since humongous regions add to the old generation occupancy. The Garbage First Collector - Humongous Objects 87
  88. 88. G1 GC Logs - Key Topics 88
  89. 89. ©2015 CodeKaram G1 GC Log 154.431: [GC pause (G1 Evacuation Pause) (young), 0.2584864 secs] [Parallel Time: 253.2 ms, GC Workers: 8] [GC Worker Start (ms): Min: 154431.3, Avg: 154431.4, Max: 154431.5, Diff: 0.1] [Ext Root Scanning (ms): Min: 0.1, Avg: 0.2, Max: 0.3, Diff: 0.1, Sum: 1.4] [Update RS (ms): Min: 3.3, Avg: 3.5, Max: 3.8, Diff: 0.6, Sum: 28.2] [Processed Buffers: Min: 3, Avg: 3.5, Max: 5, Diff: 2, Sum: 28] [Scan RS (ms): Min: 46.1, Avg: 46.4, Max: 46.7, Diff: 0.6, Sum: 371.2] [Code Root Scanning (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.5] [Object Copy (ms): Min: 202.7, Avg: 202.8, Max: 202.9, Diff: 0.3, Sum: 1622.4] [Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.5] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.6] [GC Worker Total (ms): Min: 253.0, Avg: 253.1, Max: 253.1, Diff: 0.1, Sum: 2024.7] [GC Worker End (ms): Min: 154684.5, Avg: 154684.5, Max: 154684.5, Diff: 0.1] [Code Root Fixup: 0.1 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.7 ms] [Other: 4.4 ms] [Choose CSet: 0.0 ms] [Ref Proc: 0.3 ms] [Ref Enq: 0.0 ms] [Redirty Cards: 0.3 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 3.2 ms] [Eden: 4972.0M(4972.0M)->0.0B(4916.0M) Survivors: 148.0M->204.0M Heap: 5295.8M(10.0G)->379.4M(10.0G)] [Times: user=1.72 sys=0.14, real=0.26 secs] 89
  90. 90. ©2015 CodeKaram G1 GC Log 154.431: [GC pause (G1 Evacuation Pause) (young), 0.2584864 secs] [Parallel Time: 253.2 ms, GC Workers: 8] [GC Worker Start (ms): Min: 154431.3, Avg: 154431.4, Max: 154431.5, Diff: 0.1] [Ext Root Scanning (ms): Min: 0.1, Avg: 0.2, Max: 0.3, Diff: 0.1, Sum: 1.4] [Update RS (ms): Min: 3.3, Avg: 3.5, Max: 3.8, Diff: 0.6, Sum: 28.2] [Processed Buffers: Min: 3, Avg: 3.5, Max: 5, Diff: 2, Sum: 28] [Scan RS (ms): Min: 46.1, Avg: 46.4, Max: 46.7, Diff: 0.6, Sum: 371.2] [Code Root Scanning (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.5] [Object Copy (ms): Min: 202.7, Avg: 202.8, Max: 202.9, Diff: 0.3, Sum: 1622.4] [Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.5] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.6] [GC Worker Total (ms): Min: 253.0, Avg: 253.1, Max: 253.1, Diff: 0.1, Sum: 2024.7] [GC Worker End (ms): Min: 154684.5, Avg: 154684.5, Max: 154684.5, Diff: 0.1] [Code Root Fixup: 0.1 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.7 ms] [Other: 4.4 ms] [Choose CSet: 0.0 ms] [Ref Proc: 0.3 ms] [Ref Enq: 0.0 ms] [Redirty Cards: 0.3 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 3.2 ms] [Eden: 4972.0M(4972.0M)->0.0B(4916.0M) Survivors: 148.0M->204.0M Heap: 5295.8M(10.0G)->379.4M(10.0G)] [Times: user=1.72 sys=0.14, real=0.26 secs] 90
  91. 91. ©2015 CodeKaram G1 GC Log 154.431: [GC pause (G1 Evacuation Pause) (young), 0.2584864 secs] [Parallel Time: 253.2 ms, GC Workers: 8] [GC Worker Start (ms): Min: 154431.3, Avg: 154431.4, Max: 154431.5, Diff: 0.1] [Ext Root Scanning (ms): Min: 0.1, Avg: 0.2, Max: 0.3, Diff: 0.1, Sum: 1.4] [Update RS (ms): Min: 3.3, Avg: 3.5, Max: 3.8, Diff: 0.6, Sum: 28.2] [Processed Buffers: Min: 3, Avg: 3.5, Max: 5, Diff: 2, Sum: 28] [Scan RS (ms): Min: 46.1, Avg: 46.4, Max: 46.7, Diff: 0.6, Sum: 371.2] [Code Root Scanning (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.5] [Object Copy (ms): Min: 202.7, Avg: 202.8, Max: 202.9, Diff: 0.3, Sum: 1622.4] [Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.5] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.6] [GC Worker Total (ms): Min: 253.0, Avg: 253.1, Max: 253.1, Diff: 0.1, Sum: 2024.7] [GC Worker End (ms): Min: 154684.5, Avg: 154684.5, Max: 154684.5, Diff: 0.1] [Code Root Fixup: 0.1 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.7 ms] [Other: 4.4 ms] [Choose CSet: 0.0 ms] [Ref Proc: 0.3 ms] [Ref Enq: 0.0 ms] [Redirty Cards: 0.3 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 3.2 ms] [Eden: 4972.0M(4972.0M)->0.0B(4916.0M) Survivors: 148.0M->204.0M Heap: 5295.8M(10.0G)->379.4M(10.0G)] [Times: user=1.72 sys=0.14, real=0.26 secs] 91
  92. 92. ©2015 CodeKaram G1 GC Log 154.431: [GC pause (G1 Evacuation Pause) (young), 0.2584864 secs] [Parallel Time: 253.2 ms, GC Workers: 8] [GC Worker Start (ms): Min: 154431.3, Avg: 154431.4, Max: 154431.5, Diff: 0.1] [Ext Root Scanning (ms): Min: 0.1, Avg: 0.2, Max: 0.3, Diff: 0.1, Sum: 1.4] [Update RS (ms): Min: 3.3, Avg: 3.5, Max: 3.8, Diff: 0.6, Sum: 28.2] [Processed Buffers: Min: 3, Avg: 3.5, Max: 5, Diff: 2, Sum: 28] [Scan RS (ms): Min: 46.1, Avg: 46.4, Max: 46.7, Diff: 0.6, Sum: 371.2] [Code Root Scanning (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.5] [Object Copy (ms): Min: 202.7, Avg: 202.8, Max: 202.9, Diff: 0.3, Sum: 1622.4] [Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.5] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.6] [GC Worker Total (ms): Min: 253.0, Avg: 253.1, Max: 253.1, Diff: 0.1, Sum: 2024.7] [GC Worker End (ms): Min: 154684.5, Avg: 154684.5, Max: 154684.5, Diff: 0.1] [Code Root Fixup: 0.1 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.7 ms] [Other: 4.4 ms] [Choose CSet: 0.0 ms] [Ref Proc: 0.3 ms] [Ref Enq: 0.0 ms] [Redirty Cards: 0.3 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 3.2 ms] [Eden: 4972.0M(4972.0M)->0.0B(4916.0M) Survivors: 148.0M->204.0M Heap: 5295.8M(10.0G)->379.4M(10.0G)] [Times: user=1.72 sys=0.14, real=0.26 secs] 92
  93. 93. ©2015 CodeKaram G1 GC Log 154.431: [GC pause (G1 Evacuation Pause) (young), 0.2584864 secs] [Parallel Time: 253.2 ms, GC Workers: 8] [GC Worker Start (ms): Min: 154431.3, Avg: 154431.4, Max: 154431.5, Diff: 0.1] [Ext Root Scanning (ms): Min: 0.1, Avg: 0.2, Max: 0.3, Diff: 0.1, Sum: 1.4] [Update RS (ms): Min: 3.3, Avg: 3.5, Max: 3.8, Diff: 0.6, Sum: 28.2] [Processed Buffers: Min: 3, Avg: 3.5, Max: 5, Diff: 2, Sum: 28] [Scan RS (ms): Min: 46.1, Avg: 46.4, Max: 46.7, Diff: 0.6, Sum: 371.2] [Code Root Scanning (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.5] [Object Copy (ms): Min: 202.7, Avg: 202.8, Max: 202.9, Diff: 0.3, Sum: 1622.4] [Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.5] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.6] [GC Worker Total (ms): Min: 253.0, Avg: 253.1, Max: 253.1, Diff: 0.1, Sum: 2024.7] [GC Worker End (ms): Min: 154684.5, Avg: 154684.5, Max: 154684.5, Diff: 0.1] [Code Root Fixup: 0.1 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.7 ms] [Other: 4.4 ms] [Choose CSet: 0.0 ms] [Ref Proc: 0.3 ms] [Ref Enq: 0.0 ms] [Redirty Cards: 0.3 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 3.2 ms] [Eden: 4972.0M(4972.0M)->0.0B(4916.0M) Survivors: 148.0M->204.0M Heap: 5295.8M(10.0G)->379.4M(10.0G)] [Times: user=1.72 sys=0.14, real=0.26 secs] 93
  94. 94. ©2015 CodeKaram G1 GC Log 154.431: [GC pause (G1 Evacuation Pause) (young), 0.2584864 secs] [Parallel Time: 253.2 ms, GC Workers: 8] [GC Worker Start (ms): Min: 154431.3, Avg: 154431.4, Max: 154431.5, Diff: 0.1] [Ext Root Scanning (ms): Min: 0.1, Avg: 0.2, Max: 0.3, Diff: 0.1, Sum: 1.4] [Update RS (ms): Min: 3.3, Avg: 3.5, Max: 3.8, Diff: 0.6, Sum: 28.2] [Processed Buffers: Min: 3, Avg: 3.5, Max: 5, Diff: 2, Sum: 28] [Scan RS (ms): Min: 46.1, Avg: 46.4, Max: 46.7, Diff: 0.6, Sum: 371.2] [Code Root Scanning (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.5] [Object Copy (ms): Min: 202.7, Avg: 202.8, Max: 202.9, Diff: 0.3, Sum: 1622.4] [Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.5] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.6] [GC Worker Total (ms): Min: 253.0, Avg: 253.1, Max: 253.1, Diff: 0.1, Sum: 2024.7] [GC Worker End (ms): Min: 154684.5, Avg: 154684.5, Max: 154684.5, Diff: 0.1] [Code Root Fixup: 0.1 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.7 ms] [Other: 4.4 ms] [Choose CSet: 0.0 ms] [Ref Proc: 0.3 ms] [Ref Enq: 0.0 ms] [Redirty Cards: 0.3 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 3.2 ms] [Eden: 4972.0M(4972.0M)->0.0B(4916.0M) Survivors: 148.0M->204.0M Heap: 5295.8M(10.0G)->379.4M(10.0G)] [Times: user=1.72 sys=0.14, real=0.26 secs] 94
  95. 95. ©2015 CodeKaram G1 GC Log 154.431: [GC pause (G1 Evacuation Pause) (young), 0.2584864 secs] [Parallel Time: 253.2 ms, GC Workers: 8] [GC Worker Start (ms): Min: 154431.3, Avg: 154431.4, Max: 154431.5, Diff: 0.1] [Ext Root Scanning (ms): Min: 0.1, Avg: 0.2, Max: 0.3, Diff: 0.1, Sum: 1.4] [Update RS (ms): Min: 3.3, Avg: 3.5, Max: 3.8, Diff: 0.6, Sum: 28.2] [Processed Buffers: Min: 3, Avg: 3.5, Max: 5, Diff: 2, Sum: 28] [Scan RS (ms): Min: 46.1, Avg: 46.4, Max: 46.7, Diff: 0.6, Sum: 371.2] [Code Root Scanning (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.5] [Object Copy (ms): Min: 202.7, Avg: 202.8, Max: 202.9, Diff: 0.3, Sum: 1622.4] [Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.5] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.6] [GC Worker Total (ms): Min: 253.0, Avg: 253.1, Max: 253.1, Diff: 0.1, Sum: 2024.7] [GC Worker End (ms): Min: 154684.5, Avg: 154684.5, Max: 154684.5, Diff: 0.1] [Code Root Fixup: 0.1 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.7 ms] [Other: 4.4 ms] [Choose CSet: 0.0 ms] [Ref Proc: 0.3 ms] [Ref Enq: 0.0 ms] [Redirty Cards: 0.3 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 3.2 ms] [Eden: 4972.0M(4972.0M)->0.0B(4916.0M) Survivors: 148.0M->204.0M Heap: 5295.8M(10.0G)->379.4M(10.0G)] [Times: user=1.72 sys=0.14, real=0.26 secs] 95
  96. 96. ©2015 CodeKaram G1 GC Log 154.431: [GC pause (G1 Evacuation Pause) (young), 0.2584864 secs] [Parallel Time: 253.2 ms, GC Workers: 8] [GC Worker Start (ms): Min: 154431.3, Avg: 154431.4, Max: 154431.5, Diff: 0.1] [Ext Root Scanning (ms): Min: 0.1, Avg: 0.2, Max: 0.3, Diff: 0.1, Sum: 1.4] [Update RS (ms): Min: 3.3, Avg: 3.5, Max: 3.8, Diff: 0.6, Sum: 28.2] [Processed Buffers: Min: 3, Avg: 3.5, Max: 5, Diff: 2, Sum: 28] [Scan RS (ms): Min: 46.1, Avg: 46.4, Max: 46.7, Diff: 0.6, Sum: 371.2] [Code Root Scanning (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.5] [Object Copy (ms): Min: 202.7, Avg: 202.8, Max: 202.9, Diff: 0.3, Sum: 1622.4] [Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.5] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.6] [GC Worker Total (ms): Min: 253.0, Avg: 253.1, Max: 253.1, Diff: 0.1, Sum: 2024.7] [GC Worker End (ms): Min: 154684.5, Avg: 154684.5, Max: 154684.5, Diff: 0.1] [Code Root Fixup: 0.1 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.7 ms] [Other: 4.4 ms] [Choose CSet: 0.0 ms] [Ref Proc: 0.3 ms] [Ref Enq: 0.0 ms] [Redirty Cards: 0.3 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 3.2 ms] [Eden: 4972.0M(4972.0M)->0.0B(4916.0M) Survivors: 148.0M->204.0M Heap: 5295.8M(10.0G)->379.4M(10.0G)] [Times: user=1.72 sys=0.14, real=0.26 secs] 96
  97. 97. ©2015 CodeKaram Generation Sizing [Eden: 4972.0M(4972.0M)->0.0B(4916.0M) Survivors: 148.0M->204.0M Heap: 5295.8M(10.0G)->379.4M(10.0G)] [Eden: Occupancy before GC(Eden size before GC)->Occupancy after GC(Eden size after GC) Survivors: Size before GC->Size after GC Heap: Occupancy before GC(Heap size before GC)- >Occupancy after GC(Heap size after GC)] 97
  98. 98. ©2015 CodeKaram Heap Information Plot 98 OccupancyinMBs 0 7500 15000 22500 30000 TimeStamps 2500 3750 5000 6250 7500 8750 10000 11250 12500 Eden Occupancy Before GC Eden Size After GC Survivor Size After GC Old Generation Occupancy After GC Heap Occupancy Before GC Heap Size
  99. 99. ©2015 CodeKaram Scaling: GC Times vs Sys Times [Times: user=1.72 sys=0.14, real=0.26 secs] Scaling: user time/real time = 6.6 99
  100. 100. ©2015 CodeKaram Scaling: High Sys Times 100 Swapping? IO? Transparent huge pages enabled??* Sys memory issues? ??
  101. 101. ©2015 CodeKaram Reference Processing Times [Ref Proc: 0.3 ms] Check for high time spent in ‘Ref Proc’. Also, check your Remark pause times: 9.972: [GC remark 9.972: [Finalize Marking, 0.0007865 secs] 9.973: [GC ref-proc, 0.0027669 secs] 9.976: [Unloading, 0.0075832 secs], 0.0116215 secs] If Remark pauses are high or increasing and if ‘ref- proc’ is the major contributor - use: -XX: +ParallelRefProcEnabled 101
  102. 102. ©2015 CodeKaram GC Overhead vs Elapsed Time Overhead is an indication of the frequency of stop the world GC events. The more frequent the GC events - The more likely it is to negatively impact application throughput GC Elapsed Time indicated the amount of time it takes to execute stop the world GC events The higher the GC elapsed time - the lower the application responsiveness due to the GC induced latencies 102
  103. 103. Summary 103
  104. 104. ©2015 CodeKaram Most Allocations Eden Space Eden Full? Start Young Collection: Keep Allocating :) Objects Aged in Survivor Space? Promote: Keep Aging :) Fast Path Promotions Fast Path Old Generation CMS promotes to fitting free space out of a free list. What have we learned so far? 104
  105. 105. ©2015 CodeKaram GC Tuneables - The Throughput Collector Goal: Only promote objects after you have hazed them appropriately aged Tunables: Everything related to aging objects and generation sizing - NewRatio, (Max)NewSize, SurvivorRatio, (Max)TenuringThreshold 105
  106. 106. ©2015 CodeKaram GC Tuneables - The Throughput Collector Things to remember - • Applications with steady behavior rarely need AdaptiveSizePolicy to be enabled. • Overflow gets promoted into the old generation • Provide larger survivor spaces for transient data. • In most cases, young generation sizing has the most effect on throughput • Size the young generation to maintain the GC overhead to less than 5%. 106
  107. 107. ©2015 CodeKaram GC Tuneables - The CMS Collector Goal: Only promote objects after you have hazed them appropriately aged Tunables: Everything related to aging objects and young generation sizing still applies here. The concurrent thread counts and marking threshold are addition tunables for CMS 107
  108. 108. ©2015 CodeKaram GC Tuneables - The CMS Collector Things to remember - • Premature promotions are very expensive in CMS and could lead to fragmentation • You can reduce the CMS cycle duration by adding more concurrent threads: ConcGCThreads. • Remember that this will increase the concurrent overhead. • You can manually tune the marking threshold (adaptive by default) • CMSInitiatingOccupancyFraction & UseCMSInitiatingOccupancyOnly will help fix the marking threshold. • Note: The threshold is expressed as a percentage of the old generation occupancy 108
  109. 109. ©2015 CodeKaram GC Tuneables - The G1 Collector Goal: Get the GC ergonomics to work for you and know the defaults Tunables: • Pause time goal, heap size, max and min nursery, concurrent and parallel threads • The marking threshold, number of mixed GCs after marking, liveness threshold for the old regions, garbage toleration threshold, max old regions to be collected per mixed collection 109
  110. 110. ©2015 CodeKaram GC Tuneables - The G1 Collector Things to remember - • Know your defaults! • Understand your G1HeapRegionSize - It could be any factor of two from 1MB to 32MB. G1 strives for 2048 regions. • Fixing the nursery size (using Xmn) will meddle with the GC ergonomics/adaptiveness. • Don’t set really aggressive pause time goals - this will increase the GC overhead. • Spend time taming your mixed GCs - mixed GCs are incremental collections 110
  111. 111. ©2015 CodeKaram GC Tuneables - The G1 Collector Things to remember - • Taming mixed GCs: • Adjust the marking cycle according to you live data set. • Adjust you liveness threshold - this is the live occupancy threshold per region. Any region with liveness beyond this threshold will not be included in a mixed collection. • Adjust your garbage toleration threshold - helps G1 not get too aggressive with mixed collections • Distribute mixed GC pauses over a number of mixed collections - adjust your mixed GC count target and change your max old region threshold percent so that you can limit the old regions per collection 111
  112. 112. ©2015 CodeKaram Further Reading 112
  113. 113. ©2015 CodeKaram Further Reading • Jon Masa’s blog: https://blogs.oracle.com/ jonthecollector/entry/our_collectors • A few of my articles on InfoQ: http:// www.infoq.com/author/Monica-Beckwith • Presentations: http://www.slideshare.net/ MonicaBeckwith • Mail archives on hotspot-gc-use@openjdk.java.net & hotspot-gc-dev@openjdk.java.net 113
  114. 114. Appendix 114
  115. 115. ©2015 CodeKaram Performance Analysis • Monitoring: System Under Load • Utilization - CPU, IO, Sys/ Kernel, Memory bandwidth, Java heap, … • Lock statistics • Analyzing: • Utilization and time spent - GC logs, CPU, memory and application logs • Profiling: • Application, System, Memory - Java Heap. 115
  116. 116. ©2015 CodeKaram JVM Performance Engineering Java/JVM performance engineering includes the study, analysis and tuning of the Just-in-time (JIT) compiler, the Garbage Collector (GC) and many a times tuning related to the Java Development Kit (JDK). 116
  117. 117. ©2015 CodeKaram GC Performance Engineering • Monitor the JVM - Visual VM • Monitor and collect GC information - Visual GC (online), GC logs (offline) • Develop scripts to process GC logs; use GC Histo and JFreeCharts to plot your GC logs or use specialized tools/ log analyzers that serves your purpose. 117
  118. 118. Summary 118
  119. 119. ©2015 CodeKaram What have we learned so far? - Young Generation & Collections • Young generation is always collected in its entirety. • All 3 server GCs discussed earlier follow similar mechanism for young collection. • The young collections achieve reclamation via compaction and copying of live objects. • There are a lot of options for sizing the Eden and Survivor space optimally and many GCs also have adaptive sizing and GC ergonomics for young generation collections. 119
  120. 120. ©2015 CodeKaram All 3 server GCs vary in the way they collect the old generation: • For ParallelOld GC, the old generation is reclaimed and compacted in its entirety • Luckily, the compaction cost is distributed amongst parallel garbage collector worker threads. • Unluckily, the compaction cost depends a lot on the make of the live data set since at every compaction, the GC is moving live data around. • No tuning options other than the generation size adjustment and age threshold for promotion. 120 What have we learned so far? - Old Generation & Collections
  121. 121. ©2015 CodeKaram • For CMS GC, the old generation is (mostly) concurrently marked and swept. Thus the reclamation of dead objects happen in place and the space is added to a free list of spaces. • The marking threshold can be tuned adaptively and manually as well. • Luckily, CMS GC doesn’t do compaction, hence reclamation is fast. • Unluckily, a long running Java application with CMS GC is prone to fragmentation which will eventually result in promotion failures which can eventually lead to full compacting garbage collection and sometimes even concurrent mode failures. • Full compacting GCs are singled threaded in CMS GC. What have we learned so far? - Old Generation & Collections 121
  122. 122. ©2015 CodeKaram • For G1 GC, the old generation regions are (mostly) concurrently marked and an incremental compacting collection helps with optimizing the old generation collection. • Luckily, fragmentation is not “untunable” in G1 GC as it is in CMS GC. • Unluckily, sometimes, you may still encounter promotion/evacuation failures when G1 GC runs out of regions to copy live objects. Such an evacuation failure is expensive and can eventually lead to a full compacting GC. • Full compacting GCs are singled threaded in G1 GC. • Appropriate tuning of the old generation space and collection can help avoid evacuation failures and hence keep full GCs at bay. • G1 GC has multiple tuning options so that the GC can be adapted to your application needs. 122 What have we learned so far? - Old Generation & Collections

×