SlideShare a Scribd company logo
GC Tuning Confessions
Of A Performance
Engineer
Monica Beckwith
monica@codekaram.com
@mon_beck
www.linkedin.com/in/monicabeckwith
JavaOne Conference
Oct. 26th, 2015
©2015 CodeKaram
About Me
• JVM/GC Performance Engineer/Consultant
• Worked at AMD, Sun, Oracle…
• Worked with HotSpot JVM
• JVM heuristics, JIT compiler, GCs:
Parallel(Old) GC, G1 GC, CMS GC
2
©2015 CodeKaram
About Today’s Talk
• A little bit about Performance Engineering
• Insight into Garbage Collectors
• OpenJDK HotSpot GCs
• The Tradeoffs
• GC Algorithms
• Key Topics
• Summary
• GC Tuneables
3
Performance
Engineering
4
©2015 CodeKaram
Performance Engineering
5
Requirements
Design
Performance
Requirements & Test
Plan & Development
Implementation
Performance
Analysis
©2015 CodeKaram
Performance Requirements
• Service level agreements (SLAs) for:
• throughput,
• latency and other response time related
metrics E.g. Response time (RT) metrics -
Average (RT), max or worst-case RT, 99th
percentile RT…
6
©2015 CodeKaram
Response Time Metrics
Number of GC
events
Minimum (ms)
System1 37353 7.622
System2 34920 7.258
System3 36270 6.432
System4 40636 7.353
7
©2015 CodeKaram
Number of GC
events
Minimum (ms) Average (ms)
System1 37353 7.622 307.741
System2 34920 7.258 320.778
System3 36270 6.432 321.483
System4 40636 7.353 323.143
8
Response Time Metrics
©2015 CodeKaram
Number of
GC events
Minimum
(ms)
Average
(ms)
99th
Percentile
(ms)
System1 37353 7.622 307.741 940.901
System2 34920 7.258 320.778 1006.607
System3 36270 6.432 321.483 1004.018
System4 40636 7.353 323.143 1041.225
9
Response Time Metrics
©2015 CodeKaram
Number
of GC
events
Minimum
(ms)
Average
(ms)
99th
Percentile
(ms)
Maximum
(ms)
System
1
37353 7.622 307.741 940.901 3131.331
System
2
34920 7.258 320.778 1006.607 2744.588
System
3
36270 6.432 321.483 1004.018 1681.308
System
4
40636 7.353 323.143 1041.225 20699.505
10
Response Time Metrics
©2015 CodeKaram
Average (ms)
99th
Percentile
(ms)
Maximum
(ms)
System1 307.741 940.901 3131.331
System2 320.778 1006.607 2744.588
System3 321.483 1004.018 1681.308
System4 323.143 1041.225 20699.505
5 full GCs and 10 evacuation failures
11
Response Time Metrics
Insight into GCs
12
©2015 CodeKaram
Fun Facts!
• GC can NOT eliminate your memory
leaks!
• GC (and heap dump) can provide an
insight into your application.
13
©2015 CodeKaram
Ideal GC?
Maximize Throughput
14
©2015 CodeKaram
Ideal GC?
Maximize Throughput
Minimize Latency
15
©2015 CodeKaram
Ideal GC?
Maximize Throughput
Minimize Latency
Minimize Footprint
16
©2015 CodeKaram
The Reality!
Pick Any Two! :(
17
OpenJDK HotSpot
GCs: The TRADE/OFF
18
©2015 CodeKaram
Fun Fact!
Most OpenJDK HotSpot users would like to
increase their (Java) heap space but they
fear full garbage collections.
19
©2015 CodeKaram
The Tradeoff -
• Throughput and latency are the two main drivers
towards refinement of GC algorithms.
20
©2015 CodeKaram
The Throughput Maximizer
21
Maximize
Throughput
Generational
Old
Parallel Work
Young
Concurrent
Threads
Stop-the-
world
Threads
©2015 CodeKaram
Fun Fact!
All GCs in OpenJDK HotSpot are
generational.
22
©2015 CodeKaram
Mr. Latency Sensitive
23
Latency Sensitive
No/Partial
Compaction
Concurrency
Garbage First
Mostly
Concurrent
Mark and Sweep
©2015 CodeKaram
Fun Fact!
All GCs in OpenJDK HotSpot fallback to a
fully compacting stop-the-world garbage
collection called the “full” GC.
Tuning can help avoid or postpone full GCs in
many cases.
24
OpenJDK HotSpot
GCs: Algorithms
25
©2015 CodeKaram
The Generational Java Heap
Eden S0 S1 Old Generation
Young Generation
Allocations Survivors
Tenured
26
©2015 CodeKaram
OpenJDK HotSpot
Collectors
*Similar GC
Algorithms
Different GC Algorithms
Always collected in its entirety
27
©2015 CodeKaram
The Serial Collector
Single Threaded
Single Threaded - Entire
Heap Marked, Swept and
Compacted in its entirety
Always collected in its entirety
28
©2015 CodeKaram
Young GC
Thread
Java
Application
Threads
Old GC
Thread
Java
Application
Threads
Java
Application
Threads
The Serial Collector
29
©2015 CodeKaram
The Throughput Collector
Multi-Threaded
Multi-Threaded - Entire
Heap Marked, Swept and
Compacted in its entirety
Always collected in its entirety
30
©2015 CodeKaram
Young GC
Threads
Java
Application
Threads
Old GC
Threads
Java
Application
Threads
Java
Application
Threads
The Throughput Collector
31
©2015 CodeKaram
The CMS Collector
Multi-Threaded
Multi-Threaded - Mostly
Concurrent Marked and
Swept in its entirety
Always collected in its entirety
32
©2015 CodeKaram
The CMS Collector
Young
GC
Threads
CMS
Initial
Mark
Threads
Java
Application
Threads
Young
GC
Threads
CMS
Remark
Threads
Concurrent
CMS Threads
33
Concurrent
CMS Threads
Concurrent
CMS Threads
Java
Application
Threads
Java
Application
Threads
Java
Application
Threads
Java
Application
Threads
GC Algorithms - Key
Topics
34
35
What Triggers Full (Fail-Safe)
Collections?
Promotion
Failures!
36
Promotion Failures In The
Throughput Collector
37
©2015 CodeKaram
The Throughput Collector -
Java Heap
Eden S0 S1 Old Generation
Young Generation
Allocations Survivors
Tenured
38
©2015 CodeKaram
Old Generation
Free Space
Occupied Space
The Throughput Collector -
Contiguous Old Generation
39
©2015 CodeKaram
Old Generation
To-be Promoted Object 1
The Throughput Collector -
Contiguous Old Generation
40
©2015 CodeKaram
The Throughput Collector -
Contiguous Old Generation
41
Old Generation
Free Space
Occupied Space
©2015 CodeKaram
The Throughput Collector -
Contiguous Old Generation
42
Old Generation
To-be Promoted Object 2
©2015 CodeKaram
The Throughput Collector -
Contiguous Old Generation
43
Old Generation
Free Space
Occupied Space
©2015 CodeKaram
The Throughput Collector -
Contiguous Old Generation
44
Old Generation
To-be Promoted Object 3
©2015 CodeKaram
The Throughput Collector -
Contiguous Old Generation
45
Old Generation
Free Space
Occupied Space
Promotion Failures &
Concurrent Mode Failures
In The CMS Collector
46
©2015 CodeKaram
The CMS Collector - Old Generation
Maintained By Free Lists
47
Free List
Old Generation
Free Space
Occupied Space
©2015 CodeKaram
To-be Promoted Object 1
48
Old Generation
The CMS Collector - Old Generation
Maintained By Free Lists
©2015 CodeKaram49
Free List
Old Generation
Free Space
Occupied Space
The CMS Collector - Old Generation
Maintained By Free Lists
©2015 CodeKaram50
To-be Promoted Object 2
Old Generation
The CMS Collector - Old Generation
Maintained By Free Lists
©2015 CodeKaram51
Free List
Old Generation
Free Space
Occupied Space
The CMS Collector - Old Generation
Maintained By Free Lists
©2015 CodeKaram
X
52
To-be Promoted Object 3
Old Generation
The CMS Collector - Old Generation
Maintained By Free Lists
©2015 CodeKaram
X
53
To-be Promoted Object 3
Old Generation
The CMS Collector - Old Generation
Maintained By Free Lists
©2015 CodeKaram
X
54
To-be Promoted Object 3
Old Generation
The CMS Collector - Old Generation
Maintained By Free Lists
©2015 CodeKaram
X
55
To-be Promoted Object 3
Old Generation
The CMS Collector - Old Generation
Maintained By Free Lists
©2015 CodeKaram
??
56
To-be Promoted Object 3
Old Generation
The CMS Collector - Old Generation
Maintained By Free Lists
©2015 CodeKaram
Promotion Failure!!
57
To-be Promoted Object 3
Old Generation
The CMS Collector -
Fragmented Old Generation
©2015 CodeKaram
• Your old generation is getting filled before a
concurrent cycle can complete and free up
space.
• Fragmentation has crept in.
• Causes - marking threshold is too high, heap too
small, or high application mutation rate
58
The CMS Collector -
Concurrent Mode Failures
Incremental Compaction
In The G1 Collector
59
©2015 CodeKaram
The Garbage First Collector
- Regionalized Heap
Eden
Old Old
Eden
Old
Survivor
Humongous
60
©2015 CodeKaram
The Garbage First Collector
Eden
Old Old
Eden
Old
Surv
ivor
E.g.: Current heap configuration -
61
©2015 CodeKaram
The Garbage First Collector
Eden
Old Old
Eden
Old
Surv
ivor
E.g.: During a young collection -
62
©2015 CodeKaram
The Garbage First Collector
Old Old
Old
E.g.: After a young collection -
Sur
viv
or
Ol
d
63
©2015 CodeKaram
The Garbage First Collector
Old Old
Old
E.g.: Current heap configuration -
Sur
viv
or
Ol
d
Eden Eden
Old
64
©2015 CodeKaram
Old Old
Old
E.g.: Reclamation of a garbage-filled region
during the cleanup phase -
Sur
viv
or
Ol
d
Eden Eden
Old
The Garbage First Collector
65
©2015 CodeKaram
Old Old
Old
E.g.: Reclamation of a garbage-filled region
during the cleanup phase -
Sur
viv
or
Ol
d
Eden Eden
The Garbage First Collector
66
©2015 CodeKaram
Old Old
Old
E.g.: Current heap configuration -
Sur
viv
or
Ol
d
Eden Eden
The Garbage First Collector
67
©2015 CodeKaram
Old Old
Old
E.g.: During a mixed collection -
Sur
viv
or
Ol
d
Eden Eden
The Garbage First Collector
68
©2015 CodeKaram
Old
E.g.: After a mixed collection -
O
l
d
Sur
vivo
r
Old
The Garbage First Collector
69
Promotion/Evacuation
Failures In The G1
Collector
70
©2015 CodeKaram
• When there are no more regions available for
survivors or tenured objects, G1 GC encounters
an evacuation failure.
• An evacuation failure is expensive and the usual
pattern is that if you see a couple of evacuation
failures; full GC could* soon follow.
The Garbage First Collector
- Evacuation Failures
71
©2015 CodeKaram
A heavily tuned JVM command line
may be restricting the G1 GC
ergonomics and adaptability.
Start with just your heap sizes and a
reasonable pause time goal
The Garbage First Collector -
Avoiding Evacuation Failures
72
©2015 CodeKaram
Your live data set + long live
transient data may be too large for
the old generation
Check LDS+ and increase heap to
accommodate everything in the old
generation.
The Garbage First Collector -
Avoiding Evacuation Failures
73
©2015 CodeKaram
Initiating Heap Occupancy Threshold
could be the issue.
Check IHOP and make sure it accommodates
the LDS+.
IHOP threshold too high -> Delayed marking ->
Delayed incremental compaction -> Evacuation
Failures!
The Garbage First Collector -
Avoiding Evacuation Failures
74
©2015 CodeKaram
Marking Cycle could be taking too
long to complete?
Increase concurrent marking threads
Reduce IHOP
The Garbage First Collector -
Avoiding Evacuation Failures
75
©2015 CodeKaram
to-space survivors are the problem?
Increase the G1ReservePercent, if to-space
survivors are triggering the evacuation
failures!
The Garbage First Collector -
Avoiding Evacuation Failures
76
©2015 CodeKaram
fragmentation an issue?
The Garbage First Collector -
Avoiding Evacuation Failures
77
Fragmentation In The
G1 Collector
78
©2015 CodeKaram
• G1 GC is designed to “absorb” some
fragmentation.
• Default is 5% of the total Java heap
• Tradeoff so that expensive regions are left out.
G1 Heap Waste Percentage
79
©2015 CodeKaram
G1 Mixed GC (Region)
Liveness Threshold
80
• G1 GC’s old regions are also designed to
“absorb” some fragmentation.
• Default is 85% liveness in a G1 region.
• Tradeoff so that expensive regions are left out.
©2015 CodeKaram
The Garbage First Collector
- Humongous Objects
81
An old generation region
A young generation region
©2015 CodeKaram
The Garbage First Collector
- Humongous Objects
82
Object < 50% of
G1 region size
Object >= 50% of
G1 region size
Object > G1
region size
??
©2015 CodeKaram
The Garbage First Collector
- Humongous Objects
83
Object < 50% of
G1 region size
©2015 CodeKaram
The Garbage First Collector
- Humongous Objects
84
Object >= 50% of
G1 region size
©2015 CodeKaram
The Garbage First Collector
- Humongous Objects
85
Object > G1
region size
©2015 CodeKaram
The Garbage First Collector
- Humongous Objects
86
Object NOT Humongous
Object Humongous
Object Humongous ->
Needs Contiguous Regions
Wasted
Space!
©2015 CodeKaram
Ideally, humongous objects are few in
number and are short lived.
A lot of long-lived humongous objects can
cause evacuation failures since humongous
regions add to the old generation occupancy.
The Garbage First Collector
- Humongous Objects
87
G1 GC Logs - Key
Topics
88
©2015 CodeKaram
G1 GC Log
154.431: [GC pause (G1 Evacuation Pause) (young), 0.2584864 secs]
[Parallel Time: 253.2 ms, GC Workers: 8]
[GC Worker Start (ms): Min: 154431.3, Avg: 154431.4, Max: 154431.5, Diff: 0.1]
[Ext Root Scanning (ms): Min: 0.1, Avg: 0.2, Max: 0.3, Diff: 0.1, Sum: 1.4]
[Update RS (ms): Min: 3.3, Avg: 3.5, Max: 3.8, Diff: 0.6, Sum: 28.2]
[Processed Buffers: Min: 3, Avg: 3.5, Max: 5, Diff: 2, Sum: 28]
[Scan RS (ms): Min: 46.1, Avg: 46.4, Max: 46.7, Diff: 0.6, Sum: 371.2]
[Code Root Scanning (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.5]
[Object Copy (ms): Min: 202.7, Avg: 202.8, Max: 202.9, Diff: 0.3, Sum: 1622.4]
[Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.5]
[GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.6]
[GC Worker Total (ms): Min: 253.0, Avg: 253.1, Max: 253.1, Diff: 0.1, Sum: 2024.7]
[GC Worker End (ms): Min: 154684.5, Avg: 154684.5, Max: 154684.5, Diff: 0.1]
[Code Root Fixup: 0.1 ms]
[Code Root Purge: 0.0 ms]
[Clear CT: 0.7 ms]
[Other: 4.4 ms]
[Choose CSet: 0.0 ms]
[Ref Proc: 0.3 ms]
[Ref Enq: 0.0 ms]
[Redirty Cards: 0.3 ms]
[Humongous Reclaim: 0.0 ms]
[Free CSet: 3.2 ms]
[Eden: 4972.0M(4972.0M)->0.0B(4916.0M) Survivors: 148.0M->204.0M Heap: 5295.8M(10.0G)->379.4M(10.0G)]
[Times: user=1.72 sys=0.14, real=0.26 secs]
89
©2015 CodeKaram
G1 GC Log
154.431: [GC pause (G1 Evacuation Pause) (young), 0.2584864 secs]
[Parallel Time: 253.2 ms, GC Workers: 8]
[GC Worker Start (ms): Min: 154431.3, Avg: 154431.4, Max: 154431.5, Diff: 0.1]
[Ext Root Scanning (ms): Min: 0.1, Avg: 0.2, Max: 0.3, Diff: 0.1, Sum: 1.4]
[Update RS (ms): Min: 3.3, Avg: 3.5, Max: 3.8, Diff: 0.6, Sum: 28.2]
[Processed Buffers: Min: 3, Avg: 3.5, Max: 5, Diff: 2, Sum: 28]
[Scan RS (ms): Min: 46.1, Avg: 46.4, Max: 46.7, Diff: 0.6, Sum: 371.2]
[Code Root Scanning (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.5]
[Object Copy (ms): Min: 202.7, Avg: 202.8, Max: 202.9, Diff: 0.3, Sum: 1622.4]
[Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.5]
[GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.6]
[GC Worker Total (ms): Min: 253.0, Avg: 253.1, Max: 253.1, Diff: 0.1, Sum: 2024.7]
[GC Worker End (ms): Min: 154684.5, Avg: 154684.5, Max: 154684.5, Diff: 0.1]
[Code Root Fixup: 0.1 ms]
[Code Root Purge: 0.0 ms]
[Clear CT: 0.7 ms]
[Other: 4.4 ms]
[Choose CSet: 0.0 ms]
[Ref Proc: 0.3 ms]
[Ref Enq: 0.0 ms]
[Redirty Cards: 0.3 ms]
[Humongous Reclaim: 0.0 ms]
[Free CSet: 3.2 ms]
[Eden: 4972.0M(4972.0M)->0.0B(4916.0M) Survivors: 148.0M->204.0M Heap: 5295.8M(10.0G)->379.4M(10.0G)]
[Times: user=1.72 sys=0.14, real=0.26 secs]
90
©2015 CodeKaram
G1 GC Log
154.431: [GC pause (G1 Evacuation Pause) (young), 0.2584864 secs]
[Parallel Time: 253.2 ms, GC Workers: 8]
[GC Worker Start (ms): Min: 154431.3, Avg: 154431.4, Max: 154431.5, Diff: 0.1]
[Ext Root Scanning (ms): Min: 0.1, Avg: 0.2, Max: 0.3, Diff: 0.1, Sum: 1.4]
[Update RS (ms): Min: 3.3, Avg: 3.5, Max: 3.8, Diff: 0.6, Sum: 28.2]
[Processed Buffers: Min: 3, Avg: 3.5, Max: 5, Diff: 2, Sum: 28]
[Scan RS (ms): Min: 46.1, Avg: 46.4, Max: 46.7, Diff: 0.6, Sum: 371.2]
[Code Root Scanning (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.5]
[Object Copy (ms): Min: 202.7, Avg: 202.8, Max: 202.9, Diff: 0.3, Sum: 1622.4]
[Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.5]
[GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.6]
[GC Worker Total (ms): Min: 253.0, Avg: 253.1, Max: 253.1, Diff: 0.1, Sum: 2024.7]
[GC Worker End (ms): Min: 154684.5, Avg: 154684.5, Max: 154684.5, Diff: 0.1]
[Code Root Fixup: 0.1 ms]
[Code Root Purge: 0.0 ms]
[Clear CT: 0.7 ms]
[Other: 4.4 ms]
[Choose CSet: 0.0 ms]
[Ref Proc: 0.3 ms]
[Ref Enq: 0.0 ms]
[Redirty Cards: 0.3 ms]
[Humongous Reclaim: 0.0 ms]
[Free CSet: 3.2 ms]
[Eden: 4972.0M(4972.0M)->0.0B(4916.0M) Survivors: 148.0M->204.0M Heap: 5295.8M(10.0G)->379.4M(10.0G)]
[Times: user=1.72 sys=0.14, real=0.26 secs]
91
©2015 CodeKaram
G1 GC Log
154.431: [GC pause (G1 Evacuation Pause) (young), 0.2584864 secs]
[Parallel Time: 253.2 ms, GC Workers: 8]
[GC Worker Start (ms): Min: 154431.3, Avg: 154431.4, Max: 154431.5, Diff: 0.1]
[Ext Root Scanning (ms): Min: 0.1, Avg: 0.2, Max: 0.3, Diff: 0.1, Sum: 1.4]
[Update RS (ms): Min: 3.3, Avg: 3.5, Max: 3.8, Diff: 0.6, Sum: 28.2]
[Processed Buffers: Min: 3, Avg: 3.5, Max: 5, Diff: 2, Sum: 28]
[Scan RS (ms): Min: 46.1, Avg: 46.4, Max: 46.7, Diff: 0.6, Sum: 371.2]
[Code Root Scanning (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.5]
[Object Copy (ms): Min: 202.7, Avg: 202.8, Max: 202.9, Diff: 0.3, Sum: 1622.4]
[Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.5]
[GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.6]
[GC Worker Total (ms): Min: 253.0, Avg: 253.1, Max: 253.1, Diff: 0.1, Sum: 2024.7]
[GC Worker End (ms): Min: 154684.5, Avg: 154684.5, Max: 154684.5, Diff: 0.1]
[Code Root Fixup: 0.1 ms]
[Code Root Purge: 0.0 ms]
[Clear CT: 0.7 ms]
[Other: 4.4 ms]
[Choose CSet: 0.0 ms]
[Ref Proc: 0.3 ms]
[Ref Enq: 0.0 ms]
[Redirty Cards: 0.3 ms]
[Humongous Reclaim: 0.0 ms]
[Free CSet: 3.2 ms]
[Eden: 4972.0M(4972.0M)->0.0B(4916.0M) Survivors: 148.0M->204.0M Heap: 5295.8M(10.0G)->379.4M(10.0G)]
[Times: user=1.72 sys=0.14, real=0.26 secs]
92
©2015 CodeKaram
G1 GC Log
154.431: [GC pause (G1 Evacuation Pause) (young), 0.2584864 secs]
[Parallel Time: 253.2 ms, GC Workers: 8]
[GC Worker Start (ms): Min: 154431.3, Avg: 154431.4, Max: 154431.5, Diff: 0.1]
[Ext Root Scanning (ms): Min: 0.1, Avg: 0.2, Max: 0.3, Diff: 0.1, Sum: 1.4]
[Update RS (ms): Min: 3.3, Avg: 3.5, Max: 3.8, Diff: 0.6, Sum: 28.2]
[Processed Buffers: Min: 3, Avg: 3.5, Max: 5, Diff: 2, Sum: 28]
[Scan RS (ms): Min: 46.1, Avg: 46.4, Max: 46.7, Diff: 0.6, Sum: 371.2]
[Code Root Scanning (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.5]
[Object Copy (ms): Min: 202.7, Avg: 202.8, Max: 202.9, Diff: 0.3, Sum: 1622.4]
[Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.5]
[GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.6]
[GC Worker Total (ms): Min: 253.0, Avg: 253.1, Max: 253.1, Diff: 0.1, Sum: 2024.7]
[GC Worker End (ms): Min: 154684.5, Avg: 154684.5, Max: 154684.5, Diff: 0.1]
[Code Root Fixup: 0.1 ms]
[Code Root Purge: 0.0 ms]
[Clear CT: 0.7 ms]
[Other: 4.4 ms]
[Choose CSet: 0.0 ms]
[Ref Proc: 0.3 ms]
[Ref Enq: 0.0 ms]
[Redirty Cards: 0.3 ms]
[Humongous Reclaim: 0.0 ms]
[Free CSet: 3.2 ms]
[Eden: 4972.0M(4972.0M)->0.0B(4916.0M) Survivors: 148.0M->204.0M Heap: 5295.8M(10.0G)->379.4M(10.0G)]
[Times: user=1.72 sys=0.14, real=0.26 secs]
93
©2015 CodeKaram
G1 GC Log
154.431: [GC pause (G1 Evacuation Pause) (young), 0.2584864 secs]
[Parallel Time: 253.2 ms, GC Workers: 8]
[GC Worker Start (ms): Min: 154431.3, Avg: 154431.4, Max: 154431.5, Diff: 0.1]
[Ext Root Scanning (ms): Min: 0.1, Avg: 0.2, Max: 0.3, Diff: 0.1, Sum: 1.4]
[Update RS (ms): Min: 3.3, Avg: 3.5, Max: 3.8, Diff: 0.6, Sum: 28.2]
[Processed Buffers: Min: 3, Avg: 3.5, Max: 5, Diff: 2, Sum: 28]
[Scan RS (ms): Min: 46.1, Avg: 46.4, Max: 46.7, Diff: 0.6, Sum: 371.2]
[Code Root Scanning (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.5]
[Object Copy (ms): Min: 202.7, Avg: 202.8, Max: 202.9, Diff: 0.3, Sum: 1622.4]
[Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.5]
[GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.6]
[GC Worker Total (ms): Min: 253.0, Avg: 253.1, Max: 253.1, Diff: 0.1, Sum: 2024.7]
[GC Worker End (ms): Min: 154684.5, Avg: 154684.5, Max: 154684.5, Diff: 0.1]
[Code Root Fixup: 0.1 ms]
[Code Root Purge: 0.0 ms]
[Clear CT: 0.7 ms]
[Other: 4.4 ms]
[Choose CSet: 0.0 ms]
[Ref Proc: 0.3 ms]
[Ref Enq: 0.0 ms]
[Redirty Cards: 0.3 ms]
[Humongous Reclaim: 0.0 ms]
[Free CSet: 3.2 ms]
[Eden: 4972.0M(4972.0M)->0.0B(4916.0M) Survivors: 148.0M->204.0M Heap: 5295.8M(10.0G)->379.4M(10.0G)]
[Times: user=1.72 sys=0.14, real=0.26 secs]
94
©2015 CodeKaram
G1 GC Log
154.431: [GC pause (G1 Evacuation Pause) (young), 0.2584864 secs]
[Parallel Time: 253.2 ms, GC Workers: 8]
[GC Worker Start (ms): Min: 154431.3, Avg: 154431.4, Max: 154431.5, Diff: 0.1]
[Ext Root Scanning (ms): Min: 0.1, Avg: 0.2, Max: 0.3, Diff: 0.1, Sum: 1.4]
[Update RS (ms): Min: 3.3, Avg: 3.5, Max: 3.8, Diff: 0.6, Sum: 28.2]
[Processed Buffers: Min: 3, Avg: 3.5, Max: 5, Diff: 2, Sum: 28]
[Scan RS (ms): Min: 46.1, Avg: 46.4, Max: 46.7, Diff: 0.6, Sum: 371.2]
[Code Root Scanning (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.5]
[Object Copy (ms): Min: 202.7, Avg: 202.8, Max: 202.9, Diff: 0.3, Sum: 1622.4]
[Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.5]
[GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.6]
[GC Worker Total (ms): Min: 253.0, Avg: 253.1, Max: 253.1, Diff: 0.1, Sum: 2024.7]
[GC Worker End (ms): Min: 154684.5, Avg: 154684.5, Max: 154684.5, Diff: 0.1]
[Code Root Fixup: 0.1 ms]
[Code Root Purge: 0.0 ms]
[Clear CT: 0.7 ms]
[Other: 4.4 ms]
[Choose CSet: 0.0 ms]
[Ref Proc: 0.3 ms]
[Ref Enq: 0.0 ms]
[Redirty Cards: 0.3 ms]
[Humongous Reclaim: 0.0 ms]
[Free CSet: 3.2 ms]
[Eden: 4972.0M(4972.0M)->0.0B(4916.0M) Survivors: 148.0M->204.0M Heap: 5295.8M(10.0G)->379.4M(10.0G)]
[Times: user=1.72 sys=0.14, real=0.26 secs]
95
©2015 CodeKaram
G1 GC Log
154.431: [GC pause (G1 Evacuation Pause) (young), 0.2584864 secs]
[Parallel Time: 253.2 ms, GC Workers: 8]
[GC Worker Start (ms): Min: 154431.3, Avg: 154431.4, Max: 154431.5, Diff: 0.1]
[Ext Root Scanning (ms): Min: 0.1, Avg: 0.2, Max: 0.3, Diff: 0.1, Sum: 1.4]
[Update RS (ms): Min: 3.3, Avg: 3.5, Max: 3.8, Diff: 0.6, Sum: 28.2]
[Processed Buffers: Min: 3, Avg: 3.5, Max: 5, Diff: 2, Sum: 28]
[Scan RS (ms): Min: 46.1, Avg: 46.4, Max: 46.7, Diff: 0.6, Sum: 371.2]
[Code Root Scanning (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.5]
[Object Copy (ms): Min: 202.7, Avg: 202.8, Max: 202.9, Diff: 0.3, Sum: 1622.4]
[Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.5]
[GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.6]
[GC Worker Total (ms): Min: 253.0, Avg: 253.1, Max: 253.1, Diff: 0.1, Sum: 2024.7]
[GC Worker End (ms): Min: 154684.5, Avg: 154684.5, Max: 154684.5, Diff: 0.1]
[Code Root Fixup: 0.1 ms]
[Code Root Purge: 0.0 ms]
[Clear CT: 0.7 ms]
[Other: 4.4 ms]
[Choose CSet: 0.0 ms]
[Ref Proc: 0.3 ms]
[Ref Enq: 0.0 ms]
[Redirty Cards: 0.3 ms]
[Humongous Reclaim: 0.0 ms]
[Free CSet: 3.2 ms]
[Eden: 4972.0M(4972.0M)->0.0B(4916.0M) Survivors: 148.0M->204.0M Heap: 5295.8M(10.0G)->379.4M(10.0G)]
[Times: user=1.72 sys=0.14, real=0.26 secs]
96
©2015 CodeKaram
Generation Sizing
[Eden: 4972.0M(4972.0M)->0.0B(4916.0M) Survivors:
148.0M->204.0M Heap: 5295.8M(10.0G)->379.4M(10.0G)]
[Eden: Occupancy before GC(Eden size before
GC)->Occupancy after GC(Eden size after GC)
Survivors: Size before GC->Size after GC Heap:
Occupancy before GC(Heap size before GC)-
>Occupancy after GC(Heap size after GC)]
97
©2015 CodeKaram
Heap Information Plot
98
OccupancyinMBs
0
7500
15000
22500
30000
TimeStamps
2500 3750 5000 6250 7500 8750 10000 11250 12500
Eden Occupancy Before GC Eden Size After GC
Survivor Size After GC Old Generation Occupancy After GC
Heap Occupancy Before GC Heap Size
©2015 CodeKaram
Scaling: GC Times vs Sys Times
[Times: user=1.72 sys=0.14, real=0.26 secs]
Scaling: user time/real time = 6.6
99
©2015 CodeKaram
Scaling: High Sys Times
100
Swapping?
IO?
Transparent huge
pages enabled??*
Sys memory
issues?
??
©2015 CodeKaram
Reference Processing Times
[Ref Proc: 0.3 ms]
Check for high time spent in ‘Ref Proc’.
Also, check your Remark pause times:
9.972: [GC remark 9.972: [Finalize Marking, 0.0007865 secs]
9.973: [GC ref-proc, 0.0027669 secs] 9.976: [Unloading,
0.0075832 secs], 0.0116215 secs]
If Remark pauses are high or increasing and if ‘ref-
proc’ is the major contributor - use: -XX:
+ParallelRefProcEnabled
101
©2015 CodeKaram
GC Overhead vs Elapsed
Time
Overhead is an indication of the frequency of
stop the world GC events.
The more frequent the GC events - The more
likely it is to negatively impact application
throughput
GC Elapsed Time indicated the amount of time it
takes to execute stop the world GC events
The higher the GC elapsed time - the lower the
application responsiveness due to the GC
induced latencies
102
Summary
103
©2015 CodeKaram
Most Allocations Eden Space
Eden Full? Start Young Collection: Keep Allocating :)
Objects Aged in Survivor Space? Promote: Keep Aging :)
Fast Path
Promotions
Fast Path
Old Generation
CMS promotes to fitting free space out of a free list.
What have we learned so
far?
104
©2015 CodeKaram
GC Tuneables - The
Throughput Collector
Goal:
Only promote objects after you have hazed them
appropriately
aged
Tunables:
Everything related to aging objects and
generation sizing -
NewRatio, (Max)NewSize, SurvivorRatio,
(Max)TenuringThreshold
105
©2015 CodeKaram
GC Tuneables - The
Throughput Collector
Things to remember -
• Applications with steady behavior rarely need
AdaptiveSizePolicy to be enabled.
• Overflow gets promoted into the old generation
• Provide larger survivor spaces for transient data.
• In most cases, young generation sizing has the most effect
on throughput
• Size the young generation to maintain the GC overhead to
less than 5%.
106
©2015 CodeKaram
GC Tuneables - The CMS
Collector
Goal:
Only promote objects after you have hazed them
appropriately
aged
Tunables:
Everything related to aging objects and young
generation sizing still applies here.
The concurrent thread counts and marking
threshold are addition tunables for CMS
107
©2015 CodeKaram
GC Tuneables - The CMS
Collector
Things to remember -
• Premature promotions are very expensive in CMS and could lead to
fragmentation
• You can reduce the CMS cycle duration by adding more concurrent
threads: ConcGCThreads.
• Remember that this will increase the concurrent overhead.
• You can manually tune the marking threshold (adaptive by default)
• CMSInitiatingOccupancyFraction & UseCMSInitiatingOccupancyOnly
will help fix the marking threshold.
• Note: The threshold is expressed as a percentage of the old generation
occupancy
108
©2015 CodeKaram
GC Tuneables - The G1
Collector
Goal:
Get the GC ergonomics to work for you and
know the defaults
Tunables:
• Pause time goal, heap size, max and min nursery,
concurrent and parallel threads
• The marking threshold, number of mixed GCs
after marking, liveness threshold for the old
regions, garbage toleration threshold, max old
regions to be collected per mixed collection
109
©2015 CodeKaram
GC Tuneables - The G1
Collector
Things to remember -
• Know your defaults!
• Understand your G1HeapRegionSize - It could be any factor of
two from 1MB to 32MB. G1 strives for 2048 regions.
• Fixing the nursery size (using Xmn) will meddle with the GC
ergonomics/adaptiveness.
• Don’t set really aggressive pause time goals - this will increase the
GC overhead.
• Spend time taming your mixed GCs - mixed GCs are incremental
collections
110
©2015 CodeKaram
GC Tuneables - The G1
Collector
Things to remember -
• Taming mixed GCs:
• Adjust the marking cycle according to you live data set.
• Adjust you liveness threshold - this is the live occupancy threshold
per region. Any region with liveness beyond this threshold will not
be included in a mixed collection.
• Adjust your garbage toleration threshold - helps G1 not get too
aggressive with mixed collections
• Distribute mixed GC pauses over a number of mixed collections -
adjust your mixed GC count target and change your max old region
threshold percent so that you can limit the old regions per collection
111
©2015 CodeKaram
Further Reading
112
©2015 CodeKaram
Further Reading
• Jon Masa’s blog: https://blogs.oracle.com/
jonthecollector/entry/our_collectors
• A few of my articles on InfoQ: http://
www.infoq.com/author/Monica-Beckwith
• Presentations: http://www.slideshare.net/
MonicaBeckwith
• Mail archives on hotspot-gc-use@openjdk.java.net
& hotspot-gc-dev@openjdk.java.net
113
Appendix
114
©2015 CodeKaram
Performance Analysis
• Monitoring: System Under Load
• Utilization - CPU, IO, Sys/ Kernel, Memory bandwidth, Java
heap, …
• Lock statistics
• Analyzing:
• Utilization and time spent - GC logs, CPU, memory and
application logs
• Profiling:
• Application, System, Memory - Java Heap.
115
©2015 CodeKaram
JVM Performance
Engineering
Java/JVM performance engineering includes the
study, analysis and tuning of the Just-in-time (JIT)
compiler, the Garbage Collector (GC) and many a
times tuning related to the Java Development Kit
(JDK).
116
©2015 CodeKaram
GC Performance
Engineering
• Monitor the JVM - Visual VM
• Monitor and collect GC information - Visual GC
(online), GC logs (offline)
• Develop scripts to process GC logs; use GC
Histo and JFreeCharts to plot your GC logs or
use specialized tools/ log analyzers that
serves your purpose.
117
Summary
118
©2015 CodeKaram
What have we learned so far? -
Young Generation & Collections
• Young generation is always collected in its entirety.
• All 3 server GCs discussed earlier follow similar
mechanism for young collection.
• The young collections achieve reclamation via
compaction and copying of live objects.
• There are a lot of options for sizing the Eden and
Survivor space optimally and many GCs also have
adaptive sizing and GC ergonomics for young
generation collections.
119
©2015 CodeKaram
All 3 server GCs vary in the way they collect the old generation:
• For ParallelOld GC, the old generation is reclaimed and
compacted in its entirety
• Luckily, the compaction cost is distributed amongst parallel
garbage collector worker threads.
• Unluckily, the compaction cost depends a lot on the make of
the live data set since at every compaction, the GC is moving
live data around.
• No tuning options other than the generation size adjustment
and age threshold for promotion.
120
What have we learned so far?
- Old Generation & Collections
©2015 CodeKaram
• For CMS GC, the old generation is (mostly) concurrently marked
and swept. Thus the reclamation of dead objects happen in place
and the space is added to a free list of spaces.
• The marking threshold can be tuned adaptively and manually as
well.
• Luckily, CMS GC doesn’t do compaction, hence reclamation is fast.
• Unluckily, a long running Java application with CMS GC is prone to
fragmentation which will eventually result in promotion failures
which can eventually lead to full compacting garbage collection
and sometimes even concurrent mode failures.
• Full compacting GCs are singled threaded in CMS GC.
What have we learned so far?
- Old Generation & Collections
121
©2015 CodeKaram
• For G1 GC, the old generation regions are (mostly) concurrently marked and
an incremental compacting collection helps with optimizing the old generation
collection.
• Luckily, fragmentation is not “untunable” in G1 GC as it is in CMS GC.
• Unluckily, sometimes, you may still encounter promotion/evacuation failures
when G1 GC runs out of regions to copy live objects. Such an evacuation
failure is expensive and can eventually lead to a full compacting GC.
• Full compacting GCs are singled threaded in G1 GC.
• Appropriate tuning of the old generation space and collection can help
avoid evacuation failures and hence keep full GCs at bay.
• G1 GC has multiple tuning options so that the GC can be adapted to your
application needs.
122
What have we learned so far?
- Old Generation & Collections

More Related Content

What's hot

急速に進化を続けるCNIプラグイン Antrea
急速に進化を続けるCNIプラグイン Antrea 急速に進化を続けるCNIプラグイン Antrea
急速に進化を続けるCNIプラグイン Antrea
Motonori Shindo
 
Autoscaling Flink with Reactive Mode
Autoscaling Flink with Reactive ModeAutoscaling Flink with Reactive Mode
Autoscaling Flink with Reactive Mode
Flink Forward
 
Where is my bottleneck? Performance troubleshooting in Flink
Where is my bottleneck? Performance troubleshooting in FlinkWhere is my bottleneck? Performance troubleshooting in Flink
Where is my bottleneck? Performance troubleshooting in Flink
Flink Forward
 
Garbage First Garbage Collector (G1 GC): Current and Future Adaptability and ...
Garbage First Garbage Collector (G1 GC): Current and Future Adaptability and ...Garbage First Garbage Collector (G1 GC): Current and Future Adaptability and ...
Garbage First Garbage Collector (G1 GC): Current and Future Adaptability and ...
Monica Beckwith
 
Metaspace
MetaspaceMetaspace
Metaspace
Yasumasa Suenaga
 
[오픈소스컨설팅]Java Performance Tuning
[오픈소스컨설팅]Java Performance Tuning[오픈소스컨설팅]Java Performance Tuning
[오픈소스컨설팅]Java Performance Tuning
Ji-Woong Choi
 
スレッドダンプの読み方
スレッドダンプの読み方スレッドダンプの読み方
スレッドダンプの読み方
Funato Takashi
 
OVS VXLAN Network Accelaration on OpenStack (VXLAN offload and DPDK) - OpenSt...
OVS VXLAN Network Accelaration on OpenStack (VXLAN offload and DPDK) - OpenSt...OVS VXLAN Network Accelaration on OpenStack (VXLAN offload and DPDK) - OpenSt...
OVS VXLAN Network Accelaration on OpenStack (VXLAN offload and DPDK) - OpenSt...
VirtualTech Japan Inc.
 
Improving Kafka at-least-once performance at Uber
Improving Kafka at-least-once performance at UberImproving Kafka at-least-once performance at Uber
Improving Kafka at-least-once performance at Uber
Ying Zheng
 
Keystone fernet token
Keystone fernet tokenKeystone fernet token
Keystone fernet token
Yuki Nishiwaki
 
Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latenc...
Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latenc...Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latenc...
Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latenc...
Henning Jacobs
 
Running Kafka On Kubernetes With Strimzi For Real-Time Streaming Applications
Running Kafka On Kubernetes With Strimzi For Real-Time Streaming ApplicationsRunning Kafka On Kubernetes With Strimzi For Real-Time Streaming Applications
Running Kafka On Kubernetes With Strimzi For Real-Time Streaming Applications
Lightbend
 
JVMに裏から手を出す!JVMTIに触れてみよう(オープンソースカンファレンス2020 Online/Hiroshima 講演資料)
JVMに裏から手を出す!JVMTIに触れてみよう(オープンソースカンファレンス2020 Online/Hiroshima 講演資料)JVMに裏から手を出す!JVMTIに触れてみよう(オープンソースカンファレンス2020 Online/Hiroshima 講演資料)
JVMに裏から手を出す!JVMTIに触れてみよう(オープンソースカンファレンス2020 Online/Hiroshima 講演資料)
NTT DATA Technology & Innovation
 
LINEのMySQL運用について 修正版
LINEのMySQL運用について 修正版LINEのMySQL運用について 修正版
LINEのMySQL運用について 修正版
LINE Corporation
 
今話題のいろいろなコンテナランタイムを比較してみた
今話題のいろいろなコンテナランタイムを比較してみた今話題のいろいろなコンテナランタイムを比較してみた
今話題のいろいろなコンテナランタイムを比較してみた
Kohei Tokunaga
 
Kubernetes環境で実現するWebアプリケーションセキュリティ
Kubernetes環境で実現するWebアプリケーションセキュリティKubernetes環境で実現するWebアプリケーションセキュリティ
Kubernetes環境で実現するWebアプリケーションセキュリティ
NGINX, Inc.
 
OpenJDK トラブルシューティング #javacasual
OpenJDK トラブルシューティング #javacasualOpenJDK トラブルシューティング #javacasual
OpenJDK トラブルシューティング #javacasual
Yuji Kubota
 
ゼロからはじめるKVM超入門
ゼロからはじめるKVM超入門ゼロからはじめるKVM超入門
ゼロからはじめるKVM超入門
VirtualTech Japan Inc.
 
Introduction to hazelcast
Introduction to hazelcastIntroduction to hazelcast
Introduction to hazelcast
Emin Demirci
 
Scylla Summit 2022: Scylla 5.0 New Features, Part 2
Scylla Summit 2022: Scylla 5.0 New Features, Part 2Scylla Summit 2022: Scylla 5.0 New Features, Part 2
Scylla Summit 2022: Scylla 5.0 New Features, Part 2
ScyllaDB
 

What's hot (20)

急速に進化を続けるCNIプラグイン Antrea
急速に進化を続けるCNIプラグイン Antrea 急速に進化を続けるCNIプラグイン Antrea
急速に進化を続けるCNIプラグイン Antrea
 
Autoscaling Flink with Reactive Mode
Autoscaling Flink with Reactive ModeAutoscaling Flink with Reactive Mode
Autoscaling Flink with Reactive Mode
 
Where is my bottleneck? Performance troubleshooting in Flink
Where is my bottleneck? Performance troubleshooting in FlinkWhere is my bottleneck? Performance troubleshooting in Flink
Where is my bottleneck? Performance troubleshooting in Flink
 
Garbage First Garbage Collector (G1 GC): Current and Future Adaptability and ...
Garbage First Garbage Collector (G1 GC): Current and Future Adaptability and ...Garbage First Garbage Collector (G1 GC): Current and Future Adaptability and ...
Garbage First Garbage Collector (G1 GC): Current and Future Adaptability and ...
 
Metaspace
MetaspaceMetaspace
Metaspace
 
[오픈소스컨설팅]Java Performance Tuning
[오픈소스컨설팅]Java Performance Tuning[오픈소스컨설팅]Java Performance Tuning
[오픈소스컨설팅]Java Performance Tuning
 
スレッドダンプの読み方
スレッドダンプの読み方スレッドダンプの読み方
スレッドダンプの読み方
 
OVS VXLAN Network Accelaration on OpenStack (VXLAN offload and DPDK) - OpenSt...
OVS VXLAN Network Accelaration on OpenStack (VXLAN offload and DPDK) - OpenSt...OVS VXLAN Network Accelaration on OpenStack (VXLAN offload and DPDK) - OpenSt...
OVS VXLAN Network Accelaration on OpenStack (VXLAN offload and DPDK) - OpenSt...
 
Improving Kafka at-least-once performance at Uber
Improving Kafka at-least-once performance at UberImproving Kafka at-least-once performance at Uber
Improving Kafka at-least-once performance at Uber
 
Keystone fernet token
Keystone fernet tokenKeystone fernet token
Keystone fernet token
 
Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latenc...
Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latenc...Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latenc...
Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latenc...
 
Running Kafka On Kubernetes With Strimzi For Real-Time Streaming Applications
Running Kafka On Kubernetes With Strimzi For Real-Time Streaming ApplicationsRunning Kafka On Kubernetes With Strimzi For Real-Time Streaming Applications
Running Kafka On Kubernetes With Strimzi For Real-Time Streaming Applications
 
JVMに裏から手を出す!JVMTIに触れてみよう(オープンソースカンファレンス2020 Online/Hiroshima 講演資料)
JVMに裏から手を出す!JVMTIに触れてみよう(オープンソースカンファレンス2020 Online/Hiroshima 講演資料)JVMに裏から手を出す!JVMTIに触れてみよう(オープンソースカンファレンス2020 Online/Hiroshima 講演資料)
JVMに裏から手を出す!JVMTIに触れてみよう(オープンソースカンファレンス2020 Online/Hiroshima 講演資料)
 
LINEのMySQL運用について 修正版
LINEのMySQL運用について 修正版LINEのMySQL運用について 修正版
LINEのMySQL運用について 修正版
 
今話題のいろいろなコンテナランタイムを比較してみた
今話題のいろいろなコンテナランタイムを比較してみた今話題のいろいろなコンテナランタイムを比較してみた
今話題のいろいろなコンテナランタイムを比較してみた
 
Kubernetes環境で実現するWebアプリケーションセキュリティ
Kubernetes環境で実現するWebアプリケーションセキュリティKubernetes環境で実現するWebアプリケーションセキュリティ
Kubernetes環境で実現するWebアプリケーションセキュリティ
 
OpenJDK トラブルシューティング #javacasual
OpenJDK トラブルシューティング #javacasualOpenJDK トラブルシューティング #javacasual
OpenJDK トラブルシューティング #javacasual
 
ゼロからはじめるKVM超入門
ゼロからはじめるKVM超入門ゼロからはじめるKVM超入門
ゼロからはじめるKVM超入門
 
Introduction to hazelcast
Introduction to hazelcastIntroduction to hazelcast
Introduction to hazelcast
 
Scylla Summit 2022: Scylla 5.0 New Features, Part 2
Scylla Summit 2022: Scylla 5.0 New Features, Part 2Scylla Summit 2022: Scylla 5.0 New Features, Part 2
Scylla Summit 2022: Scylla 5.0 New Features, Part 2
 

Similar to Way Improved :) GC Tuning Confessions - presented at JavaOne2015

GC Tuning Confessions Of A Performance Engineer - Improved :)
GC Tuning Confessions Of A Performance Engineer - Improved :)GC Tuning Confessions Of A Performance Engineer - Improved :)
GC Tuning Confessions Of A Performance Engineer - Improved :)
Monica Beckwith
 
GC Tuning Confessions Of A Performance Engineer
GC Tuning Confessions Of A Performance EngineerGC Tuning Confessions Of A Performance Engineer
GC Tuning Confessions Of A Performance Engineer
Monica Beckwith
 
The Performance Engineer's Guide To (OpenJDK) HotSpot Garbage Collection - Th...
The Performance Engineer's Guide To (OpenJDK) HotSpot Garbage Collection - Th...The Performance Engineer's Guide To (OpenJDK) HotSpot Garbage Collection - Th...
The Performance Engineer's Guide To (OpenJDK) HotSpot Garbage Collection - Th...
Monica Beckwith
 
Garbage First Garbage Collector: Where the Rubber Meets the Road!
Garbage First Garbage Collector: Where the Rubber Meets the Road!Garbage First Garbage Collector: Where the Rubber Meets the Road!
Garbage First Garbage Collector: Where the Rubber Meets the Road!
Monica Beckwith
 
RE-FRAC OF SHALE WELLS USING ARTIFICIAL INTELLIGENCE
RE-FRAC OF SHALE WELLS USING ARTIFICIAL INTELLIGENCERE-FRAC OF SHALE WELLS USING ARTIFICIAL INTELLIGENCE
RE-FRAC OF SHALE WELLS USING ARTIFICIAL INTELLIGENCE
iQHub
 
Lock free programming- pro tips
Lock free programming- pro tipsLock free programming- pro tips
Lock free programming- pro tips
Jean-Philippe BEMPEL
 
The Performance Engineer's Guide to Java (HotSpot) Virtual Machine
The Performance Engineer's Guide to Java (HotSpot) Virtual MachineThe Performance Engineer's Guide to Java (HotSpot) Virtual Machine
The Performance Engineer's Guide to Java (HotSpot) Virtual Machine
Monica Beckwith
 
JFokus Java 9 contended locking performance
JFokus Java 9 contended locking performanceJFokus Java 9 contended locking performance
JFokus Java 9 contended locking performance
Monica Beckwith
 
OPENJDK: IN THE NEW AGE OF CONCURRENT GARBAGE COLLECTORS
OPENJDK: IN THE NEW AGE OF CONCURRENT GARBAGE COLLECTORSOPENJDK: IN THE NEW AGE OF CONCURRENT GARBAGE COLLECTORS
OPENJDK: IN THE NEW AGE OF CONCURRENT GARBAGE COLLECTORS
Monica Beckwith
 
Lecture-5-STA.pdf
Lecture-5-STA.pdfLecture-5-STA.pdf
Lecture-5-STA.pdf
KarthikYogesh3
 
sta slide ref.pdf
sta slide ref.pdfsta slide ref.pdf
sta slide ref.pdf
quandao25
 
Java Performance Engineer's Survival Guide
Java Performance Engineer's Survival GuideJava Performance Engineer's Survival Guide
Java Performance Engineer's Survival Guide
Monica Beckwith
 
The Performance Engineer's Guide To HotSpot Just-in-Time Compilation
The Performance Engineer's Guide To HotSpot Just-in-Time CompilationThe Performance Engineer's Guide To HotSpot Just-in-Time Compilation
The Performance Engineer's Guide To HotSpot Just-in-Time Compilation
Monica Beckwith
 
Kamaelia-ACCU-20050422
Kamaelia-ACCU-20050422Kamaelia-ACCU-20050422
Kamaelia-ACCU-20050422
journeyer
 
Products & Services for Microelectronics
Products & Services for MicroelectronicsProducts & Services for Microelectronics
Products & Services for Microelectronics
Kamal Karimanal
 
Modeling the IoT with TitanDB and Cassandra
Modeling the IoT with TitanDB and CassandraModeling the IoT with TitanDB and Cassandra
Modeling the IoT with TitanDB and Cassandra
twilmes
 
QCon London: Low latency Java in the real world - LMAX Exchange and the Zing JVM
QCon London: Low latency Java in the real world - LMAX Exchange and the Zing JVMQCon London: Low latency Java in the real world - LMAX Exchange and the Zing JVM
QCon London: Low latency Java in the real world - LMAX Exchange and the Zing JVM
Azul Systems, Inc.
 
Game of Performance: A Song of JIT and GC
Game of Performance: A Song of JIT and GCGame of Performance: A Song of JIT and GC
Game of Performance: A Song of JIT and GC
Monica Beckwith
 
DataEngConf: Apache Kafka at Rocana: a scalable, distributed log for machine ...
DataEngConf: Apache Kafka at Rocana: a scalable, distributed log for machine ...DataEngConf: Apache Kafka at Rocana: a scalable, distributed log for machine ...
DataEngConf: Apache Kafka at Rocana: a scalable, distributed log for machine ...
Hakka Labs
 
Kubecon seattle 2018 workshop slides
Kubecon seattle 2018 workshop slidesKubecon seattle 2018 workshop slides
Kubecon seattle 2018 workshop slides
Weaveworks
 

Similar to Way Improved :) GC Tuning Confessions - presented at JavaOne2015 (20)

GC Tuning Confessions Of A Performance Engineer - Improved :)
GC Tuning Confessions Of A Performance Engineer - Improved :)GC Tuning Confessions Of A Performance Engineer - Improved :)
GC Tuning Confessions Of A Performance Engineer - Improved :)
 
GC Tuning Confessions Of A Performance Engineer
GC Tuning Confessions Of A Performance EngineerGC Tuning Confessions Of A Performance Engineer
GC Tuning Confessions Of A Performance Engineer
 
The Performance Engineer's Guide To (OpenJDK) HotSpot Garbage Collection - Th...
The Performance Engineer's Guide To (OpenJDK) HotSpot Garbage Collection - Th...The Performance Engineer's Guide To (OpenJDK) HotSpot Garbage Collection - Th...
The Performance Engineer's Guide To (OpenJDK) HotSpot Garbage Collection - Th...
 
Garbage First Garbage Collector: Where the Rubber Meets the Road!
Garbage First Garbage Collector: Where the Rubber Meets the Road!Garbage First Garbage Collector: Where the Rubber Meets the Road!
Garbage First Garbage Collector: Where the Rubber Meets the Road!
 
RE-FRAC OF SHALE WELLS USING ARTIFICIAL INTELLIGENCE
RE-FRAC OF SHALE WELLS USING ARTIFICIAL INTELLIGENCERE-FRAC OF SHALE WELLS USING ARTIFICIAL INTELLIGENCE
RE-FRAC OF SHALE WELLS USING ARTIFICIAL INTELLIGENCE
 
Lock free programming- pro tips
Lock free programming- pro tipsLock free programming- pro tips
Lock free programming- pro tips
 
The Performance Engineer's Guide to Java (HotSpot) Virtual Machine
The Performance Engineer's Guide to Java (HotSpot) Virtual MachineThe Performance Engineer's Guide to Java (HotSpot) Virtual Machine
The Performance Engineer's Guide to Java (HotSpot) Virtual Machine
 
JFokus Java 9 contended locking performance
JFokus Java 9 contended locking performanceJFokus Java 9 contended locking performance
JFokus Java 9 contended locking performance
 
OPENJDK: IN THE NEW AGE OF CONCURRENT GARBAGE COLLECTORS
OPENJDK: IN THE NEW AGE OF CONCURRENT GARBAGE COLLECTORSOPENJDK: IN THE NEW AGE OF CONCURRENT GARBAGE COLLECTORS
OPENJDK: IN THE NEW AGE OF CONCURRENT GARBAGE COLLECTORS
 
Lecture-5-STA.pdf
Lecture-5-STA.pdfLecture-5-STA.pdf
Lecture-5-STA.pdf
 
sta slide ref.pdf
sta slide ref.pdfsta slide ref.pdf
sta slide ref.pdf
 
Java Performance Engineer's Survival Guide
Java Performance Engineer's Survival GuideJava Performance Engineer's Survival Guide
Java Performance Engineer's Survival Guide
 
The Performance Engineer's Guide To HotSpot Just-in-Time Compilation
The Performance Engineer's Guide To HotSpot Just-in-Time CompilationThe Performance Engineer's Guide To HotSpot Just-in-Time Compilation
The Performance Engineer's Guide To HotSpot Just-in-Time Compilation
 
Kamaelia-ACCU-20050422
Kamaelia-ACCU-20050422Kamaelia-ACCU-20050422
Kamaelia-ACCU-20050422
 
Products & Services for Microelectronics
Products & Services for MicroelectronicsProducts & Services for Microelectronics
Products & Services for Microelectronics
 
Modeling the IoT with TitanDB and Cassandra
Modeling the IoT with TitanDB and CassandraModeling the IoT with TitanDB and Cassandra
Modeling the IoT with TitanDB and Cassandra
 
QCon London: Low latency Java in the real world - LMAX Exchange and the Zing JVM
QCon London: Low latency Java in the real world - LMAX Exchange and the Zing JVMQCon London: Low latency Java in the real world - LMAX Exchange and the Zing JVM
QCon London: Low latency Java in the real world - LMAX Exchange and the Zing JVM
 
Game of Performance: A Song of JIT and GC
Game of Performance: A Song of JIT and GCGame of Performance: A Song of JIT and GC
Game of Performance: A Song of JIT and GC
 
DataEngConf: Apache Kafka at Rocana: a scalable, distributed log for machine ...
DataEngConf: Apache Kafka at Rocana: a scalable, distributed log for machine ...DataEngConf: Apache Kafka at Rocana: a scalable, distributed log for machine ...
DataEngConf: Apache Kafka at Rocana: a scalable, distributed log for machine ...
 
Kubecon seattle 2018 workshop slides
Kubecon seattle 2018 workshop slidesKubecon seattle 2018 workshop slides
Kubecon seattle 2018 workshop slides
 

More from Monica Beckwith

The ilities of software engineering.pptx
The ilities of software engineering.pptxThe ilities of software engineering.pptx
The ilities of software engineering.pptx
Monica Beckwith
 
A G1GC Saga-KCJUG.pptx
A G1GC Saga-KCJUG.pptxA G1GC Saga-KCJUG.pptx
A G1GC Saga-KCJUG.pptx
Monica Beckwith
 
ZGC-SnowOne.pdf
ZGC-SnowOne.pdfZGC-SnowOne.pdf
ZGC-SnowOne.pdf
Monica Beckwith
 
QCon London.pdf
QCon London.pdfQCon London.pdf
QCon London.pdf
Monica Beckwith
 
Enabling Java: Windows on Arm64 - A Success Story!
Enabling Java: Windows on Arm64 - A Success Story!Enabling Java: Windows on Arm64 - A Success Story!
Enabling Java: Windows on Arm64 - A Success Story!
Monica Beckwith
 
Applying Concurrency Cookbook Recipes to SPEC JBB
Applying Concurrency Cookbook Recipes to SPEC JBBApplying Concurrency Cookbook Recipes to SPEC JBB
Applying Concurrency Cookbook Recipes to SPEC JBB
Monica Beckwith
 
Intro to Garbage Collection
Intro to Garbage CollectionIntro to Garbage Collection
Intro to Garbage Collection
Monica Beckwith
 
OpenJDK Concurrent Collectors
OpenJDK Concurrent CollectorsOpenJDK Concurrent Collectors
OpenJDK Concurrent Collectors
Monica Beckwith
 
Java 9: The (G1) GC Awakens!
Java 9: The (G1) GC Awakens!Java 9: The (G1) GC Awakens!
Java 9: The (G1) GC Awakens!
Monica Beckwith
 
Garbage First Garbage Collector (G1 GC) - Migration to, Expectations and Adva...
Garbage First Garbage Collector (G1 GC) - Migration to, Expectations and Adva...Garbage First Garbage Collector (G1 GC) - Migration to, Expectations and Adva...
Garbage First Garbage Collector (G1 GC) - Migration to, Expectations and Adva...
Monica Beckwith
 

More from Monica Beckwith (10)

The ilities of software engineering.pptx
The ilities of software engineering.pptxThe ilities of software engineering.pptx
The ilities of software engineering.pptx
 
A G1GC Saga-KCJUG.pptx
A G1GC Saga-KCJUG.pptxA G1GC Saga-KCJUG.pptx
A G1GC Saga-KCJUG.pptx
 
ZGC-SnowOne.pdf
ZGC-SnowOne.pdfZGC-SnowOne.pdf
ZGC-SnowOne.pdf
 
QCon London.pdf
QCon London.pdfQCon London.pdf
QCon London.pdf
 
Enabling Java: Windows on Arm64 - A Success Story!
Enabling Java: Windows on Arm64 - A Success Story!Enabling Java: Windows on Arm64 - A Success Story!
Enabling Java: Windows on Arm64 - A Success Story!
 
Applying Concurrency Cookbook Recipes to SPEC JBB
Applying Concurrency Cookbook Recipes to SPEC JBBApplying Concurrency Cookbook Recipes to SPEC JBB
Applying Concurrency Cookbook Recipes to SPEC JBB
 
Intro to Garbage Collection
Intro to Garbage CollectionIntro to Garbage Collection
Intro to Garbage Collection
 
OpenJDK Concurrent Collectors
OpenJDK Concurrent CollectorsOpenJDK Concurrent Collectors
OpenJDK Concurrent Collectors
 
Java 9: The (G1) GC Awakens!
Java 9: The (G1) GC Awakens!Java 9: The (G1) GC Awakens!
Java 9: The (G1) GC Awakens!
 
Garbage First Garbage Collector (G1 GC) - Migration to, Expectations and Adva...
Garbage First Garbage Collector (G1 GC) - Migration to, Expectations and Adva...Garbage First Garbage Collector (G1 GC) - Migration to, Expectations and Adva...
Garbage First Garbage Collector (G1 GC) - Migration to, Expectations and Adva...
 

Recently uploaded

GT degree offer diploma Transcript
GT degree offer diploma TranscriptGT degree offer diploma Transcript
GT degree offer diploma Transcript
attueb
 
IoT In Manufacturing_ Use Cases, Benefits, and Challenges.pdf
IoT In Manufacturing_ Use Cases, Benefits, and Challenges.pdfIoT In Manufacturing_ Use Cases, Benefits, and Challenges.pdf
IoT In Manufacturing_ Use Cases, Benefits, and Challenges.pdf
mohitd6
 
Google ML-Kit - Understanding on-device machine learning
Google ML-Kit - Understanding on-device machine learningGoogle ML-Kit - Understanding on-device machine learning
Google ML-Kit - Understanding on-device machine learning
VishrutGoyani1
 
Private Girls Call Navi Mumbai 🛵🚡9820252231 💃 Choose Best And Top Girl Servic...
Private Girls Call Navi Mumbai 🛵🚡9820252231 💃 Choose Best And Top Girl Servic...Private Girls Call Navi Mumbai 🛵🚡9820252231 💃 Choose Best And Top Girl Servic...
Private Girls Call Navi Mumbai 🛵🚡9820252231 💃 Choose Best And Top Girl Servic...
902basic
 
當測試開始左移
當測試開始左移當測試開始左移
當測試開始左移
Jersey (CHE-PING) Su
 
Folding Cheat Sheet #7 - seventh in a series
Folding Cheat Sheet #7 - seventh in a seriesFolding Cheat Sheet #7 - seventh in a series
Folding Cheat Sheet #7 - seventh in a series
Philip Schwarz
 
Wired_2.0_Create_AmsterdamJUG_09072024.pptx
Wired_2.0_Create_AmsterdamJUG_09072024.pptxWired_2.0_Create_AmsterdamJUG_09072024.pptx
Wired_2.0_Create_AmsterdamJUG_09072024.pptx
SimonedeGijt
 
11 Top Cross Browser Testing Tools to Know About.pdf
11 Top Cross Browser Testing Tools to Know About.pdf11 Top Cross Browser Testing Tools to Know About.pdf
11 Top Cross Browser Testing Tools to Know About.pdf
kalichargn70th171
 
VVIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 i...
VVIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 i...VVIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 i...
VVIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 i...
jealousviolet
 
welcome to presentation on Google Apps
welcome to   presentation on Google Appswelcome to   presentation on Google Apps
welcome to presentation on Google Apps
AsifKarimJim
 
Vip Girls Call ServiCe Hyderabad 0000000000 Pooja Best High Class Hyderabad A...
Vip Girls Call ServiCe Hyderabad 0000000000 Pooja Best High Class Hyderabad A...Vip Girls Call ServiCe Hyderabad 0000000000 Pooja Best High Class Hyderabad A...
Vip Girls Call ServiCe Hyderabad 0000000000 Pooja Best High Class Hyderabad A...
ashiklo9823
 
Maximizing Efficiency and Profitability: Optimizing Data Systems, Enhancing C...
Maximizing Efficiency and Profitability: Optimizing Data Systems, Enhancing C...Maximizing Efficiency and Profitability: Optimizing Data Systems, Enhancing C...
Maximizing Efficiency and Profitability: Optimizing Data Systems, Enhancing C...
OnePlan Solutions
 
Girls Call Jogeshwari 9967584737 Provide Best And Top Girl Service And No1 in...
Girls Call Jogeshwari 9967584737 Provide Best And Top Girl Service And No1 in...Girls Call Jogeshwari 9967584737 Provide Best And Top Girl Service And No1 in...
Girls Call Jogeshwari 9967584737 Provide Best And Top Girl Service And No1 in...
simran hot girls
 
TEQnation 2024: Sustainable Software: May the Green Code Be with You
TEQnation 2024: Sustainable Software: May the Green Code Be with YouTEQnation 2024: Sustainable Software: May the Green Code Be with You
TEQnation 2024: Sustainable Software: May the Green Code Be with You
marcofolio
 
Celebrity Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Servic...
Celebrity Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Servic...Celebrity Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Servic...
Celebrity Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Servic...
45unexpected
 
Independent Girls call Service Pune 000XX00000 Provide Best And Top Girl Serv...
Independent Girls call Service Pune 000XX00000 Provide Best And Top Girl Serv...Independent Girls call Service Pune 000XX00000 Provide Best And Top Girl Serv...
Independent Girls call Service Pune 000XX00000 Provide Best And Top Girl Serv...
bhumivarma35300
 
Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...
Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...
Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...
confluent
 
AWS DevOps-Tutorial CHANAKYA SRIYAN DUKKA.
AWS DevOps-Tutorial CHANAKYA SRIYAN DUKKA.AWS DevOps-Tutorial CHANAKYA SRIYAN DUKKA.
AWS DevOps-Tutorial CHANAKYA SRIYAN DUKKA.
Srinivas Dukka
 
Artificial intelligence in customer services or chatbots
Artificial intelligence  in customer services or chatbotsArtificial intelligence  in customer services or chatbots
Artificial intelligence in customer services or chatbots
kayash1656
 
How To Fill Timesheet in TaskSprint: Quick Guide 2024
How To Fill Timesheet in TaskSprint: Quick Guide 2024How To Fill Timesheet in TaskSprint: Quick Guide 2024
How To Fill Timesheet in TaskSprint: Quick Guide 2024
TaskSprint | Employee Efficiency Software
 

Recently uploaded (20)

GT degree offer diploma Transcript
GT degree offer diploma TranscriptGT degree offer diploma Transcript
GT degree offer diploma Transcript
 
IoT In Manufacturing_ Use Cases, Benefits, and Challenges.pdf
IoT In Manufacturing_ Use Cases, Benefits, and Challenges.pdfIoT In Manufacturing_ Use Cases, Benefits, and Challenges.pdf
IoT In Manufacturing_ Use Cases, Benefits, and Challenges.pdf
 
Google ML-Kit - Understanding on-device machine learning
Google ML-Kit - Understanding on-device machine learningGoogle ML-Kit - Understanding on-device machine learning
Google ML-Kit - Understanding on-device machine learning
 
Private Girls Call Navi Mumbai 🛵🚡9820252231 💃 Choose Best And Top Girl Servic...
Private Girls Call Navi Mumbai 🛵🚡9820252231 💃 Choose Best And Top Girl Servic...Private Girls Call Navi Mumbai 🛵🚡9820252231 💃 Choose Best And Top Girl Servic...
Private Girls Call Navi Mumbai 🛵🚡9820252231 💃 Choose Best And Top Girl Servic...
 
當測試開始左移
當測試開始左移當測試開始左移
當測試開始左移
 
Folding Cheat Sheet #7 - seventh in a series
Folding Cheat Sheet #7 - seventh in a seriesFolding Cheat Sheet #7 - seventh in a series
Folding Cheat Sheet #7 - seventh in a series
 
Wired_2.0_Create_AmsterdamJUG_09072024.pptx
Wired_2.0_Create_AmsterdamJUG_09072024.pptxWired_2.0_Create_AmsterdamJUG_09072024.pptx
Wired_2.0_Create_AmsterdamJUG_09072024.pptx
 
11 Top Cross Browser Testing Tools to Know About.pdf
11 Top Cross Browser Testing Tools to Know About.pdf11 Top Cross Browser Testing Tools to Know About.pdf
11 Top Cross Browser Testing Tools to Know About.pdf
 
VVIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 i...
VVIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 i...VVIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 i...
VVIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 i...
 
welcome to presentation on Google Apps
welcome to   presentation on Google Appswelcome to   presentation on Google Apps
welcome to presentation on Google Apps
 
Vip Girls Call ServiCe Hyderabad 0000000000 Pooja Best High Class Hyderabad A...
Vip Girls Call ServiCe Hyderabad 0000000000 Pooja Best High Class Hyderabad A...Vip Girls Call ServiCe Hyderabad 0000000000 Pooja Best High Class Hyderabad A...
Vip Girls Call ServiCe Hyderabad 0000000000 Pooja Best High Class Hyderabad A...
 
Maximizing Efficiency and Profitability: Optimizing Data Systems, Enhancing C...
Maximizing Efficiency and Profitability: Optimizing Data Systems, Enhancing C...Maximizing Efficiency and Profitability: Optimizing Data Systems, Enhancing C...
Maximizing Efficiency and Profitability: Optimizing Data Systems, Enhancing C...
 
Girls Call Jogeshwari 9967584737 Provide Best And Top Girl Service And No1 in...
Girls Call Jogeshwari 9967584737 Provide Best And Top Girl Service And No1 in...Girls Call Jogeshwari 9967584737 Provide Best And Top Girl Service And No1 in...
Girls Call Jogeshwari 9967584737 Provide Best And Top Girl Service And No1 in...
 
TEQnation 2024: Sustainable Software: May the Green Code Be with You
TEQnation 2024: Sustainable Software: May the Green Code Be with YouTEQnation 2024: Sustainable Software: May the Green Code Be with You
TEQnation 2024: Sustainable Software: May the Green Code Be with You
 
Celebrity Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Servic...
Celebrity Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Servic...Celebrity Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Servic...
Celebrity Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Servic...
 
Independent Girls call Service Pune 000XX00000 Provide Best And Top Girl Serv...
Independent Girls call Service Pune 000XX00000 Provide Best And Top Girl Serv...Independent Girls call Service Pune 000XX00000 Provide Best And Top Girl Serv...
Independent Girls call Service Pune 000XX00000 Provide Best And Top Girl Serv...
 
Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...
Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...
Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...
 
AWS DevOps-Tutorial CHANAKYA SRIYAN DUKKA.
AWS DevOps-Tutorial CHANAKYA SRIYAN DUKKA.AWS DevOps-Tutorial CHANAKYA SRIYAN DUKKA.
AWS DevOps-Tutorial CHANAKYA SRIYAN DUKKA.
 
Artificial intelligence in customer services or chatbots
Artificial intelligence  in customer services or chatbotsArtificial intelligence  in customer services or chatbots
Artificial intelligence in customer services or chatbots
 
How To Fill Timesheet in TaskSprint: Quick Guide 2024
How To Fill Timesheet in TaskSprint: Quick Guide 2024How To Fill Timesheet in TaskSprint: Quick Guide 2024
How To Fill Timesheet in TaskSprint: Quick Guide 2024
 

Way Improved :) GC Tuning Confessions - presented at JavaOne2015

  • 1. GC Tuning Confessions Of A Performance Engineer Monica Beckwith monica@codekaram.com @mon_beck www.linkedin.com/in/monicabeckwith JavaOne Conference Oct. 26th, 2015
  • 2. ©2015 CodeKaram About Me • JVM/GC Performance Engineer/Consultant • Worked at AMD, Sun, Oracle… • Worked with HotSpot JVM • JVM heuristics, JIT compiler, GCs: Parallel(Old) GC, G1 GC, CMS GC 2
  • 3. ©2015 CodeKaram About Today’s Talk • A little bit about Performance Engineering • Insight into Garbage Collectors • OpenJDK HotSpot GCs • The Tradeoffs • GC Algorithms • Key Topics • Summary • GC Tuneables 3
  • 5. ©2015 CodeKaram Performance Engineering 5 Requirements Design Performance Requirements & Test Plan & Development Implementation Performance Analysis
  • 6. ©2015 CodeKaram Performance Requirements • Service level agreements (SLAs) for: • throughput, • latency and other response time related metrics E.g. Response time (RT) metrics - Average (RT), max or worst-case RT, 99th percentile RT… 6
  • 7. ©2015 CodeKaram Response Time Metrics Number of GC events Minimum (ms) System1 37353 7.622 System2 34920 7.258 System3 36270 6.432 System4 40636 7.353 7
  • 8. ©2015 CodeKaram Number of GC events Minimum (ms) Average (ms) System1 37353 7.622 307.741 System2 34920 7.258 320.778 System3 36270 6.432 321.483 System4 40636 7.353 323.143 8 Response Time Metrics
  • 9. ©2015 CodeKaram Number of GC events Minimum (ms) Average (ms) 99th Percentile (ms) System1 37353 7.622 307.741 940.901 System2 34920 7.258 320.778 1006.607 System3 36270 6.432 321.483 1004.018 System4 40636 7.353 323.143 1041.225 9 Response Time Metrics
  • 10. ©2015 CodeKaram Number of GC events Minimum (ms) Average (ms) 99th Percentile (ms) Maximum (ms) System 1 37353 7.622 307.741 940.901 3131.331 System 2 34920 7.258 320.778 1006.607 2744.588 System 3 36270 6.432 321.483 1004.018 1681.308 System 4 40636 7.353 323.143 1041.225 20699.505 10 Response Time Metrics
  • 11. ©2015 CodeKaram Average (ms) 99th Percentile (ms) Maximum (ms) System1 307.741 940.901 3131.331 System2 320.778 1006.607 2744.588 System3 321.483 1004.018 1681.308 System4 323.143 1041.225 20699.505 5 full GCs and 10 evacuation failures 11 Response Time Metrics
  • 13. ©2015 CodeKaram Fun Facts! • GC can NOT eliminate your memory leaks! • GC (and heap dump) can provide an insight into your application. 13
  • 15. ©2015 CodeKaram Ideal GC? Maximize Throughput Minimize Latency 15
  • 16. ©2015 CodeKaram Ideal GC? Maximize Throughput Minimize Latency Minimize Footprint 16
  • 18. OpenJDK HotSpot GCs: The TRADE/OFF 18
  • 19. ©2015 CodeKaram Fun Fact! Most OpenJDK HotSpot users would like to increase their (Java) heap space but they fear full garbage collections. 19
  • 20. ©2015 CodeKaram The Tradeoff - • Throughput and latency are the two main drivers towards refinement of GC algorithms. 20
  • 21. ©2015 CodeKaram The Throughput Maximizer 21 Maximize Throughput Generational Old Parallel Work Young Concurrent Threads Stop-the- world Threads
  • 22. ©2015 CodeKaram Fun Fact! All GCs in OpenJDK HotSpot are generational. 22
  • 23. ©2015 CodeKaram Mr. Latency Sensitive 23 Latency Sensitive No/Partial Compaction Concurrency Garbage First Mostly Concurrent Mark and Sweep
  • 24. ©2015 CodeKaram Fun Fact! All GCs in OpenJDK HotSpot fallback to a fully compacting stop-the-world garbage collection called the “full” GC. Tuning can help avoid or postpone full GCs in many cases. 24
  • 26. ©2015 CodeKaram The Generational Java Heap Eden S0 S1 Old Generation Young Generation Allocations Survivors Tenured 26
  • 27. ©2015 CodeKaram OpenJDK HotSpot Collectors *Similar GC Algorithms Different GC Algorithms Always collected in its entirety 27
  • 28. ©2015 CodeKaram The Serial Collector Single Threaded Single Threaded - Entire Heap Marked, Swept and Compacted in its entirety Always collected in its entirety 28
  • 29. ©2015 CodeKaram Young GC Thread Java Application Threads Old GC Thread Java Application Threads Java Application Threads The Serial Collector 29
  • 30. ©2015 CodeKaram The Throughput Collector Multi-Threaded Multi-Threaded - Entire Heap Marked, Swept and Compacted in its entirety Always collected in its entirety 30
  • 31. ©2015 CodeKaram Young GC Threads Java Application Threads Old GC Threads Java Application Threads Java Application Threads The Throughput Collector 31
  • 32. ©2015 CodeKaram The CMS Collector Multi-Threaded Multi-Threaded - Mostly Concurrent Marked and Swept in its entirety Always collected in its entirety 32
  • 33. ©2015 CodeKaram The CMS Collector Young GC Threads CMS Initial Mark Threads Java Application Threads Young GC Threads CMS Remark Threads Concurrent CMS Threads 33 Concurrent CMS Threads Concurrent CMS Threads Java Application Threads Java Application Threads Java Application Threads Java Application Threads
  • 34. GC Algorithms - Key Topics 34
  • 35. 35 What Triggers Full (Fail-Safe) Collections?
  • 37. Promotion Failures In The Throughput Collector 37
  • 38. ©2015 CodeKaram The Throughput Collector - Java Heap Eden S0 S1 Old Generation Young Generation Allocations Survivors Tenured 38
  • 39. ©2015 CodeKaram Old Generation Free Space Occupied Space The Throughput Collector - Contiguous Old Generation 39
  • 40. ©2015 CodeKaram Old Generation To-be Promoted Object 1 The Throughput Collector - Contiguous Old Generation 40
  • 41. ©2015 CodeKaram The Throughput Collector - Contiguous Old Generation 41 Old Generation Free Space Occupied Space
  • 42. ©2015 CodeKaram The Throughput Collector - Contiguous Old Generation 42 Old Generation To-be Promoted Object 2
  • 43. ©2015 CodeKaram The Throughput Collector - Contiguous Old Generation 43 Old Generation Free Space Occupied Space
  • 44. ©2015 CodeKaram The Throughput Collector - Contiguous Old Generation 44 Old Generation To-be Promoted Object 3
  • 45. ©2015 CodeKaram The Throughput Collector - Contiguous Old Generation 45 Old Generation Free Space Occupied Space
  • 46. Promotion Failures & Concurrent Mode Failures In The CMS Collector 46
  • 47. ©2015 CodeKaram The CMS Collector - Old Generation Maintained By Free Lists 47 Free List Old Generation Free Space Occupied Space
  • 48. ©2015 CodeKaram To-be Promoted Object 1 48 Old Generation The CMS Collector - Old Generation Maintained By Free Lists
  • 49. ©2015 CodeKaram49 Free List Old Generation Free Space Occupied Space The CMS Collector - Old Generation Maintained By Free Lists
  • 50. ©2015 CodeKaram50 To-be Promoted Object 2 Old Generation The CMS Collector - Old Generation Maintained By Free Lists
  • 51. ©2015 CodeKaram51 Free List Old Generation Free Space Occupied Space The CMS Collector - Old Generation Maintained By Free Lists
  • 52. ©2015 CodeKaram X 52 To-be Promoted Object 3 Old Generation The CMS Collector - Old Generation Maintained By Free Lists
  • 53. ©2015 CodeKaram X 53 To-be Promoted Object 3 Old Generation The CMS Collector - Old Generation Maintained By Free Lists
  • 54. ©2015 CodeKaram X 54 To-be Promoted Object 3 Old Generation The CMS Collector - Old Generation Maintained By Free Lists
  • 55. ©2015 CodeKaram X 55 To-be Promoted Object 3 Old Generation The CMS Collector - Old Generation Maintained By Free Lists
  • 56. ©2015 CodeKaram ?? 56 To-be Promoted Object 3 Old Generation The CMS Collector - Old Generation Maintained By Free Lists
  • 57. ©2015 CodeKaram Promotion Failure!! 57 To-be Promoted Object 3 Old Generation The CMS Collector - Fragmented Old Generation
  • 58. ©2015 CodeKaram • Your old generation is getting filled before a concurrent cycle can complete and free up space. • Fragmentation has crept in. • Causes - marking threshold is too high, heap too small, or high application mutation rate 58 The CMS Collector - Concurrent Mode Failures
  • 59. Incremental Compaction In The G1 Collector 59
  • 60. ©2015 CodeKaram The Garbage First Collector - Regionalized Heap Eden Old Old Eden Old Survivor Humongous 60
  • 61. ©2015 CodeKaram The Garbage First Collector Eden Old Old Eden Old Surv ivor E.g.: Current heap configuration - 61
  • 62. ©2015 CodeKaram The Garbage First Collector Eden Old Old Eden Old Surv ivor E.g.: During a young collection - 62
  • 63. ©2015 CodeKaram The Garbage First Collector Old Old Old E.g.: After a young collection - Sur viv or Ol d 63
  • 64. ©2015 CodeKaram The Garbage First Collector Old Old Old E.g.: Current heap configuration - Sur viv or Ol d Eden Eden Old 64
  • 65. ©2015 CodeKaram Old Old Old E.g.: Reclamation of a garbage-filled region during the cleanup phase - Sur viv or Ol d Eden Eden Old The Garbage First Collector 65
  • 66. ©2015 CodeKaram Old Old Old E.g.: Reclamation of a garbage-filled region during the cleanup phase - Sur viv or Ol d Eden Eden The Garbage First Collector 66
  • 67. ©2015 CodeKaram Old Old Old E.g.: Current heap configuration - Sur viv or Ol d Eden Eden The Garbage First Collector 67
  • 68. ©2015 CodeKaram Old Old Old E.g.: During a mixed collection - Sur viv or Ol d Eden Eden The Garbage First Collector 68
  • 69. ©2015 CodeKaram Old E.g.: After a mixed collection - O l d Sur vivo r Old The Garbage First Collector 69
  • 71. ©2015 CodeKaram • When there are no more regions available for survivors or tenured objects, G1 GC encounters an evacuation failure. • An evacuation failure is expensive and the usual pattern is that if you see a couple of evacuation failures; full GC could* soon follow. The Garbage First Collector - Evacuation Failures 71
  • 72. ©2015 CodeKaram A heavily tuned JVM command line may be restricting the G1 GC ergonomics and adaptability. Start with just your heap sizes and a reasonable pause time goal The Garbage First Collector - Avoiding Evacuation Failures 72
  • 73. ©2015 CodeKaram Your live data set + long live transient data may be too large for the old generation Check LDS+ and increase heap to accommodate everything in the old generation. The Garbage First Collector - Avoiding Evacuation Failures 73
  • 74. ©2015 CodeKaram Initiating Heap Occupancy Threshold could be the issue. Check IHOP and make sure it accommodates the LDS+. IHOP threshold too high -> Delayed marking -> Delayed incremental compaction -> Evacuation Failures! The Garbage First Collector - Avoiding Evacuation Failures 74
  • 75. ©2015 CodeKaram Marking Cycle could be taking too long to complete? Increase concurrent marking threads Reduce IHOP The Garbage First Collector - Avoiding Evacuation Failures 75
  • 76. ©2015 CodeKaram to-space survivors are the problem? Increase the G1ReservePercent, if to-space survivors are triggering the evacuation failures! The Garbage First Collector - Avoiding Evacuation Failures 76
  • 77. ©2015 CodeKaram fragmentation an issue? The Garbage First Collector - Avoiding Evacuation Failures 77
  • 78. Fragmentation In The G1 Collector 78
  • 79. ©2015 CodeKaram • G1 GC is designed to “absorb” some fragmentation. • Default is 5% of the total Java heap • Tradeoff so that expensive regions are left out. G1 Heap Waste Percentage 79
  • 80. ©2015 CodeKaram G1 Mixed GC (Region) Liveness Threshold 80 • G1 GC’s old regions are also designed to “absorb” some fragmentation. • Default is 85% liveness in a G1 region. • Tradeoff so that expensive regions are left out.
  • 81. ©2015 CodeKaram The Garbage First Collector - Humongous Objects 81 An old generation region A young generation region
  • 82. ©2015 CodeKaram The Garbage First Collector - Humongous Objects 82 Object < 50% of G1 region size Object >= 50% of G1 region size Object > G1 region size ??
  • 83. ©2015 CodeKaram The Garbage First Collector - Humongous Objects 83 Object < 50% of G1 region size
  • 84. ©2015 CodeKaram The Garbage First Collector - Humongous Objects 84 Object >= 50% of G1 region size
  • 85. ©2015 CodeKaram The Garbage First Collector - Humongous Objects 85 Object > G1 region size
  • 86. ©2015 CodeKaram The Garbage First Collector - Humongous Objects 86 Object NOT Humongous Object Humongous Object Humongous -> Needs Contiguous Regions Wasted Space!
  • 87. ©2015 CodeKaram Ideally, humongous objects are few in number and are short lived. A lot of long-lived humongous objects can cause evacuation failures since humongous regions add to the old generation occupancy. The Garbage First Collector - Humongous Objects 87
  • 88. G1 GC Logs - Key Topics 88
  • 89. ©2015 CodeKaram G1 GC Log 154.431: [GC pause (G1 Evacuation Pause) (young), 0.2584864 secs] [Parallel Time: 253.2 ms, GC Workers: 8] [GC Worker Start (ms): Min: 154431.3, Avg: 154431.4, Max: 154431.5, Diff: 0.1] [Ext Root Scanning (ms): Min: 0.1, Avg: 0.2, Max: 0.3, Diff: 0.1, Sum: 1.4] [Update RS (ms): Min: 3.3, Avg: 3.5, Max: 3.8, Diff: 0.6, Sum: 28.2] [Processed Buffers: Min: 3, Avg: 3.5, Max: 5, Diff: 2, Sum: 28] [Scan RS (ms): Min: 46.1, Avg: 46.4, Max: 46.7, Diff: 0.6, Sum: 371.2] [Code Root Scanning (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.5] [Object Copy (ms): Min: 202.7, Avg: 202.8, Max: 202.9, Diff: 0.3, Sum: 1622.4] [Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.5] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.6] [GC Worker Total (ms): Min: 253.0, Avg: 253.1, Max: 253.1, Diff: 0.1, Sum: 2024.7] [GC Worker End (ms): Min: 154684.5, Avg: 154684.5, Max: 154684.5, Diff: 0.1] [Code Root Fixup: 0.1 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.7 ms] [Other: 4.4 ms] [Choose CSet: 0.0 ms] [Ref Proc: 0.3 ms] [Ref Enq: 0.0 ms] [Redirty Cards: 0.3 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 3.2 ms] [Eden: 4972.0M(4972.0M)->0.0B(4916.0M) Survivors: 148.0M->204.0M Heap: 5295.8M(10.0G)->379.4M(10.0G)] [Times: user=1.72 sys=0.14, real=0.26 secs] 89
  • 90. ©2015 CodeKaram G1 GC Log 154.431: [GC pause (G1 Evacuation Pause) (young), 0.2584864 secs] [Parallel Time: 253.2 ms, GC Workers: 8] [GC Worker Start (ms): Min: 154431.3, Avg: 154431.4, Max: 154431.5, Diff: 0.1] [Ext Root Scanning (ms): Min: 0.1, Avg: 0.2, Max: 0.3, Diff: 0.1, Sum: 1.4] [Update RS (ms): Min: 3.3, Avg: 3.5, Max: 3.8, Diff: 0.6, Sum: 28.2] [Processed Buffers: Min: 3, Avg: 3.5, Max: 5, Diff: 2, Sum: 28] [Scan RS (ms): Min: 46.1, Avg: 46.4, Max: 46.7, Diff: 0.6, Sum: 371.2] [Code Root Scanning (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.5] [Object Copy (ms): Min: 202.7, Avg: 202.8, Max: 202.9, Diff: 0.3, Sum: 1622.4] [Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.5] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.6] [GC Worker Total (ms): Min: 253.0, Avg: 253.1, Max: 253.1, Diff: 0.1, Sum: 2024.7] [GC Worker End (ms): Min: 154684.5, Avg: 154684.5, Max: 154684.5, Diff: 0.1] [Code Root Fixup: 0.1 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.7 ms] [Other: 4.4 ms] [Choose CSet: 0.0 ms] [Ref Proc: 0.3 ms] [Ref Enq: 0.0 ms] [Redirty Cards: 0.3 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 3.2 ms] [Eden: 4972.0M(4972.0M)->0.0B(4916.0M) Survivors: 148.0M->204.0M Heap: 5295.8M(10.0G)->379.4M(10.0G)] [Times: user=1.72 sys=0.14, real=0.26 secs] 90
  • 91. ©2015 CodeKaram G1 GC Log 154.431: [GC pause (G1 Evacuation Pause) (young), 0.2584864 secs] [Parallel Time: 253.2 ms, GC Workers: 8] [GC Worker Start (ms): Min: 154431.3, Avg: 154431.4, Max: 154431.5, Diff: 0.1] [Ext Root Scanning (ms): Min: 0.1, Avg: 0.2, Max: 0.3, Diff: 0.1, Sum: 1.4] [Update RS (ms): Min: 3.3, Avg: 3.5, Max: 3.8, Diff: 0.6, Sum: 28.2] [Processed Buffers: Min: 3, Avg: 3.5, Max: 5, Diff: 2, Sum: 28] [Scan RS (ms): Min: 46.1, Avg: 46.4, Max: 46.7, Diff: 0.6, Sum: 371.2] [Code Root Scanning (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.5] [Object Copy (ms): Min: 202.7, Avg: 202.8, Max: 202.9, Diff: 0.3, Sum: 1622.4] [Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.5] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.6] [GC Worker Total (ms): Min: 253.0, Avg: 253.1, Max: 253.1, Diff: 0.1, Sum: 2024.7] [GC Worker End (ms): Min: 154684.5, Avg: 154684.5, Max: 154684.5, Diff: 0.1] [Code Root Fixup: 0.1 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.7 ms] [Other: 4.4 ms] [Choose CSet: 0.0 ms] [Ref Proc: 0.3 ms] [Ref Enq: 0.0 ms] [Redirty Cards: 0.3 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 3.2 ms] [Eden: 4972.0M(4972.0M)->0.0B(4916.0M) Survivors: 148.0M->204.0M Heap: 5295.8M(10.0G)->379.4M(10.0G)] [Times: user=1.72 sys=0.14, real=0.26 secs] 91
  • 92. ©2015 CodeKaram G1 GC Log 154.431: [GC pause (G1 Evacuation Pause) (young), 0.2584864 secs] [Parallel Time: 253.2 ms, GC Workers: 8] [GC Worker Start (ms): Min: 154431.3, Avg: 154431.4, Max: 154431.5, Diff: 0.1] [Ext Root Scanning (ms): Min: 0.1, Avg: 0.2, Max: 0.3, Diff: 0.1, Sum: 1.4] [Update RS (ms): Min: 3.3, Avg: 3.5, Max: 3.8, Diff: 0.6, Sum: 28.2] [Processed Buffers: Min: 3, Avg: 3.5, Max: 5, Diff: 2, Sum: 28] [Scan RS (ms): Min: 46.1, Avg: 46.4, Max: 46.7, Diff: 0.6, Sum: 371.2] [Code Root Scanning (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.5] [Object Copy (ms): Min: 202.7, Avg: 202.8, Max: 202.9, Diff: 0.3, Sum: 1622.4] [Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.5] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.6] [GC Worker Total (ms): Min: 253.0, Avg: 253.1, Max: 253.1, Diff: 0.1, Sum: 2024.7] [GC Worker End (ms): Min: 154684.5, Avg: 154684.5, Max: 154684.5, Diff: 0.1] [Code Root Fixup: 0.1 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.7 ms] [Other: 4.4 ms] [Choose CSet: 0.0 ms] [Ref Proc: 0.3 ms] [Ref Enq: 0.0 ms] [Redirty Cards: 0.3 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 3.2 ms] [Eden: 4972.0M(4972.0M)->0.0B(4916.0M) Survivors: 148.0M->204.0M Heap: 5295.8M(10.0G)->379.4M(10.0G)] [Times: user=1.72 sys=0.14, real=0.26 secs] 92
  • 93. ©2015 CodeKaram G1 GC Log 154.431: [GC pause (G1 Evacuation Pause) (young), 0.2584864 secs] [Parallel Time: 253.2 ms, GC Workers: 8] [GC Worker Start (ms): Min: 154431.3, Avg: 154431.4, Max: 154431.5, Diff: 0.1] [Ext Root Scanning (ms): Min: 0.1, Avg: 0.2, Max: 0.3, Diff: 0.1, Sum: 1.4] [Update RS (ms): Min: 3.3, Avg: 3.5, Max: 3.8, Diff: 0.6, Sum: 28.2] [Processed Buffers: Min: 3, Avg: 3.5, Max: 5, Diff: 2, Sum: 28] [Scan RS (ms): Min: 46.1, Avg: 46.4, Max: 46.7, Diff: 0.6, Sum: 371.2] [Code Root Scanning (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.5] [Object Copy (ms): Min: 202.7, Avg: 202.8, Max: 202.9, Diff: 0.3, Sum: 1622.4] [Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.5] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.6] [GC Worker Total (ms): Min: 253.0, Avg: 253.1, Max: 253.1, Diff: 0.1, Sum: 2024.7] [GC Worker End (ms): Min: 154684.5, Avg: 154684.5, Max: 154684.5, Diff: 0.1] [Code Root Fixup: 0.1 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.7 ms] [Other: 4.4 ms] [Choose CSet: 0.0 ms] [Ref Proc: 0.3 ms] [Ref Enq: 0.0 ms] [Redirty Cards: 0.3 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 3.2 ms] [Eden: 4972.0M(4972.0M)->0.0B(4916.0M) Survivors: 148.0M->204.0M Heap: 5295.8M(10.0G)->379.4M(10.0G)] [Times: user=1.72 sys=0.14, real=0.26 secs] 93
  • 94. ©2015 CodeKaram G1 GC Log 154.431: [GC pause (G1 Evacuation Pause) (young), 0.2584864 secs] [Parallel Time: 253.2 ms, GC Workers: 8] [GC Worker Start (ms): Min: 154431.3, Avg: 154431.4, Max: 154431.5, Diff: 0.1] [Ext Root Scanning (ms): Min: 0.1, Avg: 0.2, Max: 0.3, Diff: 0.1, Sum: 1.4] [Update RS (ms): Min: 3.3, Avg: 3.5, Max: 3.8, Diff: 0.6, Sum: 28.2] [Processed Buffers: Min: 3, Avg: 3.5, Max: 5, Diff: 2, Sum: 28] [Scan RS (ms): Min: 46.1, Avg: 46.4, Max: 46.7, Diff: 0.6, Sum: 371.2] [Code Root Scanning (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.5] [Object Copy (ms): Min: 202.7, Avg: 202.8, Max: 202.9, Diff: 0.3, Sum: 1622.4] [Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.5] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.6] [GC Worker Total (ms): Min: 253.0, Avg: 253.1, Max: 253.1, Diff: 0.1, Sum: 2024.7] [GC Worker End (ms): Min: 154684.5, Avg: 154684.5, Max: 154684.5, Diff: 0.1] [Code Root Fixup: 0.1 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.7 ms] [Other: 4.4 ms] [Choose CSet: 0.0 ms] [Ref Proc: 0.3 ms] [Ref Enq: 0.0 ms] [Redirty Cards: 0.3 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 3.2 ms] [Eden: 4972.0M(4972.0M)->0.0B(4916.0M) Survivors: 148.0M->204.0M Heap: 5295.8M(10.0G)->379.4M(10.0G)] [Times: user=1.72 sys=0.14, real=0.26 secs] 94
  • 95. ©2015 CodeKaram G1 GC Log 154.431: [GC pause (G1 Evacuation Pause) (young), 0.2584864 secs] [Parallel Time: 253.2 ms, GC Workers: 8] [GC Worker Start (ms): Min: 154431.3, Avg: 154431.4, Max: 154431.5, Diff: 0.1] [Ext Root Scanning (ms): Min: 0.1, Avg: 0.2, Max: 0.3, Diff: 0.1, Sum: 1.4] [Update RS (ms): Min: 3.3, Avg: 3.5, Max: 3.8, Diff: 0.6, Sum: 28.2] [Processed Buffers: Min: 3, Avg: 3.5, Max: 5, Diff: 2, Sum: 28] [Scan RS (ms): Min: 46.1, Avg: 46.4, Max: 46.7, Diff: 0.6, Sum: 371.2] [Code Root Scanning (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.5] [Object Copy (ms): Min: 202.7, Avg: 202.8, Max: 202.9, Diff: 0.3, Sum: 1622.4] [Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.5] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.6] [GC Worker Total (ms): Min: 253.0, Avg: 253.1, Max: 253.1, Diff: 0.1, Sum: 2024.7] [GC Worker End (ms): Min: 154684.5, Avg: 154684.5, Max: 154684.5, Diff: 0.1] [Code Root Fixup: 0.1 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.7 ms] [Other: 4.4 ms] [Choose CSet: 0.0 ms] [Ref Proc: 0.3 ms] [Ref Enq: 0.0 ms] [Redirty Cards: 0.3 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 3.2 ms] [Eden: 4972.0M(4972.0M)->0.0B(4916.0M) Survivors: 148.0M->204.0M Heap: 5295.8M(10.0G)->379.4M(10.0G)] [Times: user=1.72 sys=0.14, real=0.26 secs] 95
  • 96. ©2015 CodeKaram G1 GC Log 154.431: [GC pause (G1 Evacuation Pause) (young), 0.2584864 secs] [Parallel Time: 253.2 ms, GC Workers: 8] [GC Worker Start (ms): Min: 154431.3, Avg: 154431.4, Max: 154431.5, Diff: 0.1] [Ext Root Scanning (ms): Min: 0.1, Avg: 0.2, Max: 0.3, Diff: 0.1, Sum: 1.4] [Update RS (ms): Min: 3.3, Avg: 3.5, Max: 3.8, Diff: 0.6, Sum: 28.2] [Processed Buffers: Min: 3, Avg: 3.5, Max: 5, Diff: 2, Sum: 28] [Scan RS (ms): Min: 46.1, Avg: 46.4, Max: 46.7, Diff: 0.6, Sum: 371.2] [Code Root Scanning (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.5] [Object Copy (ms): Min: 202.7, Avg: 202.8, Max: 202.9, Diff: 0.3, Sum: 1622.4] [Termination (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.5] [GC Worker Other (ms): Min: 0.0, Avg: 0.1, Max: 0.1, Diff: 0.1, Sum: 0.6] [GC Worker Total (ms): Min: 253.0, Avg: 253.1, Max: 253.1, Diff: 0.1, Sum: 2024.7] [GC Worker End (ms): Min: 154684.5, Avg: 154684.5, Max: 154684.5, Diff: 0.1] [Code Root Fixup: 0.1 ms] [Code Root Purge: 0.0 ms] [Clear CT: 0.7 ms] [Other: 4.4 ms] [Choose CSet: 0.0 ms] [Ref Proc: 0.3 ms] [Ref Enq: 0.0 ms] [Redirty Cards: 0.3 ms] [Humongous Reclaim: 0.0 ms] [Free CSet: 3.2 ms] [Eden: 4972.0M(4972.0M)->0.0B(4916.0M) Survivors: 148.0M->204.0M Heap: 5295.8M(10.0G)->379.4M(10.0G)] [Times: user=1.72 sys=0.14, real=0.26 secs] 96
  • 97. ©2015 CodeKaram Generation Sizing [Eden: 4972.0M(4972.0M)->0.0B(4916.0M) Survivors: 148.0M->204.0M Heap: 5295.8M(10.0G)->379.4M(10.0G)] [Eden: Occupancy before GC(Eden size before GC)->Occupancy after GC(Eden size after GC) Survivors: Size before GC->Size after GC Heap: Occupancy before GC(Heap size before GC)- >Occupancy after GC(Heap size after GC)] 97
  • 98. ©2015 CodeKaram Heap Information Plot 98 OccupancyinMBs 0 7500 15000 22500 30000 TimeStamps 2500 3750 5000 6250 7500 8750 10000 11250 12500 Eden Occupancy Before GC Eden Size After GC Survivor Size After GC Old Generation Occupancy After GC Heap Occupancy Before GC Heap Size
  • 99. ©2015 CodeKaram Scaling: GC Times vs Sys Times [Times: user=1.72 sys=0.14, real=0.26 secs] Scaling: user time/real time = 6.6 99
  • 100. ©2015 CodeKaram Scaling: High Sys Times 100 Swapping? IO? Transparent huge pages enabled??* Sys memory issues? ??
  • 101. ©2015 CodeKaram Reference Processing Times [Ref Proc: 0.3 ms] Check for high time spent in ‘Ref Proc’. Also, check your Remark pause times: 9.972: [GC remark 9.972: [Finalize Marking, 0.0007865 secs] 9.973: [GC ref-proc, 0.0027669 secs] 9.976: [Unloading, 0.0075832 secs], 0.0116215 secs] If Remark pauses are high or increasing and if ‘ref- proc’ is the major contributor - use: -XX: +ParallelRefProcEnabled 101
  • 102. ©2015 CodeKaram GC Overhead vs Elapsed Time Overhead is an indication of the frequency of stop the world GC events. The more frequent the GC events - The more likely it is to negatively impact application throughput GC Elapsed Time indicated the amount of time it takes to execute stop the world GC events The higher the GC elapsed time - the lower the application responsiveness due to the GC induced latencies 102
  • 104. ©2015 CodeKaram Most Allocations Eden Space Eden Full? Start Young Collection: Keep Allocating :) Objects Aged in Survivor Space? Promote: Keep Aging :) Fast Path Promotions Fast Path Old Generation CMS promotes to fitting free space out of a free list. What have we learned so far? 104
  • 105. ©2015 CodeKaram GC Tuneables - The Throughput Collector Goal: Only promote objects after you have hazed them appropriately aged Tunables: Everything related to aging objects and generation sizing - NewRatio, (Max)NewSize, SurvivorRatio, (Max)TenuringThreshold 105
  • 106. ©2015 CodeKaram GC Tuneables - The Throughput Collector Things to remember - • Applications with steady behavior rarely need AdaptiveSizePolicy to be enabled. • Overflow gets promoted into the old generation • Provide larger survivor spaces for transient data. • In most cases, young generation sizing has the most effect on throughput • Size the young generation to maintain the GC overhead to less than 5%. 106
  • 107. ©2015 CodeKaram GC Tuneables - The CMS Collector Goal: Only promote objects after you have hazed them appropriately aged Tunables: Everything related to aging objects and young generation sizing still applies here. The concurrent thread counts and marking threshold are addition tunables for CMS 107
  • 108. ©2015 CodeKaram GC Tuneables - The CMS Collector Things to remember - • Premature promotions are very expensive in CMS and could lead to fragmentation • You can reduce the CMS cycle duration by adding more concurrent threads: ConcGCThreads. • Remember that this will increase the concurrent overhead. • You can manually tune the marking threshold (adaptive by default) • CMSInitiatingOccupancyFraction & UseCMSInitiatingOccupancyOnly will help fix the marking threshold. • Note: The threshold is expressed as a percentage of the old generation occupancy 108
  • 109. ©2015 CodeKaram GC Tuneables - The G1 Collector Goal: Get the GC ergonomics to work for you and know the defaults Tunables: • Pause time goal, heap size, max and min nursery, concurrent and parallel threads • The marking threshold, number of mixed GCs after marking, liveness threshold for the old regions, garbage toleration threshold, max old regions to be collected per mixed collection 109
  • 110. ©2015 CodeKaram GC Tuneables - The G1 Collector Things to remember - • Know your defaults! • Understand your G1HeapRegionSize - It could be any factor of two from 1MB to 32MB. G1 strives for 2048 regions. • Fixing the nursery size (using Xmn) will meddle with the GC ergonomics/adaptiveness. • Don’t set really aggressive pause time goals - this will increase the GC overhead. • Spend time taming your mixed GCs - mixed GCs are incremental collections 110
  • 111. ©2015 CodeKaram GC Tuneables - The G1 Collector Things to remember - • Taming mixed GCs: • Adjust the marking cycle according to you live data set. • Adjust you liveness threshold - this is the live occupancy threshold per region. Any region with liveness beyond this threshold will not be included in a mixed collection. • Adjust your garbage toleration threshold - helps G1 not get too aggressive with mixed collections • Distribute mixed GC pauses over a number of mixed collections - adjust your mixed GC count target and change your max old region threshold percent so that you can limit the old regions per collection 111
  • 113. ©2015 CodeKaram Further Reading • Jon Masa’s blog: https://blogs.oracle.com/ jonthecollector/entry/our_collectors • A few of my articles on InfoQ: http:// www.infoq.com/author/Monica-Beckwith • Presentations: http://www.slideshare.net/ MonicaBeckwith • Mail archives on hotspot-gc-use@openjdk.java.net & hotspot-gc-dev@openjdk.java.net 113
  • 115. ©2015 CodeKaram Performance Analysis • Monitoring: System Under Load • Utilization - CPU, IO, Sys/ Kernel, Memory bandwidth, Java heap, … • Lock statistics • Analyzing: • Utilization and time spent - GC logs, CPU, memory and application logs • Profiling: • Application, System, Memory - Java Heap. 115
  • 116. ©2015 CodeKaram JVM Performance Engineering Java/JVM performance engineering includes the study, analysis and tuning of the Just-in-time (JIT) compiler, the Garbage Collector (GC) and many a times tuning related to the Java Development Kit (JDK). 116
  • 117. ©2015 CodeKaram GC Performance Engineering • Monitor the JVM - Visual VM • Monitor and collect GC information - Visual GC (online), GC logs (offline) • Develop scripts to process GC logs; use GC Histo and JFreeCharts to plot your GC logs or use specialized tools/ log analyzers that serves your purpose. 117
  • 119. ©2015 CodeKaram What have we learned so far? - Young Generation & Collections • Young generation is always collected in its entirety. • All 3 server GCs discussed earlier follow similar mechanism for young collection. • The young collections achieve reclamation via compaction and copying of live objects. • There are a lot of options for sizing the Eden and Survivor space optimally and many GCs also have adaptive sizing and GC ergonomics for young generation collections. 119
  • 120. ©2015 CodeKaram All 3 server GCs vary in the way they collect the old generation: • For ParallelOld GC, the old generation is reclaimed and compacted in its entirety • Luckily, the compaction cost is distributed amongst parallel garbage collector worker threads. • Unluckily, the compaction cost depends a lot on the make of the live data set since at every compaction, the GC is moving live data around. • No tuning options other than the generation size adjustment and age threshold for promotion. 120 What have we learned so far? - Old Generation & Collections
  • 121. ©2015 CodeKaram • For CMS GC, the old generation is (mostly) concurrently marked and swept. Thus the reclamation of dead objects happen in place and the space is added to a free list of spaces. • The marking threshold can be tuned adaptively and manually as well. • Luckily, CMS GC doesn’t do compaction, hence reclamation is fast. • Unluckily, a long running Java application with CMS GC is prone to fragmentation which will eventually result in promotion failures which can eventually lead to full compacting garbage collection and sometimes even concurrent mode failures. • Full compacting GCs are singled threaded in CMS GC. What have we learned so far? - Old Generation & Collections 121
  • 122. ©2015 CodeKaram • For G1 GC, the old generation regions are (mostly) concurrently marked and an incremental compacting collection helps with optimizing the old generation collection. • Luckily, fragmentation is not “untunable” in G1 GC as it is in CMS GC. • Unluckily, sometimes, you may still encounter promotion/evacuation failures when G1 GC runs out of regions to copy live objects. Such an evacuation failure is expensive and can eventually lead to a full compacting GC. • Full compacting GCs are singled threaded in G1 GC. • Appropriate tuning of the old generation space and collection can help avoid evacuation failures and hence keep full GCs at bay. • G1 GC has multiple tuning options so that the GC can be adapted to your application needs. 122 What have we learned so far? - Old Generation & Collections