A definitive approach to Java heap optimization

1. Memory Kinetics
for
Scale-up and Optimization
Kenny Yu
Session ID: CON5144

2. a.
A large heap (-Xmx) is not good enough --Excessive GC activitiesPremature tenuringAllocation
failuresOutOfMemoryError
Problems

3. High throughput
Low pause time
Low cost of GC
Objectives

4. Objectives explained: Good GC

5. Approaches
Generational distribution
GC intervals ⇦ Eden + SR
Minimal major GC ⇦ OC
Approaches

6. Capacity = ArrivalRate * HoldTime
Applicable to subsystems
Eden
Survivor
OldGen
Regardless of distribution of retention time
Fundamentals: Little's Law

7. 1. Optimize GC interval and memory region sizes
Approaches

8. a. The application is re-designed to polarize the
generational distribution:
Lighter sessions
Long-living caches
Generational hypothesis: enforced
Population
Age

9. 1. 180335.787: [Full GC [PSYoungGen: 57985K->0K(1850304K)]
[PSOldGen: 1945111K->1483937K(2097152K)] 2003097K-
>1483937K(3947456K) [PSPermGen: 263211K->263211K(329728K)],
10.0093850 secs]
180352.182: [GC [PSYoungGen: 1603456K->42582K(1873344K)]
3087393K->1526520K(3970496K), 0.3015250 secs]
!un-optimized
Measure fixed OldGen Utilization size
1.OUfixed = OU on full GC = 1483937K
Survivor copy rate = 42582/0.3 = 142 M/s
Full GC interval = (OC - OUF) / (ΔOU / Δt)
We want to make this 24 hrs.

10. 1. Too brief: excessive overflow to OldGen
Too long: large footprint + long pause
Determine young GC interval
1. To measure: gc-free response time (RT0) at low stress
To set: young gc interval (Tygc) > 4 * RT0

11. To measure: memory allocation rate under load
Determine Eden Capacity
13714.538: [GC 13714.538: [ParNew: 4399678K->33530K(4806016K), 0.0518020 secs]
5692328K->1326181K(6903168K), 0.0519700 secs] [Times: user=0.15 sys=0.00, real=0.05 secs]
13768.579: [GC 13768.579: [ParNew: 4402682K->27479K(4806016K), 0.0508480 secs] 5695333K-
>1320130K(6903168K), 0.0510040 secs] [Times: user=0.17 sys=0.00, real=0.05 secs]
Δm/Δt = (4 402 682 - 33 530) / (68 - 14) = 80 M/s
EC = Δm/Δt * Tygc = 80M *30 = 2400M

12. Determine Survivor Capacity
1. set: SC > 2 * max(SU)
1. In our example, SC = 200 M to handle 2x surges.
To measure: Survivor utilization on oversized SC

13. EU ∝ responses/s
SU ∝ responses/s
Fix Survivor Ratio
1. SurvivorRatio is independent of workload,
It is a property of the application.
1. In our example, 2400M/200M = 12

14. To measure: overflow rate under load
-- How fast OldGen utilization grows
Determine Promotion Rate
jstat -gc pid 10000 60
S0C S1C S0U S1U EC EU OC OU YGC FGC
436864.0 436864.0 41019.9 0.0 4369152.0 3511510.8 2097152.0 1319641.3 127 4
436864.0 436864.0 0.0 33325.8 4369152.0 197903.8 2097152.0 1319645.8 128 4
436864.0 436864.0 0.0 33325.8 4369152.0 971835.5 2097152.0 1319645.8 128 4
436864.0 436864.0 0.0 33325.8 4369152.0 2003800.7 2097152.0 1319645.8 128 4
436864.0 436864.0 0.0 33325.8 4369152.0 3422374.2 2097152.0 1319645.8 128 4
436864.0 436864.0 48272.6 0.0 4369152.0 1306482.6 2097152.0 1319646.9 129 4
PromoRate = ΔOU/Δt = (6.9-1.3)/50 = 0.11 k/s

15. With adequate full GC interval (1 day)
Determine Old Gen Capacity
1. SpareOC = PromoRate * 3600 * 24
= 10 M
OC = OUfixed + SpareOC

16. Summary of approach
Condition Measure Determine
Upon full GC fixed OldGen util OldGen capacity
Low load response time YGC interval
Load to 50% CPU memory alloc rate Eden size
Oversized
survivor
survivor utilization SurvivorRatio
Load, EC, SC promotion rate FGC interval

17. 1. -XX:SurvivorRatio=10 -XX:-
UsePSAdaptiveSurvivorSizePolicy -XX:
+UseConcMarkSweepGC
-XX:+UseParNewGC -XX:
+CMSScavengeBeforeRemark
-XX:CMSInitiatingOccupancyFraction=95 -XX:
+UseCMSInitiatingOccupancyOnly
-Xms7g -Xmx7g -Xmn5g
JVM Memory Configuration

18. 1.Keep constants:
Fixed OldGen utilization
Survivor ratio
CPU speed-dependent
Young GC interval
Proportional to CPU and Throughput:
Eden size
Spare OC
Memory scale-up to faster and more CPUs

19. 1. Full GC once at night
Memory problems resolved
GC pause time 50 ms
Application became CPU-bound
Response time outliers eliminated -- stability
Performance Results

20. • DIRECTV dev team, led by Bob Hays, Ankush Walhekar and Harsha
Anna, for the application and configuration, and for redesigning the
application for object lifetime management. Jamal Haskins for timely
assistance on build and deploy.
Owen Li, for helping with LoadRunner scripts
Louis Liao for tuning Linux OS
Rao Uppalapati, Prem Chaitanya and Abhiram Madadi, for setting up
environments
DIRECTV for support and providing the equipment and funding the trip
Acknowledgments

21. • Thank you.
Questions?