This session brings to your attention how several millions of dollars are wasted and what you can do to save money. Optimizing garbage collection performance not only saves money, but also improves the overall customer experience as well.

1. N O V E M B E R 1 0 , 2 0 2 2
Ram Lakshmanan | architect - yCrash
This is Garbage Talk.
Don’t Attend!

2. What is GARBAGE?
request
Objects
Application
Garbage

3. How are objects garbage collected?
Evolution: Manual -> Automatic
3 – 4 decades
before
Now
Developer
Writes code to Manually
evict Garbage
JVM and all modern platforms
Automatically evicts
Garbage

4. Automatic gc sounds good, right?
Yes, but for
Cpu consumption
gc pauses

5. What’s big deal about GC?
https://blog.gceasy.io/2021/07/08/i-dont-have-to-worry-about-garbage-collection-is-it-true/

6. 1. unpleasant customer experience

7. 2 . You can improve entire app’s response time
Avg response time (secs) Transactions > 25 secs (%)
Baseline 1.88 0.7
GC settings #2 1.36 0.12
GC settings #3 1.7 0.11
GC settings #4 1.48 0.08
GC settings #5 2.045 0.14
GC settings #6 1.087 0.24
GC settings #7 1.03 0.14
GC settings #8 0.95 0.31
49.46%
https://blog.gceasy.io/2022/08/25/automobile-company-optimizes-performance-using-gceasy/

8. GC
Throughput
1 day = 1440 Minutes (i.e., 24 hours x 60 minutes)
98% GC Throughput means
28.8 minutes/day in GC
7.3 days/year in GC
How does 98%
GC Throughput
sound?

9. 3. Millions of dollars wasted*
GC Throughput % 99% 98% 97% 96% 95%
Minutes wasted by 1
instance/day
14.4 min 28.8 min 43.2 min 57.6 min 72 min
Dollars wasted by mid-
size company (i.e., 1000
instances/year)
$50,072 $100,144 $150,216 $200,289 $250,361
Dollars wasted by large
size company (i.e., 10,000
instances/year)
$500,722 $1,001,443 $1,502,165 $2,002,886 $2,503,608
Dollars wasted by X-
Large size company (i.e.,
100,000 instances/year)
$5,007,216 $10,014,432 $15,021,648 $20,028,864 $25,036,080
* Calculations based on t2.2x.large 32G RHEL on-demand instance in US West (North California) - $ 0.5716/hour

10. 4. Predictive monitoring

11. HEALTHY GC PATTERN

12. ACUTE MEMORY LEAK PATTERN

13. MEMORY LEAK PATTERN

14. MORE PATTERNS
https://blog.gceasy.io/2021/10/15/interesting-garbage-collection-patterns/

15. Gc tuning tips and tricks
https://www.youtube.com/watch?v=6G0E4O5yxks

16. Key performance indicators
You can’t optimize, what you can’t measure
1. latency
GC Event’s pause time
2. Throughput
Percentage of time spent in processing
customer transactions vs time spent in
GC activity.
3. macrometrics
Memory: 2GB
CPU Time: 1 hr 8 min 8 secs
Response time: 1.2 secs
98.52%

17. -XX:+PrintGCDetails
-XX:+PrintGCDateStamps
-Xloggc:<file-path>
-Xlog:gc*:file=<file-path>
Till java 8
How to source these kpis?
Enable GC Log (always)!
From java 9
https://blog.gceasy.io/2021/08/17/overhead-added-by-garbage-collection-logging/

18. FREE GC LOG ANALYSIS TOOLS
GCeasy
02 GC Viewer
03
IBM GC and
Memory Visualizer
04 HP JMeter
01
05
Google Garbage
CAT (cms)
Freely available Garbage collection Log analysis tools

19. 1. Start from scratch
Remove all GC related JVM Arguments
Student of our
training program
Presented in DevOps/Jenkins world conference
https://tinyurl.com/jwperformance
Ryan Smith,
Senior Developer Support Engineer, CloudBees

20. Real world data from big bank

21. After removing all gc parameters

22. Real world data from big shipping company
Scenario: It was reported that, HA failover was occurring daily, leading to multiple production
outages and downtime for Jenkin users. Looking at the GC data, the following was observed:

23. 2 . STUDY gc causes

24. 3 . Remove system.gc( )
System.gc() or Runtime.getRuntime().gc()
Always causes stop-the-world Full GCs. Try to avoid it. Because you are intervening
with GC’s ergnomics
May not always originate from your source code
• 3rd party libraries, frameworks, sometimes even application servers
• External tools (like VisualVM) through use of JMX.
• RMI.
-XX:+DisableExplicitGC
This JVM argument disables System.gc() globally in your JVM.

25. Real world data from big shipping company

26. 4. Try changing generation/heap size

27. 5. High object creation rate
30 – 90% memory wasted due to inefficient programming!
https://tinyurl.com/y3q5j5oj

28. Inefficiency in collections
List<User> users = new ArrayList<>();
users.add(user);
ArrayList underlyingly maintains
Object[ ]
2 5
4 6 7 8 10
9
3
wasted
1
1
initial capacity = 10

29. Inefficiency in collections
List<User> users = new ArrayList<>();
for (int counter = 1; counter <= 11; ++counter) {
users.add(user);
}
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40
1 2 3 4 5 6 7 8 9 10 11
wasted
wasted
for (int counter = 1; counter <= 21;
++counter) {
users.add(user);
}

30. 5. More tips
https://www.youtube.com/watch?v=6G0E4O5yxks

31. Tuning Worksheet

32. Thank you friends
Ram Lakshmanan
ram@tier1app.com
@yCrash
https://www.linkedin.com/company/gceasy
This deck will be published in: https://blog.gceasy.io
CREDITS: This presentation template was created by Slidesgo, including
icons by Flaticon, and infographics & images by Freepik