This document discusses best practices for containerizing Java applications to avoid out of memory errors and performance issues. It covers choosing appropriate Java versions, garbage collector tuning, sizing heap memory correctly while leaving room for operating system caches, avoiding swapping, and monitoring applications to detect issues. Key recommendations include using the newest Java version possible, configuring the garbage collector appropriately for the workload, allocating all heap memory at startup, and monitoring memory usage to detect problems early.

1. Software Engineer
Sematext Group, Inc.
Rafał Kuć
Radu Gheorghe
Software Engineer
Sematext Group, Inc.

2. Containerizing JVM Applications
OOPs, OOMs, Oh My!

3. Our migration: the usual reasons

4. Avoid configuration drift
Better resource usage

5. Avoid configuration drift

6. Avoid configuration drift
Easy to scale, deploy and upgrade

7. BUT

8. BUT
Java

9. Container limits and:
wrong heap size
small leaks
⇒ OOMs

10. Unresponsive services:
big images
startup takes CPU
startup takes time

11. Agenda
Share our mistakes so you
don’t repeat them
And those
seen at our
clients

12. Agenda
Java 7 vs 8 vs 9 vs 10 vs 11 on
Memory: heap vs off-heap vs OS caches
Bonus: swappiness, OOM killer and other goodies
Heap sizing, OOPs and GCs

13. # JIT threads
# GC threads
# fork-join pool threads
CPU
Java - the invisible
Perm & metaspace
JIT bytecode
Java Native Interface
New I/O access
# of threads
Memory

14. Java
https://github.com/sematext/dockercon/

15. Heap Usage: healthy
Sawtooth wave indicates:
Headroom. Low risk of OOM
Old GC kicks in regularly
Old Gen

16. Heap Usage: unhealthy
Irregular pattern: spiky load
Constantly above InitiatingOccupancy: GC runs all the time
Touches 100% - likely an OOM or a long pause ⇒ all bets are off

17. Generational GC
eden survivor survivor tenured
Young generation Old generation
New allocation
On young garbage collections
Old GC cleans up here

18. Garbage Collectors
Parallel
Default until Java 9
Highest throughput
High latency
Low overhead
Good for young gen
CMS
High throughput*
Low latency*
Low memory overhead
Good for smaller heaps
Heap fragmentation over time
G1
Default from Java 9
High throughput**
Low latency**
Higher memory overhead
Good for larger heaps
* for small heaps
** for bigger heaps

19. OOPs, where’s my heap?
https://github.com/sematext/dockercon/

20. Linux + Docker + JVM gotchas
Memory leak for MMap → OOM killer on Java < 8u152*
Growing RSS? → use NativeMemoryTracking to troubleshoot
OS caches don’t count against container limits ⇒ leave room for them
OS caches are shared for all containers of a host
* https://bugs.openjdk.java.net/browse/JDK-8164293

21. Multi-processor architectures
Single NUMA node? --cpu-shares
Multiple NUMA nodes? --cpuset*
vm.zone_reclaim_mode to allocate memory only on local node
* Docker isn’t NUMA aware yet: https://github.com/moby/moby/issues/9777
But kernel automatically balances threads by default

22. kswapd
Kernel’s (single-threaded) GC: https://linux-mm.org/PageOutKswapd
Takes 100% of one core? Likely too many allocations and too little free memory
No room for the new allocation? Go to swap. No swap? OOM Killer to the rescue!
Allocate all heap on startup via AlwaysPreTouch
Increase vm.min_free_kbytes on large boxes

23. Swapped heap. Not good.

24. Lessons learned
Use newest JDK if possible - be reasonable though (it starts getting better from JDK 8u152)
Garbage collector is not your enemy (at least not always)
Leave as much memory for OS caches as possible
Spread out - large heap will cause problems
Allocate your heap during startup
Say no to swapping JVM memory
Keep things under observation - you can't fix what you can't measure