This talk was given during DockerCon EU 2018.
It ain't just a whim - to be able to continue innovating, we’ve moved our good old static production to containers. We needed to be elastic, fast, reliable and production ready at any time - that's why we chose Docker. But like in most enterprises, lots of our apps run on the JVM and most JVMs’ ergonomics assume they “own” the server they are running on. So how do you containerize JVM apps? Should you really increase JVM heap if you have spare memory? What about OS caches? What are the differences between JDK 8, 9 and 10 when it comes to container awareness? Outages because of out of memory errors? Slowness because of long garbage collection and poor environment visibility? Long story short, in this session, we’ll look at the gotchas of running JVM apps in containers and teach you how to avoid costly mistakes.
Top 3 things attendees will learn:
1. Key differences between various JVM versions relevant for containerized Java apps.
2. Best practices for running JVM in containers.
3. Avoiding common pitfalls when running containerized JVM applications.
Heap Usage: healthy
Sawtooth wave indicates:
Headroom. Low risk of OOM
Old GC kicks in regularly
Heap Usage: unhealthy
Irregular pattern: spiky load
Constantly above InitiatingOccupancy: GC runs all the time
Touches 100% - likely an OOM or a long pause ⇒ all bets are off
eden survivor survivor tenured
Young generation Old generation
On young garbage collections
Old GC cleans up here
Default until Java 9
Good for young gen
Low memory overhead
Good for smaller heaps
Heap fragmentation over time
Default from Java 9
Higher memory overhead
Good for larger heaps
* for small heaps
** for bigger heaps
OOPs, where’s my heap?
Linux + Docker + JVM gotchas
Memory leak for MMap → OOM killer on Java < 8u152*
Growing RSS? → use NativeMemoryTracking to troubleshoot
OS caches don’t count against container limits ⇒ leave room for them
OS caches are shared for all containers of a host
Single NUMA node? --cpu-shares
Multiple NUMA nodes? --cpuset*
vm.zone_reclaim_mode to allocate memory only on local node
* Docker isn’t NUMA aware yet: https://github.com/moby/moby/issues/9777
But kernel automatically balances threads by default
Kernel’s (single-threaded) GC: https://linux-mm.org/PageOutKswapd
Takes 100% of one core? Likely too many allocations and too little free memory
No room for the new allocation? Go to swap. No swap? OOM Killer to the rescue!
Allocate all heap on startup via AlwaysPreTouch
Increase vm.min_free_kbytes on large boxes
Use newest JDK if possible - be reasonable though (it starts getting better from JDK 8u152)
Garbage collector is not your enemy (at least not always)
Leave as much memory for OS caches as possible
Spread out - large heap will cause problems
Allocate your heap during startup
Say no to swapping JVM memory
Keep things under observation - you can't fix what you can't measure