Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

OOPs, OOMs, oh my! Containerizing JVM apps

819 views

Published on

This talk was given during DockerCon EU 2018.
It ain't just a whim - to be able to continue innovating, we’ve moved our good old static production to containers. We needed to be elastic, fast, reliable and production ready at any time - that's why we chose Docker. But like in most enterprises, lots of our apps run on the JVM and most JVMs’ ergonomics assume they “own” the server they are running on. So how do you containerize JVM apps? Should you really increase JVM heap if you have spare memory? What about OS caches? What are the differences between JDK 8, 9 and 10 when it comes to container awareness? Outages because of out of memory errors? Slowness because of long garbage collection and poor environment visibility? Long story short, in this session, we’ll look at the gotchas of running JVM apps in containers and teach you how to avoid costly mistakes.
Top 3 things attendees will learn:
1. Key differences between various JVM versions relevant for containerized Java apps.
2. Best practices for running JVM in containers.
3. Avoiding common pitfalls when running containerized JVM applications.

Published in: Engineering
  • Be the first to comment

  • Be the first to like this

OOPs, OOMs, oh my! Containerizing JVM apps

  1. 1. Software Engineer Sematext Group, Inc. Rafał Kuć Radu Gheorghe Software Engineer Sematext Group, Inc.
  2. 2. Containerizing JVM Applications OOPs, OOMs, Oh My!
  3. 3. Our migration: the usual reasons
  4. 4. Avoid configuration drift Better resource usage
  5. 5. Avoid configuration drift
  6. 6. Avoid configuration drift Easy to scale, deploy and upgrade
  7. 7. BUT
  8. 8. BUT Java
  9. 9. Container limits and: wrong heap size small leaks ⇒ OOMs
  10. 10. Unresponsive services: big images startup takes CPU startup takes time
  11. 11. Agenda Share our mistakes so you don’t repeat them And those seen at our clients
  12. 12. Agenda Java 7 vs 8 vs 9 vs 10 vs 11 on Memory: heap vs off-heap vs OS caches Bonus: swappiness, OOM killer and other goodies Heap sizing, OOPs and GCs
  13. 13. # JIT threads # GC threads # fork-join pool threads CPU Java - the invisible Perm & metaspace JIT bytecode Java Native Interface New I/O access # of threads Memory
  14. 14. Java https://github.com/sematext/dockercon/
  15. 15. Heap Usage: healthy Sawtooth wave indicates: Headroom. Low risk of OOM Old GC kicks in regularly Old Gen
  16. 16. Heap Usage: unhealthy Irregular pattern: spiky load Constantly above InitiatingOccupancy: GC runs all the time Touches 100% - likely an OOM or a long pause ⇒ all bets are off
  17. 17. Generational GC eden survivor survivor tenured Young generation Old generation New allocation On young garbage collections Old GC cleans up here
  18. 18. Garbage Collectors Parallel Default until Java 9 Highest throughput High latency Low overhead Good for young gen CMS High throughput* Low latency* Low memory overhead Good for smaller heaps Heap fragmentation over time G1 Default from Java 9 High throughput** Low latency** Higher memory overhead Good for larger heaps * for small heaps ** for bigger heaps
  19. 19. OOPs, where’s my heap? https://github.com/sematext/dockercon/
  20. 20. Linux + Docker + JVM gotchas Memory leak for MMap → OOM killer on Java < 8u152* Growing RSS? → use NativeMemoryTracking to troubleshoot OS caches don’t count against container limits ⇒ leave room for them OS caches are shared for all containers of a host * https://bugs.openjdk.java.net/browse/JDK-8164293
  21. 21. Multi-processor architectures Single NUMA node? --cpu-shares Multiple NUMA nodes? --cpuset* vm.zone_reclaim_mode to allocate memory only on local node * Docker isn’t NUMA aware yet: https://github.com/moby/moby/issues/9777 But kernel automatically balances threads by default
  22. 22. kswapd Kernel’s (single-threaded) GC: https://linux-mm.org/PageOutKswapd Takes 100% of one core? Likely too many allocations and too little free memory No room for the new allocation? Go to swap. No swap? OOM Killer to the rescue! Allocate all heap on startup via AlwaysPreTouch Increase vm.min_free_kbytes on large boxes
  23. 23. Swapped heap. Not good.
  24. 24. Lessons learned Use newest JDK if possible - be reasonable though (it starts getting better from JDK 8u152) Garbage collector is not your enemy (at least not always) Leave as much memory for OS caches as possible Spread out - large heap will cause problems Allocate your heap during startup Say no to swapping JVM memory Keep things under observation - you can't fix what you can't measure

×