With the increasing adoption of cloud native technologies and containerization; the gap between Java development and system administration is decreasing. Whether you are using Docker Swarm, Kubernetes or Mesos as a container orchestrator; fundamental challenges for running docker in production are common. In this talk, I would like to share some of the basic linux concepts (like memory management, CPU, IO, sockets, file descriptors, signals, OOM killer) every Java Developer should know to be able to perform effective configuration and troubleshooting for docker containers.

1. Docker Tips for Cloud Native
Java Developers
Aparna Chaudhary (@aparnachaudhary)

2. Docker in your organization
Development Environment
Build Server
Microservices + Containerization
Container Orchestration
Dockeradoptionlevels

3. Build workflow
Security
Monitoring
Resource Limiting
Logging
Secret Management
Networking
Storage Management
Image Management
Rolling Upgrades
Config Management
Tracing

4. Ops is fun. Ops is our duty.
Let’s embrace it.

5. Build workflow
Security
Monitoring
Resource Limiting
Logging
Secret Management
Networking
Storage Management
Image Management
Rolling Upgrades
Config Management
Tracing
Where do I start?

6. Docker and Memory

7. OOM Score?
OOM Killer?

8. How much memory is my process eating?
https://www.xkcd.com/1495/

9. What is top trying to tell you?

10. How much memory is my process eating?
Swap
Shared
Resident
Virtual
Virtual: Virtual address space used by the
program
Shared: Amount of Physical RAM shared with
other processes
Resident: Actual physical RAM used by the
process
Swap: OS went out of physical memory so it puts
the data from RAM to disk

11. Docker and memory management
--memory-swappiness
03
● How much swap to use between 0 and
100; 0 means do not use swap.
--memory-swap
02
● Amount of memory a container is allowed
to swap to disk
● Default is twice the memory
--memory
01
● Maximum amount of memory a container
can use

12. Do not rely on host OS settings

13. Swapping means death of
JVM performance!
Set --memory and
--memory-swap equal to
avoid use of swap space

14. Exit JVM on OutOfMemory
ExitOnOutOfMemoryError - When you enable this option, the JVM exits on the first
occurrence of an out-of-memory error. It can be used if you prefer restarting an
instance of the JVM rather than handling out of memory errors.
Set the jvm flag to ensure
auto-restart of container
(JDK 8u92).

15. Docker and CPU

16. Docker and CPU Management
Service foo is
slow everyday at
04:00
All services on
this specific node
are slow
Noisy Neighbours

17. CPU Shares
● Default CPU isolation
● Provides a priority weighting across all all cpu cycles across all cores.
● Default weight for any container is 1024
Containers
per node
increases
CPU per
container
reduces

18. CPU Set
● Limits container’s processes to specific CPU(s)
● A comma-separated range of CPU’s a container can use

19. CPU Limits
Allows container to use CPU time for duration of cpu-quota every cpu-period
--cpu-period = 100 microseconds (default)
--cpu-quota = 20 microseconds
Then application is allowed to use 20 microseconds of CPU time every 100
microseconds.

20. Scenario: Application uses all CPU quota at the start
--cpu-period=100 µs --cpu-period=100 µs --cpu-period=100 µs
80 µs20 µs 80 µs20 µs 80 µs20 µs
--cpu-period=100 µs
--cpu-quota=20 µs
Latency experienced = 80 µs
Run
Throttle

21. Scenario: STW GC Run
--cpu-period=100 µs --cpu-period=100 µs --cpu-period=100 µs
80
--cpu-period=100 µs
--cpu-quota=20 µs
Latency experienced = 95 µs
Run
Throttle
GC
5 15 805 15 805 15

22. Scenario: Concurrent GC Run (STW + non-STW phases)
--cpu-period=100 µs --cpu-period=100 µs --cpu-period=100 µs
80
--cpu-period=100 µs
--cpu-quota=20 µs
Latency experienced = 90 µs
Run
Throttle
GC
10 8010 8010
CPU time stolen by GC -
end user experiences
STW.
10 10 10
Larger the number of GC
threads, higher the latency
experienced by end user.

23. Running JVM in docker
needs thorough
understanding of how
JVM GC interacts with
cgroup CPU scheduling

24. CPU, Cgroups, JVM - What do I do?
CPU Shares:
● Unpredictable performance because of
noisy neighbours
● Simple to configure
● Allows use of idle CPU resources
● Difficult capacity planning
CPU Limits:
● Predictable performance if tuned properly
for GC behavior
● Difficult to configure properly
● Idle CPU resources are not utilized
● Better capacity planning

25. Docker CPU and JVM
● Slow GC performance
● Set appropriate JVM properties e.g. parallel GC threads
○ Use -XX:+PrintFlagsFinal to find out number of parallel GC threads
● Set thread pool sizes
Be aware of thread pools in JVM:
● Jit compiler threads
● Garbage collection threads
● Third party library thread pools

26. Remember...

27. Decide on QoS (Best
Effort, Burstable,
Guaranteed)Disable swap usage
CPU share will cause
unpredictable
performance because of
noisy neighbours
CPU limits may
cause throttling of
application during
GC.
Set JVM flags
-XX:ExitOnOutOfMemoryError
-XX:ParallelGCThreads
@aparnachaudhary

28. References
● Julia Evans @b0rk https://jvns.ca/
● Daniel Stori @turnoff_us http://turnoff.us/
● https://www.kernel.org/doc/Documentation/scheduler/sched-bwc.txt