Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Why you’re going to fail running java on docker!

1,546 views

Published on

Why you’re going to fail running java on docker! - Rafael Benavides

Published in: Software
  • Be the first to comment

Why you’re going to fail running java on docker!

  1. 1. 1 Why you’re going to FAIL running Java on docker! bit.ly/javadockerfail @burrsutter @rafabene
  2. 2. @burrsutter Change History 0.1 - JBCNConf.com 2017 - Barcelona 0.2 - JavaOne 2017 - San Francisco 0.3 - JavaDay UA 2017 - Kiev
  3. 3. Big Wins for Developers ● Highly Portable Packaging solution - for microservices, web apps ● Lightweight, Encapsulated OS abstraction - carry your OS with you ● Getting Started (docker run -it centos/wildfly) Instantly ● Dev Environments that more closely match Prod Environments ● Dev Environments that match OTHER Dev Environments (no more...but it works on my machine) ● No more waiting 3+ weeks for a VM to be provisioned by Ops just so you can run a series of tests Linux Container
  4. 4. @burrsutter
  5. 5. @burrsutter
  6. 6. Virtualization vs Containers Server Machine Host Operating System Hypervisor Guest OS Guest OS Guest OS Bins Libs Bins Libs Bins Libs App 1 App 2 App 3 Server Machine Host Operating System Bins Libs Bins Libs App 1 App 2 App 3 Docker ● Shared bins/libs and Host Operating System make containers lighter ● You can run multiple versions of app dependencies (Java 6 and 7) ● “Golden Image” deployment model ● Process isolation
  7. 7. Pros vs Cons Pros: ● Docker containers start very fast, take less memory ● Super easy to try out a new technology (e.g docker pull centos) ● Supports the “immutable image” deployment model ● It caches layers to make builds faster Cons: ● It is not a VM - it is a process running within the Host OS, do not assume you can run any flavor of container OS on any host OS ● It is not perfectly portable - you will eventually wish to get out of boot2docker-vm into a real Linux VM - one that more closely matches your production environment ● Lineage of docker images at Docker Hub is unknown
  8. 8. History of Containers 2000 2010 2005 2015 2000: JAILS ADDED TO FREEBSD 2006: GENERIC PROCESS CONTAINERS 2008: KERNEL AND USER NAMESPACES 2014: GOOGLE KUBERNETES 2008: LINUX CONTAINER PROJECT (LXC) 2015: STANDARDS VIA OCI AND CNCF 2013: RED HAT ENTERPRISE LINUX 2013: DOTCLOUD BECOMES DOCKER 2007: GPC RENAMED CONTROL GROUPS 2003: SELINUX ADDED TO LINUX MAINLINE 2015: RHT CONTAINER PLATFORM 2001: LINUX -VSERVER PROJECT 2013: DOTCLOUD PYCON LIGHTNING TALK 2005: FULL RELEASE OF SOLARIS ZONES
  9. 9. History of Java Java 1.0.2 - 30 May 1996 J2EE 1.2 - 12 Dec 1999 https://en.wikipedia.org/wiki/Java_EE_version_history#J2EE_1.2_.28December_12.2C_1999.29 https://en.wikipedia.org/wiki/Java_version_history
  10. 10. A Challenge Server Hardware Operating System Application Server .war or .ear Java Virtual Machine Custom Configuration Linux Kernel Version & Distribution Java 1.6.6_45 or Java 1.7.0_67 Weblogic 10.x.y, Tomcat 6.x.y, JBoss EAP 6.x.y JDBC driver, datasource, JMS queue, users Have you ever had “/” vs “” break your app? Or perhaps needed a unique version of a JDBC driver? Or had a datasource with a slightly misspelled JNDI name? Or received a patch for the JVM or app server that broke your code? Containerize Your App
  11. 11. Email MyApp.war has been tested with the following On my Windows 7 desktop JDK 1.8.43 Wildfly 9 Configuration: Datasource: MySQLDS Tested with: mysql-connector-java-5.1.31-bin.jar Production Environment Red Hat Enterprise Linux 6.2 JRE 1.7.3 WebSphere 8.5.5 Oracle 9
  12. 12. Dockerfile Container Guest OS Java App Server custom configuration FROM centos/wildfly COPY standalone.xml /opt/wildfly/standalone/configuration/ COPY mysql-connector-java-5.1.31-bin.jar /opt/wildfly/standalone/deployments/ COPY mysql-sample-ds.xml /opt/wildfly/standalone/deployments/ COPY myapp/target/your.war /opt/wildfly/standalone/deployments/ your.war dependencies Note: There are better ways to handle Java apps, this is for illustration purposes
  13. 13. @burrsutter Behind the Scenes
  14. 14. Cgroups and Namespaces Cgroups - cpu (cpu shares) - cpuacct - cpuset (limit processes to a CPU) - memory (swap, dirty pages) - blkio (throttle reads/writes) - devices - net_prio (packet class and priority) - freezer Namespaces - pid (processes) - net (network interfaces, routing) - ipc (system V ipc) - mnt (mount points, filesystems) - uts (hostname) - user (UIDs)
  15. 15. /sys/fs/cgroup/memory cgroup.clone_children memory.kmem.tcp.limit_in_bytes memory.move_charge_at_immigrate cgroup.event_control memory.kmem.tcp.max_usage_in_bytes memory.oom_control cgroup.procs memory.kmem.tcp.usage_in_bytes memory.pressure_level memory.failcnt memory.kmem.usage_in_bytes memory.soft_limit_in_bytes memory.force_empty memory.limit_in_bytes memory.stat memory.kmem.failcnt memory.max_usage_in_bytes memory.swappiness memory.kmem.limit_in_bytes memory.memsw.failcnt memory.usage_in_bytes memory.kmem.max_usage_in_bytes memory.memsw.limit_in_bytes memory.use_hierarchy memory.kmem.slabinfo memory.memsw.max_usage_in_bytes notify_on_release memory.kmem.tcp.failcnt memory.memsw.usage_in_bytes tasks
  16. 16. @burrsutter
  17. 17. DevOps Challenges for Multiple Containers ▪ How to scale? ▪ How to avoid port conflicts? ▪ How to manage them on multiple hosts? ▪ What happens if a host has trouble? ▪ How to keep them running? ▪ How to update them? ▪ Where are my containers? Node Node Node Node Node Logger Node
  18. 18. @burrsutter
  19. 19. ▪ Greek for “Helmsman,” also the root of the word “Governor” (from latin: gubernator) ▪ Container orchestrator ▪ Supports multiple cloud and bare-metal environments ▪ Inspired by Google’s experience with containers ▪ Open source, written in Go ▪ Manage applications, not machines Meet Kubernetes
  20. 20. SCM (Git/Svn) Master API Server Service Layer VirtualPhysical Private Public Persistent Storage Node Node Logger Node Node Node Node Dev Ops CI/CD Automation Routing Layer Registry SDN Overlay Network Controllers - Scheduler - Deployments - Services - Builds - Routes - DeploymentConfigs Kubernetes OpenShift - Builds - ImageStreams
  21. 21. SCM (Git/Svn) Master API Server Service Layer VirtualPhysical Private Public Persistent Storage Node Node Logger Node Node Node Node Dev Ops CI/CD Automation Routing Layer Registry SDN Overlay Network Controllers - Scheduler - Deployments - Services - Builds - Routes - DeploymentConfigs Kubernetes OpenShift - Builds - ImageStreams
  22. 22. SCM (Git/Svn) Master API Server Service Layer VirtualPhysical Private Public Persistent Storage Node Node Logger Node Node Node Node Dev Ops CI/CD Automation Routing Layer Registry SDN Overlay Network Controllers - Scheduler - Deployments - Services - Builds - Routes - DeploymentConfigs Kubernetes OpenShift - Builds - ImageStreams
  23. 23. @burrsutter Java
  24. 24. Why Fail? 8GB 4 cores 64GB 16 cores Developer Workstation Production Server
  25. 25. @burrsutter System.out.println("Memory: " + Runtime.getRuntime().maxMemory()/1024/1024); System.out.println("CPUs: " + Runtime.getRuntime().availableProcessors()); Why does Java Fail? It sees ALL the HOST resources, it is blind to cgroups by default Note: free also fails https://fabiokung.com/2014/03/13/memory-inside-linux-containers/
  26. 26. @burrsutter
  27. 27. @burrsutter Tomcat https://github.com/apache/tomcat/blob/trunk/java/org/apache/catalina/core/ContainerBase.java#L304 https://github.com/apache/tomcat/blob/trunk/java/org/apache/tomcat/util/net/NioEndpoint.java#L138 https://github.com/apache/tomcat/blob/trunk/java/org/apache/tomcat/websocket/AsyncChannelGroupUtil.java#L85 https://github.com/apache/tomcat/blob/trunk/java/org/apache/catalina/manager/StatusTransformer.java#L224 Wildfly https://github.com/wildfly/wildfly-common/blob/master/src/main/java/org/wildfly/common/cpu/ProcessorInfo.java#L61 Vert.x https://github.com/eclipse/vert.x/blob/master/src/main/java/io/vertx/core/VertxOptions.java#L41 Who uses those APIs? Tomcat, Wildfly, Vert.x
  28. 28. @burrsutter Memory (max heap defaults to ¼ of available memory) JIT Optimizations Mode: Client or Server Thread Management Garbage Collector Do NOT just “java -jar myVertxApp.jar” nor “java -jar mySpringBootApp.jar” Do try “java -XX:+PrintFlagsFinal -XX:+PrintGCDetails $JAVA_OPTIONS -jar myApp.jar” JVM Ergonomics https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/ergonomics.html
  29. 29. @burrsutter Memory
  30. 30. @burrsutter Heap (-Xms -Xmx) ------------------------------ Native JRE Perm (-XX:MaxPermSize), Java8 Meta (-XX:SurvivorRatio, -XX:MaxNewSize, -XX:NewRatio) JIT Bytecode JNI NIO Threads (defaults to 512K to 1024K depending on platform) Tip: Either -Xmx at 50% of container constrained memory or -XX:MaxRam=500m - your heap will be approximately 250mb * JVM Memory: Heap is about 50%
  31. 31. @burrsutter
  32. 32. @burrsutter https://hub.docker.com/r/fabric8/java-jboss-openjdk8-jdk/ https://github.com/fabric8io-images/java/blob/master/images/jboss/openjdk8/jdk/container-limits max_memory() { # High number which is the max limit until which memory is supposed to be # unbounded. local max_mem_unbounded="$(cat /sys/fs/cgroup/memory/memory.memsw.limit_in_bytes)" local mem_file="/sys/fs/cgroup/memory/memory.limit_in_bytes" if [ -r "${mem_file}" ]; then local max_mem="$(cat ${mem_file})" if [ ${max_mem} -lt ${max_mem_unbounded} ]; then echo "${max_mem}" fi fi } Workaround docker pull fabric8/java-jboss-openjdk8-jdk
  33. 33. @burrsutter Heap Patched: Java 8u131, included in Java 9 https://bugs.openjdk.java.net/browse/JDK-8170888
  34. 34. @burrsutter Try it yourself Just need Docker for Mac or Windows docker run -m 100MB openjdk:8u121 java -XshowSettings:vm -version vs docker run -m 100MB openjdk:8u131 java -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XshowSettings:vm -version UseCGroupMemoryLimitForHeap tells the JVM to look to the cgroup’s value for memory in /sys/fs/cgroup/memory/memory.limit_in_bytes
  35. 35. @burrsutter Cores
  36. 36. @burrsutter docker run --cpus="1.5" -m 400M openjdk:8u131 *docker run --cpu-quota=150000 --cpu-period=100000 -m 400M openjdk:8u131 docker run --cpu-shares=1536 -m 400M openjdk:8u131 **docker run --cpuset-cpus=0,2 -m 400M openjdk:8u131 Docker CPU Settings https://docs.docker.com/engine/admin/resource_constra ints/#configure-the-default-cfs-scheduler * Kubernetes uses this model ** JVM properly calculates availableProcessors()
  37. 37. @burrsutter docker run -it --cpus="2" -m 200M java java -XX:ParallelGCThreads=2 -XX:ConcGCThreads=2 -Djava.util.concurrent.ForkJoinPool.common.parallelism=2 -XX:MaxRam=200m // Half a core cpu-quota / cpu-period docker run -it --cpu-quota=50000 --cpu-period=100000 -m 200M openjdk:8u131 java -XX:+UseSerialGC -Xmx=100m Note: Set Max heap (-Xmx) to about 50% of total container memory Note: UseSerialGC when core count is below 2 Docker CPU Mapped to Java Some “rules of thumb”
  38. 38. @burrsutter CPU Set Patched https://bugs.openjdk.java.net/browse/JDK-6515172
  39. 39. @burrsutter Demo Boom https://github.com/burrsutter/javadockerfail Demo Script https://docs.google.com/document/d/1Td_nnjk12Miq6SpRs3RysDZbpQUjpxXk_p0J526VhcE/edit?usp=sharing
  40. 40. @burrsutter Base Docker Image FROM fabric8/java-jboss-openjdk8-jdk:1.2.7 Maven Plugin: - Docker builds leveraging base image - Deploys to Kubernetes and OpenShift easily https://fabric8.io/gitbook/mavenPlugin.html mvn io.fabric8:fabric8-maven-plugin:3.3.5:setup mvn fabric8:deploy Hot Tips https://hub.docker.com/r/fabric8/java-jboss-openjdk8-jdk/
  41. 41. @burrsutter More and More Resources https://developers.redhat.com/products/cdk/download/ https://github.com/burrsutter/javadockerfail https://developers.redhat.com/blog/2017/03/14/java-inside-docker/ https://developers.redhat.com/blog/2017/04/04/openjdk-and-containers/ https://dzone.com/articles/running-a-jvm-in-a-container-without-getting-kille https://github.com/redhat-developer-demos/java-container https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#how- pods-with-resource-limits-are-run https://github.com/fabric8io-images/java/tree/master/images/jboss/openjdk8/jdk https://github.com/fabric8io-images/java/blob/master/images/jboss/openjdk8/jdk/container-limits http://cs.oswego.edu/pipermail/concurrency-interest/2016-March/015014.html bit.ly/javadockerfail
  42. 42. 52 Why you’re going to FAIL running Java on docker! bit.ly/javadockerfail @burrsutter @rafabene
  43. 43. @burrsutter Previously Used Slides
  44. 44. Creative Commons/Flikr, SF History Center, SF Public Library, GG Bridge 1935-1936 at http://www.onlyinyourstate.com/northern-california/san-francisco/life-san-francisco-1935/
  45. 45. Creative Commons/Flikr, SF History Center, SF Public Library, GG & Bay Bridges 1935 at http://www.onlyinyourstate.com/northern-california/san-francisco/life-san-francisco-1935/
  46. 46. Creative Commons Found SF Historical Archive, GG & Bay Bridges, at http://www.onlyinyourstate.com/northern-california/san-francisco/life-san-francisco-1935/
  47. 47. Creative Commons Photo ‘For Bankrupt Hanjin, Costs and Containers Pile Up’ at https://www.workboat.com/news/bluewater/hanjin-costs-containers-pile-up-us-ports/
  48. 48. Wikimedia Commons ‘Container Port & Toxic Fire San Francisco’ at https://commons.wikimedia.org/wiki/File:Containter_port_and_toxic_fire,_San_Francisco.jpg

×