Download to read offline
USAA Mono-to-Serverless.pdf
- 1. 1 Monolith to Serverless Canmodern cloud-native Java runtimes do it all?
- 2.
- 3.
- 4. 5 Cloud Performance • Modularruntime o Low memory, high throughput o Low operating cost • CI/CD optimized o Availability on all Cloud platforms o Cloud-native tool integration o Optimized for Kubernetes • Simple developer experience o Integration with all popular IDEs and build tools o API support – Jakarta EE and MicroProfile
- 5.
- 6. 8 6 reasons whyLiberty Just enough runtime Low operating cost Continuous delivery Zero migration Kubernetes optimized 80% disk and 56% memory saving 4x increased density over Tomcat & Spring Boot Zero-effort security fixing & zero technical debt 100% v2v & fixpack migration saving Self-tuned optimal perf, production-ready, kube-native Developer experience Container & kube-native experience, rapid inner loop Lightweight, highly- efficient runtime CI/CD optimized operational experience Simple true-to- production developer experience
- 7. 9 <feature>servlet-4.0</feature> <feature>jaxrs-2.1</feature> Kernel • You controlwhich features are loaded into each Liberty instance Just Enough Runtime jsonp-1.1 servlet- 4.0 http-2.0 appmgr Java EE jaxrs-2.1 jaxrsClien t-2.1
- 8. 10 10 Zero Migration üNo configurationbehavior changes üNo runtime feature behavior changes üNo removals Never apply an ifix again Skipping a release does not introduce additional migration work Stay current with a rebuild (no app or config changes necessary)
- 9. 12 Developer experience Repositories Build IDEs DevMode APIs Testing
- 10. Plan Code Build Test Release Deploy Run Monitor Cloud-native Liberty © 2022IBM Corporation Testcontainers Zero Migration Continuous Delivery Quality Security An end-to-end cloud-native DevOps delivery experience with Liberty
- 11. 15 15 Java on Cloud Alwaysa Good Fit?
- 12. Architectural style Functions Monolith Microservice Scale Smaller Larger Macroservice Lightweight Runtimesand Frameworks Full Function Runtimes Function as a Service Java runtimes face spectrum of application scale and styles
- 13. Time Responses per second Startup Steady state Time Memory Resident Set Time Howmuch of application has runtime seen Varies with load Varies with load Varies with load Traditional focus of Java runtime technologies Traditional focus on steady state performance Rampup CPU consumption
- 14. 19 Architecture Styles vsTypical Java workloads Architecture Characteristics Compatibility Monoliths Long-lived, efficient ✅ Macroservices Long-lived ✅ Microservices Fast start-up/ramp-up for scaling up and down (to zero) to meet demand. Small footprint. ❌ Serverless/Functions Fast start-up (in MS), short-lived ❌
- 15. 20 20 The Java VirtualMachine (JVM)
- 16. 21 What is theJVM? The Java Virtual Machine (JVM) is the runtime environment to execute Java code. The JVM is responsible for taking your application bytecode and converting it to a machines language for execution. Most other languages compile code for a specific system but in Java, the compiler converts the code for a Java Virtual Machine that can be used on any system hence the phrase “Write once, run anywhere”.
- 17. 22 How the JVMworks Class Loader Bytecode Verifier Interpreter JIT Java Bytecode JVM Java Code Java Compiler javac Run Time Compile Time OS Hardware
- 18. 23 JVM Interpreter Class Loader BytecodeVerifier Interpreter JIT Java Bytecode JVM • Machine independent bytecodes are interpreted by the JVM at runtime • This ensures portability of Java programs across different architectures • But it affects performance because interpretation is relatively slow. OS Hardware
- 19. 24 Just-in-Time Compiler Class Loader BytecodeVerifier Interpreter JIT Java Bytecode JVM • The answer to the performance issue is the JIT compiler, which transforms sequences of bytecodes into machine code that is cached by the JVM • Unit of compilation is a method. To save overhead, only “hot” methods are compiled • Compiled native machine code executes 10x faster than a bytecode-by-bytecode interpreter • Generated code is saved in a "code cache" for future use for lifetime of JVM OS Hardware
- 20. 25 25 JVM on Cloud– issues?
- 21. 26 JVM – TheGood 26 • Device independent – write once, run anywhere • > 25 years of improvements • JIT produces optimized machine code through use of Profilers • Efficient garbage collection • Longer it runs, the better it runs (JVM collects more profile data, JIT compiles more methods)
- 22. 27 JVM – TheBad 27 • Initial execution run is “interpreted”, which is relatively slow • “Hot Spot” methods compiled by JIT can create CPU and memory spikes • CPU spikes cause lower QoS • Memory spikes cause OOM issues, including crashes • Slow start-up time • Slow ramp-up time
- 23. 28 JVM Spikes 28 0 50 100 150 200 250 300 350 400 0 3060 90 CPU utilization (%) Time (sec) Daytrader7 CPU consumption CPU spikes caused by JIT compilation 0 100000 200000 300000 400000 500000 600000 0 30 60 90 Resident set size (KB) Time (sec) Daytrader7 memory footprint Footprint spikes caused by JIT compilation
- 24. 29 Container Size 29 Main issues: •Needto over-provision to avoid OOM •Very hard to do – JVMs have a non-deterministic behavior 0 100000 200000 300000 400000 500000 600000 0 30 60 90 Resident set size (KB) Time (sec) Daytrader7 memory footprint Footprint spikes caused by JIT compilation
- 25. 30 Container Instances (Auto-Scaling) 30 Mainissues: •Slow start-up and ramp-up times •CPU spikes can cause auto-scaler to incorrectly launch additional instances
- 26. 31 Solution Requirements 31 •Minimize/eliminate CPUand memory spikes •Improve start-up and ramp-up times
- 27. 32 32 Eclipse OpenJ9 JVM IBMSemeru Runtimes
- 28. 33 Overview of EclipseOpenJ9 Designed from the start to span all the operating systems needed by IBM products This JVM can go from small to large Can handle constrained environments or memory rich ones Renowned for its small footprint, fast start-up and ramp-up time Is used by the largest enterprises on the planet 33
- 29. 34 Eclipse OpenJ9 OpenJ9 combineswith OpenJDK to form a full JDK Link to GitHub repo: https://github.com/eclipse-openj9/openj9 34
- 30.
- 31. 36 IBM Semeru Runtimes “Thepart of Java that’s really in the clouds” IBM-built OpenJDK runtimes powered by the Eclipse OpenJ9 JVM No cost, stable, secure, high performance, cloud optimized, multi- platform, ready for development and production use Open Edition • Open source license (GPLv2+CE) • Available for Java 8, 11, 17, 18 (soon 19) Certified Edition • IBM license • Java SE TCK certified. • Available for Java 11, 17 36
- 32. 37 37 Semeru Cloud Compiler ForMicroservices
- 33. 38 JIT-as-a-Service Decouple the JITcompiler from the JVM and let it run as an independent process Offload JIT compilation to remote process Remote JIT Remote JIT JVM JIT JVM JIT Kubernetes Control Plane Treat JIT compilation as a cloud service • Auto-managed by orchestrator • A mono-to-micro solution • Local JIT still available 38
- 34. 39 Eclipse OpenJ9 JITServer •JITServer feature is available in the Eclipse OpenJ9 JVM • “Semeru Cloud Compiler” when used with Semeru Runtimes 39
- 35. 40 JITServer advantages forJVM Clients 40 Provisioning Easier to size; only consider the needs of the application Performance Improved ramp-up time due to JITServer supplying extra CPU power when the JVM needs it the most. Reduced CPU consumption with JITServer AOT cache Cost Reduced memory consumption means increased application density and reduced operational cost. Efficient auto-scaling – only pay for what you need/use. Resiliency If the JITServer crashes, the JVM can continue to run and compile with its local JIT
- 36. 41 JITServer value inKubernetes • https://blog.openj9.org/2021/10/20/save-money-with-jitserver-on-the-cloud-an-aws- experiment/ • Experimental test bed o ROSA (RedHat OpenShift Service on AWS) • Demonstrate that JITServer is not tied to IBM HW or SW o OCP cluster: 3 master nodes, 2 infra nodes, 3 worker nodes • Worker nodes have 8 vCPUs and 16 GB RAM (only ~12.3 GB available) o Four different applications • AcmeAir Microservices • AcmeAir Monolithic • Petclinic (Springboot framework) • Quarkus o Low amount of load to simulate conditions seen in practice o OpenShift Scheduler to manage pod and node deployments/placement 41
- 37. 42 JITServer improves containerdensity and cost Default config AM 500 B 550 C 550 F 450 P 450 P 450 B 550 F 450 AM 500 A 350 AM 500 M 200 Q 350 P 450 Q 350 D 600 D 1000 F 450 B 550 Q 350 AM 500 AM 500 AM 500 B 550 B 550 A 350 C 550 F 450 M 200 P 450 P 450 P 450 Q 350 Q 350 D 1000 AM 500 B 550 P 450 AM 500 B 550 B 550 C 550 C 550 F 450 F 450 P 450 Q 350 Q 350 D 1000 D 1000 Q 350 AM 250 AM 250 P 250 P 250 F 250 F 250 B 400 C 350 Q 150 Q 150 M 150 AM 250 AM 250 P 250 P 250 F 250 B 400 Q 150 Q 150 J 1200 A 250 B 400 B 400 C 350 D 1000 D 1000 D 600 AM 250 AM 250 P 250 P 250 F 250 F 250 B 400 C 350 Q 150 Q 150 M 150 AM 250 AM 250 P 250 P 250 F 250 B 400 Q 150 Q 150 J 1200 A 250 B 400 B 400 C 350 D 1000 D 1000 JITServer config Legend: AM: AcmeAir monolithic A: Auth service B: Booking service C: Customer service D: Database (mongo/postgres) F: Flight service J: JITServer M: Main service P: Petclinic Q: Quarkus Total=8250 MB Total=8550 MB Total=8600 MB Total=9250 MB Total=9850 MB 6.3 GB less 42
- 38. 43 Throughput comparison 0 200 400 600 800 1000 1200 0 240480 720 960 1200 1440 1680 1920 Throughput (pages/sec) Time (sec) AcmeAir Microservices 0 200 400 600 800 1000 1200 0 240 480 720 960 1200 1440 1680 1920 Throughput (pages/sec) Time (sec) AcmeAir Monolithic 0 20 40 60 80 100 120 140 160 180 0 240 480 720 960 1200 1440 1680 1920 Throughput (pages/sec) Time (sec) Petclinic 0 500 1000 1500 2000 2500 3000 3500 0 240 480 720 960 1200 1440 1680 1920 Throughput (pages/sec) Time (sec) Quarkus Machine load: 17.5% from apps 7% from OpenShift 43 è JITServer and default configuration achieve the same level of throughput at steady-state JITServer Baseline
- 39. 44 Conclusions from highdensity experiments • JITServer can improve container density and reduce operational costs of Java applications running in the cloud by 20-30% • Steady-state throughput is the same despite using fewer nodes 44
- 40. 45 Horizontal Pod Autoscalingin Kubernetes • Better autoscaling behavior with JITServer due to faster ramp-up • Less risk to trick the HPA due to transient JIT compilation overhead 45 Setup: Single node Microk8s cluster (16 vCPUs, 16 GB RAM) JVMs limited to 1 CPU, 500MB JITServer limited to 8 CPUs and has AOT cache enabled Load applied with JMeter, 100 threads, 10 ms think-time, 60s ramp-up time Autoscaler: scales up when average CPU utilization exceeds 0.5P. Up to 15 AcmeAir instances 0 1000 2000 3000 4000 5000 6000 7000 8000 9000 0 60 120 180 240 300 360 420 480 Throughput (pages/sec) Time (sec) AcmeAir throughput when using Kubernetes autoscaling Baseline JITServer+AOTcache
- 41. 46 Final thoughts • JITprovides advantage, but compilation adds overhead • Disaggregate JIT from JVM è JIT compilation as a service • Eclipse OpenJ9 JITServer (a.k.a Semeru Cloud Compiler) o Available now on Linux for Java 8, Java 11 and Java 17 (IBM Semeru Runtimes) o Especially good for constrained environments (micro-containers) o Kubernetes ready (Helm chart available, Prometheus integration) o Can improve ramp-up, autoscaling and performance of short lived applications o Can reduce peak memory footprint, increase app density and reduce costs o Java solution to Java problem, with no compromise 46
- 42.
- 43.
- 44. 49 • Use LinuxCRIU capability to snapshot/restore Liberty process • Linux CRIU snapshots process state including memory pages, file handles, network connections • Liberty and OpenJ9 run pre-snapshot and post-restore hooks to enable seamless user experience What is Liberty Instant On ? Liberty JVM starts up Liberty JVM reaches snapshot point Liberty JVM runs pre snapshot hooks Liberty JVM initiates snapshot Build time Liberty JVM exits after snapshot Restore Liberty JVM from snapshot Liberty JVM runs post restore hooks Run time Restore Liberty JVM from snapshot Liberty JVM runs post restore hooks Run time Goal : “instant on” without limiting programming model/capabilities substantially Liberty JVM continues running Liberty JVM continues running 49
- 45. 50 Linux CRIU capableof reducing startup time dramatically • ~10X reduction in first response time by using Liberty Instant On prototype using CRIU • Liberty Instant On is an effort to explore productization of CRIU use with Liberty • Main design goals • Usability : make it easy for an end user to consume the feature • Functionality : make it possible for end user to run their application without changes 50 0 1000 2000 3000 4000 5000 6000 Liberty baseline Liberty Instant On prototype PingPerf - First Response Time(ms) (lower is better) 0 1000 2000 3000 4000 5000 6000 Liberty baseline Liberty Instant On prototype Rest CRUD - First Response Time(ms) (lower is better) 0 1000 2000 3000 4000 5000 6000 Liberty baseline Liberty Instant On prototype Daytrader7 - First Response Time(ms) (lower is better) 1470 128 3082 213 5462 310 System Configuration: ------------------------------- SUT: LinTel – SLES 15 sp3 - Intel(R) Xeon(R) Gold 5118 CPU @ 2.30GHz, 2 physical cores(4 cpu), 32GB RAM.
- 46. 51 Where to checkpoint? 51 Liberty InstantOnleverages Semeru to provide a seamless checkpoint/restore solution for developers. With checkpoint restore, there is a tradeoff between startup time and the complexity of the restore. Checkpoint phases: • features • deployment • applications Kernel start and feature runtime processing Process application Start application Accept requests Applications Deployment Features Later checkpoint == Faster restore time Later checkpoint == More complexity 300 – 2200ms 100 – 3000ms 0 – ???ms
- 47.
- 48. 53 Architecture Styles vsLiberty Architecture Solution Compatibility Monoliths Liberty (Semeru Runtimes/OpenJ9) ✅ Macroservices Liberty ✅ Microservices Liberty + Semeru Cloud Compiler (w/ InstantOn) ✅ Serverless/Functions Liberty + InstantOn ✅
- 49.
- 50.