The Next Frontier in Open Source Java Compilers: Just-In-TIme Compilation as a Service

1. The Next Frontier in
Open Source Java Compilers:
Just-In-Time Compilation as a Service
1

2. Who am I?
• Software Engineer and Developer
Advocate at IBM
• Based in the Austin, TX USA
• Focus on OSS cloud native Java
technologies
Rich Hagarty
@rhagarty8

3. Agenda
REASON:
JVM and JIT compiler
– the good and the
bad
4
JIT-as-a-Service
SOLUTION:
JIT-as-a-Service to
the rescue
PROBLEM:
Java on Cloud - a bad
fit for microservices

4. Java on Cloud
A Good Fit?
5

5. Legacy Java Apps
6
•Java monolith on
dedicated server
•Plenty of CPU power
and memory
•Never went down
•6 month
upgrade/refresh
schedule

6. Moving to the Cloud
7
-Running in containers
-Managed by Cloud
Provider (and K8s)
-Auto-scaling to meet
demand
Cloud native App talking to Microservices

7. Main Motivators
8
-Flexible & scalable
-Easier to roll-out new releases more frequently
-Take advantage of latest-greatest Cloud technologies
-Less infrastructure to maintain and manage
-Saving money

8. But what about performance?
9
Need to balance cost vs performance
2 variables:
-Container size
-Number of instances (efficient scaling)

9. Performance vs Cost
10
Performance
Cost
Variables – Container Size and Container Instances
Containers too small – inefficient scaling

10. Performance vs Cost
11
Performance
Cost
Variables – Container Size and Container Instances
Containers too big

11. Performance vs Cost
12
Performance
Cost
Variables – Container Size and Container Instances
Containers too big, too many instances

12. Performance vs Cost
13
Performance
Cost
Variables – Container Size and Container Instances
Ahh – just right. Why is this so hard to do?

13. JVM and JIT compiler:
The Good and The Bad
14

14. Java Virtual Machine (JVM)
15
The Good
• Device independent – write once, run anywhere
• > 25 years of improvements
• JIT produces optimized machine code through use of Profilers
• Efficient garbage collection
• Longer it runs, the better it runs (JVM collects more profile data, JIT
compiles more methods)

15. Java Virtual Machine (JVM)
16
The Bad
• Initial execution run is “interpreted”, which is relatively slow
• “Hot Spot” methods compiled by JIT can create CPU and memory
spikes
• CPU spikes cause lower QoS
• Memory spikes cause OOM issues, including crashes
• Slow start-up time
• Slow ramp-up time

16. Java Virtual Machine (JVM)
17
0
50
100
150
200
250
300
350
400
0 30 60 90
CPU
utilization
(%)
Time (sec)
Daytrader7 CPU consumption
CPU spikes caused
by JIT compilation
0
100000
200000
300000
400000
500000
600000
0 30 60 90
Resident
set
size
(KB)
Time (sec)
Daytrader7 memory footprint
Footprint spikes caused
by JIT compilation

17. Cost vs Performance - Revisited
18
Finding the ”Sweet Spot”
• Right-sized provisioned containers
• Well-behaved efficient auto-scaling
Let’s revisit our 2 variables:
Container Size + Container Instances

18. Container Size
19
Main issues:
•Need to over-provision to
avoid OOM
•Very hard to do – JVMs
have a non-deterministic
behavior
0
100000
200000
300000
400000
500000
600000
0 30 60 90
Resident
set
size
(KB)
Time (sec)
Daytrader7 memory footprint
Footprint spikes caused
by JIT compilation

19. Container Instances (Auto-Scaling)
20
Main issues:
•Slow start-up and ramp-up times
•CPU spikes can cause auto-scaler to incorrectly
launch additional instances

20. Cost vs Performance - Revisited
21
Goal:
• Minimize/eliminate CPU and memory spikes – reduce
container size
• Improve start-up and ramp-up times – improve scaling
efficiency
Solution:
• JIT-as-a-Service

21. JIT-as-a-Service
to the rescue
22

22. JIT-as-a-Service
Decouple the JIT compiler from the JVM and let it run as an independent process
Offload JIT
compilation to
remote process
Remote
JIT
Remote
JIT
JVM
JIT
JVM
JIT
Kubernetes
Control Plane
Treat JIT
compilation as a
cloud service
• Auto-managed by orchestrator
• A mono-to-micro solution
• Local JIT still available
23

23. Eclipse OpenJ9 JITServer
• JITServer feature is available in the Eclipse OpenJ9 JVM
• “Semeru Cloud Compiler” when used with Semeru Runtimes
• OpenJ9 combines with OpenJDK to form a full JDK
Link to GitHub repo: https://github.com/eclipse-openj9/openj9
24

24. Overview of Eclipse OpenJ9
Designed from the start to span all the operating
systems needed by IBM products
This JVM can go from small to large
Can handle constrained environments or memory
rich ones
Renowned for its small footprint, fast start-up and
ramp-up time
Is used by the largest enterprises on the planet
25

25. Eclipse OpenJ9 Performance
26

26. IBM Semeru Runtimes
“The part of Java that’s really in the clouds”
IBM-built OpenJDK runtimes powered by the Eclipse OpenJ9 JVM
No cost, stable, secure, high performance, cloud optimized, multi-
platform, ready for development and production use
Open Edition
• Open source license (GPLv2+CE)
• Available for Java 8 (LTS), 11 (LTS), 16, 17 (LTS), 18, 19, 20
Certified Edition
• IBM license
• Java SE TCK certified.
• Available for Java 11, 17
27

27. IBM Semeru Runtimes
“The part of Java that’s really in the clouds”
Other OpenJDK Distributions:
• OpenJDK builds by Oracle
• Adoptium Eclipse Temurin
• Azul Zulu
• BellSoft Liberica
• Amazon Corretto
• Alibaba Dragonwell
• Microsoft Build of OpenJDK
• Red Hat Build of OpenJDK
• Oracle GraalVM
28

28. JITServer advantages for JVM Clients
29
Provisioning
Easier to size; only consider the needs
of the application
Performance
Improved ramp-up time due to JITServer
supplying extra CPU power when the JVM
needs it the most.
Reduced CPU consumption with JITServer AOT
cache
Cost
Reduced memory consumption means
increased application density and reduced
operational cost.
Efficient auto-scaling – only pay for what
you need/use.
Resiliency
If the JITServer crashes, the JVM can
continue to run and compile with its
local JIT

29. Performance graphs
30

30. JITServer value in Kubernetes
• https://blog.openj9.org/2021/10/20/save-money-with-jitserver-on-the-
cloud-an-aws-experiment/
• Experimental test bed
• ROSA (RedHat OpenShift Service on AWS)
• Demonstrate that JITServer is not tied to IBM HW or SW
• OCP cluster: 3 master nodes, 2 infra nodes, 3 worker nodes
• Worker nodes have 8 vCPUs and 16 GB RAM (only ~12.3 GB available)
• Four different applications
• AcmeAir Microservices
• AcmeAir Monolithic
• Petclinic (Springboot framework)
• Quarkus
• Low amount of load to simulate conditions seen in practice
• OpenShift Scheduler to manage pod and node deployments/placement
31

31. JITServer improves container density and cost
Default config
AM 500
B 550
C 550
F 450 P 450
P 450
B 550
F 450
AM 500
A 350
AM 500
M 200
Q 350
P 450
Q 350
D 600
D 1000
F 450
B 550
Q 350
AM 500
AM 500
AM 500
B 550
B 550
A 350
C 550
F 450
M 200
P 450
P 450
P 450
Q 350
Q 350
D 1000
AM 500 B 550
P 450
AM 500
B 550
B 550
C 550
C 550
F 450
F 450 P 450
Q 350
Q 350
D 1000
D 1000
Q 350
AM 250
AM 250
P 250
P 250
F 250
F 250
B 400 C 350
Q 150
Q 150
M 150
AM 250
AM 250
P 250
P 250
F 250
B 400
Q 150
Q 150
J 1200
A 250
B 400
B 400
C 350
D 1000 D 1000
D 600
AM 250
AM 250
P 250
P 250
F 250
F 250
B 400 C 350
Q 150
Q 150
M 150
AM 250
AM 250
P 250
P 250
F 250
B 400
Q 150
Q 150
J 1200
A 250
B 400
B 400
C 350
D 1000
D 1000
JITServer config
Legend:
AM: AcmeAir monolithic
A: Auth service
B: Booking service
C: Customer service
D: Database (mongo/postgres)
F: Flight service
J: JITServer
M: Main service
P: Petclinic
Q: Quarkus
Total=8250 MB Total=8550 MB Total=8600 MB
Total=9250 MB Total=9850 MB
6.3 GB less
32

32. Throughput comparison
0
200
400
600
800
1000
1200
0 240 480 720 960 1200 1440 1680 1920
Throughput
(pages/sec)
Time (sec)
AcmeAir Microservices
0
200
400
600
800
1000
1200
0 240 480 720 960 1200 1440 1680 1920
Throughput
(pages/sec)
Time (sec)
AcmeAir Monolithic
0
20
40
60
80
100
120
140
160
180
0 240 480 720 960 1200 1440 1680 1920
Throughput
(pages/sec)
Time (sec)
Petclinic
0
500
1000
1500
2000
2500
3000
3500
0 240 480 720 960 1200 1440 1680 1920
Throughput
(pages/sec)
Time (sec)
Quarkus
Machine load:
17.5% from apps
7% from OpenShift
33
è JITServer and default configuration achieve the same level of throughput at steady-state
JITServer Baseline

33. Conclusions from high density experiments
• JITServer can improve container density and reduce operational costs
of Java applications running in the cloud by 20-30%
• Steady-state throughput is the same despite using fewer nodes
34

34. Horizontal Pod Autoscaling in Kubernetes
• Better autoscaling behavior with JITServer due to faster ramp-up
• Less risk to trick the HPA due to transient JIT compilation overhead
35
Setup:
Single node Microk8s cluster (16 vCPUs, 16 GB RAM)
JVMs limited to 1 CPU, 500MB
JITServer limited to 8 CPUs and has AOT cache enabled
Load applied with JMeter, 100 threads, 10 ms think-time,
60s ramp-up time
Autoscaler: scales up when average CPU utilization
exceeds 0.5P. Up to 15 AcmeAir instances
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
0 60 120 180 240 300 360 420 480
Throughput
(pages/sec)
Time (sec)
AcmeAir throughput when using Kubernetes autoscaling
Baseline JITServer+AOTcache

35. DEMO
36

36. Improve ramp-up time with JITServer
• Experiment in docker containers
• Show that JITServer improves ramp-up
• Show that JITServer allows a lower memory limit for JVM containers
37
OpenLiberty+
AcmeAir
1P, 400 MB
limit
OpenLiberty+
AcmeAir
1P, 200 MB
limit
OpenLiberty+
AcmeAir
1P, 200 MB
limit
MongoDB
JITServer
4P, 1GB limit
JMeter
JMeter
JMeter
InfluxDB
Grafana
Collect throughput
data from JMeter
Display throughput
data
Apply load
to AcmeAir
instances
Provide data
persistence
services
Run the AcmeAir
application
Provide JIT
compilation
services
Grafana
Prometheus
Scrape
metrics
Display
JITServer
metrics

37. 38

38. 39

39. How to use it
40

40. JITServer usage basics
• One JDK, three different personas
• Normal JVM: $JAVA_HOME/bin/java MyApp
• JITServer: $JAVA_HOME/bin/jitserver
• Client JVM: $JAVA_HOME/bin/java -XX:+UseJITServer MyApp
• Optional further configuration through JVM command line options
• At the server:
-XX:JITServerPort=… default: 38400
• At the client:
-XX:JITServerAddress=… default: ‘localhost’
-XX:JITServerPort=… default: 38400
• Full list of options: https://www.eclipse.org/openj9/docs/jitserver/
• Note: Java version and OpenJ9 release at client and server must match
41

41. JITServer usage in Kubernetes
• Typically we create/configure
• JITServer deployment
• JITServer service (clients interact with service)
• Use
• Yaml files
• Helm charts: repo https://raw.githubusercontent.com/eclipse/openj9-utils/master/helm-chart/
• Certified OpenShift/K8s Operators from Open Liberty
• Tutorial: https://developer.ibm.com/tutorials/using-openj9-jitserver-in-
kubernetes/
42

42. JITServer encryption/authentication through TLS
• Needs additional JVM options
• Server: -XX:JITServerSSLKey=key.pem -XX:JITServerSSLCert=cert.pem
• Client: -XX:JITServerSSLRootCerts=cert.pem
• Certificates and keys can be provided using Kubernetes TLS Secrets
• Create TLS secret:
• kubectl create secret tls my-tls-secret --key <private-key-filename> --cert <certificate-filename>
• Use a volume to map “pem” files
43
apiVersion: v1
kind: Pod
metadata:
name: my-pod
spec:
containers:
- name: my-container-name
image: my-image
volumeMounts:
- name: secret-volume
mountPath: /etc/secret-volume
volumes:
- name: secret-volume
secret:
secretName: my-tls-secret

43. Monitoring
• Support for custom metrics for Prometheus
• Metrics scrapping: GET request to http://<jitserveraddress>:<port>/metrics
• Command line options:
-XX:+JITServerMetrics -XX:JITServerMetricsPort=<port>
• Metrics available
• jitserver_cpu_utilization
• jitserver_available_memory
• jitserver_connected_clients
• jitserver_active_threads
• Verbose logging
• Print client/server connections
-XX:+JITServerLogConnections
• Heart-beat: periodically print to verbose log some JITServer stats
• -Xjit:statisticsFrequency=<period-in-ms>
• Print detailed information about client/server behavior
-Xjit:verbose={JITServer},verbose={compilePerformance},vlog=…
44

44. JITServer usage recommendations
• When to use it
• JVM needs to compile many methods in a relatively short time
• JVM is running in a CPU/memory constrained environment, which can worsen
interference from the JIT compiler
• The network latency between JITServer and client VM is relatively low (<1ms)
• To keep network latency low, use “latency-performance” profile for tuned and configure your VM
with SR-IOV
• Recommendations
• 10-20 client JVMs connected to a single JITServer instance
• JITServer needs 1-2 GB of RAM
• Better performance if the compilation phases from different JVM clients do not overlap
(stagger)
• Encryption adds to the communication overhead; avoid if possible
• In K8s use “sessionAffinity” to ensure a client always connects to the same server
• Enable JITServer AOT cache: -XX:+JITServerUseAOTCache (client needs to have shared
class cache enabled)
45

45. Final thoughts
• JIT provides advantage, but compilation adds overhead
• Disaggregate JIT from JVM è JIT compilation as a service
• Eclipse OpenJ9 JITServer (a.k.a Semeru Cloud Compiler)
• Available now on Linux for Java 8, Java 11 and Java 17 (IBM Semeru Runtimes)
• Especially good for constrained environments (micro-containers)
• Kubernetes ready (Helm chart available, Prometheus integration)
• Can improve ramp-up, autoscaling and performance of short lived applications
• Can reduce peak memory footprint, increase app density and reduce costs
• Java solution to Java problem, with no compromise
47

46. Open Liberty
© 2021 IBM Corporation 48
Useful Links
Why choose Liberty
for Microservices
https://ibm.biz/6ReasonsW
hyLiberty
Choosing the right
Java runtime
https://ibm.biz/ChooseJava
Runtime
How to approach
application modernization
https://ibm.biz/ModernizeJa
vaApps
Open Liberty Site
https://www.openliberty.io
Open Liberty Guides
https://www.openliberty.io/
guides https://openliberty.io

47. Workloads:
Java EE
⎼ WebSphere
Application Server
⎼ Oracle WebLogic
⎼ Red Hat JBoss
⎼ Apache Tomcat
Integration
⎼ IBM MQ
⎼ App Connect
Enterprise (IIB)
Open Source SDK
for other extenders:
https://github.com/IBM/transfo
rmation-advisor-sdk
IBM Cloud Transformation Advisor
© 2021 IBM Corporation 49
Accelerates the modernization journey by quickly discovering and
analyzing on-premise Java EE and/or messaging workloads in the
enterprise to help in determining and executing the optimum
modernization steps for each.
Optimum modernization depends on workload needs! https://ibm.biz/6ReasonsWhyLiberty
Traditional
WebSphere
network
deployment
Traditional
WebSphere
‘base’
Liberty
OpenShift OpenShift OpenShift
Monolith Modernize
Operations
Modernize
Runtime
Modernize
Architecture
Containers, Java EE, monoliths & microservices…

48. 50
a free virtual consultation
with an expert
Liberty References
Read Watch
Explore the latest on WebSphere and
Liberty
ibm.biz/LibertyTV
View our recent Expert TV episode all
about Liberty https://ibm.biz/BdPn8Y
Learn more about Liberty in containers
and Operator-based deployment
ibm.biz/LibertyContainerOperator
Experience
Try Liberty as a beginner
openliberty.io/guides/getting-
started.html
Learn Liberty, MicroProfile,
Containers, Kubernetes,
Hands-on
openliberty.io/guides
Try dev mode in Containers
https://openliberty.io/guides/doc
ker.html
Why choose Liberty for
Microservices
ibm.biz/6ReasonsWhyLiberty
Choosing the right Java
runtime
ibm.biz/ChooseJavaRuntime
How to approach application
modernization
ibm.biz/ModernizeJavaApps

49. 51
51
Thank You

50. Thank You!
Questions?
rich.hagarty@ibm.com
@rhagarty8
https://www.linkedin.com/in/rhagarty/
https://www.slideshare.net/RichHagarty/jitservertalkpdf
52