The document discusses the benefits of moving JIT compilation out of individual JVMs and into a shared, cloud-based compiler service. This "JIT-as-a-Service" approach improves efficiency by allowing optimization resources to be shared and elastic across JVMs. It also enables optimized code to be reused for applications that execute on multiple devices. Moving JIT compilation to the cloud reduces warmup time, memory footprint, and CPU usage for JVMs while improving the level of optimizations that can be performed.

1. The Cloud Native Compiler:
JIT-as-a-Service
Presented by Simon Ritter, Deputy CTO | Azul

2. 2
Why Is Java So Popular?
• It's a great language
o Easy to write, easy to read
• It has great libraries
o Pretty much anything you can think off, almost always a FOSS version as well as commercial
• It has a great community
o JUGs, Champions, conferences, etc, etc
o Easy to find Java programmers
• But really, it's the Java Virtual Machine
o Write once, run anywhere
o Excellent backwards compatibility
o A managed runtime environment

3. 3
JVM Performance Warmup
• Classfiles contain bytecodes (instructions for a Virtual Machine)
• Initially, these are interpreted
o Each bytecode is converted to equivalent native instructions
• Frequently used methods (hotspots) are identified and compiled
o Just-in-Time (JIT) compilation
o Uses two compilers, C1 (fast compile, low optimisation) and C2 (slow compile, high optimisation)
• Each time the application is started, the same process begins again
o Profile code looking for hot methods
o Compile using C1 JIT, recompile using C2 JIT when hot enough

4. 4
JVM Performance Graph
Application
Warmup

5. 5
The Way JVMs Work Now
• Isolated
• Not connected to other instances
o No memory of past runs
• Entirely self-reliant
o Limited to local resources
o Limited to local functionality
o Results in multiple trade-offs
• No Magic Cloud
• Limited compute power
• Limited storage
• Limited local functionality
• Limited analytical capabilities
• Limited knowledge

6. 6
Microservice-Based Application

7. 7
Microservice-Based Application

8. 8
Microservice-Based Application

9. 9
JVMs In The Cloud
• Connected to vast networks
• Can leverage and rely on…
o external resources
o external functionality
o external experience
• Can leverage (and contribute to) additional value
• Assume access to a “magic cloud”
o “unlimited” compute power
o “unlimited” data storage
o “unlimited” analytical capabilities
o “knowledge”, “experience”, …

10. 10
Cloud Native JVMs
• Has all the functionality of a JVM
• Outsources some of that core functionality to the cloud
• Delivers value to the cloud and to other JVMs that use that cloud

11. 11
JVM Architecture

12. 12
Basic Theory of JIT Compiler
• JVM provides “here is code I’d like to make better”
• JIT Compiler asks many “what do you know about X?” questions
• JIT Compiler eventually provides “here is code for your situation”
• JVM replaces existing code with new (hopefully better) code
• This is (currently) done with in-JVM JIT compilers

13. Speculative Optimisation
(The Secret of Java's Speed)

14. 14
Example 1: Inlining Monomorphic Sites
public class Animal {
private int color;
public int getColor() { return color };
}
myColor = animal.getColor();
can be transformed to:
myColor = animal.color;
As long as only one implementer of getColor() existslic
class Animal {

15. 15
Example 2: Branch Analysis
int computeMagnitude(int value) {
if (value > 9)
bias = computeBias(value);
else
bias = 1;
return Math.log10(bias + 99);
}
Profiling data shows that value (so far) has never been greater than 9

16. 16
Example 2: Branch Analysis
Assume that, based on profiling, value will continue to be less than 10
int computeMagnitude(int value) {
if (value > 9)
uncommonTrap(); // Deoptimise
return 2; // Math.log10(100)
}

17. 17
JVM Performance Graph
Deoptimisation

18. 18
Speculatively Optimised Code
Optimised Code
Assumptions
(That must hold true
for code to work)
Only one getColor()
method exists
value is less than 10

19. 19
JVM Performance Graph
Better optimisations

20. 20
How To Get Better Optimisations?
• Array processing example

21. 21
OpenJDK Hotspot C2 JIT
Per element jumps
2 elements per iteration

22. 22
Azul Prime JVM Falcon JIT
Using AVX2 vector instructions
32 elements per iteration
Broadwell E5-2690-v4

23. 23
Faster Code is... Faster
• Azul Prime JVM Falcon comparisson

24. 24
The Reality Of Better JIT

25. 25
Better JIT Compilation Has Cost
• We can apply more powerful optimizations and get (much) faster code
• But the JIT CPU and wall-clock time costs can grow significantly
• And so does the (JIT) memory footprint
• On powerful machines (e.g. 64 vcores) we can afford to do this
• But on more constrained/container environments…
o JIT CPU costs and wall clock time (& slowness) to optimization can become prohibitive
o JIT Memory footprint can lead to OOM Killer
• Can lead to tradeoffs between over-provisioning and foregoing higher optimization levels

26. 26
Speed and CPU Usage Over Time
• 2 Vcore container

27. 27
Speed and CPU Usage Over Time
• 2 Vcore container

28. 28
What If We Took The JIT Out Of The JVM?
JIT Compiler

29. 29
And Made It A Cloud-Based Resource
Cloud Native Compiler

30. 30
Why?

31. 31
Efficiency
Speed
Speed
Speed
Reduced
Warmup Time
Reduced
Memory Footprint
Right-sizing
$£€

32. 32
Speed and CPU Usage Over Time
• 2 Vcore container

33. 33
Speed and CPU Usage Over Time
• 2 Vcore container

34. 34
Speed and CPU Usage Over Time
• 2 Vcore container

35. 35
Isn't This Just Shifting The Cost?
• Well, Yes…
• But we are shifting it to a much more efficient place
• optimization resources are used for a tiny fraction of wall clock time
• When a JVM optimizes locally, it must carry dedicated resources to do so
• When outsourced to a Cloud Native Compiler
o The resources are shared and reused
o The resources can be elastic

36. 36
JIT-as-a-Service
Cloud Native Compiler

37. 37
JIT-as-a-Service
Cloud Native
Compiler
That's elastic

38. 38
There's More...

39. 39
Application Code Realities
• The vast majority of application code:
o Executes more than once
o Runs on more than one device
o Repeats the same tasks across runs and devices
o Has a limited set of common behavior profiles
• This is true even for rapidly changing applications…
• This is “lost” when JVMs do their own local, ephemeral optimizations

40. 40
Code Can Be Reused
Even with speculative optimisations

41. 41
Optimised Code
Assumptions
(That must hold true
for code to work)
Cloud Native
Compiler

42. 42
Isn't This Just Shifting The Cost?
• Well, Yes…
• But we are shifting it to a much more efficient place
• optimization resources are used for a tiny fraction of wall clock time
• When a JVM optimizes locally, it must carry dedicated resources to do so
• When outsourced to a Cloud Native Compiler
o The resources are shared and reused
o The resources can be elastic
• The work can be reused

43. 43
Cloud Native Compiler Is Better

44. 44
Cloud Native Compiler Is Better
Cloud Native
Compiler

45. 45
Summary

46. 46
Cloud Native Compiler
• Decoupling the JIT from the JVM has many advantages
o Better resource utilisation
o Better optimised code reuse
o Faster warmup
o and more...
• The cloud Native Compiler exists today
• Azul Platform Prime
• Try for free
https://www.azul.com/products/prime/stream-download/

47. Thank You.
Simon Ritter, Deputy CTO
@speakjava