JavaOne 2016: Code Generation with JavaCompiler for Fun, Speed and Business Profit

Medallia © Copyright 2016. Confidential. 1
Code Generation with JavaCompiler
for Fun, Speed, and Business Profit
Thorvald Natvig and Juan Cruz Nores

Medallia © Copyright 2015. 2
At Medallia, we collect, analyze, and display terabytes of structured &
unstructured feedback for our multibillion dollar clients in real time.
And what’s more: we have a lot of fun doing it.
Thorvald - Since 2010, growing from 70 to 1200 employees
JC - First team in Argentina office, 2011
Who we are
Hi, we’re JC & Thorvald @ Medallia

How to run an analytics engine at 30x speed
1/30th the cost
Optimize for maximum performance with minimum engineering time
All code shown today is on GitHub
Includes bootstrap framework to do this yourself!
What’s this all about?
Fast Code in Short Time

AGENDA 1
2
3
4
5
6
Basics of CodeGen
Problem and Java Solution
CodeGen
Learnings and Challenges
Demo!
“What’s next?”

Two ways to generate code:
Method A: Use Java ASM and generate byte-code
You spend most of the time writing a compiler
Method B: Use Java Compiler to generate byte-code
Someone else wrote the compiler
Crash Course in Code Generation
Don’t write a compiler

Crash Course in Code Generation
Field f = Fields.get("age");
int min = 20;
int max = 99;
long sum = records.stream()
.filter(x -> x.getValue(field) > min)
.filter(x -> x.getValue(field) < max)
.map(x -> x.getValue(field))
.mapToInt(x -> x)
.sum();

Autoboxing and generics in general
Stream constructs
Function call to get value
Spend most of the time in “streaming”, not in “computing”
Problems!

// Assuming the field “age” is column 3 in the datastore...
long generated(int [][] records) {
long sum = 0;
for (int i = 0; i < 88737273; ++i) {
int fieldVar1 = records[3][i];
if (fieldVar1 > 20 && fieldVar1 < 99)
sum += fieldVar1;
}
return sum;
}
Transformed code

Generate Java code as a String
Invoke Java compiler in memory
Load generated class and instantiate an object
Call function
How to guide
“Somewhat faster”

We live in the real world
Engineering time costs $$$; salary + lost opportunities
Often cheaper to buy hardware than spend time to optimize
Only apply this level of optimization if the saved server cost greatly
outweighs the engineering cost.
When should you do this?
Optimize for Total Cost

AGENDA 1
2
3
4
5
6
Basics of CodeGen
CodeGen
Demo!

An in-memory query engine
Data is a segmented 2D table, each segment an int[][]
We want to be able to execute queries such as:
newQuery()
.filter(field("a").in(1, 2, 3).or(field("b").is(3)))
.aggregate(statsAggregate("ltr"))
This is roughly equivalent to:
SELECT AVG(ltr) FROM data WHERE a IN (1, 2, 3) OR b = 3
Problem

Query Execution
b ltr
1 10
3 7
Filter
passes? Aggregate
no
yesa
5
2

newQuery()
Pure Java
OR
“a” in
[1,2,3]
“b” in
[3]

expression.visit(new ExprVisitor<Boolean>() {
public Boolean visit(AndExpr andExpr) {
return andExpr.getLeft().visit(this) && andExpr.getRight().visit(this);
}
public Boolean visit(InExpr inExpr) { /* ommitted */ }
public Boolean visit(OrExpr orExpr) {
return orExpr.getLeft().visit(this) || orExpr.getRight().visit(this);
}
/* other nodes */
});
Pure Java

Simple query:
newQuery()
Complex query:
newQuery()
.filter(field("a").in(1, 2, 3, 5).or(field("b").is(3)))
.filter(field("c").in(3, 5, 8, 1, 2).or(field("d").in(1,3,4)))
.filter(not(field("e").in(2, 5, 8,10,12,13,14,15,17)).or(field("f").in(1,4,2)))
1M rows, split in 50k row segments (avoid large allocations)
About the benchmarks

We actually built this
And ran some benchmarks on it:
Benchmark Mode Cnt Score Error Units
InterpretedQueryBenchmark.complexQuery thrpt 6 14.453 ± 5.557 ops/s
InterpretedQueryBenchmark.simpleQuery thrpt 6 46.057 ± 2.227 ops/s
Pure Java
Benchmark early+often

Idea: Reduce unnecessary per-traversal operations by using lambdas
and their closures
It also reduces overall number of method calls
Take advantage of the massive optimizations implemented in HotSpot
to support Streams and lambdas (invokedynamic)
Optimized Pure Java

@FunctionalInterface
interface Filter {
boolean eval(Segment segment, int row);
}
public Filter visit(AndExpr andExpr) {
Filter left = andExpr.getLeft().visit(this);
Filter right = andExpr.getRight().visit(this);
return (segment, row) -> left.eval(segment, row) && right.eval(segment, row);
}
Optimized Pure Java

public Filter visit(OrExpr orExpr) {
Filter left = orExpr.getLeft().visit(this);
Filter right = orExpr.getRight().visit(this);
return (segment, row) -> left.eval(segment, row) && right.eval(segment, row);
}
public Filter visit(ConstantExpr constantExpr) {
boolean val = constantExpr.value;
return (segment, row) -> val;
}
Optimized Pure Java

public Filter visit(InExpr inExpr) {
final int[] values = inExpr.getValues();
final int column = dataSet.getFieldByName(inExpr.getFieldName()).getColumn();
return (segment, row) -> {
int fieldVal = segment.rawData[column][row];
for (int value : values) {
if (value == fieldVal) return true;
}
return false;
};
}
Optimized Pure Java

So we ran some benchmarks on this too:
StreamQueryBenchmark.complexQuery thrpt 6 33.046 ± 2.625 ops/s
StreamQueryBenchmark.simpleQuery thrpt 6 124.077 ± 7.449 ops/s
~3X improvement on simple queries
~2X improvement on complex ones
Optimized Pure Java

Results

AGENDA 1
2
3
4
5
6
Basics of CodeGen
CodeGen
Demo!

Query execution boils down to:
“iterate over every record, if it matches, aggregate”
This implies a simple query structure:
for (int i = 0; i < numRecords; i++) {
if (/* record matches */) {
/* aggregate */
}
}
Only the filter and the aggregate depend on the query
CodeGen

cg.print(query.buildExpressionTree().visit(new ExprVisitor<String>() {
public String visit(AndExpr andExpr) {
return "(" +andExpr.getLeft().visit(this) +" && "+ andExpr.getRight().visit(this) + ")";
}
public String visit(InExpr inExpr) {
FieldDefinition field = dataSet.getFieldByName(inExpr.getFieldName());
return "(" + IntStream.of(inExpr.getValues())
.mapToObj(v -> String.format("rawData[%d][row] == %dL", field.getColumn(), v))
.reduce((a,b) -> a + " || " + b)
.orElseThrow(() -> new RuntimeException("empty filter"))
+ ")";
}
public String visit(NotExpr notExpr) { "!" + notExpr.getTarget().visit(this); }
public String visit(OrExpr orExpr) { return "(" +orExpr.getLeft().visit(this) + " || " + orExpr.getRight().visit(this) + ")"; }
public String visit(ConstantExpr constantExpr) { return String.valueOf(constantExpr.value); }
}));
Naive CodeGen

public void process(Segment segment) {
int[][] rawData = segment.rawData;
int nRows = rawData[0].length;
FieldStats result$1 = (FieldStats)this.result;
for (int row = 0; row < nRows; row++) {
if ((rawData[0][row] == 1L || rawData[0][row] == 2L
|| rawData[0][row] == 3L) || (rawData[1][row] == 3L)) {
result$1.count++;
result$1.sum += rawData[6][row];
}
}
}
Naive CodeGen

CompiledQuery.complexQuery thrpt 6 72.509 ± 13.526 ops/s
CompiledQuery.simpleQuery thrpt 6 208.281 ± 11.335 ops/s
~4X improvement on simple queries (over the simple interpreter)
~5X improvement on complex ones
Results

Results

Higher level transforms
extract common subexpressions
order filters to maximize the chance of rejecting a filter
...
many of these apply also to the interpreter
Low level transforms
branch reduction
perfect hashing
...
Optimizing Your CodeGen

General strategy: instead of a set.contains() or set of ‘==’ comparisons
Less than 3 values: a or b or c
Value-range <= 64: reverse inverted bitmask.
Value-range <= 1024: array of reverse inverted bitmask.
Otherwise: Generated Perfect Hash
Branch Reduction

Branch reduction
1 0 0 0 0 0 0 0 >>> test-val
0 1 0 0 0 0 0 0
0 0 0 1 0 0 0 0
0 0 0 0 1 0 0 0
0 0 0 0 0 0 0 1
1:
3:
4:
7:
0 1 0 1 1 0 0 1mask:
bitwise or
field("a")
.in(1, 3, 4, 7)

Evaluation
1:
2:
0 1 0 1 1 0 0 1(
field("a")
.in(1, 3, 4, 7)
<< fieldVal ) < 0
01 0 1 1 0 0 1
00 1 1 0 0 1 0
Field
Value
✅
❌
= -78
= 100

newQuery()
.filter(field("a").in(1, 2, 3, 9).or(field("b").is(3)))
.aggregate(statsAggregate("ltr"));
public void process(Segment segment) {
int[][] rawData = segment.rawData;
int nRows = rawData[0].length;
FieldStats result$1 = (FieldStats)this.result;
for (int row = 0; row < nRows; row++) {
if (((0x7040000000000000L << rawData[0][row]) < 0 || (rawData[1][row] == 3L))) {
result$1.count++;
result$1.sum += rawData[2][row];
}
}
}
Variant Code Generation

protected String generateInExpr(JavaCodeGenerator cg, InExpr inExpr) {
final FieldDefinition fieldDef = dataSet.getFieldByName(inExpr.getFieldName());
final FieldSpec fieldSpec = fieldDef.getFieldSpec();
final int numValues = fieldSpec.getBound() - fieldSpec.getOrigin();
final int[] inValues = inExpr.getValues();
if (inValues.length > 3 && numValues < 64) {
long mask = 0;
for (int value : inValues) {
mask |= 0x8000_0000_0000_0000L >>> (value - fieldSpec.getOrigin());
}
if (fieldSpec.getOrigin() == 0) {
return String.format("(0x%xL << rawData[%d][row]) < 0", mask, fieldDef.getColumn());
} else {
return String.format("(0x%xL << (rawData[%d][row]-%dL)) < 0", mask, fieldDef.getColumn(), fieldSpec.getOrigin());
}
}
// Other cases, fall back to the naive one
return super.generateInExpr(cg, inExpr);
}
Variant Code Generation

VariantQueryBenchmark.complexQuery thrpt 6 88.972 ± 2.206 ops/s
VariantQueryBenchmark.simpleQuery thrpt 6 210.281 ± 8.007 ops/s
about the same on simple queries (over the naive compiler)
~20% improvement on complex ones
Results

Results

AGENDA 1
2
3
4
5
6
Basics of CodeGen
CodeGen
Demo!

public void process(com.medallia.data.Segment);
Code:
// Save Segment.rawData into a local variable (var 2)
0: aload_1
1: getfield #3
4: astore_2
// get rawData[0].length and store it
// in a local variable (var 3)
5: aload_2
6: iconst_0
7: aaload
8: arraylength
9: istore_3
// cast this.result into a FieldStats object and store it in a variable (var 4)
10: aload_0
11: getfield #2 // Field result:Ljava/lang/Object;
14: checkcast #4 // class FieldStats
17: astore 4
Why not do bytecode?

// Begin the loop
// set i = 0 (var 5)
19: iconst_0
20: istore 5
// if i >= (var 3) -- number of rows
// jump to end the loop (jump to instr 113)
22: iload 5
24: iload_3
25: if_icmpge 113
// rawData[0][i] == 1 goto 78
28: aload_2
29: iconst_0
30: aaload
31: iload 5
33: laload
34: lconst_1
35: lcmp
36: ifeq 78
// rawData[0][i] == 2 goto 78
… (more conditions)
Why not do bytecode?
… (more conditions)
// fieldStats (var 4).count ++
78: aload 4
80: dup
81: getfield #9 // Field FieldStats.count:J
84: lconst_1
85: ladd
86: putfield #9 // Field FieldStats.count:J
// fieldStats (var 4).sum += rawData[6][i]
89: aload 4
91: dup
92: getfield #10 // Field FieldStats.sum:D
95: aload_2
96: bipush 6
98: aaload
99: iload 5
101: laload
102: l2d
103: dadd
104: putfield #10 // Field FieldStats.sum:D
// i++ (var 5), jump to 22 (loop start)
107: iinc 5, 1
110: goto 22
113: return
}

Javac is designed as a command line tool
but… API in javax.tools!
For the basic case we have to deal with three entities:
Source files (input)
Class files (output)
Classpath
File operations are abstracted behind JavaFileManager
Compiling in-memory

Be careful with multithreading
Cache warmup is the most time consuming part of compilation
Simple strategy:
Keep a JavaCompiler and a StandardJavaFileManager in a thread
local variable
Use a fixed size thread pool to constrain memory usage
Compiling in-memory

Classloading
The simplest strategy is to use one class per ClassLoader:
public static <T> Class<? extends T> classFromBytes(
final Class<T> baseClass, final String name, final byte[] bytecode) {
return new ClassLoader(baseClass.getClassLoader()) {
Class<? extends T> c = defineClass(name, bytecode, 0, bytecode.length)
.asSubclass(baseClass);
}.c;
}

Pros:
Easy to implement
Memory reclamation is easy
(each class/classloader pair is independent)
Cons:
Metaspace chunks are tied to a classloader
→ wasted metaspace
Constrained to generating a single class
Classloading

Most Java programs are fairly static
JVM needs to warm up new code
Interpreter
C1 (-client compiler): fast compilation
C2 (-server compiler): 20%-30% faster code for some workloads
Dynamic class loading breaks some assumptions
code size increases over application run time
compiled code size does too: HotSpot CodeCache
Classloading

HotSpot stores native code in its code cache
You can run out of it
If you do, the compilers are disabled
"CodeCache is full. Compiler has been disabled"
Some JVM options to deal with it:
-XX:InitialCodeCacheSize
-XX:ReservedCodeCacheSize
-XX:+UseCodeCacheFlushing
CodeCache

AGENDA 1
2
3
4
5
6
Basics of CodeGen
CodeGen
Demo!

Demo

AGENDA 1
2
3
4
5
6
Basics of CodeGen
CodeGen
Demo!

As in x86-64 assembly.
And yes, YES WE ARE.
We’re just one step away from ASM

Clang as a library
Compile in-memory and execute C++ code
Using medallia/unsafe (open source!)
Java↔C++ argument marshalling
Implement native methods on-the-fly!
Generating C++ on-the-fly

#include <jni.h>
/* omitted - initialization of field constants */
void process(JNIEnv* env, jobject self, jobject segment) {
jint* rawData[cols];
/* omitted - get raw data arrays */
jdouble sum = 0;
jlong count = 0;
for (int row = 0; row < (int)nRows; row++) {
if (((rawData[0][row] == 1L || rawData[0][row] == 2L || rawData[0][row] == 3L) || (rawData[1][row] == 3L))) {
count++;
sum += rawData[6][row];
}
}
/* omitted - release raw data arrays */
jobject result = env->GetObjectField(self, resultFld);
env->SetLongField(result, countFld, env->GetLongField(result, countFld) + count);
env->SetDoubleField(result, sumFld, env->GetDoubleField(result, sumFld) + sum);
}
C++ query

Unsafe.complexQuery thrpt 6 97.135 ± 4.905 ops/s
Unsafe.simpleQuery thrpt 6 251.735 ± 14.856 ops/s
~20% improvement on simple queries (over the variant compiler)
~35% improvement on complex ones
C++ query

Results

Java can now segfault.
Using codegen limits probability of “incident”
Not recommended.
But fun.
C++ violates everything Java

CodeGen is easy (and also fun)
Focus on generating String code
Faster to write
Just as fast to execute (and often faster)
Easy to debug
Spend your time wisely
Bottom Line

Code for this talk:
https://github.com/medallia/javaone2016
Clang as a library:
https://github.com/medallia/unsafe
Thank you!

JavaOne 2016: Code Generation with JavaCompiler for Fun, Speed and Business Profit

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to JavaOne 2016: Code Generation with JavaCompiler for Fun, Speed and Business Profit

Similar to JavaOne 2016: Code Generation with JavaCompiler for Fun, Speed and Business Profit (20)

Recently uploaded

Recently uploaded (20)

JavaOne 2016: Code Generation with JavaCompiler for Fun, Speed and Business Profit