Your SlideShare is downloading. ×
OSCON 2013: Apache Drill Workshop > Runtime Compilation
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

OSCON 2013: Apache Drill Workshop > Runtime Compilation

977
views

Published on

Exercises and lessons learned in Java runtime compilation using Apache Drill, CodeModel, Janino and ASM

Exercises and lessons learned in Java runtime compilation using Apache Drill, CodeModel, Janino and ASM

Published in: Technology, Education

0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
977
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
13
Comments
0
Likes
2
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. 1 Apache Drill: Compilation Workshop Jacques Nadeau, OSCON July 23, 2013 jacques@apache.org |@intjesus
  • 2. 2 Runtime Compilation is Faster  JIT is smart but more gains can be had with runtime compilation From http://bit.ly/16Xk32x
  • 3. 3 How do you do runtime compilation? Runtime Compilation Tools  javax.tools.Compiler (since 1.6): Wrapper to compile string into bytecode using javac  Janino: New BSD licensed Java-based Java compiler Which One?  Simple benchmarks shows Janino performs substantially faster than javax.tools.Compiler  Janino doesn’t support Annotations nor Generics (ugh.)
  • 4. 4 Code Play Time Get Latest Drill  git clone  git://git.apache.org/incubator-drill.git  git checkout master  cd incubator-drill/sandbox/prototype  mvn install Download OSCON Drill examples  git clone https://github.com/jacques-n/oscon-drill.git  cd oscon-drill  mvn install  cd compile
  • 5. 5 Exercise 1: Runtime Compilation Example Goal  Generate and evaluate your first runtime compiled code leveraging Janino Overview  Janino provides a simple ExpressionEvaluator interface – Takes array of Objects as parameters – Returns single Object that can be casted to appropriate output class  Simplest way to use ExpressionEvaluator is by using a variable holder class – Variables in the exercise Code  src/test/java/org/apache/drill/oscon/compile/E1JaninoTest.java
  • 6. 6 Conclusions  Runtime compilation is easy  Managing and building strings is painful  Object interface is less than elegant
  • 7. 7 Simplification is necessary  CodeModel to the rescue – CodeModel provides a simplified interface for programmatically generating Java source code – Extremely expressive, supporting all major constructs – Apache Licensed (yay!)  Use interfaces to manage things – Define an interface – Generate an entire class instead of just an evaluation block – Ensure class implements interface then generate new instance of runtime generated class
  • 8. 8 Exercise 2: CodeModel + Interfaces Goal  Clean up the previous implementation to make it more repeatable Overview  JCodeModel provides interface to generate new class  JExpr, JFieldVar, JMethod and other classes used via invocations or statically  Use Drill’s QueryClassLoader to help inject bytecode into Classloader Code  src/test/java/org/apache/drill/oscon/compile/E2CodeModelTest.java
  • 9. 9 Conclusions  Things are better…but – Large blocks of code are going to be painful to do with CodeModel  Isn’t there some way to merge compile time generated code with runtime generated code?
  • 10. 10 Solution: Runtime Bytecode Merging  CodeModel to generate runtime specific blocks  Janino to generate runtime bytecode  Precompiled bytecode templates  Use ASM package to merge the two distinct classes into one runtime class Loaded Class ASM Bytecode Merging Janino compilation CodeModel Generated Code Precompiled Bytecode Templates
  • 11. 11 Exercise 3: Template Merging Approach Goal  Leverage best of generated and pre-compiled world Overview  Drill’s ClassTransformer class does the dirty work, taking a TemplateDefinition and InternalInterface source code  ClassTransformer also marks class final and all methods private final except external ones to maximize likelihood of JVM inlining.  Drill’s QueryClassLoader is again used Code  src/test/java/org/apache/drill/oscon/compile/E3TemplateMergeTest.java  src/main/java/org/apache/drill/oscon/compile/InsideInterface.java  src/main/java/org/apache/drill/oscon/compile/OutsideInterface.java  src/main/java/org/apache/drill/oscon/compile/OutsideTemplate.java
  • 12. 12 Exercise 4: Drill example  Goal – Apply generalized knowledge to Drill  Overview – Drill utilizes a combination of templating and code generation to build query level operators – We also use source code rewriting to simplify implementation of scalar functions – To minimize function overhead, all scalar function evaluations are merged into large evaluation blocks. – FunctionDefintion and DrillFunc combine to provide implementation – drill-module.conf is used to inform Drill of available extension  Code – src/main/java/org/apache/drill/oscon/compile/AbsoluteFunction.java – src/test/java/org/apache/drill/oscon/compile/E4AbsTest.java
  • 13. 13 Drill Implementation: Expression Compilation Best of all worlds: Runtime Bytecode Merging  Balance development effort and performance needs  Interpretation overhead during record batch setup  Compile time operation for large code blocks