OSCON 2013: Apache Drill Workshop > Runtime Compilation
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

OSCON 2013: Apache Drill Workshop > Runtime Compilation

on

  • 1,215 views

Exercises and lessons learned in Java runtime compilation using Apache Drill, CodeModel, Janino and ASM

Exercises and lessons learned in Java runtime compilation using Apache Drill, CodeModel, Janino and ASM

Statistics

Views

Total Views
1,215
Views on SlideShare
1,215
Embed Views
0

Actions

Likes
2
Downloads
12
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

OSCON 2013: Apache Drill Workshop > Runtime Compilation Presentation Transcript

  • 1. 1 Apache Drill: Compilation Workshop Jacques Nadeau, OSCON July 23, 2013 jacques@apache.org |@intjesus
  • 2. 2 Runtime Compilation is Faster  JIT is smart but more gains can be had with runtime compilation From http://bit.ly/16Xk32x
  • 3. 3 How do you do runtime compilation? Runtime Compilation Tools  javax.tools.Compiler (since 1.6): Wrapper to compile string into bytecode using javac  Janino: New BSD licensed Java-based Java compiler Which One?  Simple benchmarks shows Janino performs substantially faster than javax.tools.Compiler  Janino doesn’t support Annotations nor Generics (ugh.)
  • 4. 4 Code Play Time Get Latest Drill  git clone  git://git.apache.org/incubator-drill.git  git checkout master  cd incubator-drill/sandbox/prototype  mvn install Download OSCON Drill examples  git clone https://github.com/jacques-n/oscon-drill.git  cd oscon-drill  mvn install  cd compile
  • 5. 5 Exercise 1: Runtime Compilation Example Goal  Generate and evaluate your first runtime compiled code leveraging Janino Overview  Janino provides a simple ExpressionEvaluator interface – Takes array of Objects as parameters – Returns single Object that can be casted to appropriate output class  Simplest way to use ExpressionEvaluator is by using a variable holder class – Variables in the exercise Code  src/test/java/org/apache/drill/oscon/compile/E1JaninoTest.java
  • 6. 6 Conclusions  Runtime compilation is easy  Managing and building strings is painful  Object interface is less than elegant
  • 7. 7 Simplification is necessary  CodeModel to the rescue – CodeModel provides a simplified interface for programmatically generating Java source code – Extremely expressive, supporting all major constructs – Apache Licensed (yay!)  Use interfaces to manage things – Define an interface – Generate an entire class instead of just an evaluation block – Ensure class implements interface then generate new instance of runtime generated class
  • 8. 8 Exercise 2: CodeModel + Interfaces Goal  Clean up the previous implementation to make it more repeatable Overview  JCodeModel provides interface to generate new class  JExpr, JFieldVar, JMethod and other classes used via invocations or statically  Use Drill’s QueryClassLoader to help inject bytecode into Classloader Code  src/test/java/org/apache/drill/oscon/compile/E2CodeModelTest.java
  • 9. 9 Conclusions  Things are better…but – Large blocks of code are going to be painful to do with CodeModel  Isn’t there some way to merge compile time generated code with runtime generated code?
  • 10. 10 Solution: Runtime Bytecode Merging  CodeModel to generate runtime specific blocks  Janino to generate runtime bytecode  Precompiled bytecode templates  Use ASM package to merge the two distinct classes into one runtime class Loaded Class ASM Bytecode Merging Janino compilation CodeModel Generated Code Precompiled Bytecode Templates
  • 11. 11 Exercise 3: Template Merging Approach Goal  Leverage best of generated and pre-compiled world Overview  Drill’s ClassTransformer class does the dirty work, taking a TemplateDefinition and InternalInterface source code  ClassTransformer also marks class final and all methods private final except external ones to maximize likelihood of JVM inlining.  Drill’s QueryClassLoader is again used Code  src/test/java/org/apache/drill/oscon/compile/E3TemplateMergeTest.java  src/main/java/org/apache/drill/oscon/compile/InsideInterface.java  src/main/java/org/apache/drill/oscon/compile/OutsideInterface.java  src/main/java/org/apache/drill/oscon/compile/OutsideTemplate.java
  • 12. 12 Exercise 4: Drill example  Goal – Apply generalized knowledge to Drill  Overview – Drill utilizes a combination of templating and code generation to build query level operators – We also use source code rewriting to simplify implementation of scalar functions – To minimize function overhead, all scalar function evaluations are merged into large evaluation blocks. – FunctionDefintion and DrillFunc combine to provide implementation – drill-module.conf is used to inform Drill of available extension  Code – src/main/java/org/apache/drill/oscon/compile/AbsoluteFunction.java – src/test/java/org/apache/drill/oscon/compile/E4AbsTest.java
  • 13. 13 Drill Implementation: Expression Compilation Best of all worlds: Runtime Bytecode Merging  Balance development effort and performance needs  Interpretation overhead during record batch setup  Compile time operation for large code blocks