Your SlideShare is downloading. ×
OSCON 2013: Apache Drill Workshop > Runtime Compilation
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.


Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

OSCON 2013: Apache Drill Workshop > Runtime Compilation


Published on

Exercises and lessons learned in Java runtime compilation using Apache Drill, CodeModel, Janino and ASM

Exercises and lessons learned in Java runtime compilation using Apache Drill, CodeModel, Janino and ASM

Published in: Technology, Education

  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. 1 Apache Drill: Compilation Workshop Jacques Nadeau, OSCON July 23, 2013 |@intjesus
  • 2. 2 Runtime Compilation is Faster  JIT is smart but more gains can be had with runtime compilation From
  • 3. 3 How do you do runtime compilation? Runtime Compilation Tools  (since 1.6): Wrapper to compile string into bytecode using javac  Janino: New BSD licensed Java-based Java compiler Which One?  Simple benchmarks shows Janino performs substantially faster than  Janino doesn’t support Annotations nor Generics (ugh.)
  • 4. 4 Code Play Time Get Latest Drill  git clone  git://  git checkout master  cd incubator-drill/sandbox/prototype  mvn install Download OSCON Drill examples  git clone  cd oscon-drill  mvn install  cd compile
  • 5. 5 Exercise 1: Runtime Compilation Example Goal  Generate and evaluate your first runtime compiled code leveraging Janino Overview  Janino provides a simple ExpressionEvaluator interface – Takes array of Objects as parameters – Returns single Object that can be casted to appropriate output class  Simplest way to use ExpressionEvaluator is by using a variable holder class – Variables in the exercise Code  src/test/java/org/apache/drill/oscon/compile/
  • 6. 6 Conclusions  Runtime compilation is easy  Managing and building strings is painful  Object interface is less than elegant
  • 7. 7 Simplification is necessary  CodeModel to the rescue – CodeModel provides a simplified interface for programmatically generating Java source code – Extremely expressive, supporting all major constructs – Apache Licensed (yay!)  Use interfaces to manage things – Define an interface – Generate an entire class instead of just an evaluation block – Ensure class implements interface then generate new instance of runtime generated class
  • 8. 8 Exercise 2: CodeModel + Interfaces Goal  Clean up the previous implementation to make it more repeatable Overview  JCodeModel provides interface to generate new class  JExpr, JFieldVar, JMethod and other classes used via invocations or statically  Use Drill’s QueryClassLoader to help inject bytecode into Classloader Code  src/test/java/org/apache/drill/oscon/compile/
  • 9. 9 Conclusions  Things are better…but – Large blocks of code are going to be painful to do with CodeModel  Isn’t there some way to merge compile time generated code with runtime generated code?
  • 10. 10 Solution: Runtime Bytecode Merging  CodeModel to generate runtime specific blocks  Janino to generate runtime bytecode  Precompiled bytecode templates  Use ASM package to merge the two distinct classes into one runtime class Loaded Class ASM Bytecode Merging Janino compilation CodeModel Generated Code Precompiled Bytecode Templates
  • 11. 11 Exercise 3: Template Merging Approach Goal  Leverage best of generated and pre-compiled world Overview  Drill’s ClassTransformer class does the dirty work, taking a TemplateDefinition and InternalInterface source code  ClassTransformer also marks class final and all methods private final except external ones to maximize likelihood of JVM inlining.  Drill’s QueryClassLoader is again used Code  src/test/java/org/apache/drill/oscon/compile/  src/main/java/org/apache/drill/oscon/compile/  src/main/java/org/apache/drill/oscon/compile/  src/main/java/org/apache/drill/oscon/compile/
  • 12. 12 Exercise 4: Drill example  Goal – Apply generalized knowledge to Drill  Overview – Drill utilizes a combination of templating and code generation to build query level operators – We also use source code rewriting to simplify implementation of scalar functions – To minimize function overhead, all scalar function evaluations are merged into large evaluation blocks. – FunctionDefintion and DrillFunc combine to provide implementation – drill-module.conf is used to inform Drill of available extension  Code – src/main/java/org/apache/drill/oscon/compile/ – src/test/java/org/apache/drill/oscon/compile/
  • 13. 13 Drill Implementation: Expression Compilation Best of all worlds: Runtime Bytecode Merging  Balance development effort and performance needs  Interpretation overhead during record batch setup  Compile time operation for large code blocks