Understanding and Improving Code Generation
•
0 likes•1,323 views
Code generation is integral to Spark’s physical execution engine. When implemented, the Spark engine creates optimized bytecode at runtime improving performance when compared to interpreted execution. Spark has taken the next step with whole-stage codegen which collapses an entire query into a single function.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Understanding and Improving Code Generation
Similar to Understanding and Improving Code Generation (20)
More from Databricks
More from Databricks (20)
Recently uploaded
Recently uploaded (20)
Understanding and Improving Code Generation
- 2. Understanding and Improving Code Generation Michael Chen
- 3. Agenda Spark SQL Basics Volcano Iterator Model Whole-Stage Code Generation Problems Splitting Generated Code Performance
- 4. Spark SQL https://databricks.com/blog/2015/03/24/spark-sql-graduates-from-alpha-in-spark-1-3.html
- 6. Volcano Iterator Model ▪ Each operator can be thought of as an iterator ▪ Simple to compose arbitrary operators
- 10. Whole Stage Code Generation
- 11. Whole-Stage Code Generation ▪ Inspired by Thomas Neumann’s paper – Efficiently Compiling Efficient Query Plans for Modern Hardware ▪ Collapse entire query into a single operator ▪ Generate one function for the entire query
- 12. Whole-Stage Code Generation ▪ Less virtual function calls ▪ Data placed in CPU registers ▪ Compiler optimizations
- 13. Whole-Stage Code Generation ▪ Produce method called on children until producer operator ▪ Call consume on parents to generate code for their logic ▪ A Deep Dive into Query Execution of Spark SQL
- 17. Whole-Stage Code Generation Problems ▪ Customers map external data to internal representations using case statements ▪ Accounting use case for validating input => case expressions with thousands of when branches ▪ Generated functions are 1million LOC plus
- 18. Whole-Stage Code Generation Problems
- 19. Whole-Stage Code Generation Problems ▪ Java limits method size to 64KB ▪ JIT compilation disabled when methods exceed 8KB ▪ Compiler can throw OOM exceptions with large methods
- 20. Large Generated Code Solutions ▪ Split large functions into many small functions ▪ Implemented in expression code generation ▪ Whole-stage code generation can fallback to volcano iterator
- 21. Splitting Code Generated Functions
- 22. Splitting Expression Code Generation ▪ Each operator can be thought of as an iterator ▪ Result of every operator share common interface ▪ Only input to generated function is output of next
- 23. Splitting Expression Code Generation
- 24. Splitting Expression Code Generation
- 25. Splitting Whole-Stage Code Generation ▪ Before calling parent’s consume, store output variables
- 26. Splitting Whole-Stage Code Generation
- 27. Tracking Whole-Stage Code Generation Inputs ▪ Evaluated variables from current or child expressions ▪ Rows referred by current or child expressions ▪ Eliminated subexpressions in child expressions
- 28. Split Case Expression Function
- 29. Performance
- 30. Performance Setup ▪ 1 Driver ▪ 12GB memory ▪ 1 core ▪ 3 Executors ▪ 120GB memory ▪ 28 core ▪ 50 million input rows
- 31. Performance ▪ Project case expression with 3000 branches
- 32. Feedback Your feedback is important to us. Don’t forget to rate and review the sessions.