Cost Based Optimizer in Apache Hive

1. Cost Based Optimizer
in Apache Hive
Page 1
Laljo John Pullokkaran
12/03/2014

2. Agenda
Page 2
• Status
• Performance Numbers
• Architecture
• Future Directions
• Q&A

3. © Hortonworks Inc. 2013
Status
Page 3
•Merged in to Apache Hive Trunk
• Included in Hive 14.0
• All of Hive Test passes with CBO
• ~ 3 man years of development effort
• Contributors: Harish Bhutani, Ashutosh Chauhan, Julian Hyde,
Sergey Shelukhin, Pengcheng Xiong, Mostafa Moktar, Laljo John
Pullokkaran

4. Performance Numbers
© Hortonworks Inc. 2013
Page 4
• Average gains of 2X on TPCDS queries
• 30X gains with TPCDS query 29

5. Performance Numbers - Query Break Down
© Hortonworks Inc. 2013
Page 5

6. Performance Numbers – Why?
© Hortonworks Inc. 2013
Page 6
• Reduced Intermediary data set
• Bushy Joins results in better parallelism
• Enables more Map Join as opposed to shuffle Join

7. Architecture - Overview
© Hortonworks Inc. 2013
Page 7
Hive SQL Hive AST Hive QB Tree
Hive AST Optimized Calcite Plan Calcite Plan
Hive Operator Tree Optimized Hive OP tree TEZ/MR DAG

8. Architecture - Optimizations in Calcite
© Hortonworks Inc. 2013
Page 8
• Projection Pruning
• Partition Pruning
• Predicate Push Down
• Transitive Predicate inference
• Push Down of Semi Joins
• Join Simplification
• Predicate Factorization
• Join Ordering

9. Architecture - Join Ordering
© Hortonworks Inc. 2013
Page 9
• Requires Column Statistics
• Cost Model uses NDV based heuristics
• Cost Model favors bushy join plans
• Cost Model is agnostic of execution engine

10. Future Directions
© Hortonworks Inc. 2013
Page 10
• Histogram support
• Join Algorithm selection
• Calcite return path