An Overview on Optimization in Apache Hive: Past, Present, Future

An Overview on Optimization
in Apache Hive:
Past, Present, Future
Ashutosh Chauhan
Jesús Camacho Rodríguez
DataWorks Summit San Jose
June 4, 2017

2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Apache Hive
 Initial use case: batch processing
– Read-only data
– HiveQL (SQL-like query language)
– MapReduce
 Effort to take Hive beyond its batch processing roots
– Started in Apache Hive 0.10.0 (January 2013)
– Latest released version: Apache Hive 2.1.1 (December 2016)
 Extensive renovation to improve three different axes
– Latency: allow interactive and sub-second queries
– Scalability: from TB to PB of data
– SQL support: move from HiveQL to SQL standard

Apache Hive
 Multiple execution engines: Apache Tez and Apache Spark
 More efficient join execution algorithms
 Vectorized query execution
– Integration with columnar storage formats:
Apache ORC, Apache Parquet
 LLAP (Live Long and Process)
– Persistent deamons for low-latency queries
 Semi join reduction
Important execution internals improvements

Apache Hive
 Determine the most efficient way to execute a given query
– Huge benefits for declarative query languages like SQL
– Especially if queries are generated by BI tools
 Challenging: trade-off between plan generation latency and optimality
 Helps exploiting Apache Hive capabilities
– Join reordering and most efficient algorithms selection,
static and dynamic partition pruning, column pruning, etc.
Motivation for query optimizer

Agenda
An overview on optimization in Apache Hive
Introduction
Query optimizer overview

Query preparation
✗ Optimization difficult to perform over AST
– Missing opportunities
Before Apache Hive 0.14.0
SQL Parser
Semantic
Analyzer
Logical
Optimizer
Physical
Optimizer
Execution
Engine
AST
Annotated
ASTSQL Plan
Tuned
plan

Query preparation
 Logical optimizer powered by Apache Calcite
– Each new version of Hive tightens this integration
• Improvements in the optimizer to be more effective and more efficient
• Shifting logic from Apache Hive to Apache Calcite: predicate pushdown, constant folding, etc.
After Apache Hive 0.14.0
SQL Parser
Semantic
Analyzer
Logical
Optimizer
Physical
Optimizer
Execution
Engine
Logical
plan
Physical
plan
Optimized
logical planASTSQL

Query optimization
Release Date Features (not exhaustive)
Apache Hive 0.14.0 Nov 2014 • Cost-based ordering of joins
• Cost model improvements (expressions selectivity)
• Improvements to partition pruner
Apache Hive 1.1.0 Mar 2015 • Enable Apache Calcite optimization by default
• More rules (identity project removal, union merge)
Apache Hive 1.2.0 May 2015 • Improvements in predicate inference (leads to more pruning oportunities)
• More rules (predicate folding, transitive inference)
• Chained predicate pushdown, inference, propagation, and folding
Apache Hive 2.0.0 (*) Feb 2016 • Apache Calcite rewriting rules run even without stats (subset)
• Disable rules depending on input query profile
• Improvements in metadata caching
• More rules (sort and limit pushdown, aggregate pushdown)
Apache Hive 2.1.0 Jun 2016 • Enhancement to predicate pushdown, folding and inference existing rules
• Disabling equivalent rules written in Apache Hive
Optimizer improvements since Apache Hive 0.14.0
(*) Introduction of Llap

Rules aren’t only consideration
 Collection
– Faster computation
– All statistics stored per column and partition vs per table
– Automatic gathering
 Estimation
– Improved selectivity on estimates, based on NDVs, min, max
 Retrieval
– Merge partition statistics
– Multiple ongoing efforts to improve metastore scalability by getting rid of ORM layer
• Use Apache Hbase to store metadata
• Caching in metastore process
• Redesign underlying schema for metastore
– Yahoo! recorded improvements of up to 90% for some queries
Statistics

Agenda
Introduction
Logical optimization

Apache Calcite – Architecture

 Multi-phase optimization using Apache Calcite
– Both rule-based and cost-based phases
 More than 40 different rewriting rules
– Pushdown Filter predicates
– Pushdown Project expressions
– Infer new Filter predicates
– Infer new constant expressions
– Prune unnecesary operator columns
– Simplify expressions
– Pushdown Sort and Limit operators
– Merge operators
– Pull up constants
– …

Logical optimization – Example (cost-based)

 Capable of generating bushy join operator trees
– Both inputs of a join operator can receive results from other join operators
– Well suited for parallel execution engines: increases parallelism degree, performance and cluster
utilization
Join reordering
Join
Join
Join
Join
Left-deep join tree
Join
Join
Join
Bushy join tree
Join

 Combining two star schemas
– Fact tables:
• sales, inventory
– Dimension tables:
• customer, time, product, warehouse
Join reordering
inventory warehouse
productJoin
Join
Join Join
Join time
sales customer
Dimension tables filter
fact tables records

Logical optimization – Example (rule-based)

 Predicate pushdown: push predicates as close to Scan operators as possible
 Predicate inference: infer new predicates from existant ones (maybe redundant) that
help to optimize the plan further
 Predicate propagation: propagate inferred values for expressions through the plan
 Predicate folding: fold constant expressions in predicates
Interaction of different rules allows to generate more efficient plans

SELECT quantity FROM sales s
JOIN addr a ON s.c_id=a.c_id
WHERE (a.country='US' AND a.c_id=2452276)
OR (a.country='US' AND a.c_id=2452640)
Join
Query logical plan
Project
Filter
sales addr
s.c_id=a.c_id
(a.country='US' AND a.c_id=2452276) OR
(a.country='US' AND a.c_id=2452640)
Scan Scan
quantity

Join
Project
sales addr
s.c_id=a.c_id
Scan Scan
quantity
 Predicate pushdown
 Predicate inference
 Predicate propagation
Filter (a.country='US' AND a.c_id=2452276) OR
(a.country='US' AND a.c_id=2452640)

Join
Project
sales addr
s.c_id=a.c_id
Scan Scan
quantity
 Predicate Simplification
Filter a.country='US’
AND (a.c_id=2452276 OR a.c_id=2452640)

Project
Join
sales addr
s.c_id=a.c_id
Scan Scan
quantity
 Predicate inference
Filter a.country='US’
AND (a.c_id=2452276 OR a.c_id=2452640)Filters.c_id=2452276 OR
s.c_id=2452276
SELECT quantity FROM sales s
JOIN addr a ON s.c_id=a.c_id
WHERE (a.country='US' AND a.c_id=2452276)
OR (a.country='US' AND a.c_id=2452640)

Optimizer performance
Apache Hive 2.1.1 - TPCDS (10TB) queries
* Max execution time: 1000 s
Update on Hive performance within a few weeks: http://hortonworks.com/blog/
0
20
40
60
80
100
0
200
400
600
800
1000
1200
q48 q13 q58 q21 q84 q66 q73 q20 q34 q25 q15
Gain(%)
Executiontime(s)
Optimizer off Optimizer on

Agenda
Introduction - Past
Query optimizer overview – Past
Logical optimization - Past
Logical Rewrites - Present

Rules are not only for optimization
 Extended support for subqueries in filter (where/having) and select.
 Subqueries with
– conjunction/disjunction,
– as scalar subqueries
– in select.
 Leverages calcite to rewrite subqueries.
 Select * from src where src.key in (select key from src s1 where s1.key > '9')
– Will be rewritten into Left semi join between outer src and inner src s1.

Rules are not only for optimization - II
 Extended support for
– Union Distinct
– Intersect
– Except
 Leverages calcite to rewrite into existing Hive operators.
 Select * from tb1 intersect select * from tb2;
 Select * from tb1 except select * from tb2;
 Select * from tb1 distinct select * from tb2;
 Will be rewritten using join, group by, union, UDTF etc.

Agenda
Introduction - Past
Query optimizer overview – Past
Logical optimization – Past
Logical Optimization - Present

Druid query recognition (powered by Apache Calcite)
SELECT `user`, sum(`c_added`) AS s
FROM druid_table_1
WHERE EXTRACT(year FROM `__time`)
BETWEEN 2010 AND 2011
GROUP BY `user`
ORDER BY s DESC
LIMIT 10;
 Top 10 users that have added more characters
from beginning of 2010 until the end of 2011
Apache Hive - SQL query
Query logical plan
Druid Scan
Project
Aggregate
Sort Limit
Sink
Filter

FROM druid_table_1
GROUP BY `user`
ORDER BY s DESC
LIMIT 10;
 Top 10 users that have added more characters
from beginning of 2010 until the end of 2011
Query logical plan
Druid Scan
Project
Aggregate
Sort Limit
Sink
Filter
Possible to express filters
on time dimension using
SQL standard functions

Apache Hive
Druid query
select
FROM druid_table_1
GROUP BY `user`
ORDER BY s DESC
LIMIT 10;
 Initially:
– Scan is executed in Druid (select query)
– Rest of the query is executed in Hive
Query logical plan
Druid Scan
Project
Aggregate
Sort Limit
Sink
Filter

Apache Hive
Druid query
select
Rewriting
rule
FROM druid_table_1
GROUP BY `user`
ORDER BY s DESC
LIMIT 10;
 Rewriting rules push computation into Druid
– Need to check that operator meets some
pre-conditions before pushing it to Druid
Query logical plan
Druid Scan
Project
Aggregate
Sort Limit
Sink
Filter

Apache Hive
Druid query
select
Rewriting
rule
FROM druid_table_1
GROUP BY `user`
ORDER BY s DESC
LIMIT 10;
Query logical plan
Druid Scan
Project
Aggregate
Sort Limit
Sink
Filter

Apache Hive
Druid query
groupBy
Rewriting
rule
FROM druid_table_1
GROUP BY `user`
ORDER BY s DESC
LIMIT 10;
Query logical plan
Druid Scan
Project
Aggregate
Sort Limit
Sink
Filter

{
"queryType": "groupBy",
"dataSource": "users_index",
"granularity": "all",
"dimension": "user",
"aggregations": [ { "type": "longSum", "name": "s", "fieldName": "c_added" } ],
"limitSpec": {
"limit": 10,
"columns": [ {"dimension": "s", "direction": "descending" } ]
},
"intervals": [ "2010-01-01T00:00:00.000/2012-01-01T00:00:00.000" ]
}
Physical plan transformation
Apache Hive
Druid query
groupBy
Query logical plan
Druid Scan
Project
Aggregate
Sort Limit
Sink
Filter
Select
File SinkFile Sink
Table Scan
Query physical plan
Druid JSON query
Table Scan uses
Druid Input Format

Union Remove Rewrite
select key, c, m from (
– (select key, count(key) as c, 1 as m from src group by key) s1
– union all
– (select key, count(key) as c, 4 as m from src group by key) s4
– ) where m = 1;
select key, c, 1 as m from (select key, count(key) as c from src group by key;
Union All
Filter(c=1)
TS0 TS1
Filter(c=1) Filter(c=2)
TS0
Filter(c=1)

Count Distinct rewrite
 Select sum(c1), count(distinct c1) from t1;
 Select sum(s1), count(c1) from (select sum(c1) as s1, c1 from t1 group by c1) t2;

Shared work optimizer
 Queries might scan same table more than once
– Correlated subqueries, self-joins, union queries, etc.
– Common pattern in OLAP workloads
 Scanning same table only once can reduce execution time remarkably
– Specially for I/O bounded queries
 In addition reuses similar selection predicates and other operators

Left outer join
Shared scan optimizer
SELECT e1.employee_id, e1.first_name, e1.last_name,
e1.mgr_id, e2.first_name as mgr_first_name, e2.last_name as mgr_last_name
FROM employees e1 LEFT OUTER JOIN employees e2
ON e1.mgr_id = e2.employee_id;
Find all employees and their managers
Left outer join
employees
e1.mgr_id =
e2.employee_id
Scan employees
ReduceSink key:
employee_id
Scan
ReduceSinkkey:
mgr_id
employees
ReduceSink key:
employee_id
Scan
ReduceSinkkey:
mgr_id
e1.mgr_id =
e2.employee_id

Agenda
Introduction
Logical Optimization – Present
Road Ahead - Future

Work in progress
 A materialized view contains the results of a query
 Accelerate query execution by using automatic rewriting
– Automatically benefits from LLAP, since materialized view is represented as a table
– Possibility to use other systems to store views (federation)
 Powerful side-effects: accurate statistics for query plans
Materialized views support
Query
LLAP
Disk

Work in progress
 Table cmv_table
 Syntax
CREATE MATERIALIZED VIEW cmv_mat_view ENABLE REWRITE
AS
SELECT avg(unit_price), discount
FROM cmv_table GROUP BY discount
WHERE discount > 0.2;
Materialized views support
id name unit_price discount
1 'p1' 10.30 0.15
2 'p2' 3.14 0.35
3 'p3' 172.2 0.35
4 'p4' 978.76 0.35
Enable automatic rewriting
using this materialized view

Work in progress
 Implemented as a set of rewriting rules
 Early pre-filtering of irrelevant views
 Currently supports Project, Filter, Join, Aggregate, Scan operators
SELECT avg(unit_price), discount
FROM cmv_table GROUP BY discount
WHERE discount > 0.3;
Materialized views support – Automatic rewriting
Rewrite using
Materialized view
Filter
Scancmv_mat_view
discount>0.2
Group By
Filter
Scancmv_table
discount>0.3
Avg(price)
Query plan

Future
 Collecting column stats automatically
 Making better ndv estimates
 Performance of compiler
 Exploiting constraints for optimization
 Improving stats estimation and propagation in tree
 Improving cost model

Conclusion
 Improvements on effectiveness and efficiency over the last 3 years
– Contributions to Apache Hive and Apache Calcite
 Interesting road ahead for both communities
Progress of Apache Hive optimizer

Thank You
http://hive.apache.org | @ApacheHive
http://calcite.apache.org | @ApacheCalcite
Slides contributed by:
Vineet Garg
Pengcheng Xiong
Jesus Camacho Rodriguez
Julian Hyde

Future Work
 Cost model should reflect actual execution cost
 Takes into account CPU and IO cost for operators
 Dependent on underlying execution engine
– In contrast to cardinality-based model (default cost model)
– Only available for Apache Tez
 Requires ‘tuning’ parameters
– CPU atomic operation cost, local disk byte read/write cost, HDFS byte read/write cost, network
byte transfer cost
 Allows join algorithm selection in Apache Calcite
– Sort-merge join, mapjoin, bucket mapjoin, sorted bucket mapjoin
Extended cost model

Physical optimization
 Determine how the plan will be executed
– Algorithms to use for each of the operators
– Partitioning and sorting data between operators
 Finds additional optimization opportunities
– Removal of unnecesary stages
 Completely done in Apache Hive

Physical optimization – Example

Physical optimization - Example
 Dynamic runtime filtering
– Runtime decision to cancel scan of records depending on subquery results
– Simplified semi-join reduction optimization
SELECT …
FROM sales JOIN time ON sales.time_id = time.time_id
WHERE time.year = 2014 AND time.quarter IN ('Q1', 'Q2’)

– Equivalent to semi-join reduction optimization
SELECT …
Join
sales
sales.time_id = time.time_id
Physical query plan
Scan
Filter
time
year = 2014 AND
quarter IN ('Q1', 'Q2')
ReduceSinkScan key: time_id

– Equivalent to semi-join reduction optimization
SELECT …
Join
sales
sales.time_id = time.time_id
Physical query plan
Scan
Filter
time
year = 2014 AND
quarter IN ('Q1', 'Q2')
ReduceSinkScan
Scan only subset of records
with given time_id
key: time_id

An Overview on Optimization in Apache Hive: Past, Present, Future

More Related Content

What's hot

Similar to An Overview on Optimization in Apache Hive: Past, Present, Future

More from DataWorks Summit

Recently uploaded

An Overview on Optimization in Apache Hive: Past, Present, Future

Editor's Notes