Video: https://www.youtube.com/watch?v=P5Q5GZheAEk Apache Hive is an open-source relational database system that is widely adopted by several organizations for big data analytic workloads. It combines traditional MPP (massively parallel processing) techniques with more recent cloud computing concepts to achieve the increased scalability and high performance needed by modern data intensive applications. Even though it was originally tailored towards long running data warehousing queries, its architecture recently changed with the introduction of LLAP (Live Long and Process) layer. Instead of regular containers, LLAP utilizes long-running executors to exploit data sharing and caching possibilities within and across queries. Executors eliminate unnecessary disk IO overhead and thus reduce the latency of interactive BI (business intelligence) queries by orders of magnitude. However, as container startup cost and IO overhead is now minimized, the need to effectively utilize memory and CPU resources across long-running executors in the cluster is becoming increasingly essential. For instance, in a variety of production workloads, we noticed that the memory bandwidth of early decoding all table columns for every row, even when this row is dropped later on, is starting to overwhelm the performance of single query execution. In this talk, we focus on some of the optimizations we introduced in Hive 4.0 to increase CPU efficiency and save memory allocations. In particular, we describe the lazy decoding (or row-level filtering) and composite bloom-filters optimizations that greatly improve the performance of queries containing broadcast joins, reducing their runtime by up to 50%. Over several production and synthetic workloads, we show the benefit of the newly introduced optimizations as part of Cloudera’s cloud-native Data Warehouse engine. At the same time, the community can directly benefit from the presented features as are they 100% open-source!

1. Accelerating distributed joins in
Apache Hive:
Runtime filtering enhancements
ApacheCon 2020
29th September 2020

2. © 2020 Cloudera, Inc. All rights reserved. 2
About us
Panagiotis (Panos) Garefalakis @pgaref
Software Engineer @Cloudera, Hive runtime team
Working on Apache Hive, ORC, Tez
PhD in Distributed Systems, Imperial College London
Stamatis Zampetakis
Software Engineer @Cloudera, Hive query optimizer team
Working on Apache Hive, Calcite
PhD in Data Management, INRIA & Paris-Sud University

3. © 2020 Cloudera, Inc. All rights reserved. 3
Apache Hive
Hive 1.0 Hive 2.0 Hive 3.0 Hive 4.0
GB to PB and low-latency multi-tenant
HiveQL
MapReduce
Read only
ETL
SQL
Tez & Spark
LLAP
Vectorization
BI Tools
Mat. Views
AntiJoin
Semi-join Red.
Lazy Mat.
Ad-Hoc
Batch SQL Analytics SQL Interactive SQL

4. © 2020 Cloudera, Inc. All rights reserved. 4
Hive users

5. © 2020 Cloudera, Inc. All rights reserved. 5
Hive with LLAP

6. © 2020 Cloudera, Inc. All rights reserved. 6
LLAP daemon anatomy
Query
Coordinator
Query
Coordinator
Query
Coordinator

7. © 2020 Cloudera, Inc. All rights reserved. 7
LLAP daemon anatomy
Query
Coordinator
Query
Coordinator
Query
Coordinator

8. © 2020 Cloudera, Inc. All rights reserved. 8
Outline
Data pruning
• Optimizations
• Benefit
Semijoin reduction
• Less data transfer
• Hive semijoin reduction
Lazy materialization
• ORC file format
• Row-level filtering
• Hive Probe-decode

9. © 2020 Cloudera, Inc. All rights reserved. 9
Data pruning
HOW? WHY?
Reduce memory footprint
Reduce disk & network I/O
Save CPU cycles
Semijoin reduction
Late materialization
... and many more

10. © 2020 Cloudera, Inc. All rights reserved. 10
Data pruning
Baseline
Scan
store_returns
InnerJoin
sr_reason_sk = r_reason_sk
Scan
reason
Filter
r_reason_desc=”Found...”
Project
sr_net_loss

11. © 2020 Cloudera, Inc. All rights reserved. 11
Data pruning
Baseline
Scan
store_returns
InnerJoin
sr_reason_sk = r_reason_sk
Scan
reason
Filter
r_reason_desc=”Found...”
Project
sr_net_loss

12. © 2020 Cloudera, Inc. All rights reserved. 12
Data pruning
Baseline
Scan
store_returns
InnerJoin
sr_reason_sk = r_reason_sk
Scan
reason
Filter
r_reason_desc=”Found...”
Project
sr_net_loss

13. © 2020 Cloudera, Inc. All rights reserved. 13
Data pruning
Baseline
Scan
store_returns
InnerJoin
sr_reason_sk = r_reason_sk
Scan
reason
Filter
r_reason_desc=”Found...”
Project
sr_net_loss

14. © 2020 Cloudera, Inc. All rights reserved. 14
Data pruning
Baseline
Scan
store_returns
InnerJoin
sr_reason_sk = r_reason_sk
Scan
reason
Filter
r_reason_desc=”Found...”
Project
sr_net_loss

15. © 2020 Cloudera, Inc. All rights reserved. 15
Data pruning
Baseline
Scan
store_returns
InnerJoin
sr_reason_sk = r_reason_sk
Scan
reason
Filter
r_reason_desc=”Found...”
Project
sr_net_loss

16. © 2020 Cloudera, Inc. All rights reserved. 16
Data pruning
Baseline
Scan
store_returns
InnerJoin
sr_reason_sk = r_reason_sk
Scan
reason
Filter
r_reason_desc=”Found...”
Project
sr_net_loss

17. © 2020 Cloudera, Inc. All rights reserved. 17
Data pruning
Baseline
Scan
store_returns
InnerJoin
sr_reason_sk = r_reason_sk
Scan
reason
Filter
r_reason_desc=”Found...”
Project
sr_net_loss

18. © 2020 Cloudera, Inc. All rights reserved. 18
Data pruning
Baseline
Scan
store_returns
InnerJoin
sr_reason_sk = r_reason_sk
Scan
reason
Filter
r_reason_desc=”Found...”
Project
sr_net_loss

19. © 2020 Cloudera, Inc. All rights reserved. 19
Data pruning
Filter pushdown
Scan
store_returns
InnerJoin
sr_reason_sk = r_reason_sk
Scan
reason
Project
sr_net_loss
Filter
r_reason_desc=”Found...”
BEFORE AFTER
Scan
store_returns
InnerJoin
sr_reason_sk = r_reason_sk
Scan
reason
Filter
r_reason_desc=”Found...”
Project
sr_net_loss

20. © 2020 Cloudera, Inc. All rights reserved. 20
Data pruning
Filter pushdown
BEFORE AFTER

21. © 2020 Cloudera, Inc. All rights reserved. 21
Data pruning
Project pushdown
BEFORE AFTER
Scan
store_returns
InnerJoin
sr_reason_sk = r_reason_sk
Scan
reason
Project
sr_net_loss
Filter
r_reason_desc=”Found...”
Scan
store_returns
InnerJoin
sr_reason_sk = r_reason_sk
Scan
reason
Filter
r_reason_desc=”Found...”
Project
sr_net_loss
Project
sr_reason_sk,
sr_net_loss
Project
r_reason_sk

22. © 2020 Cloudera, Inc. All rights reserved. 22
Data pruning
Project pushdown
BEFORE AFTER

23. © 2020 Cloudera, Inc. All rights reserved. 23
Data pruning
Semijoin reduction
BEFORE AFTER
Scan
store_returns
InnerJoin
sr_reason_sk = r_reason_sk
Scan
reason
Filter
r_reason_desc=”Found...”
Project
sr_net_loss
Project
sr_reason_sk,
sr_net_loss
Project
r_reason_sk
Scan
store_returns
SemiJoin
sr_reason_sk = r_reason_sk
Scan
reason
Filter
r_reason_desc=”Found...”
Project
sr_net_loss
Project
sr_reason_sk,
sr_net_loss
Project
r_reason_sk

24. © 2020 Cloudera, Inc. All rights reserved. 24
Data pruning
Semijoin reduction
BEFORE AFTER

25. © 2020 Cloudera, Inc. All rights reserved. 25
Data pruning
Late materialization
BEFORE AFTER
CE CE CE
CE
COMPRESSED
ENCODED (CE)

26. © 2020 Cloudera, Inc. All rights reserved. 26
Data pruning
Benefit
BASELINE FILTER PUSHDOWN PROJECT PUSHDOWN SEMIJOIN REDUCTION &
LATE MATERIALIZATION
CE CE CE
CE

27. © 2020 Cloudera, Inc. All rights reserved. 27
Data pruning
Benefit
BASELINE FILTER PUSHDOWN PROJECT PUSHDOWN SEMIJOIN REDUCTION &
LATE MATERIALIZATION
CE CE
CE
CE

28. Semijoin reduction

29. © 2020 Cloudera, Inc. All rights reserved. 29
Common names
Semijoin reduction
Invisible join
Selective join pushdown
Sideways information passing

30. © 2020 Cloudera, Inc. All rights reserved. 30
Benefit
Operators between the semijoin & and the reduced join treat less data:
➔ Reduce disk & network I/O
➔ Save CPU cycles
➔ Decrease memory footprint

31. © 2020 Cloudera, Inc. All rights reserved. 31
TPC-DS Query Example
Find the ten items that were returned due to a more competitive price

32. © 2020 Cloudera, Inc. All rights reserved. 32
InnerJoin
ss_item_sk=sr_item_sk AND ss_ticket_number=sr_ticket_number
Limit
10
Sort
net_loss DESC, price DESC
Scan
store_sales
Scan
store_returns
Aggregate
group=ss_item_sk, agg0=avg(ss_sales_price), agg1=(sr_net_loss)
InnerJoin
sr_reason_sk = r_reason_sk
Scan
reason
Filter
r_reason_desc=”Found...”

33. © 2020 Cloudera, Inc. All rights reserved. 33
InnerJoin
ss_item_sk=sr_item_sk AND ss_ticket_number=sr_ticket_number
Limit
10
Scan
store_sales
Scan
store_returns
Aggregate
group=ss_item_sk, agg0=avg(ss_sales_price), agg1=(sr_net_loss)
InnerJoin
sr_reason_sk = r_reason_sk
Scan
reason
Filter
r_reason_desc=”Found...” A
Sort
net_loss DESC, price DESC

34. © 2020 Cloudera, Inc. All rights reserved. 34
InnerJoin
ss_item_sk=sr_item_sk AND ss_ticket_number=sr_ticket_number
Limit
10
Scan
store_sales
Aggregate
group=ss_item_sk, agg0=avg(ss_sales_price), agg1=(sr_net_loss)
A
Sort
net_loss DESC, price DESC

35. © 2020 Cloudera, Inc. All rights reserved. 35
InnerJoin
ss_item_sk=sr_item_sk AND ss_ticket_number=sr_ticket_number
Limit
10
Scan
store_sales
Aggregate
group=ss_item_sk, agg0=avg(ss_sales_price), agg1=(sr_net_loss)
A
B
Sort
net_loss DESC, price DESC

36. © 2020 Cloudera, Inc. All rights reserved. 36
InnerJoin
ss_item_sk=sr_item_sk AND ss_ticket_number=sr_ticket_number
Scan
store_sales A
B

37. © 2020 Cloudera, Inc. All rights reserved. 37
InnerJoin
ss_item_sk=sr_item_sk AND ss_ticket_number=sr_ticket_number
Scan
store_sales A
B
2880404|46MB 8000|128KB
8000|160K

38. © 2020 Cloudera, Inc. All rights reserved. 38
InnerJoin
ss_item_sk=sr_item_sk AND ss_ticket_number=sr_ticket_number
Scan
store_sales A
B
N1 N2
N3
Node

39. © 2020 Cloudera, Inc. All rights reserved. 39
N3
N2
N1
46MB 128KB
160KB
Node

40. © 2020 Cloudera, Inc. All rights reserved. 40
InnerJoin
ss_item_sk=sr_item_sk AND ss_ticket_number=sr_ticket_number
Scan
store_sales A
B

41. © 2020 Cloudera, Inc. All rights reserved. 41
InnerJoin
ss_item_sk=sr_item_sk
AND ss_ticket_number=sr_ticket_number
Scan
store_sales A
B
SemiJoin
ss_item_sk=sr_item_sk
AND ss_ticket_number=sr_ticket_number

42. © 2020 Cloudera, Inc. All rights reserved. 42
InnerJoin
ss_item_sk=sr_item_sk
AND ss_ticket_number=sr_ticket_number
Scan
store_sales A
B
SemiJoin
ss_item_sk=sr_item_sk
AND ss_ticket_number=sr_ticket_number
2880404|46MB 8000|128KB
8000|160K
8000|128K

43. © 2020 Cloudera, Inc. All rights reserved. 43
InnerJoin
ss_item_sk=sr_item_sk
AND ss_ticket_number=sr_ticket_number
Scan
store_sales A
B
SemiJoin
ss_item_sk=sr_item_sk
AND ss_ticket_number=sr_ticket_number
N3
N2
N1
N1
Node

44. © 2020 Cloudera, Inc. All rights reserved. 44
N3
N2
N1
128KB 128KB
160KB
64KB
Node

45. © 2020 Cloudera, Inc. All rights reserved. 45
N3
N2
N1
128KB 128KB
160KB
64KB
N3
N2
N1
46MB 128KB
160KB
With semijoin
Total I/O=480KB
Without semijoin
Total I/O=46MB

46. © 2020 Cloudera, Inc. All rights reserved. 46
InnerJoin
ss_item_sk=sr_item_sk
AND ss_ticket_number=sr_ticket_number
Scan
store_sales A
B
SemiJoin
ss_item_sk=sr_item_sk
AND ss_ticket_number=sr_ticket_number
Semijoin reduction in Hive

47. © 2020 Cloudera, Inc. All rights reserved. 47
InnerJoin
ss_item_sk=sr_item_sk
AND ss_ticket_number=sr_ticket_number
Scan
store_sales A
B
Filter
ss_item_sk BETWEEN :min_sr_item_sk AND :max_sr_item
ss_ticket_number BETWEEN :min_sr_ticket_number AND :max_sr_ticket_number
IN_BLOOM_FILTER(HASH(ss_item_sk, ss_ticket_number), :bloom_sr)
Aggregate
min(sr_item_sk)
max(sr_item_sk)
min(sr_ticket_number)
max(sr_ticket_number)
bloom_filter(hash(sr_item_sk,sr_ticket_number))
Semijoin reduction in Hive

48. © 2020 Cloudera, Inc. All rights reserved. 48
InnerJoin
ss_item_sk=sr_item_sk
AND ss_ticket_number=sr_ticket_number
Scan
store_sales A
B
Filter
ss_item_sk BETWEEN :min_sr_item_sk AND :max_sr_item
ss_ticket_number BETWEEN :min_sr_ticket_number AND :max_sr_ticket_number
IN_BLOOM_FILTER(HASH(ss_item_sk, ss_ticket_number), :bloom_sr)
Aggregate
MIN(sr_item_sk)
MAX(sr_item_sk)
MIN(sr_ticket_number)
MAX(sr_ticket_number)
BLOOM_FILTER(HASH(sr_item_sk,sr_ticket_number))
Semijoin reduction in Hive

49. © 2020 Cloudera, Inc. All rights reserved. 49
InnerJoin
ss_item_sk=sr_item_sk
AND ss_ticket_number=sr_ticket_number
Scan
store_sales A
B
Filter
ss_item_sk BETWEEN :min_sr_item_sk AND :max_sr_item
ss_ticket_number BETWEEN :min_sr_ticket_number AND :max_sr_ticket_number
IN_BLOOM_FILTER(HASH(ss_item_sk, ss_ticket_number), :bloom_sr)
Aggregate
min(sr_item_sk)
max(sr_item_sk)
min(sr_ticket_number)
max(sr_ticket_number)
bloom_filter(hash(sr_item_sk,sr_ticket_number))
Semijoin reduction in Hive

50. © 2020 Cloudera, Inc. All rights reserved. 50
InnerJoin
ss_item_sk=sr_item_sk
AND ss_ticket_number=sr_ticket_number
Scan
store_sales A
B
Filter
ss_item_sk BETWEEN :min_sr_item_sk AND :max_sr_item
ss_ticket_number BETWEEN :min_sr_ticket_number AND :max_sr_ticket_number
IN_BLOOM_FILTER(HASH(ss_item_sk, ss_ticket_number), :bloom_sr)
Aggregate
min(sr_item_sk)
max(sr_item_sk)
min(sr_ticket_number)
max(sr_ticket_number)
bloom_filter(hash(sr_item_sk,sr_ticket_number))
Semijoin reduction in Hive

51. © 2020 Cloudera, Inc. All rights reserved. 51
N1
64KB
N3
N2
N1
46MB 128KB
160KB
With semijoin
Total I/O=480KB
Without semijoin
Total I/O=46MB
N3
N2
128KB 128KB
160KB
N1
9KB
With bloom semijoin
Total I/O=425KB
N3
N2
128KB 128KB
160KB

52. © 2020 Cloudera, Inc. All rights reserved. 52
Semijoin reduction in Hive
Four phase overview
Create synthetic predicate (SyntheticJoinPredicate)
Expand to semijoin reducers (DynamicPartitionPruningOptimization)
Merge to multi column semijoin reducers (SemiJoinReductionMerge)
• HIVE-21196
• HIVE-23976
Remove conditionally semijoin reducers (SemiJoinRemoval & TezCompiler)

53. © 2020 Cloudera, Inc. All rights reserved. 53
Four phase overview
Create synthetic predicate
Expand to semijoin reducers
Merge to multi column semijoin reducers
Conditionally remove semijoin reducers

54. © 2020 Cloudera, Inc. All rights reserved. 54
JOIN
ss_item_sk=sr_item_sk AND
ss_ticket_number=sr_ticket_number
TS
store_sales
JOIN RS[7]
RS[6]
store_returns
reason

55. © 2020 Cloudera, Inc. All rights reserved. 55
JOIN
ss_item_sk=sr_item_sk AND
ss_ticket_number=sr_ticket_number
TS
store_sales
JOIN RS[7]
RS[6]
store_returns
reason

56. © 2020 Cloudera, Inc. All rights reserved. 56
JOIN
ss_item_sk=sr_item_sk AND
ss_ticket_number=sr_ticket_number
TS
store_sales
JOIN RS[7]
RS[6]
store_returns
reason
FIL
ss_item_sk IN RS[7] AND
ss_ticket_number_sk IN RS[7]

57. © 2020 Cloudera, Inc. All rights reserved. 57
Four phase overview
Create synthetic predicate
Expand to semijoin reducers
Merge to multi column semijoin reducers
Conditionally remove semijoin reducers

58. © 2020 Cloudera, Inc. All rights reserved. 58
JOIN
ss_item_sk=sr_item_sk AND
ss_ticket_number=sr_ticket_number
TS
store_sales
JOIN RS[7]
RS[6]
store_returns
reason
FIL
ss_item_sk IN RS[7] AND
ss_ticket_number_sk IN RS[7]

59. © 2020 Cloudera, Inc. All rights reserved. 59
JOIN
ss_item_sk=sr_item_sk AND
ss_ticket_number=sr_ticket_number
TS
store_sales
JOIN RS[7]
RS[6]
store_returns
reason
FIL
ss_item_sk IN RS[7] AND
ss_ticket_number_sk IN RS[7]

60. © 2020 Cloudera, Inc. All rights reserved. 60
JOIN
ss_item_sk=sr_item_sk AND
ss_ticket_number=sr_ticket_number
TS
store_sales
JOIN RS[7]
RS[6]
store_returns
reason
FIL
ss_item_sk IN RS[7] AND
ss_ticket_number_sk IN RS[7]
SEL
sr_item_sk
RS[8]
GBY
MIN(0)
MAX(0)
BLOOM_FILTER(0)

61. © 2020 Cloudera, Inc. All rights reserved. 61
JOIN
ss_item_sk=sr_item_sk AND
ss_ticket_number=sr_ticket_number
TS
store_sales
JOIN RS[7]
RS[6]
store_returns
reason
FIL
ss_item_sk IN RS[7] AND
ss_ticket_number_sk IN RS[7]
SEL
sr_item_sk
RS[8]
GBY
MIN(0)
MAX(0)
BLOOM_FILTER(0)
RS[9]
GBY
MIN(0)
MAX(1)
BLOOM_FILTER(2)

62. © 2020 Cloudera, Inc. All rights reserved. 62
JOIN
ss_item_sk=sr_item_sk AND
ss_ticket_number=sr_ticket_number
TS
store_sales
JOIN RS[7]
RS[6]
store_returns
reason
SEL
sr_item_sk
RS[8]
GBY
MIN(0)
MAX(0)
BLOOM_FILTER(0)
RS[9]
GBY
MIN(0)
MAX(1)
BLOOM_FILTER(2)
FIL
ss_item_sk BETWEEN :sr_item_sk_min AND :sr_item_sk_max IN_BLOOM_FILTER(ss_item_sk, :sr_item_bloom)
ss_ticket_number_sk IN RS[7]

63. © 2020 Cloudera, Inc. All rights reserved. 63
JOIN
ss_item_sk=sr_item_sk AND
ss_ticket_number=sr_ticket_number
TS
store_sales
JOIN RS[7]
RS[6]
store_returns
reason
SEL
sr_item_sk
RS[8]
GBY
MIN(0)
MAX(0)
BLOOM_FILTER(0)
RS[9]
GBY
MIN(0)
MAX(1)
BLOOM_FILTER(2)
FIL
ss_item_sk BETWEEN :sr_item_sk_min AND :sr_item_sk_max IN_BLOOM_FILTER(ss_item_sk, :sr_item_bloom)
ss_ticket_number_sk IN RS[7]

64. © 2020 Cloudera, Inc. All rights reserved. 64
JOIN
ss_item_sk=sr_item_sk AND
ss_ticket_number=sr_ticket_number
TS
store_sales
JOIN RS[7]
RS[6]
store_returns
reason
SEL
sr_item_sk
RS[8]
GBY
MIN(0)
MAX(0)
BLOOM_FILTER(0)
RS[9]
GBY
MIN(0)
MAX(1)
BLOOM_FILTER(2)
FIL
ss_item_sk BETWEEN :sr_item_sk_min AND :sr_item_sk_max IN_BLOOM_FILTER(ss_item_sk, :sr_item_bloom)
ss_ticket_number_sk IN RS[7]

65. © 2020 Cloudera, Inc. All rights reserved. 65
JOIN
ss_item_sk=sr_item_sk AND
ss_ticket_number=sr_ticket_number
TS
store_sales
JOIN RS[7]
RS[6]
store_returns
reason
SEL
sr_item_sk
RS[8]
GBY
MIN(0)
MAX(0)
BLOOM_FILTER(0)
RS[9]
GBY
MIN(0)
MAX(1)
BLOOM_FILTER(2)
FIL
ss_item_sk BETWEEN :sr_item_sk_min AND :sr_item_sk_max IN_BLOOM_FILTER(ss_item_sk, :sr_item_bloom)
ss_ticket_number BETWEEN :sr_ticket_min AND :sr_ticket_max IN_BLOOM_FILTER(ss_item_sk, :sr_ticket_bloom)
SEL
sr_ticket_number
RS[10]
GBY
MIN(0)
MAX(0)
BLOOM_FILTER(0)
RS[11]
GBY
MIN(0)
MAX(1)
BLOOM_FILTER(2)

66. © 2020 Cloudera, Inc. All rights reserved. 66
JOIN
ss_item_sk=sr_item_sk AND
ss_ticket_number=sr_ticket_number
TS
store_sales
JOIN RS[7]
RS[6]
store_returns
reason
SEL
sr_item_sk
RS[8]
GBY
MIN(0)
MAX(0)
BLOOM_FILTER(0)
RS[9]
GBY
MIN(0)
MAX(1)
BLOOM_FILTER(2)
FIL
ss_item_sk BETWEEN :sr_item_sk_min AND :sr_item_sk_max IN_BLOOM_FILTER(ss_item_sk, :sr_item_bloom)
ss_ticket_number BETWEEN :sr_ticket_min AND :sr_ticket_max IN_BLOOM_FILTER(ss_item_sk, :sr_ticket_bloom)
SEL
sr_ticket_number
RS[10]
GBY
MIN(0)
MAX(0)
BLOOM_FILTER(0)
RS[11]
GBY
MIN(0)
MAX(1)
BLOOM_FILTER(2)

67. © 2020 Cloudera, Inc. All rights reserved. 67
Four phase overview
Create synthetic predicate
Expand to semijoin reducers
Merge to multi column semijoin reducers
Conditionally remove semijoin reducers

68. © 2020 Cloudera, Inc. All rights reserved. 68
JOIN
ss_item_sk=sr_item_sk AND
ss_ticket_number=sr_ticket_number
TS
store_sales
JOIN RS[7]
RS[6]
store_returns
reason
SEL
sr_item_sk
RS[8]
GBY
MIN(0)
MAX(0)
BLOOM_FILTER(0)
RS[9]
GBY
MIN(0)
MAX(1)
BLOOM_FILTER(2)
SEL
sr_ticket_number
RS[10]
GBY
MIN(0)
MAX(0)
BLOOM_FILTER(0)
RS[11]
GBY
MIN(0)
MAX(1)
BLOOM_FILTER(2)
FIL
ss_item_sk BETWEEN :sr_item_sk_min AND :sr_item_sk_max IN_BLOOM_FILTER(ss_item_sk, :sr_item_bloom)
ss_ticket_number BETWEEN :sr_ticket_min AND :sr_ticket_max IN_BLOOM_FILTER(ss_item_sk, :sr_ticket_bloom)

69. © 2020 Cloudera, Inc. All rights reserved. 69
JOIN
ss_item_sk=sr_item_sk AND
ss_ticket_number=sr_ticket_number
TS
store_sales
JOIN RS[7]
RS[6]
store_returns
reason
SEL
sr_item_sk, sr_ticket_number
RS[8]
GBY
MIN(0)
MAX(0)
BLOOM_FILTER(0)
RS[9]
GBY
MIN(0)
MAX(1)
BLOOM_FILTER(2)
RS[10]
GBY
MIN(0)
MAX(0)
BLOOM_FILTER(0)
RS[11]
GBY
MIN(0)
MAX(1)
BLOOM_FILTER(2)
FIL
ss_item_sk BETWEEN :sr_item_sk_min AND :sr_item_sk_max IN_BLOOM_FILTER(ss_item_sk, :sr_item_bloom)
ss_ticket_number BETWEEN :sr_ticket_min AND :sr_ticket_max IN_BLOOM_FILTER(ss_item_sk, :sr_ticket_bloom)

70. © 2020 Cloudera, Inc. All rights reserved. 70
JOIN
ss_item_sk=sr_item_sk AND
ss_ticket_number=sr_ticket_number
TS
store_sales
JOIN RS[7]
RS[6]
store_returns
reason
SEL
sr_item_sk, sr_ticket_number, HASH(sr_item_sk, sr_ticket_number)
RS[8]
GBY
MIN(0)
MAX(0)
BLOOM_FILTER(0)
RS[9]
GBY
MIN(0)
MAX(1)
BLOOM_FILTER(2)
RS[10]
GBY
MIN(0)
MAX(0)
BLOOM_FILTER(0)
RS[11]
GBY
MIN(0)
MAX(1)
BLOOM_FILTER(2)
FIL
ss_item_sk BETWEEN :sr_item_sk_min AND :sr_item_sk_max IN_BLOOM_FILTER(ss_item_sk, :sr_item_bloom)
ss_ticket_number BETWEEN :sr_ticket_min AND :sr_ticket_max IN_BLOOM_FILTER(ss_item_sk, :sr_ticket_bloom)

71. © 2020 Cloudera, Inc. All rights reserved. 71
JOIN
ss_item_sk=sr_item_sk AND
ss_ticket_number=sr_ticket_number
TS
store_sales
JOIN RS[7]
RS[6]
store_returns
reason
SEL
sr_item_sk, sr_ticket_number, HASH(sr_item_sk, sr_ticket_number)
RS[8]
GBY
MIN(0), MAX(0),
MIN(1), MAX(1),
BLOOM_FILTER(2)
RS[9]
GBY
MIN(0)
MAX(1)
BLOOM_FILTER(2)
RS[10]
RS[11]
GBY
MIN(0)
MAX(1)
BLOOM_FILTER(2)
FIL
ss_item_sk BETWEEN :sr_item_sk_min AND :sr_item_sk_max IN_BLOOM_FILTER(ss_item_sk, :sr_item_bloom)
ss_ticket_number BETWEEN :sr_ticket_min AND :sr_ticket_max IN_BLOOM_FILTER(ss_item_sk, :sr_ticket_bloom)

72. © 2020 Cloudera, Inc. All rights reserved. 72
JOIN
ss_item_sk=sr_item_sk AND
ss_ticket_number=sr_ticket_number
TS
store_sales
JOIN RS[7]
RS[6]
store_returns
reason
SEL
sr_item_sk, sr_ticket_number, HASH(sr_item_sk, sr_ticket_number)
RS
GBY
MIN(0), MAX(0),
MIN(1), MAX(1),
BLOOM_FILTER(2)
RS
GBY
MIN(0), MAX(1),
MIN(2), MAX(3),
BLOOM_FILTER(4)
FIL
ss_item_sk BETWEEN :sr_item_sk_min AND :sr_item_sk_max IN_BLOOM_FILTER(ss_item_sk, :sr_item_bloom)
ss_ticket_number BETWEEN :sr_ticket_min AND :sr_ticket_max IN_BLOOM_FILTER(ss_item_sk, :sr_ticket_bloom)

73. © 2020 Cloudera, Inc. All rights reserved. 73
JOIN
ss_item_sk=sr_item_sk AND
ss_ticket_number=sr_ticket_number
TS
store_sales
JOIN RS[7]
RS[6]
store_returns
reason
SEL
sr_item_sk, sr_ticket_number, HASH(sr_item_sk, sr_ticket_number)
RS
GBY
MIN(0), MAX(0),
MIN(1), MAX(1),
BLOOM_FILTER(2)
RS
GBY
MIN(0), MAX(1),
MIN(2), MAX(3),
BLOOM_FILTER(4)
FIL
ss_item_sk BETWEEN :sr_item_sk_min AND :sr_item_sk_max
ss_ticket_number BETWEEN :sr_ticket_min AND :sr_ticket_max
IN_BLOOM_FILTER(HASH(ss_item_sk, ss_ticket_number), :sr_item_ticket_bloom)

74. © 2020 Cloudera, Inc. All rights reserved. 74
Four phase overview
Create synthetic predicate
Expand to semijoin reducers
Merge to multi column semijoin reducers
Conditionally remove semijoin reducers

75. © 2020 Cloudera, Inc. All rights reserved. 75
Semijoin removal
Remove semijoin reduction:
• after union operator (correctness step)
• with missing statistics
• with parallel map join
• when precedes a sorted merge bucket (SMB) map join
• by benefit estimation
Code lies mostly inside TezCompiler

76. Lazy materialization

77. © 2020 Cloudera, Inc. All rights reserved. 77
Common names
Row-level filtering
Lazy decoding
Probe-decode

78. © 2020 Cloudera, Inc. All rights reserved. 78
Benefit
Avoid materializing data, not needed for the evaluation of the remaining of a
query
➔ Save CPU cycles
➔ Reduce memory allocations
➔ Eliminate data and filter operators from the pipeline

79. © 2020 Cloudera, Inc. All rights reserved. 79
ORC row group selection
• Data organized in stripes
of row groups (row x 10K)
• A single matched key of a
row group leads to Ks of
decoded rows
• Even for low selectivity
queries, most of the
row-groups might be read
and decoded

80. © 2020 Cloudera, Inc. All rights reserved. 80
ORC row group decoding

81. © 2020 Cloudera, Inc. All rights reserved. 81
ORC file structure
The file footer contains:
• Metadata
• Stripe information
• Postscript with compression details
ORC file data divided into stripes:
• Self contained sets of rows (organized by columns)
• Smallest unit of work for tasks
• ~64MB by default, often larger/smaller (16MB for Hive ACID deltas)

82. © 2020 Cloudera, Inc. All rights reserved. 82
ORC file structure
Stripe footer:
• List of col streams and locations
• Column encoding
Index data:
• Offset to seek to right row group
• Stats (min, max, sum) per column
• Orc.row.index.stride 10,000
Row data:
• Index stream
• Metadata stream
• Data stream
Stripe
N
Stripe
1

83. © 2020 Cloudera, Inc. All rights reserved. 83
ORC file structure
Stripe footer:
• List of col streams and locations
• Column encoding
Index data:
• Offset to seek to right row group
• Stats (min, max, sum) per column
• Orc.row.index.stride 10,000
Row data:
• Index stream
• Metadata stream
• Data stream
•
Stripe
N
Stripe
1

84. © 2020 Cloudera, Inc. All rights reserved. 84
ORC file structure
Stripe footer:
• List of col streams and locations
• Column encoding
Index data:
• Offset to seek to right row group
• Stats (min, max, sum) per column
• Orc.row.index.stride 10,000
Row data:
• Index stream
• Metadata stream
• Data stream
Stripe
N
Stripe
1

85. © 2020 Cloudera, Inc. All rights reserved. 85
Hive can pass Search Arguments (SArgs) filter to the Reader:
• Limited set of operations (=, <=>, <, <=, in, between, is null)
• Compare one column to literal(s) e.g., “age > 100”
Limitations:
• Can’t support filters with: inequalities, UDFs, or columns with relationships
• Can only eliminate entire row groups, stripes, or files
ORC search arguments

86. © 2020 Cloudera, Inc. All rights reserved. 86
SArgs filter RGs ONLY if they can guarantee that none of the rows can pass the
filter.
ORC row-level filtering
Row
group
1
Row data
Stripe

87. © 2020 Cloudera, Inc. All rights reserved. 87
SArgs filter RGs ONLY if they can guarantee that none of the rows can pass the
filter. [ORC-577] Finer grained row-level filtering!
Read a subset of the columns and apply more detailed row level filter.
ORC row-level filtering
Row
group
1
Row data
Stripe
Early
Expand
1

88. © 2020 Cloudera, Inc. All rights reserved. 88
SArgs filter RGs ONLY if they can guarantee that none of the rows can pass the
filter. [ORC-577] Finer grained row-level filtering!
Read a subset of the columns and apply more detailed row level filter.
ORC row-level filtering
Row
group
1
Row data
Stripe
Early
Expand
1 2
Apply
Filter
Selected rows

89. © 2020 Cloudera, Inc. All rights reserved. 89
SArgs filter RGs ONLY if they can guarantee that none of the rows can pass the
filter. [ORC-577] Finer grained row-level filtering!
Read a subset of the columns and apply more detailed row level filter.
ORC row-level filtering
Row
group
1
Row data
Stripe
Early
Expand
1 2
Apply
Filter
Selected rows
Read
Remaining
3

90. © 2020 Cloudera, Inc. All rights reserved. 90
ORC row-level filtering evaluation
[ORC-597] benchmark ORC Reader for complex types such as Decimals,
Doubles and real datasets (Taxi, Sales, GitHub)

91. © 2020 Cloudera, Inc. All rights reserved. 91
[HIVE-22959] introduce FilterContext as part of VectorRowBatch
[ORC-577] extend ORC Reader with a new option:
setRowFilter(String[] columnNames,Consumer<VectorizedRowBatch> filter)
Consumer callback to implement any kind of filtering logic
New ORC reader logic:
• Early expand the given column(s)
• Evaluate the filter callback (implemented on the consumer side)
• Decode only the selected rows from the remaining columns
ORC row-level filtering implementation

92. © 2020 Cloudera, Inc. All rights reserved. 92
Probe-decode optimization
Leveraging ORC row-level filtering
Scan
date_dim
Filter
dyear=2000 AND dqoy=2
Scan
catalog_sales
InnerJoin
cs_sold_date_sk=d_date_sk
Limit
10
Decode
d_date_sk, dyear, dqoy
2_220_880_404 1_000_000
8_000
1_000_000
8_000

93. © 2020 Cloudera, Inc. All rights reserved. 93
Probe-decode optimization
Static row-level filtering
Scan
date_dim
Filter
dyear=2000 AND dqoy=2
Scan
catalog_sales
InnerJoin
cs_sold_date_sk=d_date_sk
Limit
10
ProbeDecode
d_date_sk
2_220_880_404 1_000_000
8_000
8_000
8_000

94. © 2020 Cloudera, Inc. All rights reserved. 94
Probe-decode optimization
Dynamic MJ row-level filtering
Scan
date_dim
Filter
dyear=2000 AND dqoy=2
Scan
catalog_sales
InnerJoin
cs_sold_date_sk=d_date_sk
Limit
10
SemiJoin
cs_sold_date_sk=d_date_sk
ProbeDecode
cs_item_sk
420_504
2_220_880_404 1_000_000
8_000
8_000
8_000
ProbeDecode
d_date_sk

95. © 2020 Cloudera, Inc. All rights reserved. 95
Probe-decode plan

96. © 2020 Cloudera, Inc. All rights reserved. 96
⬢
–
–
–
–
–
DATA PLANE SERVICES
EDWS Data Analytics Studio (DAS) Management Services
COMPUTE CLUSTER
SHARED
SERVICES
Ranger
Atlas
Metastore
Tiller API Server
DAS Web Service
Query Coordinators
Query Executors
Registry
Blobstore
Indexer
RDBMS
Hive Server
Long-running kubernetes cluster
Inter-cluster communication Intra-cluster communication
Ingress Controller or
Load Balancer
Internal Service Endpoint for
ReplicaSet or StatefulSet
Ephemeral kubernetes cluster

97. © 2020 Cloudera, Inc. All rights reserved. 97
Data pruning
• Prune early → increase query efficiency
Semijoin reduction
• Reduce disk & network I/O
Lazy materialization
• Save CPU cycles and memory allocations
Conclusions

99. © 2020 Cloudera, Inc. All rights reserved. 99
References
1. Stocker, Konrad, et al. "Integrating semi-join-reducers into state-of-the-art
query processors." ICDE 2001
2. Abadi, Daniel J., Samuel R. Madden, and Nabil Hachem. "Column-stores vs.
row-stores: how different are they really?." SIGMOD 2008
3. Lang, Harald, et al. "Performance-optimal filtering: Bloom overtakes cuckoo at
high throughput." VLDB 2019