Apache Hive is an open-source relational database system that is widely adopted by several organizations for big data analytic workloads. It combines traditional MPP (massively parallel processing) techniques with more recent cloud computing concepts to achieve the increased scalability and high performance needed by modern data intensive applications. Even though it was originally tailored towards long running data warehousing queries, its architecture recently changed with the introduction of LLAP (Live Long and Process) layer. Instead of regular containers, LLAP utilizes long-running executors to exploit data sharing and caching possibilities within and across queries. Executors eliminate unnecessary disk IO overhead and thus reduce the latency of interactive BI (business intelligence) queries by orders of magnitude. However, as container startup cost and IO overhead is now minimized, the need to effectively utilize memory and CPU resources across long-running executors in the cluster is becoming increasingly essential. For instance, in a variety of production workloads, we noticed that the memory bandwidth of early decoding all table columns for every row, even when this row is dropped later on, is starting to overwhelm the performance of single query execution. In this talk, we focus on some of the optimizations we introduced in Hive 4.0 to increase CPU efficiency and save memory allocations. In particular, we describe the lazy decoding (or row-level filtering) and composite bloom-filters optimizations that greatly improve the performance of queries containing broadcast joins, reducing their runtime by up to 50%. Over several production and synthetic workloads, we show the benefit of the newly introduced optimizations as part of Cloudera’s cloud-native Data Warehouse engine. At the same time, the community can directly benefit from the presented features as are they 100% open-source!

1. Accelerating distributed joins in
Apache Hive:
Runtime filtering enhancements
ApacheCon 2020
29th September 2020

2. © 2020 Cloudera, Inc. All rights reserved. 2
About us
Panagiotis (Panos) Garefalakis @pgaref
Software Engineer @Cloudera, Hive runtime team
Working on Apache Hive, ORC, Tez
PhD in Distributed Systems, Imperial College London
Stamatis Zampetakis
Software Engineer @Cloudera, Hive query optimizer team
Working on Apache Hive, Calcite
PhD in Data Management, INRIA & Paris-Sud University

3. © 2020 Cloudera, Inc. All rights reserved. 3
Apache Hive
Hive 1.0 Hive 2.0 Hive 3.0 Hive 4.0
GB to PB and low-latency multi-tenant
HiveQL
MapReduce
Read only
ETL
SQL
Tez & Spark
LLAP
Vectorization
BI Tools
Mat. Views
AntiJoin
Semi-join Red.
Lazy Mat.
Ad-Hoc
Batch SQL Analytics SQL Interactive SQL

4. © 2020 Cloudera, Inc. All rights reserved. 4
Hive users

5. © 2020 Cloudera, Inc. All rights reserved. 5
Hive with LLAP

6. © 2020 Cloudera, Inc. All rights reserved. 6
LLAP daemon anatomy
Query
Coordinator
Query
Coordinator
Query
Coordinator

7. © 2020 Cloudera, Inc. All rights reserved. 7
LLAP daemon anatomy
Query
Coordinator
Query
Coordinator
Query
Coordinator

8. © 2020 Cloudera, Inc. All rights reserved. 8
Outline
Data pruning
• Optimizations
• Benefit
Semijoin reduction
• Less data transfer
• Hive semijoin reduction
Lazy materialization
• ORC file format
• Row-level filtering
• Hive Probe-decode

9. © 2020 Cloudera, Inc. All rights reserved. 9
Data pruning
HOW? WHY?
Reduce memory footprint
Reduce disk & network I/O
Save CPU cycles
Semijoin reduction
Late materialization
... and many more

10. © 2020 Cloudera, Inc. All rights reserved. 10
Data pruning
Baseline
SELECT sr_net_loss as net_loss
FROM store_returns
INNER JOIN reason
ON sr_reason_sk = r_reason_sk
WHERE
r_reason_desc =
'Found a better price in a store'
Scan
store_returns
InnerJoin
sr_reason_sk = r_reason_sk
Scan
reason
Filter
r_reason_desc=”Found...”
Project
sr_net_loss

11. © 2020 Cloudera, Inc. All rights reserved. 11
Data pruning
Baseline
SELECT sr_net_loss as net_loss
FROM store_returns
INNER JOIN reason
ON sr_reason_sk = r_reason_sk
WHERE
r_reason_desc =
'Found a better price in a store'
Scan
store_returns
InnerJoin
sr_reason_sk = r_reason_sk
Scan
reason
Filter
r_reason_desc=”Found...”
Project
sr_net_loss

12. © 2020 Cloudera, Inc. All rights reserved. 12
Data pruning
Baseline
SELECT sr_net_loss as net_loss
FROM store_returns
INNER JOIN reason
ON sr_reason_sk = r_reason_sk
WHERE
r_reason_desc =
'Found a better price in a store'
Scan
store_returns
InnerJoin
sr_reason_sk = r_reason_sk
Scan
reason
Filter
r_reason_desc=”Found...”
Project
sr_net_loss

13. © 2020 Cloudera, Inc. All rights reserved. 13
Data pruning
Baseline
SELECT sr_net_loss as net_loss
FROM store_returns
INNER JOIN reason
ON sr_reason_sk = r_reason_sk
WHERE
r_reason_desc =
'Found a better price in a store'
Scan
store_returns
InnerJoin
sr_reason_sk = r_reason_sk
Scan
reason
Filter
r_reason_desc=”Found...”
Project
sr_net_loss

14. © 2020 Cloudera, Inc. All rights reserved. 14
Data pruning
Baseline
SELECT sr_net_loss as net_loss
FROM store_returns
INNER JOIN reason
ON sr_reason_sk = r_reason_sk
WHERE
r_reason_desc =
'Found a better price in a store'
Scan
store_returns
InnerJoin
sr_reason_sk = r_reason_sk
Scan
reason
Filter
r_reason_desc=”Found...”
Project
sr_net_loss

15. © 2020 Cloudera, Inc. All rights reserved. 15
Data pruning
Baseline
SELECT sr_net_loss as net_loss
FROM store_returns
INNER JOIN reason
ON sr_reason_sk = r_reason_sk
WHERE
r_reason_desc =
'Found a better price in a store'
Scan
store_returns
InnerJoin
sr_reason_sk = r_reason_sk
Scan
reason
Filter
r_reason_desc=”Found...”
Project
sr_net_loss

16. © 2020 Cloudera, Inc. All rights reserved. 16
Data pruning
Baseline
SELECT sr_net_loss as net_loss
FROM store_returns
INNER JOIN reason
ON sr_reason_sk = r_reason_sk
WHERE
r_reason_desc =
'Found a better price in a store'
Scan
store_returns
InnerJoin
sr_reason_sk = r_reason_sk
Scan
reason
Filter
r_reason_desc=”Found...”
Project
sr_net_loss

17. © 2020 Cloudera, Inc. All rights reserved. 17
Data pruning
Baseline
SELECT sr_net_loss as net_loss
FROM store_returns
INNER JOIN reason
ON sr_reason_sk = r_reason_sk
WHERE
r_reason_desc =
'Found a better price in a store'
Scan
store_returns
InnerJoin
sr_reason_sk = r_reason_sk
Scan
reason
Filter
r_reason_desc=”Found...”
Project
sr_net_loss

18. © 2020 Cloudera, Inc. All rights reserved. 18
Data pruning
Baseline
SELECT sr_net_loss as net_loss
FROM store_returns
INNER JOIN reason
ON sr_reason_sk = r_reason_sk
WHERE
r_reason_desc =
'Found a better price in a store'
Scan
store_returns
InnerJoin
sr_reason_sk = r_reason_sk
Scan
reason
Filter
r_reason_desc=”Found...”
Project
sr_net_loss

19. © 2020 Cloudera, Inc. All rights reserved. 19
Data pruning
Filter pushdown
Scan
store_returns
InnerJoin
sr_reason_sk = r_reason_sk
Scan
reason
Project
sr_net_loss
Filter
r_reason_desc=”Found...”
BEFORE AFTER
Scan
store_returns
InnerJoin
sr_reason_sk = r_reason_sk
Scan
reason
Filter
r_reason_desc=”Found...”
Project
sr_net_loss

20. © 2020 Cloudera, Inc. All rights reserved. 20
Data pruning
Filter pushdown
BEFORE AFTER

21. © 2020 Cloudera, Inc. All rights reserved. 21
Data pruning
Project pushdown
BEFORE AFTER
Scan
store_returns
InnerJoin
sr_reason_sk = r_reason_sk
Scan
reason
Project
sr_net_loss
Filter
r_reason_desc=”Found...”
Scan
store_returns
InnerJoin
sr_reason_sk = r_reason_sk
Scan
reason
Filter
r_reason_desc=”Found...”
Project
sr_net_loss
Project
sr_reason_sk,
sr_net_loss
Project
r_reason_sk

22. © 2020 Cloudera, Inc. All rights reserved. 22
Data pruning
Project pushdown
BEFORE AFTER

23. © 2020 Cloudera, Inc. All rights reserved. 23
Data pruning
Semijoin reduction
BEFORE AFTER
Scan
store_returns
InnerJoin
sr_reason_sk = r_reason_sk
Scan
reason
Filter
r_reason_desc=”Found...”
Project
sr_net_loss
Project
sr_reason_sk,
sr_net_loss
Project
r_reason_sk
Scan
store_returns
SemiJoin
sr_reason_sk = r_reason_sk
Scan
reason
Filter
r_reason_desc=”Found...”
Project
sr_net_loss
Project
sr_reason_sk,
sr_net_loss
Project
r_reason_sk

24. © 2020 Cloudera, Inc. All rights reserved. 24
Data pruning
Semijoin reduction
BEFORE AFTER

25. © 2020 Cloudera, Inc. All rights reserved. 25
Data pruning
Late materialization
BEFORE AFTER
CE CE CE
CE
COMPRESSED
ENCODED (CE)

26. © 2020 Cloudera, Inc. All rights reserved. 26
Data pruning
Benefit
BASELINE FILTER PUSHDOWN PROJECT PUSHDOWN SEMIJOIN REDUCTION &
LATE MATERIALIZATION
CE CE CE
CE

27. © 2020 Cloudera, Inc. All rights reserved. 27
Data pruning
Benefit
BASELINE FILTER PUSHDOWN PROJECT PUSHDOWN SEMIJOIN REDUCTION &
LATE MATERIALIZATION
CE CE
CE
CE

28. Semijoin reduction

29. © 2020 Cloudera, Inc. All rights reserved. 29
Common names
Semijoin reduction
Invisible join
Selective join pushdown
Sideways information passing

30. © 2020 Cloudera, Inc. All rights reserved. 30
Benefit
Operators between the semijoin & and the reduced join treat less data:
➔ Reduce disk & network I/O
➔ Save CPU cycles
➔ Decrease memory footprint

31. © 2020 Cloudera, Inc. All rights reserved. 31
TPC-DS Query Example
Find the ten items that were returned due to a more competitive price
SELECT ss_item_sk, sum(sr_net_loss) as net_loss, avg(ss_sales_price) as price
FROM store_sales
INNER JOIN store_returns
ON ss_item_sk=sr_item_sk AND ss_ticket_number=sr_ticket_number
INNER JOIN reason
ON sr_reason_sk = r_reason_sk
WHERE r_reason_desc = 'Found a better price in a store'
GROUP BY ss_item_sk
ORDER BY net_loss DESC NULLS LAST, price DESC NULLS LAST
LIMIT 10;

32. © 2020 Cloudera, Inc. All rights reserved. 32
InnerJoin
ss_item_sk=sr_item_sk AND ss_ticket_number=sr_ticket_number
Limit
10
Sort
net_loss DESC, price DESC
Scan
store_sales
Scan
store_returns
Aggregate
group=ss_item_sk, agg0=avg(ss_sales_price), agg1=(sr_net_loss)
InnerJoin
sr_reason_sk = r_reason_sk
Scan
reason
Filter
r_reason_desc=”Found...”

33. © 2020 Cloudera, Inc. All rights reserved. 33
InnerJoin
ss_item_sk=sr_item_sk AND ss_ticket_number=sr_ticket_number
Limit
10
Scan
store_sales
Scan
store_returns
Aggregate
group=ss_item_sk, agg0=avg(ss_sales_price), agg1=(sr_net_loss)
InnerJoin
sr_reason_sk = r_reason_sk
Scan
reason
Filter
r_reason_desc=”Found...” A
Sort
net_loss DESC, price DESC

34. © 2020 Cloudera, Inc. All rights reserved. 34
InnerJoin
ss_item_sk=sr_item_sk AND ss_ticket_number=sr_ticket_number
Limit
10
Scan
store_sales
Aggregate
group=ss_item_sk, agg0=avg(ss_sales_price), agg1=(sr_net_loss)
A
Sort
net_loss DESC, price DESC

35. © 2020 Cloudera, Inc. All rights reserved. 35
InnerJoin
ss_item_sk=sr_item_sk AND ss_ticket_number=sr_ticket_number
Limit
10
Scan
store_sales
Aggregate
group=ss_item_sk, agg0=avg(ss_sales_price), agg1=(sr_net_loss)
A
B
Sort
net_loss DESC, price DESC

36. © 2020 Cloudera, Inc. All rights reserved. 36
InnerJoin
ss_item_sk=sr_item_sk AND ss_ticket_number=sr_ticket_number
Scan
store_sales A
B

37. © 2020 Cloudera, Inc. All rights reserved. 37
InnerJoin
ss_item_sk=sr_item_sk AND ss_ticket_number=sr_ticket_number
Scan
store_sales A
B
2880404|46MB 8000|128KB
8000|160K

38. © 2020 Cloudera, Inc. All rights reserved. 38
InnerJoin
ss_item_sk=sr_item_sk AND ss_ticket_number=sr_ticket_number
Scan
store_sales A
B
N1 N2
N3
Node

39. © 2020 Cloudera, Inc. All rights reserved. 39
N3
N2N1
46MB 128KB
160KB
Node

40. © 2020 Cloudera, Inc. All rights reserved. 40
InnerJoin
ss_item_sk=sr_item_sk AND ss_ticket_number=sr_ticket_number
Scan
store_sales A
B

41. © 2020 Cloudera, Inc. All rights reserved. 41
InnerJoin
ss_item_sk=sr_item_sk
AND ss_ticket_number=sr_ticket_number
Scan
store_sales A
B
SemiJoin
ss_item_sk=sr_item_sk
AND ss_ticket_number=sr_ticket_number

42. © 2020 Cloudera, Inc. All rights reserved. 42
InnerJoin
ss_item_sk=sr_item_sk
AND ss_ticket_number=sr_ticket_number
Scan
store_sales A
B
SemiJoin
ss_item_sk=sr_item_sk
AND ss_ticket_number=sr_ticket_number
2880404|46MB 8000|128KB
8000|160K
8000|128K

43. © 2020 Cloudera, Inc. All rights reserved. 43
InnerJoin
ss_item_sk=sr_item_sk
AND ss_ticket_number=sr_ticket_number
Scan
store_sales A
B
SemiJoin
ss_item_sk=sr_item_sk
AND ss_ticket_number=sr_ticket_number
N3
N2N1
N1
Node

44. © 2020 Cloudera, Inc. All rights reserved. 44
N3
N2N1
128KB 128KB
160KB
64KB
Node

45. © 2020 Cloudera, Inc. All rights reserved. 45
N3
N2N1
128KB 128KB
160KB
64KB
N3
N2N1
46MB 128KB
160KB
With semijoin
Total I/O=480KB
Without semijoin
Total I/O=46MB

46. © 2020 Cloudera, Inc. All rights reserved. 46
InnerJoin
ss_item_sk=sr_item_sk
AND ss_ticket_number=sr_ticket_number
Scan
store_sales A
B
SemiJoin
ss_item_sk=sr_item_sk
AND ss_ticket_number=sr_ticket_number
Semijoin reduction in Hive

47. © 2020 Cloudera, Inc. All rights reserved. 47
InnerJoin
ss_item_sk=sr_item_sk
AND ss_ticket_number=sr_ticket_number
Scan
store_sales A
B
Filterss_item_sk BETWEEN :min_sr_item_sk AND :max_sr_item
ss_ticket_number BETWEEN :min_sr_ticket_number AND :max_sr_ticket_number
IN_BLOOM_FILTER(HASH(ss_item_sk, ss_ticket_number), :bloom_sr)
Aggregatemin(sr_item_sk)
max(sr_item_sk)
min(sr_ticket_number)
max(sr_ticket_number)
bloom_filter(hash(sr_item_sk,sr_ticket_number))
Semijoin reduction in Hive

48. © 2020 Cloudera, Inc. All rights reserved. 48
InnerJoin
ss_item_sk=sr_item_sk
AND ss_ticket_number=sr_ticket_number
Scan
store_sales A
B
Filterss_item_sk BETWEEN :min_sr_item_sk AND :max_sr_item
ss_ticket_number BETWEEN :min_sr_ticket_number AND :max_sr_ticket_number
IN_BLOOM_FILTER(HASH(ss_item_sk, ss_ticket_number), :bloom_sr)
Aggregate
MIN(sr_item_sk)
MAX(sr_item_sk)
MIN(sr_ticket_number)
MAX(sr_ticket_number)
BLOOM_FILTER(HASH(sr_item_sk,sr_ticket_number))
Semijoin reduction in Hive

49. © 2020 Cloudera, Inc. All rights reserved. 49
InnerJoin
ss_item_sk=sr_item_sk
AND ss_ticket_number=sr_ticket_number
Scan
store_sales A
B
Filterss_item_sk BETWEEN :min_sr_item_sk AND :max_sr_item
ss_ticket_number BETWEEN :min_sr_ticket_number AND :max_sr_ticket_number
IN_BLOOM_FILTER(HASH(ss_item_sk, ss_ticket_number), :bloom_sr)
Aggregatemin(sr_item_sk)
max(sr_item_sk)
min(sr_ticket_number)
max(sr_ticket_number)
bloom_filter(hash(sr_item_sk,sr_ticket_number))
Semijoin reduction in Hive

50. © 2020 Cloudera, Inc. All rights reserved. 50
InnerJoin
ss_item_sk=sr_item_sk
AND ss_ticket_number=sr_ticket_number
Scan
store_sales A
B
Filter
ss_item_sk BETWEEN :min_sr_item_sk AND :max_sr_item
ss_ticket_number BETWEEN :min_sr_ticket_number AND :max_sr_ticket_number
IN_BLOOM_FILTER(HASH(ss_item_sk, ss_ticket_number), :bloom_sr)
Aggregatemin(sr_item_sk)
max(sr_item_sk)
min(sr_ticket_number)
max(sr_ticket_number)
bloom_filter(hash(sr_item_sk,sr_ticket_number))
Semijoin reduction in Hive

51. © 2020 Cloudera, Inc. All rights reserved. 51
N1
64KB
N3
N2N1
46MB 128KB
160KB
With semijoin
Total I/O=480KB
Without semijoin
Total I/O=46MB
N3
N2
128KB 128KB
160KB
N1
9KB
With bloom semijoin
Total I/O=425KB
N3
N2
128KB 128KB
160KB

52. © 2020 Cloudera, Inc. All rights reserved. 52
Semijoin reduction in Hive
Four phase overview
Create synthetic predicate (SyntheticJoinPredicate)
Expand to semijoin reducers (DynamicPartitionPruningOptimization)
Merge to multi column semijoin reducers (SemiJoinReductionMerge)
• HIVE-21196
• HIVE-23976
Remove conditionally semijoin reducers (SemiJoinRemoval & TezCompiler)

53. Lazy materialization

54. © 2020 Cloudera, Inc. All rights reserved. 54
Common names
Row-level filtering
Lazy decoding
Probe-decode

55. © 2020 Cloudera, Inc. All rights reserved. 55
Benefit
Avoid materializing data, not needed for the evaluation of the remaining of a
query
➔ Save CPU cycles
➔ Reduce memory allocations
➔ Eliminate data and filter operators from the pipeline

56. © 2020 Cloudera, Inc. All rights reserved. 56
ORC row group selection
• Data organized in stripes
of row groups (row x 10K)
• A single matched key of a
row group leads to Ks of
decoded rows
• Even for low selectivity
queries, most of the
row-groups might be read
and decoded

57. © 2020 Cloudera, Inc. All rights reserved. 57
ORC row group decoding

58. © 2020 Cloudera, Inc. All rights reserved. 58
ORC file structure
The file footer contains:
• Metadata
• Stripe information
• Postscript with compression details
ORC file data divided into stripes:
• Self contained sets of rows (organized by columns)
• Smallest unit of work for tasks
• ~64MB by default, often larger/smaller (16MB for Hive ACID deltas)

59. © 2020 Cloudera, Inc. All rights reserved. 59
ORC file structure
Stripe footer:
• List of col streams and locations
• Column encoding
Index data:
• Offset to seek to right row group
• Stats (min, max, sum) per column
• Orc.row.index.stride 10,000
Row data:
• Index stream
• Metadata stream
• Data stream
StripeNStripe1

60. © 2020 Cloudera, Inc. All rights reserved. 60
ORC file structure
Stripe footer:
• List of col streams and locations
• Column encoding
Index data:
• Offset to seek to right row group
• Stats (min, max, sum) per column
• Orc.row.index.stride 10,000
Row data:
• Index stream
• Metadata stream
• Data stream
•
StripeNStripe1

61. © 2020 Cloudera, Inc. All rights reserved. 61
ORC file structure
Stripe footer:
• List of col streams and locations
• Column encoding
Index data:
• Offset to seek to right row group
• Stats (min, max, sum) per column
• Orc.row.index.stride 10,000
Row data:
• Index stream
• Metadata stream
• Data stream
StripeNStripe1

62. © 2020 Cloudera, Inc. All rights reserved. 62
Hive can pass Search Arguments (SArgs) filter to the Reader:
• Limited set of operations (=, <=>, <, <=, in, between, is null)
• Compare one column to literal(s) e.g., “age > 100”
Limitations:
• Can’t support filters with: inequalities, UDFs, or columns with relationships
• Can only eliminate entire row groups, stripes, or files
SELECT x,y,z FROM foo WHERE x <> 1;
ORC search arguments

63. © 2020 Cloudera, Inc. All rights reserved. 63
SArgs filter RGs ONLY if they can guarantee that none of the rows can pass the
filter.
ORC row-level filteringRowgroup1
Row data
Stripe
SELECT x,y,z FROM foo WHERE x <> 1;

64. © 2020 Cloudera, Inc. All rights reserved. 64
SArgs filter RGs ONLY if they can guarantee that none of the rows can pass the
filter. [ORC-577] Finer grained row-level filtering!
Read a subset of the columns and apply more detailed row level filter.
ORC row-level filteringRowgroup1
Row data
Stripe
Early
Expand
1
SELECT x,y,z FROM foo WHERE x <> 1;

65. © 2020 Cloudera, Inc. All rights reserved. 65
SArgs filter RGs ONLY if they can guarantee that none of the rows can pass the
filter. [ORC-577] Finer grained row-level filtering!
Read a subset of the columns and apply more detailed row level filter.
ORC row-level filteringRowgroup1
Row data
Stripe
Early
Expand
1 2
Apply
Filter
Selected rows
SELECT x,y,z FROM foo WHERE x <> 1;

66. © 2020 Cloudera, Inc. All rights reserved. 66
SArgs filter RGs ONLY if they can guarantee that none of the rows can pass the
filter. [ORC-577] Finer grained row-level filtering!
Read a subset of the columns and apply more detailed row level filter.
ORC row-level filteringRowgroup1
Row data
Stripe
Early
Expand
1 2
Apply
Filter
Selected rows
Read
Remaining
3
SELECT x,y,z FROM foo WHERE x <> 1;

67. © 2020 Cloudera, Inc. All rights reserved. 67
ORC row-level filtering evaluation
[ORC-597] benchmark ORC Reader for complex types such as Decimals,
Doubles and real datasets (Taxi, Sales, GitHub)

68. © 2020 Cloudera, Inc. All rights reserved. 68
[HIVE-22959] introduce FilterContext as part of VectorRowBatch
[ORC-577] extend ORC Reader with a new option:
setRowFilter(String[] columnNames,Consumer<VectorizedRowBatch> filter)
Consumer callback to implement any kind of filtering logic
New ORC reader logic:
• Early expand the given column(s)
• Evaluate the filter callback (implemented on the consumer side)
• Decode only the selected rows from the remaining columns
ORC row-level filtering implementation

69. © 2020 Cloudera, Inc. All rights reserved. 69
Probe-decode optimization
Leveraging ORC row-level filtering
SELECT *
FROM catalog_sales
INNER JOIN date_dim
ON cs_sold_date_sk = d_date_sk
WHERE d_year = 2000 and d_qoy = 2
LIMIT 10
Scan
date_dim
Filter
dyear=2000 AND dqoy=2
Scan
catalog_sales
InnerJoin
cs_sold_date_sk=d_date_sk
Limit
10
Decode
d_date_sk, dyear, dqoy
2_220_880_404 1_000_000
8_000
1_000_000
8_000

70. © 2020 Cloudera, Inc. All rights reserved. 70
Probe-decode optimization
Static row-level filtering
SELECT *
FROM catalog_sales
INNER JOIN date_dim
ON cs_sold_date_sk = d_date_sk
WHERE d_year = 2000 and d_qoy = 2
LIMIT 10
Scan
date_dim
Filter
dyear=2000 AND dqoy=2
Scan
catalog_sales
InnerJoin
cs_sold_date_sk=d_date_sk
Limit
10
ProbeDecode
d_date_sk
2_220_880_404 1_000_000
8_000
8_000
8_000

71. © 2020 Cloudera, Inc. All rights reserved. 71
Probe-decode optimization
Dynamic MJ row-level filtering
SELECT *
FROM catalog_sales
INNER JOIN date_dim
ON cs_sold_date_sk = d_date_sk
WHERE d_year = 2000 and d_qoy = 2
LIMIT 10
Scan
date_dim
Filter
dyear=2000 AND dqoy=2
Scan
catalog_sales
InnerJoin
cs_sold_date_sk=d_date_sk
Limit
10
SemiJoin
cs_sold_date_sk=d_date_sk
ProbeDecode
cs_item_sk
420_504
2_220_880_404 1_000_000
8_000
8_000
8_000ProbeDecode
d_date_sk

72. © 2020 Cloudera, Inc. All rights reserved. 72
Probe-decode plan
Map 1 <- Map 4 (BROADCAST_EDGE), Map 5 (BROADCAST_EDGE)
Reducer 2 <- Map 1 (SIMPLE_EDGE)
Reducer 3 <- Reducer 2 (SIMPLE_EDGE)
Vertices:
Map 1
Map Operator Tree:
TableScan
alias: store_sales
probeDecodeDetails: cacheKey:HASH_MAP_MAPJOIN_48_container, bigKeyColName:ss_item_sk, smallTablePos:1, keyRatio:1.6
Statistics: Num rows: 82510879939 Data size: 10343396725952 Basic stats: COMPLETE Column stats: COMPLETE
TableScan Vectorization:
native: true
Select Operator
expressions: ss_item_sk (type: bigint), ss_ext_sales_price (type: decimal(7,2)), ss_sold_date_sk (type: bigint)
outputColumnNames: _col0, _col1, _col2
Select Vectorization:
className: VectorSelectOperator
native: true
projectedOutputColumnNums: [1, 14, 22]
Statistics: Num rows: 82510879939 Data size: 10343396725952 Basic stats: COMPLETE Column stats: COMPLETE
Map Join Operator
condition map:
Inner Join 0 to 1
keys:"
0 _col2 (type: bigint)
1 _col0 (type: bigint)
Map Join Vectorization:
bigTableKeyColumns: 22:bigint
className: VectorMapJoinInnerBigOnlyLongOperator
input vertices:
1 Map 4
SET hive.optimize.scan.probedecode=true

73. © 2020 Cloudera, Inc. All rights reserved. 73
⬢
–
–
–
–
–
DATA PLANE SERVICES
EDWS Data Analytics Studio (DAS) Management Services
COMPUTE CLUSTER
SHARED
SERVICES
Ranger
Atlas
Metastore
Tiller API Server
DAS Web Service
Query Coordinators
Query Executors
Registry
Blobstore
Indexer
RDBMS
Hive Server
Long-running kubernetes cluster
Inter-cluster communication Intra-cluster communication
Ingress Controller or
Load Balancer
Internal Service Endpoint for
ReplicaSet or StatefulSet
Ephemeral kubernetes cluster

74. © 2020 Cloudera, Inc. All rights reserved. 74
Data pruning
• Prune early → increase query efficiency
Semijoin reduction
• Reduce disk & network I/O
Lazy materialization
• Save CPU cycles and memory allocations
Conclusions

76. © 2020 Cloudera, Inc. All rights reserved. 76
References
1. Stocker, Konrad, et al. "Integrating semi-join-reducers into state-of-the-art
query processors." ICDE 2001
2. Abadi, Daniel J., Samuel R. Madden, and Nabil Hachem. "Column-stores vs.
row-stores: how different are they really?." SIGMOD 2008
3. Lang, Harald, et al. "Performance-optimal filtering: Bloom overtakes cuckoo at
high throughput." VLDB 2019