Databricks, profile picture
Uploaded byDatabricks
PDF, PPTX3,462 views

What’s New in the Upcoming Apache Spark 3.0

The document outlines the new features and enhancements in Apache Spark 3.0, including adaptive query execution, dynamic partition pruning, and improved performance for various workloads. It introduces richer APIs, an accelerator-aware scheduler, enhancements to built-in functions, and improvements in SQL compatibility and data sources. Additional features include support for JDK 11, Hadoop 3, and Hive 3.x, as well as significant monitoring and debugging capabilities in structured streaming.

Embed presentation

Download as PDF, PPTX
What’s New in the Upcoming Apache Spark 3.0 Xiao Li gatorsmile June 2020
• Engineering Manager at Databricks • Apache Spark Committer and PMC • Previously, IBM Master Inventor • Ph.D. in CISE at University of Florida • GitHub: gatorsmile About Me
Unified data analytics platform for accelerating innovation across data science, data engineering, and business analytics Original creators of popular data and machine learning open source projects Global company with 5,000 customers and 450+ partners
3400+ Resolved JIRAs in Spark 3.0 rc2
Adaptive Query Execution Dynamic Partition Pruning Query Compilation Speedup Join Hints Performance Richer APIs Accelerator-aware Scheduler Built-in Functions pandas UDF Enhancements DELETE/UPDATE/ MERGE in Catalyst Reserved Keywords Proleptic Gregorian Calendar ANSI Store Assignment Overflow Checking SQL Compatibility Built-in Data Sources Parquet/ORC Nested Column Pruning Parquet: Nested Column Filter Pushdown CSV Filter Pushdown New Binary Data Source Data Source V2 API + Catalog Support JDK 11 Support Hadoop 3 Support Hive 3.x Metastore Hive 2.3 Execution Extensibility and Ecosystem Structured Streaming UI DDL/DML Enhancements Observable Metrics Event Log Rollover Monitoring and Debuggability
Performance Achieve high performance for interactive, batch, streaming and ML workloads Adaptive Query Execution Dynamic Partition Pruning Query Compilation Speedup Join Hints
Performance Achieve high performance for interactive, batch, streaming and ML workloads Adaptive Query Execution Dynamic Partition Pruning Join Hints Query Compilation Speedup
Spark Catalyst Optimizer Spark 1.x, Rule Spark 2.x, Rule + Cost
Query Optimization in Spark 2.x § Missing statistics Expensive statistics collection § Out-of-date statistics Compute and storage separated § Suboptimal Heuristics Local § Misestimated costs Complex environments User-defined functions
Spark Catalyst Optimizer Spark 1.x, Rule Spark 2.x, Rule + Cost Spark 3.0, Rule + Cost + Runtime
adaptive planning Based on statistics of the finished plan nodes, re-optimize the execution plan of the remaining queries Adaptive Query Execution
Blog post: https://databricks.com/blog/2020/05/29/adaptive- query-execution-speeding-up-spark-sql-at-runtime.html Based on statistics of the finished plan nodes, re-optimize the execution plan of the remaining queries • Convert Sort Merge Join to Broadcast Hash Join • Shrink the number of reducers • Handle skew join Adaptive Query Execution
One of the Most Popular Performance Tuning Tips § Choose Broadcast Hash Join? § Increase “spark.sql.autoBroadcastJoinThreshold”? § Use “broadcast” hint? However ▪ Hard to tune ▪ Hard to maintain over time ▪ OOM…
Why Spark not Making the Best Choice Automatically? § Inaccurate/missing statistics; § File is compressed; columnar store; § Complex filters; black-box UDFs; § Complex query fragments…
Convert Sort Merge Join to Broadcast Hash Join Sort Merge Join Filter Scan Execute Shuffle Sort Scan Shuffle Sort Sort Merge Join Filter Scan Shuffle Sort Scan Shuffle Sort Stage 1 Stage 2 Stage 1 Stage 2 Estimate size: 100 MB Estimate size: 30 MB Actual size: 86 MB Actual size: 8 MB Optimize Broadcast Hash Join Filter Scan Shuffle Broadcast Scan Shuffle Stage 1 Stage 2 Actual size: 86 MB Actual size: 8 MB
One More Popular Performance Tuning Tip § Tuning spark.sql.shuffle.partitions § Default magic number: 200 !?! However § Too small: GC pressure; disk spilling § Too large: Inefficient I/O; scheduler pressure ▪ Hard to tune over the whole query plan ▪ Hard to maintain over time
Dynamically Coalesce Shuffle Partitions Filter Scan Execute Shuffle (50 part.) Sort Stage 1 OptimizeFilter Scan Shuffle (50 part.) Sort Stage 1 Filter Scan Shuffle (50 part.) Sort Stage 1 Coalesce (5 part.) ▪ Set the initial partition number high to accommodate the largest data size of the entire query execution ▪ Automatically coalesce partitions if needed after each query stage
Another Popular Performance Tuning Tip § Symptoms of data skew § Frozen/long-running tasks § Disk spilling § Low resource utilization in most nodes § OOM § Various ways § Find the skew values and rewrite the queries § Adding extra skew keys…
TABLE A - MAP 1 TABLE A - MAP 2 TABLE A - MAP 3 PART. A0 PART. A1 PART. A2 PART. A3 PART. B0 PART. B1 PART. B2 PART. B3 TABLE B - MAP 1 TABLE B - MAP 2 JOIN Data Skew in Sort Merge Join
Dynamically Optimize Skew Joins Sort Merge Join Filter Scan Execute Shuffle Sort Scan Shuffle Sort Sort Merge Join Filter Scan Shuffle Sort Scan Shuffle Sort Stage 1 Stage 2 Stage 1 Stage 2 Optimize Sort Merge Join Filter Scan Shuffle Sort Scan Shuffle Sort Stage 1 Stage 2 Skew Reader Skew Reader • Detect skew from partition sizes using runtime statistics • Split skew partitions into smaller sub-partitions
Skew-optimized sort merge join -- with skew shuffle reader: A0 - S2 TABLE A - MAP 1 B0 TABLE A - MAP 2 TABLE A - MAP 3 Split A0 PART. A1 PART. A2 PART. A3 PART. B1 PART. B2 PART. B3 TABLE B - MAP 1 TABLE B - MAP 2 A0 - S1 A0 - S0 B0 B0 Duplicate B0 JOIN Dynamically Optimize Skew Joins
Adaptive Query Execution
Performance Achieve high performance for interactive, batch, streaming and ML workloads Adaptive Query Execution Dynamic Partition Pruning Join Hints Query Compilation Speedup
Dynamic Partition Pruning • Avoid partition scanning based on the query results of the other query fragments. • Important for star-schema queries. • Significant speedup in TPC-DS.
Dynamic Partition Pruning 60 / 102 TPC-DS queries: a speedup between 2x and 18x
t1: a large fact table with many partitions t2.id < 2 t2: a dimension table with a filter SELECT t1.id, t2.pKey FROM t1 JOIN t2 ON t1.pKey = t2.pKey AND t2.id < 2 t1.pKey = t2.pKey Dynamic Partition Pruning Project Join Filter Scan Scan Optimize
t1: a large fact table with many partitions t2.id < 2 t2: a dimension table with a filter SELECT t1.id, t2.pKey FROM t1 JOIN t2 ON t1.pKey = t2.pKey AND t2.id < 2 t1.pKey = t2.pKey Dynamic Partition Pruning Project Join Filter Scan Scan Optimize Scan all the partitions of t1 Filter pushdown t1.pkey IN ( SELECT t2.pKey FROM t2 WHERE t2.id < 2) t2.id < 2 Project Join Filter + Scan Filter Optimize Scan t1.pKey = t2.pKey
Dynamic Partition Pruning Scan all the partitions of t1 t2.id < 2 Project Join Filter + Scan Filter Scan t1.pKey = t2.pKey Scan the required partitions of t2 t1.pKey in DPPFilterResult
Dynamic Partition Pruning Scan all the partitions of t1 t2.id < 2 Project Join Filter + Scan Filter Scan t1.pKey = t2.pKey Scan the required partitions of t2 t1.pKey in DPPFilterResultOptimize Scan the required partitions of t1 t2.id < 2 Project Join Filter + Scan Filter + Scan Scan the required partitions of t2 t1.pKey in DPPFilterResult
Dynamic Partition Pruning Scan all the partitions of t1 t2.id < 2 Project Join Filter + Scan Filter Scan t1.pKey = t2.pKey Scan the required partitions of t2 t1.pKey in DPPFilterResultOptimize Scan the required partitions of t1 t2.id < 2 Project Join Filter + Scan Filter + Scan Scan the required partitions of t2 t1.pKey in DPPFilterResult 90+% less file scan33 X faster Optimize
Performance Achieve high performance for interactive, batch, streaming and ML workloads Adaptive Query Execution Dynamic Partition Pruning Join Hints Query Compilation Speedup
Optimizer Hints § Join hints influence optimizer to choose the join strategies § Broadcast hash join § Sort-merge join § Shuffle hash join § Shuffle nested loop join § Should be used with extreme caution. § Difficult to manage over time. NEW NEW NEW
§ Broadcast Hash Join SELECT /*+ BROADCAST(a) */ id FROM a JOIN b ON a.key = b.key § Sort-Merge Join SELECT /*+ MERGE(a, b) */ id FROM a JOIN b ON a.key = b.key § Shuffle Hash Join SELECT /*+ SHUFFLE_HASH(a, b) */ id FROM a JOIN b ON a.key = b.key § Shuffle Nested Loop Join SELECT /*+ SHUFFLE_REPLICATE_NL(a, b) */ id FROM a JOIN b How to Use Join Hints?
Broadcast Hash Join Requires one side to be small. No shuffle, no sort, very fast. Sort-Merge Join Robust. Can handle any data size. Needs to shuffle and sort data, slower in most cases when the table size is small. Shuffle Hash Join Needs to shuffle data but no sort. Can handle large tables, but will OOM too if data is skewed. Shuffle Nested Loop Join Doesn’t require join keys.
Enable new use cases and simplify the Spark application development Accelerator-aware Scheduler Built-in Functions pandas UDF enhancements DELETE/UPDATE/ MERGE in Catalyst Richer APIs
Enable new use cases and simplify the Spark application development Accelerator-aware Scheduler Built-in Functions pandas UDF enhancements DELETE/UPDATE/ MERGE in Catalyst Richer APIs
Accelerator-aware Scheduling § Widely used for accelerating special workloads, e.g., deep learning and signal processing. § Supports Standalone, YARN and K8S. § Supports GPU now, FPGA, TPU, etc. in the future. § Needs to specify required resources by configs § Application level. Will support job/stage/task level in the future.
Enable new use cases and simplify the Spark application development Richer APIs Accelerator-aware Scheduler Built-in Functions pandas UDF enhancements DELETE/UPDATE/ MERGE in Catalyst
32 New Built-in Functions
Enable new use cases and simplify the Spark application development Richer APIs Accelerator-aware Scheduler Built-in Functions pandas UDF enhancements DELETE/UPDATE/ MERGE in Catalyst
Python UDF for SQL Python lambda functions for RDDs Session- specific Python UDF JAVA UDF in Python API New Pandas UDF Python Type Hints Py UDF V 3.0V 0.7 V 1.2 2013 2015 20182014 2019/20202016 2017 V 2.0 V 2.1 V 2.3/2.4
Scalar Pandas UDF [pandas.Series to pandas.Series] SPARK 2.3 SPARK 3.0 Python Type Hints
Grouped Map Pandas Function API [pandas.DataFrame to pandas.DataFrame] SPARK 2.3 SPARK 3.0 Python Type Hints
Grouped Aggregate Pandas UDF [pandas.Series to Scalar] SPARK 2.4 SPARK 3.0 Python Type Hints
New Pandas UDF Types
Map Pandas UDF Cogrouped Map Pandas UDF New Pandas Function APIs
Make monitoring and debugging Spark applications more comprehensive and stable Structured Streaming UI DDL/DML Enhancements Observable Metrics Event Log Rollover Monitoring and Debuggability
Make monitoring and debugging Spark applications more comprehensive and stable Monitoring and Debuggability Structured Streaming UI DDL/DML Enhancements Observable Metrics Event Log Rollover
Structured Streaming UI
Make monitoring and debugging Spark applications more comprehensive and stable Monitoring and Debuggability Structured Streaming UI DDL/DML Enhancements Observable Metrics Event Log Rollover
New Command EXPLAIN FORMATTED *(1) Project [key#5, val#6] +- *(1) Filter (isnotnull(key#5) AND (key#5 = Subquery scalar-subquery#15, [id=#113])) : +- Subquery scalar-subquery#15, [id=#113] : +- *(2) HashAggregate(keys=[], functions=[max(key#21)]) : +- Exchange SinglePartition, true, [id=#109] : +- *(1) HashAggregate(keys=[], functions=[partial_max(key#21)]) : +- *(1) Project [key#21] : +- *(1) Filter (isnotnull(val#22) AND (val#22 > 5)) : +- *(1) ColumnarToRow : +- FileScan parquet default.tab2[key#21,val#22] Batched: true, DataFilters: [isnotnull(val#22), (val#22 > 5)], Format: Parquet, Location: InMemoryFileIndex[file:/user/hive/warehouse/tab2], PartitionFilters: [], PushedFilters: [IsNotNull(val), GreaterThan(val,5)], ReadSchema: struct<key:int,val:int> +- *(1) ColumnarToRow +- FileScan parquet default.tab1[key#5,val#6] Batched: true, DataFilters: [isnotnull(key#5)], Format: Parquet, Location: InMemoryFileIndex[file:/user/hive/warehouse/tab1], PartitionFilters: [], PushedFilters: [IsNotNull(key)], ReadSchema: struct<key:int,val:int> EXPLAIN FORMATTED SELECT * FROM tab1 WHERE key = (SELECT max(key) FROM tab2 WHERE val > 5
* Project (4) +- * Filter (3) +- * ColumnarToRow (2) +- Scan parquet default.tab1 (1) (1) Scan parquet default.tab1 Output [2]: [key#5, val#6] Batched: true Location: InMemoryFileIndex [file:/user/hive/warehouse/tab1] PushedFilters: [IsNotNull(key)] ReadSchema: struct<key:int,val:int> (2) ColumnarToRow [codegen id : 1] Input [2]: [key#5, val#6] (3) Filter [codegen id : 1] Input [2]: [key#5, val#6] Condition : (isnotnull(key#5) AND (key#5 = Subquery scalar-subquery#27, [id=#164])) (4) Project [codegen id : 1] Output [2]: [key#5, val#6] Input [2]: [key#5, val#6] EXPLAIN FORMATTED SELECT * FROM tab1 WHERE key = (SELECT max(key) FROM tab2 WHERE val > 5
(5) Scan parquet default.tab2 Output [2]: [key#21, val#22] Batched: true Location: InMemoryFileIndex [file:/user/hive/warehouse/tab2] PushedFilters: [IsNotNull(val), GreaterThan(val,5)] ReadSchema: struct<key:int,val:int> (6) ColumnarToRow [codegen id : 1] Input [2]: [key#21, val#22] (7) Filter [codegen id : 1] Input [2]: [key#21, val#22] Condition : (isnotnull(val#22) AND (val#22 > 5)) ===== Subqueries ===== Subquery:1 Hosting operator id = 3 Hosting Expression = Subquery scalar-subquery#27, [id=#164] * HashAggregate (11) +- Exchange (10) +- * HashAggregate (9) +- * Project (8) +- * Filter (7) +- * ColumnarToRow (6) +- Scan parquet default.tab2 (5) (8) Project [codegen id : 1] Output [1]: [key#21] Input [2]: [key#21, val#22] (9) HashAggregate [codegen id : 1] Input [1]: [key#21] Keys: [] Functions [1]: [partial_max(key#21)] Aggregate Attributes [1]: [max#35] Results [1]: [max#36] (10) Exchange Input [1]: [max#36] Arguments: SinglePartition, true, [id=#160] (11) HashAggregate [codegen id : 2] Input [1]: [max#36] Keys: [] Functions [1]: [max(key#21)] Aggregate Attributes [1]: [max(key#21)#33] Results [1]: [max(key#21)#33 AS max(key)#34]
Reduce the time and complexity of enabling applications that were written for other relational database products to run in Spark SQL. Reserved Keywords in Parser Proleptic Gregorian Calendar ANSI Store Assignment Overflow Checking SQL Compatibility
Enhance the performance and functionalities of the built-in data sources Parquet/ORC Nested Column Pruning Parquet: Nested Column Filter Pushdown New Binary Data Source CSV Filter Pushdown Built-in Data Sources
Improve the plug-in interface and extend the deployment environments Data Source V2 API + Catalog Support Hive 3.x Metastore Hive 2.3 Execution Hadoop 3 Support JDK 11 Support Extensibility and Ecosystem
Spark 3.0 Builds • Only builds with Scala 2.12 • Deprecates Python 2 (already EOL) • Can build with various Hadoop/Hive versions – Hadoop 2.7 + Hive 1.2 – Hadoop 2.7 + Hive 2.3 (supports Java 11) [Default] – Hadoop 3.2 + Hive 2.3 (supports Java 11) • Supports the following Hive metastore versions: – "0.12", "0.13", "0.14", "1.0", "1.1", "1.2", "2.0", "2.1", "2.2", "2.3", "3.0", "3.1"
32 New Built-in Functions § map Documentation • Web UI • SQL reference • Migration guide • Semantic versioning guidelines
Adaptive Query Execution Dynamic Partition Pruning Query Compilation Speedup Join Hints Performance Richer APIs Accelerator-aware Scheduler Built-in Functions pandas UDF Enhancements DELETE/UPDATE/ MERGE in Catalyst Reserved Keywords Proleptic Gregorian Calendar ANSI Store Assignment Overflow Checking SQL Compatibility Built-in Data Sources Parquet/ORC Nested Column Pruning Parquet: Nested Column Filter Pushdown CSV Filter Pushdown New Binary Data Source Data Source V2 API + Catalog Support JDK 11 Support Hadoop 3 Support Hive 3.x Metastore Hive 2.3 Execution Extensibility and Ecosystem Structured Streaming UI DDL/DML Enhancements Observable Metrics Event Log Rollover Monitoring and Debuggability
Try Databricks Runtime 7.0 Beta For Free https://databricks .com/try- databricks
Thank you for your contributions!
THE VIRTUAL EVENT FOR DATA TEAMS ● Extended to 5 days with over 200 sessions ● 4x the pre-conference training ● Keynotes by visionaries and thought leaders June 22-26 | Organized by

More Related Content

PDF
Parquet performance tuning: the missing guide
byRyan Blue
 
PDF
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...
byDatabricks
 
PDF
Apache Spark Core—Deep Dive—Proper Optimization
byDatabricks
 
PDF
Making Apache Spark Better with Delta Lake
byDatabricks
 
PDF
Dynamic Partition Pruning in Apache Spark
byDatabricks
 
PPT
Apache Spark Introduction and Resilient Distributed Dataset basics and deep dive
bySachin Aggarwal
 
PDF
Efficient Data Storage for Analytics with Apache Parquet 2.0
byCloudera, Inc.
 
PDF
Introduction to Spark with Python
byGokhan Atil
 
Parquet performance tuning: the missing guide
byRyan Blue
 
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...
byDatabricks
 
Apache Spark Core—Deep Dive—Proper Optimization
byDatabricks
 
Making Apache Spark Better with Delta Lake
byDatabricks
 
Dynamic Partition Pruning in Apache Spark
byDatabricks
 
Apache Spark Introduction and Resilient Distributed Dataset basics and deep dive
bySachin Aggarwal
 
Efficient Data Storage for Analytics with Apache Parquet 2.0
byCloudera, Inc.
 
Introduction to Spark with Python
byGokhan Atil
 

What's hot

PDF
Simplify CDC Pipeline with Spark Streaming SQL and Delta Lake
byDatabricks
 
PDF
Spark Performance Tuning .pdf
byAmit Raj
 
PPTX
TechEvent Databricks on Azure
byTrivadis
 
PPTX
Delta lake and the delta architecture
byAdam Doyle
 
PDF
Databricks Delta Lake and Its Benefits
byDatabricks
 
PDF
The Parquet Format and Performance Optimization Opportunities
byDatabricks
 
PDF
Introduction to Apache Spark
byAnastasios Skarlatidis
 
PDF
Cosco: An Efficient Facebook-Scale Shuffle Service
byDatabricks
 
PDF
Parquet Strata/Hadoop World, New York 2013
byJulien Le Dem
 
PPTX
Introduction to spark
byHome
 
PDF
Iceberg: A modern table format for big data (Strata NY 2018)
byRyan Blue
 
PDF
Batch Processing at Scale with Flink & Iceberg
byFlink Forward
 
PDF
Apache Spark Streaming in K8s with ArgoCD & Spark Operator
byDatabricks
 
PDF
Data engineering design patterns
byValdas Maksimavičius
 
PPTX
Data Lakehouse Symposium | Day 4
byDatabricks
 
PDF
Spark tuning
byGMO-Z.com Vietnam Lab Center
 
PDF
Iceberg: a fast table format for S3
byDataWorks Summit
 
PDF
Amazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud
byNoritaka Sekiyama
 
PPTX
Using Apache Arrow, Calcite, and Parquet to Build a Relational Cache
byDremio Corporation
 
PDF
Hive Bucketing in Apache Spark with Tejas Patil
byDatabricks
 
Simplify CDC Pipeline with Spark Streaming SQL and Delta Lake
byDatabricks
 
Spark Performance Tuning .pdf
byAmit Raj
 
TechEvent Databricks on Azure
byTrivadis
 
Delta lake and the delta architecture
byAdam Doyle
 
Databricks Delta Lake and Its Benefits
byDatabricks
 
The Parquet Format and Performance Optimization Opportunities
byDatabricks
 
Introduction to Apache Spark
byAnastasios Skarlatidis
 
Cosco: An Efficient Facebook-Scale Shuffle Service
byDatabricks
 
Parquet Strata/Hadoop World, New York 2013
byJulien Le Dem
 
Introduction to spark
byHome
 
Iceberg: A modern table format for big data (Strata NY 2018)
byRyan Blue
 
Batch Processing at Scale with Flink & Iceberg
byFlink Forward
 
Apache Spark Streaming in K8s with ArgoCD & Spark Operator
byDatabricks
 
Data engineering design patterns
byValdas Maksimavičius
 
Data Lakehouse Symposium | Day 4
byDatabricks
 
Spark tuning
byGMO-Z.com Vietnam Lab Center
 
Iceberg: a fast table format for S3
byDataWorks Summit
 
Amazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud
byNoritaka Sekiyama
 
Using Apache Arrow, Calcite, and Parquet to Build a Relational Cache
byDremio Corporation
 
Hive Bucketing in Apache Spark with Tejas Patil
byDatabricks
 

Similar to What’s New in the Upcoming Apache Spark 3.0

PDF
Deep Dive into the New Features of Apache Spark 3.0
byDatabricks
 
PDF
Adaptive Query Execution: Speeding Up Spark SQL at Runtime
byDatabricks
 
PDF
From Query Plan to Query Performance: Supercharging your Apache Spark Queries...
byDatabricks
 
PDF
Spark SQL Join Improvement at Facebook
byDatabricks
 
PDF
Cost-Based Optimizer in Apache Spark 2.2
byDatabricks
 
PDF
Spark + AI Summit recap jul16 2020
byGuido Oswald
 
PDF
Optimizations in Spark; RDD, DataFrame
byKnoldus Inc.
 
PDF
Apache Spark 3.0: Overview of What’s New and Why Care
byDatabricks
 
PDF
An Adaptive Execution Engine for Apache Spark with Carson Wang and Yucai Yu
byDatabricks
 
PDF
Cost-Based Optimizer in Apache Spark 2.2 Ron Hu, Sameer Agarwal, Wenchen Fan ...
byDatabricks
 
PDF
SQLDAY 2023 Chodkowski Adrian Databricks Performance Tuning
bySeeQuality.net
 
PPTX
Data Con LA 2018 - Applying Probabilistic Algorithms by Grant Kushida
byData Con LA
 
PDF
Shark SQL and Rich Analytics at Scale
byDataWorks Summit
 
PDF
Meetup talk
byArpit Tak
 
PDF
Spark SQL Adaptive Execution Unleashes The Power of Cluster in Large Scale wi...
byDatabricks
 
PDF
Deep Dive into Spark
byEric Xiao
 
PDF
Spark Streaming Tips for Devs and Ops
byFrancisco Pérez Paradas
 
PDF
Spark Streaming Tips for Devs and Ops by Fran perez y federico fernández
byJ On The Beach
 
PDF
The internals of Spark SQL Joins, Dmytro Popovich
bySigma Software
 
PDF
Beyond shuffling - Scala Days Berlin 2016
byHolden Karau
 
Deep Dive into the New Features of Apache Spark 3.0
byDatabricks
 
Adaptive Query Execution: Speeding Up Spark SQL at Runtime
byDatabricks
 
From Query Plan to Query Performance: Supercharging your Apache Spark Queries...
byDatabricks
 
Spark SQL Join Improvement at Facebook
byDatabricks
 
Cost-Based Optimizer in Apache Spark 2.2
byDatabricks
 
Spark + AI Summit recap jul16 2020
byGuido Oswald
 
Optimizations in Spark; RDD, DataFrame
byKnoldus Inc.
 
Apache Spark 3.0: Overview of What’s New and Why Care
byDatabricks
 
An Adaptive Execution Engine for Apache Spark with Carson Wang and Yucai Yu
byDatabricks
 
Cost-Based Optimizer in Apache Spark 2.2 Ron Hu, Sameer Agarwal, Wenchen Fan ...
byDatabricks
 
SQLDAY 2023 Chodkowski Adrian Databricks Performance Tuning
bySeeQuality.net
 
Data Con LA 2018 - Applying Probabilistic Algorithms by Grant Kushida
byData Con LA
 
Shark SQL and Rich Analytics at Scale
byDataWorks Summit
 
Meetup talk
byArpit Tak
 
Spark SQL Adaptive Execution Unleashes The Power of Cluster in Large Scale wi...
byDatabricks
 
Deep Dive into Spark
byEric Xiao
 
Spark Streaming Tips for Devs and Ops
byFrancisco Pérez Paradas
 
Spark Streaming Tips for Devs and Ops by Fran perez y federico fernández
byJ On The Beach
 
The internals of Spark SQL Joins, Dmytro Popovich
bySigma Software
 
Beyond shuffling - Scala Days Berlin 2016
byHolden Karau
 

More from Databricks

PDF
5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop
byDatabricks
 
PPTX
DW Migration Webinar-March 2022.pptx
byDatabricks
 
PDF
Stage Level Scheduling Improving Big Data and AI Integration
byDatabricks
 
PDF
Massive Data Processing in Adobe Using Delta Lake
byDatabricks
 
PDF
Learn to Use Databricks for Data Science
byDatabricks
 
PDF
Scaling your Data Pipelines with Apache Spark on Kubernetes
byDatabricks
 
PDF
Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink
byDatabricks
 
PPT
Data Lakehouse Symposium | Day 1 | Part 2
byDatabricks
 
PDF
Processing Large Datasets for ADAS Applications using Apache Spark
byDatabricks
 
PDF
Simplify Data Conversion from Spark to TensorFlow and PyTorch
byDatabricks
 
PPTX
Data Lakehouse Symposium | Day 1 | Part 1
byDatabricks
 
PDF
Re-imagine Data Monitoring with whylogs and Spark
byDatabricks
 
PDF
Democratizing Data Quality Through a Centralized Platform
byDatabricks
 
PDF
Sawtooth Windows for Feature Aggregations
byDatabricks
 
PDF
Scaling and Unifying SciKit Learn and Apache Spark Pipelines
byDatabricks
 
PDF
Raven: End-to-end Optimization of ML Prediction Queries
byDatabricks
 
PDF
The Function, the Context, and the Data—Enabling ML Ops at Stitch Fix
byDatabricks
 
PPTX
Data Lakehouse Symposium | Day 2
byDatabricks
 
PDF
Why APM Is Not the Same As ML Monitoring
byDatabricks
 
PDF
Machine Learning CI/CD for Email Attack Detection
byDatabricks
 
5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop
byDatabricks
 
DW Migration Webinar-March 2022.pptx
byDatabricks
 
Stage Level Scheduling Improving Big Data and AI Integration
byDatabricks
 
Massive Data Processing in Adobe Using Delta Lake
byDatabricks
 
Learn to Use Databricks for Data Science
byDatabricks
 
Scaling your Data Pipelines with Apache Spark on Kubernetes
byDatabricks
 
Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink
byDatabricks
 
Data Lakehouse Symposium | Day 1 | Part 2
byDatabricks
 
Processing Large Datasets for ADAS Applications using Apache Spark
byDatabricks
 
Simplify Data Conversion from Spark to TensorFlow and PyTorch
byDatabricks
 
Data Lakehouse Symposium | Day 1 | Part 1
byDatabricks
 
Re-imagine Data Monitoring with whylogs and Spark
byDatabricks
 
Democratizing Data Quality Through a Centralized Platform
byDatabricks
 
Sawtooth Windows for Feature Aggregations
byDatabricks
 
Scaling and Unifying SciKit Learn and Apache Spark Pipelines
byDatabricks
 
Raven: End-to-end Optimization of ML Prediction Queries
byDatabricks
 
The Function, the Context, and the Data—Enabling ML Ops at Stitch Fix
byDatabricks
 
Data Lakehouse Symposium | Day 2
byDatabricks
 
Why APM Is Not the Same As ML Monitoring
byDatabricks
 
Machine Learning CI/CD for Email Attack Detection
byDatabricks
 

Recently uploaded

PPTX
First introduction to pipeline pilot a datascience software
byJamesImmanuel4
 
PDF
Introducing Cohesity NetBackup (2).pdf
byNestorNavarro20
 
PPTX
Presentation 1-PUM Singapore 2015- 2D.pptx
byWilliamChong71
 
PPTX
[DSC Europe 25] Petar Zivanov - AI meets documents From chatbots to AI-powere...
byDataScienceConferenc1
 
PPTX
software define network authentication .pptx
byAhmedGouda93
 
PPTX
Time serise forcasting of non-durble consumer goods industry
byvkumaranand89
 
PPTX
essential in computer science structure and other
byhncseven7
 
PPTX
Betsi- Zero-Trust Security Securing the Modern Enterprise.pptx
bymershaabdisa
 
PPT
Perceptron and Activation Function in deep learning
byShrideviS7
 
PPTX
Customer Segmentation: Seeing the Trees and the Forest Simultaneously - V2
bySK Palu
 
PPTX
Two Factor Authentication for CSE Students.pptx
byfizarcse
 
PDF
[DSC Europe 25] Marija Vlajkovic & Andrea Radonjanin - Integration of AI tool...
byDataScienceConferenc1
 
PPTX
Understanding Double Bar Graphs for grade 5
byJhonnalynMaeSia1
 
PDF
20251202 Frank_Melis Vittal_Prabhu Platform_Talk v3b.pdf
byhome
 
PPTX
Test Planner for allen 2025-26 samarth batch
byamaanahmed1312
 
PPTX
[DSC Europe 25] Dragan Vucic - Building the Learning Organization - How AI Tr...
byDataScienceConferenc1
 
PPT
How To Write A Scientific Paper- IMRAD pattern
bydrmeenajain
 
PPTX
ELECTRICITY SECTOR IN PHILIPHINE .pptx
byahmadnabil090406
 
PPTX
1. Prepare for the application of health survey.pptx
byDegaregeNibret
 
PDF
Kate Scrivens-OECD-Community well-being and climate risks.pdf
byStatsCommunications
 
First introduction to pipeline pilot a datascience software
byJamesImmanuel4
 
Introducing Cohesity NetBackup (2).pdf
byNestorNavarro20
 
Presentation 1-PUM Singapore 2015- 2D.pptx
byWilliamChong71
 
[DSC Europe 25] Petar Zivanov - AI meets documents From chatbots to AI-powere...
byDataScienceConferenc1
 
software define network authentication .pptx
byAhmedGouda93
 
Time serise forcasting of non-durble consumer goods industry
byvkumaranand89
 
essential in computer science structure and other
byhncseven7
 
Betsi- Zero-Trust Security Securing the Modern Enterprise.pptx
bymershaabdisa
 
Perceptron and Activation Function in deep learning
byShrideviS7
 
Customer Segmentation: Seeing the Trees and the Forest Simultaneously - V2
bySK Palu
 
Two Factor Authentication for CSE Students.pptx
byfizarcse
 
[DSC Europe 25] Marija Vlajkovic & Andrea Radonjanin - Integration of AI tool...
byDataScienceConferenc1
 
Understanding Double Bar Graphs for grade 5
byJhonnalynMaeSia1
 
20251202 Frank_Melis Vittal_Prabhu Platform_Talk v3b.pdf
byhome
 
Test Planner for allen 2025-26 samarth batch
byamaanahmed1312
 
[DSC Europe 25] Dragan Vucic - Building the Learning Organization - How AI Tr...
byDataScienceConferenc1
 
How To Write A Scientific Paper- IMRAD pattern
bydrmeenajain
 
ELECTRICITY SECTOR IN PHILIPHINE .pptx
byahmadnabil090406
 
1. Prepare for the application of health survey.pptx
byDegaregeNibret
 
Kate Scrivens-OECD-Community well-being and climate risks.pdf
byStatsCommunications
 
In this document
Powered by AI

Overview of Apache Spark 3.0 features and speaker's background as an engineering manager.

Description of Apache Spark as a unified data analytics platform with a global presence and significant customer base.

Highlights over 3400 resolved JIRAs in Spark 3.0 release candidate with various new features including query execution improvements.

Focus on achieving high performance for various workloads through adaptive query execution and optimizations.

Comparison of Spark Catalyst Optimizer evolution from rules-based to cost-based with runtime enhancements introduced in version 3.0.

Adaptive planning and execution by re-optimizing query execution based on statistics to improve performance.

Popular performance tuning tips including handling broadcast joins, shuffle partition tuning, and remedies for data skew.

Strategies to optimize joins in the presence of data skew using dynamic join optimization techniques.

Dynamic partition pruning effectively reduces file scans and enhances query performance in Spark applications.

Discussion on various join hints influencing optimizer strategies and details on enabling new use cases through built-in functions.

Introduction of new Pandas UDF types and enhancements such as type hints to improve usability in Spark SQL.

Improvements in monitoring capabilities for Spark applications, including a new structured streaming UI.

Enhancements aimed at improving SQL compatibility with new reserved keywords and performance in built-in data sources.

Invitation to try Databricks and appreciation for contributions, along with details about a virtual event.

What’s New in the Upcoming Apache Spark 3.0

  • 1.
    What’s New inthe Upcoming Apache Spark 3.0 Xiao Li gatorsmile June 2020
  • 2.
    • Engineering Managerat Databricks • Apache Spark Committer and PMC • Previously, IBM Master Inventor • Ph.D. in CISE at University of Florida • GitHub: gatorsmile About Me
  • 3.
    Unified data analyticsplatform for accelerating innovation across data science, data engineering, and business analytics Original creators of popular data and machine learning open source projects Global company with 5,000 customers and 450+ partners
  • 4.
  • 5.
    Adaptive Query Execution Dynamic Partition Pruning QueryCompilation Speedup Join Hints Performance Richer APIs Accelerator-aware Scheduler Built-in Functions pandas UDF Enhancements DELETE/UPDATE/ MERGE in Catalyst Reserved Keywords Proleptic Gregorian Calendar ANSI Store Assignment Overflow Checking SQL Compatibility Built-in Data Sources Parquet/ORC Nested Column Pruning Parquet: Nested Column Filter Pushdown CSV Filter Pushdown New Binary Data Source Data Source V2 API + Catalog Support JDK 11 Support Hadoop 3 Support Hive 3.x Metastore Hive 2.3 Execution Extensibility and Ecosystem Structured Streaming UI DDL/DML Enhancements Observable Metrics Event Log Rollover Monitoring and Debuggability
  • 6.
    Performance Achieve high performancefor interactive, batch, streaming and ML workloads Adaptive Query Execution Dynamic Partition Pruning Query Compilation Speedup Join Hints
  • 7.
    Performance Achieve high performancefor interactive, batch, streaming and ML workloads Adaptive Query Execution Dynamic Partition Pruning Join Hints Query Compilation Speedup
  • 8.
    Spark Catalyst Optimizer Spark1.x, Rule Spark 2.x, Rule + Cost
  • 9.
    Query Optimization inSpark 2.x § Missing statistics Expensive statistics collection § Out-of-date statistics Compute and storage separated § Suboptimal Heuristics Local § Misestimated costs Complex environments User-defined functions
  • 10.
    Spark Catalyst Optimizer Spark1.x, Rule Spark 2.x, Rule + Cost Spark 3.0, Rule + Cost + Runtime
  • 11.
    adaptive planning Based onstatistics of the finished plan nodes, re-optimize the execution plan of the remaining queries Adaptive Query Execution
  • 12.
    Blog post: https://databricks.com/blog/2020/05/29/adaptive- query-execution-speeding-up-spark-sql-at-runtime.html Basedon statistics of the finished plan nodes, re-optimize the execution plan of the remaining queries • Convert Sort Merge Join to Broadcast Hash Join • Shrink the number of reducers • Handle skew join Adaptive Query Execution
  • 13.
    One of theMost Popular Performance Tuning Tips § Choose Broadcast Hash Join? § Increase “spark.sql.autoBroadcastJoinThreshold”? § Use “broadcast” hint? However ▪ Hard to tune ▪ Hard to maintain over time ▪ OOM…
  • 14.
    Why Spark notMaking the Best Choice Automatically? § Inaccurate/missing statistics; § File is compressed; columnar store; § Complex filters; black-box UDFs; § Complex query fragments…
  • 15.
    Convert Sort MergeJoin to Broadcast Hash Join Sort Merge Join Filter Scan Execute Shuffle Sort Scan Shuffle Sort Sort Merge Join Filter Scan Shuffle Sort Scan Shuffle Sort Stage 1 Stage 2 Stage 1 Stage 2 Estimate size: 100 MB Estimate size: 30 MB Actual size: 86 MB Actual size: 8 MB Optimize Broadcast Hash Join Filter Scan Shuffle Broadcast Scan Shuffle Stage 1 Stage 2 Actual size: 86 MB Actual size: 8 MB
  • 16.
    One More PopularPerformance Tuning Tip § Tuning spark.sql.shuffle.partitions § Default magic number: 200 !?! However § Too small: GC pressure; disk spilling § Too large: Inefficient I/O; scheduler pressure ▪ Hard to tune over the whole query plan ▪ Hard to maintain over time
  • 17.
    Dynamically Coalesce ShufflePartitions Filter Scan Execute Shuffle (50 part.) Sort Stage 1 OptimizeFilter Scan Shuffle (50 part.) Sort Stage 1 Filter Scan Shuffle (50 part.) Sort Stage 1 Coalesce (5 part.) ▪ Set the initial partition number high to accommodate the largest data size of the entire query execution ▪ Automatically coalesce partitions if needed after each query stage
  • 18.
    Another Popular PerformanceTuning Tip § Symptoms of data skew § Frozen/long-running tasks § Disk spilling § Low resource utilization in most nodes § OOM § Various ways § Find the skew values and rewrite the queries § Adding extra skew keys…
  • 19.
    TABLE A -MAP 1 TABLE A - MAP 2 TABLE A - MAP 3 PART. A0 PART. A1 PART. A2 PART. A3 PART. B0 PART. B1 PART. B2 PART. B3 TABLE B - MAP 1 TABLE B - MAP 2 JOIN Data Skew in Sort Merge Join
  • 20.
    Dynamically Optimize SkewJoins Sort Merge Join Filter Scan Execute Shuffle Sort Scan Shuffle Sort Sort Merge Join Filter Scan Shuffle Sort Scan Shuffle Sort Stage 1 Stage 2 Stage 1 Stage 2 Optimize Sort Merge Join Filter Scan Shuffle Sort Scan Shuffle Sort Stage 1 Stage 2 Skew Reader Skew Reader • Detect skew from partition sizes using runtime statistics • Split skew partitions into smaller sub-partitions
  • 21.
    Skew-optimized sort mergejoin -- with skew shuffle reader: A0 - S2 TABLE A - MAP 1 B0 TABLE A - MAP 2 TABLE A - MAP 3 Split A0 PART. A1 PART. A2 PART. A3 PART. B1 PART. B2 PART. B3 TABLE B - MAP 1 TABLE B - MAP 2 A0 - S1 A0 - S0 B0 B0 Duplicate B0 JOIN Dynamically Optimize Skew Joins
  • 22.
  • 23.
    Performance Achieve high performancefor interactive, batch, streaming and ML workloads Adaptive Query Execution Dynamic Partition Pruning Join Hints Query Compilation Speedup
  • 24.
    Dynamic Partition Pruning •Avoid partition scanning based on the query results of the other query fragments. • Important for star-schema queries. • Significant speedup in TPC-DS.
  • 25.
    Dynamic Partition Pruning 60/ 102 TPC-DS queries: a speedup between 2x and 18x
  • 26.
    t1: a large facttable with many partitions t2.id < 2 t2: a dimension table with a filter SELECT t1.id, t2.pKey FROM t1 JOIN t2 ON t1.pKey = t2.pKey AND t2.id < 2 t1.pKey = t2.pKey Dynamic Partition Pruning Project Join Filter Scan Scan Optimize
  • 27.
    t1: a large facttable with many partitions t2.id < 2 t2: a dimension table with a filter SELECT t1.id, t2.pKey FROM t1 JOIN t2 ON t1.pKey = t2.pKey AND t2.id < 2 t1.pKey = t2.pKey Dynamic Partition Pruning Project Join Filter Scan Scan Optimize Scan all the partitions of t1 Filter pushdown t1.pkey IN ( SELECT t2.pKey FROM t2 WHERE t2.id < 2) t2.id < 2 Project Join Filter + Scan Filter Optimize Scan t1.pKey = t2.pKey
  • 28.
    Dynamic Partition Pruning Scanall the partitions of t1 t2.id < 2 Project Join Filter + Scan Filter Scan t1.pKey = t2.pKey Scan the required partitions of t2 t1.pKey in DPPFilterResult
  • 29.
    Dynamic Partition Pruning Scanall the partitions of t1 t2.id < 2 Project Join Filter + Scan Filter Scan t1.pKey = t2.pKey Scan the required partitions of t2 t1.pKey in DPPFilterResultOptimize Scan the required partitions of t1 t2.id < 2 Project Join Filter + Scan Filter + Scan Scan the required partitions of t2 t1.pKey in DPPFilterResult
  • 30.
    Dynamic Partition Pruning Scanall the partitions of t1 t2.id < 2 Project Join Filter + Scan Filter Scan t1.pKey = t2.pKey Scan the required partitions of t2 t1.pKey in DPPFilterResultOptimize Scan the required partitions of t1 t2.id < 2 Project Join Filter + Scan Filter + Scan Scan the required partitions of t2 t1.pKey in DPPFilterResult 90+% less file scan33 X faster Optimize
  • 31.
    Performance Achieve high performancefor interactive, batch, streaming and ML workloads Adaptive Query Execution Dynamic Partition Pruning Join Hints Query Compilation Speedup
  • 32.
    Optimizer Hints § Joinhints influence optimizer to choose the join strategies § Broadcast hash join § Sort-merge join § Shuffle hash join § Shuffle nested loop join § Should be used with extreme caution. § Difficult to manage over time. NEW NEW NEW
  • 33.
    § Broadcast HashJoin SELECT /*+ BROADCAST(a) */ id FROM a JOIN b ON a.key = b.key § Sort-Merge Join SELECT /*+ MERGE(a, b) */ id FROM a JOIN b ON a.key = b.key § Shuffle Hash Join SELECT /*+ SHUFFLE_HASH(a, b) */ id FROM a JOIN b ON a.key = b.key § Shuffle Nested Loop Join SELECT /*+ SHUFFLE_REPLICATE_NL(a, b) */ id FROM a JOIN b How to Use Join Hints?
  • 34.
    Broadcast Hash Join Requiresone side to be small. No shuffle, no sort, very fast. Sort-Merge Join Robust. Can handle any data size. Needs to shuffle and sort data, slower in most cases when the table size is small. Shuffle Hash Join Needs to shuffle data but no sort. Can handle large tables, but will OOM too if data is skewed. Shuffle Nested Loop Join Doesn’t require join keys.
  • 35.
    Enable new usecases and simplify the Spark application development Accelerator-aware Scheduler Built-in Functions pandas UDF enhancements DELETE/UPDATE/ MERGE in Catalyst Richer APIs
  • 36.
    Enable new usecases and simplify the Spark application development Accelerator-aware Scheduler Built-in Functions pandas UDF enhancements DELETE/UPDATE/ MERGE in Catalyst Richer APIs
  • 37.
    Accelerator-aware Scheduling § Widelyused for accelerating special workloads, e.g., deep learning and signal processing. § Supports Standalone, YARN and K8S. § Supports GPU now, FPGA, TPU, etc. in the future. § Needs to specify required resources by configs § Application level. Will support job/stage/task level in the future.
  • 38.
    Enable new usecases and simplify the Spark application development Richer APIs Accelerator-aware Scheduler Built-in Functions pandas UDF enhancements DELETE/UPDATE/ MERGE in Catalyst
  • 39.
    32 New Built-inFunctions
  • 40.
    Enable new usecases and simplify the Spark application development Richer APIs Accelerator-aware Scheduler Built-in Functions pandas UDF enhancements DELETE/UPDATE/ MERGE in Catalyst
  • 41.
    Python UDF for SQL Python lambda functionsfor RDDs Session- specific Python UDF JAVA UDF in Python API New Pandas UDF Python Type Hints Py UDF V 3.0V 0.7 V 1.2 2013 2015 20182014 2019/20202016 2017 V 2.0 V 2.1 V 2.3/2.4
  • 42.
    Scalar Pandas UDF [pandas.Seriesto pandas.Series] SPARK 2.3 SPARK 3.0 Python Type Hints
  • 43.
    Grouped Map PandasFunction API [pandas.DataFrame to pandas.DataFrame] SPARK 2.3 SPARK 3.0 Python Type Hints
  • 44.
    Grouped Aggregate PandasUDF [pandas.Series to Scalar] SPARK 2.4 SPARK 3.0 Python Type Hints
  • 45.
  • 46.
    Map Pandas UDF CogroupedMap Pandas UDF New Pandas Function APIs
  • 47.
    Make monitoring anddebugging Spark applications more comprehensive and stable Structured Streaming UI DDL/DML Enhancements Observable Metrics Event Log Rollover Monitoring and Debuggability
  • 48.
    Make monitoring anddebugging Spark applications more comprehensive and stable Monitoring and Debuggability Structured Streaming UI DDL/DML Enhancements Observable Metrics Event Log Rollover
  • 49.
  • 50.
    Make monitoring anddebugging Spark applications more comprehensive and stable Monitoring and Debuggability Structured Streaming UI DDL/DML Enhancements Observable Metrics Event Log Rollover
  • 51.
    New Command EXPLAINFORMATTED *(1) Project [key#5, val#6] +- *(1) Filter (isnotnull(key#5) AND (key#5 = Subquery scalar-subquery#15, [id=#113])) : +- Subquery scalar-subquery#15, [id=#113] : +- *(2) HashAggregate(keys=[], functions=[max(key#21)]) : +- Exchange SinglePartition, true, [id=#109] : +- *(1) HashAggregate(keys=[], functions=[partial_max(key#21)]) : +- *(1) Project [key#21] : +- *(1) Filter (isnotnull(val#22) AND (val#22 > 5)) : +- *(1) ColumnarToRow : +- FileScan parquet default.tab2[key#21,val#22] Batched: true, DataFilters: [isnotnull(val#22), (val#22 > 5)], Format: Parquet, Location: InMemoryFileIndex[file:/user/hive/warehouse/tab2], PartitionFilters: [], PushedFilters: [IsNotNull(val), GreaterThan(val,5)], ReadSchema: struct<key:int,val:int> +- *(1) ColumnarToRow +- FileScan parquet default.tab1[key#5,val#6] Batched: true, DataFilters: [isnotnull(key#5)], Format: Parquet, Location: InMemoryFileIndex[file:/user/hive/warehouse/tab1], PartitionFilters: [], PushedFilters: [IsNotNull(key)], ReadSchema: struct<key:int,val:int> EXPLAIN FORMATTED SELECT * FROM tab1 WHERE key = (SELECT max(key) FROM tab2 WHERE val > 5
  • 52.
    * Project (4) +-* Filter (3) +- * ColumnarToRow (2) +- Scan parquet default.tab1 (1) (1) Scan parquet default.tab1 Output [2]: [key#5, val#6] Batched: true Location: InMemoryFileIndex [file:/user/hive/warehouse/tab1] PushedFilters: [IsNotNull(key)] ReadSchema: struct<key:int,val:int> (2) ColumnarToRow [codegen id : 1] Input [2]: [key#5, val#6] (3) Filter [codegen id : 1] Input [2]: [key#5, val#6] Condition : (isnotnull(key#5) AND (key#5 = Subquery scalar-subquery#27, [id=#164])) (4) Project [codegen id : 1] Output [2]: [key#5, val#6] Input [2]: [key#5, val#6] EXPLAIN FORMATTED SELECT * FROM tab1 WHERE key = (SELECT max(key) FROM tab2 WHERE val > 5
  • 53.
    (5) Scan parquetdefault.tab2 Output [2]: [key#21, val#22] Batched: true Location: InMemoryFileIndex [file:/user/hive/warehouse/tab2] PushedFilters: [IsNotNull(val), GreaterThan(val,5)] ReadSchema: struct<key:int,val:int> (6) ColumnarToRow [codegen id : 1] Input [2]: [key#21, val#22] (7) Filter [codegen id : 1] Input [2]: [key#21, val#22] Condition : (isnotnull(val#22) AND (val#22 > 5)) ===== Subqueries ===== Subquery:1 Hosting operator id = 3 Hosting Expression = Subquery scalar-subquery#27, [id=#164] * HashAggregate (11) +- Exchange (10) +- * HashAggregate (9) +- * Project (8) +- * Filter (7) +- * ColumnarToRow (6) +- Scan parquet default.tab2 (5) (8) Project [codegen id : 1] Output [1]: [key#21] Input [2]: [key#21, val#22] (9) HashAggregate [codegen id : 1] Input [1]: [key#21] Keys: [] Functions [1]: [partial_max(key#21)] Aggregate Attributes [1]: [max#35] Results [1]: [max#36] (10) Exchange Input [1]: [max#36] Arguments: SinglePartition, true, [id=#160] (11) HashAggregate [codegen id : 2] Input [1]: [max#36] Keys: [] Functions [1]: [max(key#21)] Aggregate Attributes [1]: [max(key#21)#33] Results [1]: [max(key#21)#33 AS max(key)#34]
  • 54.
    Reduce the timeand complexity of enabling applications that were written for other relational database products to run in Spark SQL. Reserved Keywords in Parser Proleptic Gregorian Calendar ANSI Store Assignment Overflow Checking SQL Compatibility
  • 55.
    Enhance the performanceand functionalities of the built-in data sources Parquet/ORC Nested Column Pruning Parquet: Nested Column Filter Pushdown New Binary Data Source CSV Filter Pushdown Built-in Data Sources
  • 56.
    Improve the plug-ininterface and extend the deployment environments Data Source V2 API + Catalog Support Hive 3.x Metastore Hive 2.3 Execution Hadoop 3 Support JDK 11 Support Extensibility and Ecosystem
  • 57.
    Spark 3.0 Builds •Only builds with Scala 2.12 • Deprecates Python 2 (already EOL) • Can build with various Hadoop/Hive versions – Hadoop 2.7 + Hive 1.2 – Hadoop 2.7 + Hive 2.3 (supports Java 11) [Default] – Hadoop 3.2 + Hive 2.3 (supports Java 11) • Supports the following Hive metastore versions: – "0.12", "0.13", "0.14", "1.0", "1.1", "1.2", "2.0", "2.1", "2.2", "2.3", "3.0", "3.1"
  • 58.
    32 New Built-inFunctions § map Documentation • Web UI • SQL reference • Migration guide • Semantic versioning guidelines
  • 62.
    Adaptive Query Execution Dynamic Partition Pruning QueryCompilation Speedup Join Hints Performance Richer APIs Accelerator-aware Scheduler Built-in Functions pandas UDF Enhancements DELETE/UPDATE/ MERGE in Catalyst Reserved Keywords Proleptic Gregorian Calendar ANSI Store Assignment Overflow Checking SQL Compatibility Built-in Data Sources Parquet/ORC Nested Column Pruning Parquet: Nested Column Filter Pushdown CSV Filter Pushdown New Binary Data Source Data Source V2 API + Catalog Support JDK 11 Support Hadoop 3 Support Hive 3.x Metastore Hive 2.3 Execution Extensibility and Ecosystem Structured Streaming UI DDL/DML Enhancements Observable Metrics Event Log Rollover Monitoring and Debuggability
  • 63.
    Try Databricks Runtime 7.0Beta For Free https://databricks .com/try- databricks
  • 64.
    Thank you foryour contributions!
  • 65.
    THE VIRTUAL EVENT FORDATA TEAMS ● Extended to 5 days with over 200 sessions ● 4x the pre-conference training ● Keynotes by visionaries and thought leaders June 22-26 | Organized by