Databricks, profile picture
Uploaded byDatabricks
PDF, PPTX2,868 views

Deep Dive into the New Features of Apache Spark 3.0

The document presents an overview of the new features in Apache Spark 3.0, highlighting enhancements in performance through adaptive query execution, dynamic partition pruning, and optimized join strategies. It details improvements in APIs, support for newer technologies such as Java 11 and Hadoop 3, and introduces richer built-in functions and an accelerator-aware scheduler to enhance user experience. Other notable features include better monitoring and debugging capabilities, structured streaming UI enhancements, and advances in UDFs for improved functionality.

Embed presentation

Download as PDF, PPTX
Deep Dive into the New Features of Upcoming Apache Spark 3.0 Xiao Li gatorsmile June 2020 Wenchen Fan cloud-fan
• Open Source Team at • Apache Spark Committer and PMC About Us Xiao Li (Github: gatorsmile) Wenchen Fan (Github: cloud-fan)
Unified data analytics platform for accelerating innovation across data science, data engineering, and business analytics Original creators of popular data and machine learning open source projects Global company with 5,000 customers and 450+ partners
3400+ Resolved JIRAs in Spark 3.0 rc2
Adaptive Query Execution Dynamic Partition Pruning Query Compilation Speedup Join Hints Performance Richer APIs Accelerator-aware Scheduler Built-in Functions pandas UDF Enhancements DELETE/UPDATE/ MERGE in Catalyst Reserved Keywords Proleptic Gregorian Calendar ANSI Store Assignment Overflow Checking SQL Compatibility Built-in Data Sources Parquet/ORC Nested Column Pruning Parquet: Nested Column Filter Pushdown CSV Filter Pushdown New Binary Data Source Data Source V2 API + Catalog Support Java 11 Support Hadoop 3 Support Hive 3.x Metastore Hive 2.3 Execution Extensibility and Ecosystem Structured Streaming UI DDL/DML Enhancements Observable Metrics Event Log Rollover Monitoring and Debuggability
Performance Achieve high performance for interactive, batch, streaming and ML workloads Adaptive Query Execution Dynamic Partition Pruning Query Compilation Speedup Join Hints
Performance Achieve high performance for interactive, batch, streaming and ML workloads Adaptive Query Execution Dynamic Partition Pruning Join Hints Query Compilation Speedup
Spark Catalyst Optimizer Spark 1.x, Rule Spark 2.x, Rule + Cost
Query Optimization in Spark 2.x ▪ Missing statistics Expensive statistics collection ▪ Out-of-date statistics Compute and storage separated ▪ Suboptimal Heuristics Local ▪ Misestimated costs Complex environments User-defined functions
Spark Catalyst Optimizer Spark 1.x, Rule Spark 2.x, Rule + Cost Spark 3.0, Rule + Cost + Runtime
adaptive planning Based on statistics of the finished plan nodes, re-optimize the execution plan of the remaining queries Adaptive Query Execution [AQE]
Blog post: https://databricks.com/blog/2020/05/29/adaptive- query-execution-speeding-up-spark-sql-at-runtime.html Based on statistics of the finished plan nodes, re-optimize the execution plan of the remaining queries • Convert Sort Merge Join to Broadcast Hash Join • Shrink the number of reducers • Handle skew join Adaptive Query Execution
One of the Most Popular Performance Tuning Tips ▪ Choose Broadcast Hash Join? ▪ Increase “spark.sql.autoBroadcastJoinThreshold”? ▪ Use “broadcast” hint? However ▪ Hard to tune ▪ Hard to maintain over time ▪ OOM…
Why Spark not Making the Best Choice Automatically? ▪ Inaccurate/missing statistics; ▪ File is compressed; columnar store; ▪ Complex filters; black-box UDFs; ▪ Complex query fragments…
Estimate size: 30 MB Actual size: 8 MB Convert Sort Merge Join to Broadcast Hash Join Sort Merge Join Filter Scan Shuffle Sort Scan Shuffle Sort Stage 1 Stage 2 Estimate size: 100 MB Execute Sort Merge Join Filter Scan Shuffle Sort Scan Shuffle Sort Stage 1 Stage 2 Actual size: 86 MB Optimize Broadcast Hash Join Filter Scan Shuffle Broadcast Scan Shuffle Stage 1 Stage 2 Actual size: 86 MB Actual size: 8 MB
One More Popular Performance Tuning Tip ▪ Tuning spark.sql.shuffle.partitions ▪ Default magic number: 200 !?! However ▪ Too small: GC pressure; disk spilling ▪ Too large: Inefficient I/O; scheduler pressure ▪ Hard to tune over the whole query plan ▪ Hard to maintain over time
Dynamically Coalesce Shuffle Partitions Filter Scan Execute Shuffle (50 part.) Sort Stage 1 OptimizeFilter Scan Shuffle (50 part.) Sort Stage 1 Filter Scan Shuffle (50 part.) Sort Stage 1 Coalesce (5 part.) Set the initial partition number high to accommodate the largest data size of the entire query execution Automatically coalesce partitions if needed after each query stage
Another Popular Performance Tuning Tip ▪ Symptoms of data skew ▪ Frozen/long-running tasks ▪ Disk spilling ▪ Low resource utilization in most nodes ▪ OOM ▪ Various ways ▪ Find the skew values and rewrite the queries ▪ Adding extra skew keys…
TABLE A Table A - Part 0 Table A - Part 1 Table B - Part 0 TABLE B Data Skew in Sort Merge Join Shuffle Sort Table B - Part 1 Table A - Part 2 Table B - Part 2 Table A - Part 3 Table B - Part 3
Table A - Part 0 Table A - Part 1 Table B - Part 0 Data Skew in Sort Merge Join Sort Merge-Join Table B - Part 1 Table A - Part 2 Table B - Part 2 Table A - Part 3 Table B - Part 3 Table A – Sorted Part 0 Table B – Sorted Part 0 Table B – Sorted Part 1 Table B – Sorted Part 2 Table B – Sorted Part 3 Table A – Sorted Part 1 Table A – Sorted Part 2 Table B – Sorted Part 3 Merge-Join Merge-Join Merge-Join
Dynamically Optimize Skew Joins Sort Merge Join Filter Scan Execute Shuffle Sort Scan Shuffle Sort Sort Merge Join Filter Scan Shuffle Sort Scan Shuffle Sort Stage 1 Stage 2 Stage 1 Stage 2 Optimize Sort Merge Join Filter Scan Shuffle Sort Scan Shuffle Sort Stage 1 Stage 2 Skew Reader Skew Reader • Detect skew from partition sizes using runtime statistics • Split skew partitions into smaller sub-partitions
TABLE A Table A - Part 1 Table B - Part 0 TABLE B Shuffle Sort Table B - Part 1 Table A - Part 2 Table B - Part 2 Table A - Part 3 Table B - Part 3 Table B - Part 0 Table B - Part 0 Table A - Part 0 – Split 0 Table A - Part 0 – Split 1 Table A - Part 0 – Split 2 Dynamically Optimize Skew Joins
Table A - Part 1 Table B - Part 0 Sort Table B - Part 1 Table A - Part 2 Table B - Part 2 Table A - Part 3 Table B - Part 3 Table B - Part 0 Table B - Part 0 Table A - Part 0 – Split 0 Table A - Part 0 – Split 1 Table A - Part 0 – Split 2 Table A - Part 1 [Sorted] TabB.P0.S1 [Sorted] Table B - Part 1 [Sorted] Table A - Part 2 [Sorted] Table B - Part 2 [Sorted] Table A - Part 3 [Sorted] Table B - Part 3 [Sorted] TabB.P0.S1 [Sorted] TabB.P0.S0 [Sorted] TabA.P0.S0 [Sorted] TabA.P0.S1 [Sorted] TabA.P0.S2 [Sorted] Merge-Join Merge-Join Merge-Join Merge-Join Merge-Join Merge-Join
Adaptive Query Execution
Performance Achieve high performance for interactive, batch, streaming and ML workloads Adaptive Query Execution Dynamic Partition Pruning Join Hints Query Compilation Speedup
Dynamic Partition Pruning • Avoid partition scanning based on the query results of the other query fragments. • Important for star-schema queries. • Significant speedup in TPC-DS.
Dynamic Partition Pruning 60 / 102 TPC-DS queries: a speedup between 2x and 18x
t1: a large fact table with many partitions t2.id < 2 t2: a dimension table with a filter SELECT t1.id, t2.pKey FROM t1 JOIN t2 ON t1.pKey = t2.pKey AND t2.id < 2 t1.pKey = t2.pKey Dynamic Partition Pruning Project Join Filter Scan Scan Optimize
SELECT t1.id, t2.pKey FROM t1 JOIN t2 ON t1.pKey = t2.pKey AND t2.id < 2 Dynamic Partition Pruning Scan all the partitions of t1 Filter pushdown t1.pkey IN ( SELECT t2.pKey FROM t2 WHERE t2.id < 2) t2.id < 2 Project Join Filter + Scan Filter Optimize Scan t1.pKey = t2.pKey t1: a large fact table with many partitions t2.id < 2 t2: a dimension table with a filter t1.pKey = t2.pKey Project Join Filter Scan Scan Optimize
Dynamic Partition Pruning Scan all the partitions of t1 t2.id < 2 Project Join Filter + Scan Filter Scan t1.pKey = t2.pKey Scan the required partitions of t2 t1.pKey in DPPFilterResult
Dynamic Partition Pruning Optimize Scan the required partitions of t1 t2.id < 2 Project Join Filter + Scan Filter + Scan Scan the required partitions of t2 t1.pKey in DPPFilterResult Scan all the partitions of t1 t2.id < 2 Project Join Filter + Scan Filter Scan t1.pKey = t2.pKey Scan the required partitions of t2 t1.pKey in DPPFilterResult
Dynamic Partition Pruning 90+% less file scan33 X faster Optimize Optimize Scan the required partitions of t1 t2.id < 2 Project Join Filter + Scan Scan the required partitions of t2 t1.pKey in DPPFilterResult Scan all the partitions of t1 t2.id < 2 Project Join Filter + Scan Filter Scan t1.pKey = t2.pKey Scan the required partitions of t2 t1.pKey in DPPFilterResult Filter + Scan
Performance Achieve high performance for interactive, batch, streaming and ML workloads Adaptive Query Execution Dynamic Partition Pruning Join Hints Query Compilation Speedup
Optimizer Hints ▪ Join hints influence optimizer to choose the join strategies ▪ Broadcast hash join ▪ Sort-merge join NEW ▪ Shuffle hash join NEW ▪ Shuffle nested loop join NEW ▪ Should be used with extreme caution. ▪ Difficult to manage over time.
▪ Broadcast Hash Join SELECT /*+ BROADCAST(a) */ id FROM a JOIN b ON a.key = b.key ▪ Sort-Merge Join SELECT /*+ MERGE(a, b) */ id FROM a JOIN b ON a.key = b.key ▪ Shuffle Hash Join SELECT /*+ SHUFFLE_HASH(a, b) */ id FROM a JOIN b ON a.key = b.key ▪ Shuffle Nested Loop Join SELECT /*+ SHUFFLE_REPLICATE_NL(a, b) */ id FROM a JOIN b How to Use Join Hints?
Broadcast Hash Join Requires one side to be small. No shuffle, no sort, very fast. Sort-Merge Join Robust. Can handle any data size. Needs to shuffle and sort data, slower in most cases when the table size is small. Shuffle Hash Join Needs to shuffle data but no sort. Can handle large tables, but will OOM too if data is skewed. Shuffle Nested Loop Join Doesn’t require join keys.
Enable new use cases and simplify the Spark application development Accelerator-aware Scheduler Built-in Functions pandas UDF enhancements DELETE/UPDATE/ MERGE in Catalyst Richer APIs
Enable new use cases and simplify the Spark application development Richer APIs Accelerator-aware Scheduler Built-in Functions pandas UDF enhancements DELETE/UPDATE/ MERGE in Catalyst
Python UDF for SQL Python lambda functions for RDDs Session- specific Python UDF JAVA UDF in Python API New Pandas UDF Python Type Hints Py UDF V 3.0V 0.7 V 1.2 2013 2015 20182014 2019/20202016 2017 V 2.0 V 2.1 V 2.3/2.4
Scalar Pandas UDF [pandas.Series to pandas.Series] SPARK 2.3 SPARK 3.0Python Type Hints
Grouped Map Pandas Function API [pandas.DataFrame to pandas.DataFrame] SPARK 2.3 SPARK 3.0Python Type Hints
Grouped Aggregate Pandas UDF [pandas.Series to Scalar] SPARK 2.4 SPARK 3.0Python Type Hints
New Pandas UDF Types
Map Pandas UDF Cogrouped Map Pandas UDF New Pandas Function APIs
Enable new use cases and simplify the Spark application development Accelerator-aware Scheduler Built-in Functions pandas UDF enhancements DELETE/UPDATE/ MERGE in Catalyst Richer APIs
Accelerator-aware Scheduling ▪ Widely used for accelerating special workloads, e.g., deep learning and signal processing. ▪ Supports Standalone, YARN and K8S. ▪ Supports GPU now, FPGA, TPU, etc. in the future. ▪ Needs to specify required resources by configs ▪ Application level. Will support job/stage/task level in the future.
The workflow User Spark Cluster Manager 0. Auto-discover resources. 1. Submit an application with resource requests. 2. Pass resource requests to cluster manager. 4. Register executors. 3. Allocate executors with resource isolation. 5. Submit a Spark job. 6. Schedule tasks on available executors. 7. Dynamic allocation. 8. Retrieve assigned resources and use them in tasks. 9. Monitor and recover failed executors.
Discover and request accelerators Admin can specify a script to auto-discover accelerators (SPARK-27024) ● spark.driver.resource.${resourceName}.discoveryScript ● spark.executor.resource.${resourceName}.discoveryScript ● e.g., `nvidia-smi --query-gpu=index ...` User can request accelerators at application level (SPARK-27366) ● spark.executor.resource.${resourceName}.amount ● spark.driver.resource.${resourceName}.amount ● spark.task.resource.${resourceName}.amount
Retrieve assigned accelerators User can retrieve assigned accelerators from task context (SPARK- 27366) context = TaskContext.get() assigned_gpu = context.resources().get(“gpu”).get.addresses.head with tf.device(assigned_gpu): # training code ...
Cluster manager support Standalone SPARK-27360 YARN SPARK-27361 Kubernetes SPARK-27362 Mesos (not started) SPARK-27363
Web UI for accelerators
Enable new use cases and simplify the Spark application development Richer APIs Accelerator-aware Scheduler Built-in Functions pandas UDF enhancements DELETE/UPDATE/ MERGE in Catalyst
32 New Built-in Functions
Make monitoring and debugging Spark applications more comprehensive and stable Structured Streaming UI DDL/DML Enhancements Observable Metrics Event Log Rollover Monitoring and Debuggability
Make monitoring and debugging Spark applications more comprehensive and stable Monitoring and Debuggability Structured Streaming UI DDL/DML Enhancements Observable Metrics Event Log Rollover
Structured Streaming UI
Make monitoring and debugging Spark applications more comprehensive and stable Monitoring and Debuggability Structured Streaming UI DDL/DML Enhancements Observable Metrics Event Log Rollover
New Command EXPLAIN FORMATTED *(1) Project [key#5, val#6] +- *(1) Filter (isnotnull(key#5) AND (key#5 = Subquery scalar-subquery#15, [id=#113])) : +- Subquery scalar-subquery#15, [id=#113] : +- *(2) HashAggregate(keys=[], functions=[max(key#21)]) : +- Exchange SinglePartition, true, [id=#109] : +- *(1) HashAggregate(keys=[], functions=[partial_max(key#21)]) : +- *(1) Project [key#21] : +- *(1) Filter (isnotnull(val#22) AND (val#22 > 5)) : +- *(1) ColumnarToRow : +- FileScan parquet default.tab2[key#21,val#22] Batched: true, DataFilters: [isnotnull(val#22), (val#22 > 5)], Format: Parquet, Location: InMemoryFileIndex[file:/user/hive/warehouse/tab2], PartitionFilters: [], PushedFilters: [IsNotNull(val), GreaterThan(val,5)], ReadSchema: struct<key:int,val:int> +- *(1) ColumnarToRow +- FileScan parquet default.tab1[key#5,val#6] Batched: true, DataFilters: [isnotnull(key#5)], Format: Parquet, Location: InMemoryFileIndex[file:/user/hive/warehouse/tab1], PartitionFilters: [], PushedFilters: [IsNotNull(key)], ReadSchema: struct<key:int,val:int>
* Project (4) +- * Filter (3) +- * ColumnarToRow (2) +- Scan parquet default.tab1 (1) (1) Scan parquet default.tab1 Output [2]: [key#5, val#6] Batched: true Location: InMemoryFileIndex [file:/user/hive/warehouse/tab1] PushedFilters: [IsNotNull(key)] ReadSchema: struct<key:int,val:int> (2) ColumnarToRow [codegen id : 1] Input [2]: [key#5, val#6] (3) Filter [codegen id : 1] Input [2]: [key#5, val#6] Condition : (isnotnull(key#5) AND (key#5 = Subquery scalar-subquery#27, [id=#164])) (4) Project [codegen id : 1] Output [2]: [key#5, val#6] Input [2]: [key#5, val#6] EXPLAIN FORMATTED SELECT * FROM tab1 WHERE key = (SELECT max(key) FROM tab2 WHERE val > 5
(5) Scan parquet default.tab2 Output [2]: [key#21, val#22] Batched: true Location: InMemoryFileIndex [file:/user/hive/warehouse/tab2] PushedFilters: [IsNotNull(val), GreaterThan(val,5)] ReadSchema: struct<key:int,val:int> (6) ColumnarToRow [codegen id : 1] Input [2]: [key#21, val#22] (7) Filter [codegen id : 1] Input [2]: [key#21, val#22] Condition : (isnotnull(val#22) AND (val#22 > 5)) ===== Subqueries ===== Subquery:1 Hosting operator id = 3 Hosting Expression = Subquery scalar-subquery#27, [id=#164] * HashAggregate (11) +- Exchange (10) +- * HashAggregate (9) +- * Project (8) +- * Filter (7) +- * ColumnarToRow (6) +- Scan parquet default.tab2 (5) (8) Project [codegen id : 1] Output [1]: [key#21] Input [2]: [key#21, val#22] (9) HashAggregate [codegen id : 1] Input [1]: [key#21] Keys: [] Functions [1]: [partial_max(key#21)] Aggregate Attributes [1]: [max#35] Results [1]: [max#36] (10) Exchange Input [1]: [max#36] Arguments: SinglePartition, true, [id=#160] (11) HashAggregate [codegen id : 2] Input [1]: [max#36] Keys: [] Functions [1]: [max(key#21)] Aggregate Attributes [1]: [max(key#21)#33] Results [1]: [max(key#21)#33 AS max(key)#34]
DDL/DML Enhancements Make monitoring and debugging Spark applications more comprehensive and stable Monitoring and Debuggability Structured Streaming UI Observable Metrics Event Log Rollover
A flexible way to monitor data quality. Observable Metrics
Reduce the time and complexity of enabling applications that were written for other relational database products to run in Spark SQL. Reserved Keywords in Parser Proleptic Gregorian Calendar ANSI Store Assignment Overflow Checking SQL Compatibility
Reduce the time and complexity of enabling applications that were written for other relational database products to run in Spark SQL. Reserved Keywords in Parser Proleptic Gregorian Calendar ANSI Store Assignment Overflow Checking SQL Compatibility
A safer way to do table insertion and avoid bad data. ANSI store assignment + overflow check
A safer way to do table insertion and avoid bad data. ANSI store assignment + overflow check
Enhance the performance and functionalities of the built-in data sources Parquet/ORC Nested Column Pruning Parquet: Nested Column Filter Pushdown New Binary Data Source CSV Filter Pushdown Built-in Data Sources
Enhance the performance and functionalities of the built-in data sources Parquet/ORC Nested Column Pruning Parquet: Nested Column Filter Pushdown New Binary Data Source CSV Filter Pushdown Built-in Data Sources
▪ Skip reading useless data blocks when only a few inner fields are selected. Better performance for nested fields
▪ Skip reading useless data blocks when there are predicates with inner fields. Better performance for nested fields
Improve the plug-in interface and extend the deployment environments Data Source V2 API + Catalog Support Hive 3.x Metastore Hive 2.3 Execution Hadoop 3 Support Java 11 Support Extensibility and Ecosystem
Improve the plug-in interface and extend the deployment environments Data Source V2 API + Catalog Support Hive 3.x Metastore Hive 2.3 Execution Hadoop 3 Support Java 11 Support Extensibility and Ecosystem
Catalog plugin API Users can register customized catalogs and use Spark to access/manipulate table metadata directly. JDBC data source v2 is coming in Spark 3.1
To developers: When to use Data Source V2? ▪ Pick V2 if you want to provide catalog functionalities, as V1 doesn’t have such ability. ▪ Pick V2 If you want to support both batch and streaming, as V1 uses separated APIs for batch and streaming which makes it hard to reuse code. ▪ Pick V2 if you are sensitive to the scan performance, as V2 allows you to report data partitioning to skip shuffle, and implement vectorized reader for better performance. Data source v2 API is not as stable as V1!
Improve the plug-in interface and extend the deployment environments Data Source V2 API + Catalog Support Hive 3.x Metastore Hive 2.3 Execution Hadoop 3 Support Java 11 Support Extensibility and Ecosystem
Spark 3.0 Builds • Only builds with Scala 2.12 • Deprecates Python 2 (already EOL) • Can build with various Hadoop/Hive versions – Hadoop 2.7 + Hive 1.2 – Hadoop 2.7 + Hive 2.3 (supports Java 11) [Default] – Hadoop 3.2 + Hive 2.3 (supports Java 11) • Supports the following Hive metastore versions: – "0.12", "0.13", "0.14", "1.0", "1.1", "1.2", "2.0", "2.1", "2.2", "2.3", "3.0", "3.1"
32 New Built-in Functions ▪ map Documentation • Web UI • SQL reference • Migration guide • Semantic versioning guidelines
Adaptive Query Execution Dynamic Partition Pruning Query Compilation Speedup Join Hints Performance Richer APIs Accelerator-aware Scheduler Built-in Functions pandas UDF Enhancements DELETE/UPDATE/ MERGE in Catalyst Reserved Keywords Proleptic Gregorian Calendar ANSI Store Assignment Overflow Checking SQL Compatibility Built-in Data Sources Parquet/ORC Nested Column Pruning Parquet: Nested Column Filter Pushdown CSV Filter Pushdown New Binary Data Source Data Source V2 API + Catalog Support Java 11 Support Hadoop 3 Support Hive 3.x Metastore Hive 2.3 Execution Extensibility and Ecosystem Structured Streaming UI DDL/DML Enhancements Observable Metrics Event Log Rollover Monitoring and Debuggability
Try Databricks Runtime 7.0 Beta For Free https://databricks. com/try-databricks
Thank you for your contributions!

More Related Content

PDF
Everyday I'm Shuffling - Tips for Writing Better Spark Programs, Strata San J...
byDatabricks
 
PDF
Apache Spark Performance tuning and Best Practise
byKnoldus Inc.
 
PDF
Dynamic Partition Pruning in Apache Spark
byDatabricks
 
PDF
Adaptive Query Execution: Speeding Up Spark SQL at Runtime
byDatabricks
 
PDF
Memory Management in Apache Spark
byDatabricks
 
PDF
A Deep Dive into Query Execution Engine of Spark SQL
byDatabricks
 
PDF
From Query Plan to Query Performance: Supercharging your Apache Spark Queries...
byDatabricks
 
PPTX
Optimizing Apache Spark SQL Joins
byDatabricks
 
Everyday I'm Shuffling - Tips for Writing Better Spark Programs, Strata San J...
byDatabricks
 
Apache Spark Performance tuning and Best Practise
byKnoldus Inc.
 
Dynamic Partition Pruning in Apache Spark
byDatabricks
 
Adaptive Query Execution: Speeding Up Spark SQL at Runtime
byDatabricks
 
Memory Management in Apache Spark
byDatabricks
 
A Deep Dive into Query Execution Engine of Spark SQL
byDatabricks
 
From Query Plan to Query Performance: Supercharging your Apache Spark Queries...
byDatabricks
 
Optimizing Apache Spark SQL Joins
byDatabricks
 

What's hot

PDF
Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...
byDatabricks
 
PDF
The Parquet Format and Performance Optimization Opportunities
byDatabricks
 
PDF
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...
byDatabricks
 
PDF
Parquet performance tuning: the missing guide
byRyan Blue
 
PDF
Understanding Query Plans and Spark UIs
byDatabricks
 
PDF
Spark + Parquet In Depth: Spark Summit East Talk by Emily Curtin and Robbie S...
bySpark Summit
 
PDF
Deep Dive: Memory Management in Apache Spark
byDatabricks
 
PDF
A Thorough Comparison of Delta Lake, Iceberg and Hudi
byDatabricks
 
PDF
Spark with Delta Lake
byKnoldus Inc.
 
PDF
Apache Spark 2.0: A Deep Dive Into Structured Streaming - by Tathagata Das
byDatabricks
 
PDF
The Apache Spark File Format Ecosystem
byDatabricks
 
PDF
Simplify CDC Pipeline with Spark Streaming SQL and Delta Lake
byDatabricks
 
PDF
The Rise of ZStandard: Apache Spark/Parquet/ORC/Avro
byDatabricks
 
PDF
How We Optimize Spark SQL Jobs With parallel and sync IO
byDatabricks
 
PDF
Physical Plans in Spark SQL
byDatabricks
 
PDF
Apache Spark Core—Deep Dive—Proper Optimization
byDatabricks
 
PDF
Amazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud
byNoritaka Sekiyama
 
PDF
Spark tuning
byGMO-Z.com Vietnam Lab Center
 
PDF
The delta architecture
byPrakash Chockalingam
 
PDF
Making Apache Spark Better with Delta Lake
byDatabricks
 
Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...
byDatabricks
 
The Parquet Format and Performance Optimization Opportunities
byDatabricks
 
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...
byDatabricks
 
Parquet performance tuning: the missing guide
byRyan Blue
 
Understanding Query Plans and Spark UIs
byDatabricks
 
Spark + Parquet In Depth: Spark Summit East Talk by Emily Curtin and Robbie S...
bySpark Summit
 
Deep Dive: Memory Management in Apache Spark
byDatabricks
 
A Thorough Comparison of Delta Lake, Iceberg and Hudi
byDatabricks
 
Spark with Delta Lake
byKnoldus Inc.
 
Apache Spark 2.0: A Deep Dive Into Structured Streaming - by Tathagata Das
byDatabricks
 
The Apache Spark File Format Ecosystem
byDatabricks
 
Simplify CDC Pipeline with Spark Streaming SQL and Delta Lake
byDatabricks
 
The Rise of ZStandard: Apache Spark/Parquet/ORC/Avro
byDatabricks
 
How We Optimize Spark SQL Jobs With parallel and sync IO
byDatabricks
 
Physical Plans in Spark SQL
byDatabricks
 
Apache Spark Core—Deep Dive—Proper Optimization
byDatabricks
 
Amazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud
byNoritaka Sekiyama
 
Spark tuning
byGMO-Z.com Vietnam Lab Center
 
The delta architecture
byPrakash Chockalingam
 
Making Apache Spark Better with Delta Lake
byDatabricks
 

Similar to Deep Dive into the New Features of Apache Spark 3.0

PDF
What’s New in the Upcoming Apache Spark 3.0
byDatabricks
 
PDF
Apache Spark 3.0: Overview of What’s New and Why Care
byDatabricks
 
PDF
Spark + AI Summit recap jul16 2020
byGuido Oswald
 
PDF
Deep Dive into Spark
byEric Xiao
 
PDF
SQLDAY 2023 Chodkowski Adrian Databricks Performance Tuning
bySeeQuality.net
 
PDF
An Adaptive Execution Engine for Apache Spark with Carson Wang and Yucai Yu
byDatabricks
 
PDF
Spark SQL Join Improvement at Facebook
byDatabricks
 
PDF
Meetup talk
byArpit Tak
 
PDF
The internals of Spark SQL Joins, Dmytro Popovich
bySigma Software
 
PDF
Shark SQL and Rich Analytics at Scale
byDataWorks Summit
 
PPTX
2018 data warehouse features in spark
byChester Chen
 
PDF
Cost-Based Optimizer in Apache Spark 2.2
byDatabricks
 
PDF
Cost-Based Optimizer in Apache Spark 2.2 Ron Hu, Sameer Agarwal, Wenchen Fan ...
byDatabricks
 
PDF
Common Strategies for Improving Performance on Your Delta Lakehouse
byDatabricks
 
PDF
Cost-Based Optimizer Framework for Spark SQL: Spark Summit East talk by Ron H...
bySpark Summit
 
PDF
Optimizations in Spark; RDD, DataFrame
byKnoldus Inc.
 
PDF
Advanced Apache Spark Meetup Spark and Elasticsearch 02-15-2016
byChris Fregly
 
PDF
Best Practices for Building Robust Data Platform with Apache Spark and Delta
byDatabricks
 
PDF
Spark Streaming Tips for Devs and Ops by Fran perez y federico fernández
byJ On The Beach
 
PDF
Spark Streaming Tips for Devs and Ops
byFrancisco Pérez Paradas
 
What’s New in the Upcoming Apache Spark 3.0
byDatabricks
 
Apache Spark 3.0: Overview of What’s New and Why Care
byDatabricks
 
Spark + AI Summit recap jul16 2020
byGuido Oswald
 
Deep Dive into Spark
byEric Xiao
 
SQLDAY 2023 Chodkowski Adrian Databricks Performance Tuning
bySeeQuality.net
 
An Adaptive Execution Engine for Apache Spark with Carson Wang and Yucai Yu
byDatabricks
 
Spark SQL Join Improvement at Facebook
byDatabricks
 
Meetup talk
byArpit Tak
 
The internals of Spark SQL Joins, Dmytro Popovich
bySigma Software
 
Shark SQL and Rich Analytics at Scale
byDataWorks Summit
 
2018 data warehouse features in spark
byChester Chen
 
Cost-Based Optimizer in Apache Spark 2.2
byDatabricks
 
Cost-Based Optimizer in Apache Spark 2.2 Ron Hu, Sameer Agarwal, Wenchen Fan ...
byDatabricks
 
Common Strategies for Improving Performance on Your Delta Lakehouse
byDatabricks
 
Cost-Based Optimizer Framework for Spark SQL: Spark Summit East talk by Ron H...
bySpark Summit
 
Optimizations in Spark; RDD, DataFrame
byKnoldus Inc.
 
Advanced Apache Spark Meetup Spark and Elasticsearch 02-15-2016
byChris Fregly
 
Best Practices for Building Robust Data Platform with Apache Spark and Delta
byDatabricks
 
Spark Streaming Tips for Devs and Ops by Fran perez y federico fernández
byJ On The Beach
 
Spark Streaming Tips for Devs and Ops
byFrancisco Pérez Paradas
 

More from Databricks

PPTX
DW Migration Webinar-March 2022.pptx
byDatabricks
 
PPTX
Data Lakehouse Symposium | Day 1 | Part 1
byDatabricks
 
PPT
Data Lakehouse Symposium | Day 1 | Part 2
byDatabricks
 
PPTX
Data Lakehouse Symposium | Day 2
byDatabricks
 
PPTX
Data Lakehouse Symposium | Day 4
byDatabricks
 
PDF
5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop
byDatabricks
 
PDF
Democratizing Data Quality Through a Centralized Platform
byDatabricks
 
PDF
Learn to Use Databricks for Data Science
byDatabricks
 
PDF
Why APM Is Not the Same As ML Monitoring
byDatabricks
 
PDF
The Function, the Context, and the Data—Enabling ML Ops at Stitch Fix
byDatabricks
 
PDF
Stage Level Scheduling Improving Big Data and AI Integration
byDatabricks
 
PDF
Simplify Data Conversion from Spark to TensorFlow and PyTorch
byDatabricks
 
PDF
Scaling your Data Pipelines with Apache Spark on Kubernetes
byDatabricks
 
PDF
Scaling and Unifying SciKit Learn and Apache Spark Pipelines
byDatabricks
 
PDF
Sawtooth Windows for Feature Aggregations
byDatabricks
 
PDF
Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink
byDatabricks
 
PDF
Re-imagine Data Monitoring with whylogs and Spark
byDatabricks
 
PDF
Raven: End-to-end Optimization of ML Prediction Queries
byDatabricks
 
PDF
Processing Large Datasets for ADAS Applications using Apache Spark
byDatabricks
 
PDF
Massive Data Processing in Adobe Using Delta Lake
byDatabricks
 
DW Migration Webinar-March 2022.pptx
byDatabricks
 
Data Lakehouse Symposium | Day 1 | Part 1
byDatabricks
 
Data Lakehouse Symposium | Day 1 | Part 2
byDatabricks
 
Data Lakehouse Symposium | Day 2
byDatabricks
 
Data Lakehouse Symposium | Day 4
byDatabricks
 
5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop
byDatabricks
 
Democratizing Data Quality Through a Centralized Platform
byDatabricks
 
Learn to Use Databricks for Data Science
byDatabricks
 
Why APM Is Not the Same As ML Monitoring
byDatabricks
 
The Function, the Context, and the Data—Enabling ML Ops at Stitch Fix
byDatabricks
 
Stage Level Scheduling Improving Big Data and AI Integration
byDatabricks
 
Simplify Data Conversion from Spark to TensorFlow and PyTorch
byDatabricks
 
Scaling your Data Pipelines with Apache Spark on Kubernetes
byDatabricks
 
Scaling and Unifying SciKit Learn and Apache Spark Pipelines
byDatabricks
 
Sawtooth Windows for Feature Aggregations
byDatabricks
 
Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink
byDatabricks
 
Re-imagine Data Monitoring with whylogs and Spark
byDatabricks
 
Raven: End-to-end Optimization of ML Prediction Queries
byDatabricks
 
Processing Large Datasets for ADAS Applications using Apache Spark
byDatabricks
 
Massive Data Processing in Adobe Using Delta Lake
byDatabricks
 

Recently uploaded

PPTX
Recommended Practices for system studies with high shares of wind and solar
byIEA-ETSAP
 
PPTX
normal labor normal labor normal labor normal labor normal labor normal labor...
bymeksbaba1
 
PPTX
[DSC Europe 25] Milos Belcevic - Product Professional's Journey to Full-Stack...
byDataScienceConferenc1
 
PDF
Machine Learning Lectures -- Outliers in Data
byZahra Sadeghi
 
PPTX
PPT sidang disertasi terbuka terbaru.pptx
byRifaiteguhSaputra
 
PDF
Documenting Architectural Styles using CIDOC- CRM
bymagas4
 
PPTX
Data Collection lessons for grade 5 third quarter
byLorizaMElwas
 
PDF
mythbusting-cdc-what-employers-need-to-know.pdf
byHenry Tapper
 
PPTX
Working-at-Height-Rescue plan for construction.pptx
byHendraMessa1
 
PDF
Semantic Technology
byRathachai Chawuthai
 
PDF
Operations and Data Support Portfolio | Delbert Ross Wanjala Investor Relatio...
byDelbertRossWanjala
 
PDF
Data Migration Best Practices: A Visual Guide for Secure & Accurate Data Tran...
byalexmartincaneda
 
PDF
Presentation - 5 Essential Steps for Successful Data Migration
byalexmartincaneda
 
PPTX
Population & Sample in Bio-Statistics for Allied Health Science
byEzhumalaiG3
 
PPTX
[DSC Europe 25] Josip Saban - Career building for data professionals.pptx
byDataScienceConferenc1
 
PDF
SFBA Splunk Usergroup meeting Jan 21, 2026
byBecky Burwell
 
PDF
Let's build machine learning CNN in RUST + bonus_ ML in water meter with ESP...
byMarcin Bielak
 
PDF
Data architecture for Data Engineer: Overview of Architecture.pdf
byGeekyGangsta
 
PDF
Introduction to deep learning Important mmmmmmmm
byhossainmm2
 
PPTX
gas chromatograohy by Rasha Ibrahim.pptx
byri7061584
 
Recommended Practices for system studies with high shares of wind and solar
byIEA-ETSAP
 
normal labor normal labor normal labor normal labor normal labor normal labor...
bymeksbaba1
 
[DSC Europe 25] Milos Belcevic - Product Professional's Journey to Full-Stack...
byDataScienceConferenc1
 
Machine Learning Lectures -- Outliers in Data
byZahra Sadeghi
 
PPT sidang disertasi terbuka terbaru.pptx
byRifaiteguhSaputra
 
Documenting Architectural Styles using CIDOC- CRM
bymagas4
 
Data Collection lessons for grade 5 third quarter
byLorizaMElwas
 
mythbusting-cdc-what-employers-need-to-know.pdf
byHenry Tapper
 
Working-at-Height-Rescue plan for construction.pptx
byHendraMessa1
 
Semantic Technology
byRathachai Chawuthai
 
Operations and Data Support Portfolio | Delbert Ross Wanjala Investor Relatio...
byDelbertRossWanjala
 
Data Migration Best Practices: A Visual Guide for Secure & Accurate Data Tran...
byalexmartincaneda
 
Presentation - 5 Essential Steps for Successful Data Migration
byalexmartincaneda
 
Population & Sample in Bio-Statistics for Allied Health Science
byEzhumalaiG3
 
[DSC Europe 25] Josip Saban - Career building for data professionals.pptx
byDataScienceConferenc1
 
SFBA Splunk Usergroup meeting Jan 21, 2026
byBecky Burwell
 
Let's build machine learning CNN in RUST + bonus_ ML in water meter with ESP...
byMarcin Bielak
 
Data architecture for Data Engineer: Overview of Architecture.pdf
byGeekyGangsta
 
Introduction to deep learning Important mmmmmmmm
byhossainmm2
 
gas chromatograohy by Rasha Ibrahim.pptx
byri7061584
 

Deep Dive into the New Features of Apache Spark 3.0

  • 1.
    Deep Dive intothe New Features of Upcoming Apache Spark 3.0 Xiao Li gatorsmile June 2020 Wenchen Fan cloud-fan
  • 2.
    • Open SourceTeam at • Apache Spark Committer and PMC About Us Xiao Li (Github: gatorsmile) Wenchen Fan (Github: cloud-fan)
  • 3.
    Unified data analyticsplatform for accelerating innovation across data science, data engineering, and business analytics Original creators of popular data and machine learning open source projects Global company with 5,000 customers and 450+ partners
  • 4.
  • 5.
    Adaptive Query Execution Dynamic Partition Pruning QueryCompilation Speedup Join Hints Performance Richer APIs Accelerator-aware Scheduler Built-in Functions pandas UDF Enhancements DELETE/UPDATE/ MERGE in Catalyst Reserved Keywords Proleptic Gregorian Calendar ANSI Store Assignment Overflow Checking SQL Compatibility Built-in Data Sources Parquet/ORC Nested Column Pruning Parquet: Nested Column Filter Pushdown CSV Filter Pushdown New Binary Data Source Data Source V2 API + Catalog Support Java 11 Support Hadoop 3 Support Hive 3.x Metastore Hive 2.3 Execution Extensibility and Ecosystem Structured Streaming UI DDL/DML Enhancements Observable Metrics Event Log Rollover Monitoring and Debuggability
  • 6.
    Performance Achieve high performancefor interactive, batch, streaming and ML workloads Adaptive Query Execution Dynamic Partition Pruning Query Compilation Speedup Join Hints
  • 7.
    Performance Achieve high performancefor interactive, batch, streaming and ML workloads Adaptive Query Execution Dynamic Partition Pruning Join Hints Query Compilation Speedup
  • 8.
    Spark Catalyst Optimizer Spark1.x, Rule Spark 2.x, Rule + Cost
  • 9.
    Query Optimization inSpark 2.x ▪ Missing statistics Expensive statistics collection ▪ Out-of-date statistics Compute and storage separated ▪ Suboptimal Heuristics Local ▪ Misestimated costs Complex environments User-defined functions
  • 10.
    Spark Catalyst Optimizer Spark1.x, Rule Spark 2.x, Rule + Cost Spark 3.0, Rule + Cost + Runtime
  • 11.
    adaptive planning Based onstatistics of the finished plan nodes, re-optimize the execution plan of the remaining queries Adaptive Query Execution [AQE]
  • 12.
    Blog post: https://databricks.com/blog/2020/05/29/adaptive- query-execution-speeding-up-spark-sql-at-runtime.html Basedon statistics of the finished plan nodes, re-optimize the execution plan of the remaining queries • Convert Sort Merge Join to Broadcast Hash Join • Shrink the number of reducers • Handle skew join Adaptive Query Execution
  • 13.
    One of theMost Popular Performance Tuning Tips ▪ Choose Broadcast Hash Join? ▪ Increase “spark.sql.autoBroadcastJoinThreshold”? ▪ Use “broadcast” hint? However ▪ Hard to tune ▪ Hard to maintain over time ▪ OOM…
  • 14.
    Why Spark notMaking the Best Choice Automatically? ▪ Inaccurate/missing statistics; ▪ File is compressed; columnar store; ▪ Complex filters; black-box UDFs; ▪ Complex query fragments…
  • 15.
    Estimate size: 30 MB Actualsize: 8 MB Convert Sort Merge Join to Broadcast Hash Join Sort Merge Join Filter Scan Shuffle Sort Scan Shuffle Sort Stage 1 Stage 2 Estimate size: 100 MB Execute Sort Merge Join Filter Scan Shuffle Sort Scan Shuffle Sort Stage 1 Stage 2 Actual size: 86 MB Optimize Broadcast Hash Join Filter Scan Shuffle Broadcast Scan Shuffle Stage 1 Stage 2 Actual size: 86 MB Actual size: 8 MB
  • 16.
    One More PopularPerformance Tuning Tip ▪ Tuning spark.sql.shuffle.partitions ▪ Default magic number: 200 !?! However ▪ Too small: GC pressure; disk spilling ▪ Too large: Inefficient I/O; scheduler pressure ▪ Hard to tune over the whole query plan ▪ Hard to maintain over time
  • 17.
    Dynamically Coalesce ShufflePartitions Filter Scan Execute Shuffle (50 part.) Sort Stage 1 OptimizeFilter Scan Shuffle (50 part.) Sort Stage 1 Filter Scan Shuffle (50 part.) Sort Stage 1 Coalesce (5 part.) Set the initial partition number high to accommodate the largest data size of the entire query execution Automatically coalesce partitions if needed after each query stage
  • 18.
    Another Popular PerformanceTuning Tip ▪ Symptoms of data skew ▪ Frozen/long-running tasks ▪ Disk spilling ▪ Low resource utilization in most nodes ▪ OOM ▪ Various ways ▪ Find the skew values and rewrite the queries ▪ Adding extra skew keys…
  • 19.
    TABLE A Table A- Part 0 Table A - Part 1 Table B - Part 0 TABLE B Data Skew in Sort Merge Join Shuffle Sort Table B - Part 1 Table A - Part 2 Table B - Part 2 Table A - Part 3 Table B - Part 3
  • 20.
    Table A -Part 0 Table A - Part 1 Table B - Part 0 Data Skew in Sort Merge Join Sort Merge-Join Table B - Part 1 Table A - Part 2 Table B - Part 2 Table A - Part 3 Table B - Part 3 Table A – Sorted Part 0 Table B – Sorted Part 0 Table B – Sorted Part 1 Table B – Sorted Part 2 Table B – Sorted Part 3 Table A – Sorted Part 1 Table A – Sorted Part 2 Table B – Sorted Part 3 Merge-Join Merge-Join Merge-Join
  • 21.
    Dynamically Optimize SkewJoins Sort Merge Join Filter Scan Execute Shuffle Sort Scan Shuffle Sort Sort Merge Join Filter Scan Shuffle Sort Scan Shuffle Sort Stage 1 Stage 2 Stage 1 Stage 2 Optimize Sort Merge Join Filter Scan Shuffle Sort Scan Shuffle Sort Stage 1 Stage 2 Skew Reader Skew Reader • Detect skew from partition sizes using runtime statistics • Split skew partitions into smaller sub-partitions
  • 22.
    TABLE A Table A- Part 1 Table B - Part 0 TABLE B Shuffle Sort Table B - Part 1 Table A - Part 2 Table B - Part 2 Table A - Part 3 Table B - Part 3 Table B - Part 0 Table B - Part 0 Table A - Part 0 – Split 0 Table A - Part 0 – Split 1 Table A - Part 0 – Split 2 Dynamically Optimize Skew Joins
  • 23.
    Table A -Part 1 Table B - Part 0 Sort Table B - Part 1 Table A - Part 2 Table B - Part 2 Table A - Part 3 Table B - Part 3 Table B - Part 0 Table B - Part 0 Table A - Part 0 – Split 0 Table A - Part 0 – Split 1 Table A - Part 0 – Split 2 Table A - Part 1 [Sorted] TabB.P0.S1 [Sorted] Table B - Part 1 [Sorted] Table A - Part 2 [Sorted] Table B - Part 2 [Sorted] Table A - Part 3 [Sorted] Table B - Part 3 [Sorted] TabB.P0.S1 [Sorted] TabB.P0.S0 [Sorted] TabA.P0.S0 [Sorted] TabA.P0.S1 [Sorted] TabA.P0.S2 [Sorted] Merge-Join Merge-Join Merge-Join Merge-Join Merge-Join Merge-Join
  • 24.
  • 25.
    Performance Achieve high performancefor interactive, batch, streaming and ML workloads Adaptive Query Execution Dynamic Partition Pruning Join Hints Query Compilation Speedup
  • 26.
    Dynamic Partition Pruning •Avoid partition scanning based on the query results of the other query fragments. • Important for star-schema queries. • Significant speedup in TPC-DS.
  • 27.
    Dynamic Partition Pruning 60/ 102 TPC-DS queries: a speedup between 2x and 18x
  • 28.
    t1: a large facttable with many partitions t2.id < 2 t2: a dimension table with a filter SELECT t1.id, t2.pKey FROM t1 JOIN t2 ON t1.pKey = t2.pKey AND t2.id < 2 t1.pKey = t2.pKey Dynamic Partition Pruning Project Join Filter Scan Scan Optimize
  • 29.
    SELECT t1.id, t2.pKey FROMt1 JOIN t2 ON t1.pKey = t2.pKey AND t2.id < 2 Dynamic Partition Pruning Scan all the partitions of t1 Filter pushdown t1.pkey IN ( SELECT t2.pKey FROM t2 WHERE t2.id < 2) t2.id < 2 Project Join Filter + Scan Filter Optimize Scan t1.pKey = t2.pKey t1: a large fact table with many partitions t2.id < 2 t2: a dimension table with a filter t1.pKey = t2.pKey Project Join Filter Scan Scan Optimize
  • 30.
    Dynamic Partition Pruning Scanall the partitions of t1 t2.id < 2 Project Join Filter + Scan Filter Scan t1.pKey = t2.pKey Scan the required partitions of t2 t1.pKey in DPPFilterResult
  • 31.
    Dynamic Partition Pruning Optimize Scanthe required partitions of t1 t2.id < 2 Project Join Filter + Scan Filter + Scan Scan the required partitions of t2 t1.pKey in DPPFilterResult Scan all the partitions of t1 t2.id < 2 Project Join Filter + Scan Filter Scan t1.pKey = t2.pKey Scan the required partitions of t2 t1.pKey in DPPFilterResult
  • 32.
    Dynamic Partition Pruning 90+%less file scan33 X faster Optimize Optimize Scan the required partitions of t1 t2.id < 2 Project Join Filter + Scan Scan the required partitions of t2 t1.pKey in DPPFilterResult Scan all the partitions of t1 t2.id < 2 Project Join Filter + Scan Filter Scan t1.pKey = t2.pKey Scan the required partitions of t2 t1.pKey in DPPFilterResult Filter + Scan
  • 33.
    Performance Achieve high performancefor interactive, batch, streaming and ML workloads Adaptive Query Execution Dynamic Partition Pruning Join Hints Query Compilation Speedup
  • 34.
    Optimizer Hints ▪ Joinhints influence optimizer to choose the join strategies ▪ Broadcast hash join ▪ Sort-merge join NEW ▪ Shuffle hash join NEW ▪ Shuffle nested loop join NEW ▪ Should be used with extreme caution. ▪ Difficult to manage over time.
  • 35.
    ▪ Broadcast HashJoin SELECT /*+ BROADCAST(a) */ id FROM a JOIN b ON a.key = b.key ▪ Sort-Merge Join SELECT /*+ MERGE(a, b) */ id FROM a JOIN b ON a.key = b.key ▪ Shuffle Hash Join SELECT /*+ SHUFFLE_HASH(a, b) */ id FROM a JOIN b ON a.key = b.key ▪ Shuffle Nested Loop Join SELECT /*+ SHUFFLE_REPLICATE_NL(a, b) */ id FROM a JOIN b How to Use Join Hints?
  • 36.
    Broadcast Hash Join Requiresone side to be small. No shuffle, no sort, very fast. Sort-Merge Join Robust. Can handle any data size. Needs to shuffle and sort data, slower in most cases when the table size is small. Shuffle Hash Join Needs to shuffle data but no sort. Can handle large tables, but will OOM too if data is skewed. Shuffle Nested Loop Join Doesn’t require join keys.
  • 37.
    Enable new usecases and simplify the Spark application development Accelerator-aware Scheduler Built-in Functions pandas UDF enhancements DELETE/UPDATE/ MERGE in Catalyst Richer APIs
  • 38.
    Enable new usecases and simplify the Spark application development Richer APIs Accelerator-aware Scheduler Built-in Functions pandas UDF enhancements DELETE/UPDATE/ MERGE in Catalyst
  • 39.
    Python UDF for SQL Python lambda functionsfor RDDs Session- specific Python UDF JAVA UDF in Python API New Pandas UDF Python Type Hints Py UDF V 3.0V 0.7 V 1.2 2013 2015 20182014 2019/20202016 2017 V 2.0 V 2.1 V 2.3/2.4
  • 40.
    Scalar Pandas UDF [pandas.Seriesto pandas.Series] SPARK 2.3 SPARK 3.0Python Type Hints
  • 41.
    Grouped Map PandasFunction API [pandas.DataFrame to pandas.DataFrame] SPARK 2.3 SPARK 3.0Python Type Hints
  • 42.
    Grouped Aggregate PandasUDF [pandas.Series to Scalar] SPARK 2.4 SPARK 3.0Python Type Hints
  • 43.
  • 44.
    Map Pandas UDF CogroupedMap Pandas UDF New Pandas Function APIs
  • 45.
    Enable new usecases and simplify the Spark application development Accelerator-aware Scheduler Built-in Functions pandas UDF enhancements DELETE/UPDATE/ MERGE in Catalyst Richer APIs
  • 46.
    Accelerator-aware Scheduling ▪ Widelyused for accelerating special workloads, e.g., deep learning and signal processing. ▪ Supports Standalone, YARN and K8S. ▪ Supports GPU now, FPGA, TPU, etc. in the future. ▪ Needs to specify required resources by configs ▪ Application level. Will support job/stage/task level in the future.
  • 47.
    The workflow User SparkCluster Manager 0. Auto-discover resources. 1. Submit an application with resource requests. 2. Pass resource requests to cluster manager. 4. Register executors. 3. Allocate executors with resource isolation. 5. Submit a Spark job. 6. Schedule tasks on available executors. 7. Dynamic allocation. 8. Retrieve assigned resources and use them in tasks. 9. Monitor and recover failed executors.
  • 48.
    Discover and requestaccelerators Admin can specify a script to auto-discover accelerators (SPARK-27024) ● spark.driver.resource.${resourceName}.discoveryScript ● spark.executor.resource.${resourceName}.discoveryScript ● e.g., `nvidia-smi --query-gpu=index ...` User can request accelerators at application level (SPARK-27366) ● spark.executor.resource.${resourceName}.amount ● spark.driver.resource.${resourceName}.amount ● spark.task.resource.${resourceName}.amount
  • 49.
    Retrieve assigned accelerators Usercan retrieve assigned accelerators from task context (SPARK- 27366) context = TaskContext.get() assigned_gpu = context.resources().get(“gpu”).get.addresses.head with tf.device(assigned_gpu): # training code ...
  • 50.
  • 51.
    Web UI foraccelerators
  • 52.
    Enable new usecases and simplify the Spark application development Richer APIs Accelerator-aware Scheduler Built-in Functions pandas UDF enhancements DELETE/UPDATE/ MERGE in Catalyst
  • 53.
    32 New Built-inFunctions
  • 57.
    Make monitoring anddebugging Spark applications more comprehensive and stable Structured Streaming UI DDL/DML Enhancements Observable Metrics Event Log Rollover Monitoring and Debuggability
  • 58.
    Make monitoring anddebugging Spark applications more comprehensive and stable Monitoring and Debuggability Structured Streaming UI DDL/DML Enhancements Observable Metrics Event Log Rollover
  • 59.
  • 60.
    Make monitoring anddebugging Spark applications more comprehensive and stable Monitoring and Debuggability Structured Streaming UI DDL/DML Enhancements Observable Metrics Event Log Rollover
  • 61.
    New Command EXPLAINFORMATTED *(1) Project [key#5, val#6] +- *(1) Filter (isnotnull(key#5) AND (key#5 = Subquery scalar-subquery#15, [id=#113])) : +- Subquery scalar-subquery#15, [id=#113] : +- *(2) HashAggregate(keys=[], functions=[max(key#21)]) : +- Exchange SinglePartition, true, [id=#109] : +- *(1) HashAggregate(keys=[], functions=[partial_max(key#21)]) : +- *(1) Project [key#21] : +- *(1) Filter (isnotnull(val#22) AND (val#22 > 5)) : +- *(1) ColumnarToRow : +- FileScan parquet default.tab2[key#21,val#22] Batched: true, DataFilters: [isnotnull(val#22), (val#22 > 5)], Format: Parquet, Location: InMemoryFileIndex[file:/user/hive/warehouse/tab2], PartitionFilters: [], PushedFilters: [IsNotNull(val), GreaterThan(val,5)], ReadSchema: struct<key:int,val:int> +- *(1) ColumnarToRow +- FileScan parquet default.tab1[key#5,val#6] Batched: true, DataFilters: [isnotnull(key#5)], Format: Parquet, Location: InMemoryFileIndex[file:/user/hive/warehouse/tab1], PartitionFilters: [], PushedFilters: [IsNotNull(key)], ReadSchema: struct<key:int,val:int>
  • 62.
    * Project (4) +-* Filter (3) +- * ColumnarToRow (2) +- Scan parquet default.tab1 (1) (1) Scan parquet default.tab1 Output [2]: [key#5, val#6] Batched: true Location: InMemoryFileIndex [file:/user/hive/warehouse/tab1] PushedFilters: [IsNotNull(key)] ReadSchema: struct<key:int,val:int> (2) ColumnarToRow [codegen id : 1] Input [2]: [key#5, val#6] (3) Filter [codegen id : 1] Input [2]: [key#5, val#6] Condition : (isnotnull(key#5) AND (key#5 = Subquery scalar-subquery#27, [id=#164])) (4) Project [codegen id : 1] Output [2]: [key#5, val#6] Input [2]: [key#5, val#6] EXPLAIN FORMATTED SELECT * FROM tab1 WHERE key = (SELECT max(key) FROM tab2 WHERE val > 5
  • 63.
    (5) Scan parquetdefault.tab2 Output [2]: [key#21, val#22] Batched: true Location: InMemoryFileIndex [file:/user/hive/warehouse/tab2] PushedFilters: [IsNotNull(val), GreaterThan(val,5)] ReadSchema: struct<key:int,val:int> (6) ColumnarToRow [codegen id : 1] Input [2]: [key#21, val#22] (7) Filter [codegen id : 1] Input [2]: [key#21, val#22] Condition : (isnotnull(val#22) AND (val#22 > 5)) ===== Subqueries ===== Subquery:1 Hosting operator id = 3 Hosting Expression = Subquery scalar-subquery#27, [id=#164] * HashAggregate (11) +- Exchange (10) +- * HashAggregate (9) +- * Project (8) +- * Filter (7) +- * ColumnarToRow (6) +- Scan parquet default.tab2 (5) (8) Project [codegen id : 1] Output [1]: [key#21] Input [2]: [key#21, val#22] (9) HashAggregate [codegen id : 1] Input [1]: [key#21] Keys: [] Functions [1]: [partial_max(key#21)] Aggregate Attributes [1]: [max#35] Results [1]: [max#36] (10) Exchange Input [1]: [max#36] Arguments: SinglePartition, true, [id=#160] (11) HashAggregate [codegen id : 2] Input [1]: [max#36] Keys: [] Functions [1]: [max(key#21)] Aggregate Attributes [1]: [max(key#21)#33] Results [1]: [max(key#21)#33 AS max(key)#34]
  • 64.
    DDL/DML Enhancements Make monitoring anddebugging Spark applications more comprehensive and stable Monitoring and Debuggability Structured Streaming UI Observable Metrics Event Log Rollover
  • 65.
    A flexible wayto monitor data quality. Observable Metrics
  • 66.
    Reduce the timeand complexity of enabling applications that were written for other relational database products to run in Spark SQL. Reserved Keywords in Parser Proleptic Gregorian Calendar ANSI Store Assignment Overflow Checking SQL Compatibility
  • 67.
    Reduce the timeand complexity of enabling applications that were written for other relational database products to run in Spark SQL. Reserved Keywords in Parser Proleptic Gregorian Calendar ANSI Store Assignment Overflow Checking SQL Compatibility
  • 68.
    A safer wayto do table insertion and avoid bad data. ANSI store assignment + overflow check
  • 69.
    A safer wayto do table insertion and avoid bad data. ANSI store assignment + overflow check
  • 70.
    Enhance the performanceand functionalities of the built-in data sources Parquet/ORC Nested Column Pruning Parquet: Nested Column Filter Pushdown New Binary Data Source CSV Filter Pushdown Built-in Data Sources
  • 71.
    Enhance the performanceand functionalities of the built-in data sources Parquet/ORC Nested Column Pruning Parquet: Nested Column Filter Pushdown New Binary Data Source CSV Filter Pushdown Built-in Data Sources
  • 72.
    ▪ Skip readinguseless data blocks when only a few inner fields are selected. Better performance for nested fields
  • 73.
    ▪ Skip readinguseless data blocks when there are predicates with inner fields. Better performance for nested fields
  • 74.
    Improve the plug-ininterface and extend the deployment environments Data Source V2 API + Catalog Support Hive 3.x Metastore Hive 2.3 Execution Hadoop 3 Support Java 11 Support Extensibility and Ecosystem
  • 75.
    Improve the plug-ininterface and extend the deployment environments Data Source V2 API + Catalog Support Hive 3.x Metastore Hive 2.3 Execution Hadoop 3 Support Java 11 Support Extensibility and Ecosystem
  • 76.
    Catalog plugin API Userscan register customized catalogs and use Spark to access/manipulate table metadata directly. JDBC data source v2 is coming in Spark 3.1
  • 77.
    To developers: Whento use Data Source V2? ▪ Pick V2 if you want to provide catalog functionalities, as V1 doesn’t have such ability. ▪ Pick V2 If you want to support both batch and streaming, as V1 uses separated APIs for batch and streaming which makes it hard to reuse code. ▪ Pick V2 if you are sensitive to the scan performance, as V2 allows you to report data partitioning to skip shuffle, and implement vectorized reader for better performance. Data source v2 API is not as stable as V1!
  • 78.
    Improve the plug-ininterface and extend the deployment environments Data Source V2 API + Catalog Support Hive 3.x Metastore Hive 2.3 Execution Hadoop 3 Support Java 11 Support Extensibility and Ecosystem
  • 79.
    Spark 3.0 Builds •Only builds with Scala 2.12 • Deprecates Python 2 (already EOL) • Can build with various Hadoop/Hive versions – Hadoop 2.7 + Hive 1.2 – Hadoop 2.7 + Hive 2.3 (supports Java 11) [Default] – Hadoop 3.2 + Hive 2.3 (supports Java 11) • Supports the following Hive metastore versions: – "0.12", "0.13", "0.14", "1.0", "1.1", "1.2", "2.0", "2.1", "2.2", "2.3", "3.0", "3.1"
  • 80.
    32 New Built-inFunctions ▪ map Documentation • Web UI • SQL reference • Migration guide • Semantic versioning guidelines
  • 95.
    Adaptive Query Execution Dynamic Partition Pruning QueryCompilation Speedup Join Hints Performance Richer APIs Accelerator-aware Scheduler Built-in Functions pandas UDF Enhancements DELETE/UPDATE/ MERGE in Catalyst Reserved Keywords Proleptic Gregorian Calendar ANSI Store Assignment Overflow Checking SQL Compatibility Built-in Data Sources Parquet/ORC Nested Column Pruning Parquet: Nested Column Filter Pushdown CSV Filter Pushdown New Binary Data Source Data Source V2 API + Catalog Support Java 11 Support Hadoop 3 Support Hive 3.x Metastore Hive 2.3 Execution Extensibility and Ecosystem Structured Streaming UI DDL/DML Enhancements Observable Metrics Event Log Rollover Monitoring and Debuggability
  • 96.
    Try Databricks Runtime 7.0Beta For Free https://databricks. com/try-databricks
  • 97.
    Thank you foryour contributions!