The document discusses the advancements in lakehouse models with Spark 3.0, focusing on overcoming challenges associated with traditional data warehousing methods like Kimball. Key features include the use of Slowly Changing Dimensions (SCD), dynamic partition pruning, and adaptive query execution (AQE) that enhance performance and simplify complex tasks. The Delta file format facilitates these advancements by enabling familiar SQL patterns and improving analytics capabilities.

1. Achieving Lakehouse Models
with Spark 3.0
Simon Whiteley
Director of Engineering, Advancing Analytics

2. Agenda
Why Lakehouse?
Kimball Problems
Delta & Spark 3.0
▪ SCD & SQL Merge
▪ Dynamic Partition Pruning
▪ Adaptive Query Execution
Enabling the Lakehouse

3. The Data Lakehouse

4. Analytics Evolution
PARQUET
Delta Lake

6. The Modern Warehouse
RAW BASE
PARQUET

7. The Lakehouse
RAW BASE ENRICHED
DELTA DELTA

8. Lakehouse Barriers

9. When you think Warehouse…
We automatically think of Star
Schemas and Kimball warehousing
approaches.
A large central fact table with smaller
reference dimensions… some of which
aren’t so small

10. Literally Everyone
(All The time)
“You can’t use Kimball in a
Data Lake”

11. Three Historical Challenges
▪ Slowly Changing Dimensions
▪ Filtering Dimensions
▪ General SQL Performance

12. Slowly Changing Dimensions

13. SCD - Enabling the Familiar
PrimaryKey Address Current EffectiveDate EndDate
11 A new customer address TRUE 03/08/2020 null
58 Yet another address TRUE 03/08/2020 null
41 A different address TRUE 03/08/2020 null
PrimaryKey Address Current EffectiveDate EndDate
11 A new customer address FALSE 03/08/2020 22/10/2020
11 An updated address TRUE 22/10/2020 null
58 Yet another address TRUE 03/08/2020 null
41 A different address TRUE 03/08/2020 null

14. SCD - Merge Commands
MERGE INTO dataai.addresses as original
USING updates
ON original.primaryKey = updates.primaryKey
WHEN MATCHED THEN
UPDATE SET *
WHEN NOT MATCHED THEN
INSERT *
Available in SQL, Scala and Python APIs - the merge command has made many complex warehousing
jobs accessible to the wider Analytics community
This is enabled by the Delta file format

15. Dynamic Partition Pruning

16. Spark Partitioning
SELECT * FROM Sales WHERE Month
= 3
SQL Query Action
Filtering performed by
selectively reading files
SALES
Month=1 Month=2
Month=3 Month=4

17. Cross-Filter Spark 2.4
SELECT * FROM Sales JOIN Date
WHERE DateMonth = 3
SQL Query Action
SALES
Month=1 Month=2
Month=3 Month=4
DimDATE
Partition Keys not hit when
filtering on joined tables

18. Cross-Filter Spark 3.0
SELECT * FROM Sales JOIN Date
WHERE DateMonth = 3
SQL Query Action
SALES
Month=1 Month=2
Month=3 Month=4
DimDATE
Dynamic Partition Pruning
determines partition filters
during runtime

19. Adaptive Query Execution

20. AQE in Spark 3.0
AQE will speed up common queries in a number of ways:
▪ Coalescing Shuffle Partitions
▪ Switching Join Strategies
▪ Optimizing Skew Joins

21. Before AQE - Shuffle Coalescing
Read RDDs
Read RDDs
Shuffle RDDs
Shuffle RDDs
Shuffle RDDs
Shuffle RDDs
Shuffle RDDs
Shuffle RDDs
Shuffle RDDs
Shuffle RDDs
Shuffle RDDs
Shuffle RDDs
Shuffle RDDs
Write RDDs
Write RDDs
2 Tasks 200 Tasks 2 Tasks

22. Using Spark 3.0 AQE - Shuffle Coalescing
Read RDDs
Read RDDs
Write RDDs
Write RDDs
2 Tasks 2 Tasks 2 Tasks
Shuffle RDDs
Shuffle RDDs

23. DEMO: Let’s see it in action

24. The Data Lakehouse
Delta & Spark 3.0 enable the Lakehouse through:
▪ Enabling familiar (SQL) patterns
▪ Removing technical barriers
▪ Targeting performance of common
warehousing activities

25. Feedback
Your feedback is important to us.
Don’t forget to rate
and review the sessions.