Timely data in a data warehouse is a challenge many of us face, often with there being no straightforward solution. Using a combination of batch and streaming data pipelines you can leverage the Delta Lake format to provide an enterprise data warehouse at a near real-time frequency. Delta Lake eases the ETL workload by enabling ACID transactions in a warehousing environment. Coupling this with structured streaming, you can achieve a low latency data warehouse. In this talk, we’ll talk about how to use Delta Lake to improve the latency of ingestion and storage of your data warehouse tables. We’ll also talk about how you can use spark streaming to build the aggregations and tables that drive your data warehouse.

1. WIFI SSID:Spark+AISummit | Password: UnifiedDataAnalytics

2. Jasper Groot, Eventbrite
Near Real-Time Data
Warehousing with
Apache Spark and Delta
Lake
#UnifiedDataAnalytics #SparkAISummit

3. Introduction
Personal Introduction
- Data engineering in the event
industry for 4+ years
- Using spark for 3+ years
- Currently at Eventbrite
3#UnifiedDataAnalytics #SparkAISummit

4. Outline
• Structured Streaming
– In a nutshell
• Delta Lake
– How it works
• Data Warehousing
– Detailed example using these tools
– Gotchas
4#UnifiedDataAnalytics #SparkAISummit

5. Structured Streaming
In a nutshell
• Introduced in Spark 2.0
• Streams are unbounded dataframes
• Familiar API for anyone who has used
Dataframes
5#UnifiedDataAnalytics #SparkAISummit

6. Structured Streaming
6#UnifiedDataAnalytics #SparkAISummit

7. Structured Streaming
How streaming dataframes differ
• More restrictive operations
– Distinct
– Joins
– Aggregations
• Must be after joins
7#UnifiedDataAnalytics #SparkAISummit

8. Structured Streaming - Recovery
Recovery is done through checkpointing
• Checkpointing uses write-ahead logs
• Stores running aggregates and progress
• Must be a HDFS compatible FS
There are limitations on resuming from a
checkpoint after updating application logic
8#UnifiedDataAnalytics #SparkAISummit

9. Structured Streaming + Data
Warehousing
• Importance of watermarking
• Managing late data
• Using foreachBatch
9#UnifiedDataAnalytics #SparkAISummit

10. 10#UnifiedDataAnalytics #SparkAISummit

11. 11#UnifiedDataAnalytics #SparkAISummit

12. Delta Lake
• Open Sourced in 2019
• Parquet under the hood
• Enables ACID transactions
• Supports looking back in time
• UPDATE & DELETE existing records
• Schema management options
12#UnifiedDataAnalytics #SparkAISummit

13. Delta Lake
• Files can be backed by
– AWS S3
– Azure Blob Store
– HDFS
• Able to convert datasets between parquet and
delta lake
• Some SQL Support
13#UnifiedDataAnalytics #SparkAISummit

14. Delta Lake - ACID Transactions
Works using a transaction log
• Transaction log tracks state
• Files will not be deleted during read
• Optimistic conflict resolution
14#UnifiedDataAnalytics #SparkAISummit

15. Delta Lake - ACID Transactions
15#UnifiedDataAnalytics #SparkAISummit

16. Delta Lake - ACID Transactions
16#UnifiedDataAnalytics #SparkAISummit

17. Delta Lake - ACID Transactions
17#UnifiedDataAnalytics #SparkAISummit

18. Delta Lake - ACID Transactions
18#UnifiedDataAnalytics #SparkAISummit

19. Delta Lake - ACID Transactions
19#UnifiedDataAnalytics #SparkAISummit

20. Delta Lake - ACID Transactions
20#UnifiedDataAnalytics #SparkAISummit
Update
Insert

21. Delta Lake - ACID Transactions
21#UnifiedDataAnalytics #SparkAISummit

22. Delta Lake - ACID Transactions
22#UnifiedDataAnalytics #SparkAISummit
Defining our dataset
Aliases for merge
Join condition

23. Delta Lake - ACID Transactions
23#UnifiedDataAnalytics #SparkAISummit
Values to update

24. Delta Lake - ACID Transactions
24#UnifiedDataAnalytics #SparkAISummit
If the join condition is not met, insert

25. Delta Lake - ACID Transactions
25#UnifiedDataAnalytics #SparkAISummit

26. Delta Lake - ACID Transactions
26#UnifiedDataAnalytics #SparkAISummit

27. Delta Lake - ACID Transactions
27#UnifiedDataAnalytics #SparkAISummit

28. Delta Lake - ACID Transactions
Delta tracks operations on files
• Not all operations are effective immediately
• New log file is created for each transaction
28#UnifiedDataAnalytics #SparkAISummit

29. Delta Lake - ACID Transactions
29#UnifiedDataAnalytics #SparkAISummit

30. Delta Lake - ACID Transactions
30#UnifiedDataAnalytics #SparkAISummit

31. Delta Lake - ACID Transactions
31#UnifiedDataAnalytics #SparkAISummit

32. Delta Lake - File Management
Cleaning up
• Delta provides VACUUM commands
• VACUUM can be run with a retention period
– Default 7 day retention period
– VACUUM with a low retention period can corrupt
active writers
• VACUUM does not get logged
32#UnifiedDataAnalytics #SparkAISummit

33. Delta Lake - File Management
33#UnifiedDataAnalytics #SparkAISummit

34. Pulling it all together
Structured Streaming
• Leverages many strengths of the Dataframe
API
• Gives a clean way to manage late data
• Makes it manageable to join multiple streams
34#UnifiedDataAnalytics #SparkAISummit

35. Pulling it all together
Delta Lake
• Gives us ACID transactions
• Logs what has taken place
• Requires some file management
35#UnifiedDataAnalytics #SparkAISummit

36. Data Warehousing
There are many ways to model data, let’s stick to
an example:
• Star Schema
• Source is MySQL
• Sink is S3
• Possibilities to export from S3
36#UnifiedDataAnalytics #SparkAISummit

37. Data Warehousing
37#UnifiedDataAnalytics #SparkAISummit

38. 38#UnifiedDataAnalytics #SparkAISummit
Schema

39. Data Warehousing
39#UnifiedDataAnalytics #SparkAISummit

40. Data Warehousing
40#UnifiedDataAnalytics #SparkAISummit

41. Data Warehousing
41#UnifiedDataAnalytics #SparkAISummit
Read stream from Kafka
Value comes in as a binary
Parse the message using a
fixed schema

42. Data Warehousing
42#UnifiedDataAnalytics #SparkAISummit
Parse the MySQL data
Write the stream as Delta
leveraging checkpoints

43. Data Warehouse
Type 2 Dimension
• A valid new record must
– update the previous version
– insert itself as the new version
• Delta merge is the way to go
• Process batches using
foreachBatch
43#UnifiedDataAnalytics #SparkAISummit

44. Data Warehousing
44#UnifiedDataAnalytics #SparkAISummit

45. Data Warehousing
45#UnifiedDataAnalytics #SparkAISummit

46. Data Warehousing
46#UnifiedDataAnalytics #SparkAISummit
foreachBatch method
• Takes a dataframe,
batchId, and more
• You are free to handle
the dataframe as you
see fit

47. Data Warehousing
47#UnifiedDataAnalytics #SparkAISummit
NULL merge key guarantees insert
Join to table to merge into
Filter to most recent records
Select only the batch data and merge key

48. Data Warehousing
48#UnifiedDataAnalytics #SparkAISummit

49. Data Warehousing
49#UnifiedDataAnalytics #SparkAISummit
Match condition, only
match current records
Set the previous latest
record as non-current
Insert all data if there is
no previous iteration

50. Data Warehousing
50#UnifiedDataAnalytics #SparkAISummit

51. Data Warehousing - Gotchas
51#UnifiedDataAnalytics #SparkAISummit
File Management
• Smaller trigger windows mean more files
– More files mean slower reads
• How useful is table history
• File size optimization

52. Data Warehousing - Gotchas
52#UnifiedDataAnalytics #SparkAISummit
Streaming joins
• Watermarks required for stream-to-stream joins
• Be aware of the latency of your streams
• Handle late data beyond watermark
– Set failure conditions for you streaming applications

53. DON’T FORGET TO RATE
AND REVIEW THE SESSIONS
SEARCH SPARK + AI SUMMIT