Arbitrary Stateful Aggregation and MERGE INTO - Data + AI Summit EU 2020
Report
Share
•0 likes•572 views
Arbitrary Stateful Aggregation and MERGE INTO - Data + AI Summit EU 2020
•0 likes•572 views
Report
Share
Download to read offline
Jules Damji and Denny Lee from Databricks Developer Relations will recap some keynote highlights, and each will briefly present personal picks from sessions that resonated well with them. Next, Jacek Laskowski, an independent consultant, will speak about Spark 3.0 internals, and Scott Haines from Twilio, Inc. will give a talk about structured streaming microservice architectures. This live coding session and technical deep dive are not to be missed!
Recommended
More Related Content
What's hot
What's hot(20)
Similar to Arbitrary Stateful Aggregation and MERGE INTO - Data + AI Summit EU 2020
Similar to Arbitrary Stateful Aggregation and MERGE INTO - Data + AI Summit EU 2020(20)
More from Databricks
More from Databricks(20)
Recently uploaded
![[DSC Europe 23][AI:CSI] Aleksa Stojanovic - Applying AI for Threat Detection ...](https://cdn.slidesharecdn.com/ss_thumbnails/jytahfl2tqyatpbxfkpe-aleksa-stojanovic-applying-ai-for-threat-detection-in-the-world-of-cybersecurit-231130130512-290ee1ec-thumbnail.jpg?width=384&fit=bounds)
Recently uploaded(20)
![[DSC Europe 23][AI:CSI] Aleksa Stojanovic - Applying AI for Threat Detection ...](https://cdn.slidesharecdn.com/ss_thumbnails/jytahfl2tqyatpbxfkpe-aleksa-stojanovic-applying-ai-for-threat-detection-in-the-world-of-cybersecurit-231130130512-290ee1ec-thumbnail.jpg?width=768&fit=bounds)
![[DSC Europe 23][AI:CSI] Dragan Pleskonjic - AI Impact on Cybersecurity and P...](https://cdn.slidesharecdn.com/ss_thumbnails/sq09ljvgsbqiyv2sraq9-dragan-pleskonjic-ai-impact-on-cybersecurity-and-privacy-231130130513-51a0f434-thumbnail.jpg?width=768&fit=bounds)
Arbitrary Stateful Aggregation and MERGE INTO - Data + AI Summit EU 2020
- 1. Arbitrary Stateful Aggregation and MERGE INTO Spark Structured Streaming + Delta Lake = “Double Metrics” Jacek Laskowski jaceklaskowski / November 2020
- 2. About the Speaker Jacek Laskowski is an IT Freelancer specializing in Apache Spark, Delta Lake, Apache Kafka and Kafka Streams. Contact me at jacek@japila.pl or DM on twitter @jaceklaskowski to discuss opportunities. Best known by "The Internals Of" online books @ https://books.japila.pl
- 3. The Internals of Delta Lake 1. Available for free @ https://books.japila.pl/delta-lake-internals
- 4. Friendly Reminder Should you have any questions, Feel free to ask them in the chat window. I’m going to answer them at the end of the talk. Thank you!
- 5. Client Requirements and Recommendations 1. A client wants to load Kafka records at regular intervals ● Spark Structured Streaming 2. A client wants to do a stateful aggregation in a custom per-group way ● KeyValueGroupedDataset.flatMapGroups WithState 3. A client wants to update a Delta table with aggregation results ● MERGE INTO ● DataStreamWriter.foreachBatch
- 6. Arbitrary Stateful Aggregation 1. KeyValueGroupedDataset.flatMapGroupsWithState (scaladoc) 2. A user-defined per-group state 3. For a static batch Dataset, the function will be invoked once per group 4. For a streaming Dataset, the function will be invoked for each group repeatedly in every trigger, and updates to each group's state will be saved across invocations
- 7. The Code 1. Code?! Open Intellij IDEA! 😎
- 8. Delta Lake Users Mailing List 1. Multiple executions of flatMapGroupsWithState when DeltaTable.merge
- 9. Possible Way-Outs (“Solutions”) 1. Separate Delta table for state? a. Avoid multiple passes over flatMapGroupsWithState
- 10. O’Reilly Learning Spark 2nd Edition 1. Available for free @ https://dbricks.co/get-ebook 2. Chapter 9 “Building Reliable Data Lakes with Apache Spark” touches Delta Lake a. Also the competitors: Apache Hudi and Apache Iceberg
- 11. That’s all folks! Thank you! ❤ /me Answering questions... Jacek Laskowski / @jaceklaskowski / jacek@japila.pl