Download to read offline
Uploaded bySigmoid
Structured Streaming Using Spark 2.1
Structured Streaming is a scalable and fault-tolerant stream processing engine in Spark SQL that allows expressing streaming computations like batch computations. It includes three output modes: append, complete, and update, allowing different ways of handling results as new data arrives. The implementation is still in alpha stage in Spark 2.1, with experimental APIs for streaming aggregation and event-time windows.
More Related Content
Similar to Structured Streaming Using Spark 2.1
Structured Streaming Using Spark 2.1
- 1. Structured Streaming Using Spark2.1 BY YASWANTH YADLAPALLI
- 2. What is StructuredStreaming? Structured Streaming is a scalable and fault-tolerant stream processing engine built on the Spark SQL engine. Express streaming computation the same way as a batch computation on static data. The Spark SQL engine takes care of running it incrementally and continuously and updating the final result as streaming data continues to arrive. Use the Dataset/DataFrame API to express streaming aggregations, event-time windows, stream-to-batch joins, etc. Structured Streaming is still ALPHA in Spark 2.1 and the APIs are still experimental
- 3.
- 4. Output modes Thereare 3 output modes in Spark Structured streaming Append mode - Only the new rows added to the Result Table since the last trigger will be outputted to the sink. This is supported for only those queries where rows added to the Result Table is never going to change Complete mode - The whole Result Table will be outputted to the sink after every trigger. This is supported for aggregation queries. Update mode - Only the rows in the Result Table that were updated since the last trigger will be outputted to the sink.
- 5.
- 6. Windowing import spark.implicits._ val words= ... // streaming DataFrame of schema { timestamp: Timestamp, word: String } // Group the data by window and word and compute the count of each group val windowedCounts = words.groupBy( window($"timestamp", "10 minutes", "5 minutes"), $"word" ).count()
- 7.
- 8.
- 9.
- 10.
Editor's Notes
- #5 Currently spark supports the following output sinks File sink Foreach sink Console sink (for debugging) Memory Sink Append Mode for parquet writing and in general file sinks
- #6 Imagine our quick example is modified and the stream now contains lines along with the time when the line was generated. Instead of running word counts, we want to count words within 10 minute windows, updating every 5 minutes. That is, word counts in words received between 10 minute windows 12:00 - 12:10, 12:05 - 12:15, 12:10 - 12:20, etc
- #8 Now consider what happens if one of the events arrives late to the application. For example, say, a word generated at 12:04 (i.e. event time) could be received received by the application at 12:11. The application should use the time 12:04 instead of 12:11 to update the older counts for the window 12:00 - 12:10. This occurs naturally in our window-based grouping – Structured Streaming can maintain the intermediate state for partial aggregates for a long period of time such that late data can update aggregates of old windows correctly, as illustrated below.