This talk will cover the details of Continuous Processing in Structured Streaming and my work implementing the initial version in Spark 2.3 as well as the updates for 2.4. DStreams was Spark’s first attempt at streaming, and through dstream Spark became the first framework to provide both batch and streaming functionalities in one unified execution engine. The way streaming execution happens is through this “micro-batch” model, in which the underlying execution engine simply runs on batches of data over and over again. Dstream’s design tightly couples the user-facing APIs with the execution model, and as a result was very difficult to accomplish certain tasks important in streaming, e.g. using event time and working with late data, without breaking the user-facing APIs. Structured Streaming was the 2nd (and the latest) major streaming effort in Spark. Its design decouples the frontend (user-facing APIs) and backend (execution), and allows us to change the execution model without any user API change. However, the (historical) minimum possible latency for any record for DStreams or Structured Streaming was bounded by the amount of time that it takes to launch a task. This limitation is a result of the fact that the engine requires us to know both the starting and the ending offset, before any tasks are launched. In the worst case, the end-to-end latency is actually closer to the average batch time + task launching time. Continuous Processing removes this constraints and allows users to achieve sub-millisecond end-to-end latencies with the new execution engine. This talk will take a technical deep dive into its capabilities, what it took to implement, and discuss the future developments.

1. Jose Torres, Databricks
Continuous Processing in
Structured Streaming
#Dev4SAIS

2. Continuous Processing Overview
● Unified Spark SQL API
● No microbatches
● Low (~1ms) latency
2#Dev4SAIS
Continuous Processing Microbatch

3. DStream API
● Non-declarative, similar to RDDs
● Scala/Java only
● Checkpoints only through complete snapshots
● No event time
3#Dev4SAIS

4. Structured Streaming
● Data represented as a virtual append-only table
● Unified Spark SQL query API
● Batch and streaming queries return same
results
4#Dev4SAIS

5. Structured Streaming
5#Dev4SAIS

6. Structured Streaming Features
● Dataframes and Datasets
● SQL, Python, and R language APIs
● Delta-based aggregation state
6#Dev4SAIS

7. Microbatches
7#Dev4SAIS

8. Continuous Processing
8#Dev4SAIS

9. Chandy-Lamport Checkpoints
9#Dev4SAIS
Writer TaskReader
Aggregation
Task
Driver
save checkpoint to state store
epoch marker
partition level commit
checkpoint complete
ready for global commit
Data Stream
● Asynchronous
● Consistent

10. Checkpointing - Detailed
10#Dev4SAIS
AggregationReader Processing Writer
Shuffle
Aggregation WriterReader Processing
Reader Processing
Driver
epoch markers

11. Checkpointing - Detailed
11#Dev4SAIS
AggregationReader Processing Writer
Shuffle
Aggregation WriterReader Processing
Reader Processing

12. Checkpointing - Detailed
12#Dev4SAIS
AggregationReader Processing Writer
Shuffle
Aggregation WriterReader Processing
Reader Processing

13. Checkpointing - Detailed
13#Dev4SAIS
AggregationReader Processing Writer
Shuffle
Aggregation WriterReader Processing
Reader Processing

14. Checkpointing - Detailed
14#Dev4SAIS
AggregationReader Processing Writer
Shuffle
Aggregation WriterReader Processing
Reader Processing

15. Checkpointing - Detailed
15#Dev4SAIS
AggregationReader Processing Writer
Shuffle
Aggregation WriterReader Processing
Reader Processing
aggregation checkpoint
aggregation checkpoint

16. Checkpointing - Detailed
16#Dev4SAIS
AggregationReader Processing Writer
Shuffle
Aggregation WriterReader Processing
Reader Processing
commit partition writer
commit partition writer
commit partition writer
Driver
ready for global commit

17. Continuous Processing API
● It’s just Structured Streaming
● Run the same queries in continuous mode
17#Dev4SAIS

18. Continuous Processing in 2.3
● Initial experimental release
● Supports ETL use cases
18#Dev4SAIS

19. Ongoing And Future Work
● Shuffles (SPARK-24036)
● Event time (SPARK-24459)
● Metrics (SPARK-23887)
● Exactly-once semantics mode (SPARK-24460)
● Performance testing (TBD)
● Additional data sources (TBD)
19

20. Q&A
20