The document provides an overview of continuous processing in Databricks structured streaming, highlighting features such as the unified Spark SQL API, low latency, and the representation of data as a virtual append-only table. It discusses the differences between continuous processing and microbatches, and outlines the ongoing work for improvements in event time handling, performance testing, and additional data sources. The content also addresses checkpointing and data processing mechanisms within the continuous processing framework.

1. Jose Torres, Databricks
Continuous Processing in
Structured Streaming
#Dev4SAIS

2. Continuous Processing Overview
● Unified Spark SQL API
● No microbatches
● Low (~1ms) latency
2#Dev4SAIS
Continuous Processing Microbatch

3. DStream API
● Non-declarative, similar to RDDs
● Scala/Java only
● Checkpoints only through complete snapshots
● No event time
3#Dev4SAIS

4. Structured Streaming
● Data represented as a virtual append-only table
● Unified Spark SQL query API
● Batch and streaming queries return same
results
4#Dev4SAIS

5. Structured Streaming
5#Dev4SAIS

6. Structured Streaming Features
● Dataframes and Datasets
● SQL, Python, and R language APIs
● Delta-based aggregation state
6#Dev4SAIS

7. Microbatches
7#Dev4SAIS

8. Continuous Processing
8#Dev4SAIS

9. Chandy-Lamport Checkpoints
9#Dev4SAIS
Writer TaskReader
Aggregation
Task
Driver
save checkpoint to state store
epoch marker
partition level commit
checkpoint complete
ready for global commit
Data Stream
● Asynchronous
● Consistent

10. Checkpointing - Detailed
10#Dev4SAIS
AggregationReader Processing Writer
Shuffle
Aggregation WriterReader Processing
Reader Processing
Driver
epoch markers

11. Checkpointing - Detailed
11#Dev4SAIS
AggregationReader Processing Writer
Shuffle
Aggregation WriterReader Processing
Reader Processing

12. Checkpointing - Detailed
12#Dev4SAIS
AggregationReader Processing Writer
Shuffle
Aggregation WriterReader Processing
Reader Processing

13. Checkpointing - Detailed
13#Dev4SAIS
AggregationReader Processing Writer
Shuffle
Aggregation WriterReader Processing
Reader Processing

14. Checkpointing - Detailed
14#Dev4SAIS
AggregationReader Processing Writer
Shuffle
Aggregation WriterReader Processing
Reader Processing

15. Checkpointing - Detailed
15#Dev4SAIS
AggregationReader Processing Writer
Shuffle
Aggregation WriterReader Processing
Reader Processing
aggregation checkpoint
aggregation checkpoint

16. Checkpointing - Detailed
16#Dev4SAIS
AggregationReader Processing Writer
Shuffle
Aggregation WriterReader Processing
Reader Processing
commit partition writer
commit partition writer
commit partition writer
Driver
ready for global commit

17. Continuous Processing API
● It’s just Structured Streaming
● Run the same queries in continuous mode
17#Dev4SAIS

18. Continuous Processing in 2.3
● Initial experimental release
● Supports ETL use cases
18#Dev4SAIS

19. Ongoing And Future Work
● Shuffles (SPARK-24036)
● Event time (SPARK-24459)
● Metrics (SPARK-23887)
● Exactly-once semantics mode (SPARK-24460)
● Performance testing (TBD)
● Additional data sources (TBD)
19

20. Q&A
20