Spark Streaming 2.0 introduces Structured Streaming which addresses some areas for improvement in Spark Streaming 1.X. Structured Streaming builds streaming queries on the Spark SQL engine, providing implicit benefits like extending the primary batch API to streaming and gaining an optimizer. It introduces a more seamless API between batch and stream processing, supports event time semantics, and provides end-to-end fault tolerance guarantees through checkpointing. Structured Streaming also aims to simplify streaming application development by managing streaming queries and allowing continuous queries to be started, stopped, and modified more gracefully.

1. © 2015 IBM Corporation1
! Agenda
- Spark Streaming 1.X
• Features
• Areas for Improvement
- Spark Streaming 2.0 – Structured Streaming
• Addressing the Improvement Areas
• API
• Fault Tolerance
• Event Time
• Managing Streaming queries
- Structured Streaming Examples
https://github.com/agsachin/spark-meetup/tree/master/sparkStructuredStreaming
- Summary thoughts

2. © 2015 IBM Corporation2
Spark Streaming 1. X
! Features of Spark Streaming
- High Level API (stateful, joins, aggregates, windows etc.)
• Overlap with RDD API (batch)
- Fault – Tolerant (exactly once semantics achievable)
- Back Pressure
- Deep Integration with Spark Ecosystem (MLlib, SQL, GraphX etc.)
!
Apache
Hadoop
Day
2015

3. © 2015 IBM Corporation3
Spark Streaming 1. X – Areas of improvement
! Fault-tolerance
For end-2-end exactly once guarantees, user needs to do all the heavy lifting in
the Sink
Can that be handled in a very simple way for the end-user ?
Apache
Hadoop
Day
2015

4. © 2015 IBM Corporation4
Fault-Tolerant Semantics
Exactly
Once,
If
Outputs
are
Idempotent
or
transac6onal
Exactly
Once,
as
long
as
received
data
is
not
lost
Exactly
Once
needs
re-­‐playable
sources
(e.g.
Ka?a
Direct)
Source
Receiver
Transforming
Outputting
Sink

5. © 2015 IBM Corporation5
Spark Streaming 1. X – Areas of improvement
! Fault-tolerance
- For end-2-end exactly once guarantees, user needs to do all the heavy lifting in the Sink
! API
- Request for more seamless API between Batch & Stream
- Reduce complexities of streaming app *
! No Event Time support
- Hard to support when processing time/batch time exposed in externals
! Streaming Query Management
! Micro-batch
!
Apache
Hadoop
Day
2015

6. © 2015 IBM Corporation6
Spark Streaming 2.0 API
! Built on top of Spark SQL Engine
! Implicit Benefits
- Extend the primary Batch API even to Streaming
- Gain an Optimizer and all other enhancements done in SparkSQL.
! Challenge
- Remove/Keep streaming complexities to minimum
!

7. © 2015 IBM Corporation7
Lets Dive in

8. © 2015 IBM Corporation8
SQL Batch vs SQL Streaming- Conceptually

9. © 2015 IBM Corporation9
Batch vs Streaming - Programmatically

10. © 2015 IBM Corporation10
Output Modes - Sink
! Defined as what gets written from the Result table to external storage (Sink)
! Output modes
- Complete – Entire updated Result table is written to external storage.
- Append – Only new rows added in the Result table since last incremental query execution is
written to external storage.
- Update - Only the rows updated in the Result table since last incremental query execution is
written to external storage.
Upto implementation of Storage connector to decide how to write.
* Aggregate queries only support complete mode and non-aggregate queries append mode

11. © 2015 IBM Corporation11
Supported Sinks & Modes in 2.0
*DEBUG
ONLY
*DEBUG
ONLY

12. © 2015 IBM Corporation12
Windowing in Structured Streaming

13. © 2015 IBM Corporation13
Window operations
! Continuous time based aggregations are most common in Streaming applications.
- Sliding window & Tumbling window
E.g. Top x hashtags on Twitter in last half hour, every 5 minutes
! New function that treats windowing as a regular aggregation
! Used in a Group By clause
Can be used in Batch as well

14. © 2015 IBM Corporation14
Event Time Windows
! Event-Time is time embedded within the data itself
It is not the time Spark received the data
! What about processing time windows if you want them

15. © 2015 IBM Corporation15
Handling Late Arrival in Event-Time
! Since the ‘Result’ table is updated by Spark, the late data is put in its correct
window group
! Use a normal filter in the SQL ?
! Watermarks

16. © 2015 IBM Corporation16
Fault Tolerance
! Why Care?
! Different guarantees for Data Loss
! Atleast Once
! Exactly Once
! What all can fail?
! Driver
! Executor

17. © 2015 IBM Corporation17
Spark 1.x Best Fault tolerance - Kafka Direct API
• Simplified Parallelism
• Less Storage Need
• Exactly Once Semantics.
source & processing
Benefits
of
this
approach

18. © 2015 IBM Corporation18
Fault Tolerance in Structured Streaming
Active
Driver
Checkpoint
to
HDFS
! Structured Streaming Checkpointing
Decided Offsets ranges for a trigger interval is logged to checkpoint Directory *before* any
processing is started for that trigger
Nth record in log indicates data that is currently being processed
N-1 entry in log indicates offsets idempotent written to Sink
Log entries are monotonically increasing integers
! On Recovery
Restart processing of nth entry in WAL

19. © 2015 IBM Corporation19
Fault Tolerance in Structured Streaming
! End-to-End Exactly Once guarantees with
- idempotent Sinks (built-in for commonly used sinks e.g. Files / JDBC)
- Built-in Sources will *mostly* be only ones that support replay
https://issues.apache.org/jira/browse/SPARK-15842

20. © 2015 IBM Corporation20
Managing Streaming Queries
! Streaming in 1.x was definetly lacking in
- Starting / Stopping individual Streaming Queries
- Changing the computation done in a Query.
- When a Streaming Query abnormally terminates handle more gracefully than app crash.

21. © 2015 IBM Corporation21
Managing Streaming Queries

22. © 2015 IBM Corporation22
Managing Streaming Queries

23. © 2015 IBM Corporation23
Summary
! Overall has a good set of features
- Easier code share between Batch and Streaming (No different type hierarchies)
- Window not tied to Batch interval
- No Streaming context
- Optimizer now available for your queries.
! Getting started
- Combining of 3 things (Output Mode & Sink Type & Query type) needs some time to wrap your head around *
And not much control over those.
- Only get Runtime exceptions when you mess with above
! How does it compare to Apache Beam ?

24. © 2015 IBM Corporation24
For Each Sink

25. © 2015 IBM Corporation25
Thank YOU