In this talk, we will introduce some of the new available APIs around stateful aggregation in Structured Streaming, namely flatMapGroupsWithState. We will show how this API can be used to power many complex real-time workflows, including stream-to-stream joins, through live demos using Databricks and Apache Kafka.

1. Arbitrary Stateful Aggregations
using Structured Streaming
in Apache Spark™
Burak Yavuz
5/16/2017

2. 2
Outline
• Structured Streaming Concepts
• Stateful Processing in Structured Streaming
• Use Cases
• Demos

3. 3
The simplest way to perform streaming analytics
is not having to reason about streaming at all

4. 4

5. 5
Input: data from source as an
append-only table
Trigger: how frequently to check
input for new data
Query: operations on input
usual map/filter/reduce
new window, session ops
Trigger: every 1 sec
1 2 3
Time
data up
to 1
Input data up
to 2
data up
to 3
Query
New Model

6. 6
Trigger: every 1 sec
1 2 3
result
for data
up to 1
Result
Query
Time
data up
to 1
Input data up
to 2
result
for data
up to 2
data up
to 3
result
for data
up to 3
Output
[complete mode]
output all the rows in the result table
New Model
Result: final operated table
updated every trigger interval
Output: what part of result to
write to data sink after every
trigger
Complete output: Write full result
table every time

7. 7
Trigger: every 1 sec
1 2 3
result
for data
up to 1
Result
Query
Time
data up
to 1
Input data up
to 2
result
for data
up to 2
data up
to 3
result
for data
up to 3
Output
[append mode]
output only new rows since
last trigger
Result: final operated table updated
every trigger interval
Output: what part of result to write
to data sink after every trigger
Complete output: Write full result table
every time
Append output: Write only new rows that got
added to result table since previous batch
*Not all output modes are feasible with all queries
New Model

8. 8

9. 9
Output Modes
• Append mode (default) - New rows added to the Result Table
since the last trigger will be outputted to the sink. Rows will be
output only once, and cannot be rescinded.
Example use cases: ETL

10. 10
Output Modes
• Complete mode - The whole Result Table will be outputted to
the sink after every trigger. This is supported for aggregation
queries.
Example use cases: Monitoring

11. 11
Output Modes
• Update mode - (Available since Spark 2.1.1) Only the rows in the
Result Table that were updated since the last trigger will be
outputted to the sink.
Example use cases: Alerting, Sessionization

12. 12
Outline
• Structured Streaming Concepts
• Stateful Processing in Structured Streaming
• Use Cases
• Demos

13. 13
Event time Aggregations
Many use cases require aggregate statistics by event time
E.g. what's the #errors in each system in 1 hour windows?
Many challenges
Extracting event time from data, handling late, out-of-order data
DStream APIs were insufficient for event time operations

14. 14
Event time Aggregations
Windowing is just another type of grouping in Struct.
Streaming
number of records every hour
parsedData
.groupBy(window("timestamp","1 hour"))
.count()
parsedData
.groupBy(
"device",
window("timestamp","10 mins"))
.avg("signal")
avg signal strength of each
device every 10 mins
Use built-in functions to extract event-time
No need for separate extractors

15. 15
Advanced Aggregations
Powerful built-in
aggregations
Multiple simultaneous
aggregations
Custom aggs using
reduceGroups, UDAFs
parsedData
.groupBy(window("timestamp","1 hour"))
.agg(avg("signal"), stddev("signal"), max("signal"))
variance, stddev, kurtosis, stddev_samp, collect_list,
collect_set, corr, approx_count_distinct, ...
// Compute histogram of age by name.
val hist = ds.groupBy(_.type).mapGroups {
case (type, data: Iter[DeviceData]) =>
val buckets = new Array[Int](10)
data.map(_.signal).foreach { a => buckets(a/10)+=1 }
(type, buckets)
}

16. 16
Stateful Processing for Aggregations
In-memory,
streaming state
maintained for
aggregations
12:00 - 13:00 1 12:00 - 13:00 3
13:00 - 14:00 1
12:00 - 13:00 3
13:00 - 14:00 2
14:00 - 15:00 5
12:00 - 13:00 5
13:00 - 14:00 2
14:00 - 15:00 5
15:00 - 16:00 4
12:00 - 13:00 3
13:00 - 14:00 2
14:00 - 15:00 6
15:00 - 16:00 4
16:00 - 17:00 3
13:00 14:00 15:00 16:00 17:00
Keeping state allows late data to
update counts of old windows
But size of the state increases
indefinitely if old windows not dropped
red = state updated
with late data

17. 17

18. 18
Watermarking and Late Data
Watermark [Spark 2.1] - a
moving threshold that trails
behind the max seen event time
Trailing gap defines how late
data is expected to be
event time
max event time
watermark data older
than
watermark
not expected
12:30 PM
12:20 PM
trailing gap
of 10 mins

19. 19
Watermarking and Late Data
Data newer than watermark may
be late, but allowed to aggregate
Data older than watermark is "too
late" and dropped
State older than watermark
automatically deleted to limit the
amount of intermediate state
max event time
event time
watermark
late data
allowed to
aggregate
data too
late,
dropped

20. 20
Watermarking and Late Data
max event time
event time
watermark
allowed
lateness
of 10 mins
parsedData
.withWatermark("timestamp", "10 minutes")
.groupBy(window("timestamp","5 minutes"))
.count()
late data
allowed to
aggregate
data too
late,
dropped
Control the tradeoff between state
size and lateness requirements
Handle more late à keep more state
Reduce state à handle less lateness

21. 21
Watermarking to Limit State [Spark 2.1]
data too late,
ignored in counts,
state dropped
Processing Time12:00
12:05
12:10
12:15
12:10 12:15 12:20
12:07
12:13
12:08
EventTime
12:15
12:18
12:04
watermark updated to
12:14 - 10m = 12:04
for next trigger,
state < 12:04 deleted
data is late, but
considered in counts
parsedData
.withWatermark("timestamp", "10 minutes")
.groupBy(window("timestamp","5 minutes"))
.count()
system tracks max
observed event time
12:08
wm = 12:04
10min
12:14
More details in blog post!

22. 22

23. 23
Working With Time
df.withWatermark("timestampColumn", "5 hours")
.groupBy(window("timestampColumn", "1 minute"))
.count()
.writeStream
.trigger("10 seconds")
Separate processing details (output rate, late data tolerance)
from query semantics.

24. 24
Working With Time
df.withWatermark("timestampColumn", "5 hours")
.groupBy(window("timestampColumn", "1 minute"))
.count()
.writeStream
.trigger("10 seconds")
How to group
data by time
Same in streaming & batch

25. 25
Working With Time
df.withWatermark("timestampColumn", "5 hours")
.groupBy(window("timestampColumn", "1 minute"))
.count()
.writeStream
.trigger("10 seconds")
How late
data can be

26. 26
Working With Time
df.withWatermark("timestampColumn", "5 hours")
.groupBy(window("timestampColumn", "1 minute"))
.count()
.writeStream
.trigger("10 seconds")
How often
to emit updates

27. 27
Arbitrary Stateful Operations [Spark 2.2]
mapGroupsWithState
allows any user-defined
stateful ops to a
user-defined state
Direct support for per-key
timeouts in event-time or
processing-time
supports Scala and Java
ds.groupByKey(groupingFunc)
.mapGroupsWithState
(timeoutConf)
(mappingWithStateFunc)
def mappingWithStateFunc(
key: K,
values: Iterator[V],
state: GroupState[S]): U = {
// update or remove state
// set timeouts
// return mapped value
}

28. 28
flatMapGroupsWithState
• Applies the given function to each group of data, while
maintaining a user-defined per-group state
• Invoked once per group in batch
• Invoked each trigger (with the existence of data) per group in
streaming
• Requires user to provide an output mode for the function

29. 29
flatMapGroupsWithState
• mapGroupsWithState is a special case with
• Output mode: Update
• Output size: 1 row per group
• Supports both Processing Time and Event Time timeouts

30. 30
Outline
• Structured Streaming Concepts
• Stateful Processing in Structured Streaming
• Use Cases
• Demos

31. 31
Alerting
val monitoring = stream
.as[Event]
.groupBy(_.id)
.flatMapGroupsWithState(Append, GST.ProcessingTimeTimeout) {
(id: Int, events: Iterator[Event], state: GroupState[…]) =>
...
}
.writeStream
.queryName("alerts")
.foreach(new PagerdutySink(credentials))
Monitor a stream using custom stateful logic with timeouts.

32. 32
Sessionization
val monitoring = stream
.as[Event]
.groupBy(_.session_id)
.mapGroupsWithState(GST.EventTimeTimeout) {
(id: Int, events: Iterator[Event], state: GroupState[…]) =>
...
}
.writeStream
.parquet("/user/sessions")
Analyze sessions of user/system behavior

33. 33
Demo

34. 34
SPARK SUMMIT 2017
DATA SCIENCE AND ENGINEERING AT SCALE
JUNE 5 – 7 | MOSCONE CENTER | SAN FRANCISCO
ORGANIZED BY spark-summit.org/2017
Discount Code: Databricks

35. We are hiring!
https://databricks.com/company/careers

36. Thank You
“Does anyone have any questions for my answers?” - Henry Kissinger