Kostas Kloudas presented on stateful stream processing with Apache Flink. He discussed how Flink handles state management, fault tolerance, and time semantics to allow for continuous and accurate processing of streaming data. Flink embeds local state with keyed streams, takes consistent snapshots of distributed state, and uses watermarks to process events in event time to produce correct results even for out-of-order data. This allows Flink to provide a robust stream processing engine that scales to large deployments.

1. 1
Kostas Kloudas
@kkloudas
HUG @ Warsaw
JULY 3, 2017
Stateful Stream Processing
with Apache Flink®

2. 2
Original creators of Apache
Flink®
Providers of the
dA Platform

3. 3 questions and some history
▪ What is stateful stream processing?
▪ Why care about it?
▪ How does Flink do it?
▪ The evolution of Flink.
3

4. Stateful Stream Processing
4

5. Stateful Stream Processing
5
Continuous Processing
for Continuously Arriving Data

6. 6
Batch
jobs
t
2017-06-14
01:00am
2017-06-14
00:00am
2017-06-13
11:00pm
2017-06-13
10:00pm
... ● Continuously ingesting
data
● Time-bounded batch
files
● Periodic batch jobs
The ol’ traditional batch way

7. 7
intermediate
state
t
2017-06-14
01:00am
2017-06-14
00:00am
2017-06-13
11:00pm
2017-06-13
10:00pm
...
● Compute a counter:
#(A) per hour / 2 min
● What if:
● interval crosses batch
boundaries?
→ carry intermediate
results to next batch
● events out of order?
→ ???
The ol’ traditional batch way

8. ▪ So, for a simple counting program:
• Custom logic for handling state
• Custom logic for handling time
• Custom logic for fault tolerance
8
The ol’ traditional batch way

9. ▪ So, for a simple counting program:
• Custom logic for handling state
• Custom logic for handling time
• Custom logic for fault tolerance
9
The ol’ traditional batch way
Difficult and has nothing to do with your
program.

10. Why should we care?
▪...this is just for continuous data, right?
10

11. Why should we care?
▪...this is just for continuous data, right?
11
Most datasets are
continuously arriving streams.

12. Stream Processing
12
Computation over an endless stream of data
Your
Code
...

13. Distributed Stream Processing
13
Your
Code
...
...
...
Your
Code
Your
Code
● Partitions input by some key
● Distributes computation
across multiple instances
● Each instance is responsible
for some keys

14. qwe
Stateful Stream Processing
14
...
...
Your
Code
Your
Code
update local
variables/structures
var x = …
if (condition(x)) {
…
}

15. Stateful Stream Processing
15
...
...
Your
Code
Your
Code
qwe
update local
variables/structures
var x = …
if (condition(x)) {
…
}
● Embedded local state
● State co-partitioned with
the input stream by key

16. A practical stream processor
16
state
●Fault-tolerance
●Scalability
●Efficiency
●Event-time
(out-of-order events)
●Allows you to work in
event-time (e.g. timers)
time

17. 17
Stateful Stream Processor
that handles
consistently, robustly, and efficiently
Large
Distributed State
Time / Order /
Completeness
● Stateful stream processing as
a new paradigm to
continuously process
continuously arriving data
● Produce accurate results
● Real-time is only a natural
consequence of the model
A practical stream processor

18. This is where Flink shines...
▪ Supports out-of-order streams
▪ Manages state transparently
• exactly-once processing
▪ Offers high throughput and low latency
▪ Scales to large deployments
• https://data-artisans.com/blog/blink-flink-alibaba-search
• https://data-artisans.com/blog/rbea-scalable-real-time-analytics-at-king
18

19. Apache Flink®
19

20. About time ...
20
...
...
Your
Code
Your
Code
When are my results complete?

21. 21
...
...
Your
Code
Your
Code
When are my results complete?
Processing Time drawbacks:
• Incorrect results
• Irreproducible results
About time ...

22. About time ...
22

23. Event Time: Watermarks
23
● Special markers,
called Watermarks
● Flow with elements
● A watermark of
timestamp t means
that no records with
timestamp < t should
be expected

24. Event Time: Watermarks
24

25. 25
Documentation:
https://ci.apache.org/projects/flink/flink-docs-release-
1.3/dev/event_time.html
Event Time

26. Fault tolerance
▪ How to ensure exactly-once semantics?
26

27. Fault tolerance simple case
27
event log
single process
main memoryperiodically take a
Snapshot of the memory

28. 28
event log
single process
main memoryRecovery
restore snapshot and replay
events since snapshot
persists events
(temporarily)
Fault tolerance simple case

29. Fault tolerance distributed
▪ How to create consistent snapshots of
distributed state?
▪ How to do it efficiently?
29

30. Distributed Snapshots
30
Coordination via markers, injected into the streams

31. 31
State index
(Hash Table
or
RocksDB)
Events flow without replication or synchronous writes
stateful
operation
source
Distributed Snapshots

32. 32
Trigger checkpoint
Inject checkpoint barrier
stateful
operation
source
Distributed Snapshots

33. 33
stateful
operation
source
Take state snapshot
Trigger state
copy-on-write
Distributed Snapshots

34. 34
stateful
operation
source
DFS
Durably persist
snapshots
asynchronously
Processing pipeline continues
Distributed Snapshots

35. 35
...
Your
Code
Your
Code
Your
Code
State
State
State
Your
Code
State
● Consistent snapshotting:
Fault tolerance

36. 36
...
Your
Code
Your
Code
Your
Code
State
State
State
Your
Code
State
checkpointed
state
checkpointed
state
checkpointed
state
File System
Checkpoint
● Consistent snapshotting:
Fault tolerance

37. 37
...
Your
Code
Your
Code
Your
Code
State
State
State
Your
Code
State
checkpointed
state
checkpointed
state
checkpointed
state
File System
Restore
● Recover all embedded state
● Reset position in input stream
Fault tolerance

38. 38
Documentation:
https://ci.apache.org/projects/flink/flink-docs-release-
1.3/internals/stream_checkpointing.html
Fault tolerance

39. State Management: misc.
39
▪ Savepoints
▪ Rescaling
▪ Queryable State

40. Apache Flink Ecosystem
40
Integration
POSIX Java/Scala
Collections
POSIX

41. Apache Flink Stack
41
DataStream API
Stream Processing
DataSet API
Batch Processing
Runtime
Distributed Streaming Data Flow
Libraries
Streaming and batch as first class citizens.

42. Levels of abstraction
42
Process Function (events, state, time)
DataStream API (streams, windows)
Table API (dynamic tables)
Stream SQL
low-level (stateful
stream processing)
stream processing &
analytics
declarative DSL
high-level langauge

43. API and Execution
43
Source
DataStream<String> lines = env.addSource(new FlinkKafkaConsumer010(…));
DataStream<Event> events = lines.map(line -> parse(line));
DataStream<Statistic> stats = stream
.keyBy("id")
.timeWindow(Time.seconds(5))
.sum(new MyAggregationFunction());
stats.addSink(new BucketingSink(path));
map()
[1]
keyBy()/
window()/
apply()
[1]
Transformation
Transformation
Sink
Streaming
DataflowkeyBy()/
window()/
apply()
[2]
map()
[1]
map()
[2]
Source
[1]
Source
[2]
Sink
[1]

44. Evolution of Flink
44

45. Programming APIs
45

46. Large State Handling
46

47. Conclusion
47

48. TL;DR
▪ Stateful stream processing as a paradigm for
continuous data processing
▪ Flink is a sophisticated and tested stateful stream
processor
▪ Efficiency, management, and operational issues for
state are taken very seriously
48

49. 4
Thank you!
@kkloudas
@ApacheFlink
@dataArtisans

50. 50
Stream Processing
and Apache Flink®'s
approach to it
@StephanEwen
Apache Flink PMC
CTO @ data ArtisansFLINK FORWARD IS COMING BACK TO BERLIN
SEPTEMBER 11-13, 2017
BERLIN.FLINK-FORWARD.ORG -

51. We are hiring!
data-artisans.com/careers