The document discusses the rethinking of the Apache Flink stack and APIs to unify batch and streaming processing. It highlights the challenges of maintaining separate APIs and processing styles, as well as the benefits of creating a unified system to improve efficiency and ease of use. Future improvements include a new unified source interface and enhancements to existing components to better support both data processing paradigms.

1. © 2019 Ververica
Aljoscha Krettek – Software Engineer, Flink PMC
Stephan Ewen – CTO, Flink PMC/Chair
Towards Flink 2.0: Rethinking the Stack and APIs
to unify Batch & Streaming

2. © 2019 Ververica2
This is joint work with many members of
the Apache Flink community
Timo, Dawid, Shaoxuan, Kurt, Guowei, Becket, Jincheng, Fabian, Till, Andrey, Gary, Piotr, Stefan, etc.
And many others …

3. © 2019 Ververica3
Some of this presents work that is in
progress in the Flink community. Other
things are planned and/or have design
documents. Some were discussed at one
point or another on the mailing lists or in
person.
This represents our understaning of the
current state, this is not a fixed roadmap,
Flink is an open-source Apache project.

4. © 2019 Ververica
Batch and Streaming

5. © 2019 Ververica5
Everything Is a Stream

6. © 2019 Ververica6
What changes faster? Data or Query?
ad-hoc queries, data exploration,
ML training and
(hyper) parameter tuning
continuous applications,
data pipelines, standing queries,
anomaly detection, ML evaluation, …
Data changes slowly
compared to fast
changing queries
Data changes fast
application logic
is long-lived
“batch” “streaming”

7. © 2019 Ververica7
Latency vs. Completeness (for geeks)
1977 1980 1983 1999 2002 2005 2015
Processing Time
Episode
IV
Episode
V
Episode
VI
Episode
I
Episode
II
Episode
III
Episode
VII
Event Time
2016
Rogue
One
III.5
2017
Episode
VIII

8. © 2019 Ververica8
Latency vs. Completeness (more formally)
*from the excellent Streaming 101 by Tyler Akidau:
https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-

9. © 2019 Ververica9
Latency vs. Completeness
Bounded/
Batch
Unbounded/
Streaming
Data is as complete
as it gets within that
Batch Job
No fine latency control
Trade of latency
versus completeness
Processing-time
timers for latency
control

10. © 2019 Ververica10
What does this mean for processing?
older more recent
files,
message
bus, or …

11. © 2019 Ververica11
Stream-style processing
older more recent
watermark
unprocesse
d

12. © 2019 Ververica12
Batch-style processing
older more recent
watermark
unprocesse
d

13. © 2019 Ververica13
Batch and Streaming Processing Styles
S S S
M M M
R R R
S S S
S S S
M M M
R R R
S S S
more batch-y more stream-y
running
not running
can do things one-by-one everything is always-on
running

14. © 2019 Ververica14
So this is the reason why we have
different APIs, different batch and
stream processing systems?
• Different requirements
• Optimization potential for batch and
streaming
• Also: historic developments and slow-
changing organizations
💡

15. © 2019 Ververica
Current Stack

16. © 2019 Ververica16
Runtime
Tasks / JobGraph / Network Interconnect / Fault Tolerance
DataSet
“Operators“ and Drivers / Operator Graph / Visitors
DataStream
StreamOperators / StreamTransformation Graph* /
Monolithic Translators
Table API / SQL
Logical Nodes* / Different translation paths

17. © 2019 Ververica17
What could be improved?
• Each API has its own internal graph representation  code duplication
• Multiple translation components between the different graphs  code duplication
– DataStream API has an intermediate graph structure: StreamTransformation  StreamGraph 
JobGraph
• Separate (incompatible) operator implementations
– DataStream API has StreamOperator, DataSet API has Drivers  two map operators, two flatMap
operators
– These are run by different lower-level Tasks
– DataSet operators are optimized for different requirements than DataSet operators
• Table API is translated to two distinct lower-level APIs  two different translation
stacks
– ”project operator” for DataStream and for DataSet
• Connectors for each API are separate  a whole bunch of connectors all over the
From a system design/code quality/architecture/development perspective

18. © 2019 Ververica18
What does this mean for users?
• You have to decide between DataSet and DataStream when writing a job
– Two (slightly) different APIs, with different capabilities
– Different set of supported connectors: no Kafka DataSet connector, no HBase DataStream connector
– Different performance characteristics
– Different fault-tolerance behavior
– Different scheduling logic
• With Table API, you only have to learn one API
– Still, the set of supported connectors depends on the underlying execution API
– Feature set depends on whether there is an implementation for your underlying API
• You cannot combine more batch-y with more stream-y sources/sinks
• A “soft problem”: with two stacks of everything, less developer power will go into
each one individual stack  less features, worse performance, more bugs that are
fixed slower

19. © 2019 Ververica
Future Stack

20. © 2019 Ververica20
DataSet
“Operators“ and Drivers / Operator
Graph / Visitors
DataStream
StreamOperator /
StreamTransformation Graph* /
Monolithic Translators
Batch is a subset of streaming!
Can’t we just?
✅❌
Done!
🥳

21. © 2019 Ververica21
Unifying the Batch and Streaming APIs
• DataStream API functionality is already a superset of DataSet API functionality
• We need to introduce BoundedStream to harness optimization potential, semantics
are clear from earlier:
–No processing-time timers
–Watermark “jumps” from –Infinity to +Infinity at end of processing
• DataStream translation and runtime (operators) need to be enhanced to use the
added optimization potential
• Streaming execution is the generic case that always works, “batch” enables
additional “optimization rules”: bounded operators, different scheduling  we get
feature parity automatically ✅
• Sources need to be unified as well  see later

22. © 2019 Ververica22
A typical “unified” use case: Bootstrapping state
* See for example Bootstrapping State in Flink by Gregory Fee https://sf-2018.flink-
“stream
”
source
“batch”
source
Stateful
operatio
n
batch-y partstream-y part
• We have a streaming use
case
• We want to bootstrap the
state of some operations
from a historical source
• First execute bounded parts
of the graph, then start the
rest

23. © 2019 Ververica23
Under-the-hood Changes
• StreamTransformation/StreamGraph
need to be beefed up to carry the
additional information about
boundedness
• Translation, scheduling, deployment,
memory management and network stack
needs to take this into account
Graph Representation / DAG Operator / Task
• StreamOperator needs to support
batch-style execution  see next slide
• Network stack must eventually support
blocking inputs

24. © 2019 Ververica24
Selective Push Model Operator FLINK-11875
batch: pull-based operator (or Driver) streaming: push-based StreamOperator
• StreamOperator needs additional API
to tell the runtime which input to
consume
• Network stack/graph needs to be
enhanced to deal with blocking inputs,
😱

25. © 2019 Ververica25
Current Source Interfaces
InputFormat
createInputSplits(): splits
openSplit(split)
assignInputSplit()
nextRecord(): T
closeCurrentSplit()
SourceFunction
run(OutputContext)
close()
batch streaming

26. © 2019 Ververica26
Batch InputFormat Processing
TaskManager TaskManager TaskManager
JobManager
(1) request split
(2) send split
(3) process split
• Splits are assigned to TaskManagers by the JobManager, which runs a copy of
the InputFormat  Flink knows about splits and can be clever about scheduling,
be reactive
• Splits can be processed in arbitrary order
• Split processing pulls records from the InputFormat
• InputFormat knows nothing about watermarks, timestamps, checkpointing  bad
for streaming

27. © 2019 Ververica27
Stream SourceFunction Processing
• Source have a run-loop that they manage completely on their own
• Sources have flexibility and can efficiently work with the source system: batch
accesses, dealing with multiple topics from one consumer, threading model,
etc…
• Flink does not know what’s going on inside and can’t be clever about it
• Sources have to implement their own per-partition watermarking, idleness
tracking, what have you
TaskManagerTaskManager TaskManager
(1) do your thing

28. © 2019 Ververica28
A New (unified) Source Interface
• This must support both batch and streaming use cases, allow Flink to be clever, be
able to deal with event-time, watermarks, source idiosyncrasies, and enable
snapshotting
• This should enable new features: generic idleness detection, event-time alignment
FLIP-27
Source
createSplitEnumerator()
createSplitReader
SplitEnumerator
discoverNewSplits()
nextSplit()
snapshotState()
isDone()
SplitReader
addSplit()
hasAvailable(): Future
snapshotState()
emitNext(Context): Status
* FLINK-10886: Event-time alignment for sources; Jamie Grier (Lyft) contributed the first parts of this

29. © 2019 Ververica29
A New (unified) SourceInterface: Execution Style I
TaskManager TaskManager TaskManager
JobManager
(1) request split
(2) send split
(3) process split
• Splits are assigned to TaskManagers by the JobManager, which runs a copy of
the SplitEnumerator  Flink knows about splits and can be clever about
scheduling, be reactive
• Splits can be processed in arbitrary order
• Split processing is driven by the TaskManager working with SplitReader
• SplitReader emits watermarks but Flink deals with idleness, per-split
watermarking

30. © 2019 Ververica30
A New (unified) SourceInterface: Execution Style II

31. © 2019 Ververica31
Table API / SQL
• API: easy, it’s already unified ✅
• Translation and runtime (operators) need to be enhanced to use the added
optimization potential but use the StreamOperator for both batch and streaming
style execution
• Streaming execution is the generic case that always works, “batch” enables
additional “optimization rules”: bounded operators, different scheduling  we get
feature parity automatically ✅
• Sources will be unified from the unified source interface ✅
• This is already available in the Blink fork (by Alibaba), FLINK-11439 is the effort of
getting that into Flink

32. © 2019 Ververica32
StreamTransformation DAG / StreamOperator
DataStream
“Physical“ Application API
Table API / SQL
Declarative API
Runtime
The Future
Stack

33. © 2019 Ververica
Closing

34. © 2019 Ververica34
Summary
• Semantics of unified processing are quite clear already
• For some things, work is ongoing and there are design documents (FLIPs)
• Some things are farther in the future
• This project requires changes in all components/layers of Flink:
–API, deployment, network stack, scheduling, fault tolerance
• You can follow all of this on the public mailing lists and FLIPs!
• The Flink tech stack is going to be quite nice! 😎

35. © 2019 Ververica
www.ververica.com @VervericaDataaljoscha@ververica.com
stephan@ververica.com

36. © 2019 Ververica
Thank you!
Questions?