1. Aljoscha Krettek – PMC of Apache Flink and Apache Beam, co-founder at data Artisans
The Past, Present, and
Future of Apache Flink®

2. © 2018 data Artisans2
About Data Artisans
Original Creators of
Apache Flink®
Real Time Stream Processing Enterprise
Ready

3. © 2018 data Artisans3
Past

4. © 2018 data Artisans4
It all started in 2014
2009 - 2014 since 2014
● Batch processor on top of streaming runtime
● First Apache Flink 0.6.0 release August 2014

5. © 2018 data Artisans5
Flink learns to stream in real time
DataStream API
Stream Processing
DataSet API
Batch Processing
Runtime
Distributed Streaming Data Flow

6. © 2018 data Artisans6
Flink learns to remember

7. © 2018 data Artisans7
Flink learns to remember

8. © 2018 data Artisans8
Flink learns to remember

9. © 2018 data Artisans9
Flink learns to remember
Remember where we left off

10. © 2018 data Artisans10
Latency vs. Throughput?
high latency
low high throughput
Prevailing
belief
● 10s of millions of events/s
● Latency down to 1 ms
≠

11. © 2018 data Artisans11
Flink becomes event-time aware
Episode
IV
Episode
V
Episode
VI
Episode
I
Episode
II
Episode
III
Episode
VII
Episode
VIII
1977 1980 1983 1999 2002 2005 2015 2017
Flink becomes event-time aware
Processing time

12. © 2018 data Artisans12
Flink becomes event-time aware
Episode
I
Episode
II
Episode
III
Episode
IV
Episode
V
Episode
VI
Episode
VII
Episode
VIII
1999 2002 2005 1977 1980 1983 2015 2017
Processing time
Event time

13. © 2018 data Artisans13
More than just analytics: ProcessFunction
class MyFunction extends ProcessFunction[MyEvent, Result] {
// declare state to use in the program
lazy val state: ValueState[CountWithTimestamp] = getRuntimeContext().getState(…)
def processElement(event: MyEvent, ctx: Context, out: Collector[Result]): Unit = {
// work with event and state and schedule timers
}
def onTimer(timestamp: Long, ctx: OnTimerContext, out: Collector[Result]): Unit = {
// handle callback when event-/processing- time instant is reached
}
}
● ProcessFunction gives access to state, time
and events
● Low level API
● Enables data-driven applications

14. © 2018 data Artisans14
Present & Future

15. © 2018 data Artisans15
Large, larger, Flink
Time
State
Incremental
snapshots
● Snapshot only state diff
● Incremental snapshots allow to handle very large state

16. © 2018 data Artisans16
Faster failover is always better

17. © 2018 data Artisans17
Varying workloads
• Violating SLAs vs. wasting money
• Varying workloads require to adapt resources

18. © 2018 data Artisans18
Revamped distributed architecture
ResourceManag
er
ClusterManager
TaskManagerJobManager
Dispatcher
Client
1. Submit job 2. Start job
3. Request
slots
4. Allocate resources
5. Start TaskManager
6. Execute job
● Support for full resource elasticity
● Application parallelism can be dynamically changed

19. © 2018 data Artisans19
Flink as a library (and still as a framework)
• Deploying Flink applications should be as easy as starting a process
• Bundle application code and Flink into a single image
• Process connects to other application processes and figures out its role
• Removing the cluster out of the equation
P2 P3
P1
P4
New process

20. © 2018 data Artisans20
How much control do I need?
Batch
processing
Continuous
processing
Real-time &
data-driven
applications
● Multiple short lived stages
● Different resource requirements
per stage
● Efficient execution requires
control over resources
● Flink allocates actively resources
● Continuously processing operators
● Constrained by external systems,
SLAs and application logic
● External system can assign
resources
● Flink reacts to available resources

21. © 2018 data Artisans21
Active vs. reactive mode
• Active mode
‒ Flink is aware of underlying cluster framework
‒ Flink allocate resources
‒ E.g. existing YARN and Mesos integration
• Reactive mode
‒ Flink is oblivious to its runtime environment
‒ External system allocates and releases
resources
‒ Flink scales with respect to available resources
‒ Relevant for environments: Kubernetes, Docker,
as a library

22. © 2018 data Artisans22
Scaling automatically
• Latency
• Throughput
• Resource utilization
• Connector signals

23. © 2018 data Artisans23
How we create Flink Jobs
Flink APIs
Stream/Batch Processing
Runtime
Distributed Streaming Data Flow
Java/Scala

24. © 2018 data Artisans24
Flink SQL
Flink APIs
Stream/Batch Processing
Runtime
Distributed Streaming Data Flow
Java/Scala SQL
“NO CODING REQUIRED”
Source/Sink
definition in YAML
Configuration in
YAML
SQL commandline
User-defined
functions
Streaming and
Batch
Event time and
processing time
*since Flink 0.9.0 (June 2015)

25. © 2018 data Artisans25

26. © 2018 data Artisans26
“Join” me for some trading
buy buy sell buy
Join
$ 17
£ 42
12.5
buy sell
₪

27. © 2018 data Artisans27
Introducing Time-versioned Table Joins
buy buy sell buy
Join
buy sell
curr rate time
£ 42 3
£ 12 17
1453
31753
14
event time

28. © 2018 data Artisans28
SQL for pattern analysis?
SELECT * from ?

29. © 2018 data Artisans29
Introducing MATCH_RECOGNIZE
SELECT *
FROM TaxiRides
MATCH_RECOGNIZE (
PARTITION BY driverId
ORDER BY rideTime
MEASURES
S.rideId as sRideId
AFTER MATCH SKIP PAST LAST ROW
PATTERN (S M{2,} E)
DEFINE
S AS S.isStart = true,
M AS M.rideId <> S.rideId,
E AS E.isStart = false
AND E.rideId = S.rideId
)

30. © 2018 data Artisans30
Todays processing landscape
Streaming Batch

31. © 2018 data Artisans31
Batch/streaming unification

32. © 2018 data Artisans32
Into the Future
Big state
SQL

33. © 2018 data Artisans33
Thank
s!