Flink is a unified stream and batch processing framework that natively supports streaming topologies, long-running batch jobs, machine learning algorithms, and graph processing through a pipelined dataflow execution engine. It provides high-level APIs, automatic optimization, efficient memory management, and fault tolerance to execute all of these workloads without needing to treat the system as a black box. Flink achieves native support through its ability to execute everything as data streams, support iterative and stateful computation through caching and managed state, and optimize jobs through cost-based planning and local execution strategies like sort merge join.

1. Apache Flink™ deep-dive
Unified Batch and Stream Processing
Robert Metzger
@rmetzger_
Hadoop Summit 2015,
San Jose, CA

2. Flink’s Recent History
April 2014 April 2015Dec 2014
Top Level
Project
Graduation
0.70.60.5 0.90.9-m1

3. What is Flink
3
Gelly
Table
ML
SAMOA
DataSet (Java/Scala) DataStream
HadoopM/R
Local Remote YARN Tez Embedded
Dataflow
Dataflow(WiP)
MRQL
Table
Cascading(WiP)
Streaming dataflow runtime
Zeppelin

4. Program compilation
4
case class Path (from: Long, to:
Long)
val tc = edges.iterate(10) {
paths: DataSet[Path] =>
val next = paths
.join(edges)
.where("to")
.equalTo("from") {
(path, edge) =>
Path(path.from, edge.to)
}
.union(paths)
.distinct()
next
}
Optimizer
Type extraction
stack
Task
scheduling
Dataflow
metadata
Pre-flight (Client)
Master
Workers
Data Source
orders.tbl
Filter
Map DataSource
lineitem.tbl
Join
Hybrid Hash
buildHT probe
hash-part [0] hash-part [0]
GroupRed
sort
forward
Program
Dataflow Graph
Independent of
batch or
streaming job
deploy
operators
track
intermediate
results
 Layered Architecture
allows plugging of
components

5. Native workload support
5
Flink
Streaming
topologies
Long batch
pipelines
Machine Learning at scale
How can an engine natively support all these workloads?
And what does "native" mean?
Graph Analysis
 Low latency
 resource utilization  iterative algorithms
 Mutable state

6. E.g.: Non-native iterations
6
Step Step Step Step Step
Client
for (int i = 0; i < maxIterations; i++) {
// Execute MapReduce job
}
 Teaching an old elephant new tricks
 Treat system as a black box

7. E.g.: Non-native streaming
7
stream
discretizer
Job Job Job Job
while (true) {
// get next few records
// issue batch job
}
Data Stream
 Simulate stream processor with batch system

8. Native workload support
8
Flink
Streaming
topologies
Long batch
pipelines
Machine Learning at scale
How can an engine natively support all these workloads?
And what does "native" mean?
Graph Analysis
 Low latency
 resource utilization  iterative algorithms
 Mutable state

9. Ingredients for “native” support
1. Execute everything as streams
Pipelined execution, push model
2. Special code paths for batch
Automatic job optimization, fault tolerance
3. Allow some iterative (cyclic) dataflows
4. Allow some mutable state
5. Operate on managed memory
Make data processing on the JVM robust
9

10. Flink by Use Case
10

11. Stream data processing
streaming dataflows
11
Full talk tomorrow:
3:10PM, Grand Ballroom 220A
Stream processing with Flink

12. Pipelined stream processor
12
Streaming
Shuffle!
 Low latency
 Operators push data
forward

13. Expressive APIs
13
case class Word (word: String, frequency: Int)
val lines: DataStream[String] = env.fromSocketStream(...)
lines.flatMap {line => line.split(" ").map(word => Word(word,1))}
.window(Time.of(5,SECONDS)).every(Time.of(1,SECONDS))
.groupBy("word").sum("frequency")
.print()
val lines: DataSet[String] = env.readTextFile(...)
lines.flatMap {line => line.split(" ").map(word => Word(word,1))}
.groupBy("word").sum("frequency")
.print()
DataSet API (batch):
DataStream API (streaming):

14. Checkpointing / Recovery
14
Chandy-Lamport Algorithm for consistent asynchronous distributed snapshots
Pushes checkpoint barriers
through the data flow
Data Stream
barrier
Before barrier =
part of the snapshot
After barrier =
Not in snapshot
(backup till next snapshot)
 Guarantees exactly-once
processing

15. Batch processing
Batch on Streaming
15

16. Batch on an streaming engine
16
File in HDFS
Filter Map Result 1
Map Result 2
 Batch program, completely pipelined
 Data is never materialized anywhere (in this example)

17. Batch on an streaming engine
Map
Operator
Map
Operator
Map
Operator
17
Data
Source
(small)
Stream
Data
Sink
Data
Sink
Data
Sink
Join
Operator
in parallel
Data
Source
(large)
Data
Sink
in parallel (once build side finished)
Map

18. Batch processing requirements
 Get the data processed as fast as possible
• Automatic job optimizer
• Efficient memory management
 Robust processing
• provide fault-tolerance
• again, memory management
18

19. Optimizer
 Cost-based optimizer
 Select data shipping strategy (forward, partition, broadcast)
 Local execution (sort merge join/hash join)
 Caching of loop invariant data (iterations)
19
case class Path (from: Long, to:
Long)
val tc = edges.iterate(10) {
paths: DataSet[Path] =>
val next = paths
.join(edges)
.where("to")
.equalTo("from") {
(path, edge) =>
Path(path.from, edge.to)
}
.union(paths)
.distinct()
next
}
Optimizer
Type extraction
stack
Pre-flight (Client)
Data
Source
orders.tbl
Filter
Map
DataSourc
e
lineitem.tbl
Join
Hybrid Hash
build
HT
probe
hash-part [0] hash-part [0]
GroupRed
sort
forward
Program
Dataflow
Graph

20. Two execution plans
20
DataSource
orders.tbl
Filter
Map DataSource
lineitem.tbl
Join
Hybrid Hash
buildHT probe
broadcast forward
Combine
GroupRed
sort
DataSource
orders.tbl
Filter
Map DataSource
lineitem.tbl
Join
Hybrid Hash
buildHT probe
hash-part [0] hash-part [0]
hash-part [0,1]
GroupRed
sort
forward
Best plan
depends on
relative sizes
of input files

21. Memory Management
21

22. Operators on managed memory
22

23. Smooth out-of-core performance
23
More at: http://flink.apache.org/news/2015/03/13/peeking-into-Apache-Flinks-Engine-Room.html
Blue bars are in-memory, orange bars (partially) out-of-core

24. Machine Learning Algorithms
Iterative data flows
24

25. Iterate in the Dataflow
26
 API and runtime support
 Automatic caching of loop invariant
data
IterationState state =
getInitialState();
while (!terminationCriterion()) {
state = step(state);
}
setFinalState(state);

26. Example: Matrix Factorization
27
Factorizing a matrix with
28 billion ratings for
recommendations
More at: http://data-artisans.com/computing-recommendations-with-flink.html
Setups:
• 40 medium instances ("n1-highmem-8" - 8
cores, 52 GB)
• 40 large instances ("n1-highmem-16" - 16
cores, 104 GB)

27. Flink ML – Machine Learning
 Provide a complete toolchain
• scikit-learn style pipelining
• Data pre-processing
 various algorithms
• Recommendations: ALS
• Supervised learning: Support Vector Machines
• …
 ML on streams: SAMOA. We are planning to add support for
streaming into ML
28

28. Graph Analysis
Stateful Iterations
29

29. Graph processing characteristics
0
5000000
10000000
15000000
20000000
25000000
30000000
35000000
40000000
45000000
1 6 11 16 21 26 31 36 41 46 51 56 61
#ofelementsupdated
iteration

30. Iterate natively with state/deltas
31
 Keep state in an controlled way by having a partitioned hash-
map
 Relax immutability assumption of batch processing

31. … fast graph analysis
32More at: http://data-artisans.com/data-analysis-with-flink.html

32. Gelly – Graph Processing API
33
 Transformations: map, filter, subgraph, union, reverse,
undirected
 Mutations: add vertex/edge, remove …
 Pregel style vertex centric iterations
 Library of algorithms
 Utilities: Special data types, loading, graph properties

33. Gelly and Flink ML:
 Available in Flink 0.9 (so far only beta release)
 Still under heavy development
 Seamlessly integrate with DataSet abstraction
Preprocess data as needed
Use results as needed
 Easy entry point for new contributors
34

34. Closing
35

35. Flink Meetup Groups
 SF Spark and Friends
• June 16, San Francisco
 Bay Area Flink Meetup
• June 17, Redwood City
 Chicago Flink Meetup
• June 30
 Stockholm, Sweden
 Berlin, Germany
36

36. Flink Forward registration & call for
abstracts is open now
flink.apache.org 37
• 12/13 October 2015
• Meet developers and users of Flink!
• With Flink Workshops / Trainings!

37. flink.apache.org
@ApacheFlink