These are slides presented at our first Flink London meetup (http://www.meetup.com/Apache-Flink-London-Meetup/). They are a non-technical introduction to Flink and highlight some of the motivations for selecting this tool for streaming data processing. See the associated video recording here: https://youtu.be/Hgkmnuj1vUw

1. Motivation

2. The Evolution of Massive-Scale Data Processing
Tyler Akidau, Staff Software Engineer @ Google
https://goo.gl/5k0xaL

3. The Evolution of Massive-Scale Data Processing
Tyler Akidau, Staff Software Engineer @ Google
https://goo.gl/5k0xaL

4. We’re not the newest!

5. APACHE FLINK LONDON
MEETUP
3rd March 2016 | Bonhill House, London

6. What we’ll cover today
¨ Hand-wavey bit
¨ Practical bit
¨ Textbook bit

7. Part 1: The hand-wavey bit
¨ Aim:
¤ Make sure we all have same basic understanding of
what Flink is
¤ Introduce key concepts
n Not exhaustive
n Not explaining much!

8. WTF is Flink?

9. Flink basics…
¨ Apache Foundation top level open source project…
¨ …for distributed data processing…
¨ …with a “streaming first” architecture…
¨ …running on the JVM.
Or:
A ‘free’ way to process a lot of data (especially
streaming data) on ‘commodity’ hardware, with a
codebase that is continually improving. Useful for
reporting, analytics, log processing, machine learning,
etc.

10. Some key terms
¨ DataStream
A possibly unbounded immutable collection of data items of the same type
¨ DataSet
An abstract representation of a finite immutable collection of data of the
same type that may contain duplicates
¨ Source
Can be file-based, socket-based, collection-based, Custom (e.g. Kaea)
¨ Sink
Consumes DataSets/DataStreams and forwards them to files, sockets,
external systems, or prints them
¨ Operator
Represents an operation (or a data processing step) in the ‘Job Graph’ –
includes properties like the actual code and desired parallelism.

11. Application architecture

12. Flink ‘skeleton’ program structure
DataStream
1. Obtain a
StreamExecutionEnvironment
2. Connect to data stream
sources
3. Specify transformations
on the data streams
4. Specify output for the
processed data
5. Execute the program
[env.execute()]
DataSet
1. Obtain an
ExecutionEnvironment
2. Load/create the initial
data
3. Specify transformations
on the data
4. Specify where to put
results
5. Execute the program
[env.execute(), print(),
collect()]

13. (in future meetups Guest Speakers will give us
the juicy details…)
Key Flink features

14. High Performance
Support for out-of-
order events
Low latency
Exactly-once
semantics
Flexible streaming
windows
One runtime for
stream & batch /
ecosystem
Back pressure
Delta iterate
operators

15. One runtime for
stream & batch /
ecosystem
Delta iterate
operators
High Performance
Support for out-of-
order events
Low latency
Exactly-once
semantics
Flexible streaming
windows
Back pressure
According to the
Apache Flink site
(http://flink.apache.org/)

16. High performance / Low latency
High
throughput
Low
latency

17. Flow control and backpressure
¨ Backpressure bottleneck: ‘pressure’ building up
because data is arriving faster than it can be
processed.
¤ Temporary process slow-down (e.g. GC on JVM)
¤ Temporary traffic spike
¨ “Flink achieves the maximum throughput
allowed by the slowest part of the pipeline”
¤ Not a configurable ‘feature’
¤ Inherent in architecture (buffer-based)

18. Exactly-once semantics for state
¨ In the event of failure “Pick up where you left off”.
¤ Means you need to remember where you left off (data and state)
¨ 3 levels of risk appetite:
¤ L1 – Accept misses (“At most once”)
¤ L2 – Accept duplicates (“At least once”)
¤ L3 – Don’t accept either (“Exactly once”)
¨ Checkpointing / snapshots
¤ Dependent on stream source – e.g. Kaea
¤ Orchestration is tricky (see next slide)

19. Checkpointing orchestration

20. Support for Out-of-Order events
¨ Real life: messages will be delayed
¨ Every event is time-stamped
¨ It’s harder than it sounds
(‘kinds’ of time, windows, watermarks, etc)
t1 t2 t3 t5 t6 t7 t4 t8

21. Highly flexible streaming windows
The start and end of the data stream that is being
processed.
¨ Different ways to define the window, including:
¤ Time (from 9:00:00 to 9:00:04)
¤ Count (from item 12 to item 18)
¤ Session (from first ‘keyed event’ until we don’t see same
key for X time – analogous to cookie session)
¤ More complex logic driven by the data, and more
complex windows depending on what is needed

22. (Delta) iterate operators
Iterate operator
Delta iterate operator
Work on ‘hot’
Don’t touch ‘cold’

23. (Delta) iterate operators

24. One runtime / library ecosystem
NB:
• Libraries in beta
• APIs in Java, Scala, [Python]
• Flink CEP too?