The document provides an overview of Apache Flink, an open-source stream processing framework, highlighting its basic concepts, including stateful processing, fault tolerance, and window functions. It details its architecture, programming models, and APIs for data processing in various environments, such as standalone clusters and cloud services. Additionally, it covers examples of data flow operations and checkpointing mechanisms for ensuring data consistency.

1. Apache Flink
In action

2. About me
Name: Artsem Semianenka
Position: BigData engineer in adform.com
LinkedIn: https://www.linkedin.com/in/artsemsemianenka/
GitHub: https://github.com/art4ul
https://www.facebook.com/art4ulFacebook:

3. Agenda
• Basic concepts
• Stateful processing
• Practice
• Fault tolerance
• Window functions

4. What is Apache Flink?
Apache Flink is an open source stream
processing framework

5. Stream

6. What is Apache Flink?
True
Streaming
Low latency
High Throughput
Back
pressure
Stateful
Computation
Savepoint
Exactly-once
semantic
User defined
state
Windows
Functions
Event time
Flexible windows
functions
API &
Libraries CEP
Flink MLGelly

7. Standalone Cluster
Mesos/Yarn/Kubernete
s
Cloud
AWS / Google
Engine)
Deploy
Streaming dataflow runtimeCore
DataStream API ( Java / Scala ) DataSet API ( Java / Scala)
Api
CEP Table API Flink ML Gelly Table API
Libraries
Apache Flink stack

8. Standalone Cluster
Mesos/Yarn/Kubernete
s
Cloud
AWS / Google
Engine)
Deploy
Streaming dataflow runtimeCore
DataStream API ( Java / Scala ) DataSet API ( Java / Scala)
Api
CEP Table API Flink ML Gelly Table API
Libraries
Apache Flink stack

9. Stream processing
approaches
Micro batch
True streaming

10. Micro batch
Micro Batch

11. True streaming

12. Source
Transformation
s
map(..)
flatMap(..)
filter(..)
reduce(..)fold(..)
join(..)
union(..)
Sink
Dataflow Programming Model

13. DAG
Source
Transformation
s Sink

14. Operators
map(..)/
keyBy(..)
Sink
/windows(..)/
apply(..)
Source
Parallelism:3 Parallelism:1Parallelism:3 Parallelism:3

15. Operators
map(..)/
keyBy(..)
Sink
/windows(..)/
apply(..)
Source
Parallelism:3
Parallelism:1

16. Operator subtasks
map(..)
keyBy(..)
windows(..)/
apply(..)
Source
map(..)/
keyBy(..) Sink
windows(..)/
apply(..)
Source
map(..)
keyBy(..)
windows(..)/
apply(..)
Source

17. Distributed Runtime Environment

18. Distributed Runtime Environment
Job Manager
JVM

19. Distributed Runtime Environment
Job Manager
JVM
Task Manager
JVM
Task Manager
JVM

20. Distributed Runtime Environment
Job Manager
JVM
Task Manager
JVM
Task Manager
JVM
map(..) Sinkwindow
Sourc
e

21. Distributed Runtime Environment
Job Manager
JVM
Task Manager
JVM
Task Manager
JVM
Sourc
e
Sourc
e
map(..)
window
window
map(..)
map(..) Sinkwindow
Sourc
e

22. Distributed Runtime Environment
Job Manager
JVM
Task Manager
JVM
Task Manager
JVM
map(..)
window
window
map(..)
map(..)
Sinkwindow
Sourc
e
Sourc
e
Sourc
e
Sourc
e
Sourc
e
map(..)
window
window
map(..)
map(..) Sinkwindow
Sourc
e

23. Example

24. Example
Datacenter
DeviceId: 1234
Value: 26.0

25. Example: Data Flow
Datacenter
Source KeyBy
(DeviceId)
Map
(Metric)
Sink

26. Example: Data Flow
Source KeyBy
(DeviceId)
Map
(Metric)
Sink
KeyBy
(DeviceId)
Map
(Metric)
Sink
KeyBy
(DeviceId)
Map
(Metric)
Sink

27. Let’s code

28. Example
DatacenterDeviceId: 1234
Value: 26.0
Value: 26.0
Device1 -> User1
Device2 -> User2

29. Example: Data Flow
Source KeyBy
(DeviceId)
Map
(Metric)
Sink

30. FlatMap
(UserMapping)
Example: Data Flow
KeyBy
(DeviceId)
Map
(Metric)
Sink
Control
Source
Source KeyBy
(DeviceId)
DeviceId UserId
Operator State
Device1 User1
Device2 User2
DeviceId:Device1
UserId: User1
DeviceId:Device1
Value: 23.0
DeviceId:Device1
UserId: User1
Value: 23.0
DeviceId:Device2
UserId: User2
DeviceId:Device3
Value: 30.0
DeviceId:Device2
Value: 10.0
DeviceId:Device2
UserId: User2
Value: 10.0

31. <code/>

32. Storage
Checkpointing: Barriers
State: sum =
4 13 2N5
01366
Operator

33. Storage
Checkpointing: Barriers
Operator1Source Operator2 Sink
State1 State2
N=1 N=2
Source
Operator 1
Operator 2
Sink
Src State Sink State
12
Src State State1 State2Src State
Ack
State1

34. Operator
Barrier alining

35. Operator
Barrier alining
1

36. Operator
Barrier alining
11

37. Operator
Barrier alining
112

38. Operator
Barrier alining
113 2

39. Operator
Barrier alining
113 2

40. Operator
Barrier alining (Exactly once)
113 2

41. Windows functions

42. Window functions
Unbounded event stream
Window 1Window 2Window 3Window 4
Time driven
(example: every 30 seconds)

43. Window functions
Unbounded event stream
Window 1Window 2Window 3
Data driven
(example: every 2 elements)

44. Tumbling Windows
User1
User2
User3
Time
30 sec30 sec 30 sec 30 sec

45. Tumbling Windows
User1
User2
User3
Time

46. Sliding Windows
User1
User2
User3
Time

47. Sliding Windows
User1
User2
User3
TimeWindow
Slide
Interval

48. Sliding Windows
User1
User2
User3
Time

49. User1
User2
User3
Time
Session Windows
Window
Session
Gap

50. User1
User2
User3
Time
Session Windows
Window
Session
Gap

51. Time
Window
Function
Window
Function
Source
Source
Event time Ingestion time Processing time

52. Time
Window
Function
Window
Function
Source
Sourc
e

53. Watermarks
Window
Operator
Window( 00:00 - 0:10)
Window( 00:05 - 0:15)
00:0
0
00:0
2
00:0
3
00:0
5
00:0
5
Watermark
t = 00:05
00:0
8
00:1
1
Watermark
t = 00:11

54. Thank you for
attention
Questions?