Flink streaming ( Tutorial, Consideration before apply to system )

1. LADP ( LINE AD Data-Pipeline ) 김용휘
2018.12.18
Consideration for build-up streaming service
Why Flink streaming

5. ???

6. Streaming
1
Intro
Dataflow Model
Triggers & Incremental
Processing
Stream API
2
ParDo / GroupByKey
Window
Process Function
Rich Function
Broadcast
Connector
Etc
Operations
3
Monitoring
Alert
Resource Management
Scale in&out
Index
Q&A
4

7. What is Streaming

8. ※출처 : https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101
Unbounded / Bounded Intro

9. ※출처 : https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101
Unbounded / Bounded Intro

10. ※출처 : https://static.googleusercontent.com/media/research.google.com/ko//pubs/archive/43864.pdf
Dataflow Model

11. ※출처 : https://static.googleusercontent.com/media/research.google.com/ko//pubs/archive/43864.pdf
Dataflow Model
Almost Flink API is related to This Model

12. ※출처 : https://static.googleusercontent.com/media/research.google.com/ko//pubs/archive/43864.pdf
Dataflow Model
ParDo : for generic parallel processing. Each input element to be processed (which itself may
be a finite collection) is provided to a user-defined function (called a DoFn in Dataflow), which
can yield zero or more output elements per input
GroupByKey : for key-grouping (key, value) pairs

13. ※출처 : https://static.googleusercontent.com/media/research.google.com/ko//pubs/archive/43864.pdf
Dataflow Model
ParDo : for generic parallel processing. Each input element to be processed (which itself may
be a finite collection) is provided to a user-defined function (called a DoFn in Dataflow), which
can yield zero or more output elements per input
GroupByKey : for key-grouping (key, value) pairs

14. ※출처 : https://beam.apache.org/blog/2017/08/28/timely-processing.html
Dataflow Model
ParDo : for generic parallel processing. Each input element to be processed (which itself may
be a finite collection) is provided to a user-defined function (called a DoFn in Dataflow), which
can yield zero or more output elements per input

15. ※출처 : https://static.googleusercontent.com/media/research.google.com/ko//pubs/archive/43864.pdf
Dataflow Model
ParDo : for generic parallel processing. Each input element to be processed (which itself may
be a finite collection) is provided to a user-defined function (called a DoFn in Dataflow), which
can yield zero or more output elements per input
GroupByKey : for key-grouping (key, value) pairs

16. ※출처 : https://static.googleusercontent.com/media/research.google.com/ko//pubs/archive/43864.pdf
Dataflow Model
GroupByKey : for key-grouping (key, value) pairs

17. ※출처 : https://static.googleusercontent.com/media/research.google.com/ko//pubs/archive/43864.pdf
Windowing

18. ※출처 : https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101
Time Domains
• Event time
• Processing time
Intro

19. ※출처 : https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101
Lateness Problem Intro

20. ※출처 : https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101
Lateness Problem Intro

21. Event time based window Intro
• How long wait for Lateness data
• Trigger Timing
• Store aggregation result
• How to operate Lateness data

22. Dataflow Model 에는 빠진 내용이지만.. Intro
• Scaling
• Optimize Task
• Etc

23. Stream API

24. ※출처 : https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/stream/operators/index.html
ParDo & GroupByKey API API
• Map, FlatMap
• Filter
• KeyBy
• Reduce, Fold, Aggregation
• Etc.. (Connect, CoMap, CoFlatMap, Split, Select)

25. ※출처 : https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/stream/operators/index.html
ParDo & GroupByKey API API
• Map, FlatMap
• Filter
• KeyBy
• Reduce, Fold, Aggregation
• Etc.. (Connect, CoMap, CoFlatMap, Split, Select)
Same Task Slot

26. ※출처 : https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/stream/operators/index.html
ParDo & GroupByKey API API
• Map, FlatMap
• Filter
• KeyBy
• Reduce, Fold, Aggregation
• Etc.. (Connect, CoMap, CoFlatMap, Split, Select)
Same Task Slot
Shuffle & new Task Slot

27. ※출처 : https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/event_time.html
Window API

28. ※출처 : https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/event_time.html
Watermark ( fixed amount of lateness ) API

29. ※출처 : https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/stream/operators/windows.html#window-assigners
Windowed Functions API

30. ※출처 : https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/stream/operators/windows.html#window-assigners
Windowed Functions API

31. ※출처 : https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/stream/operators/windows.html#window-assigners
Windowed Functions API

32. ※출처 : https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/stream/operators/windows.html#window-assigners
Windowed Functions API

33. ※출처 : https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/stream/operators/process_function.html
ProcessFunction ( Low-level Operation ) ( e.g session window ) API
1
2
3

34. ※출처 : https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/common/functions/AbstractRichFunction.java
Rich Function API

35. ※출처 : https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/common/functions/AbstractRichFunction.java
Rich Function API

36. ※출처 : https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/common/functions/AbstractRichFunction.java
Rich Function API
…

37. ※출처 : https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/common/functions/AbstractRichFunction.java
Rich Function API
…
1 Init State
2 Setting Connections
3 Etc

38. ※출처 : https://data-artisans.com/blog/a-practical-guide-to-broadcast-state-in-apache-flink
Broadcast State in Flink API

39. ※출처 : https://data-artisans.com/blog/a-practical-guide-to-broadcast-state-in-apache-flink
Broadcast State in Flink API

40. ※출처 : https://data-artisans.com/blog/a-practical-guide-to-broadcast-state-in-apache-flink
Broadcast State in Flink API

41. ※출처 : https://data-artisans.com/blog/a-practical-guide-to-broadcast-state-in-apache-flink
Broadcast State in Flink API

42. ※출처 : https://data-artisans.com/blog/a-practical-guide-to-broadcast-state-in-apache-flink
Broadcast State in Flink API

43. ※출처 : https://data-artisans.com/blog/a-practical-guide-to-broadcast-state-in-apache-flink
Broadcast State in Flink API

44. ※출처 : https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/connectors/guarantees.html
Connector ( integration other system ) API

45. ※출처 : https://ci.apache.org/projects/flink/flink-docs-release-1.7/dev/connectors/guarantees.html
Connector ( integration other system ) API

46. ※출처 : https://ci.apache.org/projects/flink/flink-docs-release-1.7
만약 Flink Connector 가 없다면? API
• Kafka consumer 를 구현하기 위해..
• State Init 을 해주고
• Partition Discover Thread 를 만들어주고,
• Kafka Consumer Thread, Fetch Thread 를 따로 만든 다음
• Consumer 와 Fetcher thread 의 message serving 을 담당하는 memory Queue(handover)
를 하나 만들어서 통신하게 하고~~
• 주기적으로 Checkpointing 을 하는 로직을 짠다음
• Close 할땐 Thread 정리와 checkpointing 을 잘 하면 되겠다!
• Monitoring 도 할거니까 metric 도 노출해야지!

47. ※출처 : https://ci.apache.org/projects/flink/flink-docs-release-1.7
만약 Flink Connector 가 없다면? API
• Kafka consumer 를 구현하기 위해..
• State Init 을 해주고
• Partition Discover Thread 를 만들어주고,
• Kafka Consumer Thread, Fetch Thread 를 따로 만든 다음
• Consumer 와 Fetcher thread 의 message serving 을 담당하는 memory Queue(handover)
를 하나 만들어서 통신하게 하고~~
• 주기적으로 Checkpointing 을 하는 로직을 짠다음
• Close 할땐 Thread 정리와 checkpointing 을 잘 하면 되겠다!
• Monitoring 도 할거니까 metric 도 노출해야지!

48. ※출처 : https://ci.apache.org/projects/flink/flink-docs-release-1.7
Others.. API
• Physical Partitioning ( rebalance, custom partition, rescaling )
• Side outputs
• Custom Serde
• Checkpointing
• Etc..

49. Operations

50. Flink Dashboard( DAG, Task Slot )
※출처 : https://ci.apache.org/projects/flink/flink-docs-stable/tutorials/local_setup.html

51. Flink Dashboard ( Metric )

52. Flink Dashboard ( Checkpointing )
※출처 : https://ci.apache.org/projects/flink/flink-docs-release-1.7/monitoring/checkpoint_monitoring.html

53. Flink Dashboard ( Back Pressure Testing )
※출처 : https://ci.apache.org/projects/flink/flink-docs-release-1.7/monitoring/back_pressure.html

54. Flink Dashboard
But… No history…
※출처 : https://static.googleusercontent.com/media/research.google.com/ko//pubs/archive/43864.pdf

55. Monitoring ( with HTTP Rest API )
※출처 : https://static.googleusercontent.com/media/research.google.com/ko//pubs/archive/43864.pdf

56. Monitoring ( with HTTP Rest API )
Flink Cluster
Job Manager
Task Manager
Task Manager
…
Task Manager
Daemon
(HTTP Client)
Monitoring System
…

57. Monitoring ( with Pure JMX Client )
Flink Cluster
Job Manager
Task Manager
Task Manager
…
Task Manager
Daemon
(JMX Client)
Monitoring System
…

58. Monitoring ( Others .. )
※출처 : https://www.slideshare.net/FlinkForward/flink-forward-berlin-2018-dongwon-kim-realtime-driving-score-service-using-flink

59. Abnormal Status Detect ( metric )
• CPU / Memory / Disk usage
• Gabage Collection ( count / time )
• Network I/O
• Job Downtime
• Latency Tracking
• Customized Metric
• Etc …
Alert

60. ※출처 : http://colingorrie.github.io/outlier-detection.html
Abnormal Status Detect ( method )
• Z-score method
• Modified Z-score method
• IQR method
• Pattern matching
Alert

61. ※출처 : http://colingorrie.github.io/outlier-detection.html
Abnormal Status Detect ( method )
• Z-score method
• Modified Z-score method
• IQR method
• Pattern matching
Alert
Statistics Based

62. ※출처 : http://colingorrie.github.io/outlier-detection.html
Abnormal Status Detect ( method )
• Z-score method
• Modified Z-score method
• IQR method
• Pattern matching
Alert
Statistics Based
Rule Based

63. ※출처 : https://slides.com/yonghweekim/streaming-system#/
Abnormal Status Detect ( Channel ) Alert
• Email
• SMS
• Slack
• SMS

64. Should Alerting?
• CPU / Memory / Disk usage
• Gabage Collection ( count / time )
• Network I/O
• Job Downtime
• Latency Tracking
• Customized Metric
• Etc …
Alert

65. Should Alerting?
“As with alerts, an
information radiator
that always shows red
has no value. If a
condition shown on the
radiator isn’t important
enough to fix
immediately, then
remove it.
Alert
※출처 : O'Reilly Media, Inc. Infrastructure as code

66. Should Alerting?
• CPU / Memory / Disk usage
• Gabage Collection ( count / time )
• Network I/O
• Job Downtime
• Latency Tracking
• Customized Metric
• Etc …
Alert

67. Resources Management

68. Resources Management

69. Network managing is difficult

70. Network managing is difficult
※출처 : http://blog.woosum.net/archives/1524

71. Solution - separate Batch and Streaming
Cluster
Batch Job
Batch Job
Batch Job
…
Batch Job
Streaming Job
Streaming Job
Streaming Job
…
Streaming Job

72. Solution - separate Batch and Streaming
Cluster
Batch Job
Batch Job
Heavy Batch!!
…
Batch Job
Streaming Job
Streaming Job
Streaming Job
…
Streaming Job

73. Solution - separate Batch and Streaming
Cluster
Batch Job
Batch Job
Heavy Batch!!
…
Batch Job
Streaming Job
Streaming Job
Streaming Job
…
Streaming Job
Slow
Slow
Slow
Slow
Slow
Slow
Slow
Slow
Slow
Slow

74. Solution - separate Batch and Streaming
Batch Cluster
Batch Job
Batch Job
Batch Job
…
Batch Job
Streaming Job
Streaming Job
Streaming Job
…
Streaming Job
Streaming Cluster

75. Solution - separate Batch and Streaming
Batch Cluster
Batch Job
Batch Job
Heavy Batch!!
…
Batch Job
Streaming Job
Streaming Job
Streaming Job
…
Streaming Job
Streaming Cluster

76. Solution - separate Batch and Streaming
Batch Cluster
Batch Job
Batch Job
Heavy Batch!!
…
Batch Job
Streaming Job
Streaming Job
Streaming Job
…
Streaming Job
Slow
Slow
Slow
Slow
Slow
Streaming Cluster

77. Other Solutions
• Avoid to consuming from kafka-earliest-offsets
• Be careful of GroupByKey operator
• Do it first to Predicative / Filter-out operator
• Repartition / Rescaling for bottleneck ( sync logic )
• Use Async Logic

78. 78
6. FAQ
Questions and answers

79. 발표를 끝내기전에…

80. Flink 한국 사용자 모임
Flink 관련된 정보 공유 및 Networking
개발 하시다 어려운 점이나 궁금하신점 있으시면 편하게 질문해주세요! 

81. Flink 한국 사용자 모임
Flink 관련된 정보 공유 및 Networking
개발 하시다 어려운 점이나 궁금하신점 있으시면 편하게 질문해주세요! 

82. THANK YOU