Downloaded 18 times
Uploaded byIndizen Technologies
Flink. Pure Streaming
The document discusses the transition from batch processing to stream processing in data applications, highlighting Apache Flink as a key player in this paradigm shift. It covers Flink's features, including low latency, exactly-once guarantees, and its approach to handling time and state management in data streams. The document emphasizes the importance of stream processing as the backbone of modern data infrastructure and the need for explicit time handling and fault tolerance in streaming applications.
More Related Content
Similar to Flink. Pure Streaming
Flink. Pure Streaming
- 1. Flink Pure Streaming Paco Guerrero BigData & Solutions Architect 9/21/16
- 2.
- 4.
- 5.
- 6.
- 7.
- 8. “Abstraction of realityused to facilitate information processing” Batch
- 9.
- 10.
- 11.
- 12.
- 13. Batch Batch Job All Input All Output Nothing abouttime Timestamps used as trick to keep real time fingerprint
- 14. Streaming “Continuous processing ofdata that is continuously produced”
- 15. Streaming “Streaming is thenext programming paradigm for data applications, and you need to start thinking in terms of streams” “Continuous processing of data that is continuously produced”
- 16. Streaming “Streaming is thenext programming paradigm for data applications, and you need to start thinking in terms of streams” “Continuous processing of data that is continuously produced” Data Stream: Infinite sequence of data arriving in a continuous fashion.
- 17. Streaming “Streaming is thenext programming paradigm for data applications, and you need to start thinking in terms of streams” “Continuous processing of data that is continuously produced” Data Stream: Infinite sequence of data arriving in a continuous fashion. Stream processing is the backbone of the new data infrastructure.
- 18. Streaming “Streaming is thenext programming paradigm for data applications, and you need to start thinking in terms of streams” “Continuous processing of data that is continuously produced” Data Stream: Infinite sequence of data arriving in a continuous fashion. Stream processing is the backbone of the new data infrastructure. “The world beyond batch” A high-level tour of modern data-processing concepts. By Tyler Akidau August 5, 2015 https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101
- 19.
- 20.
- 21.
- 22.
- 23. Streaming is thebiggest change in data infraestructure since Hadoop Streaming
- 24. The biggest changeis moving from batch to streaming is handling time explicitly Streaming
- 25.
- 26.
- 27.
- 28. Batch Job Batch Job2 All Output Batch Job 1 Micro Batch
- 29. Batch Job Batch Job2 All Input All OutputBatch Job 3 All Output All Output Batch Job 1 Batch Frequency ? Timestamps keeps real time fingerprint Micro Batch
- 30. Streaming Technologies Batch StreamingMicroBatch StateLess – Record acknowledgements CPU bounded performance Not expressive declarative functional API – Low Level API Not auto scaling Low level programmatic topology Poor Streaming Windows funcionalities Not compatible with Hadoop APIs Streams
- 31. Streaming Technologies Batch StreamingMicroBatch StateLess – Record acknowledgements CPU bounded performance Not expressive declarative functional API – Low Level API Not auto scaling Low level programmatic topology Poor Streaming Windows funcionalities Not compatible with Hadoop APIs Streams
- 32. Streaming Technologies Batch StreamingMicroBatch StateLess – Record acknowledgements CPU bounded performance Not expressive declarative functional API – Low Level API Not auto scaling Low level programmatic topology Poor Streaming Windows funcionalities Not compatible with Hadoop APIs Streams
- 33. Streaming Technologies Batch StreamingMicroBatch StateLess – Record acknowledgements CPU bounded performance Not expressive declarative functional API – Low Level API Not auto scaling Low level programmatic topology Poor Streaming Windows funcionalities Not compatible with Hadoop APIs Streams
- 34.
- 35.
- 36.
- 37.
- 38.
- 39. Flink Open Source StreamProcessing Framework. Last available Release 1.1.1 Top Level Apache Project since Dec '14
- 40. Flink Open Source StreamProcessing Framework. Last available Release 1.1.1 Top Level Apache Project since Dec '14 Main Features Native Stream Low Latency High throughput Stateful Exactly-one guarantees Distributed Expressive APIs And more ….
- 41. Flink Open Source StreamProcessing Framework. Last available Release 1.1.1 Top Level Apache Project since Dec '14 Main Features Native Stream Low Latency High throughput Stateful Exactly-one guarantees Distributed Expressive APIs And more ….
- 42.
- 43.
- 44.
- 45.
- 46.
- 47.
- 48.
- 49.
- 50.
- 51.
- 52. Distributed pipelined processing Executeeverything as Stream Iterative ( cyclic ) dataflows Mutable state in operations Operate on managed memory (*) Also works on batch !! Job Manager Client Optimizer Dataflow Graph Flink Runtime Engine
- 53. Distributed pipelined processing Executeeverything as Stream Iterative ( cyclic ) dataflows Mutable state in operations Operate on managed memory (*) Also works on batch !! Workers ( Task Managers ) Job Manager Client Optimizer Dataflow Graph Execution Graph Flink Runtime Engine
- 54. Stream Job Batch Job MLJob Flink Runtime Engine Graph Job optimizer optimizer optimizer optimizer
- 55. Stream Job Batch Job MLJob Flink Runtime Engine Graph Job optimizer optimizer optimizer optimizer
- 56. Tasks scheduled andexecuted in workers ( slots ) Tasks as chain of operators Run operator logic in a pipelined fashion Stream Job Batch Job ML Job Flink Runtime Engine Graph Job optimizer optimizer optimizer optimizer
- 57. Tasks scheduled andexecuted in workers ( slots ) Tasks as chain of operators Run operator logic in a pipelined fashion Stream Job Batch Job ML Job Flink Runtime Engine Graph Job optimizer optimizer optimizer optimizer
- 58. Tasks scheduled andexecuted in workers ( slots ) Tasks as chain of operators Run operator logic in a pipelined fashion Stream Job Batch Job ML Job Flink Runtime Engine Graph Job optimizer optimizer optimizer optimizer
- 59. If you wantto know one thing about Flink is that you don't need to know the internals of Flink
- 60. Events Time & Windows Fault Tolerance & Correctness StateHandling Low Latency & High Throughput API Libraries SQL Building Blocks
- 61. Events Time & Windows Fault Tolerance & Correctness StateHandling Low Latency & High Throughput API Libraries SQL Building Blocks lTime references lOut of order events lPowerful Windowing
- 62. Event Times &Windowing
- 63. Event Times &Windowing Event Time Event Time
- 64. Event Times &Windowing Flink Data Source Event Time Event Time Ingestion Time
- 65. Event Times &Windowing Flink Data Source Flink Window Operator Event Time Event Time Ingestion Time Processing Time
- 66. Event Time: whendata is generated Ingestion time: when data is loaded from source Processing time: when data is processed Event time help to process out- of-order events and replay elements as the ocurred ( deterministic results ) Explicit handling of time. 3 choices: Event Times & Windowing
- 67. Event Times &Windowing
- 68. Event time. Outor Order
- 69. 1 2 35 7 4 6 8 9 10 Event time. Out or Order
- 70. 1 2 35 7 4 6 8 9 10 Event time. Out or Order Out or Order 1 2 3 5 74 6 8 9 10
- 71. 1 2 35 7 4 6 8 9 10 1 2 3 5 74 6 8 9 104 Event time. Out or Order Ingestion Time WindowsOut or Order 1 2 3 5 74 6 8 9 10
- 72. 1 2 35 7 4 6 8 9 10 1 2 3 5 74 6 8 9 10 1 2 3 4 4 5 Event time. Out or Order 6 7 8 9 10 Event Time Windows Ingestion Time WindowsOut or Order 1 2 3 5 74 6 8 9 10
- 73.
- 74. 1 2 35 7 4 6 8 9 10 Event time. Watermarks
- 75. 1 2 35 7 4 6 8 9 10 1 2 3 54 6 8 1 2 3 54 6 8 1 2 3 4 4 5 Event time. Watermarks 6 8 Event Time Windows Ingestion Time WindowsOut or Order
- 76. 1 2 35 7 4 6 8 9 10 1 2 3 54 6 8 910 1 2 3 54 6 8 910 1 2 3 4 4 5 Event time. Watermarks 6 8 9 10 Event Time Windows Ingestion Time WindowsOut or Order Not event time before 5 will come Late Time of 2 5
- 77. 1 2 35 7 4 6 8 9 10 1 2 3 5 74 6 8 910 1 2 3 5 74 6 8 910 1 2 3 4 4 5 Event time. Watermarks 6 7 8 9 10 Event Time Windows Ingestion Time WindowsOut or Order Not event time before 10 will come Late Time of 2 10
- 78. Windowing Windows: grouping ofevents according to time, session*, count
- 79. Windowing Windows: grouping ofevents according to time, session*, count Powerful built-in windows:
- 80. Windowing Windows: grouping ofevents according to time, session*, count Powerful built-in windows: Count: number of events to trigger the window. Process X last events each Y events.
- 81. Windowing Windows: grouping ofevents according to time, session*, count Powerful built-in windows: Count: number of events to trigger the window. Process X last events each Y events. Time: lTumbling: trigger every X time with received events lSliding: trigger every X time with received events in last Y time
- 82. Windowing Windows: grouping ofevents according to time, session*, count Powerful built-in windows: Count: number of events to trigger the window. Process X last events each Y events. Time: lTumbling: trigger every X time with received events lSliding: trigger every X time with received events in last Y time Session: all events from session/user X until session time expired ( Gap )
- 83. Windowing Windows: grouping ofevents according to time, session*, count Powerful built-in windows: Count: number of events to trigger the window. Process X last events each Y events. Time: lTumbling: trigger every X time with received events lSliding: trigger every X time with received events in last Y time Session: all events from session/user X until session time expired ( Gap ) High level API for user windows: Window Assigner, Trigger, Evictor
- 84. Events Time & Windows Fault Tolerance & Correctness StateHandling Low Latency & High Throughput API Libraries SQL Building Blocks lManaged operator state for backup/recovery lSavepoints
- 85.
- 86. Stateful Streaming Op Op State StatelessStream Processing Stateful Stream Processing lBuilt-in internal state in each operator for exactly-once semantics lUser state can be declared in each operator to be saved locally in memory ( API, key/value pars ) lSnapshots: periodically local states in memory are persisted in lightweight distributed snapshots. No global pause !! lCheckpoint as global consistent point-in-time snapshot build by set of distributed snapshots. lPluggable state backend for snapshots: JobManager, HDFS, RocksDB lSavepoints: user-triggered retained checkpoint
- 87. Events Time & Windows Fault Tolerance & Correctness StateHandling Low Latency & High Throughput API Libraries SQL Building Blocks lExactly-once semantics with managed operator state lDistributed Snapshotting Algorithm
- 88. Periodically Chandy-Lamport Snapshots “The global-state-detectionalgorithm is to be superimposed on the underlying computation: It must run concurrently with, but no alter, this underlying computation” . Triggers snapshots asynchronously . Embedded snapshots algorithm in stream of data ( barriers ) . No global pause, lightweight impact in performance Handling Checkpoints
- 89. Periodically Chandy-Lamport Snapshots “The global-state-detectionalgorithm is to be superimposed on the underlying computation: It must run concurrently with, but no alter, this underlying computation” . Triggers snapshots asynchronously . Embedded snapshots algorithm in stream of data ( barriers ) . No global pause, lightweight impact in performance Handling Checkpoints
- 90. Periodically Chandy-Lamport Snapshots “The global-state-detectionalgorithm is to be superimposed on the underlying computation: It must run concurrently with, but no alter, this underlying computation” . Triggers snapshots asynchronously . Embedded snapshots algorithm in stream of data ( barriers ) . No global pause, lightweight impact in performance Handling Checkpoints
- 91. snapshot Job Manager Periodically pushes barriers for newstate New state X+1 Ack for Snapshot state X from Task N Handling Checkpoints
- 92.
- 93.
- 94. snapshot Job Manager Handling Checkpoints AllAcks received Register Checkpoint for restore in case of fail
- 95. Streaming Fault Tolerance Incase of fail, last global checkpoint is recovered ( recovery from partial checkpoint / individual snapshots is coming ) Need of stateful source like kafka to ensure end-to-end exactly-once semantic in case of fail. Kafka sink doesn't guarantee end-to-end exactly-once ( multiple writes in topic ) ( at least-once ) Semantics in Flink: At Least Once: never loses events, events might be reprocessed Exactly once: neither reprocessed nor lost events. Exactly once by default, with low impact in performance
- 96. If you wantto know one thing about Flink is that you don't need to know the internals of Flink
- 97. Events Time & Windows Fault Tolerance & Correctness StateHandling Low Latency & High Throughput API Libraries SQL Building Blocks lPipelined runtime lLatency vs throughput tunning
- 98. Exactly-once semantic withlow impact in performance Controllable checkpointing overhead Higher throughput using processing time Performance improvements thanks to: . operator chaining during optimization phase . own optimized serialization stack with code generation Performance Tunning
- 99. Benchmark for “StreamingComputation” published by Yahoo. Dec 18, 2015 https://yahooeng.tumblr.com/post/135321837876/benchmarking-streaming-computation-engines-at Production use-case lcounting ad impressions group by campaign laggregations over a 10 second window lsave current aggregate value to Redis every second Streaming Benchmark
- 100. Throughput vs LatencyGraph Throughput ( 1000 events / sec ) 99 Percentile Latency ( ms ) Not Operator combinig in Storm, more complicate topology, more steps for events and more overhead
- 101. Apache Storm WithoutTrident lAt least once / Double counting after fail / Lost state after Failures lCPU bounded Apache Spark lLatency increase with throughput Apache Flink lExactly once / No double counting / No state loss lLimited by bandwidth between Kafka and Flink cluster l(1 GigE). lkafka brokers within Kafka Cluster ( 10 GigE ) lAchieved 15 million messages /sec l( before 3 million m/sec) with exactly once semantic 10,000,000 20,000,000 1 GigE 10 GigE Performance Tunning
- 102. Events Time & Windows Fault Tolerance & Correctness StateHandling Low Latency & High Throughput API Libraries SQL Building Blocks lHigh Level API lWide range of basic and advanced operators lJava , Scala. Python soon !!
- 103.
- 104. API Working on datastreams ( bounded ? )
- 105. API Working on datastreams ( bounded ? ) Stream Processing: Explicit Handling of Time
- 106. API Working on datastreams ( bounded ? ) Stream Processing: Explicit Handling of Time Java & Scala. Python coming. Java: Bean type classes vs Tuples with position addresses. Scala: case classes.
- 107. API Working on datastreams ( bounded ? ) Stream Processing: Explicit Handling of Time Java & Scala. Python coming. Java: Bean type classes vs Tuples with position addresses. Scala: case classes. Operators: Sources: kafka, FileSystem, Cassandra … Sinks: Kafka, HDFS, Cassandra …. Transformations: Basic: map, flatmap, filter, grouping, iterate, project, join, cross, … Streaming: Windowing + Aggregations, Temporal Binary Iterative Stream operators
- 108. API Working on datastreams ( bounded ? ) Stream Processing: Explicit Handling of Time Java & Scala. Python coming. Java: Bean type classes vs Tuples with position addresses. Scala: case classes. Operators: Sources: kafka, FileSystem, Cassandra … Sinks: Kafka, HDFS, Cassandra …. Transformations: Basic: map, flatmap, filter, grouping, iterate, project, join, cross, … Streaming: Windowing + Aggregations, Temporal Binary Iterative Stream operators DataStream<?> DataSet<?> Core API 1 implementation*, 2 interfaces
- 109. Source Map Reduce Fliter JoinSum Sink Map Source Operators
- 110. Source Map Reduce Fliter JoinSum Sink Map Source Operators
- 111. Source Map Reduce Fliter JoinSum Sink Source Filter Operators
- 112. Source Map Reduce Fliter JoinSum Sink Source Filter Operators
- 113. Source Map Reduce Fliter JoinSum Sink Source Reduce Operators
- 114. Source Map Reduce Fliter JoinSum Sink Source Reduce Operators
- 115. Source Map Reduce Fliter JoinSum Sink Source Join Operators
- 116. Source Map Reduce Fliter JoinSum Sink Source Join Operators
- 117. Source Map Reduce Fliter JoinSum Sink Source Operators
- 118. Events Time & Windows Fault Tolerance & Correctness StateHandling Low Latency & High Throughput API Libraries SQL Building Blocks lEasy to use. SQL !! lBased on Apache Calcite
- 119. API extension forDataSets y DataStreams Based on relational Table abstraction Table <=> Source / DataSet / DataStream Operators like: where, select, as, groupBy, join, union, minus, distinct, orderBy, ... Table API
- 120. Execute SQL-Like sentenceson DataSets and Datastreams Resuts returned as Table ( Table API ), convertible to DataStream or DataSets SQL and Table API can be seamlessly mixed over DataStream/DataSets Flink’s SQL support is not feature complete, yet. Queries that include unsupported SQL will fail !! SQL Support SQL
- 121. Parsing and Logicalplan for Table operators and SQL are optimized using Apache Calcite Only supported a Subset of the comprehensive SQL standard Apache Calcite provides with: SQL Parsing API for building expressions in relational algebra Query planning engine Provides SQL for Streaming Queries with windows aggregations SELECT STREAM TUMBLE_END(rowtime, INTERVAL '1' HOUR) AS rowtime, productId, COUNT(*) AS c, SUM(units) AS units FROM Orders Apache Calcite SQL Sentence Apache Calcite: SQL to Logical Plan as Relational Algebra Flink Optimizer: Logical Plan to Execution Plan
- 122. If you wantto know one thing about Flink is that you don't need to know the internals of Flink So … Batch
- 123.
- 124. Stream: Unbounded DataStream Unbounded Data Stream Batch on Stream
- 125. Stream: Unbounded DataStream Batch: Bounded stream ( dataset ) on a stream processor Global window over the entire dataset Optimization in operators for joins and grouping, with blocking data exchange if needed Unbounded Data Stream Bounded Data Set Batch on Stream
- 126. Stream: Unbounded DataStream Batch: Bounded stream ( dataset ) on a stream processor Global window over the entire dataset Optimization in operators for joins and grouping, with blocking data exchange if needed Unbounded Data Stream Bounded Data Set Batch on Stream
- 127. Stream: Unbounded DataStream Batch: Bounded stream ( dataset ) on a stream processor Global window over the entire dataset Optimization in operators for joins and grouping, with blocking data exchange if needed Batch specific optimizations: Cost-based optimizer: dataset size known before hand Manage memory on / off-heap for join, sort, … Optimization serialization stack for user-types Bounded Data Set Batch on Stream Unbounded Data Stream
- 128.
- 129.
- 130. Conclusions Flink Pure streamingengine matches real life. No Abstraction
- 131. Conclusions Flink Pure streamingengine matches real life. No Abstraction Batch on streaming
- 132. Conclusions Flink Pure streamingengine matches real life. No Abstraction Batch on streaming Flexible Windowing Semantics with Explicit Time handling
- 133. Conclusions Flink Pure streamingengine matches real life. No Abstraction Batch on streaming Flexible Windowing Semantics with Explicit Time handling Competitive Performance, low latency and hight throughput
- 134. Conclusions Flink Pure streamingengine matches real life. No Abstraction Batch on streaming Flexible Windowing Semantics with Explicit Time handling Competitive Performance, low latency and hight throughput Apache Beam, open sourced by Google, uses Flink as its first order runner for Batch and Streaming processing in partnership with Data Artisans. 100% Compliance of data processing model “what, where, when, how “
- 136.