Tech Talk @ Google on Flink Fault Tolerance and HA
•
7 likes•1,941 views
The document summarizes Apache Flink's approach to exactly-once stream processing through distributed snapshots. It discusses how Flink takes asynchronous snapshots of streaming jobs using barriers to define consistent cuts. Snapshots include operator states and records in transit, allowing the job to be reset from the snapshot state. The approach works for both DAG and cyclic dataflow topologies. Flink implements distributed snapshots using a coordinator that triggers snapshots and handles recovery. Snapshots are stored asynchronously to avoid blocking the streaming job execution.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Tech Talk @ Google on Flink Fault Tolerance and HA
Similar to Tech Talk @ Google on Flink Fault Tolerance and HA (20)
More from Paris Carbone
More from Paris Carbone (11)
Recently uploaded
Recently uploaded (20)
Tech Talk @ Google on Flink Fault Tolerance and HA
- 1. Apache Flink Streaming Resiliency and Consistency Paris Carbone - PhD Candidate KTH Royal Institute of Technology <parisc@kth.se, senorcarbone@apache.org> 1 Technical Talk @ Google
- 2. Overview • The Flink Execution Engine Architecture • Exactly-once-processing challenges • Using Snapshots for Recovery • The ABS Algorithm for DAGs and Cyclic Dataflows • Recovery, Cost and Performance • Job Manager High Availability 2
- 3. Current Focus 3 Streaming APIBatch API Flink Optimiser Flink Runtime Table ML Gelly ML Gelly StateManagement
- 4. Executing Pipelines • Streaming pipelines translate into job graphs 4 Flink Runtime Flink Job Graph Builder/Optimiser Flink ClientUser Pipeline Job Graph
- 5. Unbounded Data Processing Architectures 5 (Hadoop, Spark) (Spark Streaming) 1) Streaming (Distributed Data Flow) LONG-LIVED TASK EXECUTION STATE IS KEPT INSIDE TASKS 2) Micro-Batch
- 6. The Flink Runtime Engine • Tasks run operator logic in a pipelined fashion • They are scheduled among workers • State is kept within tasks 6 LONG-LIVED TASK EXECUTION Job Manager • scheduling • monitoring
- 7. Task Failures • Task failures are guaranteed to occur • We need to make them transparent • Can we simply recover a task from scratch? 7 task state consistent state no duplicatesno loss
- 8. Record Acknowledgements • We can monitor the consumption of every event • It guarantees no loss • Duplicates and inconsistent state are not handled 8 task state consistent state no duplicatesno loss bookkeeping
- 9. Atomic transactions per update • It guarantees no loss and consistent state. • No duplication can be achieved e.g. with bloom filters or caching • Fine grained recovery • Non-constant load in external storage can lead to unsustainable execution 9 task state consistent state no duplicatesno loss state state state Distributed Storage Trident
- 10. Lessons Learned from Batch 10 batch-1batch-2 • If a batch computation fails, simply repeat computation as a transaction • Transaction rate is constant • Can we apply these principles to a true streaming execution?
- 12. Distributed Snapshots 11 “A collection of operator states and records in transit (channels) that reflects a moment at a valid execution”
- 13. Distributed Snapshots 11 “A collection of operator states and records in transit (channels) that reflects a moment at a valid execution” p1 p2 p3 p4
- 14. Distributed Snapshots 11 “A collection of operator states and records in transit (channels) that reflects a moment at a valid execution” p1 p2 p3 p4
- 15. Distributed Snapshots 11 “A collection of operator states and records in transit (channels) that reflects a moment at a valid execution” p1 p2 p3 p4 consistent cut
- 16. Distributed Snapshots 11 “A collection of operator states and records in transit (channels) that reflects a moment at a valid execution” p1 p2 p3 p4 consistent cut
- 17. Distributed Snapshots 11 “A collection of operator states and records in transit (channels) that reflects a moment at a valid execution” p1 p2 p3 p4 consistent cut causality violation
- 18. Distributed Snapshots 11 “A collection of operator states and records in transit (channels) that reflects a moment at a valid execution” p1 p2 p3 p4 consistent cut inconsistent cut causality violation
- 19. Distributed Snapshots 11 “A collection of operator states and records in transit (channels) that reflects a moment at a valid execution” p1 p2 p3 p4 consistent cut inconsistent cut causality violation Idea: We can resume from a snapshot that defines a consistent cut
- 20. Distributed Snapshots 11 Assumptions • repeatable sources • reliable FIFO channels “A collection of operator states and records in transit (channels) that reflects a moment at a valid execution” p1 p2 p3 p4 consistent cut inconsistent cut causality violation Idea: We can resume from a snapshot that defines a consistent cut
- 25. Distributed Snapshots 12 t2t1 snap - t1 snap - t2
- 26. Distributed Snapshots 12 t2t1 snap - t1 snap - t2
- 27. Distributed Snapshots 12 t3t2t1 snap - t1 snap - t2
- 28. Distributed Snapshots 12 t3t2t1 reset from snap t2 snap - t1 snap - t2
- 29. Distributed Snapshots 12 t3t2t1 reset from snap t2 snap - t1 snap - t2 Assumptions • repeatable sources • reliable FIFO channels
- 30. Taking Snapshots 13 t2t1 execution snapshots Initial approach (see Naiad) • Pause execution on t1,t2,.. • Collect state • Restore execution
- 31. Lamport On the Rescue 14 “The global-state-detection algorithm is to be superimposed on the underlying computation: it must run concurrently with, but not alter, this underlying computation”
- 32. Chandy-Lamport Snapshots 15 p1 p2 p1 Using Markers/Barriers • Triggers Snapshots • Separates preshot-postshot events • Leads to a consistent execution cut markers propagate the snapshot execution under continuous ingestion markers
- 33. Asynchronous Snapshots 16 t2t1 snap - t1 snap - t2 snapshotting snapshotting propagating markers
- 51. Asynchronous Barrier Snapshotting Benefits 27 • Taking advantage of the execution graph structure • No records in transit included in the snapshot • Aligning has lower impact than halting
- 58. Snapshots on Cyclic Dataflows • Checkpointing should eventually terminate • Records in transit within loops should be included in the checkpoint
- 67. Implementation in Flink 38 • Coordinator/Driver Actor per job in JM • sends periodic markers to sources • collects acknowledgements with state handles and registers complete snapshots • injects references to latest consistent state handles and back-edge records in pending execution graph tasks • Tasks • snapshot state transparently in given backend upon receiving a barrier and send ack to coordinator • propagate barriers further like normal records
- 68. Recovery 39 • Full rescheduling of the execution graph from the latest checkpoint - simple, no further modifications • Partial rescheduling of the execution graph for all upstream dependencies - trickier, duplicate elimination should be guaranteed
- 70. Node Failures 41 Job Manager Task Manager Task Manager Task Manager task task task Client job graph optimiser JM State • Pending Execution Graphs • Snapshots (State Handles)
- 71. Node Failures 41 Job Manager Task Manager Task Manager Task Manager task task task Client job graph optimiser JM State • Pending Execution Graphs • Snapshots (State Handles)
- 72. Node Failures 41 Job Manager Task Manager Task Manager Task Manager task task task task Client job graph optimiser JM State • Pending Execution Graphs • Snapshots (State Handles)
- 73. Node Failures 41 Job Manager Task Manager Task Manager Task Manager task task task task Client job graph optimiser JM State • Pending Execution Graphs • Snapshots (State Handles)
- 74. Node Failures 41 Job Manager Task Manager Task Manager Task Manager task task task task Client job graph optimiser JM State • Pending Execution Graphs • Snapshots (State Handles) Single Point Of Failure
- 75. Passive Standby JM 42 JM JM JM Zookeeper State • Leader JM Address • Pending Execution Graphs • State Snapshots Client
- 77. Eventual Leader Election 43 JM JM JMt0 time
- 78. Eventual Leader Election 43 JM JM JMt0 t1 JM JM JM time
- 79. Eventual Leader Election 43 JM JM JMt0 t1 t2 JM JM JM JM JM JM …recovering time
- 80. Eventual Leader Election 43 JM JM JMt0 t1 t2 t3 JM JM JM JM JM JM …recovering JM JMJM time
- 81. Scheduling Jobs 44 Job Manager Task Manager Task Manager Task Manager Client optimiser Zookeeper
- 86. Scheduling Jobs 44 Job Manager Task Manager Task Manager Task Manager task task task task Client optimiser Zookeeperjob-1 execution graph task blocking write in Zookeeper
- 87. Logging Snapshots 45 Job Manager Task Manager Task Manager Task Manager task task task task Zookeeper State Backend asynchronous writes in Zookeeper
- 88. Logging Snapshots 45 Job Manager Task Manager Task Manager Task Manager task task task task Zookeeper State Backend asynchronous writes in Zookeeper Checkpointing states
- 89. Logging Snapshots 45 Job Manager Task Manager Task Manager Task Manager task task task task Zookeeper State Backend asynchronous writes in Zookeeper Checkpointing states
- 90. Logging Snapshots 45 Job Manager Task Manager Task Manager Task Manager task task task task Zookeeper State Backend asynchronous writes in Zookeeper state handle Checkpointing states
- 91. Logging Snapshots 45 Job Manager Task Manager Task Manager Task Manager task task task task Zookeeper State Backend asynchronous writes in Zookeeper state handle Checkpointing states
- 92. Logging Snapshots 45 Job Manager Task Manager Task Manager Task Manager task task task task Zookeeper State Backend asynchronous writes in Zookeeper state handle Checkpointing states
- 93. Logging Snapshots 45 Job Manager Task Manager Task Manager Task Manager task task task task Zookeeper State Backend asynchronous writes in Zookeeper state handle Checkpointing states
- 94. Logging Snapshots 45 Job Manager Task Manager Task Manager Task Manager task task task task Zookeeper State Backend asynchronous writes in Zookeeper state handle Checkpointing states
- 95. Logging Snapshots 45 Job Manager Task Manager Task Manager Task Manager task task task task Zookeeper State Backend asynchronous writes in Zookeeper global snapshot state handle Checkpointing states
- 96. HA in Zookeeper • Task Managers query ZK for the current leader before connecting (with lease) • Upon standby failover restart pending execution graphs • Inject state handles from the last global snapshot
- 97. Summary • The Flink execution monitors long running tasks • Establishing exactly-once-processing guarantees with snapshots can be achieved without halting the execution • The ABS algorithm persists the minimal state at a low cost • Master HA is achieved through Zookeeper and passive failover