### SlideShare for iOS

by Linkedin Corporation

FREE - On the App Store

A distributed algorithm for stateful fault recovery

A distributed algorithm for stateful fault recovery

- Total Views
- 598
- Views on SlideShare
- 594
- Embed Views

- Likes
- 0
- Downloads
- 2
- Comments
- 0

http://www.linkedin.com | 3 |

https://www.linkedin.com | 1 |

Uploaded via SlideShare as Adobe PDF

© All Rights Reserved

- 1. TAMC 2007 25 th May, 2007 A Distributed Algorithm of Fault Recovery For Stateful Failover Indranil Saha HTS (Honeywell Technology Solutions) Research Bangalore, India Email: indranil.saha@honeywell.com and Debapriyay Mukhopadhyay Ixia Technologies Kolkata, India Email: dmukhopadhyay@ixiacom.com A Distributed Algorithm of Fault Recovery For Stateful Failover 1
- 2. TAMC 2007 25 th May, 2007 Presentation Outline I will talk about • Introduction • System Models • Distributed Algorithm for Automated Fault Recovery • Formal veriﬁcation of the Distributed Algorithm • Conclusion A Distributed Algorithm of Fault Recovery For Stateful Failover 2
- 3. TAMC 2007 25 th May, 2007 Presentation Outline • Introduction • System Models • Distributed Algorithm for Automated Fault Recovery • Formal veriﬁcation of the Distributed Algorithm • Conclusion A Distributed Algorithm of Fault Recovery For Stateful Failover 3
- 4. TAMC 2007 25 th May, 2007 Introduction • Critical business processes and mission critical systems should provide a high degree of availability and reliability to the end users. • Redundancy techniques are mostly used to achieve fault-tolerance. • Redundancy can be achieved by using extra copies of its components which include hardware, software and network components. A Distributed Algorithm of Fault Recovery For Stateful Failover 4
- 5. TAMC 2007 25 th May, 2007 Stateful and Stateless Failover • Stateless Failover: - Occasional loss of application state information or data is tolerable. - The system can restart without any state or data restoration after a failure. - Any live node in the network is a promising candidate to take over the processes of any failed node • Stateful Failover - Restoration of the state or data pertaining to the application is required for highly accurate recovery. - How to distribute the state information of a node across the network is an important issue. A Distributed Algorithm of Fault Recovery For Stateful Failover 5
- 6. TAMC 2007 25 th May, 2007 Related Works • Graph theoretic models have been extensively used to represent processor-to-processor interconnection structure of fault tolerant designs for speciﬁc multi-processor architectures (Kuhl80, Yang88, Sridhar91, Mukhopadhyay92, Sung00, Hung01). • Minimum k-Hamilton graphs are widely used to meet reliability considerations for loop type communication networks (Mukhopadhyay92, Sung00, Hung01). • Fault tolerant networks based on de Bruijn graph are proposed, which can tolerate up to k − 2 node faults, where the graph is regular of degree k and have k n number of vertices for some n (Sridhar91). None of these works talk about stateful failover. A Distributed Algorithm of Fault Recovery For Stateful Failover 6
- 7. TAMC 2007 25 th May, 2007 Presentation Outline • Introduction • System Models • Distributed Algorithm for Automated Fault Recovery • Formal veriﬁcation of the Distributed Algorithm • Conclusion A Distributed Algorithm of Fault Recovery For Stateful Failover 7
- 8. TAMC 2007 25 th May, 2007 System Model • The network consists of the set of nodes N with |N | = n • Each node is labeled with a unique id from 0 to n − 1. • Each node handles one process initially, and is capable of executing at most m processes simultaneously. • Pi is the process node i starts executing initially when the network becomes functional. • Failures are of failstop kind, i.e., the nodes in the network can stop operating at any point of time due to a crash. • With a processor failed, all the links incident on that node also becomes non-functional. • k node faults are allowed in the network. A Distributed Algorithm of Fault Recovery For Stateful Failover 8
- 9. TAMC 2007 25 th May, 2007 Network Topology Each node i ∈ N, (0 ≤ i ≤ n − 1), in the network is connected to the set of nodes Pi ⊆ N, such that |Pi | = l = k + x, where k + x(≤ n − 1) is even, and Pi = {j ∈ N : j = (i + p)(mod n), where − l/2 ≤ p ≤ l/2, p = 0} Underlying undirected graph modeling the network can be written as (N, E) where n−1 E = ∪i=0 {(i, j) : j ∈ Pi }. The state information of processor i, i ∈ N , is periodically forwarded to all the nodes in the set Fi ⊆ N such that |Fi | = k and Fi = {j ∈ S : j = (i + p)(mod n), where − k/2 ≤ p ≤ k/2 , p = 0} A Distributed Algorithm of Fault Recovery For Stateful Failover 9
- 10. TAMC 2007 25 th May, 2007 Connectivity - The graph (N, E) represents a regular network, for, the degree of each node is l. - For any n and k, the graph (N, E) corresponds to the Harary Graph Hl,n , where k + 2, for k even, l =k+x≥ k + 1, for k odd, The network is l-connected with χ(G) ≥ l(> k), χ(G) denotes the connectivity of G. A Distributed Algorithm of Fault Recovery For Stateful Failover 10
- 11. TAMC 2007 25 th May, 2007 Theoretical Results Theorem 1. A. Forwarding state information of each process to k other nodes in the network ensures k-fault tolerance. B. A suﬃcient condition to ensure k-fault tolerance is to forward the state information by each node to at least k other nodes in the network. Theorem 2. As long as k ≤ m−1 .n , no live node has to execute m more than m processes including one of its own and an algorithm to attain the same under the proposed framework can also be found. Theorem 3. Minimum number of nodes with which any node in a network with n > 2k (or n = 2k) is required to be connected directly is 2k (or 2k − 1) to ensure that all the eligible nodes corresponding to a process can be updated about its state information all the time in one hop. A Distributed Algorithm of Fault Recovery For Stateful Failover 11
- 12. TAMC 2007 25 th May, 2007 Network Example Illustration of a network with n = 10, m = 2 and k = 4 A Distributed Algorithm of Fault Recovery For Stateful Failover 12
- 13. TAMC 2007 25 th May, 2007 Presentation Outline • Introduction • System Models • Distributed Algorithm for Automated Fault Recovery • Formal veriﬁcation of the Distributed Algorithm • Conclusion A Distributed Algorithm of Fault Recovery For Stateful Failover 13
- 14. TAMC 2007 25 th May, 2007 Message Types 1. INFO • In the ﬁrst round, each node i sends an IN F O message to all the nodes in the set Fi . • Message consists of the tuple (j, Fj ) 2. STATUS • Starting from the second round in each successive rounds, every live node i sends ST AT U S message for every process pj that is running on it to all the live nodes in the set Fj . • Message consists of the tuple (pj , Spj ) • Ommision of the Status message for a process for a round indicates the failure of the process. 3. RESOLVED Message is sent to all the nodes in Fj by the node who has resolved the failure of process j. A Distributed Algorithm of Fault Recovery For Stateful Failover 14
- 15. TAMC 2007 25 th May, 2007 Preference for the Neighbours i prefj . denotes the preference of node i to take process j in case of its failure among the nodes in Fj . A Distributed Algorithm of Fault Recovery For Stateful Failover 15
- 16. TAMC 2007 25 th May, 2007 Distributed Algorithm: Example Illustration of the distributed algorithm for a network with n = 10, m = 2 and k = 4 Every node is running its own process. A Distributed Algorithm of Fault Recovery For Stateful Failover 16
- 17. TAMC 2007 25 th May, 2007 Distributed Algorithm: Example Node 9 is faulty A Distributed Algorithm of Fault Recovery For Stateful Failover 17
- 18. TAMC 2007 25 th May, 2007 Distributed Algorithm: Example Node 1 takes the process of node 9 after one round as it is the highest preference node for process 9. A Distributed Algorithm of Fault Recovery For Stateful Failover 18
- 19. TAMC 2007 25 th May, 2007 Distributed Algorithm: Example Node 2 is faulty A Distributed Algorithm of Fault Recovery For Stateful Failover 19
- 20. TAMC 2007 25 th May, 2007 Distributed Algorithm: Example Node 4 takes the process of node 2 after one round as it is the highest preference node for process 2. A Distributed Algorithm of Fault Recovery For Stateful Failover 20
- 21. TAMC 2007 25 th May, 2007 Distributed Algorithm: Example Node 8 is faulty A Distributed Algorithm of Fault Recovery For Stateful Failover 21
- 22. TAMC 2007 25 th May, 2007 Distributed Algorithm: Example Node 0 takes the process of node 8 after one round as it is the highest preference node for process 8. A Distributed Algorithm of Fault Recovery For Stateful Failover 22
- 23. TAMC 2007 25 th May, 2007 Distributed Algorithm: Example Node 0 is faulty. Real problem begins... A Distributed Algorithm of Fault Recovery For Stateful Failover 23
- 24. TAMC 2007 25 th May, 2007 Distributed Algorithm: Example Node 7 takes the process of node 8 after 3 rounds as it is the third preference node for process 8. A Distributed Algorithm of Fault Recovery For Stateful Failover 24
- 25. TAMC 2007 25 th May, 2007 Distributed Algorithm: Example Node 1 stops running process 9 and starts running process 0 after 6 rounds of node 0’s failure. According to Theorem 1 there is at least one node available to take process 9. A Distributed Algorithm of Fault Recovery For Stateful Failover 25
- 26. TAMC 2007 25 th May, 2007 Distributed Algorithm: Example Node 7 stops running process 8 and starts running process 9 after 8 rounds when node 1 stops running process 9. A Distributed Algorithm of Fault Recovery For Stateful Failover 26
- 27. TAMC 2007 25 th May, 2007 Distributed Algorithm: Example Node 6 starts running process 8 after 4 rounds when node 7 stops running process 8. No more failure is possible. A Distributed Algorithm of Fault Recovery For Stateful Failover 27
- 28. TAMC 2007 25 th May, 2007 Analysis of the Algorithm • At most 2k rounds are required to resolve a single fault. • To resolve a single fault, the maximum number of RESOLV ED messages that is required to be sent across the network is (k − 2)m + 1, where m is the maximum number of processes that a node is capable of executing. A Distributed Algorithm of Fault Recovery For Stateful Failover 28
- 29. TAMC 2007 25 th May, 2007 Presentation Outline • Introduction • System Models • Distributed Algorithm for Automated Fault Recovery • Formal veriﬁcation of the Distributed Algorithm • Conclusion A Distributed Algorithm of Fault Recovery For Stateful Failover 29
- 30. TAMC 2007 25 th May, 2007 Correctness of the Algorithm • We show the correctness of the distributed algorithm through formal veriﬁcation. • We use Spin Model checker for modeling and veriﬁcation of the algorithm. • We have been able to verify our model for N=8, K=3 and M=2 and all lower instances. • Due to the state-space explosion problem inherent in model checker SPIN, we could not verity our algorithm for more than 8 processors. A Distributed Algorithm of Fault Recovery For Stateful Failover 30
- 31. TAMC 2007 25 th May, 2007 Spin Model Checker • Tool for automatically model checking distributed algorithms • Promela is a language for modeling systems of concurrent processes that can interact via shared variables and message channels • Given a concurrent system modeled by a Promela program, SPIN can check for deadlock, dead code, violations of user speciﬁed assertions, and temporal properties expressed by LTL formulas • When a violation of a property is detected, SPIN reports a scenario, i.e., a sequence of transitions, violating the property. A Distributed Algorithm of Fault Recovery For Stateful Failover 31
- 32. TAMC 2007 25 th May, 2007 Properties Safety 1 Whenever a node becomes faulty, at least one of its neighboring nodes is non-faulty. Safety 2 No node has to take more than M processes at any point of time. Liveness Whenever a node becomes faulty, its process is eventually taken up by some other live nodes. Timeliness Every fault is recovered in no more than 2K rounds. A Distributed Algorithm of Fault Recovery For Stateful Failover 32
- 33. TAMC 2007 25 th May, 2007 Presentation Outline • Introduction • System Models • Distributed Algorithm for Automated Fault Recovery • Formal veriﬁcation of the Distributed Algorithm • Conclusion A Distributed Algorithm of Fault Recovery For Stateful Failover 33
- 34. TAMC 2007 25 th May, 2007 Conclusion • We have presented a distributed algorithm of automated fault recovery for stateful failover in a network. • In whatever way the fault may arise the algorithm can handle that fault • In at most 2k rounds the processes of the faulty processor are taken up by a(some) eligible live node(nodes) in the network. • The message complexity of our algorithm is linear with the number of nodes. • The correctness of the algorithm has been proved by modeling the algorithm in SPIN and verifying its desired properties. A Distributed Algorithm of Fault Recovery For Stateful Failover 34
- 35. TAMC 2007 25 th May, 2007 Thank You!! A Distributed Algorithm of Fault Recovery For Stateful Failover 35

Full NameComment goes here.