1       Toward Optimal Network Fault Correction in         Externally Managed Overlay Networks                            ...
2                                                                                                    logical              ...
3                                                                     Logical tree                                    Bad ...
4nodes. However, we do not consider the repair cost since             complexity of finding the expected cost of a diagnosi...
5                                        1                                                                                ...
6                1                                i   pi    ci    φ(i, T )                                                ...
7a best node for a given two-path bad tree of size |N |           Algorithm 2 Pa-Naive-Probunder different distributions o...
8 Proportion of correct instances     1                                                                                   ...
9                        10                                                                                               ...
10costs can lead to a large performance improvement of              18%). Figure 12(b) shows the performance of the multi-...
Toward optimal network fault correction in externally managed overlay networks
Toward optimal network fault correction in externally managed overlay networks
Toward optimal network fault correction in externally managed overlay networks
Toward optimal network fault correction in externally managed overlay networks
Upcoming SlideShare
Loading in …5

Toward optimal network fault correction in externally managed overlay networks


Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Toward optimal network fault correction in externally managed overlay networks

  1. 1. 1 Toward Optimal Network Fault Correction in Externally Managed Overlay Networks Patrick P. C. Lee, Vishal Misra, and Dan Rubenstein Abstract—We consider an end-to-end approach of inferring probabilistic data-forwarding failures in an externally managed overlay network, where overlay nodes are independently operated by various administrative domains. Our optimization goal is to minimize the expected cost of correcting (i.e., diagnosing and repairing) all faulty overlay nodes that cannot properly deliver data. Instead of first checking the most likely faulty nodes as in conventional fault localization problems, we prove that an optimal strategy should start with checking one of the candidate nodes, which are identified based on a potential function that we develop. We propose several efficient heuristics for inferring the best node to be checked in large-scale networks. By extensive simulation, we show that we can infer the best node in at least 95% of time, and that first checking the candidate nodes rather than the most likely faulty nodes can decrease the checking cost of correcting all faulty nodes. Index Terms—network management, network diagnosis and correction, fault localization and repair, reliability engineering. ! Note: An earlier and shorter conference version of this deploying and maintaining numerous monitoring pointspaper appeared in IEEE INFOCOM ’07 [23]. in a large-scale network introduces heavy computational overhead in collecting network statistics [10] and in- volves complicated administrative management [7]. In1 I NTRODUCTION particular, it is difficult to directly monitor and access allNetwork components are prone to a variety of faults overlay nodes in an externally managed network, whosesuch as packet loss, link cut, or node outage. To prevent routing nodes are independently operated by variousthe faulty components from hindering network applica- administrative domains. In this case, we can only infertions, it is important to diagnose (i.e., detect and localize) the network condition from end-to-end information.the components that are the root cause of network faults, Here, we consider an end-to-end inference approachas in [18], [19], [32]. However, it is also desirable to repair which, using end-to-end measurements, infers compo-the faulty components to enable them to return to their nents that are probably faulty in forwarding data in anoperational states. Therefore, we focus on network fault application-layer overlay network whose overlay nodescorrection, by which we mean not only to diagnose, but are externally managed by independent administrativealso to repair all faulty components within a network. domains. We start with a routing tree topology with a setIn addition, it has been shown that a network outage of overlay nodes, since a tree-based setting is typicallycan bring significant economic loss. For example, the seen in destination-based routing (e.g., CAN [30] andrevenue loss due to a 24-hour outage of a Switzerland- Chord [33]), where each overlay node builds a routingbased Internet service provider can be more than CHF tree with itself as a root, as well as in multicast routing,30 million [15]. As a result, we want to devise a cost- where a routing tree is built to connect members ineffective network fault correction mechanism that corrects a multicast group. We then monitor every root-to-leafall network faults at minimum cost. overlay path. If a path exhibits any “anomalous be- To diagnose (but not repair) network faults, recent havior” in forwarding data, then some “faulty” overlayapproaches like [4], [28], [36] use all network nodes to node on the path must be responsible. In practice, thecollaboratively achieve this. For instance, in hop-by-hop precise definition of an “anomalous behavior” dependsauthentication [4], each hop inspects packets received on specific applications. For instance, a path is said to befrom its previous hop and reports errors when packets anomalous if it fails to deliver a number of correct pack-are found to be corrupted. While such a distributed ets within a time window. Using the path informationinfrastructure can accurately pinpoint network faults, collected at the application endpoints (i.e., leaf nodes), we can narrow down the space of possibly faulty overlay• P. Lee is with the Department of Computer Science and Engineering, The nodes. Chinese University of Hong Kong, Hong Kong. In the above end-to-end solution, one can tell whether E-mail: pclee@cse.cuhk.edu.hk• V. Misra and D. Rubenstein are with the Department of Computer Science, a path behaves anomalously, but cannot specifically tell Columbia University, New York, NY 10027. which and how many overlay nodes on the path are E-mail: {misra, danr}@cs.columbia.edu. faulty. Since we cannot directly monitor and access externally managed overlay nodes, in order to correct
  2. 2. 2 logical badthe faulty nodes, we need to contact the administrators Network tree T Retaining only tree T Inference best node(s)of the corresponding domains to manually check a se- topology bad paths algorithmquence of potentially faulty nodes and fix any nodes failure checkingthat are found to be actually faulty. Given the anomalous probabilities {pi} costs {ci}paths in a tree, our main goal is to infer the best node(or the best set of nodes) that should be first checked so Fig. 1. End-to-end inference approach for a network faultas to minimize the expected cost of correcting all faulty correction scheme.nodes. In this paper, we develop several optimality results for 2 E XTERNALLY M ANAGED OVERLAY N ET-inferring the best node that should be first checked by WORKSa network fault correction scheme, with an objective to In this paper, we are interested in diagnosing and repair-minimize the expected cost of correcting all faulty nodes. ing faulty nodes in an externally managed overlay network,Our contributions include the following: in which overlay nodes are independently operated by multiple administrative domains. By an administra- • Conventional failure localization problems (e.g., tive domain, we mean a single administrative authority [18], [19], [32]) seek to identify the most likely faulty that controls a collection of resources (e.g., routers and node (i.e., the node with the highest conditional fail- servers) [8]. Examples of externally managed overlay ure probability given a network with faulty nodes). networks include Resilient Overlay Network (RON) [3], Here, we show that first checking the node that which provides routing resilience toward Internet path is most likely faulty or has the least checking cost outages, and Service Overlay Network (SON) [14], which does not necessarily minimize the expected cost of provides end-to-end quality-of-service guarantees. Both correcting all faulty nodes. RON and SON deploy overlay nodes over multiple • We formally identify a subset of nodes termed can- administrative domains that cooperatively accomplish didate nodes, one of which should be first checked in certain network services. To ensure the availability of order to minimize the expected cost of correcting all these network services, an effective network fault mech- faulty nodes. We develop a potential function that anism is therefore necessary. determines the candidate nodes. Researchers also advocate the notion of security- • Based on the potential function and candidate nodes oriented overlay architectures, such as Secure Overlay that we propose, we devise various heuristics for Services (SOS) [20] and Mayday [2], to defend against the best node inference in a single tree and mul- denial-of-service attacks. In both SOS and Mayday, data tiple trees, where the latter forms a more general is securely tunneled over an overlay network. As shown topology. We show via simulation that the candidate in [20], successful data delivery is preserved with a very node with the highest potential value is in fact the high probability even if a subset of overlay nodes are best node that should be first checked by an optimal failed (e.g., shut down by attackers). While data may strategy in at least 95% of time under a special be re-routed to bypass failed nodes, robustness of data setting. In addition, we conduct simulation using delivery will be degraded if the failed nodes are not large-scale network topologies. As compared to the immediately repaired because attackers can now devote strategies that first check the most likely faulty node resources to attacking the remaining non-failed nodes. (or the set of most likely faulty nodes), we show Note that security-oriented overlay networks might be that by first checking the candidate nodes, we can deployed over a number of end sites rather than a decrease the checking cost of correcting all faulty single ISP [20], and can be viewed as externally managed nodes, for example, by more than 30% in some networks since each end site is independently operated. scenarios. Thus, our proposed network fault correction mechanism is also essential for this type of networks. The remainder of the paper is organized as follows.In Section 2, we describe a class of externally managed 3 P ROBLEM F ORMULATIONoverlay networks where our network fault correction In this section, we formulate our end-to-end inferencemechanism is applicable. In Section 3, we formulate approach for network fault correction. We are interestedthe network fault correction problem. Section 4 demon- in diagnosing and repairing faulty nodes in an externallystrates that naive strategies that are intuitively optimal managed overlay network, in which the overlay nodesare in fact not optimal in general. In Section 5, we intro- cannot be directly accessed. In order to diagnose failures,duce a potential function for identifying the candidate we need to contact the administrators of the correspond-nodes and show the optimality results. In Section 6, we ing domains to manually check potentially faulty nodespropose several heuristics for the best node inference. In (e.g., by conducting a set of sanity tests). Any nodes thatSection 7, we evaluate the proposed heuristics in large- are found to be actually faulty will then be repaired.scale networks. We review related work in Section 8 and Figure 1 summarizes the end-to-end inference ap-conclude the paper in Section 9. proach. We consider a logical tree as T = (N, {pi }, {ci }),
  3. 3. 3 Logical tree Bad treewhere N is the set of overlay nodes, pi is the failure 1 1 good nodeprobability of node i ∈ N , and ci is the checking cost of 2 3 4 2 3deciding if node i ∈ N is faulty. The overlay node set Retaining bad pathsN provides the topological information and specifies the 5 6 7 8 5 6 7 bad bad bad good bad bad badsequence of overlay nodes, and hence the corresponding path path path path path path pathadministrative domains, along which each data packet istraversed. On the other hand, the construction of {pi } Fig. 2. Given a logical tree, we retain only the bad pathsand {ci } are based on the following failure and cost and indicate any good node. Since path 1, 4, 8 is a goodmodels, respectively. path, it is known that nodes 1, 4, and 8 are good. Nodes Failure model. Our analysis focuses on overlay node 4 and 8 can be pruned from the tree, and node 1 can befailures such that nodes cannot properly forward data. indicated as good. The resulting set of bad paths will leadFor example, nodes can delay or drop packets to be to a bad tree.forwarded because of power outage, full transmissionqueues, hardware errors, or route misconfiguration. Inaddition, our analysis focuses on fail-stop failures, mean- discussion, each node in T is referred to as bad if it ising that a node completely stops its operations upon faulty, or as good otherwise. We say a path is bad if itfailures. Examples of fail-stop failures include power contains at least one bad node, and is otherwise good.outage or machine shutdown. Under the fail-stop model, Also, we call T a bad tree if every path in the tree is athe failure probabilities {pi } can then be characterized bad path.via statistical measurements of reliability indexes [17] Since our failure model focuses on fail-stop failures,or vulnerability modeling [13]. However, we note that we further assume that a node exhibits the same be-constructing accurate failure probabilities for overlay havior across all paths upon which it lies. With thisnodes is difficult in practice. Thus, we also evaluate assumption, if a node lies on at least one good path,the impact on our proposed approach due to inaccurate then we infer that it is a good node. Note that a goodestimates of failure probabilities (see Section 7). Here, we node may still lie on one or more bad paths, but thisconsider the case where each overlay node is a physical only means each such bad path contains some other badnode that possesses a single identity, and hence we node. On the other hand, if a node exhibits differentassume that node failures and the corresponding failure behaviors across different paths, our analysis becomesprobabilities {pi } are all independent. more complicated, as a good path now possibly contains Cost model. We characterize the checking costs {ci } bad nodes as well. We pose the analysis of differentusing the personnel hours and wages required for trou- behaviors of a node as future work.bleshooting problems or the cost of test equipment. Thus, Given a logical tree T , we determine whether a paththe checking costs can be highly varying, depending is good or bad via end-to-end measurements that areon the administrative domains in which the checked carried out between the root and leaf nodes of T . Fornodes resides. Note that we do not consider the cost example, the root can send probes to the leaf nodes,of repairing overlay nodes, and that the total checking from which we collect the measurement results. Since wecost is the only cost component being considered in our focus on the data-forwarding failures, the probes shouldoptimization problem, as explained later in this section. represent the regular data packets that can be forwarded In our analysis, we assume that pi and ci can be any by overlay nodes. Since a good path contains only goodvalues in [0, 1] and [0, ∞), respectively. For instance, if nodes that need not be checked, we only need to focusci = 1 for all i, then the total checking cost denotes the on the bad paths in T . Also, any node that lies on bothnumber of nodes that have been checked. Also, depend- good and bad paths is indicated to be good. To illustrate,ing on the fault definition, the failure probabilities {pi } Figure 2 shows how to retain only the bad paths in Tcan be significantly small for general failures [18], but and indicate the good nodes. The resulting set of badcan also be non-negligible if we are concerned with how paths will then form a bad tree. With a slight abuse oflikely a node is brought down due to catastrophic events notation, we denote this bad tree by T as well.that lead to large-scale failures. We then pass the bad tree T to the inference algorithm, Each node in a logical tree T is classified as faulty or which determines, from the set of nodes that are notnon-faulty, depending on how we first define whether indicated as good, the “best” node (or the “best” set ofa (root-to-leaf) path exhibits any “anomalous behavior”. nodes) to be checked, and repaired if necessary.In practice, such a definition varies across applications. Before formalizing the notion of “best”, we first stateFor example, we say that a path behaves anomalously our optimization goal, namely, to minimize the expectedif it fails to deliver a number of correct packets within cost of correcting all faulty nodes in a given bad tree T .a time window, and that some node on the path is Here, we assume that using end-to-end measurementsfaulty if it causes severe packet loss and delay. Note that (i.e., sending probe packets from the root to leaf nodes)the inference approach only knows whether a path is to determine bad paths incurs negligible cost. Thus, theanomalous, but does not specifically know which and correction cost has two main components: the cost ofhow many nodes on the path are faulty. To help our checking all nodes and the cost of repairing all faulty
  4. 4. 4nodes. However, we do not consider the repair cost since complexity of finding the expected cost of a diagnosis se-all faulty nodes have to be recovered eventually, and quence is Θ(|N |3 )). Therefore, in the following sections,any successful repair strategy has the same total cost of we seek to answer the following question: Given a badrepairing all faulty nodes. As a result, by cost, we here tree T , how can the inference algorithm determine the bestrefer to the checking cost only. node in polynomial time? Because of our optimization goal, we can focus onthe sequential case where we check one node at a time, Algorithm 1 Brute-force inference algorithmsince checking multiple nodes simultaneously does not Input: Bad tree T = (N, {pi }, {ci })improve the expected checking cost, even though it 1: S ∗ = φ, c∗ = ∞reduces the time required to repair all bad nodes on all 2: for all diagnosis sequence S dobad paths. As a result, our theoretical analysis assumes 3: compute c = the expected cost of S 4: if c < c∗ thenthat the inference algorithm returns only a single best 5: S ∗ = S, c∗ = cnode, while we evaluate via simulation the impact of 6: return the first node in S ∗simultaneously inferring and checking multiple nodesin Section 7. With the optimization goal, we select the “best” nodebased on a diagnosis sequence S = l1 , l2 , · · · , l|N | , defined 4 N AIVE H EURISTICS FOR THE I NFERENCEas the order of nodes to be examined given a bad tree T . A LGORITHMWhen node li is examined, it is either checked or skipped. Intuitively, the best node returned by the inference al-If node li lies on bad paths only, we cannot tell whether gorithm could be either the node that has the highestit is good or bad. In this case, we have to check node li , conditional failure probability given a bad tree T (seeand repair it if it is determined to be bad. On the other Appendix A for its calculation, whose complexity ishand, if node li lies on a good path, it is known to be a Θ(|N |2 )), or the node that has the least checking cost.good node and does not need to be checked. In this case, In this section, we show that these naive choices do notwe say we skip node li . After node li has been checked necessarily minimize the expected cost of correcting allor skipped, it is known to be good. Thus, given a bad faulty nodes.tree T , the expected (checking) cost with respect to S is: We first give a simple counter-example that disproves|N | the above naive choices. Figure 3 illustrates a bad tree bad tree T , and nodes l1 , rooted at node 1 and the corresponding failure proba- cli Pr(node li is checked ). · · ·, li−1 known to be good bilities {pi } and checking costs {ci }. As verified by thei=1 brute-force approach in Algorithm 1, the best node isWe detail in Appendix A the calculation of the expected node 2, where a possible optimal diagnosis sequence iscost of a diagnosis sequence. A diagnosis sequence S is 2, 1, 3, 4 and has expected cost 1.044. However, node 2said to be optimal if its expected cost is minimum among is neither the node with the highest conditional failureall possible diagnosis sequences. Therefore, among all probability, nor the node with the least checking cost,the nodes in a bad tree T , we formally define the best nor the node with the highest ratio of the conditionalnode as the first node in an optimal diagnosis sequence failure probability to the checking cost.for T . In other words, the best node should be the nodeto be first checked in order to minimize the expected cost 1 i pi ci Cond. pi (Cond. pi ) : ciof correcting all faulty nodes. T 1 0.1 1 0.6586 0.6586 We point out that the optimal decision of the inference 2 2 0.05 0.3 0.3293 1.0977algorithm is derived from the current topology. The 3 0.08 0.2 0.1214 0.6072 3 4 4 0.1 0.1 0.1405 1.405decision will be revised for a new topology when thefaults are actually checked and repaired. In spite of Fig. 3. A counter-example of showing the best nodethis, the topology may still change between the time (which is node 2) is not the one chosen by the naiveof identifying the physical topology and performing the choices.inference algorithm. However, as stated in [29], topologychange occurs in a coarse time scale (on the order of To further understand the performance of the naiveminutes and hours). Thus, as long as the network fault choices, we evaluate three naive heuristics for the in-correction scheme has its monitoring period bounded ference algorithm: (i) Naive-Prob, which returns the nodewithin the time scale of topology changes, the topology with the highest conditional failure probability, (ii) Naive-should remain fairly stable. Cost, which returns the node with the least checking cost, A straightforward way to implement the inference and (iii) Naive-Prob-Cost, which returns the node with thealgorithm is based on the brute-force approach as shown highest ratio of the conditional failure probability to thein Algorithm 1, which enumerates all possible diagnosis checking cost. We compare their performance to that ofsequences in order to determine the best node. How- the brute-force inference algorithm in Algorithm 1 usingever, the brute-force approach has factorial complexity a special small-scale setting where a bad tree comprisesΘ(|N |3 |N |!) (note that as shown in Appendix A, the only two bad paths (e.g., see Figure 3). Such a setting
  5. 5. 5 1 1 0.75 Proportion of correct instances Naive-Prob Proportion of correct instances Proportion of correct instances 0.9 0.7 Naive-Cost Naive-Prob 0.8 Naive-Cost 0.8 Naive-Prob-Cost 0.65 0.7 Naive-Prob-Cost 0.6 0.6 0.6 Naive-Prob 0.5 Naive-Cost 0.55 Naive-Prob-Cost 0.4 0.4 0.5 0.3 0.45 0.2 0.2 0.1 0.4 0 0 0.35 4 6 8 10 12 14 16 4 6 8 10 12 14 16 4 6 8 10 12 14 16 |N| = Number of nodes |N| = Number of nodes |N| = Number of nodes (a) pi ∼ U (0, 1), ci = 1 (b) pi = 0.1, ci ∼ U (0, 1) (c) pi ∼ U (0, 1), ci ∼ U (0, 1)Fig. 4. Proportions of instances in which Naive-Prob, Naive-Cost, and Naive-Prob-Cost return a correct best node.Note that in (a), Naive-Prob and Naive-Prob-Cost are identical.is to describe a scenario where most routing paths are 5.1 Definitions of a Candidate Node and Potentialdisjoint and at most two paths share the same physical Definition 1: The potential of node i in a tree T iscomponents. defined as the value returned by the potential function: Our evaluation setup is described as follows. For a Pr(T |Xi , Ai )pifixed number of nodes |N |, we randomly generate 200 φ(i, T ) = .bad trees, each of which comprises two bad paths such ci (1 − pi )that the position of the only non-leaf node that has Intuitively, the best node should be a node with a hightwo child nodes is randomly chosen. Since, as stated in potential, since such a node in general has a small check-Section 3, pi and ci can be any values in [0, 1] and [0, ∞), ing cost, a large failure probability, and a large likelihood pirespectively, we arbitrarily choose three distributions of leading to a bad tree. Note that the term ci (1−pi )for our evaluation: (a) pi ∼ U (0, 1) and ci = 1, (b) denotes the potential of a node for a single-bad-pathpi = 0.1 and ci ∼ U (0, 1), as well as (c) pi ∼ U (0, 1) case (see Corollary 1 and [17]). Therefore, the potentialand ci ∼ U (0, 1), where U (u1 , u2 ) denotes the uniform function φ(i, T ) is to combine the potential of a single pidistribution between u1 and u2 . path (i.e., ci (1−pi ) ) and the likelihood of having a bad Figure 4 illustrates the proportions of instances (out tree with respect to the tree topology (i.e., Pr(T |Xi , Ai )).of 200) in which Naive-Prob, Naive-Cost, and Naive- The potential function is derived based on the optimalityProb-Cost return correctly a best node (which may not results presented in the following subsection.be unique). Depending on the distributions of pi and ci , The potential of node i can be obtained by first com-the proportion of instances where the best choice is made puting Pr(T |Xi , Ai ) = Pr(T | pi = 1, pj = 0 ∀j ∈ Ai ),can be as low as 10% for Naive-Prob and Naive-Cost where Ai is the set of ancestors of node i. As shown in(see Figures 4(a) and 4(b)), and less than 55% for Naive- Appendix A, we can compute the potential of a node inProb-Cost (see Figure 4(c)). Therefore, to determine the Θ(|N |2 ) time.best node to first check, our inference algorithm should Definition 2: For each path W in a tree T , we selectuse a measure that better combines failure probabilities, a single node i to be a candidate node if node i lies onchecking costs, as well as the structure of the bad tree. W and its potential φ(i, T ) is the highest among all the nodes on W . If more than one node on W has the highest potential, then we choose the one that is closest to the root node of T as the candidate node.5 C ANDIDATE N ODES We can view the selection of candidate nodes as a mapping function that returns a single candidate nodeInstead of the naive choices described in the previous given W and T . Note that it is possible for a path insection, we show in this section that we should first T to have multiple candidate nodes that correspond tocheck a candidate node, which is selected based on the different paths in T .maximization of a potential function as described below. We first give the notation and definitions that we will 5.2 Optimality Resultsuse. Given a tree T , we define ancestors of node i to be We now state the main optimality result as follows. Inthe nodes (not including node i) on the path from the the interest of space, the formal proofs are detailed inroot of T to node i, and descendants of node i to be the [22].nodes that have node i as one of their ancestors. Let T Theorem 1: Given a bad tree T , there always exists anbe the event that T is a bad tree, and Xi be the event optimal diagnosis sequence that starts with a candidatethat node i is a bad node. Let Ai be the event that the node.ancestors of node i are all good. If node r is the root In other words, the best node returned by our infer-node, then we let Ar be always true and Pr(Ar ) = 1. ence approach should be one of the candidate nodes. We
  6. 6. 6 1 i pi ci φ(i, T ) i pi ci T 1 0.4 1 0.67 T 1 T’ 1 0.939 1 2 2 0.4 1 0.67 3 2 0.806 1 3 0.9 1 8.46 2 3 3 0.042 1 3 4 4 0.9 0.9 9.9 4 5 4 0.979 1 5 0.9 1 8.46 4 5 6 7 5 0.326 1 5 6 6 0.4 1 0.66 6 7 6 0.992 1 7 0.997 1Fig. 5. Counter-example 1. Fig. 7. Counter-example 3. i pi ci φ(i, T ) 1 T 1 0.1 1 0.111 2 0.9 1 7.2 2 3 3 0.8 1 3.6 best node (so is node 5). A possible optimal diagnosis sequence is 3, 5, 1, 2, 4, 6 , whose expected cost 4.309,Fig. 6. Counter-example 2. while the expected cost of any diagnosis sequence that starts with node 4 is at least 4.344. Counter-example 2: Checking simultaneously all candidatepresent a sketch of proof of Theorem 1 in Appendix B nodes in a bad tree does not minimize the expected cost ofto illustrate the intuition of how the potential function correcting all faulty nodes. Figure 6 illustrates a bad tree Tis derived and why a candidate node should be first that has nodes 2 and 3 as the candidate nodes. To checkchecked. The formal proof is presented in [22]. simultaneously all candidate nodes, we can construct a In some special cases, we can identify the best node diagnosis sequence S = 2, 3, 1 , whose expected cost isfrom a set of candidate nodes, as shown in the following given by 2.134. However, the optimal diagnosis sequencecorollaries. is S ∗ = 2, 1, 3 , whose expected cost is 2.107. Corollary 1: Given a bad tree T which is a single path, pi To explain this counter-example, note that after node 2the best node is the one with the maximum ci (1−pi ) . has been checked, we have a new tree T = (N, {pi |p2 = Remark: Corollary 1 implies that the optimal diagnosis pi 0}, {ci }), where setting p2 = 0 is to indicate that node 2 issequence is in non-increasing order of ci (1−pi ) . This now known to be good (see Section 3). The potentials ofconforms to the result in [17]. nodes 1 and 3 in T are respectively φ(1, T ) = 0.111 and In general, whether a node is the best depends on the φ(3, T ) = 0. Thus, node 1 is the only candidate node infailure probabilities {pi }, the checking costs {ci }, as well T , and it must be the next node to check.as the tree topology T . In spite of this, there are cases Counter-example 3: The best node for a bad tree is notwhere the most likely faulty node is in fact the best node. necessarily the best node for a subtree. Consider a bad tree Corollary 2: Given a bad tree T , if every node i has T in Figure 7. The best node is node 7, and the optimalpi = p and ci = c, where p and c are some fixed constants, diagnosis sequence is 7, 1, 2, 4, 3, 5, 6 . However, for thethen the best node is the root node of T . subtree T rooted at node 3, if it is a bad tree, then the Remark: If T is a bad tree, then the root node r is also best node is node 4, and the optimal diagnosis sequencethe node that has the maximum conditional failure prob- is 4, 3, 7, 5, 6 . The reason is that the subtree T , whenability since Pr(Xr |T ) = Pr(T ) ≥ Pr(T |X) )p = Pr(Xi |T ) p i Pr(T residing in T , may or may not be a bad tree. Evenfor every node i. if we have found the best node in T , the best node Corollary 3: Consider a two-level tree T whose root can still vary if we know that subtree T is a bad tree.node is attached to |N | − 1 leaf nodes. If T is a bad tree Therefore, we cannot determine the optimal solution toand every node i has ci = c for some constant c, the best a problem by first solving for the optimal solutions tonode is the one with the maximum conditional failure the subproblems, and this makes the best node inferenceprobability. difficult.5.3 Why Selecting the Best Node from CandidateNodes is Difficult? 5.4 Evaluation of Candidate-based HeuristicsWhile we have proved that one of the candidate nodes Given the difficulty of finding the best node among ais the best node, we have not yet identified which set of candidate nodes, we evaluate the performance ofcandidate node should be selected as the best node in three candidate-based heuristics that approximate thethe general case. We show that deciding the best node best node selection of the inference algorithm. Theseis non-trivial using several counter-examples that violate heuristics are: (1) Cand-Prob, which selects the candidateour intuition. In the following discussion, the optimality node with the highest conditional failure probabilityresults are verified by the brute-force inference algorithm given a bad tree, (2) Cand-Cost, which selects the can-in Algorithm 1. didate node with the least checking cost, and (3) Cand- Counter-example 1: Given a bad tree, the best node is not Pot, which selects the candidate node with the highestthe one with the highest potential. Figure 5 shows a bad potential. Our evaluation setting is the same as that intree T in which the candidate nodes are nodes 3 and 4. Section 4, i.e., we determine the proportion of instancesAlthough node 4 has the highest potential, node 3 is the (out of 200) in which a candidate-based heuristic selects
  7. 7. 7a best node for a given two-path bad tree of size |N | Algorithm 2 Pa-Naive-Probunder different distributions of pi and ci . Input: a bad tree T = (N, {pi }, {ci }) Figure 8 plots the results. Comparing the results to 1: for all node i ∈ N in reverse breadth-first-search order dothose in Figure 4, we find that all candidate-based heuris- 2: if node i is a leaf node then 3: s(i) = pi ; mark node i /* s(i) denotes the score of itics are more likely to make the best choice than the naive */ones. In particular, Cand-Pot outperforms all naive and 4: else if node i is a non-leaf node thencandidate-based heuristics. In most cases, the candidate 5: if pi > j∈C s(j) then /* Ci = set of child nodes of inode with the highest potential is actually the best node i */in at least 95% of time. 6: s(i) = pi ; mark node i 7: else 8: s(i) = j∈C s(j) i5.5 Complexity of Finding the Node with Maximum 9: Ipnp = φ; Q = φ; enqueue root node of T to QPotential 10: while Q = φ do 11: dequeue node i from QSince the computation of the potential of a node needs 12: if node i is marked thenΘ(|N |2 ) time (see Section 5.1), the complexity of Cand- 13: Ipnp = Ipnp ∪ {i}Pot, which searches among all nodes for the one with 14: elsethe highest potential, is Θ(|N |3 ). This may lead to the 15: enqueue all child nodes of i to Qscalability issue when the network grows. However, in 16: return Ipnpmost cases, the bad tree is only a subset of the entirerouting tree. Hence, the bad tree that is parsed by Cand-Pot is likely to be within a manageable size in general. We also implement a candidate-based heuristic for inferring multiple nodes termed Pa-Cand, which returns6 H EURISTICS FOR THE I NFERENCE A LGO - the minimum-sized subset Ipc of candidate nodes that cover all bad paths in a bad tree. The algorithm ofRITHM finding Ipc is shown in Algorithm 3, whose complexityWhile the brute-force inference algorithm (see Algo- is Θ(|N |3 ) due to the search of candidate nodes.rithm 1) returns the best node, its factorial complex-ity prohibits its use in large-scale networks. Thus, we Algorithm 3 Pa-Candpropose three different groups of efficient heuristics Input: a bad tree T = (N, {pi }, {ci })for the inference algorithm that are suitable for large- 1: determine the set of candidate nodes in Tscale networks. Each heuristic belongs to one of the 2: Ipc = φ; Q = φ; enqueue root node of T to Qtwo classes: (i) naive heuristics, which consider only the 3: while Q = φ domost likely faulty nodes based on the conditional failure 4: dequeue node i from Qprobability distribution, and (ii) candidate-based heuristics, 5: if node i is a candidate node then 6: Ipc = Ipc ∪ {i}which consider the candidate nodes based on both the 7: elseconditional failure probability as well as the checking 8: enqueue all child nodes of i to Qcost distributions. 9: return Ipc6.1 Single Node Inference for a Single TreeWe consider two heuristics Naive-Prob (see Section 4) 6.3 Multiple Node Inference for Multiple Treesand Cand-Pot (see Section 5), which respectively return A limitation of the above heuristics is that a single logicalthe node that is most likely faulty given a bad tree and tree only includes a subset of physical components inthe candidate node with the highest potential. the entire network. In order to cover more physical components, it is important to conduct the inference6.2 Multiple Node Inference for a Single Tree algorithm on multiple bad trees, which collectively formInstead of sequentially checking one node at a time, we a directed acyclic graph. In general, finding a minimumcan also infer and check multiple nodes in parallel so as subset of nodes that cover a general set of paths can beto reduce the time needed to repair all bad nodes. viewed as a set-cover problem, which is NP-hard [11]. We first extend Naive-Prob to Pa-Naive-Prob (where Therefore, we consider two multi-tree-based heuristics“Pa” stands for “parallel”), which returns the most likely termed Mt-Pa-Naive-Prob and Mt-Pa-Cand (where “Mt”faulty subset Ipnp of nodes that cover all the bad paths stands for “multiple trees”), which respectively call Pa-in a given bad tree T , i.e., Ipnp = arg maxI Pr(XI |T ), Naive-Prob and Pa-Cand independently on each of thewhere T is the event that T is a bad tree, and XI is the bad trees and return the union of the results.event that the subset I of nodes are all bad and cover all Note that in typical overlay networks, an overlaybad paths in T . Since Pr(T |XI ) is one, it is equivalent to node may possess multiple logical identities, or mayevaluate Ipnp = arg maxI Pr(XI ) = arg maxI i∈I pi . We be shared by multiple overlay networks. Thus, multiplecan determine Ipnp using Algorithm 2, whose complexity failed points may actually correspond to the failure of ais Θ(|N |). single physical node, and this implies that node failures
  8. 8. 8 Proportion of correct instances 1 1 1 Proportion of correct instances Proportion of correct instances 0.95 0.98 Cand-Prob 0.95 Cand-Prob 0.96 0.9 Cand-Cost Cand-Cost Cand-Prob Cand-Pot 0.9 Cand-Pot 0.94 Cand-Cost 0.85 0.92 Cand-Pot 0.8 0.85 0.9 0.75 0.88 0.8 0.86 0.7 0.75 0.84 0.65 0.82 0.6 0.7 0.8 4 6 8 10 12 14 16 4 6 8 10 12 14 16 4 6 8 10 12 14 16 |N| = Number of nodes |N| = Number of nodes |N| = Number of nodes (a) pi ∼ U (0, 1), ci = 1 (b) pi = 0.1, ci ∼ U (0, 1) (c) pi ∼ U (0, 1), ci ∼ U (0, 1)Fig. 8. Proportions of correct instances in which Cand-Prob, Cand-Cost, and Cand-Pot, start with a best node.are correlated. Our multi-tree-based heuristics provide a choose from the shortest-path tree a subset of K pathspreliminary attempt to illustrate how we can correlate to be included in the logical tree. Our experiments willnode failures in multiple tree-based topologies. study the results with different values of K. For the heuristics that have multiple logical trees, we simply7 E XPERIMENTS IN L ARGE -S CALE repeat the logical tree construction by selecting a random subset of routers to be the roots of the logical trees.N ETWORKS Finding the efficient set of trees that cover all networkIn this section, we evaluate via simulation the naive components has been considered in [1], [6] and is beyondheuristics (i.e., Naive-Prob, Pa-Naive-Prob, and Mt-Pa- the scope of this paper.Naive-Prob) and the candidate-based heuristics (i.e., Given a single or multiple logical trees, we simulateCand-Pot, Pa-Cand, and Mt-Pa-Cand) described in Sec- the faults by letting each overlay node fail independentlytion 6 in correcting faulty nodes in large-scale networks. with its assigned failure probability. To ensure that the inference algorithm is actually executed, we require that7.1 Experimental Setup at least one bad node resides in a logical tree. Given a logical tree, we first retain only the bad paths and formWe use the BRITE generator [27] to generate random a bad tree. We then check the node (or nodes) returnedoverlay networks based on the router Waxman model. from the inference algorithm that is implemented usingIn our prior conference version [23], we also consider different heuristics. For any located bad node, we “re-the router-level Barb´ si-Albert model [5]. While unstruc- a pair” it by switching it to a good node. We repeatedlytured overlay networks (e.g., Gnutella [9]) generally update the set of bad paths and execute the inferencefollow the Barb´ si-Albert model, structured networks a algorithm until all bad nodes are repaired.(e.g., CAN [30] and Chord [33]) are similar to random Due to the large network size, we cannot apply thenetworks in the sense that nodes have similar degrees brute-force inference algorithm in Algorithm 1 to de-[12]. As shown in our simulation results, both graph termine the proportion of correct identification as inmodels lead to similar observations. Thus, we focus on Sections 4 and 5. Therefore, our experiments mainlythe Waxman model in this paper. focus on two metrics to evaluate our heuristics: Each network topology that we generate contains 200 • Total checking cost, the sum of costs of checking thenodes and 800 links. We treat the nodes and links as node (or nodes) returned by the inference algorithmoverlay nodes and overlay links, respectively. We then until all bad nodes are repaired.construct 10 topologies with random node placement • Number of rounds, the number of times the infer-and random link weights. Each topology is further ence algorithm is executed until all bad nodes areassigned 20 sets of parameters using different seeds, repaired.leading to a total of 200 instances. Our simulation focuseson monitoring overlay nodes so as to demonstrate theimportance of first checking candidate nodes. In each 7.2 Experimental Resultsof the instances, each overlay node is assigned a failure In the following results, each data point is averagedprobability pi and a checking cost ci . Unless otherwise over 200 instances and is plotted with its 95% confidencespecified, pi and ci follows one of the three distributions: interval.(a) pi ∼ U (0, 0.2) and ci = 1, (b) pi = 0.1 and ci ∼ U (0, 1), Experiment 1 (Comparison of single-tree-basedas well as (c) pi ∼ U (0, 0.2) and ci ∼ U (0, 1), where heuristics). We first consider the performance of theU (u1 , u2 ), as defined in Section 4, denotes the uniform heuristics when we only monitor a single logical tree.distribution between u1 and u2 . Figure 9 shows the performance of Naive-Prob, Cand- To construct a logical tree, we first create a shortest- Pot, Pa-Naive-Prob, and Pa-Cand versus K, the num-path tree rooted at a randomly selected router based ber of paths that are included in a logical tree. As Kon the given overlay link weights, and then randomly increases, more nodes are being monitored, and hence
  9. 9. 9 10 6 6 Naive-Prob Naive-Prob Naive-Prob Cand-Pot Cand-Pot Cand-Pot 8 Pa-Naive-Prob 5 Pa-Naive-Prob 5 Pa-Naive-Prob Total checking cost Total checking cost Total checking cost Pa-Cand Pa-Cand Pa-Cand 4 4 6 3 3 4 2 2 2 1 1 0 0 0 0 5 10 15 20 25 30 35 0 5 10 15 20 25 30 35 0 5 10 15 20 25 30 35 K = number of paths K = number of paths K = number of paths (a) pi ∼ U (0, 0.2), ci = 1 (b) pi = 0.1, ci ∼ U (0, 1) (c) pi ∼ U (0, 0.2), ci ∼ U (0, 1) 12 12 12 Naive-Prob Naive-Prob Naive-Prob Cand-Pot Cand-Pot Cand-Pot 10 Pa-Naive-Prob 10 Pa-Naive-Prob 10 Pa-Naive-Prob Number of rounds Number of rounds Number of rounds Pa-Cand Pa-Cand Pa-Cand 8 8 8 6 6 6 4 4 4 2 2 2 0 0 0 0 5 10 15 20 25 30 35 0 5 10 15 20 25 30 35 0 5 10 15 20 25 30 35 K = number of paths K = number of paths K = number of paths (d) pi ∼ U (0, 0.2), ci = 1 (e) pi = 0.1, ci ∼ U (0, 1) (f) pi ∼ U (0, 0.2), ci ∼ U (0, 1)Fig. 9. Experiment 1: Comparison of Naive-Prob, Cand-Pot, Pa-Naive-Prob, and Pa-Cand in terms of the total checkingcost (see (a) to (c)) and the number of rounds (see (d) to (f)), where K = 5, 10, 15, 20, 25, and 30. For clarity ofpresentation, data points are shifted slightly on x-axis. Naive-Prob Naive-Prob 20 Cand-Pot 20 Cand-Pot Pa-Naive-Prob Pa-Naive-Prob Total checking cost Total checking cost Pa-Cand Pa-Cand 15 15 10 10 5 5 0 0 0 1 2 3 4 5 6 7 8 0 1 2 3 4 5 6 7 8 b b (a) K = 5 (b) K = 20Fig. 10. Experiment 2: Comparison of single-based heuristics in terms of the total checking cost under a contrivedchecking cost distribution.the total checking costs of all heuristics increase as outage is generally huge [15]. This demonstrates thewell. We note that the distributions of pi and ci can competence of the candidate-based heuristics when theinfluence the difference between the total checking costs physical components have varying checking costs (seeof the naive heuristics (i.e., Naive-Prob and Pa-Naive- Section 3).Prob) and candidate-based heuristics (i.e., Cand-Pot and We also note that Naive-Prob and Cand-Pot, the se-Pa-Cand). When pi is varied and ci is fixed (see Fig- quential heuristics that infer one node at a time, takeure 9(a)), both naive and candidate-based heuristics have more rounds to execute the inference algorithm as Knearly identical total checking cost. This also confirms grows. On the other hand, Pa-Naive-Prob and Pa-Cand,the effectiveness of conventional fault localization ap- which return multiple nodes at a time, significantlyproaches, whose focus is to identify the most likely faults reduce the number of rounds, while only slightly increas-without considering the variance of the cost component. ing the total checking cost (by less than 1% in general)However, the candidate-based heuristics reduce the total as compared to their respective sequential heuristics.checking costs of the naive heuristics when ci is var- Furthermore, both Pa-Naive-Prob and Pa-Cand take aied, for instance, by 8-18% when pi is fixed and ci is similar number of rounds to execute the inference algo-varied (see Figure 9(b)), and by 7-12% when both pi rithm, with less than one round of difference.and ci are varied (see Figure 9(c)). It should be notedthat such cost reduction can imply significant revenue Experiment 2 (Comparison of single-tree-basedsaving, given that the revenue loss due to a network heuristics under a contrived checking cost distribu- tion). We point out that some distributions of checking
  10. 10. 10costs can lead to a large performance improvement of 18%). Figure 12(b) shows the performance of the multi-the candidate-based heuristics over the naive heuristics. tree-based heuristics Pa-Naive-Prob and Pa-Cand (as inIntuitively, if checking costs are fixed for all nodes in a Experiment 2, where we also set K= 20 here), and Pa-bad tree, then both naive and candidate-based heuristics Cand still gives smaller total checking costs than Pa-prefer to first check the nodes near the root node, as Naive-Prob for different estimates of failure probabilities.such nodes have higher conditional failure probabilities Summary: We show that the candidate-based heuris-On the other hand, if the nodes around the leaf positions tics can decrease the total checking cost of the naivehave small checking costs, then they may be preferred heuristics that focus on most likely faults, especiallyto be first checked by the candidate-based heuristics when the overlay nodes have varying checking costs.based on the potential calculation, but not by the naive Also, multiple node inference can speed up the faultheuristics, which do not consider checking costs. We correction process, with only a slight increase in theexamine this observation in this experiment using a total checking cost as compared to single node inference.contrived checking cost distribution. Furthermore, we observe similar improvement of the For a given logical tree T that we monitor, we first candidate-based heuristics over the naive heuristics evenidentify the depth di of each node i, i.e., the path length though the estimates of failure probabilities are inaccu-from root to node i (we let dr = 0 for root r). Also, we let rate.dmax be the maximum depth of all nodes in T . Then welet the checking cost ci of node i be ci = (1.5−di /dmax )b ,where b is some exponent that we vary in our evaluation.Note that as b increases, nodes closer to the root node 8 R ELATED W ORKwill have increasingly higher checking costs than thosenear the leaf nodes. Here, we set pi ∼ U (0, 0.2), and Our end-to-end inference solution is inspired by [7], [10],consider the cases where K = 5 and K = 20 in a logical [16], [26], which also use end-to-end measurements totree. infer link statistics or the underlying network topology. Figure 10 shows the total checking costs of different We extend this end-to-end approach to the applicationheuristics versus b. The improvement of the candidate- of network fault management.based heuristics over the naive heuristics increases with Fault diagnosis is an important topic in network man-b. Specifically, when b ≥ 4, Cand-Pot (resp. Pa-Cand) agement (see survey in [31]). For example, the codebookreduces the total checking cost of Naive-Prob (resp. Pa- approach [35] and the Bayesian approach [21] considerNaive-Prob) by more than 30% when K = 5 and by more a deterministic setting where network faults are equallythan 20% when K = 20. likely to occur. To address probabilistic faults, [18], [19], Experiment 3 (Comparison of multi-tree-based [32] seek to localize the most probable subset of faultsheuristics). We now evaluate the performance of the given the observed symptoms. Katzela and Schwartzheuristics when we monitor multiple logical trees. Fig- [19] show that such a problem is generally NP-hard.ure 11 illustrates the performance of Mt-Pa-Naive-Prob In view of this, Steinder and Sethi [32] formulate theand Mt-Pa-Cand with respect to the number of logical problem based on bipartite belief networks and proposetrees, where we fix K = 20 in each logical tree. Similar to efficient techniques on localizing end-to-end multi-layerExperiment 1, Mt-Pa-Cand can reduce the total checking failures. Kandula et al. [18] propose to infer the mostcost of Mt-Pa-Naive-Prob, for example, by 10-16% when likely faulty IP links with the assumption of significantlypi is fixed and ci is varied (see Figure 10(b)), and by small failure probabilities. All these schemes focus on7-11% when both pi and ci are varied. Also, both Mt-Pa- localizing faults. In this paper, in addition to probabilisticNaive-Prob and Mt-Pa-Cand call the inference algorithm fault localization, we take the subsequent step of repair-with about the same number of rounds. ing faults as well. We show that checking first the most Experiment 4 (Inaccuracies in failure probability es- likely faults does not give optimal results in general.timation). In general, it is difficult to accurately estimate References [25], [36] consider various dimensions ofthe failure probabilities of all overlay nodes. In this performance of failure detection in overlay networks,experiment, we study how our heuristics are affected in which nodes periodically probe their neighbors tosubject to the inaccurate failure probability estimation. determine failures. Our work considers the case whereWe let pi and pi be the actual and estimated failure nodes are externally managed and they need to beprobability for node i, respectively. We let pi ∼ U (0, 0.2). manually checked in order to determine failures.We then consider three cases (i) pi = pi (i.e., no error), (ii) Fault correction has also been studied extensively inpi = 0.1, and (iii) pi = 0.2. We assume that ci ∼ U (0, 1). reliability engineering (see survey in [24]). The work Figure 12(a) shows the performance of the single-tree- that is closest to ours is in [17], [34], whose objectivebased heuristics Naive-Prob and Cand-Pot versus K (as is also to minimize the cost of correcting all faulty units.in Experiment 1). As expected, the inaccurate estimation However, the optimality assumes a series system, whichof failuire probabilities increases the total checking costs is equivalent to a single path in a network setting. Inof all heuristics. However, Cand-Pot still reduces the contrast, we consider a routing tree whose paths aretotal checking costs of Naive-Prob in all cases (by 14- shared and dependent.