Bayesian network based software reliability prediction
1.
The Journal of Systems and Software 77 (2005) 103–112 www.elsevier.com/locate/jss Bayesian network based software reliability prediction with an operational proﬁle q Cheng-Gang Bai * Department of Automatic Control, Beihang University (Beijing University of Aeronautics and Astronautics), Beijing 100083, China Received 29 June 2004; received in revised form 11 November 2004; accepted 26 November 2004 Available online 24 December 2004Abstract This paper uses a Bayesian network to model software reliability prediction with an operational proﬁle. Due to the complexity ofsoftware products and development processes, software reliability models need to possess the ability to deal with multiple param-eters. A Bayesian network exhibits a strong ability to adapt to problems involving complex variant factors. A special kind of Bayes-ian network named a Markov Bayesian network has been applied successfully into modeling software reliability prediction.However, the existing research did not pay enough attention to the fact that the failure characteristics of many software systemsoften depend on the speciﬁc operation performed. In this paper, an extended Markov Bayesian network is developed to model soft-ware reliability prediction with an operational proﬁle. The extended Markov Bayesian network proposed in the paper is focused ondiscrete-time failure data. Methods to solve the network are proposed, and an example is used to illustrate the utilization of themodel.Ó 2004 Elsevier Inc. All rights reserved.Keywords: Software reliability; Software failure rate; Operational proﬁle; Bayesian network; Markov Bayesian network; Discrete-time failure data;Software run1. Introduction such as the Jelinski–Moranda model (Jelinski and Mor- anda, 1972) and the Goel–Okumoto NHPP model (Goel Research on how to predict software reliability accu- and Okumoto, 1979). This kind of model suggests that arately is of great practical importance. Many software stochastic process, especially a non-homogeneous Pois-reliability prediction models have been developed and son process (NHPP) (Boland and Singh, 2002, 2003),described in the literature (Goel, 1985; Xie, 1991, can describe the failure process of software.1993), among which many classic probability models However, many of these models make unrealisticare widely discussed. For some recent references, see, restrictive assumptions to ensure tractability and solv-e.g. (Cai, 1998b; Sahinoglu et al., 2001; Malaiya et al., ability (Bishop and Pullen, 1988; Lyu, 1996), such as2002; Zheng, 2002; Ozekici and Soyer, 2003; Reussner removing defects completely (no new defects introducedet al., 2003). The traditional software reliability predic- when removing the one detected), or data completenesstion models mainly belong to stochastic process models, (no missing data). These restrictive assumptions have limited the development and applications of these probability models in recent years, and as a result q researchers have begun to explore other approaches. Supported by the National Natural Science Foundation of China(Grant No. 60473067 and Grant No. 60233020). The assumptions have been relaxed by making no distri- * Tel.: +861082315087; fax: +861082317328. bution assumptions on the failure process (Pfeﬀerman E-mail address: bcg@buaa.edu.cn and Cemuschi-Frias, 2002), but this advantage is0164-1212/$ - see front matter Ó 2004 Elsevier Inc. All rights reserved.doi:10.1016/j.jss.2004.11.034
2.
104 C.-G. Bai / The Journal of Systems and Software 77 (2005) 103–112 Nomenclature ð1Þ {D, S, P} denote Bayesian network Hj the failure rate of Oj at the beginning of test- D the set of variables (or nodes) of Bayesian ing process j = 1, 2, . . ., m ðiÞ network Hj the failure rate of Oj after the (i À 1)th run, S the set of conditional probability distribu- i = 2, . . . , n + 1, j = 1, 2, . . ., m tion(CPD) of Bayesian network Oj the j th operation, j = 1,2, . . ., m P the set of marginal probability distributions Ai the ith run, Ai = j if the jth operation Oj is of Bayesian network carried out at ith run, i = 1,2, . . ., n p(X) if X is a continuous variable, it denotes the Qi the operational proﬁle for the ith run, probability density function of X; if X is a i = 1,2, . . ., n discrete variable, it denotes the probability Qn+1 the operational proﬁle for the possible next distribution of X run after n runs MBN Markov Bayesian network Q operational proﬁle if it is invariant Xi(1 6 i 6 n) the remaining defects number after the Zi the result of the i th run, Zi = 1 if a failure ith failure occurs at ith run, and otherwise Zi = 0, i = Yi(1 6 i 6 n) the time between the (i À 1)th and ith 1, 2, . . ., n failure d denotes the decrement of failure rate after a EMBN extended Markov Bayesian network failure occurs and the failure-causing defects n number of software testing runs are removed m number of operations Beta(Æ ja, b) denotes Beta distribution with para- f failure order meters a,b gf the number of runs from the fth failure to Dir(Æ jq1, . . . , qm) denotes Dirichlet distribution with (f + 1)th failure parameters q1, . . ., qm R the software reliability after n runscompensated by the added noise in the software reliabil- atively new technique, the Bayesian network, can be ap-ity prediction results. Non-probability models, e.g. neu- plied to model and solve such problems. Bayesianral networks, have been applied (Karunanithi and networks can describe many factors of the system simul-Whitley, 1992; Khoshgoftaar and Szabo, 1996; Sitte, taneously and also can express the interrelations be-1999; Cai et al., 2001). Recent research has shown some tween them eﬃciently. It combines precise probabilityadvantage of neural networks in predictive capabilities distributions and prior expertise information in a wayover the traditional NHPP model (Ho et al., 2003). that is more suitable to solving problems with manyHowever, there are still some problems with this ap- uncertainties. Due to these desirable properties, Bayes-proach. First, due to the randomness of the network ian networks have been increasingly applied in manystructure, it is diﬃcult to express the interrelationships ﬁelds (Bickmore, 1994; Madigan et al., 1996; Lewisbetween the input variables. Second, it is necessary to and Ransing, 1997; Castillo et al., 1999; Zhu, 2003),ensure the integrity of the neural network, which means although this approach has found applications in reli-that if the data is absent in some input nodes, it would ability only in the last seven years (Sigurdsson et al.,be diﬃcult for the neural network to process eﬀectively. 2001). A Bayesian framework combining diverse sources Amongst many unreasonable assumptions in most of evidence was developed to assess system dependabil-traditional models, one of the most serious limitations ity (Neil et al., 1996; Bouissou et al., 1999). It has beenis the reliance on past failure data, alone, to predict soft- used to predict software reliability in the early phases ofware reliability. The future failure time not only relies on the development by incorporating information ahead ofthe past failure data, but also relies on other variables testing (Smidts et al., 1998) and to develop a causalsuch as how the software is used. model for software defect rates prediction (Fenton and Models of such complex problems as software reli- Neil, 1999). It has also been combined with fault treesability prediction in large software systems should incor- for reliability analysis (Pai, 2001), and used for the cer-porate prior information and available expert tiﬁcation of COTS (Commercial Oﬀ-The-Shelf) softwareexperience. The Bayesian method provides a proper tool reliability (Yu and Johnson, 2003). In our previous workto deal with this problem (Kuo and Yang, 1995,1996; (Bai et al., 2005), a software failure prediction modelKuo and Lee, 1997; Basu and Ebrahimi, 2003). How- based on Markov Bayesian networks was developed.ever, it is diﬃcult for traditional Bayesian models to deal However, an undesirable restriction in almost all ofwith problems involving complex variant factors. A rel- the software reliability models in the literature is that
3.
C.-G. Bai / The Journal of Systems and Software 77 (2005) 103–112 105the parameters of the software failure process are inde- (3) P is a set of marginal probability distributions.pendent of the operation that the software performs P = {p(D)jD 2 D} stands for the probability distri-(Ozekici and Soyer, 2003). Musa (1993, 1996) describes bution of variable D.how operational proﬁles can be built and states thatthere is substantial beneﬁt to be gained by applying In the Bayesian network, variables are used to ex-them. An operational proﬁle describes how users em- press the events or objects. The problem could be mod-ploy a system (Musa, 1993). There has been some work eled with the behavior of these variables. In general, weon the topic of software reliability with an operational ﬁrst calculate (or determine from expert experience) theproﬁle. Cai (1998a) developed various methods to model probability distribution of each variable and the condi-operational proﬁles. Ozekici and Soyer (2003) provide a tional probability distribution between them. Then fromstochastic and statistical framework to model software these distributions we can obtain the joint distributionsreliability in the presence of an operational proﬁle. Their of these variables. Finally, some deductions can bemethod is based on the assumption that software reli- developed for some variables of interest using someability behavior can be measured in terms of continuous other known variables.time. However, many systems depart from this assump- A Bayesian network is generally based on Bayesiantion (Cai, 2000). For example, the reliability behavior of statistical theory. Compared with other traditional dataa bank transaction processing software system should be analysis methods, a Bayesian network has the followingmeasured in terms of how many transactions are suc- properties:cessful, rather than how long the software system oper-ates without failure. Cai (2003) discussed how to assess (1) A Bayesian network can be connected to all thesoftware reliability with an operational proﬁle during variables of interest, so it is expected to increasethe software validation phase. In the validation phase, the precision of the analysis. For example, in ait is assumed that no detected defects are removed. problem with two input and one output variables, Diﬀerent from these earlier works, this paper de- the two input variables might be correlated. If bothscribes an extended Markov Bayesian network that of the input variables can be observed, most of thecan deal with software reliability prediction with an models will be able to predict the output. However,operational proﬁle. This paper is organized as follows. if one of the inputs cannot be observed, then fewSection 2 presents the background of the paper. In Sec- models can provide a precise prediction, and thetion 3, we develop an extended Markov Bayesian Net- correlation between these two variables is not used.work model for software reliability prediction. In On the contrary, it is natural for a Bayesian networkSection 4, we discuss the techniques for solving the to measure and use this kind of correlation.model under various assumptions. In Section 5, a case (2) A Bayesian network can model the causal relation-study is given with real software failure data, and ship between variables. Thus, the interdependencesatisfactory results are obtained. of the variables could be studied in order to improve prediction. For example, one market analyst may want to know whether it is worthwhile to increase2. Background the intensity of some advertising campaign in order to gain more sales of a certain product. To solve this2.1. Bayesian network problem, the analyst needs to know whether the advertisement is an inﬂuencing factor, and the The deﬁnition of a Bayesian network can be found in extent of its inﬂuence. Bayesian networks can helpmany versions, but the basic form (Pearl, 1986) is stated to solve problems like this.as follows: a Bayesian network is a directed probabilitygraph, connecting the relative variables with arcs, and A Bayesian network with Bayesian statistical meth-this kind of connection expresses the conditional depen- ods can combine the relative knowledge in the problemdence between the variables. The formal deﬁnition domain and the available data eﬀectively. No one withfollows. practical experiences can deny the importance of the A Bayesian network is deﬁned as the set of {D, S, P}, prior information or the problem domain knowledge,where especially in circumstances where little or no data are available or when data collection is costly.(1) D is a set of variables (or nodes).(2) S is a set of conditional probability distributions 2.2. Bayesian network based software reliability (CPD). prediction model S = {p(DjParents(D))jD 2 D}, Parents(D) & D stands for all the parent nodes for D, p(DjParents(D)) is the Let Xi (1 6 i 6 n) denote the number of remaining conditional distribution of variable D. defects after the ith failure, and Yi (1 6 i 6 n) the time
4.
106 C.-G. Bai / The Journal of Systems and Software 77 (2005) 103–112 X1 X2 X3 Xn 3. Bayesian network model with an operational proﬁle … 3.1. Extended Markov Bayesian network Assume that there are m operations and n runs. Let … Oj = j denote the jth operation of them operations. Let Ai (i = 1, . . . , n) denote the ith run. Ai = j (i = 1, . . . , n) if the jth operation Oj is carried out at ith run. Let Qi Y1 Y2 Y2 Yn (i = 1, . . . , n) denote operational proﬁle at ith run Fig. 1. A Markov Bayesian network (MBN). n o ðiÞ ðiÞ Qi ¼ qj j qj ¼ Pr½Ai ¼ j ð1 6 j 6 mÞ ði ¼ 1; . . . ; nÞ:between the (i À 1)th and ith failure. During the defects Letremoval process, it is possible to both remove and intro- duce defects, and hence Xi+1 is dependent on Xi. The 1 if a failure occurs at the i th run Zi ¼ ði ¼ 1; . . . ; nÞtime interval between software failures is inﬂuenced 0 if no failure occurs at the ith runby the number of remaining defects. Assuming thatp(xi+1jxi) and p(yijxi) are independent of time, then a ð1Þ Let H j ðj ¼ 1; . . . ; mÞ denote the failure rate of Oj at theMarkov Bayesian network can be used to model this ðiÞ beginning of the testing. Let H j ði ¼ 2; . . . ; n þ 1;problem (Jordan et al., 1999). The network graph is j ¼ 1; . . . ; mÞ denote the failure rate of Oj after thegiven in Fig. 1 (Bai et al., 2005). (i À 1)th run. Comparing with our previous work (Bai et al., 2005), we use failure rate in this study rather than2.3. Operational proﬁle the number of remaining defects. The main reasons are (1) it is easier to get the relationship between failure rate Musa (1993) said Ôa proﬁle is simply a set of disjoint and testing results than that between the number of(only one can occur at a time) alternatives with the prob- remaining defects and testing results; (2) it is easier toability that each will occur.Õ An operational proﬁle sim- predict software reliability based on the failure rateply consists of the set of all operations that a system is rather than on remaining defects. So, failure rate isdesigned to perform and their probabilities of occur- much more interesting.rence. It provides a quantitative characterization of Let R denote the software reliability after the total nhow the system will be used in the ﬁeld, making it an runs. Qn+1 denotes the operational proﬁle after the totalessential ingredient of software reliability engineering. n runs.Technically speaking, one can think of an operational Fig. 2 shows the extended Markov Bayesian networkproﬁle as a generic random variable that indicates the (EMBN).operations that will be performed by the system (Ozekiciand Soyer, 2003). 3.2. Determining the distributions2.4. Software run In order to use the EMBN model to predict software reliability, the following distributions are essential: The Poisson model (Bai et al., 2005) is based on the ð1Þ ð1Þassumption that software reliability behavior can be (1) The initial distributions of H j . pðH j Þ ðj ¼ 1;measured in terms of calendar time, clock time or . . . ; mÞ. These could be estimated using softwareCPU execution time. Although this assumption is metrics from the development of the software prod-appropriate for a wide scope of systems, there are many uct, or be given by experts.systems, which depart from this assumption. For exam- (2) The initial distributions of Qi (i = 1, . . . , n + 1). Qi isple, the reliability behavior of a bank transaction pro- the operational proﬁle at the ith run. It can becessing software system should be measured in terms assumed that the initial distributions of Qi can beof how many transactions are successful, rather than gotten from historical data.of how long the software system operates without failure (3) The conditional distribution of Zi (i = 1, . . . , n)(Cai, 2000). h i ðiÞ ðiÞ ðjÞ Cai (2000) proposed the conception of ‘‘a run’’ to de- Pr Z i ¼ 1 j Ai ; H 1 ; . . . ; H ðiÞ ¼ H j Á qi mscribe discrete time. A run is minimum execution unit of h isoftware. Any software execution process can be divided ðiÞ ðiÞ ðjÞ Pr Z i ¼ 0 j Ai ; H 1 ; . . . ; H ðiÞ ¼ 1 À H j Á qi minto a series of runs.
5.
C.-G. Bai / The Journal of Systems and Software 77 (2005) 103–112 107 H1 (1) H1 (2) H1 (n) H1(n+1) 3.3. Software reliability prediction … Although Eq. (1) is useful in evaluating the prediction probabilities, the densities in Eq. (1) can be diﬃcult to compute directly. Here, we propose to use the Gibbs sampling approach to approximate Eq. (1). According … … … … … to the Gibbs sampling theory (Gelfand and Smith, (2) 1985), the key point is to sample from the conditional Hm(1) Hm Hm(n) Hm (n+1) distributions of every variable under condition that all … other variables are known. ðkÞ (1) H l ( ðkÞ ) … ðiÞ H l j R; H j ði ¼ 1;...;n þ 1; j ¼ 1;...;m; ði;jÞ 6¼ ðk;lÞÞ p Z1 Z2 Zn Ai ði ¼ 1;...;nÞ; Z i ði ¼ 1;...;nÞ; Qi ði ¼ 1;...;n þ 1Þ R ( ðiÞ ) R; H j ði ¼ 1;...;n þ 1; j ¼ 1;...;mÞ; Ai ði ¼ 1;...;nÞ /p A1 A2 An Z i ði ¼ 1;...;nÞ; Qi ði ¼ 1;...;n þ 1Þ … Y nþ1 Y n ¼ pfR j Qnþ1 ; H nþ1 ðj ¼ 1;...;mÞg Á j pfQi g Á pfAi j Qi g i¼1 i¼1 Y m ð1Þ YY nþ1 m ðiÞ ðiÀ1Þ Á pfH j g Á pfH j j H j ; AiÀ1 ; Z iÀ1 g … j¼1 iÀ2 j¼1 Q1 Q2 Qn Qn+1 Y n ðiÞ Á pfZ i j H j ðj ¼ 1;...;mÞ; Ai g Fig. 2. Extended Markov Bayesian network. i¼1 if 1 < k < n + 1(4) The conditional distribution of Ai (i = 1, . . . , n) ( ðkÞ ðiÞ ) ðiÞ ðiÞ H l j R;H j ði ¼ 1;...;n þ 1; j ¼ 1;...;m; ði;jÞ 6¼ ðk;lÞÞ pfAi ¼ j j Qi ¼ fqj ; j ¼ 1; . . . ; mgg ¼ qj p ðiÞ Ai ði ¼ 1;...;nÞ; Z i ði ¼ 1;...;nÞ; Qi ði ¼ 1;...;n þ 1Þ(5) The conditional distribution of H j ði ¼ 2; . . . ; n þ 1Þ. ðkÞ ðkÀ1Þ ðkÞ Cai (2000) proposed ﬁve models to describe the / pfH l j H l ;AkÀ1 ;Z kÀ1 g Á pfZ k j H l ;Ak g changing trend about hazard rate function (the prob- ðkþ1Þ ðkÞ ability that the software fails). Therefore, the 5 mod- Á pfH l j H l ;Ak ;Z k g els can be applied into this step. For example, when ðiþ1Þ ðiÞ if k = 1 Ai = j 0 , it is feasible to let H j0 ¼ H j0 À dZ i ; ðiþ1Þ ðiÞ 0 ( ðkÞ ðiÞ ) Hj ¼ H j ; j 6¼ j . H l j H j ði ¼ 1; . . . ; n; j ¼ 1; . . . ; m; ði; jÞ 6¼ ðk; lÞÞ(6) The conditional distribution of R p X ðnþ1Þ ðnþ1Þ m Ai ði ¼ 1; . . . ; nÞ; Z i ði ¼ 1; . . . ; nÞ; Qi ði ¼ 1; . . . ; nÞ R¼1À H j qj ð1Þ ð1Þ ð2Þ ð1Þ j¼1 / pfH l g Á pfZ 1 j H l ; A1 g Á pfH l j H l ; A1 ; Z 1 g As in standard Bayesian network, the joint probabil- if k = n + 1 ðiÞ ity density of R, H j ði ¼ 1; . . . ; n þ 1; j ¼ 1; . . . ; mÞ, Ai ( ðkÞ ðiÞ ) H l j R;H j ði ¼ 1; . . . ; n þ 1; j ¼ 1; . . . ; m; ði; jÞ 6¼ ðk; lÞÞ (i = 1, . . . , n), Zi (i = 1, . . . , n), Qi (i = 1, . . . , n + 1) is p ( ) Ai ði ¼ 1; .. . ; nÞ; Z i ði ¼ 1;. . . ; nÞ; Qi ði ¼ 1;. . . ; n þ 1Þ ðiÞ R; H j ði ¼ 1; . . . ; n þ 1; j ¼ 1; . . . ; mÞ; Ai ði ¼ 1; . . . ; nÞ; p Z i ði ¼ 1; . . . ; nÞ; Qi ði ¼ 1; . . . ; n þ 1Þ / pfH l ðnþ1Þ ðnÞ j H l ; An ; Z n g Y nþ1 Y n ¼ pfR j Qnþ1 ; H nþ1 ðj ¼ 1; . . . ; mÞg Á j pfQi g Á pfAi j Qi g So, the distribution i¼1 i¼1 ( ðkÞ ðiÞ ) Y m ð1Þ YY nþ1 m ðiÞ ðiÀ1Þ H l j R; H j ði ¼ 1;...;n þ 1; j ¼ 1;...;m; ði;jÞ 6¼ ðk;lÞÞ Á pfH j g Á pfH j j H j ; AiÀ1 ; Z iÀ1 g p j¼1 iÀ2 j¼1 Ai ði ¼ 1;...;nÞ; Z i ði ¼ 1;...;nÞ; Qi ði ¼ 1;...;n þ 1Þ Y n ðiÞ Á pfZ i j H j ðj ¼ 1; . . . ; mÞ; Ai g ð1Þ i¼1 can be sampled.
6.
108 C.-G. Bai / The Journal of Systems and Software 77 (2005) 103–112(2) Zk. If k < n (4) Qk ( ) 8 9 ðiÞ < Z k j R; H ðiÞ ði ¼ 1;. ..; n þ 1; j ¼ 1;.. .;mÞ = Qk j R;H j ði ¼ 1; .. .; n þ 1; j ¼ 1; .. .; mÞ j pp Ai ði ¼ 1; .. .; nÞ; Z i ði ¼ 1; .. .; nÞ; Qi ði ¼ 1;. .. ;n þ 1; i 6¼ kÞ : ; ( ) Ai ði ¼ 1;.. .;nÞ; Z i ði ¼ 1; ... ;n;i 6¼ kÞ; Qi ði ¼ 1;.. .;n þ 1Þ ðiÞ R; H j ði ¼ 1;. .. ;n þ 1; j ¼ 1;. .. ;mÞ; Ai ði ¼ 1; .. .; nÞ 8 9 /p < R; H ðiÞ ði ¼ 1;. ..; n þ 1; j ¼ 1;.. .;mÞ; Ai ði ¼ 1;.. .;nÞ = Z i ði ¼ 1;. .. ;nÞ; Qi ði ¼ 1; .. .; n þ 1Þ j /p Y nþ1 Y n : ; ¼ pfR j Qnþ1 ; H nþ1 ðj ¼ 1; .. .; mÞg Á pfQi g Á pfAi j Qi g Z i ði ¼ 1;. ..; nÞ; Qi ði ¼ 1;.. .;n þ 1Þ j i¼1 i¼1 Y nþ1 Y n Y m ð1Þ YY nþ1 m ðiÞ ðiÀ1Þ ¼ pfR j Qnþ1 ; H nþ1 ðj ¼ 1; ... ;mÞg Á pfQi g Á pfAi j Qi g Á pfH j g Á pfH j j H j ; AiÀ1 ; Z iÀ1 g j i¼1 i¼1 j¼1 iÀ2 j¼1 Y n ðiÞ Y m YY nþ1 m Á pfZ i j H j ðj ¼ 1;. .. ;mÞ; Ai g ð1Þ ðiÞ ðiÀ1Þ Á pfH j g Á pfH j j Hj ; AiÀ1 ; Z iÀ1 g i¼1 j¼1 iÀ2 j¼1 if k < n + 1 Y n ( ðiÞ ) ðiÞ Qk j R; H j ði ¼ 1; . . . ; n þ 1; j ¼ 1; .. . ; mÞ Á pfZ i j H j ðj ¼ 1;.. .;mÞ; Ai g p i¼1 Ai ði ¼ 1; . . . ;nÞ; Z i ði ¼ 1; . .. ; nÞ; Qi ði ¼ 1; . . . ;n þ 1; i 6¼ kÞ ðiÞ / pfZ i j H j ðj ¼ 1;. ..; mÞ; Ai g Á pfH Ak ðkþ1Þ ðkÞ j H Ak ; A k ; Z k g / pfQk g Á pfAk j Qk g if k = n + 1So, the distribution ( ) ðiÞ Qk j R; H j ði ¼ 1; .. . ; n þ 1; j ¼ 1; . . . ; mÞ 8 9 p < Z k j R; H ðiÞ ði ¼ 1;. ..;n þ 1; j ¼ 1;...; mÞ j = Ai ði ¼ 1; . . .; nÞ; Z i ði ¼ 1; . . . ; nÞ; Qi ði ¼ 1; . . . ; n þ 1; i 6¼ kÞp : ; nþ1 Ai ði ¼ 1;...; nÞ; Z i ði ¼ 1; ...;n;i 6¼ kÞ; Qi ði ¼ 1; ...;n þ 1Þ / pfR j Qnþ1 ; H j ðj ¼ 1; . . . ; mÞg Á pfQnþ1 g So, the distributioncan be sampled. ( ) ðiÞ Qk j R; H j ði ¼ 1; . . . ; n þ 1; j ¼ 1; . . . ; mÞ(3) Ak p 8 9 Ai ði ¼ 1; . . . ; nÞ; Z i ði ¼ 1; . . . ; nÞ; Qi ði ¼ 1; . . . ; n þ 1; i 6¼ kÞ < Ak j R; H ðiÞ ði ¼ 1; . . . ; n þ 1; j ¼ 1; . . . ; mÞ j =p can be sampled. : ; Ai ði ¼ 1; . . . ; n; i 6¼ kÞ; Z i ði ¼ 1; . .. ; nÞ; Qi ði ¼ 1; . . . ; n þ 1Þ (5) R 8 9 ( ðiÞ ) < R; H ðiÞ ði ¼ 1; . . . ; n þ 1; j ¼ 1; .. . ; mÞ; Ai ði ¼ 1; . . .; nÞ = j R j H j ði ¼ 1;...;n þ 1; j ¼ 1;...;mÞ /p p : ; Ai ði ¼ 1;...;nÞ; Z i ði ¼ 1;...;nÞ; Qi ði ¼ 1;...;n þ 1Þ Z i ði ¼ 1; .. . ; nÞ; Qi ði ¼ 1; . . . ; n þ 1Þ ( ) ðiÞ R;H j ði ¼ 1;...;n þ 1; j ¼ 1;...;mÞ; Ai ði ¼ 1;...;nÞ Y nþ1 Y n /p ¼ pfR j Qnþ1 ; H nþ1 ðj ¼ 1; . . . ;mÞg Á j pfQi g Á pfAi j Qi g Z i ði ¼ 1;...;nÞ; Qi ði ¼ 1;...;n þ 1Þ i¼1 i¼1 Y nþ1 Y n Y m YY nþ1 m ¼ pfR j Qnþ1 ;H nþ1 ðj ¼ 1;...;mÞg Á j pfQi g Á pfAi j Qi g ð1Þ ðiÞ ðiÀ1Þ Á pfH j g Á pfH j j H j ; AiÀ1 ; Z iÀ1 g i¼1 i¼1 j¼1 iÀ2 j¼1 Y m ð1Þ YY nþ1 m ðiÞ ðiÀ1Þ Y n Á pfH j g Á pfH j j H j ; AiÀ1 ; Z iÀ1 g ðiÞ Á pfZ i j H j ðj ¼ 1; .. . ; mÞ; Ai g / pfAk j Qk g j¼1 iÀ2 j¼1 i¼1 Y n ðiÞ ðkþ1Þ ðkÞ ðkÞ Á pfZ i j H j ðj ¼ 1;...;mÞ; Ai g Á pfH j j H j ; Ak ; Z k g Á pfZ k j H Ak ; Ak g i¼1 ¼ pfR j Qnþ1 ;H nþ1 ðj ¼ 1;...;mÞg jSo, the distribution 8 9 So, the distribution < Ak j R; H ðiÞ ði ¼ 1; . . . ; n þ 1; j ¼ 1; . . . ; mÞ j = ( ) ðiÞp R j H j ði ¼ 1; . . . ; n þ 1; j ¼ 1; . . . ; mÞ : ; p Ai ði ¼ 1; . . . ; n; i 6¼ kÞ; Z i ði ¼ 1; . . . ; nÞ; Qi ði ¼ 1; . . . ; n þ 1Þ Ai ði ¼ 1; . . . ; nÞ; Z i ði ¼ 1; . . . ; nÞ; Qi ði ¼ 1; . . . ; n þ 1Þcan be sampled. can be sampled.
7.
C.-G. Bai / The Journal of Systems and Software 77 (2005) 103–112 109 From the above discussion it is derived that the (15) Predict the software reliabilityconditional distributions of every variable can be Xm ðnþ1Þsampled. Therefore, if we know the values of any R¼1À qj pðH j Þsubset of all variables in the model, the others can j¼1be inferred by the Gibbs sampling approach. We (16) End.call this algorithm the Gibbs sampling-basedAlgorithm. Although the time which the algorithm consumes is 4. A case studymanageable, if all Ai (i = 1, . . . , n) , Zi (i = 1, . . . , n)are known, a simpler algorithm to predict software We apply this model to software known as the Spacereliability can be derived by a message-passing program. Rothermel et al. (2001) describe the programscheme (Pearl, 1986). We call this the Simpliﬁed as follows: ‘‘Space consists of 9,564 lines of C codeAlgorithm. (6,218 executable), and functions as an interpreter for Assume that operational proﬁle is invariable, that is an array deﬁnition language (ADL). The program readsQ1 = Q2 = Á Á Á a ﬁle that contains several ADL statements, and checks So, let Q = {qjjqj = Pr[Ai = j], (1 6 j 6 m)}. the contents of the ﬁle for adherence to the ADL gram- Given Ai = ai (i = 1, . . . , n), Zi = zi (i = 1, . . . , n), the mar and to speciﬁc consistency rules. If the ADL ﬁle issimpliﬁed algorithm is as follows: correct, space outputs an array data ﬁle containing a list of array elements, positions, and excitations; otherwise, (1) Give Ai = ai(i = 1, . . . , n), Zi = zi(i = 1, . . . , n) the program outputs an error message’’. We apply ð1Þ ð1Þ (2) Give the prior distributions of H j ) pðH j Þ, EMBN to this data set to predict the software reliability (j = 1, . . . , m) behavior in this program. (3) Give the prior distribution of Q = {qj,j = 1, . . . , With the assumption that the initial failure rates of m})p(Q) each operation follow a Beta distribution and the oper- (4) 1 ) i ational proﬁle follows a Dirichlet distribution, we apply ðiÞ ðiÞ ðiÞ (5) pðH j Þ ) p ðH j Þ, p ðH j Þ (j = 1, . . . , m) denote the model for software reliability prediction. ðiÞ the prior distribution of H j . Assume that ðiÞ (6) Calculate the posterior distribution of H j ðj ¼ ð1Þ ðiÞ ðiÞ H j $ Betaðaj ; bðiÞ Þ; j Q $ Dirððq1 ; . . . ; qm Þ 1; . . . ; mÞ by Bayesian rule. Let hðH j j Z i ; Ai Þ ðiÞ j q1 ; . . . ; qm Þ denote the posterior distribution of H j , respectively. where Beta(a, b) is Beta distribution with parameters a, ðiÞ ðiÞ ðiÞ b. hðH j j Z i ¼ zi ; Ai ¼ ai Þ / pfZ i ¼ zi j H j gpfH j g j ¼ ai ðiÞ ðiÞ Dir((q1, . . . , qm)jq1, . . . , qm) is a Dirichlet distribution, hðH j j Z i ¼ zi ; Ai ¼ ai Þ ¼ pfH j g j 6¼ ai if ðiÞ ðiÞ (7) hðH j j Z i ¼ zi ; Ai ¼ ai Þ ) pðH j Þ pðq1 ;q2 ; .. . ;qm j q1 ; q2 ; . .. ;qm Þ ðiþ1Þ (8) Calculate the distribution of H j (j = 1, . . . , m) by 8 m C Pq ! ðiþ1Þ ðiÞ j¼1 j Q qj À1 m Pm Hj ¼ H j À zi d; j ¼ ai Qm qi ; 0 6 pi 6 1; i ¼ 1;. . .; m; pj ¼ 1 ¼ and j¼1 Cðqj Þ j¼1 j¼1 ðiþ1Þ ðiÞ : Hj ¼ Hj ; j 6¼ ai 0 otherwise ðiþ1Þ Let pðH j Þ (j = 1, . . . , m) denote the distribution The simpliﬁed algorithm is adopted in this case study. of ðiþ1Þ Hj : For step (6) of the algorithm, we have (9) Calculate the posterior distribution of Q by ð1Þ ð1Þ ð1Þ H j j Z 1 ¼ z1 ; A1 ¼ a1 $ Betaðaj þ z1 ; bj þ 1 À z1 Þ; Bayesian rule. j ¼ ai hðQ j Ai ¼ ai Þ / pðAi ¼ ai j QÞpðQÞ ðiþ1Þ(10) h(QjAi = ai) ) p(Q) In order to keep H j still a Beta distribution, we use ðiþ1Þ ðiÞ ðiþ1Þ ðiÞ(11) If i = n, then go (12); else i + 1 ) i and go (5) EH j ¼ EH j À d other than H j ¼ H j À d in the ðnþ1Þ step (8). From this we have(12) Output pðH j Þ (j = 1, . . . , m) !(13) Output p(Q) ð1Þ ð1Þ ð1Þ aj þ z1 À dz1 ðaj þ bj þ 1Þ ð2Þ(14) Estimate the operational proﬁle: Q = {qjjj = 1, . . . , H j $ Beta ð1Þ m} from p(Q) bð1Þ þ 1 À z1 þ dz1 ðaj þ bj þ 1Þ j ð1Þ
8.
110 C.-G. Bai / The Journal of Systems and Software 77 (2005) 103–112So, Table 1 ( ð2Þ ð1Þ ð1Þ ð1Þ Software reliability prediction by EMBN model aj ¼ aj þ z1 À dz1 ðaj þ bj þ 1Þ Failure Operational proﬁle Software reliability j ¼ a1 ð2Þ ð1Þ order f bj ¼ bð1Þ j þ 1 À z1 þ dz1 ðaj þ ð1Þ bj þ 1Þ For Set A For Set B( 1 0.1995, 0.3112, 0.4894 0.5001 0.6967 ð2Þ ð1Þ 2 0.1991, 0.3115, 0.4894 0.8571 0.8889 aj ¼ aj j 6¼ a1 3 0.1991, 0.3112, 0.4897 0.9231 0.9286 bj ¼ bð1Þ ð2Þ j 4 0.1995, 0.3112, 0.4894 0.9444 0.9347 5 0.1991, 0.3112, 0.4897 0.9600 0.9583Furthermore 6 0.1991, 0.3115, 0.4894 0.9643 0.9630 ( ðiþ1Þ ðiÞ ðiÞ 7 0.1991, 0.3112, 0.4897 0.9697 0.9688 aj ¼ aj þ zi À dzi ðaj þ bðiÞ þ 1Þ j 8 0.1991, 0.3112, 0.4897 0.9744 0.9730 ðiÞ j ¼ ai 9 0.1991, 0.3112, 0.4897 0.9773 0.9762 bj ¼ bðiÞ þ 1 À zi þ dzi ðaj þ bðiÞ þ 1Þ ð2Þ j j 10 0.1995, 0.3112, 0.4894 0.9792 0.9787( 11 0.1995, 0.3112, 0.4894 0.9815 0.9808 ðiþ1Þ ðiÞ aj ¼ aj 12 0.1991, 0.3115, 0.4894 0.9833 0.9828 ðiþ1Þ j 6¼ ai 13 0.1991, 0.3112, 0.4897 0.9848 0.9844 bj ¼ bðiÞ j 14 0.1991, 0.3112, 0.4897 0.9861 0.9855 15 0.1991, 0.3112, 0.4897 0.9882 0.9878In the step (9), let 16 0.1991, 0.3112, 0.4897 0.9891 0.9888 17 0.1991, 0.3115, 0.4894 0.9901 0.9897 ð1Þ ðjÞ ðmÞhðQ j Ai ¼ ai Þ $ Dir qi ; . . . ; qi þ 1; . . . ; qi j ¼ aj 18 0.1991, 0.3115, 0.4894 0.9907 0.9903 19 0.1991, 0.3115, 0.4894 0.9915 0.9912 20 0.1995, 0.3112, 0.4894 0.9922 0.9919and assume that q1 = 2, q2 = 2, q3 = 2, d = 0.0002. 21 0.1991, 0.3112, 0.4897 0.9927 0.9925 In this study, it is interesting to see to what extent the 22 0.1991, 0.3112, 0.4897 0.9932 0.9930prediction is dependent on diﬀerent initial values of the 23 0.1991, 0.3112, 0.4894 0.9937 0.9935parameters. Two kind of initial values of parameters are 24 0.1991, 0.3112, 0.4897 0.9941 0.9939as follows: 25 0.1991, 0.3112, 0.4897 0.9945 0.9944 26 0.1991, 0.3112, 0.4897 0.9949 0.9948Set A: a11 ¼ 1; b11 ¼ 1; a12 ¼ 1; b12 ¼ 1; a13 ¼ 1; b13 ¼ 1:Set B: a11 ¼ 1; b11 ¼ 3; a12 ¼ 2; b12 ¼ 2; a13 ¼ 1; b13 ¼ 4: In order to evaluate the performance of the proposed The software is tested for 3000 runs in total, and there model, we compare our results with those of the Kaani-are three operations (m = 3). Twenty-six failures occur che–Kanoun model (KK model) (Kaaniche andin total during testing. Because we are interested in the Kanoun, 1992). The KK model is one of the few dis-trend of the software reliability prediction along with crete software reliability models. Most discrete softwarethe testing process, all the software reliability predic- reliability models assume that all design faults can betions for n = 1, . . . , 3000 are calculated and studied. removed from the software. However, this is not typicalHowever, for the sake of space limitation, only those of most large software systems. The KK model, as welldata with failures occurring are listed in Table 1. as ours, does not need this assumption. Due to space Table 1 shows the results of software reliability by limitation, only the parameter Set A is used for ourEMBN model. The serial numbers of failure order are model. Table 2 shows the failure interval (the numberlisted in the ﬁrst column with the title ‘‘Failure Order’’. of runs from one failure to the next failure) predictionsThe estimations of operational proﬁle after f failures oc- for each of the models, and Fig. 3 illustrates the results.curred are listed in the second column. Table 1 shows From the results, we see that the EMBN model has goodthe estimations of operational proﬁle basically keep con- estimation and prediction capabilities. From the results,stant, especially when f is large. The software reliability we can see that when the failure order is small, the KKprediction for parameters values Set A and Set B are model is unable to predict the failure intervals. Inlisted in the third and the fourth column, respectively. contrast, the EMBN model is always able to make pre-Table 1 shows that at the beginning of the test the third dictions. It seems that better predictions can be achievedcolumn is quite diﬀerent from the fourth column. That by combining the prior experience and the failure datameans that at the beginning of the test the prior infor- from software testing. However, in the very late stagemation play a key role in predicting software reliability. of software testing the behaviors of software are sud-It tells us that if the number of test runs is not very big denly changed. This means that it is diﬃcult to ﬁndthe prior distribution should be selected carefully. How- more defects by the current software testing methods.ever, the trend of software reliability becomes stable In this situation, both of the two methods fail to giveregardless of the initial values of the parameters. That proper predictions. If we want to ﬁnd more defects,means that if the number of test runs is large, assuming the software testing methods should be changed (Baidiﬀerent prior distributions donÕt make much diﬀerence. et al., 2003). This means that software testing proﬁles
9.
C.-G. Bai / The Journal of Systems and Software 77 (2005) 103–112 111Table 2 obtain, so prior information should be incorporated.The SPACE failure data (* means no value) For this, the Bayesian approach seems appropriateFailure Failure interval Predictions by Predictions by (Pham and Pham, 2001). From this starting point, aorder f data gf EMBN model KK Model Bayesian network model is proposed to predict software1 5 2 * reliability. This is desirable for complex software sys-2 14 7 * tems where prior information may be available from ex-3 16 13 *4 27 18 * perts or former development processes and experience.5 17 25 20 However, Bayesian network models require the user6 31 28 19 to develop prior distributions. There exist several ways7 34 33 20 to do this, such as relying on expert opinion. The expert8 35 39 25 opinion includes empirical experience with other similar9 37 44 2810 41 48 31 software, knowledge of software testing, and other11 54 54 38 background information. Although speciﬁc assumptions12 58 60 44 were made in the development of our model, the initial13 59 66 50 distributions can also be obtained by combining infor-14 122 72 54 mation gained in the former development process.15 79 85 6716 99 92 73 Moreover, other distributions can be assumed to reﬂect17 79 101 83 diﬀerent testing conﬁgurations.18 135 107 89 Bayesian models often involve high complexity and19 136 117 99 computational eﬀort. As a result, the suggested algo-20 137 128 110 rithm, the Gibbs sampling-based Algorithm, would be21 142 137 11622 163 147 126 time consuming. However, it is unreasonable to expect23 185 158 138 that a simple model would be able to model a complex24 193 170 150 system. A very simple algorithm for a special case, which25 224 183 164 often happens in practice, is proposed in the paper.26 865 197 178 Beta distributions are adopted as priors for the case study. The Beta distribution has many interesting prop- erties. The most important property of the Beta distribu- 900 tion is that it is conjugated with the binomial distribution. Thus the Beta distribution ﬁts very well 800 for the variables that are deﬁned on (0, 1). However, 700 studies with other distributions present avenues for fur- Failure Data ther research. 600ηf 500 400 Acknowledgment ENBN Model 300 The author would like to thank the two anonymous K K Model 200 referees and the editor for providing helpful comments. 100 0 References 0 5 10 15 20 25 30 f Bai, C.G., Hu, Q.P., Xie, M., 2005. Software failure prediction based Fig. 3. Comparison between EMBN model and KK model. on a Markov Bayesian network model. Journal of Systems and Software 74 (3), 275–282. Bai, C.G., Cai, K.Y., Chen, T.Y., 2003. An eﬃcient defect estimation method for software defect curves. In: Proceedings of the 27thwill be changed. Due to the complexity of this problem, Annual International Computer Software and Applications Con-it is diﬃcult to solve in the current paper. However, this ference, Dallas, USA, pp. 534–539.problem will be studied further in the future. Basu, S., Ebrahimi, N., 2003. Bayesian software reliability models based on Martingale processes. Technometrics 45 (2), 150–158. Bickmore, T.W., 1994. Real-time sensor data validation. NASA Contractor Report 195295, National Aeronautics and Space5. Conclusion Administration. Bishop, P., Pullen, F.D., 1988. Probabilistic modeling of software For most safety-critical software with high reliability failure characteristics. in: Proceedings of the IFAC Workshoprequirements, an adequate sample of failures is hard to SAFECOMPÕ88, pp. 87–93.
10.
112 C.-G. Bai / The Journal of Systems and Software 77 (2005) 103–112Boland, P.J., Singh, H., 2002. Determining the optimal release time for Lewis, R.W., Ransing, R.S., 1997. A semantically constrained Bayes- software in the geometric Poisson reliability model. International ian network for manufacturing diagnosis. International Journal of Journal of Reliability, Quality and Safety Engineering 9 (3), 201– Production Research 35, 2171–2187. 213. Lyu, M.R. (Ed.), 1996. Handbook of Software Reliability Engineering.Boland, P.J., Singh, H., 2003. A birth-process approach to MorandaÕs McGraw-Hill, New York. geometric software-reliability model. IEEE Transactions on Reli- Madigan, D., Mosurski, K., Almond, R.G., 1996. Explanation in ability 52 (2), 168–174. belief networks. Journal of Computational and Graphical StatisticsBouissou, M., Martin, F., Ourghanlian, A., 1999. Assessment of a 6, 160–187. safety-critical system including software: A Bayesian belief network Malaiya, Y.K., Li, M.N., Bieman, J.M, 2002. Software reliability for evidence sources. in: Proceedings of the Annual Reliability and growth with test coverage. IEEE Transactions on Reliability 51 (4), Maintainability Symposium, pp. 142–150. 420–426.Cai, K.Y., 1998a. Software defect and operational proﬁle modeling. Musa, J.D., 1993. Operational proﬁles in software reliability engineer- Kluwer Academic Publishers, Dordrecht. ing. IEEE Software 10 (1993), 14–32.Cai, K.Y., 1998b. On estimating the number of defects remaining in Musa, J.D., 1996. The operational proﬁle. In: Ozekici (Ed.), Reliability software. Journal of Systems and Software 40, 93–114. and Maintenance of Complex Systems, NATO ASI Series, vol.Cai, K.Y., 2000. Towards a conceptual framework of software run F154. Springer, Berlin, pp. 332–343. reliability modeling. Information Sciences 126 (1–4), 137–163. Neil, M., Littlewood, B., Fenton, N., 1996. Applying Bayesian beliefCai,, K.Y., 2003. In: Proceedings of the 3rd International Conference networks to system dependability assessment. in: Proceedings on Quality Software. IEEE Computer Society Press, Silver Spring, of Safety Critical Systems Club Symposium, Springer-Verlag, MD, pp. 32–39. Berlin.Cai, K.Y., Cai, L., Wang, W.D., 2001. On the neural network Ozekici, S., Soyer, R., 2003. Reliability of software with an operational approach in software reliability modeling. Journal of Systems and proﬁle. European Journal of Operational Research 149 (2), 459– Software 58, 47–62. 474.Castillo, E., Sarabia, J.M., Solares, C., 1999. Uncertainty analyses in Pai, G.J., 2001. Combining Bayesian belief networks with fault tree to fault trees and Bayesian networks using FORM SORM methods. enhance software reliability analysis. in: Proceedings of the IEEE Reliability Engineering and Systems Safety 65, 29–40. International Symposium on Software Reliability Engineering.Fenton, N., Neil, M., 1999. Software metrics: successes, failures and Pearl, J., 1986. Fusion, propagation and structuring in belief network. new directions. Journal of Systems and Software 47, 149–157. Artiﬁcial Intelligence 29, 241–288.Gelfand, A., Smith, A., 1985. Sampling based approaches to calculat- Pfeﬀerman, J.D., Cemuschi-Frias, B., 2002. A non-parametric non- ing marginal densities. Journal of the American Statistical Asso- stationary procedure for failure prediction. IEEE Transactions on ciation 85, 385–409. Reliability 51 (4), 434–442.Goel, A.L., 1985. Software reliability models: assumptions, limitations, Pham, L., Pham, H., 2001. A Bayesian predictive software reliability and applicability. IEEE Transactions on Software Engineering SE- model with pseudo-failures. IEEE Transactions on Systems, Man 11, 1411–1423. and Cybernetics 31 (3), 233–238.Goel, A.L., Okumoto, K., 1979. Time dependent error detection rate Reussner, R.H., Schmidt, H.W., Poernomo, I.H., 2003. Reliability model for software reliability and other performance measure. prediction for component-based software architectures. Journal of IEEE Transactions on Reliability 28 (3), 206–211. Systems and Software 66 (3), 241–252.Ho, S.L., Xie, M., Goh, T.N., 2003. A study of the connectionist Rothermel, G., Untch, R., Chu, C., Harrold, M.J., 2001. Prioritizing models for software reliability prediction. Computers and Math- test cases for regression testing. IEEE Transactions on Software ematics with Application 46 (7), 1037–1045. Engineering 27, 929–948.Jelinski, Z., Moranda, P.B., 1972. Software reliability research. In: Sahinoglu, M., Deely, J.J., Capar, S., 2001. Stochastic Bayes measures Proceedings of the Statistical Methods for the Evaluation of to compare forecast accuracy of software-reliability models. IEEE Computer System Performance. Academic Press, New York, pp. Transactions on Reliability 50 (1), 92–97. 465–484. Sigurdsson, J.H., Walls, L.A., Quigley, J.L., 2001. Bayesian belief netsJordan, M.I., Ghahramani, Z., Jaakkola, T.S., Saul, L.K., 1999. An for managing expert judgment and modeling reliability. Quality introduction to variational methods for graphical models. In: and Reliability Engineering International 17, 181–190. Jordan, M. (Ed.), Learning in Graphical Models. MIT Press, Sitte, R., 1999. Comparison of software-reliability-growth predictions: Cambridge, MA. neural networks vs parametric-recalibration. IEEE TransactionsKaaniche, M., Kanoun, K., 1992. The discrete-time hyperexponential on Reliability 48, 285–291. model for software reliability growth evaluation. in: Proceedings of Smidts, C., Stoddard, R.W., Stutzke, M., 1998. Software reliability the Third International Symposium on Software Reliability Engi- models: an approach to early reliability prediction. IEEE Trans- neering, pp. 64–75. actions on Reliability 47, 268–278.Karunanithi, N., Whitley, D., 1992. Prediction of software reliability Xie, M., 1991. Software reliability modeling. World Scientiﬁc Pub- using feedforward and recurrent neural nets. in: International Joint lisher, Singapore. Conference on Neural Networks, vol. 1, pp. 800–805. Xie, M., 1993. Software reliability models: a selected bibliography.Khoshgoftaar, T.M., Szabo, R.M., 1996. Using neural networks to Journal of Software Testing, Veriﬁcation and Reliability 3, 32– predict software faults during testing. IEEE Transactions on 38. Reliability 45 (4), 56–462. Yu, Y., Johnson, B.W., 2003, A BBN approach to certifying theKuo, L., Lee, J.C., 1997. Bayes inference for S-shaped software- reliability of COTS software system reliability. in: Proceedings reliability growth models. IEEE Transactions on Reliability 46 (1), Annual Reliability and Maintainability Symposium, pp. 19–24. 76–80. Zheng, S.H., 2002. Dynamic release policies for software systems withKuo, L., Yang, T.Y., 1996. Bayesian computation for nonhomogene- a reliability constraint. IIE Transactions 34 (3), 253–262. ous Poisson processes in software reliability. Journal of the Zhu, W.P., 2003. Using Bayesian network on network tomography. American Statistical Association 91 (434), 763–773. Computer Communication 26, 155–163.
Clipping is a handy way to collect and organize the most important slides from a presentation. You can keep your great finds in clipboards organized around topics.
Be the first to comment