Successive Software Reliability Growth Model: A Modular Approach

2012 ARS, India: Chennai
Track 2, Session 11
Begins at 2:40 PM, Thursday, October 11

Successive Software Reliability
Growth Model: A Modular Approach
Dr. Ajeet Kumar Pandey
Cognizant Technology Solution, Hyderabad, India

PRESENTATION SLIDES
The following presentation was delivered at the:

International Applied Reliability Symposium, India
October 10 - 12, 2012: Chennai, India
http://www.ARSymposium.org/india/2012/

The International Applied Reliability Symposium (ARS) is intended to be a forum for reliability and maintainability
practitioners within industry and government to discuss their success stories and lessons learned regarding
the application of reliability techniques to meet real world challenges. Each year, the ARS issues an open
"Call for Presentations" at http://www.ARSymposium.org/india/presenters/index.htm and the presentations
delivered at the Symposium are selected on the basis of the presentation proposals received.

Although the ARS may edit the presentation materials as needed to make them ready to print, the content of the
presentation is solely the responsibility of the author. Publication of these presentation materials in the
ARS Proceedings does not imply that the information and methods described in the presentation have been
verified or endorsed by the ARS and/or its organizers.

The publication of these materials in the ARS presentation format is
Copyright © 2012 by the ARS, All Rights Reserved.

Brief: Myself and Proposal Vocabulary
 Dr. Ajeet Kumar Pandey, Ph.D. (Software Reliability) from IIT Kharagpur,  COMP Error: Commission/Omission/Misinterpretation/Performance Error
working as Sr. RAMS Engineer at Cognizant Technology Solution,
Hyderabad, India.  DC: Defect Checklist

 Division: CoE (Centre of Excellence).  FDR : Fault Density Indicator at Requirement Phase
 Area: Reliability and Safety (Assessment and Prediction), Regulatory and  FDD: Fault Density Indicator at Design Phase
Compliance of Safety Critical System (Rail/ Automotive/ Avionics and
Medical).  FDC: Fault Density Indicator at Coding Phase

 Proposal: To improve the reliability of software successively by predicting  FDT: Fault Density Indicator at Testing Phase
and fixing the faults before they propagate.
Applied Reliability Symposium, India 2012

 RRSM: Reliability Relevant Software Metrics
 Key Points: Realistic (Birth to Death) approach for software, early reliability
assurance before testing, reliability relevant software metrics and review  SDLC: Software Development Life Cycle
defect checklist.
 SRGM: Software Reliability Growth Model

 SSRGM: Successive Software Reliability Growth Model

Dr. Ajeet Kumar Pandey, Cognizant Technology Solution Track 2 Session 11 Slide Number: 2 Dr. Ajeet Kumar Pandey, Cognizant Technology Solution Track 2 Session 11 Slide Number: 3

Agenda Introduction: Software Reliability

 Introduction 5 min  Applicability of software keeps on increasing, from basic home
appliances to safety critical business applications.
 Earlier Works 8 min
 Observation and Motivation 7 min  Size, complexity and dependency on software based systems are
growing.
 Proposed SSRGM Model 20 min
 Results and Discussion 5 min  Software reliability becomes a challenging objective for both
developer as well as user.
 Summary 5 min
 Relevant References • Developer: How to develop fault free software (system)?


 Questions 10 min • User: How to choose a reliable (failure free) system?
 System failures due to a software failure are very common and
can result in undesirables situations.


Introduction: H/W Reliability vs. S/W Reliability Introduction: Software Reliability

Reliability is the probability that a Software reliability: probability that a
system or component performs its software system or component
required functions under stated performs its intended function under
conditions for a specified period of the specified operating conditions
time. over the specified period of time.

A software failure is defined as “the
Software faults are the deviation of the program behavior
root cause of failures, from requirements,” whereas a fault
making the software is defined as “the defect in the

unreliable. program that causes failures when
executed.
One measure of software
reliability is the number of
residual faults, and it has been The proposal is to develop a new model
observed that the more to predict and fix the number of faults at
residual faults a software has, each phase of SDLC before they
the less reliable it is. propagate, thus growing the reliability
successively.


Earlier Works: Software Reliability Models Earlier Works: Software Reliability Models
 A software reliability model usually refers to the mathematical form  SRGM assumes that the reliability of S/W will continue to grow if
of the equation that is used in estimating/predicting the number of the observed error (during testing) are removed (i.e., number of
faults/failures in a software. residual faults decreases with progression of testing).
 Software reliability models can be broadly categorized into two  SRGM Limitations:
types (Pham, 2006): Deterministic and Probabilistic.
 Can be applied once coding is done, and is useful only if
 Some probabilistic models are: failure rate models (times between failure data ID is available.
failure models), failure or fault count models (NHPP models),
 Can’t do much with requirements and design phase in term of
error or fault seeding models, Markov structure models, reliability
reliability.


growth models, etc.
 Costly and unrealistic reliability improvement approach.
 Reliability growth: Fix the defect, grow the reliability.
 Reliability is a Birth to Death process, so it will be good enough if
 A SRGM is a mathematical equation by which version (i) reliability
the reliability growth process is applied since the beginning.
is improved by using data of version (i-1) or any earlier version.
 Key References: [1], [2], and [3].


Earlier Works: Early Software Reliability Models Earlier Works: Affecting Factors
 Predicting the reliability of a software system before the testing phase is  Around 40 reliability relevant software measures are given IEEE STD-982.2
known as early software reliability prediction. to produce reliable software.

 Early prediction attracts both software professionals and managers  A study was conducted by Zhang and Pham (2000) to find the factors
because it provides insight towards optimal development strategies. affecting software reliability.

 Failure data are not available in the early phase of the software  Li, et al. (2003) have shown that there are 30 software metrics associated
development life cycle, and reliability can be predicted on the basis of the with different phases of the software life cycle, and among these metrics,
software metrics, developer’s process maturity level and expert opinions. some are relevant to reliability and can be identified at the early stage of the
life cycle.
 Early reliability prediction seems to be useful, but the problem with early
 The Capability Maturity Model (CMM) has become a popular methodology to
software reliability predictions are :

develop high-quality software within budget and time. Harter et al. (2000)
 First, how to find the software failure intensity function without found that a 1% improvement in process maturity resulted in 1.6% increase
executing the software, which is required to calculated the software in product quality.
reliability?  Krishnan and Kellner (1999) found process maturity and personnel capability
 Second, how the time parameter of reliability evaluation can be to be significant predictors (both at the 10% level) of the number of defects.
found during the early stage of software development?  Ref: [7], [8], [9],[10] and [11].
 Key References: [4], [5], and [6].


Observation & Motivation Proposed SSRGM Model
 Around hundreds of software reliability growth models (SRGMs) have  The proposed model assumes that the software is being developed
been developed to date. Limitation with SRGM are: late applicability, through a waterfall process model.
cost of fixing, failure data availability, not suitable for requirement/design
 A software engineer collects, measures and develops metrics so that
phase, etc.
indicators will be obtained. An indicator is a metric or combination of
 SRGM approaches for reliability prediction are not very useful in a metrics that provides insight into a software process, a software project,
practical scenario because version (i) reliability depends on the data of or the product itself.
version (i-1) or any earlier version.
 The proposed model utilizes software metrics and finds fault density
 Due to the several practical limitations with earlier software reliability indicators for each development phase using a fuzzy inference system
models, this work focuses on fault prediction model. A software system (FIS).


fails only if the residual faults are executed, causing failure and making it
 Also, the proposal is to use a defect checklist (DC) to fix the common
unreliable.
defects quickly and update the fault density indicator value, before
 The reliability of a software system depends on the number of residual passing it unto the next phase.
faults sitting dormant inside. Therefore, this work aims to predict and fix
 Finally, using the fault density indicator of the testing phase, the number
residual faults across the SDLS, growing reliability successively.
of residual faults is predicted.


Proposed Model Architecture Proposed Model Architecture
# Phase Metrics
1 Requirement RC, RS, RIW # Phase Input Variables Output
RRSM 2 Design FDR, DTE, PM Variables
Extracts 1 Requirement RC, RS, RIW FDR
3 Coding FDD, CTE, DPF
4 Testing FDT, TTE, SI, 2 Design FDR, DTE, PM FDD
SIZE 3 Coding FDD, CTE, DPF FDC
4 Testing FDT, TTE, SI, FDT
SIZE
Defect Checklists (Req. Phase) 5 Fault FDT Faults
Prediction
Derive COMP Error # Description [Prob.] Severity Reason

phase wise defects 1. Missing 0.8 M Change
and make a defect Var. Request
checklist. 2. … ….. …. …..
3.
4.
Sources: [7], [8], [9], and [12]


Proposed SSRGM Approach Proposed SSRGM Approach (cont’d)
 Model assumption: Waterfall development process model. Step 1: Identification of independent and dependent variables

 The model is based on fuzzy logic and implemented in MATLAB. The model
consists of the following steps: Table: Independent Variables Table: Dependent Variables
No. Independent Variables No Dependent Variables
 Identification of independent/dependent variables
1 Requirements Complexity (RC) 1 Fault density indicator at
 Development of fuzzy profile (on the basis of nature variables) 2 Requirements Stability (RS) requirements phase (FDR)
 Developing fuzzy rules (expert opinions) 3 Review, Inspection and 2 Fault density indicator at
Walkthrough (RIW) design phase (FDD)
 Information Processing (Mamdani FIS)
4 Design Team Experience (DTE) 3 Fault density indicator at
 Residual fault prediction


5 Process Maturity (PM) coding phase (FDC)
 Software metrics (independent variables) are considered as input variables 6 Coding Team Experience (CTE) 4 Fault density indicator at
to the model to get dependent variables (output). Independent variables are 7 Defined Process Followed (DPF) testing phase (FDT)
taken from PROMISE repository [14]. 8 Testing Team Experience (TTE) 5 Total number of residual faults
 Fault density indictors and residual faults are the dependent variables in this 9 Stake holder Involvement (SI) predicted (Faults)
study. There are four fault density indicators (FDR, FDD, FDC and FDT) 10 Size of program in LOC (SIZE)
associated with requirements, design, coding and testing phase,
respectively.


Proposed SSRGM Approach (cont’d) Proposed SSRGM Approach (cont’d)
Step 2: Development of fuzzy profile
 For logarithmic nature software metrics,
 Software metrics may follow either linear scale or logarithmic scale.
 Out of ten input variables, only three variables (RIW, PM and DPF) The profiles may take the values as VL (0; 0; 0.14), L (0; 0.14; 0.32), M
variation follow a linear nature. The remaining variables follow a (0.14; 0.32; 0.57), H (0.32; 0.57; 1.00), and VH (0.57; 1.00; 1.00).
logarithmic nature.
 For linear nature software metrics,
 All output variables are assumed to follow a logarithmic nature.
 On the basis of their nature, fuzzy profiles of software metrics are
developed and triangular fuzzy profiles are considered. The profiles may take the values as VL (0; 0; 0.25), L (0; 0.25; 0.50), M

(0.25; 0.50; 0.75), H (0.50; 0.75; 1.00), and VH (0.75; 1.00; 1.00).
 For all input variables, we have considered five levels, i.e., very low (VL)
to very high (VH).  For outputs,
 For all output variables, we have considered seven levels, i.e., very very
low (VVL) to very very high (VVH).
The profiles may take the values as VVL (0; 0; 0.08), VL (0; 0.08; 0.17), L
(0.08; 0.17; 0.29), M (0.14; 0.32; 0.57), H (0.17; 0.29; 0.44), VH (0.44; 0.64;
1.00), and VVH (0.64; 1.00; 1.00).

Proposed SSRGM Approach (cont’d) Proposed SSRGM Approach (cont’d)
Step 3: Development of fuzzy rules
Rules are developed
Table : Rules at Req. Phase
using two or more
Rule RC RS RIW FDR domain expert engineers.
1 L L L VL
2 L L M L
3 L L H M
. . . . .

Step 4: Information Processing: The Mamdani fuzzy inference system is
used. For defuzzification process, “Centroid Method” is considered.


RC RS RIW DTE PM CTE DPF TTE SI Size
VL 0.05 0.05 0.08 0.05 0.08 0.05 0.08 0.05 0.05 0.05
L 0.15 0.15 0.25 0.15 0.25 0.15 0.25 0.15 0.15 0.15
M 0.34 0.34 0.50 0.34 0.50 0.34 0.50 0.34 0.34 0.34
H 0.63 0.63 0.75 0.63 0.75 0.63 0.75 0.63 0.63 0.63
VH 0.86 0.86 0.92 0.86 0.92 0.86 0.92 0.86 0.86 0.86
Figures: Examples of Fuzzy profiles
Ajeet Kumar, Cognizant Track 2 Session 11 Slide Number: 20 Dr. Ajeet Kumar Pandey, Cognizant Technology Solution Track 2 Session 11 Slide Number: 21

Proposed SSRGM Approach (cont’d) Results and Discussion
ROMOSE repository (http://promisedata.org/repository/data/qqdefects) [12] dataset
Step 5: Fault Prediction are used for validation.
 Fault density indictor value is refined using DC, before sending to the next RC RS RIW DTE PM CTE DPF TTE SI SIZE Faults

phase. # Project F1 S7 S3 D1 P9 D2 D3 T2 P5 K TD
1 1 M L VH L H H H H H 6.02 148
 On the basis of fault density indicator of testing phase, total number of 2 2 L H VH L H H H H H 0.9 31
faults is computed as: 3 3 H H VH H VH VH H H VH 53.86 209
4 5 H M H L H M H M M 14 373
5 7 L M VH M H VH H M VH 21 204
6 8 M H H H M H M M H 5.79 53
 Fault detection process is not exactly linear with size. As size of a software 7 10 M H H H H H H M H 4.84 29
8 11 H H H H H H H H H 4.37 71
increases, portion of faults detected decreases due to saturation, time and 9 12 H L H VH H M M H H 19 90
experience.

10 13 H L M H H H H M H 49.1 129
11 14 VH H H H H H H H H 58.3 672
 Therefore, the FTP value is modified as provided in the equation below. 12 15 H VL H H H H H H VH 154 1768

The C2 value scales the effect of LOC value. Thus, residual faults can be 13 16 L M H H H H H H VH 26.67 109
14 17 L M M M H M H L M 33 688
predicted as, 15 19 H M H H H H H M H 87 476
16 20 VH VL M VL H VL L VL H 50 928
17 21 L M H H H H H H H 22 196
18 22 M L M H H M L M H 44 184
 C1 and C2 are constants obtained through recursive learning. The value of 19 23 H M VH L H H H H H 61 680
C1 and C2 for current projects are found to be 0.04 and 107 respectively. 20 27 H M VH M H L M M M 52 412
21 29 M VH VH VH H VH H VH VH 11 91
22 30 L VH VH H H H H H VH 1 5


Results and Discussion (cont’d)
Results and Discussion (cont’d)
Table: Project Data Set After Conversion Table : Fault Density Indicators, Actual Faults and Faults Predicted
# FDR FDD FDC FDT Size (LOC) Actual Proposed Fenton et al.
# Project RC RS RIW DTE PM CTE DPF TTE SI SIZE Faults
Faults Model (2008)
1 1 0.34 0.15 0.92 0.15 0.75 0.63 0.75 0.63 0.63 0.15 148
2 2 0.15 0.63 0.92 0.15 0.75 0.63 0.75 0.63 0.63 0.15 31 1 0.0925 0.1937 0.1387 0.3042 900 31 5.48 52
3 3 0.63 0.63 0.92 0.63 0.92 0.86 0.75 0.63 0.86 0.86 209 2 0.0815 0.0834 0.0739 0.3008 1000 5 6.02 46
4 5 0.63 0.34 0.75 0.15 0.75 0.34 0.75 0.34 0.34 0.34 373 3 0.457 0.3002 0.1802 0.4645 4370 71 40.61 51
5 7 0.15 0.34 0.92 0.34 0.75 0.86 0.75 0.34 0.86 0.63 204 4 0.1827 0.1378 0.1332 0.2703 4840 29 26.17 203
6 8 0.34 0.63 0.75 0.63 0.5 0.63 0.5 0.34 0.63 0.15 53 5 0.1827 0.3002 0.1802 0.4645 5790 53 53.81 48
7 10 0.34 0.63 0.75 0.63 0.75 0.63 0.75 0.34 0.63 0.15 29 6 0.3171 0.4571 0.392 0.4645 6020 148 55.94 75
8 11 0.63 0.63 0.75 0.63 0.75 0.63 0.5 0.63 0.63 0.15 71 7 0.2087 0.1898 0.142 0.5 11000 91 110.06 116
9 12 0.63 0.15 0.75 0.86 0.75 0.34 0.5 0.63 0.63 0.34 90 8 0.6306 0.6523 0.6802 0.8297 14000 373 232.48 349
10 13 0.63 0.15 0.5 0.63 0.75 0.63 0.75 0.34 0.63 0.63 129 9 0.2892 0.2543 0.185 0.4634 19000 90 176.25 347
11 14 0.86 0.63 0.75 0.63 0.75 0.63 0.75 0.63 0.63 0.86 672 10 0.186 0.2806 0.1723 0.2698 21000 204 113.43 262


12 15 0.63 0.05 0.75 0.63 0.75 0.63 0.75 0.63 0.86 0.86 1768 11 0.1825 0.1372 0.133 0.2395 22000 196 105.51 259
13 16 0.05 0.34 0.75 0.63 0.75 0.63 0.75 0.63 0.86 0.63 109 12 0.1825 0.1372 0.133 0.2395 26670 109 127.94 145
14 17 0.05 0.34 0.5 0.34 0.75 0.34 0.75 0.15 0.34 0.63 688 13 0.1825 0.2807 0.1725 0.2053 33000 688 135.74 444
15 19 0.63 0.34 0.75 0.63 0.75 0.63 0.75 0.34 0.63 0.86 476 14 0.5653 0.3024 0.3001 0.33 44000 184 291.02 501
16 20 0.86 0.05 0.5 0.05 0.75 0.05 0.25 0.05 0.63 0.86 928 15 0.457 0.3002 0.2322 0.3419 49100 129 336.59 516
17 21 0.05 0.34 0.75 0.63 0.75 0.63 0.75 0.63 0.63 0.63 196 16 0.7184 0.7334 0.8448 0.8671 50000 928 869.23 986
18 22 0.34 0.15 0.5 0.63 0.75 0.34 0.25 0.34 0.63 0.63 184 17 0.4211 0.4331 0.1801 0.3838 52000 412 400.17 430
19 23 0.63 0.34 0.92 0.15 0.75 0.63 0.75 0.63 0.63 0.86 680 18 0.2908 0.2554 0.1552 0.195 53860 209 210.61 210
20 27 0.34 0.34 0.92 0.34 0.75 0.15 0.5 0.34 0.34 0.86 412 19 0.458 0.3002 0.2322 0.5964 58300 672 697.48 674
21 29 0.34 0.86 0.92 0.86 0.75 0.86 0.75 0.86 0.86 0.34 91 20 0.4778 0.4944 0.3929 0.564 61000 680 690.17 722
22 30 0.05 0.86 0.92 0.63 0.75 0.63 0.75 0.63 0.86 0.15 5 21 0.6306 0.305 0.2499 0.3281 87000 476 573.44 581
22 0.4572 0.3002 0.2322 0.5319 154000 1768 1650.89 1526


Results and Discussion (cont’d) Summary
 Proposed model prediction result is compared with a model using  Software faults are the root causes of failures; thus, degrading the
Bayesian Nets (Fenton et al., 2008), as listed in the following table. reliability. This work aims to improve the reliability successively
through fault prediction modeling.
 From the table it is clear that the proposed approach, which is
based on fuzzy inference system, provides more accurate results  The model has discussed a comprehensive framework to gather the
than the model based on Bayesian Nets provided by Fenton et al. relevant metrics and defect checklist from each phases of SDLC,
(2008). processing it, and integrating it with the fuzzy logic system to predict
residual faults.
Evaluation Proposed Fenton et al. (2008)
 This model will be useful to software professionals by providing an
Measures Approach Approach [16]
insight to software metrics and its impact on software fault during the

SSE 440943.00 500895.00 development process.
MSE 20042.86 22767.95  Another benefits of this kind of fault prediction is to help developers
produce software with a minimum number of residual faults.
RMSE 141.57 150.89
MAPE 37.49 116.81


Where to Get More Information Where to Get More Information
1. Yamada, S., Ohba, M., and Osaki, S. (1983), S-shaped Reliability Growth Modelling 9. Li, M. and Smidts, C. (2003), A Ranking of Software Engineering Measures Based
for Software Error Detection, IEEE Transaction on Reliability, Vol. R-32, pp. 475–478. on Expert Opinion, IEEE Transaction on Software Engineering, Vol. 29, No. 9, pp.
2. Goel, A.L. (1985), Software Reliability Models: Assumptions, Limitations, and 811–24.
Applicability, IEEE Transaction on Software Engineering, Vol. SE–11, No. 12, pp. 10. Harter, D.E., Krishnan, M.S. and Slaughter, S.A. (2000), Effects of Process Maturity
1411–1423. on Quality, Cycle Time and Effort in Software Product Development, Management
3. Kapur, P. K. and Younes, S. (1995), Software Reliability Growth Model with Error Science, Vol. 46, pp. 451–466.
Dependency, Microelectronics and Reliability, Vol. 35, No. 2, pp. 273-278. 11. Krishnan, M. S. and Kellner, M. I. (1999), Measuring Process Consistency:
4. Rome Laboratory (1992), Methodology for Software Reliability Prediction and Implications Reducing Software Defects, IEEE Transaction on Software Engineering,
Assessment, Technical Report RL-TR-92-52, vol. 1 & 2. Vol. 25, No. 6, pp. 800–815.
5. Gaffney, G. E. and Pietrolewiez, J., (1990), An Automated Model for Software Early 12. PROMISE repository (2007), http://promisedata.org/repository/data/qqdefects.
Error Prediction (SWEEP), Proceeding of 13th Minnow Brook Workshop on Software 13. Pham, H. (2006), System Software Reliability, Reliability Engineering Series,


Reliability. Springer.
6. Pandey, A. K. and Goyal, N. K. (2009), A Fuzzy Model for Early Software Fault 14. Ross, T. J. (2005), Fuzzy Logic with Engineering Applications, Willy–India 2nd
Prediction using Process Maturity and Software Metrics, International Journal of Edition.
Electronics Engineering, Vol.1, No. 2, pp. 239–245. 15. Zadeh, L. A. (1965), Fuzzy Sets — Information and Control, Vol. 8, No. 3, pp. 338–
7. IEEE (1988), IEEE Guide for the Use of IEEE Standard Dictionary of Measures to 353.
Produce Reliable Software, IEEE Std. 982.2. 16. Fenton, N., Neil, N., Marsh, W., Hearty, P., Radlinski, L. and Krause, P. (2008), On
8. Zhang, X. and Pham, H. (2000), An Analysis of Factors Affecting Software the effectiveness of early life cycle defect prediction with Bayesian Nets, Empirical of
Reliability, The Journal of Systems and Software, Vol. 50, No. 1, pp. 43–56. Software Engineering, Vol. 13, pp. 499–537.


Dr. Ajeet Kumar Pandey Questions
 Qualifications: Ph.D. (Software Reliability) from IIT Kharagpur,
Kharagpur, W.B. India.
 Working as Sr. RAMS Engineer at Cognizant Technology
Solution, Hyderabad, India.
Thank you for your attention.
 Work Area: Reliability and Safety (Assessment and Prediction),
Regulatory and Compliance of Safety Critical System (Rail/
Automotive/ Avionics and Medical).
Do you have any questions?

 Email: ajeet.kumar3@cognizant.com, ajeet.mnnit@gmail.com
 Voice: (+91) 40 44518085, 888 6411889


Successive Software Reliability Growth Model: A Modular Approach

More Related Content

What's hot

Viewers also liked

Similar to Successive Software Reliability Growth Model: A Modular Approach

Recently uploaded

Successive Software Reliability Growth Model: A Modular Approach