This document discusses predicting software reliability using software reliability growth models (SRGMs). It aims to determine the best SRGMs for optimal testing resource allocation and assessing release readiness. A multiple case study of large industrial projects is conducted. The research questions examine which SRGMs are best for resource allocation and release readiness, and whether using historical data improves assessments. Metrics like mean square error and balanced predicted relative error are used for evaluation. Results show recommended SRGMs depend on the software development process, observed defect profile shape, and whether historical data is used. The document also discusses characteristics that can help determine the predicted defect profile shape and most effective SRGMs.

1. Predicting reliability of software systems
under development
A multiple case study of large industrial embedded software projects

2. Support organizations in making decisions with respect to:
–Optimal allocation of test resources
–Asses the release readiness of software under
development
SRGMs: Software Reliability Growth Models
Objectives
Image: http://flylib.com/books/1/428/1/html/2/files/10fig07.gif

3. • RQ1: Which SRGMs are best to assist decisions for
optimal allocation of testing resources?
• RQ2: Which SRGMs are best for assessing the release
readiness of a software system?
• RQ3: Does using information from earlier projects
improve release readiness assessment?
• RQ4: How to make the choice of SRGM more
effective?
Research Questions

4. CASE STUDY DESIGN
Company
(unit of analysis)
Application
domain
Software development process for
studied projects
Volvo Cars
Corporation
Automotive
V-shaped software development mostly
using sub-suppliers for implementation
Ericsson Telecom Agile development, mostly in-house
SAAB EDS Defense Equipment
Waterfall development (old projects)
with development concentrated in-house

5. Software Development Process

6. SRGMs: Software Reliability Growth Models
No Model Name Shape Structure Mean Value Function Ref.
1 Musa-Okumoto (MO) Concave NHPP 𝑚 𝑡 = 𝑎 ln(1 + 𝑏𝑡) [28]
2 Goel-Okumoto (GO) Concave NHPP 𝑚 𝑡 = 𝑎 (1 − 𝑒−𝑏𝑡
) [29]
3 Inflection-S model S-shaped NHPP 𝑚 𝑡 =
𝑎 (1 − 𝑒−𝑏𝑡
)
(1 + 𝛽𝑒−𝑏𝑡)
[30]
4 Delayed-S model S-shaped NHPP 𝑚 𝑡 = 𝑎 (1 − 1 + 𝑏𝑡 𝑒−𝑏𝑡
) [31]
5 Rayleigh model S-shaped NHPP 𝑚 𝑡 = 𝑎 (1 − 𝑒
−
𝑡
𝑏
2
)
[32]
6 Logistic model S-shaped Trend 𝑚 𝑡 =
𝑎
(1 + 𝑒−𝑏(𝑡−𝑐))
[33]
7 Gompertz model S-shaped Trend 𝑚 𝑡 = 𝑎 𝑒−𝑏𝑒−𝑐𝑡
[34]
8 Linear model Linear Trend 𝑚 𝑡 = 𝑔 ∗ 𝑡 + 𝑐 [27]

7. Metrics used for evaluation
𝑀𝑆𝐸 =
1
𝑛
1
𝑛
𝑌𝑖 − 𝑌𝑖
2
𝐵𝑃𝑅𝐸 =
𝑃𝑟𝑒𝑑𝑖𝑐𝑡𝑒𝑑 − 𝐴𝑐𝑡𝑢𝑎𝑙
𝑃𝑟𝑒𝑑𝑖𝑐𝑡𝑒𝑑 + 2𝜂 ∗ (𝐴𝑐𝑡𝑢𝑎𝑙 − 𝑃𝑟𝑒𝑑𝑖𝑐𝑡𝑒𝑑)
;
Mean Square Error (MSE)
Balanced Predicted Relative Error (BPRE)
𝑊ℎ𝑒𝑟𝑒, 𝜂 =
0 𝑖𝑓 𝑃𝑟𝑒𝑑𝑖𝑐𝑡𝑒𝑑 > 𝐴𝑐𝑡𝑢𝑎𝑙
1 𝑖𝑓 𝑃𝑟𝑒𝑑𝑖𝑐𝑡𝑒𝑑 < 𝐴𝑐𝑡𝑢𝑎𝑙
Image: http://flylib.com/books/1/428/1/html/2/files/10fig07.gif

8. a) Which SRGMs are best to assist decisions
for optimal allocation of testing resources?
Data

9. Metrics used for evaluation
Mean Square Error (MSE) Balanced Predicted Relative Error (BPRE)
Image: http://flylib.com/books/1/428/1/html/2/files/10fig07.gif
RQ1: Which SRGMs are best to assist decisions for optimal allocation of testing resources?
RQ2: Which SRGMs are best for assessing the release readiness of a software system?
RQ3: Does using information from earlier projects improve release readiness assessment?

10. a) Which SRGMs are best to assist decisions
for optimal allocation of testing resources?

11. b) Which SRGMs are best for assessing the
release readiness of a software system?

12. c) Does using information from earlier projects
improve release readiness assessment?

13. Summary of results
Case unit (domain)
Software
development
process
Observed shape
of defect inflow
profile
Recommended SRGMs
For testing
resource(s)
allocation
For release readiness assessment
Only using current
project data
Using historic
information
1. Automotive V-model S-shape, Concave Logistic Logistic Logistic
2. Telecom Lean + Agile Concave, Convex Gompertz Logistic Musa-Okumoto
3. Defense Equip Waterfall S-shape, Concave Logistic Gompertz Logistic
RQ4: How to make the choice of SRGM more effective?

14. How to make the choice of SRGM more effective?

15. How to make the choice of SRGM more effective?

16. How to make the choice of SRGM more effective?
Projects/
Releases
Defect inflow intensity trend until half-way through the project Predicted
shape of
defect inflow
profile
Overall
trend
Trend after
reaching
maximum
Defect inflow intensity trend characteristics
A1, A3, A4
& C1
Increasing Decreasing
Defect inflow intensity first increases, maximizes near
to half-way and then decreases
S-shape
B1, B3 &
B4
Decreasing Decreasing
Early defects, defect inflow intensity maximum early
then decreases smoothly
Convex
A2, B2, B5
& C2
Increasing Increasing
Late defects, defect inflow intensity trend is positive
throughout half-way of project timeline
Concave

17. How to make the choice of SRGM more effective?
Predicted
shape of
defect inflow
profile
Recommended SRGMs
For testing resource(s)
allocation
For release readiness
assessment using
current project data
S-shape Logistic Logistic
Convex Gompertz Gompertz
Concave Delayed-S Logistic

18. Thank You