Defect Prediction & Prevention In Automotive Software Development Dec, 2013 Göteborg, Sweden Get full text of publication at: http://rakeshrana.website/index.php/work/publications/

1. Defect Prediction & Prevention In
Automotive Software Development
11-Dec-2013
Rakesh Rana

2. Agenda
• Introduction: Why do we need to focus on reliability?
• Research Question
• Structure of thesis
• Reliability growth models – theory
• Defect Prediction: Chapters 2 – 4
• Defect Prevention: Chapters 5 – 6
• Conclusions
• Future research directions

3. Road fatalities in the EU since 2001
• 2011 more than 30,000 people died on the roads of the EU (equivalent of a medium town)
• For every death on Europe's roads there are an estimated 4 permanently disabling injuries
such as damage to the brain or spinal cord, 8 serious injuries and 50 minor injuries.
Source: EU Commission, Mobility and Transport, Road Safety; http://ec.europa.eu/transport/road_safety/specialist/statistics/

4. Cars: Safety Goal
“Our aim for 2020 is that no one
should be killed or seriously
injured in a Volvo”
- Thomas Broberg, Volvo’s senior safety
adviser, 2009
Source: The Volvo S60 concept, The New York Times; http://wheels.blogs.nytimes.com/2009/10/14/volvo-sets-a-lofty-safety-goal/?_r=0
http://www.industryweek.com/product-development/volvo-eyes-no-death-goal-its-new-cars-2020
"The car of the future will be just like the farmer's horse. The
farmer can steer the horse and carriage but if he falls asleep
the horse can still (get) back home. And if the farmer tries to
steer the carriage against a tree or off a cliff, the horse will
refuse"
- Anders Eugensson, Volvo's head of government affairs, Dec 2012 to Wall Street Journal

5. Increasing role of Software in Cars
 First automotive ECU -- single-function
controller -- 1977 -- GM.
 By 1981, GM was using ~50000 lines of
code across its entire domestic passenger
car production.
 Today premium-class automobiles contains
~100 m lines of software code running on
70-100 microprocessors based ECUs.
Source: http://www2.teknat.uu.se/forskning/program.php?vetenskapsid=1&hforskomr=6&id=39&lang=en
http://spectrum.ieee.org/green-tech/advanced-cars/this-car-runs-on-code
“It takes dozens of microprocessors
running 100 million lines of code to get
a premium car out of the driveway, and
this software is only going to get more
complex”
- Robert N. Charette, 2009 in IEEE Spectrum

6. Cars and Software: Challenges
 Cost of software and electronics can reach ~35 to 40% of the cost of a car.
Complexity also brings with it reliability issues.
In 2005, Toyota voluntarily recalled 160 000 -- Prius hybrids -- software problem. Time
needed to repair the software ~90 minutes per vehicle!
May 2008, Chrysler recalled 24 535 -- Jeep Commanders -- problem in the automatic-
transmission software.
June 2008, Volkswagen recalled about 4000 Passats and about 2500 Tiguans --
problem in the engine-control-module software.
Nov 2008, GM recalled 12 662 -- Cadillac CTS -- software problem.
Problems with repair and warranty
More than 50% of the ECUs that mechanics replace in cars are technically error free:
They exhibit neither a hardware nor a software problem.
50% of warranty costs (IBM).
Source: http://spectrum.ieee.org/green-tech/advanced-cars/this-car-runs-on-code

7. With so much more software in cars and its increasing
complexity, how do we control the reliability issue?
Cars and Software: The Big Question

8. Reliability
Source: A. Avizienis, J. C. Laprie, and B. Randell, “Fundamental concepts of dependability,” Tech. Rep. Ser.-Univ. Newctle. Tyne Comput. Sci., 2001.
IEEE standard 1633, recommended practice on software
reliability definition for software reliability:
(A) The probability that software will not cause the failure of
a system for a specfied time under specified conditions.
(B) The ability of a program to perform a required function
under stated conditions for a stated period of time.

9. Reliability
Source: A. Avizienis, J. C. Laprie, and B. Randell, “Fundamental concepts of dependability,” Tech. Rep. Ser.-Univ. Newctle. Tyne Comput. Sci., 2001.
Dependability
Attributes
Safety Reliability

10. Reliability
Source: A. Avizienis, J. C. Laprie, and B. Randell, “Fundamental concepts of dependability,” Tech. Rep. Ser.-Univ. Newctle. Tyne Comput. Sci., 2001.
Threats
Reliability
Means
Faults
Errors
Failures
Fault Prevention
Fault Tolerance
Fault Removal
Fault Forecasting

11. Research Focus & Questions
(Addressed in chapters 2, 3 & 4) (Addressed in chapters 5 & 6)
Thesis
RG1. Evaluating the applicability of
Software Reliability Growth Models
(SRGMs) in the context of automotive
software development?
RG2. Propose and evaluate methods that
can potentially increase the reliability of
software in the automotive domain
RQ1. Do SRGMs fit
defect inflow data
from automotive
domain?
RQ2. Differences
between the widely
used parameter
estimation methods?
RQ3. Which
SRGMs have the
best long-term
predictive power?
RQ4. How to use
fault injection &
mutation testing
@models?
RQ3. How to test
models better in
simulating
environment?
Prediction Prevention

12. Chapters 2 – 4: Defect prediction
(IEEE standard 1044)
o defect: An imperfection or deficiency in a work product where that
work product does not meet its requirements or specifications and
needs to be either repaired or replaced.

13. Software reliability growth models: Theory
Source: Wood, Alan. "Software reliability growth models." Tandem Technical Report 96 (1996).
Concave Models
• Assumption
• Examples: Exponential Model,
Goel-Okumoto; Musa-Okumoto
S-Shaped Models
• Assumption
• Examples: InflectionS, DelayedS,
gompertz, logistic model.

14. Chapter 2: Evaluation of standard reliability growth models
in the context of automotive software systems*
*Proceedings of 14th Product-Focused Software Process Improvement (PROFES) 2013, Paphos, Cyprus
RQ1. Do SRGMs fit
defect inflow data from
automotive domain?
• Objective: Do widely used software reliability growth models fit to defect
inflow data from the automotive domain?
• Method: Analytical study, we evaluated eight commonly used software
reliability growth models on defect inflow data from the automotive
domain (large project on active safety function).
• Results: While three parameter models provide good fit to the defect
data, better results can be obtained by accounting for changes in the
testing effort over calendar time.

15. Chapter 2: Evaluation of standard reliability growth models
in the context of automotive software systems*
*Proceedings of 14th Product-Focused Software Process Improvement (PROFES) 2013, Paphos, Cyprus
RQ1. Do SRGMs fit
defect inflow data from
automotive domain?
Where ai is actual values, pi predicted values of total number
of defects for data set of size k and q is the number of
parameters of software reliability growth model equation.
𝑴𝑺𝑬 =
𝟏
𝒌
(𝒂𝒊 − 𝒑𝒊)
𝒌 − 𝒑

16. Chapter 3: Comparing between Maximum Likelihood
Estimator and Non-Linear Regression estimation
procedures for Software Reliability Growth Modelling*
*Proceedings of 23nd International Workshop on Software Measurement, IWSM-Mensura 2013, Turkey.
RQ2. Differences between
the widely used parameter
estimation methods?
• Objective: To explore applicability and practical considerations for
applying two widely recommended and used parameter estimation
methods.
– Maximum likelihood estimation
– Non-linear regression estimation
• Method: Analytical study comparing the parameter estimates obtained
from these two methods for same data set and also comparing them to
results obtained via empirical equations and those reported in earlier
study.
• Results: While MLE is the recommended estimator with superior
statistical properties, its usability and applicability in all situations is
questionable. We further provide an improvised metric (BPRE) for
comparing the predictive accuracy.

17. BPRE, Balanced Predicted Relative Error
Metric Relative Error
(RE)
Predicted Relative Error
(PRE)
Balanced Predicted Relative
Error (BPRE)
Formula 𝑹𝑬 =
𝑷 − 𝑨
𝑨
𝑷𝑹𝑬 =
𝑷 − 𝑨
𝑷
𝑩𝑷𝑹𝑬 =
𝑷 − 𝑨
𝜼𝑷 + 𝟏 − 𝜼 (𝟐𝑨 − 𝑷)
Where η=1 if P > A, 0 otherwise
𝑹𝒂𝒏𝒈𝒆: (−∞, ∞) [𝟎, 𝟏) [𝟎, 𝟏)
Over
Prediction
+20% +16.67% +16.67%
Under
Prediction
-20% -25.0% -16.67%

18. Chapter 3: Comparing between Maximum Likelihood
Estimator and Non-Linear Regression estimation
procedures for Software Reliability Growth Modelling*
*Proceedings of 23nd International Workshop on Software Measurement, IWSM-Mensura 2013, Turkey.
RQ2. Differences between
the widely used parameter
estimation methods?
𝑩𝑷𝑹𝑬 =
𝑷𝒓𝒆𝒅𝒊𝒄𝒕𝒆𝒅 − 𝑨𝒄𝒕𝒖𝒂𝒍
𝜼 ∗ 𝑷𝒓𝒆𝒅𝒊𝒄𝒕𝒆𝒅 + 𝟏 − 𝜼 ∗ (𝟐 ∗ 𝑨𝒄𝒕𝒖𝒂𝒍 − 𝑷𝒓𝒆𝒅𝒊𝒄𝒕𝒆𝒅)
; 𝒘𝒉𝒆𝒓𝒆 𝜼 =
𝟏 𝒊𝒇 𝑷𝒓𝒆𝒅𝒊𝒄𝒕𝒆𝒅 > 𝑨𝒄𝒕𝒖𝒂𝒍
𝟎 𝒊𝒇 𝑷𝒓𝒆𝒅𝒊𝒄𝒕𝒆𝒅 < 𝑨𝒄𝒕𝒖𝒂𝒍

19. Chapter 4: Evaluating long-term predictive power of
standard reliability growth models on automotive systems*
*Proceedings of the 24th IEEE International Symposium on Software Reliability Engineering, Pasadena,2013
RQ3. Which SRGMs have the
best long-term predictive
power?
• Objective:
– Which SRGMs fit best to the defect data from automotive software projects?
– Which SRGMs have the best long-term predictive power?
– Which models growth rates are consistent between projects over time?
• Method: Analytical study comparing seven widely used SRGMs on full & partial
defect inflow data from four large automotive software projects.

20. Chapter 4: Evaluating long-term predictive power of
standard reliability growth models on automotive systems*
*Proceedings of the 24th IEEE International Symposium on Software Reliability Engineering, Pasadena,2013
RQ3. Which SRGMs have the
best long-term predictive
power?
• Results:
– Which SRGMs fit best to the defect data from automotive software projects?
– Which SRGMs have the best long term predictive power?
– Which models growth rates are consistent between projects over time?
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Musa-Okumoto Goel-Okumoto Inflection-S Delayed-S Rayleigh Gompertz Logistic
BPRE for different modelsusinggrowth ratefrom project A (100% data)
PRE-100%
PRE-90%
PRE-70%
PRE-50%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Musa-Okumoto Goel-Okumoto Inflection-S Delayed-S Rayleigh Gompertz Logistic
BPRE values(averageacrossprojects) for SRGMsusingfull & partial data
PRE-100%
PRE-90%
PRE-70%
PRE-50%
BPRE ~100% --
too bad prediction
BPRE ~0% -- predicted total
#Defects ~Actual total #Defects
Same SRGM, using full and
partial data (forecasting)

21. Research Focus & Questions
(Addressed in chapters 5 & 6)
Thesis
RG2. Propose and evaluate methods that
can potentially increase the reliability of
software in the automotive domain
Prevention
Models SRGMs
Models Functional Models
w.r.t. MBD
Prediction

22. Chapter 5: Increasing Efficiency of ISO-26262 Verification and
Validation by Combining Fault Injection and Mutation Testing
with Model Based Development*
*8th International Joint Conference on Software Technologies - ICSOFT-EA, Reykjavik, Iceland, July 2013
RQ4. How to use
fault injection &
mutation testing
@models?
• Objective:
– How models can be used more effectively for early verification and validation?
• Method: Descriptive-Qualitative case study based on empirical observations to
propose a framework which combines methods of fault injection and mutation
testing to be used at the model level that can be used for increasing the efficiency
of ISO-26262 compliance.

23. Chapter 5: Increasing Efficiency of ISO-26262 Verification and
Validation by Combining Fault Injection and Mutation Testing
with Model Based Development*
*8th International Joint Conference on Software Technologies - ICSOFT-EA, Reykjavik, Iceland, 2013
RQ4. How to use
fault injection &
mutation testing
@models?
a) Assign TSRs corresponding
to FSRs to Z-outputs
b) Inject Faults (simulating
common defects) to X-inputs
c) Identify critical fault scenarios;
Study fault propagation properties;
Build fault tolerance
e) Repeat Steps (b) & (c) to
test, correct & validate
the function for its “d”
dependencies
d) Cause Mutation to “n” blocks of
function & assess effectiveness of
given test suit using M.Testing
f) Examine Mutation not killed; Update
test cases or build new to detect such
failure scenarios/defects

24. Chapter 6: Improving Fault Injection in Automotive Model
Based Development using Fault Bypass Modelling*
*2nd Workshop on Software-Based Methods for Robust Embedded Systems, INFORMATIK, Germany,2 013
RQ3. How to test models better
in simulating environment?
Environment Model
SW system Model
Out_1
Output
Inp_2
Inp_1
Out_2
Natural/State
parameter(s)
• Objective:
– How simulations of functional models be used effectively for early verification
and validation?
• Method: Descriptive-Qualitative case study based on experiment, we propose
and provide proof-of-concept for “fault bypass modelling”, a simple yet effective
framework for correct analysis of simulation in closed loop mode.

25. Chapter 6: Improving Fault Injection in Automotive Model
Based Development using Fault Bypass Modelling*
*2nd Workshop on Software-Based Methods for Robust Embedded Systems, INFORMATIK, Germany,2 013
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
0
10
20
30
40
50
60
Time in sec
SpeedinRPM
Vehicle and wheel speed with fault injection (FBM)
Vehicle Speed
Wheel Speed
0 1 2 3 4 5 6 7 8 9 10
0
20
40
60
80
100
120
140
Time in sec
SpeedinRPM
Vehicle and wheel speed with fault injection
Vehicle Speed
Wheel Speed
RQ3. How to test models better
in simulating environment?
Vehicle Speed
Control Signal
Wheel Speed
Vehicle Speed
Wheel Speed
Relative Slip
ABS Model
Environment Model

26. Conclusions
RG1. Evaluating the applicability of software reliability growth models
in the context of automotive software development?
1. SRGMs are able to fit the defect inflow data from automotive domain.
2. MLE Vs. NLR
– MLE has superior statistical properties.
– MLE is not applicable in all cases.
– A new metric for predictive accuracy is introduced.
3. Logistic and Gompertz model provide best fit among widely used
SRGMs, these models also provide best long-term predictive power. The
results also indicates that information (in form of growth rates) can be used
to improve the predictive power of most SRGMs.

27. Conclusions
RG2. Propose and evaluate methods that can potentially increase
the reliability of software in the automotive domain.
4. A framework to combine fault injection and mutation testing approach
applied at behavioural models is introduced.
5. Fault Bypass Modelling is introduced that helps to develop robust
software, a proof-of-concept is provided.

28. Conclusions
Why predict and prevent software defects in automotive domain?
• Predicting defects inflow help us manage defects and testing resources
effectively.
• Effective defect management and defect prevention increases reliability
of software in cars and thus cars themselves. It also translates to lower
development costs and time to market.
• Consumers get cars that are cheaper, safe and reliable.
• Lower ownership costs and it allows adding more innovations to your
cars with the help of software.

29. Future Research Directions
• Exploratory analysis of defect inflow data distribution from
industrial software projects
• Evaluating SRGMs and their long-term predictive power for
embedded software projects from industry
• Explore Machine Learning to Predicts Defects and Analysing Risks
in Large Software Development Projects
• SRGMs based on functional/behavioural models of software
• Measuring impact of software quality and/or software reliability
assessment on software development projects

30. Thank You

31. Back Up Slides
Source: Healing with Art, community on Facebook, https://www.facebook.com/photo.php?fbid=10151903164088141&set=a.378605758140.163024.14524668140&type=1&theater