The document discusses software defect prediction techniques in the automotive domain, emphasizing the complexity and cost associated with automotive software development. It outlines key research questions, methodologies, and significant contributions related to improving defect prediction accuracy and aiding release readiness assessments. Additionally, the document presents industrial impacts and frameworks for the adoption of machine learning-based techniques for software defect prediction.

1. Software Defect Prediction Techniques in the Automotive Domain:
Evaluation, Selection and Adoption
Rakesh Rana
Slide | 01

2. Outline:
 Part 1: Introduction
 Part 2: The Main Problem
 Part 3: Research Questions & Methodology
 Part 4: Scientific Contributions
 Part 5: Industrial Impact
 Part 6: Conclusions
Agenda
Slide | 02
Introduction
The Main
Problem
Research
Questions &
Methodology
Scientific
Contributions
Industrial
Impact
Conclusions

3. Software (SW) in Automotive Domain
“It takes dozens of microprocessors running 100 million
lines of code to get a premium car out of the driveway,
and this software is only going to get more complex”
- IEEE spectrum
Driving Factors:
o Size & Complexity: >1 GB of software distributed over
~100 ECUs.
o Cost of SW & Electronics: for today’s premium cars
o ~40% of a vehicle’s development costs are
determined by electronics and software, and
o 50 - 70% of the development costs for an ECU
(Electronic Control Unit) are related to software.
o Innovation & Trends: today, ~90% of all innovations
are driven by electronics and software.
V&V
FunctionalityMixed Blessing
Introduction
The Main
Problem
Research
Questions &
Methodology
Scientific
Contributions
Industrial
Impact
Conclusions
Slide | 03
S. Fürst, “Challenges in the design of automotive software,” in Proceedings of the Conference on Design, Automation and Test in Europe, 2010, pp. 256–258.R. N. Charette, “This car runs on code,” IEEE Spectrum, vol. 46, no. 3, p. 3, 2009.

4. Software Defect – Definition
Defect: An imperfection or deficiency in a work product where that work
product does not meet its requirements or specifications and needs to be
either repaired or replaced.
Error: A human action that produces an incorrect result.
Failure: (A) Termination of the ability of a product to perform a required
function or its inability to perform within previously specified limits Or
(B) An event in which a system or system component does not perform a
required function within specified limits.
Fault: A manifestation of an error in software.
Problem: (A) Difficulty or uncertainty experienced by one or more
persons, resulting from an unsatisfactory encounter with a system in use
or (B) a negative situation to overcome.
IEEE standard 1044, Classification for Software Anomalies
Introduction
The Main
Problem
Research
Questions &
Methodology
Scientific
Contributions
Industrial
Impact
Conclusions
Slide | 04

5. Software Defect – an example of defect description
Information type Attribute (example) Example defect description
Basic information Unique ID SWI-1234
Date & time stamp 12-Jul-201X, 16:33:04
Problem Status
Open/ Resolved/ Closed/
Cancelled
Open
Severity Major/ Minor/ Insignificant Minor
Problem Type
Requirement defect/ Design
defect/ Code defect
Code defect
Title and Description
Title Misplaced white pixels on the home screen
Description
Precondition
System is up and running.
Action
Enter home screen in default mode (all tiles in standard size).
Note! This error is only applicable first time the user enters the home screen.
Once tiles in the home screen have been expanded, the misplaced pixels
disappear.
Observation
In the middle of the home screen there are some white and grey pixels in line
with the top of the phone tile (between the phone tile and the media tile).
Expected
There should not be any colour deviation of “stains” on the background screen.
Probability
100%
Additional Information Attachments Attached Log file
Comments
Update 201X-08-26:
When entering and exiting the Settings list, the misplaced pixels reappear, even
if tiles in the home screen have previously been expanded (and shrunk back to
default tile size again).
Introduction
The Main
Problem
Research
Questions &
Methodology
Scientific
Contributions
Industrial
Impact
Conclusions
Slide | 05

6. Applying SDP Techniques in the Automotive Domain
Introduction
The Main
Problem
Research
Questions &
Methodology
Scientific
Contributions
Industrial
Impact
Conclusions
Main Objective
1. Optimize allocation of test resources:
o Predicting the shape of defect inflow profile, and
o Predicting expected pre-release defect count for an on-going project.
Help experts within the automotive domain to:
2. Assess release readiness of software before release:
Slide | 06

7. Applying SDP Techniques in the Automotive Domain
Introduction
The Main
Problem
Research
Questions &
Methodology
Scientific
Contributions
Industrial
Impact
Conclusions
The Main Problem
1. SW Development Process
o SW sourced through suppliers.
o Model-Based Development, and
2. Evaluation
Slide | 07

8. Research Questions
RQ1: Which defect prediction techniques are applicable at different stages of software life cycle in the
automotive domain?
RQ2: How software reliability growth models can be used for defect predictions in the automotive domain?
RQ3: What are the consequences of mispredicting total number of defects and release readiness?
RQ4: How can correlation based models be used for defect prediction in the automotive domain?
RQ5: How to evaluate reliability characteristics of software at early stages of development using only
behavioural models?
RQ6: What are important factors for industrial adoption of machine learning based defect prediction models?
To evaluate how software defect prediction techniques can be effectively applied
over the software development life cycle within the automotive domain.
Introduction
The Main
Problem
Research
Questions &
Methodology
Scientific
Contributions
Industrial
Impact
Conclusions
Slide | 08

9. Descriptive
Analytical
Fundamental
Applied
Quantitative
Qualitative
Conceptual
Empirical
Descriptive-
Reviewof
literature
Descriptive
other
Evaluative-
deductive
Evaluative-
interpretive
Formulative-
model
Formulative-
framework
Formulative-
guidelines
Conceptual
analysis
Literature
review/analy
Casestudy
Action
research
Dataanalysis
Simulation
Fieldstudy
Paper 1
Paper 2
Paper 3
Paper 4
Paper 5
Paper 6
Paper 7
Paper 8
Paper 9
Paper 10
Paper
No
Research approachResearch type Research methods
Research Methodology
Introduction
The Main
Problem
Research
Questions &
Methodology
Scientific
Contributions
Industrial
Impact
Conclusions
Slide | 09
Research approach and methods are classification based on: R. L. Glass, I. Vessey, and V. Ramesh, “Research in software engineering: an analysis of the
literature,” Information and Software technology, vol. 44, no. 8, pp. 491–506, 2002.
Research type classification is based on: C. R. Kothari, Research Methodology: Methods and Techniques. New Age International, 2011.

10. RQ1: Which defect prediction techniques are applicable at different stages of
software life cycle in the automotive domain?
Introduction
The Main
Problem
Research
Questions &
Methodology
Scientific
Contributions
Industrial
Impact
Conclusions
Main contributions:
o Overview of automotive SW development life cycle for large platform projects,
o Seven different SDP techniques were mapped according to the development phase they can
be applied,
o Mapped SDP techniques were also classified based on the purpose and application level, and
o Roadmap for increasing the efficiency of defect predictions by using field data.
Production Software
Slide | 10

11. RQ2: How software reliability growth models can be used for defect
predictions in the automotive domain?
Evaluation:
o Two commonly used parameter estimation procedure (MLE, and NLR) were compared,
o Eight SRGMs were evaluated at function level (best: logistic Model),
o Eight SRGMs were further evaluated at Project/Release level using defect data from three
different companies from embedded software domain (best: logistic & Gompertz Model),
o How to use historical projects data to make defect count prediction more accurate,
Selection:
o Proposed method for predicting shape of defect inflow profile, and
o The statistical distribution family of defect inflow data was also explored where beta
distribution was shown to be prominent family among tested distributions.
Introduction
The Main
Problem
Research
Questions &
Methodology
Scientific
Contributions
Industrial
Impact
Conclusions
Slide | 11

12. Defect inflow intensity until half-way through the project Predicted shape of
defect inflow profileOverall trend Trend after reaching local maximum
Increasing Decreasing S-shape
Decreasing Decreasing Concave
Increasing Increasing Convex
ConcaveS-shape Convex
Empirical Evaluation at Project/Release level
Introduction
The Problem
Research
Questions &
Methodology
Scientific
Contributions
Industrial
Impact
Conclusions
RQ2: How software reliability growth models can be used for defect
predictions in the automotive domain?
Slide | 12

13. Predicted
shape of
defect inflow
profile
Recommended SRGMs
For testing
resource(s)
allocation
For release readiness
assessment using
current project data
S-shape Logistic Logistic
Concave Gompertz Gompertz
Convex Delayed-S Logistic
Recommendation on applying SRGMs for SDP in industry:
o Collect the right data from testing,
o Perform exploratory analysis of data,
o Define goals for applying SRGMs,
o Select appropriate model based on predicted shape/distribution, and
o Use historical information when available
Introduction
The Problem
Research
Questions &
Methodology
Scientific
Contributions
Industrial
Impact
Conclusions
RQ2: How software reliability growth models can be used for defect
predictions in the automotive domain?
Slide | 13

14. Introduction
The Problem
Research
Questions &
Methodology
Scientific
Contributions
Industrial
Impact
Conclusions
RQ3: What are the consequences of mispredicting total number of defects and
release readiness?
Consequences of mispredictions:
o Two axis of accuracy of prediction were explicitly identified –
1) the prediction of the asymptote or the total number of defects, and
2) when the total number of defects are discovered or the release readiness.
o Consequences of mispredictions the shape of defect inflow profile were examined, and
o The consequence model was validated at two companies which provided insights into
which mispredictions are critical in industry and how these are managed.
Slide | 14

15. Introduction
The Main
Problem
Research
Questions &
Methodology
Scientific
Contributions
Industrial
Impact
Conclusions
Empirical Evaluation at Function (Feature) and ECU level
o Confirmed the Pareto principle of defect distribution (20-60 rule) for automotive
software projects,
o Correlation between defects count in consecutive integration points - not found to be
strong, and
o Evidence for strong correlation between number of defects found at IP4/IP5 with total
pre-release defect count.
RQ4: How can correlation based models be used for defect prediction in the
automotive domain?
Slide | 15

16. Introduction
The Main
Problem
Research
Questions &
Methodology
Scientific
Contributions
Industrial
Impact
Conclusions
RQ5: How to evaluate reliability characteristics of software at early stages of
development using only behavioural models?
Evaluation at Model/Code level
o Behavioural models in MBD offers significant opportunity to do reliability analysis early
in the development process,
o Proposed a framework for early identification of design flaws and evaluating the efficacy
of test suite to detect potential implementation defects that can potentially violate the
safety goals, and
o The framework was validation to provide a proof-of-concept - 2 out of 24 mutants
produced unexpected results and exposed the deficiency of the current test protocol.
Slide | 16

17. Introduction
The Main
Problem
Research
Questions &
Methodology
Scientific
Contributions
Industrial
Impact
Conclusions
RQ6: What are important factors for industrial adoption of machine learning
based defect prediction
Adoption:
o Proposed a framework for adoption of machine learning based techniques in industry
and its adaptation for software defect prediction,
o Identified nine main factors and various sub-attributes that affect the decisions of
adoption of new techniques for SDP in the industry, and
o The framework was validation at two companies and specific challenges for adoption
were identified.
Slide | 17

18. Software Reliability Growth Models (SRGMs)
Analogy Based Predictions
Correlation Based Models
Reliability analysis of Simulink Models
Introduction
The Main
Problem
Research
Questions &
Methodology
Scientific
Contributions
Industrial
Impact
Conclusions
Supporting experts with data based model
o Applying different SDP models based on testing data provides empirical evidence and
predictions that support experts in making their own forecasts,
o Data based models brings more objectivity in the predictions thus helping experts to
potentially avoid some biases.
Slide | 18

19. Func Increase Calib/Optimization
SRGMs
Correlation
Simulink Model Analysis
ML
Introduction
The Main
Problem
Research
Questions &
Methodology
Scientific
Contributions
Industrial
Impact
Conclusions
Efficient resource planning (cost/release timing)
o Automotive SW development is characterised by high time pressure/constraints, and
o Large part of verification & validation at vehicle level is done in late stages of
development life cycle, thus:
o Reliability analysis early in development life-cycle can help in developing robust
SW, and
o SDP about mid-way can provide useful estimates for efficient resource planning of
testing resources.
Slide | 19

20. Introduction
The Main
Problem
Research
Questions &
Methodology
Scientific
Contributions
Industrial
Impact
Conclusions
Conclusions
Slide | 20
Software Defect Prediction Techniques in the Automotive Domain:
Evaluation:
1. SRGMs
 To support test resource allocation decisions,
 For assessment of release readiness, and
 Using historical data to make better predictions.
2. Correlation based models
 For predicting defect count in next integration point, and
 For pre-release defect count predictions.
3. Behavioural models
 For early reliability analysis.
Selection:
1. How appropriate SRGMs could be selected by predicting the shape of defect inflow, and
2. Using defect inflow distribution family to select appropriate SRGMs.
Adoption:
1. Explored factors that play important role for adoption (of ML based SDP techniques),
2. Adoption framework is expected to help with wider adoption of such techniques.