ICST 2020

1. Can we predict the quality of
spectrum-based fault
localization?
Mojdeh Golagha
fortiss
Lionel C. Briand
University of Luxembourg, University of Ottawa
Alexander Pretschner
Technical University of Munich

2. Problem
Spectrum-based Fault localization
25.10.2020
Why is the effectiveness of spectrum based fault localization techniques so unpredictable?
2
Source code Test Suite
Block Test 1 Test 2 Test 3 Tarantula Ochiai DStar
1 ∞ ∞
2 ∞ ∞ ∞
3 ∞
„ Not widely applied in practice yet.
„ Effectiveness varies greatly from case to case.
„ New algorithms and ideas as well as adjustments to the
test suites to improve effectiveness.
„ Why is the effectiveness of these techniques so
unpredictable?
„ What are the factors that influence the effectiveness of
fault localization?
„ Can we accurately predict fault localization effectiveness?

3. Solution
25.10.2020
Based on precise hypotheses, we define metrics and assess their influence on fault localization
effectiveness.
3
I. Define metrics.
II. Generate a data set.
III. Apply classification analysis to assess the influence of metrics.
IV. Build the most accurate prediction model for effectiveness.

4. Define Metrics

5. Metrics
25.10.2020
Metrics capture static aspects of the source code, dynamic aspects of test executions, properties of the
test suite, and fault properties.
5

6. Generate Data Set

7. Data Set
25.10.2020
Our final data set has 341 instances and 70 variables. The “effective” label has been assigned to
193 of these instances.
7
Each observation is a faulty version extracted from
Defects4J
Method-level SBFL.
• Rank of the faulty
method
• 1 to 10 -> “effective”
• Otherwise “ineffective”
Variables Class Labels

8. Data Analysis

9. Analysis of Significance
25.10.2020
To assess the significance and impact of each metric individually.
9
Pre-analysis: Pearson’s Correlation to remove correlated metrics
I. Odds ratio analysis in univariate logistic regression
II. Average information gain

10. Multivariate Analysis
25.10.2020
To generate prediction models. One model for each group and one model for all groups combined.
10
I. Tree
II. Logistic Regression
III. SVM
IV. Random Forest
V. Adaboost

11. Multivariate Analysis
25.10.2020
Considering the StatDynaTest metrics, random forest and logistic regression yield the best, almost
identical, results.
11
Logistic Regression Model Random Forest Model

12. Multivariate Analysis
25.10.2020
There are eight metrics in common among them:
Static - % Methods with Nesting Depth>5, Mean # of Fields per Type,
Dynamic - Mean Node Degree, Max Node Out Degree, Graph Diameter, Response for Class,
Test - % Method Coverage, % Methods Covered in Failing Tests.
12

13. Results
25.10.202013
• A combination of only a few static, dynamic, and test metrics enables the construction of a prediction model with
excellent discrimination power between levels of effectiveness:
- eight metrics yielding an AUC of .86 (Logistic Regression on Selected-SDT metrics)
- fifteen metrics yielding an AUC of .88 (Random Forest on StatDynaTest metrics)
• A confidence factor that can be used to assess the potential effectiveness of fault localization.
• The most influential metrics can also be used as a guide for corrective actions on code and test suites leading to
more effective fault localization.
• The effectiveness of fault localization depends more on the complexity of the code and test suite than on the fault
type and location.
• Selected dynamic metrics measure the degree to which the call graph is entangled. If the dynamic call graph of
the tests is highly entangled and coupled, it is difficult to localize a fault, no matter where it happens.

14. ©2020
Mojdeh Golagha
golagha@fortiss.org