This document summarizes a study on reducing failure analysis time for software testing in the automotive industry. The proposed approach clusters failing tests based on their coverage profiles and hypothesized faults, then selects representative tests from each cluster for analysts to examine. It was evaluated on 9 automotive components totaling 850 KLoC and over 7,000 tests, reducing failure analysis time by over 80% on average while maintaining high purity and accuracy in fault localization. The approach utilizes coverage profiles from software-in-the-loop testing to aid hardware-in-the-loop analysis and could help address challenges of failure analysis for automotive cyber-physical systems.

1. Reducing Failure Analysis Time:
An Industrial Evaluation
26.05.2017
Mojdeh Golagha, Alexander Pretschner
Technical University of Munich
Department of Informatics
LS XXII – Chair for Software Engineering
Dominik Fisch, Roman Nagy
BMW
Research and Development

2. /15
Motivation
Problem
• Big number of SiL/HiL
tests
• A few faults cause a lot
of failures
• Significantly high test
analysis time
• How can we reduce the
failure analysis time?
Solution Idea
• Cluster failing tests with
respect to the
hypothesized faults
• Select representative
test(s) for each cluster
• Analyze only the
representatives
2

3. /15
Approach
Running tests and collecting coverage information
To generate features for clustering
3
Generating failure tree
To cluster failing executions
Cutting the failure tree by Fault Localization *
To decide the best number of clusters (the number of hypothesized faults)
Selecting representative tests
To start the debugging process
1
STEP
2
STEP
3
STEP
4
STEP
* Based on James A. Jones, James F. Bowring, and Mary Jean Harrold. 2007. Debugging in Parallel. ISSTA '07
Utilizing SiL coverage profiles for HiL analysis
5
STEP

4. /15
Step 1: Running Tests and Collecting Coverage
Information
Blocks
TC
1
TC
2
TC
3
TC
4
TC
5
TC
6
TC
7
TC
8
IsOdd() 0 0 1 1 0 1 1 0
GetMainCppPath() 0 1 1 1 1 1 0 1
GetOutputBinaryPath() 0 1 0 1 1 0 0 0
4
*
Spectralizer
• Prepare test execution
• Execute tests while recording the coverage information using
OpenCPPCoverage *
• Generate coverage profile at desired level of granularity (Function)

5. /15
Step 2: Generating Hierarchical Tree of Failing Test
cases (FTC)
5
FTC1
FTC2
FTC3
FTC4
FTC5 FTC6

6. /15
Step 3: Cutting the Failure Tree by Fault Localization
66
FTC1
FTC2
FTC3
FTC4
FTC5 FTC6

7. /15
Spectrum-based Fault Localization
Function TC1 TC2 TC3 TC4 TC5 TC6 TC7 TC8
Fun1 ● ● ● ● ● ● ● ●
Fun2 ● ● ● ● ● ● ● ●
Fun3 ● ● ● ● ● ● ● ●
Fun4 ● ● ● ● ●
Fun5 ● ● ● ● ● ● ● ●
Fun6 ● ● ● ●
Fun7 ● ● ● ● ● ● ● ●
Result fail fail fail fail fail fail pass pass
7
Function Suspiciousness Rank
Fun4 125 1
Fun1 108 2
Fun2 108 2
Fun3 108 2
Fun5 108 2
Fun7 108 2
Fun6 1.33 3
𝐷∗ =
(𝑁𝐶𝐹)∗
𝑁 𝑈𝐹 + 𝑁𝐶𝑆
NCF: # failing, covered
NUF: # failing, not covered
NCS: # passing, covered

8. /15
Step 3: Cutting the Failure Tree by Fault Localization
88
FTC1
FTC2
FTC5
FTC3
FTC4
FTC6
Suspicious functions:
[{Fun1}
{Fun18}
{Fun7}
{Fun2}
{Fun3}
…]
Suspicious functions:
[{Fun1}
{Fun18}
{Fun5}
{Fun2}
{Fun4}
…]
FTC3
FTC4
FTC6
FTC1
FTC2
FTC5

9. /15
Step 4: Selecting Representative Tests
• Clusters are unlikely to be
100 percent pure
• Calculate the center of cluster
• Find the K-nearest neighbors
to the center
• K=1, Analyze 2 failing tests
instead of 6
999
FTC1
FTC2
FTC3
FTC4
Ftc5 FTC6

10. /15
Training Phase
• Clustering Metric and
Method Selection
• 42 different combination
UPGMA and Euclidean
• Fault localization
similarity metric
• 72 metrics
DStar
• 25 components
• 100 first-order mutated
versions
10
Mutations
Negate condition in “for” or “while” statement
Replace integer constant by another integer
Delete a statement
Replace an operator by another operator
Assign null in assignment statements
Replace return expression with null

11. /15
Evaluation
• 9 different components, ca. 850 KLoC
11
Lines of Code (KLoC) Number of TCs
SWC1-Old1 146 1103
SWC1-Old2 146 1103
SWC1* 146 1103
SWC2* 113 1303
SWC3 14 23
SWC4 59 890
SWC5 42 793
SWC6 88 687
SWC7 7 424
SWC8 30 234
SWC9 59 163
* Using HiL verdicts

12. /15
# Failures # Faults # Clusters Purity F-measure Entropy Reduction Achieved Red
SWC1-Old1 24 2 8 1 0.51 1.36 66.66 72.72
SWC1-Old2 240 3 11 0.98 0.26 3.12 95.41 96.61
SWC1 32 8 11 1 0.85 1.32 65.62 87.49
SWC2 25 4 4 0.88 0.8 0.93 84 100
SWC3 7 1 1 1 1 0 85.71 100
SWC4 39 5 6 0.77 0.48 0.98 84.61 94.28
SWC5 30 3 3 0.97 0.96 0.69 90 100
SWC6 19 4 4 1 0.89 0.69 78.94 100
SWC7 66 1 2 1 0.97 0.042 96.96 98.45
SWC8 9 4 6 1 0.59 1.07 33.33 60
SWC9 8 1 1 1 1 0 87.5 100
Results
12

13. /15
Remarks
• Large-scale industrial case study
• Utilizing SiL Coverage Profiles for HiL Analysis
– if a test is passing at SiL, but failing at HiL
• Potential usefulness of fault localization techniques if
multiple faults are present
13

14. /15
Conclusion
• Failure analysis a challenge in testing automotive CPSs
• Assisting tool for testers
• Save more than 80% in failure analysis time
• Useful for unit testing with mocks and testing of the
integrated system with actual implementations
14

15. /1515
Mojdeh Golagha
golagha@in.tum.de
https://www22.in.tum.de/golagha