1. Speeding-Up Software Testing
With Computational Intelligence
Annibale Panichella
a.panichella@tudelf.nl
@AnniPanic
!1

2. Search-Based Software Testing
!2
Computational
Intelligence
Software_
Testing_
Search
Based
Software
TestingOptimization Search
Genetic Algorithms
Ant Colony
Test Case
Coverage
Assertions
Failures

3. Search-Based Software Testing
!3
Computational
Intelligence
Software_
Testing_
AI-based
Software
Testing

4. Why SBSE?
!4
class Triangle {
int a, b, c; //sides
String type = "NOT_TRIANGLE";
Triangle (int a, int b, int c){…}
void computeTriangleType() {
1. if (a == b) {
2. if (b == c)
3. type = "EQUILATERAL";
else
4. type = "ISOSCELES";
} else {
5. if (a == c) {
6. type = "ISOSCELES";
} else {
7. if (b == c)
8. type = “ISOSCELES”;
else
9. type = “SCALENE”;
}
}
}
Class Under Test (CUT)
@Test
public void test(){
// Constructor (init)
// Method Calls
// Assertions (check)
}
Test Case
@Test
public void test(){
Triangle t = new Triangle (1,2,3);
t.computeTriangleType();
String type = t.getType();
assertTrue(type.equals(“SCALENE”));
}

5. !5
Why SBSE in Unit Testing?
Class = Pass2Verifier.java
Project = Apache commons BCEL

6. Beyond Unit Testing?
!6

7. Automated Test Design and Execution
!7
Random Fuzzer Human Developers
(Are the tests good?)
AI
Fitness
Function

8. Case Study:
Testing Advanced Driver
Assistance Systems (ADAS)
8

9. Advanced Driver Assistance Systems (ADAS)
Traffic Sign Recognition (TSR)
Pedestrian Protection (PP) Lane Departure Warning (LDW)
!9
Automated Emergency Braking (AEB)

10. Feature Interactions
Sensors /
Camera
Autonomous
Feature
Actuator
Braking (over time)
!10
Sensors /
Camera
Autonomous
Feature
Actuator
Sensors /
Camera
Autonomous
Feature
Actuator
.
.
.
30% 20% … 80%
Acceleration (over time)
60% 10% … 20%
Steering (over time)
30% 20% … 80%

11. Feature Interactions
Sensors /
Camera
Autonomous
Feature
Actuator
!11
Sensors /
Camera
Autonomous
Feature
Actuator
Sensors /
Camera
Autonomous
Feature
Actuator
.
.
.
Priority?

12. Integration Components
!12
Pedestrian
Protection
(PP)
Autom. Emerg.
Braking
(AEB)
Lane Dep.
Warning
(LDW)
The integration is a rule set:
each condition checks a
specific feature interaction
situation and resolves
potential conflicts that may
arise under that condition

13. Testing Using Physics-Based Simulation
Simulator
(Matlab/Simulink)
Test Input
Test Output
!13
Software
Under Test
(SUT)

14. Test Inputs
Environment
Position
and speed
Road Shape
Traffic lights
position and
status
!14
Weather
Ego Car:
- Initial Position
- Initial Speed
Car Under Test:
- Initial Position
- Initial Speed

15. Testing Target: Feature Interactions Failures
!15
Stop

16. Infinite Test Space
!16

17. AI-Based Testing
Test Case
Selection
Initial
Tests
Search
Test
Execution
Variants
Generation
!17

18. AI-Based Testing
Test Case
Selection
Initial
Tests
Search
Test
Execution
Variants
Generation
!18
Test 1
Test 2

19. AI-Based Testing
Test Case
Selection
Initial
Tests
Search
Test
Execution
Variants
Generation
!19
Minimum distance within the
simulation time window
Results of Test 1
2m
Results of Test 2
1m
Results of Test 3
1.5m

20. AI-Based Testing
Test Case
Selection
Initial
Tests
Search
Test
Execution
Variants
Generation
!20
The best test case it the one closer to
violate the safe distance (fitness)
Results of Test 2
1m
Results of Test 1
2m

21. AI-Based Testing
Initial
Tests
Search
Test
Execution
Variants
Generation
!21
Test 2
Test Case
Selection
Mutation and/or
Crossover

22. Case Study
• Two case study systems from IEE (industrial partner)
• Designed by experts
• Manually tested for more than six months
• Different rules to integrated feature actuator commands
• Both systems consist of four self-driving features
• Adaptive Cruise Control (ACC)
• Automated Emergency Braking (AEB)
• Traffic Sign Recognition (TSR)
• Pedestrian Protection (PP)
!22

23. Many-Objective Search in Action
!23
0 min
0
1
2
3
4
F1 F2 F3 F4 F5 F6 F7 F8 F9 F10
Distancetoafailure
Different types of
(potential) failures with
feature interactions
A zero distance means we
found a test that exposes
the failure

24. Many-Objective Search in Action
!24
510 min
0
0,25
0,5
0,75
1
F1 F2 F3 F4 F5 F6 F7 F8 F9 F10
Distancetoafailure
Different types of
(potential) failures with
feature interactions
A zero distance means we
found a test that exposes
the failure

25. Some Results
!25
#DiscoveredFailures
0
2
4
6
8
10
0 3 6 9 12
Coverage-based Fuzzing
AI-based Testing

26. Example of Failures
!26

27. Feedback From Domain Experts
• The failure we found were due to undesired feature
interactions
• The failures were not previously known to the experts
(new tests for regression testing)
• We identified ways to improve the feature interaction
logic to avoid such failures
!27

28. AI-Based Testing at TUDelft
!28
https://github.com/STAMP-project/botsing https://github.com/SERG-Delft/evosql
https://github.com/apanichella/evosuite https://github.com/dappelt/xavier-grammar

29. References
[1] R. Abdessalem, A. Panichella, S. Nejati, L. Briand, T. Stifter, "Testing Autonomous Cars for Feature Interaction
Failures using Many-Objective Search". The 33rd IEEE/ACM International Conference on Automated Software
Engineering (ASE 2018), Montpellier, France.
[2] D. Di Nucci, A. Panichella, A. Zaidman, A. De Lucia. "A Test Case Prioritization Genetic Algorithm guided by the
Hypervolume Indicator". Transactions on Software Engineering (TSE) 2018 | To appear.
[3] A. Panichella, F. M. Kifetew, P. Tonella. " A Large Scale Empirical Comparison of State-of-the-art Search-based
Test Case Generators". Information and Software Technology (IST) 2018 | To appear
[4] D. Appelt, C. D. Nguyen, A. Panichella, L. Briand. "A Machine Learning-Driven Evolutionary Approach for
Testing Web Application Firewalls". IEEE Transactions on Reliability (TR) 2018 | To appear.
[5] S. Jan, A. Panichella, A. Arcuri, L. Briand. Automatic Generation of Tests to Exploit XML Injection Vulnerabilities
in Web Applications. Transactions on Software Engineering (TSE) 2018. To appear
[6] A. Panichella, F. M. Kifetew, P. Tonella. Automated Test Case Generation as a Many-Objective Optimisation
Problem with Dynamic Selection of the Targets. Transactions on Software Engineering (TSE) 2018, volume 44,
issue 2, pp 122-158. doi:10.1109/TSE.2017.2663435
!29