Sajad Khatiri, Prasun Saurabh, Timothy Zimmermann, Charith Munasinghe, Christian Birchler, Sebastiano Panichella: SBFT Tool Competition 2024 - CPS-UAV Test Case Generation Track 17th International Workshop on Search-Based and Fuzz Testing

1. SBFT Tool Competition 2024
CPS-UAV Test Case Generation Track
IEEE/ACM International Workshop on Search-Based and Fuzz Testing (SBFT) 2024
15 April 2024
Sajad
Khatiri
Prasun
Saurabh
Timothy
Zimmermann
Charith
Munasinghe
Christian
Birchler
Sebastiano
Panichella

2. • `
Unmanned Aerial Vehicles (UAVs)
2

3. 3
Sample PX4 Flight Log

4. Aerialist
UAV Test Bench
4
"Simulation-based testing of unmanned aerial vehicles with Aerialist“, ICSE 2024
Khatiri, Sajad, Sebastiano Panichella, and Paolo Tonella
UAV Config.
Env. Config.
Commands
Expectation
Test Description

5. Given an autonomous UAV flight mission
Generate test cases that violate safety distance to the
obstacles by placing obstacles in the environment
5
Test Generation

6. Competition Rules
• Use the provided platform for test definition
• Test Generation using a Search-based approach
• Place up to 4 box-shaped obstacles
• Size (length, width, height)
• Position (x, y, z)
• Orientation (r)
• Obstacles should
• Keep the mission physically possible
• Fit in the predefined area
• Be Taller than the flight altitude (10m)
6

7. Evaluation
• 6 competing test generation tools (7 submitted)
• 1 baseline approach
• Generated tests for 6 flight missions
• With 200 Simulation Budget
• Using our K8S evaluation platform
• Report the failing ones
• 20 Tests from each test suite were evaluated
• Simulated 5 times
• Assign points to each execution
• Assign score to each test case
• Estimate Test Suite Score
7

8. Tests Count
8
0
20
40
60
80
100
120
140
160
mission 2 mission 3 mission4 mission 5 mission 6 mission 7
TAIiST Surrealist AmbieGen DeepHyperion-UAV CAMBA TUMB WOGAN-UAV

9. Test Suite Score
9
0.00
2.00
4.00
6.00
8.00
10.00
12.00
14.00
16.00
18.00
mission 2 mission 3 mission 4 mission 5 mission 6 mission 7
TAIiST Surrealist AmbieGen DeepHyperion-UAV CAMBA TUMB WOGAN-UAV

10. Ranking
10
0.00 10.00 20.00 30.00 40.00 50.00 60.00
WOGAN-UAV
TUMB
CAMBA
DeepHyperion-UAV
AmbieGen
TAIiST
Surrealist [baseline]
Total Score

11. 11
0.00 10.00 20.00 30.00 40.00 50.00 60.00
WOGAN-UAV
TUMB
CAMBA
DeepHyperion-UAV
AmbieGen
TAIiST
Surrealist [baseline]
Total Score
github.com/skhatiri/UAV-Testing-Competition
github.com/skhatiri/Aerialist
mazr@zhaw.ch
Sajad Khatiri
ICSE Demonstration Presentation on Aerialist
Time: Friday 19.4, 15:15
Room: Fernando Pessoa