This document proposes a method called SURREALIST to generate realistic simulated test cases for unmanned aerial vehicles (UAVs) using real flight logs. It aims to address limitations of field testing such as lack of reproducibility and limited test scenarios. SURREALIST works in two steps: 1) It systematically replicates real flights in simulation by finding optimal drone and environment configurations that minimize differences between real and simulated flight trajectories. 2) It generates new challenging test cases by manipulating drone and environment configurations according to a difficulty measure, such as violating safety distances to obstacles. The approach is evaluated on examples of replicating and modifying an existing flight to evaluate its ability to find bugs. SURREALIST aims to generate tests that can discover non

1. Simulation-based Test Case Generation
for Unmanned Aerial Vehicles
in the Neighborhood of Real Flights
IEEE International Conference on Software Testing, Verification and Validation (ICST) 2023
17 April 2023
Sajad
Khatiri
Sebastiano
Panichella
Paolo
Tonella

2. • `
Unmanned Aerial Vehicles (UAVs)
2

3. Field Testing
o Reliable
o Not Reproducible
o Limited Test Scenarios
o Expensive
o Time Consuming
o Unsafe
3

4. Simulation-based Testing
o Reproducible
o Scalable & Automatable
o Affordable
o Safe
o Reliable Test Results?
o Reproducing Real-world Bugs?
o Generating Realistic Test Cases?
4

5. 5
Latest Github Issue Sample PX4 Flight Log

6. 6
Latest Github Issue Sample PX4 Flight Log
How can PX4 developers use the log to fix the issue?

7. SURREALIST
Generate Realistic and Effective simulated test cases
1. Systematically Replicate Field Tests in simulation using flight logs
2. Systematically Generate Challenging, but similar test cases
7

8. UAV System Test
UAV Configurations
Autopilot Parameter
Config Files (mission plan)
Environment Configurations
Weather Condition
Surrounding Objects
Runtime Commands
Radio Controller Commands
Starting Mission
Expected Behavior
Flight Trajectory
Safety Requirements
8

9. 1. Flight Replication
Systematically Replicate field tests in simulation
• Find optimal drone/environment configurations
• Replicate a Specific Logged Behavior
• Minimizing differences in real vs simulation records
9
UAV Config.
Env. Config.
Commands
Expectation
Replicated Test

10. Given the flight log of an autonomous real-world flight,
Can we place an obstacle in the simulation environment
to faithfully replicate the flight’s trajectory?
Evaluation Scenario
1. Flight Replication
10

11. 3×3×3 box
Seed Solution
11
1. Flight Replication
Distance: DTW (Dynamic Time Warping)

12. Move (∆x, ∆y)
Resize (∆l, ∆w, ∆h)
Rotate (∆r)
Mutations
12
1. Flight Replication

13. Obstacle Move (∆y = +4)
Obstacle Move (∆y = -4)
Mutations
13
1. Flight Replication

14. Adapt the mutation
for next round
Mutations
Obstacle Move (∆y = +4)
14
1. Flight Replication

15. Final Solution
15
1. Flight Replication

16. Final Solution
16
o Obstacle was moved 2m up, rotated 30˚ and made 2.8m taller
o Almost identical replication of the flight trajectory
Main Findings
1. Flight Replication

17. 2. Test Generation
Systematically Generate Challenging but similar test cases
• Manipulate drone/environment configurations
• According to a predefined Difficulty measure
17
UAV Config.
Env. Config.
Commands
Expectation
Generated Test

18. Given a simulated test case for autonomous UAV flight,
Can we generate test cases that violate safety distance
to obstacles by placing an additional obstacle?
18
2. Test Generation
Evaluation Scenario

19. Same size box
Seed Solution
d1
d2
Distance = 2×d1 + d2
19
2. Test Generation
Move (∆x, ∆y)

20. Obstacle Move (∆x = -8)
Obstacle Move (∆x = -4)
Mutations
20
2. Test Generation

21. Final Solution
21
o Obstacle was moved 8m to the left and 1m up
o Final test revealed a nondeterministic behavior
o A critical and reproducible bug in PX4 which leads to crashes
Main Findings
2. Test Generation

22. github.com/skhatiri/Surrealist
mazras@usi.ch
www.skhatiri.ir
22
Sajad Khatiri