1. “Testing and Development Challenges for
Complex Cyber-Physical Systems:
Insights from the COSMOS H2020 Project”
20 April, 2023 - Ireland Co-located with ICST 2023
Sebastiano Panichella
Zurich University of Applied Sciences
https://spanichella.github.io/
AIST 2023:
3rd International Workshop on Arti
fi
cial
Intelligence in Software Testing

2. Zurich University of
Applied Science
Senior Computer Science Researcher
Since August 2018
PhD
June 2014
October 2014 - August 2018
About me
Academic & Industrial
Collaborations
Investigated Research Topics related to
- “Development Automation, Test automation” & “Human Computer Interaction” for
- Software Systems
- Cyber-physical systems (CPSs)
- and, AI-based systems
- “Software Engineering for AI”
- “AI for Software Engineering”
G
2

3. Outline
3
• DevOps shortcomings for Complex CPSs
• What types of bugs occur in open-source CPSs?
• How to enable cost-effective testing for Self-driving cars?
• How to address the Reality Gap problem when testing UAVs?
• Context: Cyber-physical Systems (CPSs)
The COSMOS Project has
received funding from
the European Union’s
Horizon 2020 Research
and Innovation
Programme under grant
agreement No. 957254.

4. “Emerging Cyber-physical Systems (CPS) will play a crucial role in the quality of
life of European citizens and the future of the European economy”
Context
• CPS relevant sectors:
• Healthcare
• Automotive
• Water Monitoring
• Railway
• Manufacturing
• Avionics
• etc.
MEDICAL DELIVERY
FOOD DELIVERY
• Avionics
4

5. FOOD DELIVERY

6. Background
First aerodynamic
fl
ight on another planet. Landed with Perseverance rover on 18 February 2021
SPACE EXPLORATION

7. 7
UAVs
“But do we have, today UAVs, that would autonomously
map the disaster area at the Fukushima nuclear power
plant or spot the location of people stranded and isolated
after such disaster?”
Fukushima disaster
Unmanned Aerial Vehicles (UAVs) - a specific case of “CPSs”
Problem Statement (1)

8. • -
• Our (Software Engineering) view of DevOps and AI for IoT systems:
• DevOps and Continuous Delivery (CD): Whats is it?
• Present, Challenges, and Opportunities
• Relevant Research Questions
• Arti
fi
cial Intelligence (AI) and Testing Automation:
• Present, Challenges, and Opportunities
• User-oriented Testing Automation
• Relevant Research Questions
“We all recognize the relevance and capacity of contemporary cyber-
physical systems for building the future of our society, but ongoing research
in the
fi
eld is also clearly failing in making the right countermeasures to
avoid that CPS usage a
ff
ects human being safety”. In
“Self-driving Uber kills Arizona
woman in first fatal crash involving
pedestrian”
“Swiss Post drone
crashes in Zurich
Problem Statement (2)
“A simple software update was
the direct cause of the fatal
crashes of the Boeing 737”
8

9. Question:
What are the main Challenges of
Testing Cyber-physical Systems?
9

10. Question:
What are the main Challenges of
Testing Cyber-physical Systems?
Answers from the Audience (1)
https://www.slideshare.net/sebastianopanichella/testing-with-fewer-resources-toward-adaptive-
approaches-for-costeffective-test-generation-and-selection
International Summer School on Search- and Machine Learning-based Software Engineering: https://www.uco.es/smileseng/

11. Question:
What are the main Challenges of
Testing Cyber-physical Systems?
Answers from the Audience (2)
https://www.slideshare.net/sebastianopanichella/testing-with-fewer-resources-toward-adaptive-
approaches-for-costeffective-test-generation-and-selection
International Summer School on Search- and Machine Learning-based Software Engineering: https://www.uco.es/smileseng/

12. • -
• Our (Software Engineering) view of DevOps and AI for IoT systems:
• DevOps and Continuous Delivery (CD): Whats is it?
• Present, Challenges, and Opportunities
• Relevant Research Questions
• Arti
fi
cial Intelligence (AI) and Testing Automation:
• Present, Challenges, and Opportunities
• User-oriented Testing Automation
• Relevant Research Questions
“Self-driving Uber kills Arizona
woman in first fatal crash involving
pedestrian”
“Swiss Post drone
crashes in Zurich
Challenge
“A simple software update was
the direct cause of the fatal
crashes of the Boeing 737”
Observability, testability, and predictability of the behavior of emerging
CPS is highly limited and, unfortunately, their usage in the real world can
lead to fatal crashes sometimes tragically involving also humans
12

13. Research Challenges and Opportunities
As reported by National Academies:
[“A 21st Century Cyber-Physical Systems Education”]
“today's practice of IoT system design and
implementation are often unable to support
the level of ``complexity, scalability, security,
safety, […] required to meet future needs”
13

14. Research Challenges and Opportunities
“The main problem is that contemporary
development methodologies for CPS need to
incorporate core aspects of both systems and
software engineering communities, with the
goal to explicitly embrace and consider the
several direct and indirect physical effects of
software”
[“Complexity challenges in development of cyber-physical systems”]
(Martin Törngren, Ulf Sellgren Pages 478-503)
14
Crash of
Boeing 737
As reported by National Academies:
[“A 21st Century Cyber-Physical Systems Education”]
“today's practice of IoT system design and
implementation are often unable to support
the level of ``complexity, scalability, security,
safety, […] required to meet future needs”

15. Research Challenges and Opportunities
[“Complexity challenges in development of cyber-physical systems”]
(Martin Törngren, Ulf Sellgren Pages 478-503)
“As identi
fi
ed by agile methodologies, the
development of modern/emerging systems
(e.g., e-health, automotive, satellite, and IoT
manufacturing systems) should evolve with
the systems,as development never ends”
15
Tools
“The main problem is that contemporary
development methodologies for CPS need to
incorporate core aspects of both systems and
software engineering communities, with the
goal to explicitly embrace and consider the
several direct and indirect physical effects of
software”
[“Complexity challenges in development of cyber-physical systems”]
(Martin Törngren, Ulf Sellgren Pages 478-503)
Crash of
Boeing 737
As reported by National Academies:
[“A 21st Century Cyber-Physical Systems Education”]
“today's practice of IoT system design and
implementation are often unable to support
the level of ``complexity, scalability, security,
safety, […] required to meet future needs”

16. Research Challenges and Opportunities
These concepts are closely related to DevOps and
Arti
fi
cial Intelligence technologies, and several
researchers and practitioners advocate them as a
promising solutions for the development,
maintenance, testing, and evolution of these
complex systems
16
[“Complexity challenges in development of cyber-physical systems”]
(Martin Törngren, Ulf Sellgren Pages 478-503)
“As identi
fi
ed by agile methodologies, the
development of modern/emerging systems
(e.g., e-health, automotive, satellite, and IoT
manufacturing systems) should evolve with
the systems, ``as development never ends”
Tools
“The main problem is that contemporary
development methodologies for CPS need to
incorporate core aspects of both systems and
software engineering communities, with the
goal to explicitly embrace and consider the
several direct and indirect physical effects of
software”
[“Complexity challenges in development of cyber-physical systems”]
(Martin Törngren, Ulf Sellgren Pages 478-503)
Crash of
Boeing 737
As reported by National Academies:
[“A 21st Century Cyber-Physical Systems Education”]
“today's practice of IoT system design and
implementation are often unable to support
the level of ``complexity, scalability, security,
safety, […] required to meet future needs”

17. Research Challenges and Opportunities
Challenge 1: Observability, testability, and
predictability of the behavior of emerging
CPS is highly limited and, unfortunately,
their usage in the real world can lead to fatal
crashes sometimes tragically involving also
humans
Challenge 2: Contemporary DevOps and
AI practices and tools are potentially the
right solution to this problem, but they are
not developed to be applied in CPS
domains
These concepts are closely related to DevOps and
Arti
fi
cial Intelligence technologies, and several
researchers and practitioners advocate them as a
promising solutions for the development,
maintenance, testing, and evolution of these
complex systems

18. Traditional DevOps Pipeline
18
“General lack of DevOps solutions supporting the physical dimension of CPS…”
Lint

19. Sebastiano Panichella Sajad Khatiri
Christian Birchler
COSMOS:
DevOps for Complex Cyber-physical Systems
https://www.cosmos-devops.org/ https://twitter.com/COSMOS_DEVOPS https://lnkd.in/eUVeaYaz

20. COSMOS Vision
■ Develop novel DevOps tools,
methodologies, and techniques that enable
effective, continuous development and
evolution of CPS
■ Increase the level of reliability,
dependability, trustworthiness, and
adaptability of CPS
■ Delivers proven DevOps advantages and
benefits to Europe’s CPS development
community
https://www.cosmos-devops.org/ https://twitter.com/COSMOS_DEVOPS https://lnkd.in/eUVeaYaz

21. 21
COSMOS DevOps Pipeline

22. Three Methodological Pillars
22
KPIs
Scientific and Technological
Foundations of COSMOS
Empirical Validation of
COSMOS Innovations
DevOps Technological
Foundations of COSMOS

23. Industrial Use Cases
AVIATION
E-HEALTH
WATER MONITORING
SATELLITES
AUTOMOTIVE
RAILWAYS
DRONES SELF-DRIVING CARS
Reference Use Cases
23
COSMOS Use Cases

24. 24
Innovation Area 1: DevOps Pipelines for CPS
WP3: Methodology for Setting-Up and Maintaining
COSMOS DevOps Pipelines
■ CI/CD Antipatterns Identification for CPS
■ Definition of a DevOps-based Methodology to Support
the Development of Self-Adaptive CPS
■ COSMOS Pipeline Optimization
COMPONENTS

25. 25
Innovation Area 2: V&V and Security Assessment of DevOps pipelines
COMPONENTS
WP4: V&V and security assessment of COSMOS DevOps
pipelines
■ Development of Automated Techniques for Software Testing
for CPS
■ Development of Run-time Verification Techniques for
Checking and Diagnosing CPS Executions
■ Development of Solutions for Detecting Security
Vulnerabilities in CPS

26. 26
Innovation Area 3: Tools for High Quality CPS Software Evolution
WP5: Development of Tools to support High Quality CPS
Software Evolution
■ Design and Development of Refactoring Framework
for Secure and Reliable CPS
■ Development of Test Case Generation Tools for Rapid
DevOps Iterations
■ Development of Tools to support User-oriented
Maintenance and Testing
COMPONENTS

27. Innovation Area 4: Tools for Monitoring, Self-healing and Self-adaptability of CPS
WP6: Development of Tools to support Monitoring, Self-
healing, and Self-adaptability of CPS in the Field
■ Development and Assessment of CPS Change &
Behavioral Models
■ Developing AI-based Solutions to Support Two-speed
DevOps Cycles for CPS
■ Automated Quality Assessment and Monitoring of CPS
in the Field
■ Development of AI-based Solutions to Increase CPS
Self-adaptability to Diverse Contexts
COMPONENTS
https://www.cosmos-devops.org/ https://twitter.com/COSMOS_DEVOPS https://lnkd.in/eUVeaYaz

28. Innovation Area 4: Tools for Monitoring, Self-healing and Self-adaptability of CPS
WP6: Development of Tools to support Monitoring, Self-
healing, and Self-adaptability of CPS in the Field
■ Development and Assessment of CPS Change &
Behavioral Models
■ Developing AI-based Solutions to Support Two-speed
DevOps Cycles for CPS
■ Automated Quality Assessment and Monitoring of CPS
in the Field
■ Development of AI-based Solutions to Increase CPS
Self-adaptability to Diverse Contexts
COMPONENTS
https://www.cosmos-devops.org/ https://twitter.com/COSMOS_DEVOPS https://lnkd.in/eUVeaYaz

29. Outline
29
• DevOps shortcomings for Complex CPSs
• What types of bugs occur in open-source CPSs?
• How to enable cost-effective testing for Self-driving cars?
• How to address the Reality Gap problem when testing UAVs?
• Context: Cyber-physical Systems (CPSs)
The COSMOS Project has
received funding from
the European Union’s
Horizon 2020 Research
and Innovation
Programme under grant
agreement No. 957254.

30. What types of bugs occur in open-source CPSs?
CPS
Bugs
J1 presentation: “An Empirical Characterization of Software Bugs in Open-Source Cyber-Physical Systems” - Journal of Systems & Software (JSS) - Presented at ICST

31. 31
Famous Software Failure in a CPS: The Case of Ariane 5

32. 32
Bugs and Failures in CPS
CPS Bug is
“a
fl
aw in the hardware (not properly
handled by the software), or an incorrect
interaction between the software and
hardware components leading to a CPS
misbehavior’’
A CPS bug can manifest as a CPS failure,
which makes a CPS unable to deliver its
required functionality or not ful
fi
lling some
non-functional properties
Properties

33. 33
Question:
What types of bugs occur in open-source CPSs?

34. 34
Bugs in the PX4 Project
https://github.com/PX4/PX4-Autopilot/issues/8980
Px4 Issue 8980: Unsuccessful
fl
ight
“ Autopilot receiving noisy sensor-data…“

35. 35
Bugs in the OpenPilot Project
Openpilot Issue 2103: A CAN bus error
https://github.com/commaai/openpilot/issues/2103
“ Software update on unsupported hardware devices…“

36. 36
Fiorella Zampetti, Ritu Kapur, Massimiliano Di Penta,
Sebastiano Panichella: An Empirical Characterization
of Software Bugs in Open-Source Cyber-Physical
Systems. Journal of Systems & Software (JSS).
What types of bugs occur in Open-source CPSs?
CPS
Bugs
1,151 closed issues sampled from
14 open-source CPS projects
CPS bugs taxonomy comprises
8 di
ff
erent high-level categories

37. 37
What types of bugs occur in Open-source CPSs?
Process for designing a taxonomy of bugs occurring in CPSs
from GitHub (Arduino, drones, robotics, automotive, etc.)

38. 38
Analyzed Projects
Fiorella Zampetti, Ritu Kapur, Massimiliano Di Penta, Sebastiano Panichella: An Empirical Characterization of Software Bugs in Open-Source Cyber-Physical Systems. Journal of Systems & Software (JSS).
14 open-source CPS projects CPS Domains Issues
Closed
Issues
Bug-related
Issues

39. 39
What types of bugs occur in Open-source CPSs?
33% of the
bugs are
CPS-speci
fi
c
7
6
5
4
3
2
1
8
Grouped into
8 high-level categories
CPS Bug taxonomy:
22 different root causes

40. 40
Hardware Bugs in Open-source CPSs
Energy Faulty Sensors
Hardware failure
Hardware
Not Supported/Compatible
1
Memory

41. 41
Hardware Bugs in Open-source CPSs
Energy
Faulty Sensors
Hardware failure
1
Bug #21033 in openpilot points out the
presence of a CAN bus error on a
speci
fi
c device (i.e., Rav4 Prime).
Main Findings:
Hardware-speci
fi
c bugs are peculiar to our taxonomy, and, unsurprisingly, all of
them are CPS-speci
fi
c.
Recognizing (and simulating) hardware failures has paramount importance in
V&V. Also, developers should take particular care of hardware compatibility,
especially for CPSs targeting multiple devices.
The interaction with the hardware makes particularly crucial the analysis of non-
functional properties such as performance, memory, and energy consumption.

42. 42
Network & Interface Bugs in Open-source CPSs

43. 43
Network & Interface Bugs in Open-source CPSs
Bug #4302 in Arduino, where there is a memory leak while
doing repeated connections to a server, causing the loss of around
8KB for each connection.
Bug #6546 in PX4-AutoPilot that has been inherited from the third-
party library being used while interfacing with GPS (dealing with GPS
‘‘jamming’’ that has already been reported as an issue in the library aimed at supporting
the Intel Aero Platform)

44. 44
Network & Interface Bugs in Open-source CPSs
Main Findings:
Networking plays a paramount role for CPSs
and can be the origin of bugs.
The CPS infrastructure should include
network monitors and V&V techniques may
contemplate CPS misbehavior caused by
network-speci
fi
c aspects.
Main Findings:
Interfacing bugs are challenging for
developers coping with CPSs, and testing
e
ff
orts should focus on them.

45. 45
Question:
What types of bugs are we still missing?
CPS
Bugs

46. 46
CPS Safety Related Issues of UAVs
Andrea Di Sorbo, Fiorella Zampetti, Corrado A. Visaggio, Massimiliano Di Penta, and Sebastiano
Panichella: Automated Identification and Qualitative Characterization of Safety Concerns Reported in
UAV Software Platforms. Transactions on Software Engineering and Methodology.
What are the main Hazards and Accidents Emerging from Safety Issues
Reported in UAV Software Platforms?

47. 47
Co-occurrences
of hazard
categories
and accident
categories
Hazard Accident
Hazard categories and
corresponding occurrences in our
dataset of 273 safety-related
issues and pull requests.
What are the main Hazards and Accidents Emerging from Safety Issues
Reported in UAV Software Platforms?

48. 48
Co-occurrences
of hazard
categories
and accident
categories
Hazard Accident
Hazard categories and
corresponding occurrences in our
dataset of 273 safety-related
issues and pull requests.
What are the main Hazards and Accidents Emerging from Safety Issues
Reported in UAV Software Platforms?
Main Findings:
We need automated approaches to monitor and address safety concerns within a
CPS DevOps pipeline

49. 49
DevOps Challenges for Dealing with CPS Bugs and Complexity
Interview-based methodology
Interviews’
transcripts
Card Sorting
Early feedback from
COSMOS partners
Bad (and good)
practices,
Challenges,
Barriers,
Mitigation
Analysis Triangulation
Validation outside COSMOS
(survey questionnaire)
Pull Requests (PRs) Mining
20 CPS related projects

50. 50
DevOps Challenges for Dealing with CPS Bugs and Complexity
Zampetti, Fiorella; Tamburri, Damian ; Panichella, Sebastiano;
Panichella, Annibale; Canfora, Gerardo; Di Penta, Massimiliano:
Continuous Integration and Delivery practices for Cyber-Physical
systems: An interview-based study. Transactions on Software
Engineering and Methodology.
Finding Overview:

51. 51
DevOps Challenges for Dealing with CPS Bugs and Complexity
Finding Overview:
Zampetti, Fiorella; Tamburri, Damian ; Panichella, Sebastiano;
Panichella, Annibale; Canfora, Gerardo; Di Penta, Massimiliano:
Continuous Integration and Delivery practices for Cyber-Physical
systems: An interview-based study. Transactions on Software
Engineering and Methodology.

52. 52
DevOps Challenges for Dealing with CPS Bugs and Complexity
Finding Overview:
Zampetti, Fiorella; Tamburri, Damian ; Panichella, Sebastiano;
Panichella, Annibale; Canfora, Gerardo; Di Penta, Massimiliano:
Continuous Integration and Delivery practices for Cyber-Physical
systems: An interview-based study. Transactions on Software
Engineering and Methodology.

53. 53
Challenges vs. Mitigation Strategies
Mitigation Stategies Mitigation Stategies
Challenges Challenges

54. Challenges vs. Mitigation Strategies
Simulation Simulation
Challenges Mitigation Stategies
1
2
…

55. 55
• Context: Cyber-physical Systems (CPSs)
The COSMOS Project has
received funding from
the European Union’s
Horizon 2020 Research
and Innovation
Programme under grant
agreement No. 957254.
• DevOps shortcomings for Complex CPSs
• What types of bugs occur in open-source CPSs?
• How to enable cost-effective testing for Self-driving cars?
• How to address the Reality Gap problem when testing UAVs?
Outline

56. How to enable cost-effective testing
for Self-driving cars?
56
Christian Birchler

57. Tesla Car
Autonomous Driving Systems (ADSs)
Multi-sensing Systems:
• Autonomous systems capture surrounding
environmental data at run-time via
multiple sensors (e.g. camera, radar, lidar)
as inputs
• Processes these data with Deep Neural
Networks (DNNs) and outputs control
decisions (e.g. steering).
• Requires robust testing that
• creates realistic, diverse test cases
57

58. .
.
.
Driving
Actions
Sensors /
Camera
Autonomous
Feature
Actuator
58
Autonomous Driving Systems (ADSs)
1. Pedestrians
2. Lane Position
4. Other Cars
3. Traf
fi
c Signs
DNNs • steering
• stop
• acceleration/
deceleration
• …

59. ADSs
59
Traditional DevOps Pipeline ADSs
“Manual Testing is still
Dominant…”

60. Testing Steps in ADSs
60
Requirements of Testing ADSs
• Generate Diversi
fi
ed Test
Inputs (or Scenarios)
• Evaluation based Failures
Detection
“Manual Testing is still
Dominant…”

61. National, January 2022
Reuters, September 2021
Testing Autonomous Driving Systems
The New York Times, April 2021

62. 62
class Triangle {
int a, b, c; //sides
String type = "NOT_TRIANGLE";
Triangle (int a, int b, int c){…}
void computeTriangleType() {
1. if (a == b) {
2. if (b == c)
3. type = "EQUILATERAL";
else
4. type = "ISOSCELES";
} else {
5. if (a == c) {
6. type = "ISOSCELES";
} else {
7. if (b == c)
8. type = “ISOSCELES”;
else
9. type = “SCALENE”;
}
}
}
Java Class Under Test (CUT)
@Test
public void test(){
Triangle t = new Triangle (1,2,3);
t.computeTriangleType();
String type = t.getType();
assertTrue(type.equals(“SCALENE”));
}
Test Case
Code Coverage:
The main
Quality Assessment
Criteria
Traditional Development Pipeline:
Coding v.s. Testing

63. 63
class Triangle {
int a, b, c; //sides
String type = "NOT_TRIANGLE";
Triangle (int a, int b, int c){…}
void computeTriangleType() {
1. if (a == b) {
2. if (b == c)
3. type = "EQUILATERAL";
else
4. type = "ISOSCELES";
} else {
5. if (a == c) {
6. type = "ISOSCELES";
} else {
7. if (b == c)
8. type = “ISOSCELES”;
else
9. type = “SCALENE”;
}
}
}
Java Class Under Test (CUT)
@Test
public void test(){
Triangle t = new Triangle (1,2,3);
t.computeTriangleType();
String type = t.getType();
assertTrue(type.equals(“SCALENE”));
}
Test Case
Traditional Development Pipeline:
Coding v.s. Testing
Code Coverage:
Not Suf
fi
cient as
Quality Assessment
Criteria

64. Challenges of Testing ADSs
64
Challenge 1:
Code coverage
vs.
Scenario Coverage
Challenge 2:
Code coverage
&
CPU & Memory
consumption
Challenge 3:
Unit-Test
v.s.
System-level Testing

65. In
fi
nite Test Space
65

66. 66
COSMOS DevOps Testing Pipeline

67. 67
World of Agile, 2018
Testing Autonomous Driving Systems

68. 68
World of Agile, 2018
Testing Autonomous Driving Systems
Testing on-the-road
!

69. 69
World of Agile, 2018
Testing Autonomous Driving Systems
Testing on-the-road
!
Simulation-based Testing

70. 70
Testing Autonomous Driving Systems
Real-world testing:
➡Realistic
➡Trustworthy
➡Costly
➡Nondeterministic
Testing on-the-road
!
Simulation-based Testing
Simulation is:
➡Cheaper
➡Faster
➡Less reliable
➡Complex CI/CD integration

71. 71
Simulation-based Testing for ADSs

72. Simulation-Based Test Case
Simulator
(Matlab/Simulink)
Test Input
Test Output
72
Software
Under Test
(SUT)

73. Test Output: Safe and unsafe Tests in Autonomous Driving Systems
Test Output
Safe Behaviour
Unsafe Behaviour
https://sbft23.github.io/

74. Regression Testing
“Regression testing is re-running
functional and non-functional tests to
ensure that previously developed and
tested software still performs after a
change”
Anirban Basu
74

75. Why Do We Need
Regression Testing?
75

76. A Typical Scenario
76
Class 1 Class 2 Class 3
Class 4 Class 5 Class 6
Let’s add a new
functionality
Class N
Production Code
Test Code

77. A Typical Scenario
77
Class 1 Class 2 Class 3
Class 4 Class 5 Class 6
Now, let’s test
the new code
Class N+1 Class N+2
Class N
Great!
Production Code
Test Code

78. A Typical Scenario
78
Class 1 Class 2 Class 3
Class 4 Class 5 Class 6
Class N
Production Code
Test Code
Let’s push our
changes to the
CI server
Class N+1 Class N+2
Great!

79. A Typical Scenario
79
Class 1 Class 2 Class 3
Class 4 Class 5 Class 6
Class N
Why? We haven’t
changed Class 6
Class N+1 Class N+2
Production Code
Test Code

80. Why Regression Testing?
Many developers don’t want to believe it, small changes to one part of a
system often cause problems in distant (other) parts of the system
80

81. Why Regression Testing?
Change can have an unexpected impact on other part of the systems
81
Projects can be
large (e.g., OS)
Interconnected
Components
Software always
Evolves

82. Regression Testing
82
Yoo et al. 2013
Selection
Prioritization
Minimization
Re-test All
V.S.

83. Regression Testing
Birchler et al.,
SANER 2022.
github.com/ChristianBirchler/sdc-scissor
Birchler et al., TOSEM 2022
Birchler et al.,
EMSE 2023.

84. Regression Testing
84
Selection
Birchler et al.,
SANER 2022.
Birchler et al.,
EMSE 2023.

85. Soft-body Simulator: BeamNG
85
Sebastiano Panichella
Zurich University of
Applied Sciences
Alessio Gambi
IMC University of
Applied Science Krems
Ongoing
Collaboration with

86. SDC-Scissor: Test Regression Pipeline
86
Test Generation Oracle
Training Prediction
github.com/ChristianBirchler/sdc-scissor

87. Test Selection for Self-driving Cars
87
How does a test
look like?
{
‘road_points’: [
(x0,y0),
(x1,y1),
(x2,y2),
(x3,y3),
…
]
}
test.json

88. Safe Tests
88

89. Unsafe Tests
89

90. Regression Testing with SDC-scissor: Test Generation
90
{
‘road_points’: [
(x0,y0),
(x1,y1),
(x2,y2),
(x3,y3),
…
]
}
test.json
{
‘road_points’: [
(x0,y0),
(x1,y1),
(x2,y2),
(x3,y3),
…
]
‘test_outcome’: ‘PASS’
}
test.json
{
‘road_points
’: [
(x0,y0),
(x1,y1),
(x2,y2),
(x3,y3)]
‘test_outc
ome’:
‘PASS’
}
test.json
{
‘road_points
’: [
(x0,y0),
(x1,y1),
(x2,y2),
(x3,y3)]
‘test_outc
ome’:
‘PASS’
}
test.json
{
‘road_points
’: [
(x0,y0),
(x1,y1),
(x2,y2),
(x3,y3)]
‘test_outc
ome’:
‘PASS’
}
test.json
{
‘road_points
’: [
(x0,y0),
(x1,y1),
(x2,y2),
(x3,y3)]
‘test_outc
ome’:
‘PASS’
}
test.json

91. Test Selection
91

92. Test Selection
92

93. Our Intuition
93
Road Features

94. Regression Testing with SDC-scissor: Test Outcome Prediction
94
dist, turns, angle, length,...
194, 5, 94,286, 99234,2,39
194, 5, 94,286, 99234,2,39
194, 5, 94,286, 99234,2,39
194, 5, 94,286, 99234,2,39
194, 5, 94,286, 99234,2,39
194, 5, 94,286, 99234,2,39
194, 5, 94,286, 99234,2,39
194, 5, 94,286, 99234,2,39
194, 5, 94,286, 99234,2,39
194, 5, 94,286, 99234,2,39
features.csv
ML

95. Dataset Summary
95
More than 10,000 evaluated test scenarios in simulation!
Distribution of safe and unsafe scenarios
0%
25%
50%
75%
100%
BeamNG AI cautious BeamNG AI moderate BeamNG AI reckless Driver.AI
Safe Unsafe
Unsafe
Unsafe
Unsafe
Unsafe

96. SDC-Scissor Cost-effectiveness
𝑆
𝑖
𝑚
𝑢
𝑙
𝑎
𝑡
𝑖
𝑜
𝑛
𝑡
𝑖
𝑚
𝑒
𝑏
𝑦
𝑟
𝑎
𝑛
𝑑
𝑜
𝑚
𝑠
𝑒
𝑙
𝑒
𝑐
𝑡
𝑖
𝑜
𝑛
𝑆
𝑖
𝑚
𝑢
𝑙
𝑎
𝑡
𝑜
𝑛
𝑡
𝑖
𝑚
𝑒
𝑏
𝑦
𝑆
𝐷
𝐶
−
𝑆
𝑐
𝑖
𝑠
𝑠
𝑜
𝑟
Dataset
and ML
model
96
We spend 41% less computational time to
detect the same number of faults!
57% < F1-score < 97%
Finding:
Logistic and Naïve Bayes
classi
fi
ers are the more accurate

97. Regression Testing
97
Prioritization
Selection
Minimization
Birchler et al.,
TOSEM 2022

98. Test case prioritization: executing the available test cases in a speci
fi
c
order that increases the likelihood of revealing regression faults earlier.
98
Test Suite
T1 T2 T3 T4
Ordered Test Suite
T4 T2 T1 T3
Prioritization
Procedure
Test Case Prioritization

99. Black-Box Test Case Prioritization
99
@Test
public void testTriangle_invalid2() {
assertEquals(Triangle2.Type.INVALID,
Triangle2.triangle(-5,1,3));
}
@Test
public void testTriangle_isoscele() {
assertEquals(Triangle2.Type.ISOSCELE,
Triangle2.triangle(3,3,4));
}
@Test
public void testTriangle_isoscele() {
assertEquals(Triangle2.Type.ISOSCELE,
Triangle2.triangle(3,4,3));
}
@Test
public void testTriangle_isoscele() {
assertEquals(Triangle2.Type.SCALENE,
Triangle2.triangle(4,9,6));
}
T1
T2
T3
T4
Input diversity (e.g., Euclidean distance):
T1 T2 T3 T4
T1 - 5.91 6.40 12.40
T2 5.91 - 1.41 8.54
T3 6.40 1.41 - 8.30
T4 12.40 8.54 8.30 -
Euclidean distance between
input vectors of T4 and T2

100. Test Prioritization
100

101. Our Intuition
101
Cost to minimize:
Time needed to run the whole test suite
(Only the simulation-based tests)

102. 102
We extract these road features from
simulation-based tests
Roads with more diverse set of
features correspond to more
diverse test case inputs
Diversity is measure using the
Euclidean distance between two
feature vectors
Road Features
Our Intuition

103. Multi-Objective Genetic Algorithms
We can use Genetic Algorithms to search for optimal
test case permutation/prioritization of the test case
103
Initial Population
Selection
Crossover
Mutation
End?
YES NO
T1 T4 T2 T8 … Tn
We can encode a solution as a permutation of n test cases
In this example, T4 is executed before T2 and after T1

104. Multi-Objective Genetic Algorithms
We need search objectives to measure the quality of
each solution. We have two objectives:
104
Initial Population
Selection
Crossover
Mutation
End?
YES NO
Objective 1: the sum of the
distances between each pair
of consecutive test cases ti
and ti-1 in the permutation
Objective 2: the sum of the
(past) execution cost for all
test cases. The cost of the
test ti is divided by its position
i in the permutation

105. Multi-Objective Genetic Algorithms
New solutions (o
ff
spring) are generated by
recombining the genes of two parent solutions
105
Initial Population
Selection
Crossover
Mutation
End?
YES NO
T1 T4 T2 T5 T6 T3
T3 T1 T2 T6 T4 T5
Parent solutions

106. Multi-Objective Genetic Algorithms
106
Initial Population
Selection
Crossover
Mutation
End?
YES NO
T1 T4 T2 T5 T6 T3
T3 T1 T2 T6 T4 T5
Parent solutions
T1 T4 T2
O
ff
spring
New solutions (o
ff
spring) are generated by
recombining the genes of two parent solutions

107. Multi-Objective Genetic Algorithms
107
Initial Population
Selection
Crossover
Mutation
End?
YES NO
T1 T4 T2 T5 T6 T3
T3 T1 T2 T6 T4 T5
T1 T4 T2 T3 T6 T5
Parent solutions O
ff
spring
New solutions (o
ff
spring) are generated by
recombining the genes of two parent solutions

108. Multi-Objective Genetic Algorithms
108
Initial Population
Selection
Crossover
Mutation
End?
YES NO
T1 T4 T2 T5 T6 T3
T3 T1 T2 T6 T4 T5
Parent solutions
T1 T4 T2 T3 T6 T5
O
ff
spring 1
T3 T1 T2 T4 T5 T6
O
ff
spring 2
New solutions (o
ff
spring) are generated by
recombining the genes of two parent solutions

109. Multi-Objective Genetic Algorithms
The newly generated permutation will be further
changed by applying random small mutation
109
O
ff
spring
T1 T4 T2 T3 T6 T5
Mutated O
ff
spring
Initial Population
Selection
Crossover
Mutation
End?
YES NO
T1 T6 T2 T3 T4 T5
The swap mutation operator randomly swaps two
genes in a (o
ff
spring) solution

110. Our Results
110
We compare the
fi
nal solution produced by GA
against two baselines:
• Random prioritization (average of 100K
randomly generated permutations)
• Greedy algorithm, which build a permutation
by greedily selects the test cases that is the
more diverse from the previously selected
one
Safe Driving Style

111. Cost-effectiveness with
111
Safe Driving Style Aggressive Driving Style Driving Style of a Different AI

112. Cost-effectiveness
112
SO-SDC-Prioritizer
MO-SDC-Prioritizer
Greedy

113. Application to AICAS (Automotive) Use Case in COSMOS
113
Birchler et al.,
EMSE 2023.
Simulated World Physical World

114. CAN Bus
114
ECU
BCM
Instr.
Cluster
ABS
Airbag
HVAC
Traditionally
Multiple connections between each control unit to
exchange data
- Complicated / heavy cable harness
- Error prone
- Expensive

115. CAN Bus
115
Control Area Network
Protocol used for communication between control
units over a shared network
Used in
- Cars
- Robots
- Automated Production Facilities
- Etc
ECU
BCM
Instr.
Cluster
ABS
Airbag
HVAC

116. CAN Bus
116
Control Area Network
One shared network between all controllers.
All stations can publish and receive messages
ECU
BCM
Instr.
Cluster
ABS
Airbag
HVAC
CAN BUS

117. Limitations
117
Testing Can Bus Messages are generated manually
- Takes a lot of time and effort
- Limited Diversity of Inputs
- Limited Realism of Inputs
ECU
BCM
Instr.
Cluster
ABS
Airbag
HVAC

118. Our Goal & Requirements
118
DATA
Initiates
Collects
Generates
CAN Messages
Use the data created during
simulations and create CAN
Messages from that data
+ Data are generated simultaneously
+ Data should be realistic
+ More/Diverse messages

119. What Data do we get from the Simulator?
119

120. 120
Can Bus Handler – Message Generation

121. 121
Achievements:
• Ability to generate “real time” CAN
messages
• Signals generated by realistic driving
behaviours from a simulation
environment
Ongoing:
• Validating the fault detection of
generated messages
Simulated World Physical World

122. 122
Achievements:
• Ability to generate “real time” CAN
messages
• Signals generated by realistic driving
behaviours from a simulation
environment
Ongoing:
• Validating the fault detection of
generated messages
Simulated World Physical World
Main Findings:
ML and GA approaches can be exploited to enable cost-e
ff
ective regression testing
approaches
More meaningful types of features need to be designed
Moving from simulated to real world in each CPS domain and use case requires
• Extensive development and technological e
ff
ort
• Ad-hoc metrics to assess correctness of generated signals and assessing fault detection

123. 123
Achievements:
• Ability to generate “real time” CAN
messages
• Signals generated by realistic driving
behaviours from a simulation
environment
Ongoing:
• Validating the fault detection of
generated messages
Simulated World Physical World
Main Findings:
ML and GA approaches can be exploited to enable cost-e
ff
ective regression testing
approaches
More meaningful types of features need to be designed
Moving from simulated to real world in each CPS domain and use case requires
• Extensive development and technological e
ff
ort
• Ad-hoc metrics to assess correctness of generated signals and assessing fault detection

124. 124
• Context: Cyber-physical Systems (CPSs)
The COSMOS Project has
received funding from
the European Union’s
Horizon 2020 Research
and Innovation
Programme under grant
agreement No. 957254.
• DevOps shortcomings for Complex CPSs
• What types of bugs occur in open-source CPSs?
• How to enable cost-effective testing for Self-driving cars?
• How to address the Reality Gap problem when testing UAVs?
Outline

125. 125
How to address the Reality Gap problem when
testing UAVs?
Sajad Khatiri

126. Field Testing
o Reliable
o Not Reproducible
o Limited Test Scenarios
o Expensive
o Time Consuming
o Unsafe
126

127. Simulation-based Testing
o Reproducible
o Scalable & Automatable
o Affordable
o Safe
o Reliable Test Results?
o Reproducing Real-world Bugs?
o Generating Realistic Test Cases?
127

128. Reality Gap
“The extent to which, we can rely on
simulations to faithfully mirror the
real-world performance of CPS”
128

129. 129
Latest Github Issue
X
Y
Z
A set of Time Series:
PX4
Velocity
Acceleration
CPU/RAM
Temperature
How can PX4 developers use PX4 logs
to fix the issue?

130. 130
SURREALIST:
UAV Test Case Generation in the Neighborhood of Real Flights
Sajad Khatiri, Sebastiano Panichella, Paolo Tonella: Simulation-based Test Case Generation for Unmanned Aerial Vehicles in
the Neighborhood of Real Flights. International Conference on Software Testing, Verification and Validation. (ICST 2023)
▪Unsafe / Misbehavior

131. SURREALIST
Generate Realistic and Effective simulated test cases
1. Systematically Replicate Field Tests in simulation
2. Systematically Generate Challenging, but similar test cases
131

132. UAV System Test
UAV Configurations
Autopilot Parameter
Config Files (mission plan)
Environment Configurations
Weather Condition
Surrounding Objects
Runtime Commands
Radio Controller Commands
Starting Mission
Expected Behavior
Flight Trajectory
Safety Requirements
132

133. 1. Flight Replication
Systematically Replicate field tests in simulation
• Find optimal drone/environment configurations
• Replicate a Specific Logged Behavior
• Minimizing differences in real vs simulation records
133
UAV Config.
Env. Config.
Commands
Expectation
Replicated Test

134. Given the flight log of an autonomous real-world flight,
Can we place an obstacle in the simulation environment
to faithfully replicate the flight’s trajectory?
Evaluation Scenario
1. Flight Replication
134

135. 3×3×3 box
Seed Solution
135
1. Flight Replication
Distance: DTW (Dynamic Time Warping)

136. Move (∆x, ∆y)
Resize (∆l, ∆w, ∆h)
Rotate (∆r)
Mutations
136
1. Flight Replication

137. Obstacle Move (∆y = +4)
Obstacle Move (∆y = -4)
Mutations
137
1. Flight Replication

138. Adapt the mutation
for next round
Mutations
Obstacle Move (∆y = +4)
138
1. Flight Replication

139. Final Solution
139
1. Flight Replication

140. Final Solution
140
1. Flight Replication
o Obstacle was moved 2m up, rotated 30˚and made 2.8m taller
o Almost identical replication of the flight trajectory
Main Findings

141. 2. Test Generation
Systematically Generate Challenging but similar test cases
• Manipulate drone/environment configurations
• According to a predefined Difficulty measure
141
UAV Config.
Env. Config.
Commands
Expectation
Generated Test

142. Given a simulated test case for autonomous UAV flight,
Can we generate test cases that violate safety distance
to obstacles by placing an additional obstacle?
142
2. Test Generation
Evaluation Scenario

143. Same size box
Seed Solution
d1
d2
Distance = 2×d1 + d2
143
2. Test Generation
Move (∆x, ∆y)

144. Obstacle Move (∆x = -8)
Obstacle Move (∆x = -4)
Mutations
144
2. Test Generation

145. Final Solution
145
2. Test Generation
o Obstacle was moved 8m to the left and 1m up
o Final test revealed a nondeterministic behavior
o A critical and reproducible bug in PX4 which leads to crashes
Main Findings

146. Ongoing Research and Application to GMV (Avionic) Use Case in COSMOS
146
“Comparing Simulation and Field tests
using execution logs”
Simulated
World
Physical
World
Testing Path Planning Systems
of airplanes requires considering
contrasting goals:
V.S.

147. Summary
147
• Context: Cyber-physical Systems (CPSs) The COSMOS Project has
received funding from
the European Union’s
Horizon 2020 Research
and Innovation
Programme under grant
agreement No. 957254.
CPS
Bugs
• DevOps shortcomings for Complex CPSs
• What types of bugs occur in open-source CPSs?
• How to enable cost-effective testing for Self-driving cars?
• How to address the Reality Gap problem when testing UAVs?

148. Thanks for the Attention!
• Any Questions?
“Testing and Development Challenges for
Complex Cyber-Physical Systems:
Insights from the COSMOS H2020 Project”
20 April, 2023 - Ireland Co-located with ICST 2023
Sebastiano Panichella
Zurich University of Applied Sciences
https://spanichella.github.io/