Presentation at the IEEE/ACM International Conference on Automated Software Engineering (ASE 2023): “Automated Identification and Qualitative Characterization of Safety Concerns Reported in UAV Software Platforms” - Transactions on Software Engineering and Methodology

1. J1 presentation:
“Automated Identification and Qualitative Characterization of Safety Concerns
Reported in UAV Software Platforms”
Transactions on Software Engineering and Methodology.
Mon 11 - Fri 15 September 2023, Luxembourg
Sebastiano Panichella
Zurich University of Applied Sciences
https://spanichella.github.io/
Andrea Di Sorbo Fiorella Zampetti Aaron Visaggio Massimiliano Di Penta
University of Sannio
IEEE/ACM International Conference on
Automated Software Engineering (ASE 2023)

2. Outline
2
• DevOps shortcomings for Complex CPSs
• What types of safety bugs occur in open-source CPSs?
• Context: Cyber-physical Systems (CPSs)
The COSMOS Project has
received funding from
the European Union’s
Horizon 2020 Research
and Innovation
Programme under grant
agreement No. 957254.
UAV

3. “Emerging Cyber-physical Systems (CPS) will play a crucial role in the quality of
life of European citizens and the future of the European economy”
Context
• CPS relevant sectors:
• Healthcare
• Automotive
• Water Monitoring
• Railway
• Manufacturing
• Avionics
• etc.
MEDICAL DELIVERY
FOOD DELIVERY
• Avionics
3

4. FOOD DELIVERY

5. Background
First aerodynamic flight on another planet. Landed with Perseverance rover on 18 February 2021
SPACE EXPLORATION

6. • -
• Our (Software Engineering) view of DevOps and AI for IoT systems:
• DevOps and Continuous Delivery (CD): Whats is it?
• Present, Challenges, and Opportunities
• Relevant Research Questions
• Artificial Intelligence (AI) and Testing Automation:
• Present, Challenges, and Opportunities
• User-oriented Testing Automation
• Relevant Research Questions
“We all recognize the relevance and capacity of contemporary cyber-
physical systems for building the future of our society, but ongoing research
in the field is also clearly failing in making the right countermeasures to
avoid that CPS usage affects human being safety”. In
“Self-driving Uber kills Arizona
woman in first fatal crash involving
pedestrian”
“Swiss Post drone
crashes in Zurich
Problem Statement
“A simple software update was
the direct cause of the fatal
crashes of the Boeing 737”
6

7. • -
• Our (Software Engineering) view of DevOps and AI for IoT systems:
• DevOps and Continuous Delivery (CD): Whats is it?
• Present, Challenges, and Opportunities
• Relevant Research Questions
• Artificial Intelligence (AI) and Testing Automation:
• Present, Challenges, and Opportunities
• User-oriented Testing Automation
• Relevant Research Questions
“Self-driving Uber kills Arizona
woman in first fatal crash involving
pedestrian”
Challenge
“A simple software update was
the direct cause of the fatal
crashes of the Boeing 737”
Observability, testability, and predictability of the behavior of emerging
CPS is highly limited and, unfortunately, their usage in the real world can
lead to fatal crashes sometimes tragically involving also humans
7

8. Sebastiano Panichella Sajad Khatiri
Christian Birchler
COSMOS:
DevOps for Complex Cyber-physical Systems
https://www.cosmos-devops.org/ https://twitter.com/COSMOS_DEVOPS https://lnkd.in/eUVeaYaz

9. 9
What types of bugs occur in open-source CPSs?
CPS
Bugs

10. 10
Famous Software Failure in a CPS: The Case of Ariane 5

11. 11
Bugs and Failures in CPS
CPS Bug is
“a flaw in the hardware (not properly
handled by the software), or an incorrect
interaction between the software and
hardware components leading to a CPS
misbehavior’’
A CPS bug can manifest as a CPS failure,
which makes a CPS unable to deliver its
required functionality or not fulfilling some
non-functional properties
Properties

12. 12
Bugs in the PX4 Project
https://github.com/PX4/PX4-Autopilot/issues/8980
Px4 Issue 8980: Unsuccessful flight
“ Autopilot receiving noisy sensor-data…“

13. 13
Fiorella Zampetti, Ritu Kapur, Massimiliano Di Penta,
Sebastiano Panichella: An Empirical Characterization
of Software Bugs in Open-Source Cyber-Physical
Systems. Journal of Systems & Software (JSS).
What types of bugs occur in Open-source CPSs?
CPS
Bugs
1,151 closed issues sampled from
14 open-source CPS projects
CPS bugs taxonomy comprises
8 different high-level categories

14. 17
What about Safety Bugs of CPSs?
CPS
Bugs

15. 18
What are the main Hazards and Accidents Emerging from Safety Issues
Reported in UAV Software Platforms?
Andrea Di Sorbo, Fiorella Zampetti, Corrado A. Visaggio, Massimiliano Di Penta, and Sebastiano
Panichella: Automated Identification and Qualitative Characterization of Safety Concerns Reported in
UAV Software Platforms. Transactions on Software Engineering and Methodology.
150 Projects

16. 19
What are the main Hazards and Accidents Emerging from Safety Issues
Reported in UAV Software Platforms?
CPS Safety Related Issues of UAVs
Andrea Di Sorbo, Fiorella Zampetti, Corrado A. Visaggio, Massimiliano Di Penta, and Sebastiano
Panichella: Automated Identification and Qualitative Characterization of Safety Concerns Reported in
UAV Software Platforms. Transactions on Software Engineering and Methodology.

17. 20
RQ1: To what extent can machine learning models
automatically identify safety-related concerns in
issue reports of UAV software platforms?
RQ2: What are the main hazards and accidents emerging
from safety issues reported in UAV software platforms?
CPS Safety Related Issues of UAVs
RQ1: To what extent can machine learning models
automatically identify safety-related concerns in
issue reports of UAV software platforms?
RQ2: What are the main hazards and accidents emerging
from safety issues reported in UAV software platforms?
What are the main Hazards and Accidents Emerging from Safety Issues Reported in UAV
Software Platforms?

18. 21
Gold Standard Definition
1,916 sentences
Cohen’s k = 0.68
(substantial)
837 (43.7%) safety
1,079 (56.3%) non-safety

19. 22
Safety Bugs in open-source CPSs
Preprocessing of CPS Safety Related Issues
Tf Matrix
GitHub
Issues
Tf Matrix Tf-idf Matrix
RQ1: To what extent can machine learning models
automatically identify safety-related concerns in
issue reports of UAV software platforms?
RQ2: What are the main hazards and accidents emerging
from safety issues reported in UAV software platforms?

20. 23
Safety Bugs in open-source CPSs
ML prediction & Hyperparameter optimization
Tf Matrix
Tf-idf Matrix
RQ1: To what extent can machine learning models
automatically identify safety-related concerns in
issue reports of UAV software platforms?
RQ2: What are the main hazards and accidents emerging
from safety issues reported in UAV software platforms?

21. 24
Safety Bugs in open-source CPSs
Performance of the ML classifiers
for classifying safety-related
issues in pull requests
Tf-idf wins over other
processing approaches
> 80% Precision,
Recall, and F-measure (10-
fold validation) using word
embeddings with FastText
Link to the tool behind our research
https://github.com/spanichella/SALIENT-TOOL

22. 26
What are the main Hazards and Accidents Emerging from Safety Issues
Reported in UAV Software Platforms?
273 safety-related issues and PR 19 Hazard categories
7 Accident categories

23. 27
Co-occurrences
of hazard
categories
and accident
categories
Hazard Accident
Hazard categories and
corresponding occurrences in our
dataset of 273 safety-related
issues and pull requests.
What are the main Hazards and Accidents Emerging from Safety Issues
Reported in UAV Software Platforms?

24. What are the main Hazards and Accidents Emerging from Safety Issues
Reported in UAV Software Platforms?

25. 29
Co-occurrences
of hazard
categories
and accident
categories
Hazard Accident
Hazard categories and
corresponding occurrences in our
dataset of 273 safety-related
issues and pull requests.
Main Findings:
We need automated approaches to monitor and address safety concerns within a
CPS DevOps pipeline
What are the main Hazards and Accidents Emerging from Safety Issues
Reported in UAV Software Platforms?

26. Summary
30
• DevOps shortcomings for Complex CPSs
• What types of safety bugs occur in open-source CPSs?
• Context: Cyber-physical Systems (CPSs)
The COSMOS Project has
received funding from
the European Union’s
Horizon 2020 Research
and Innovation
Programme under grant
agreement No. 957254.
UAV

27. Thanks for the Attention!
• Any Questions?
J1 presentation:
“Automated Identification and Qualitative
Characterization of Safety Concerns Reported in UAV
Software Platforms”. Transactions on Software
Engineering and Methodology.
Mon 11 - Fri 15 September 2023, Luxembourg
Sebastiano Panichella
Zurich University of Applied Sciences
https://spanichella.github.io/
Link to the tool behind our research
https://github.com/spanichella/SALIENT-TOOL