COSMOS: DevOps for Complex Cyber-physical Systems Sebastiano Panichella Zurich University of Applied Sciences (ZHAW) Workshop on Adaptive CPSoS (WASOS) 2023

1. COSMOS:
DevOps for Complex Cyber-physical Systems
Sebastiano Panichella
Zurich University of Applied Sciences (ZHAW)
Workshop on Adaptive CPSoS (WASOS) 2023
1
www.COSMOS-DevOps.org
@COSMOS_DEVOPS
The COSMOS Project has
received funding from
the European Union’s
Horizon 2020 Research
and InnovaCon
Programme under grant
agreement No. 957254.

2. ■ Context & Challenges
■ Vision & Objectives
■ Work Packages (WPs)
⧫ WPs and Progress
■ Impact
Outline
2

3. COSMOS Technology Vision 3
■ Emerging Cyber-physical Systems (CPS) play a crucial role
in the quality of life of European citizens and the future of
the European “smart everywhere” economy
■ CPS relevant sectors
⧫ Healthcare
⧫ Avionics
⧫ Automotive
⧫ Utilities
⧫ Railway
⧫ Manufacturing
⧫ Smart Cities
⧫ Many others…
Context

4. COSMOS Technology Vision 4
Challenges
■ C1: Observability, testability, and predictability of behaviour of CPS is highly
limited and, unfortunately, their usage in the real world can lead to fatal
crashes sometimes tragically involving also humans
■ C2: Contemporary DevOps practices and tools are potentially the right solution
to this problem, but are currently not developed to be applied in CPS domains

5. COSMOS Technology Vision 5
Traditional DevOps Pipeline
Lint
“General lack of DevOps solutions supporting the physical dimension of CPS…”

6. COSMOS Technology Vision 6
COSMOS Vision
■ Develop novel DevOps tools,
methodologies, and techniques that
enable effective, continuous
development and evolution of CPS
■ Increase the level of reliability,
dependability, trustworthiness, and
adaptability of CPS
■ Delivers proven DevOps
advantages and benefits to Europe’s
CPS development community

7. COSMOS Technology Vision 7
COSMOS DevOps Pipeline
Lint

8. COSMOS Technology Vision 8
Project Objectives
Technical objectives
■ Realise innovative DevOps automation solutions specifically tailored for CPS
■ Automate V&V and security assessment of CPS within DevOps pipelines to ensure
high levels of dependability
■ Enable monitoring and evolving CPS behaviour in the field to provide high
adaptability of CPS to unexpected changes
Integration and Evaluation objectives
■ Develop and evolve COSMOS approaches, technologies, and services so they can be
integrated into different DevOps and CPS toolchains
■ Validate effectiveness of COSMOS technologies in 5 industrial demonstrators
■ Broadly disseminate the COSMOS technologies to create a European DevOps for
CPS ecosystem

9. COSMOS Technology Vision 9
Three Methodological Pillars
KPIs
Scien&fic and Technological
Founda&ons of COSMOS
Empirical Valida&on of
COSMOS Innova&ons
DevOps Technological
Founda&ons of COSMOS

10. COSMOS Technology Vision 10
Work Packages Overview

11. Automotive
11
Industrial CPS Evaluations
Railways
Medical
Evaluations conducted at both mid-project and during final project months
Avionics
Utilities
COSMOS Technology Vision

12. 12
Innovation Area 1: DevOps Pipelines for CPS
WP3: Methodology for Setting-Up and Maintaining
COSMOS DevOps Pipelines
■ CI/CD Antipatterns Identification for CPS
■ Definition of a DevOps-based Methodology to
Support the Development of Self-Adaptive CPS
■ COSMOS Pipeline Optimization
COMPONENTS
COSMOS Technology Vision

13. 13
Progress (Innovation Area 1)
COSMOS Technology Vision
WP3 - Setting-Up and Maintaining DevOps Pipelines (UoS)
■ (Completed) Catalogue of CI/CD CPS good (and bad) practices
■ (Completed) Detector of bad practices in CPS pipelines
■ (Completed) Approach for the smart allocation of jobs on HiL
and simulators
■ Ongoing
⧫ Tool for the smart allocation of jobs on HiL and
simulators, and build prioritization
⧫ Automated bad practice resolution recommender for CPS
⧫ Methodology for setting-up CI/CD pipelines for CPS
Modular extension of specification-based vulnerability
analysis

14. 14
Elicitation of CPS DevOps Challenges
Interview-based methodology
Interviews’
transcripts
Card Sorting
Early feedback from
COSMOS partners
Validation outside COSMOS
(survey questionnaire)
Pull Requests (PRs) Mining
20 CPS related projects
Bad (and good)
practices,
Challenges,
Barriers,
Mitigation
Analysis Triangulation

15. 15
Finding overview - CPS DevOps challenges
Zampetti, Fiorella; Tamburri, Damian ; Panichella, Sebastiano;
Panichella, Annibale; Canfora, Gerardo; Di Penta, Massimiliano:
Continuous Integration and Delivery practices for Cyber-Physical
systems: An interview-based study. Transactions on Software
Engineering and Methodology.

16. 16
Finding overview - CPS DevOps challenges
Zampetti, Fiorella; Tamburri, Damian ; Panichella, Sebastiano;
Panichella, Annibale; Canfora, Gerardo; Di Penta, Massimiliano:
Continuous Integration and Delivery practices for Cyber-Physical
systems: An interview-based study. Transactions on Software
Engineering and Methodology.

17. 17
Finding overview - CPS DevOps challenges
Zampetti, Fiorella; Tamburri, Damian ; Panichella, Sebastiano;
Panichella, Annibale; Canfora, Gerardo; Di Penta, Massimiliano:
Continuous Integration and Delivery practices for Cyber-Physical
systems: An interview-based study. Transactions on Software
Engineering and Methodology.

18. 18
Challenges vs. mitigation strategies
Zampetti, Fiorella; Tamburri,
Damian ; Panichella, Sebastiano;
Panichella, Annibale; Canfora,
Gerardo; Di Penta, Massimiliano:
Continuous Integration and
Delivery practices for Cyber-
Physical systems: An interview-
based study. Transactions on
Software Engineering and
Methodology.

19. 19
Pipeline analysis tools
Examples of rules:
■ Missing stable branch
■ Arbitrary skip of
steps/stages
■ Build time aging
■ Inappropriate cache
handling
■ Different outcome on
different targets
■ Build stages not properly
ordered

20. 20
Example: Different outcome on different targets
Consistently different build outcome on different targets
linux-aarch64
linux-x86_64
macos-x86_64
macos-aarch64
windows-x86_64
✘
🇭
🇭
🇭
🇭
commit1
linux-aarch64
linux-x86_64
macos-x86_64
macos-aarch64
windows-x86_64
✘
🇭
🇭
🇭
🇭
commit2
……
linux-aarch64
linux-x86_64
macos-x86_64
macos-aarch64
windows-x86_64
✘
🇭
🇭
🇭
🇭
commitN
✘ ✘ ✘
✘
🇭
🇭
Targets

21. 21
Innovation Area 2: V&V and Security
Assessment of DevOps pipelines
WP4: V&V and security assessment of COSMOS DevOps
pipelines
■ Development of Automated Techniques for Software
Testing for CPS
■ Development of Run-time Verification Techniques for
Checking and Diagnosing CPS Executions
■ Development of Solutions for Detecting Security
Vulnerabilities in CPS
COMPONENTS
COSMOS Technology Vision

22. 22
Progress (Innovation Area 2)
WP4 - V&V and Security Assessment of DevOps pipelines (UoL)
■ (Completed) Automated testing (in co-simulated
environments) and specification-based functional
security testing
■ (Completed) Definition of a high-level specification
language for signal-based properties of CPS (SCSL) and
its corresponding (online and offline) monitoring algorithm
■ (Completed) Toolset for code analysis of CPS
■ Ongoing:
⧫ HIL Testing
⧫ Runtime verification for SCSL
⧫ ML-based generation of metamorphic security relations
⧫ Modular extension of specification-based vulnerability
analysis

23. 23
Runtime Verification for SCSL
SCSL, designed for capturing signal
and source code-level behaviour
Instrumentation of the CPS’
source code is performed
automatically, with respect
to the given SCSL specification
Monitoring can either take
place online or offline,
depending on the use case
requirements
Work on diagnosing
violations of SCSL
specifications is ongoing
Team contact: joshua.dawes@uni.lu

24. 24
V&V of DevOps pipelines for CPS
AutoPilot

25. 25
Violation Scenario found by TEA - CARLA

26. 26
Innovation Area 3: Tools for High Quality
CPS Software Evolution
WP5: Development of Tools to support High
Quality CPS Software Evolution
■ Design and Development of Refactoring
Framework for Secure and Reliable CPS
■ Development of Test Case Generation Tools
for Rapid DevOps Iterations
■ Development of Tools to support User-
oriented Maintenance and Testing
COMPONENTS
COSMOS Technology Vision

27. 27
Progress (Innovation Area 3)
■ WP5 - Tools for High Quality CPS Software Evolution
(TUD)
⧫ (Completed) Handbook of refactoring of
production code
⧫ (Completed) Prototype for CPS refactorings
⧫ Ongoing:
● Test Case Generation Tools for Rapid
DevOps Iterations
● Tools to support User-oriented Maintenance and
Testing

28. ■ Goal: Designing a performance antipatterns
detector and a refactoring framework CPS’s
■ First step:
⧫ Gather information
⧫ Reviewed previous studies
⧫ Analyzing code history of open-source CPS
projects
28
Progress (Innovation Area 3)

29. 29
Results: Antipatterns in CPS Projects Waiting time

30. ■ Goal: generating simulation-based
test cases for CPS based on user-
feedback information
■ Context: self-driving cars
■ Challenge: creating realistic road-
maps that are difficult for drivers
(both human and autonomous)
Test Case Generation based on Real-world Maps
30

31. Seeding Strategies based on Real-world Streets
31
Road from Google
Map
Simulation-based
Driving Scenario
Map reconstruction
via Spline
interpolation

32. Seeding Strategies based on Real-world Streets
32
■ How: extracted
“dangerous” roads are
evolved using evolutionary
algorithms (initial
population)
■ Many-objective testing:
⧫ FITEST by Abdessalem et al
(ASE 2018)
⧫ REWOSA (Real-world Street
Search Algorithms)
With better Failures Detection Results

33. 33
Innovation Area 4: Tools for Monitoring,
Self-healing and Self-adaptability of CPS
WP6: Development of Tools to support Monitoring, Self-
healing, and Self-adaptability of CPS in the Field
■ Development and Assessment of CPS Change &
Behavioral Models
■ Developing AI-based Solutions to Support Two-speed
DevOps Cycles for CPS
■ Automated Quality Assessment and Monitoring of
CPS in the Field
■ Development of AI-based Solutions to Increase CPS
Self-adaptability to Diverse Contexts
COMPONENTS
COSMOS Technology Vision

34. 34
Progress (Innovation Area 4)
■ WP6 - Tools for Monitoring, Self-healing, & Self-adaptability of CPS in the
field (ZHAW)
⧫ (Completed) Monitoring CPS Changes in CI/CD pipelines for CPS
⧫ (Completed) Monitoring CPS States and Flaky Behaviors of CPS
⧫ (Completed) Support to Regression Testing for CPS
⧫ (Ongoing)
⧫ Handling and monitoring CPS Behavioral changes in CI/CD
pipelines for CPS
⧫ Quality Assessment, Monitoring, and Self-Adaptability
● Change Propagation
● Vulnerability proneness
● AI-based Support for CPS Self-Adaptability

35. 35
Progress (Innovation Area 4)
Use Cases & CD/CI assets
Change Analysis
&
Regression Testing
Optimization of
CD/CI Assets usage

36. CPS Safety Related Issues of UAVs
Andrea Di Sorbo, Fiorella Zampetti, Corrado A. Visaggio, Massimiliano Di Penta, and Sebastiano
Panichella: Automated Identification and Qualitative Characterization of Safety Concerns Reported
in UAV Software Platforms. Transactions on Software Engineering and Methodology.
36
Progress (Innovation Area 4)
CPS Bugs Taxonomy
Designed a taxonomy of bugs occurring in over 10 CPS projects from
GitHub (Arduino, drones, robotics, automotive, etc.)
Fiorella Zampetti, Ritu Kapur, Massimiliano Di Penta, Sebastiano Panichella: An Empirical
Characterization of Software Bugs in Open-Source Cyber-Physical Systems. Journal of Systems
& Software (JSS).

37. 37
Progress (Innovation Area 4)
Co-occurrences
of hazard
categories
and accident
categories
Hazard Accident
Hazard categories and
corresponding occurrences in our
dataset of 273 safety-related
issues and pull requests.

38. 38
Progress (Innovation Area 4)
CPS Change Taxonomy
CPS
Specific
Non-CPS
Specific
Developed a pipeline analizing
■ Code changes of CPS
■ Dependencies changes of
CPS
■ Changes in operational or
configuration data of
CPSto Diverse Context

39. 39
UAV Test Case Generation in the Neighborhood of Real Flights

40. 40
UAV Test Case Generation in the Neighborhood of Real Flights
Sajad Khatiri, Sebastiano Panichella, Paolo Tonella: Simulation-based Test Case Generation for Unmanned Aerial Vehicles
in the Neighborhood of Real Flights. International Conference on Software Testing, Verification and Validation. (ICST 2023)
§Unsafe / Misbehavior

41. 41
Simulation-based Regression Testing for CPS
Test Generation Oracle Training Prediction
github.com/ChristianBirchler/sdc-scissor
ML-based test selection:
• reduces test execution time of
about 50%
• with identical fault detection
effectiveness

42. 42
WP7: Integration and Validation
WP7: Integration and Validation of COSMOS Solutions on
Industrial Use Cases
■ Task T7.1: Platform Integration and Customization for Use
Cases
■ Task T7.2: Design of Evaluation Methodology
■ Tasks T7.3 - 7: COSMOS Evaluation on 5 Use cases
CPS Use Cases

43. 43
Targeted Impacts
■ Industrial Impacts
⧫ Decreasing percentage of changes that result in CPS failure
⧫ Reducing CPS test execution time and computational resource consumption
⧫ Replacing manually generated tests with automated CPS test coverage
⧫ Improving test effectiveness through tests able to discover more bugs
⧫ Reducing number of security vulnerabilities in CPS
⧫ Reducing component integration and deployment time
⧫ Reducing time to implement a change and make updated CPS operational
⧫ Reducing downtime when deploying new CPS hardware or software

44. 44
Targeted Impacts
■ CPS DevOps Ecosystem
⧫ Project technologies available in open source with actions to build a
European community and ecosystem exploiting DevOps for CPS
■ Standardisation
⧫ Usage of existing industry standards and proposed new standards and
extensions to ensure “plug-n-play” of DevOps tools for CPS
development
■ Academic impact
⧫ Partners produced publications and are contributing to educational
content

45. DevOps for Complex Cyber-physical Systems
45
www.COSMOS-DevOps.org
@COSMOS_DEVOPS
The COSMOS Project has
received funding from
the European Union’s
Horizon 2020 Research
and InnovaCon
Programme under grant
agreement No. 957254.
COSMOS Technology Vision