This document proposes a methodology for model testing of cyber-physical systems (CPSs) using SysML and Simulink models. It presents a case study of modeling an attitude determination and control system. Key points: - A modeling methodology is introduced to specify testable CPS models in SysML and integrate them with Simulink models. This allows for co-simulation of software and functional models. - An execution framework is developed to efficiently co-simulate SysML and Simulink models without user intervention, while generating traces for test evaluation. - An evaluation on the case study shows the approach enables overnight model testing of realistic CPS models. However, integrating Simulink models and specifying software

1. .lusoftware verification & validation
VVS
Enabling Model Testing of
Cyber Physical Systems
Carlos A. González1, Mojtaba Varmazyar1,
Shiva Nejati1, Lionel C. Briand1, and Yago Isasi2
MODELS 2018
1 SnT Centre / University of Luxembourg
2 LuxSpace Sàrl

2. Cyber-Physical Systems (CPSs)
2

3. Testing Cyber-physical Systems:
Challenges
3
complex
systems
complex
interactions
with
environment
hybrid
behavior
intertwining of
software and
hardware
complex
requirements
uncertainty

4. CPS Development Process
4

5. Context

6. Case Study
6
Attitude Determination and Control System
(ADCS)
Earth

7. Model Testing

8. CPS Development Process
8

9. Simulink vs. SysML
• Two industry-strength and complementary modeling languages
for CPS
• Simulink
• Functional and algorithmic computations (MiL-level)
• E.g., controllers, plant and environment models
• SysML
• Software architecture modeling (SiL-level)
• E.g., State machines integrating different controllers, real-time behaviors
9

10. Model Testing
• Enables early testing of CPSs
• Is Scalable
• Can execute a large number of test scenarios (simulations)
• Explore the design space
• Guides the testing process at later stages by identifying high-risk test scenarios
10
• Testing the implemented system on the deployment
platform is necessarily limited
• Shift the bulk of testing from implemented systems
to models of such systems and their environments:

11. Requirements
• Efficient model execution
• To execute thousands of test cases within practical time
• Co-Simulation of heterogeneous models
• Software (SysML) and function (Simulink) models
• No user intervention
• Controllability
• To emulate, as precisely as possible, the expected runtime behavior of the
future deployed system and its environment
• Observability
• To enable automated failure detection
11

12. Our Approach
• A modeling methodology
• To specify testable models of a CPS (relying on SysML)
• To integrate software models with function (Simulink) models capturing
hardware and environment
• A co-simulation framework
• To execute software and function models in a synchronized way
• To enable the generation of adequate execution traces (oracles, guiding test
generation)
• Entirely generated from models
12

13. Related Work
Tool Efficiency Co-Simulation No User Intervention Controllability Observability
TASTE 1 High Medium Low High Low
INTO-CPS 2 ? Medium Low High Low
OpenMETA 3 Medium Low Low Low Low
CyPhySim 4 ? Medium Low Medium Low
13
Even though a great deal of research has been conducted on co-
simulation in the last few years, the most promising approaches fall
short to fulfill the requirements for enabling model testing
1 https://taste.tuxfamily.org/
2 http://into-cps.org/
3 https://openmeta.metamorphsoftware.com/
4 https://ptolemy.berkeley.edu/projects/chess/cyphysim/

14. A SysML-based Methodology
for the Systematic Modeling of
Testable CPS Models

15. What to Model?
15
The level of detail at which the different concepts are modeled is
driven by the minimum information required to enable the faithful
simulation of the SUT.

16. Model Testing Profile
• SysML is not expressive enough to produce testable CPS models, as per
our requirements.
• <<SUT>>: To denote the SUT block
• <<Schedulable>>: To denote SUT tasks
• <<Data>>: To denote data exchanges
• <<Initialization>>: To denote SUT environment operations executed
when the simulation starts
• <<Background>>: To denote SUT environment operations executed
periodically
• <<NotLoggable>>: To control the verbosity of the execution trace
16

17. Modeling Steps
1. SUT Environment
2. SUT Architecture
3. SUT Behavior
17

18. Step 1: Specifying the SUT
Environment (IBD)
18
IBD: SUT, Sensors, actuators, external systems

19. Step 1: Specifying the SUT
Environment (BDD)
19
BDD: Environment blocks, data blocks

20. Step 2: Specifying the SUT
Architecture
20
IBD: Subsystems, data flow

21. Step 3: Specifying the SUT
Behavior (BDD)
21
Interacts with sun sensor
BDD: Subsystem and data blocks

22. Step 3: Specifying the SUT
Behavior (Task)
22
Task: <<Schedulable>>

23. Step 3: Specifying the SUT
Behavior (Activity or State)
23
getSunSensorReading()
SysML specialized actions
Opaque action

24. Simulink and SysML Integration
24
Opaque actions are used to indicate integration points
Simulink

25. A Model Execution Framework
for the Co-Simulation of SysML
and Simulink Models

26. CPS Simulation & Test
26
Test
inputs
CPS SysML
Models
CPS Function
Models
Integrates
Modeling
Co-Simulation
Matlab
Runtime
C Code
Code
Generation Executes
Execution
Traces
Calls
Data
Test input:
- initial state of satellite (speed,
attitude, orbit, date),
- a sequence of time-stamped
external events (tele-commands)

27. Characteristics of the CPS
Simulator
• Simulated time: It emulates the passing of physical time
• Static Cycle Scheduling:
ØSimulated time is divided into time slots
ØTask schedule is computed statically
ØSUT tasks are assigned to time slots depending on their period
• The generated C++ code orchestrates the co-simulation of Simulink models
27

28. Characteristics of the CPS
Simulator
• Behavioral aspects: SysML specialized actions match C++ constructs
• State machines: State design pattern
• Trace generation code: Based on analysis of model
• Traces (timestamped): data is read or written, internal operations or
Simulink models are invoked, or how the system transitions from
one state to another.
28

29. 29
INPUTS: initialStatus, duration, eventList
OUTPUTS: trace
PROCESS:
1 time = 0, timeStep = 0, trace = “”;
2 setSimulatorStatus(initialStatus, trace);
3 runInitializationOperations(trace);
4 while (timeStep < duration) {
5 time = timeStep * timeStepSize;
6 checkExternalEvents(time, eventList, trace);
7 runBackgroundOperations(trace);
8 taskSchedule = generateTaskSchedule(timeStep);
9 foreach (task in taskSchedule) {
10 runTask(task, trace);
11 time = time + task.estimatedCompletionTime;
12 }
13 timeStep = timeStep + 1;
14 }
Simulation Loop

30. Synchronized Co-Simulation
30
CPS
Simulator SW HW
t
0
2 · t
3 · t
1
2 · t
⇥
⇥
⇥
• Both SysML and Simulink models
are simulated in time steps.
• They should preferably be aligned.
• True in our case study.
• Otherwise, simulation results are
more approximative.
Simulator updates or retrieve model state (Simulink blocks)

31. Do we Fulfill Requirements?
• Efficiency: See case study. Main bottleneck: Simulink execution.
• Co-simulation: Integration points helps generate code that handle the interaction
with the Matlab runtime.
• No user intervention: Models contain all the information required for fully
automated co-simulation.
• Controllability: Modeling methodology enables sufficiently precise simulation, i.e.,
task scheduling, data flow.
• Observability: Sufficiently detailed and configurable traces. DSL for property
definition and efficient analysis of traces.
31

32. Evaluation

33. Research Questions
• RQ1: How long does it take for the CPS simulator to execute
test cases?
• RQ2: Is our modeling methodology applicable for realistic
CPSs?
33

34. Answering RQ1 (I)
• Experiment design: execute the models from the case study with the
following two test suites:
Ø Test Suite 1: 20 test cases, each with a simulation time span of 5000 seconds
Ø Test Suite 2: 1000 test cases, each with a simulation time span of 100 seconds
• Coverage criteria: path coverage in the activity diagrams of the SysML model
• Goal: report on the performance of the co-simulation framework, which
affects its applicability
34

35. Answering RQ1 (II)
35
Test Suite 1 Test Suite 2
Number of test cases 20 1000
Simulation time span per test
case
5000 seconds 100 seconds
Aggregate simulation time span ≅ 27.8 hours ≅ 27.8 hours
Time needed to initialize
Simulink models per test case
5198 seconds 4337 seconds
Total execution time 8.25 hours 9.66 hours
Ratio of total execution time to
aggregate simulation time span
0.297 0.348
Experiment results

36. Answering RQ1 (III)
36
• Our co-simulation framework is efficient enough to
enable model testing of CPSs in practical time:
Ø Realistic CPS models can be verified overnight, every day.
Ø Efficiency can be increased by only rerunning the test cases
affected by model changes, or by parallelizing the process.

37. Answering RQ2 (I)
• The SysML model from the case study is of manageable size: 40
blocks, 30 activity diagrams, 1 state machine.
• Building the model required an effort of approximately 2 man-
months, plus an additional man-week to train LuxSpace control
engineers.
But two tasks were particularly laborious…
37

38. Answering RQ2 (II)
• Integrating Simulink models with the SysML model
ØSimulink models must be analyzed to match their numerous
input and output parameters with SysML block attributes.
ØCommunication between software engineers and control and
mechanical engineers can be a source of confusion and requires
special attention.
38

39. Answering RQ2 (III)
• The specification of software behavioral aspects
A high level of precision was needed during the specification of
software behavioral aspects to reduce the complexity of the code
generation process.
39

40. Answering RQ2 (IV)
40
We consider that the cost of applying our modeling
methodology is acceptable:
• CPSs are critical systems that often need to be certified based
on design models.
• Models are typically built once and then updated over the
(long) life span of the system.
• No other way to perform early, design-time testing

41. .lusoftware verification & validation
VVS
Enabling Model Testing of
Cyber Physical Systems
Carlos A. González1, Mojtaba Varmazyar1,
Shiva Nejati1, Lionel C. Briand1, and Yago Isasi2
MODELS 2018
1 SnT Centre / University of Luxembourg
2 LuxSpace Sàrl

42. Next Steps

43. Next Steps
• Search-based test case generation
ØThe test input space is very large, we need to effectively find critical scenarios. We plan on
using test case generation strategies based on metaheuristic search
• Trace execution analysis
ØDefinition and automatic generation of oracles supporting the checking of temporal
properties, timing properties, data assertions, etc
43

44. Next Steps
• Uncertainty
Ø Identify sources of uncertainty in the context of our case study (e.g., sensor precision,
data transmission errors, etc.)
ØExtend our modeling methodology to model uncertainty in an effective manner
ØStudy the impact of uncertainty on the co-simulation of models
ØStudy the impact of uncertainty on the testing process (test case generation, oracle
definition)
44