MODELS 2018 paper presentation

1. .lusoftware verification & validation
VVS
HITECS: A UML Profile and Analysis
Framework for Hardware-in-the-Loop
Testing of Cyber Physical Systems
Seung Yeob Shin1, Karim Chaouch1, Shiva Nejati1, Mehrdad Sabetzadeh1,
Lionel C. Briand1, and Frank Zimmer2
1 University of Luxembourg
2 SES Networks
MODELS 2018

2. Hardware-in-the-Loop (HiL) Testing
of Cyber Physical Systems
• Actual hardware platform
• Acceptance testing
• Motivating case study
• In-orbit satellite testing
2
Launch effect:
e.g., vibration
Environment:
e.g., space

3. Motivating Case Study:
In-orbit Satellite Testing
3
Source
Synthesizer
Pilot
Synthesizer
Spectrum
Analyzer
High Power
Amplifier
Low Noise
Amplifier
Test instruments
Satellite under test

4. Motivating Case Study:
In-orbit Satellite Testing
4
Test instruments
Satellite under test
• Manipulating hardware devices
• Time-consuming
• Environmental uncertainty
• Risks of hardware damage

5. Risks of Hardware Damage
Manipulating hardware devices may entail potential damage to
the system under test or its environment
5
> threshold

6. Risks of Hardware Damage
Manipulating hardware devices may entail potential damage to
the system under test or its environment
6
Satellite communication block diagram
Channel 1
Channel 2
Damaged amplifier Out of service

7. Time Budget Constraints
• Business constraints (e.g., time to market)
• Environmental constraints
7
Satellite life-cycle
V&V on ground Commercial services
2 months of in-orbit testing
SUTNeighboring satellites
Risk of interference
Launch

8. Uncertainties in Execution Time
• The amount of the actual movement of
hardware depends on the environment
• Hardware recalibration in response to
changes in the environment
(e.g., temperature)
8
Recalibrate
hardware
devices

9. Uncertainties in Execution Time
• The amount of the actual movement of
hardware depends on the environment
• Hardware recalibration in response to
changes in the environment
(e.g., temperature)
Recalibrate
hardware
devices
Risks of Hardware Damage
Manipulating hardware devices may entail potential damage to
the system under test or its environment
> threshold
Risks of Hardware Damage
Manipulating hardware devices may entail potential damage to
the system under test or its environment
Satellite communication block diagram
Channel 1
Channel 2
Damaged amplifier Out of service
Time Budget Constraints
• Business constraints (e.g., time to market)
• Environmental constraints
Satellite life-cycle
V&V on ground Commercial services
2 months of in-orbit testing
SUTNeighboring satellites
Risk of interference
Launch
Problem Statement
How to ensure that
(1) HiL test cases do not accidentally damage hardware
(2) HiL test cases can execute within the time budget
9

10. HITECS: Hardware-In-the-loop
TEst Case Specification and
Analysis Framework

11. Approach Overview
11
Model
Checking
Specify
Test Cases
Simulation
HITECS
Spec.

12. Background:
HITECS Language Development
12
An OMG®
Action Language for Foundational UML™ Publication
Action Language for Foundational UML (Alf)
Concrete Syntax for a UML Action Language
Version 1.1
____________________________________________________
OMG Document Number: formal/2017-07-04
Date: July 2017
Normative reference: http://www.omg.org/spec/ALF/1.1
Machine readable file(s):
Normative: http://www.omg.org/spec/ALF/20170201/Alf-Syntax.xmi
http://www.omg.org/spec/ALF/20170201/Alf-Library.xmi
http://www.omg.org/spec/ALF/20120827/ActionLanguage-Profile.xmi
____________________________________________________
UML Testing Profile (UTP)
Version 2.0 - Beta
____________________________________________________
OMG Document Number: ptc/2017-09-29
Publication Date: September 2017
Normative reference: http://www.omg.org/spec/UTP/2.0/
Machine readable file(s):
Normative: http://www.omg.org/spec/UTP/20170501/utp.xmi
http://www.omg.org/spec/UTP/20170501/utptypes.xmi
http://www.omg.org/spec/UTP/20170501/utpauxiliary.xmi
____________________________________________________
This OMG document replaces the submission document (ad/17-05-01, Alpha). It is an OMG
Adopted Beta Specification and is currently in the finalization phase. Comments on the content
of this document are welcome, and should be directed to issues@omg.org by January 31, 2018.
UML Testing Profile (UTP) UML Action Language (Alf)
Specifying common concepts
in various testing approaches
Specifying executable
behaviors of UML models

13. HITECS Language Profile
13
«metaclass»
TestItem
«stereotype»
HiLComponent
«metaclass»
TestAction
«stereotype»
Initialize
«stereotype»
Cleanup
«stereotype»
Act
«stereotype»
CompOperation
*
«metaclass»
TestComponent
«metaclass»
Property
«stereotype»
CompProperty
*
«stereotype»
Assertion
«metaclass»
Constraint
«metaclass»
Comment
«stereotype»
Annotation
«metaclass»
Activity
«stereotype»
AnnotationSemantics«defines»
1
«stereotype»
HiLTestSuite
«metaclass»
TestSet
«stereotype»
HiLTestSchedule
«metaclass»
TestExecutionSchedule
«runs»
1..*
«stereotype»
HiLTestCase
«stereotype»
Setup
«stereotype»
Teardown
«stereotype»
Oracle
«stereotpe»
TestResult
certainty: Real
verdict: Verdict
«stereotype»
Main
«stereotype»
TestCaseOperation
«metaclass»
TestCase
«metaclass»
TestProcedure
1..3
«determines»
«metaclass»
ArbitrationSpecification
«metaclass»
Property
«stereotype»
TestCaseProperty
*
«enumeration»
ProcedurePhaseKind
SETUP
MAIN
TEARDOWN
«specialized by»
0..1
«metaclass»
Verdict
«has an instance of»
«metaclass»
LiteralSpecification
«stereotype»
Unknown
1
1
1

14. HITECS Language Profile
14
HiL Platform
(SUT and test instruments)
Test Analysis
(assertions and annotations)
Test Behavior
(test cases)
Test Schedule
(test suites and schedulers)

15. HiL Platform
• Components required to execute HiL test cases
• SUT and test instruments
• Blackbox components
15
Synthesizer
generateSignal()

16. Test Behavior
Test cases: parameters, test procedures, and oracles
16
setup
test scenario
teardown
Test case Components
manipulates Synthesizer
Spectrum
analyzer
Amplifier

17. Uncertainties in Environment
A value that can be determined only at the time of actual testing
17
[HITECS specification]
…
noise = unknown;
powerLv = calculatePowerLvBasedOn(noise);
…

18. Confidence in Test Results
18
Application-specific notion capturing the degree of confidence
in test results
Triggering manual inspection
acceptable
boundary
acceptable
boundary
Test result 1 Test result 2

19. Test Analysis
19
Assertions and annotations used by HITECS analysis tasks
assert ( target == satellite.position )
antenna.point ( target )
@exeTime ( historical record )
antenna.point ( target )
Note:
User-defined annotation semantics

20. Test Scheduler
20
A particular order of executing test cases
setup
test scenario
teardown
Test case

21. Approach Overview
Model
Checking
Specify
Test Cases
Simulation
HITECS
Spec.
HITECS Analysis Support

22. HITECS Model Checking
Well-behavedness requirements
• Valid inputs and outputs of HiL component operations
• States of HiL components where they can process data
• Proper initialization and cleaning up of HIL components
22

23. Well-behavedness Verification
23
Domain expertise
e.g., aerospace engineering
Verification techniques
Guidelines
</>

24. Well-behavedness Guidelines
24
Requirements
• Valid inputs and outputs of HiL component operations
• States of HiL components where they can process data
• Proper initialization and cleaning up of HIL components
Guidelines
• Guiding the content of assertions and their expected
locations in HITECS specifications
assert ( target == satellite.position )
antenna.point ( target )
prescribe
ensure

25. HITECS Simulation
Custom annotations
25
HITECS
Simulation
ExeTime
Simulation
HwRisk
Simulation
Custom simulator that analyzes the executions of HiL test cases
@exeTime ( historical record )
antenna.point ( target )

26. Execution Time Simulation
26
Historical record
Execution time of hardware operations
@exeTime
semantics routine
Test suite
(@exeTime annotated)
Estimating distributions of test execution time time
probability
setup
test scenario
teardown
Test case
HITECS simulation
engine

27. Empirical Evaluation
RQ1 Assertion guidelines: Are our guidelines for defining well-
behavedness assertions useful?
RQ2 Model checking: Can HITECS conclusively verify HiL test
case assertions in practical time?
RQ3 Simulation: Can HITECS accurately estimate the execution
times of HiL test cases via simulation?
27

28. Experiment Setting
• Model checking
• Assertion verification
• 609 fault-seeded test cases
(SES: In-orbit test cases)
• Simulation
• Execution time estimation
• Historical data
(SES: Previous in-orbit testing)
28
assert ( target == satellite.position )
antenna.point ( target )
@exeTime ( historical record )
antenna.point ( target )

29. RQ1 Assertion Guidelines
• Result
• All the fault-seeded test cases are detected by guideline-based
assertions
• HITECS helps engineers
• Defining complete and effective assertions for checking
the well-behavedness of HiL test cases
29
# fault-seeded
test cases
# detected test cases
ad-hoc guideline
609 382 609

30. RQ2 Model Checking
• Result
• HITECS model checking verifies all the test cases in less than 2h
• HITECS helps engineers
• Verifying the well-behavedness of HiL test cases in practical time
30
Undetected
Detected
0 5 10 15 20 25
Verification time (second)

31. RQ3 Simulation
• Result
• The estimated execution time distributions
of the test cases include their actual
execution time samples
• HITECS helps engineers
• Accurately estimating the execution time of HiL test cases
31

32. Motivating Case Study:
In-orbit Satellite Testing
Source
Synthesizer
Pilot
Synthesizer
Spectrum
Analyzer
High Power
Amplifier
Low Noise
Amplifier
Test instruments
Satellite under test
HITECS Language Profile
HiL Platform
(SUT and test instruments)
Test Analysis
(assertions and annotations)
Test Behavior
(test cases)
Test Schedule
(test suites and schedulers)
Well-behavedness Verification
Domain expertise
e.g., aerospace engineering
Verification techniques
Guidelines
</>
Execution Time Simulation
Historical record
Execution time of hardware operations
@exeTime
semantics routine
Test suite
(@exeTime annotated)
Estimating distributions of test execution time time
probability
setup
test scenario
teardown
Test case
HITECS simulation
engine
Conclusions
• Executable, uncertainty-aware test modeling language
• Verification method to ensure the well-behavedness of HiL test cases
• Simulation method to estimate the execution time of HiL test cases
• Industrial case study from the satellite domain
32