The document describes a technique called TAUC that combines use case specifications and timed automata to automatically generate test cases for validating timing requirements of safety-critical systems. TAUC models the system functionality and environment as timed automata, identifies test inputs from use case scenarios to trigger state transitions, and employs a coverage-based and metaheuristic search approach to generate a test suite that stresses timing constraints. An evaluation on a case study shows TAUC achieves a 91% fault detection rate, significantly outperforming random and manual testing.

1. .lusoftware verification & validation
VVS
System Testing of Timing Requirements
based on Use Cases and Timed Automata
ChunhuiWang, Fabrizio Pastore, Lionel Briand
ICST’17, Tokyo, March 14th, 2017
University of Luxembourg,
Luxembourg
{chunhui.wang, fabrizio.pastore, lionel.briand}@uni.lu

2. Timing Requirements are crucial for
safety-critical systems
2

3. Timing Requirements: Automotive Domain
Airbag should be fully deployed within 50ms when
a crash event is detected.
3

4. Timing Requirements: Automotive Domain
Car control system should signal overheating to driver when
temperature exceeds a safety threshold for a given time.
Most of safety critical
components are not reliable
when car is overheated
(e.g. Airbag Control).
4

5. Timing Requirements: overheating
•A Temperature Error can be qualified (signaled)
if overheating is detected for at least 3100ms
(avoid signal toggling)
•A Temperature Error should be qualified no
later than 4800ms after overheating is
detected
5

6. Objectives
• Automatically generate test cases to stress timing
requirements with minimal modelling overhead
• Identification of input sequences that increase the
likelihood that the system will break timing constraints
6

7. Context
• FunctionalRequirementsare captured throughan
analyzableform of use case specifications
• Timing Requirementsare modeled by UML state
charts or Timed Automata
• Functionaltest cases automaticallygeneratedfrom
use case specificationswith UMTG
Wang, C., Pastore, F., Goknil, A., Briand, L. C., & Iqbal, Z. (2015, August). UMTG: a toolsetto automaticallygenerate system testcases
from usecase specifications. In Proceedings of the 2015 10th Joint Meeting on Foundations ofSoftwareEngineering (pp. 942-945). ACM.
BASIC FLOW
1. The system REQUESTS temperature FROM temperature sensor.
2. The system VALIDATE THAT the temperature is valid.
3. The system SEND occupant status TO airbag control unit.
……
ALTERNATIVE FLOW
RFS 2
1. The system set temperature error as detected.
7

8. THE ACTOR SEND
THE SYSTEM VALI
THE SYSTEM DIS
THE ACTOR SEND
THE ACTOR SEND
THE SYSTEM VALI
THE SYSTEM DIS
THE ACTOR SEND
THE ACTOR SEND
THE SYSTEM VALI
THE SYSTEM DIS
THE ACTOR SEND
Use Cases
Domain Model
Errors.size()== 0
Status != null
t > 0 && t < 50
OCL constraints
Test Cases
Test Scenarios
UMTG [ISSTA’15]
8

9. Context
• FunctionalRequirementsare captured throughan
analyzableform of use cases specifications
• Timing Requirementsare modeled by UML state
charts or Timed Automata
• Functionaltest cases automaticallygeneratedfrom
use case specificationswith UMTG
Wang, C., Pastore, F., Goknil, A., Briand, L. C., & Iqbal, Z. (2015, August). UMTG: a toolsetto automaticallygenerate system testcases
from usecase specifications. In Proceedings of the 2015 10th Joint Meeting on Foundations ofSoftwareEngineering (pp. 942-945). ACM. 9

10. Strategy
• Use Timed Automata to model Timing Constraints
• Automata are high-level, missing information about concrete test
inputs that may lead to state transitions
• Rely upon Use Case Scenarios to identify the test inputs that
trigger state transitions
• Rely upon edge coverage criteria to identify test sequences
• Integrate an ad-hoc metaheuristic search approach to build a test
suite that effectively stresses timing requirements by maximizing
test case diversity
10

11. Test Generation combining Timed
Automata and Use Case
Specifications
(TAUC)
11

12. Msg1?
Environment Automata
scenario1?
[Error == true]
[Error == false]
scenario2?
TimingReq Automata
ANALYSIS AND DESIGN
1
12

13. Modelling
• Automatathat capture timing propertiesof the functionality
under test (timing requirement automata)
• E.g. qualificationof temperatureerror (overheating)
• Automatathat model the environment:
• Capture frequencyof arrival of interrupts/ messages / inputs
from sensors
13

14. Example: TimingReqAutomata
• Automatathat capture timing propertiesof the functionality
under test, e.g. qualificationof of temperatureerror
14
x >= 3100
unDetected?
isDetected = false, x = 0
detected?
isDetected = true, x = 0
isDetected = false, x = 0
isDetected = true, x = 0
x >= 6100
NotDetectedNotQualified DetectedNotQualified
DetectedQualifiedNotDetectedQualified
x <= 4800
x <= 8100
Invariant
Guard
Updates
unDetected?
detected?
isQualified = false
isQualified = trueA Temperature Error can be qualified if overheating is detected for at
least 3100ms.
A Temperature Error should be qualified no later than 4800ms after
overheating is detected

15. • Automatathat model the environment:
• Capture frequencyof arrival of interrupts/ messages / inputs
from sensors
15
x = 0
x > 41
DMAInterrupt!
x = 0
x >= 50
TimerInterrupt!
x <= 50
x = 0
x > 40
OccupancyInfo!
x > 40
CarInfo!
x > 40
AirbagInfo!
Example: EnvironmentAutomata

16. Msg1?
Environment Automata
scenario1?
[Error == true]
[Error == false]
scenario2?
TimingReq Automata
scenario1 scenario2 scenario3
ANALYSIS AND DESIGN
1
IDENTIFY FUNCTIONAL SCENARIOS
2
IDENTIFY
DEPENDENCIES
3
Automated by UMTG
• Transitionsare triggeredby scenario outputs
• Scenarios can be executed only in specific states
• Useful to translate abstract test activities into concrete test
inputs
16

17. 17
Coverage-based Test Generation
x >= 3100
unDetected?
isDetected = false, x = 0
detected?
isDetected = true, x = 0
isDetected = false, x = 0
isDetected = true, x = 0
x >= 6100
NotDetectedNotQualified
DetectedNotQualified
DetectedQualifiedNotDetectedQualified
x <= 4800
x <= 8100
unDetected?
detected?
isQualified = false
isQualified = true

18. 18
Coverage-based Test Generation
x >= 3100
unDetected?
isDetected = false, x = 0
detected?
isDetected = true, x = 0
isDetected = false, x = 0
isDetected = true, x = 0
x >= 6100
NotDetectedNotQualified
DetectedNotQualified
DetectedQualifiedNotDetectedQualified
x <= 4800
x <= 8100
unDetected?
detected?
isQualified = false
isQualified = true
Test Case
Detected
Wait 4800ms
unDetected
Wait 8100ms

19. How to translate
“error detected”
into a concrete test input?
By Reusing Information
Available in
Use Case Specifications
19

20. How to translate
“error detected”
to a concrete test input?
20
TemperatureSensor.allInstances()->forAll(t | t.temperature > 0 and t.temperature < 45)
UMTG
generates the input
(e.g. temperature = 50)
that leads to the detection of a
temperature error

21. 21
A scenario brings the system into a specific state
(i.e. assigns values to variables)

22. 22
Errors.allInstances()->forAll(e | e.isQualified = false)
Precondition to execute the scenario:
At least one error is in the states with the state variable isQualified = true

23. 23
x >= 3100
unDetected?
detected?
x >= 6100
NotDetectedNotQualified
DetectedNotQualified
x <= 4800
x <= 8100
unDetected?
detected?
isQualified = false
isQualified = true
NotDetectedQualified
DetectedQualified
Scenario
isQualified = true

24. 24
• A scenario bringsthe system into a specific state
(i.e. assign values to variables)
• A scenario can be executed only if the system has reached a
specific state
(i.e. requiresthat state variables have specific values)

25. Msg1?
Environment Automata
scenario1?
[Error == true]
[Error == false]
scenario2?
TimingReq Automata
scenario1 scenario2 scenario3
ANALYSIS AND DESIGN
1
IDENTIFY FUNCTIONAL SCENARIOS
2
IDENTIFY
DEPENDENCIES
3
25

26. scenario1?
[Error == true]
[Error == false]
scenario2?
Msg1?
Timeliness Test Model
Environment Automata
C
scenario1!
Error := true
Scenario Automata
TimingReq Automata
4800ms
Scenario1
ScenarioX
8100ms
Message1
…
GENERATE TIMELINESS
TEST MODEL
4 GENERATE
EXECUTION SCENARIOS
5
TAUC
26

27. 4800ms
Scenario1
ScenarioX
8100ms
Message1
…
4800ms
Scenario1
ScenarioY
8100ms
Message2
…
ScenarioX
ScenarioZ
8100ms
Message1
…
Message3
GeneratedTest Suites
…
4800ms
Scenario1
ScenarioX
8100ms
Message1
…
Maximize test case diversity:
• Execute more, diverse paths, including the same relevant edges
• Execute paths with a diversity of input, interrupts, and messages
• Increase the chances of timeliness failure detection
TAUC
27

28. Case Study
• BodySense,embedded system for detecting occupancy
statusin a car
• Evaluation:
• Effectivenessin terms of fault detection rate
compared to current practice at IEE
28

29. Effectiveness Evaluation
• Derived 323 faultyversion of BodySenseby means of model
mutation
• Each faultyversion of BodySensemodels was generated by
executing a single mutation operatoron the originalmodel
• Comparethe Fault Detection Rate with the test suites
generated by
• Randomtesting
• Expertise-based manualtesting
29

30. 30
Fault Detection Rate
25 50 75 100 122
TAUC 85% 88% 91% 91% 91%
Random 7% 12% 22% 30% 40%
Manual 60%

31. 31
Evaluation of TAUC Test Suite
scenario1?
[Error == true]
[Error == false]
scenario2?
Original model
Mutate
scenario1?
[Error == true]
[Error == false]
scenario2?
scenario1?
[Error == true]
[Error == false]
scenario2?
…
scenario1?
[Error == true]
[Error == false]
scenario2?
TAUC
scenario1?
[Error == true]
[Error == false]
scenario2?
Msg1?
Environment Automata
C
scenario1!
Error := true
Scenario Automata
TimingReq Automata
4800ms
Scenario1
ScenarioX
8100ms
Message1
…
4800ms
Scenario1
ScenarioY
8100ms
Message2
…
ScenarioX
ScenarioZ
8100ms
Message1
…
Message3
Test Suites
…
GENERATETIMELINESS
TEST MODEL
4
GENERATE
EXECUTION SCENARIOS
5
IDENTIFY
DEPENDENCIES
3
scenario1 scenario2 scenario3
IDENTIFY FUNCTIONAL SCENARIOS
2
Msg1?
Environment Automata
scenario1?
[Error == true]
[Error == false]
scenario2?
TimingReq Automata
ANALYSIS AND DESIGN
1
Timing Requirements are crucial for
safety-critical systems
Fault Detection Rate
25 50 75 100 122
TAUC 85% 88% 91% 91% 91%
Random 7% 12% 22% 30% 40%
Manual 60%