EPIcuRus is an approach to automatically generate assumptions for software components in cyber-physical systems modeled in Simulink. It uses machine learning techniques to mine assumptions from test case results. The approach includes generating tests cases using an important feature boundary test generation method, model checking candidate assumptions, and selecting the most informative safe assumptions. An evaluation on industrial case studies found it can learn non-vacuous assumptions for most requirements within a practical time limit, with the important feature boundary test generation performing best.

1. .lusoftware verification & validation
VVS
Mining Assumptions for Software
Components using Machine Learning
Khouloud Gaaloul
Claudio Menghi
Shiva Nejati
Lionel C. Briand
QRA Corp, Canada
David Wolfe
University of Luxembourg,
Luxembourg
University of Ottawa,
Canada
University of Luxembourg,
Luxembourg
University of Ottawa,
Canada
University of Luxembourg,
Luxembourg
University of Luxembourg,
Luxembourg

2. Assumption Generation
Problem
2

3. Cyber-Physical Systems (CPS)
Systems in which
the physical and software components
are deeply intertwined
!3

4. Requirements Check
Model
Requirements
Check
!4
PHASE 1:
Modeling
(Simulink)
PHASE 2:
Verification
PHASE 3:
Coding
PHASE 2:
Verification
Requirements
Check

5. Problem
• Usually, exhaustive verification can not analyse complex
industrial models
• However, model checking can verify models’ components
• When a sub-component is analysed, assumptions are usually
not explicitly documented.
!5

6. • A software component to guide small aircrafts
• Controls the aircraft orientation (Pitch, Roll, and Yaw)
The Autopilot Case Study
!6
Yaw
Roll Pitch
Yaw
Roll Pitch
Yaw
Yaw
Autopilot
Indicators
Actuators

7. The Autopilot Case Study
!7
Altitude Control
Component
When the autopilot is enabled, the aircraft
altitude should reach the desired altitude
within 500 seconds in calm air
= ?
0% 100%
tt+500s
desired altitude
20%40%60%80%

8. Goal
!8
To provide the aircraft with enough boost so that it can reach the desired
altitude, the pilot should manually adjust the power given to the engines
of the aircraft to ensure that the aircraft does not enter a stall condition
Advanced Avionics Handbook

9. Goal
Our goal is to mine assumptions for software
components
!9
Altitude Control
Component
>

10. An assumption is v-safe for a model ! and its requirement ! if
!
M
hAiMh i > v
!10
V-safe Assumption
RequirementModelAssumption
! : Degree of satisfactionv
an assumption is considered 0-safe, if the requirement is satisfied
under the assumption
v 0 v < 0

11. !11
The throttle is
higher than 60%
The throttle is
higher than 80%
is
More informative
than
Informativeness
0% 100%
60%
Altitude Control
Component
>
0% 100%
80%
Altitude Control
Component
>
A1 A2

12. Goal
!12
The goal is to generate the most informative v-safe
assumptions for software components

13. Preriquisites
The model is specified in Simulink
Preriquisite-1 Preriquisite-2
The requirement is in a logical languageThe model is supported by a model checker
Preriquisite-3
The model satisfies neither the requirement
nor its negation
!13
Preriquisite-4

14. Our Solution
(EPIcuRus)
14

15. EPIcuRus (assumPtIon
geneRation approach for CPS)
We propose EPIcuRus, an automated approach to infer
environment assumptions for system components
!15

16. EPIcuRus (assumPtIon
geneRation approach for CPS)
!16

17. EPIcuRus (assumPtIon
geneRation approach for CPS)
!17

18. EPIcuRus (assumPtIon
geneRation approach for CPS)
!18

19. EPIcuRus (assumPtIon
geneRation approach for CPS)
!19
IFBT

20. Generates input signals that are meaningful
The signals are encoded using these parameters:
Test Generation
!20
1
• The input domain
• The number of control points
• The interpolation function

21. Model
Requirements
Check
Input (U) Output (Y)
Test Generation1
Test
Generation
Policy
UR,
ART,
… !21
Pass/Fail
Altitude Control
Component

22. Assumption Generation
!22
2
Node 1
Total data: 1000
Node 2
Total data: 494
98%
Label: fail
Node 3
Total data: 532
Node 4
Total data: 200
100%
Label: pass
Throttle1 < 60Throttle1 < 60Throttle1 < 60 Throttle1 ≥ 60Throttle1 ≥ 60Throttle1 ≥ 60
Throttle1 < 90Throttle1 < 90Throttle1 < 90 Throttle1 ≥ 90Throttle1 ≥ 90Throttle1 ≥ 90
Node 5
Total data: 332
Node 6
Total data: 145
32%
Label: pass
Node 7
Total data: 187
85%
Label: fail
Pwheel < 2Pwheel < 2Pwheel < 2 Pwheel ≥ 2Pwheel ≥ 2Pwheel ≥ 2
Node 1
Total data: 1000
Node 2
Total data: 494
98%
Label: fail
Node 3
Total data: 532
Node 4
Total data: 200
100%
Label: pass
Throttle1 < 60Throttle1 < 60Throttle1 < 60 Throttle1 ≥ 60Throttle1 ≥ 60Throttle1 ≥ 60
Throttle1 < 90Throttle1 < 90Throttle1 < 90 Throttle1 ≥ 90Throttle1 ≥ 90Throttle1 ≥ 90
Node 5
Total data: 332
Node 6
Total data: 145
32%
Label: pass
Node 7
Total data: 187
85%
Label: fail
Pwheel < 2Pwheel < 2Pwheel < 2 Pwheel ≥ 2Pwheel ≥ 2Pwheel ≥ 2
8t 2 [0, t1] : 60  Throttle(t) < 908t 2 [0, t1] : 60  Throttle(t) < 908t 2 [0, t1] : 60  Throttle(t) < 90
60  Throttle1 < 9060  Throttle1 < 9060  Throttle1 < 90

23. Model Checking
!23
3
Exhaustively checks if the obtained assumption is accurate
• QVTrace from QRA Corp, Canada
• SMT-based model checker for Simulink
• Z3, Mathematica
2.4.Accessing QVtrace:
Once the QVtrace server is running, QVtrace will be accessed through a web browser with
the address: http://localhost:2999
If accessing the QVtrace server on a networked computer then use the address:
http://[server_name]:2999.
QVtrace has been fully tested to be accessed with the Google Chrome web browser.
Although other browsers may render QVtrace appropriately, these have not been fully
tested and their performance is not well known. We recommend you use the Google
Chrome browser for QVtrace.
3. Using QVtrace
3.1.Understanding the QVtrace user interface
QVtrace has been designed to optimize the workflow for model-based design analysis. The
interface has three main sections as shown in the image below and described in detail on
the next page.
QVtrace User Manual v0.11.7 qracorp.com of4 21
1
2
3

24. IFBT-Important Feature Boundary
Test
[ ]][ ][
0% 100%
boundary areas
Conjecture-1Conjecture-2
Identifying control points with high
impact on the fitness value and
focusing the search on them
!24
Generating test cases in boundary
areas of the input domain
Two conjectures enable more effective learning of v-safe
assumptions.

25. !25
IFBT-Important Feature Boundary
Test
1. Build a regression tree
How do we generate the test case?
Node 1
Total data: 1000
Node 2
Total data: 494
-0.1
Node 3
Total data: 532
Node 4
Total data: 200
0.1
Throttle1 < 60Throttle1 < 60Throttle1 < 60 Throttle1 ≥ 60Throttle1 ≥ 60Throttle1 ≥ 60
Throttle1 < 90Throttle1 < 90Throttle1 < 90 Throttle1 ≥ 90Throttle1 ≥ 90Throttle1 ≥ 90
Node 5
Total data: 332
Node 6
Total data: 145
-0.3
Node 7
Total data: 187
0.8
Pwheel < 2Pwheel < 2Pwheel < 2 Pwheel ≥ 2Pwheel ≥ 2Pwheel ≥ 2

26. !26
IFBT-Important Feature Boundary
Test
2. Get the most important feature among the
control points (Conjecture-1)
How do we generate the test case?
Node 1
Total data: 1000
Node 2
Total data: 494
-0.1
Node 3
Total data: 532
Node 4
Total data: 200
0.1
Throttle1 < 60Throttle1 < 60Throttle1 < 60 Throttle1 ≥ 60Throttle1 ≥ 60Throttle1 ≥ 60
Throttle1 < 90Throttle1 < 90Throttle1 < 90 Throttle1 ≥ 90Throttle1 ≥ 90Throttle1 ≥ 90
Node 5
Total data: 332
Node 6
Total data: 145
-0.3
Node 7
Total data: 187
0.8
Pwheel < 2Pwheel < 2Pwheel < 2 Pwheel ≥ 2Pwheel ≥ 2Pwheel ≥ 2

27. Node 1
Total data: 1000
Node 2
Total data: 494
-0.1
Node 3
Total data: 532
Node 4
Total data: 200
0.1
Throttle1 < 60Throttle1 < 60Throttle1 < 60 Throttle1 ≥ 60Throttle1 ≥ 60Throttle1 ≥ 60
Throttle1 < 90Throttle1 < 90Throttle1 < 90 Throttle1 ≥ 90Throttle1 ≥ 90Throttle1 ≥ 90
Node 5
Total data: 332
Node 6
Total data: 145
-0.3
Node 7
Total data: 187
0.8
Pwheel < 2Pwheel < 2Pwheel < 2 Pwheel ≥ 2Pwheel ≥ 2Pwheel ≥ 2
3. Extract the test cases that are the closest to the
boundary (Conjecture-2)
!27
IFBT-Important Feature Boundary
Test
How do we generate the test case?

28. Node 1
Total data: 1000
Node 2
Total data: 494
-0.1
Node 3
Total data: 532
Node 4
Total data: 200
0.1
Throttle1 < 60Throttle1 < 60Throttle1 < 60 Throttle1 ≥ 60Throttle1 ≥ 60Throttle1 ≥ 60
Throttle1 < 90Throttle1 < 90Throttle1 < 90 Throttle1 ≥ 90Throttle1 ≥ 90Throttle1 ≥ 90
Node 5
Total data: 332
Node 6
Total data: 145
-0.3
Node 7
Total data: 187
0.8
Pwheel < 2Pwheel < 2Pwheel < 2 Pwheel ≥ 2Pwheel ≥ 2Pwheel ≥ 2
!28
IFBT-Important Feature Boundary
Test
4. Get the ranges associated with the most
important feature
[54 , 66]
[81 , 99]
How do we generate the test case?

29. !29
IFBT-Important Feature Boundary
Test
5. For each test case, get the ranges associated
with the most important feature
How do we generate the test case?
Node 1
Total data: 1000
Node 2
Total data: 494
-0.1
Node 3
Total data: 532
Node 4
Total data: 200
0.1
Throttle1 < 60Throttle1 < 60Throttle1 < 60 Throttle1 ≥ 60Throttle1 ≥ 60Throttle1 ≥ 60
Throttle1 < 90Throttle1 < 90Throttle1 < 90 Throttle1 ≥ 90Throttle1 ≥ 90Throttle1 ≥ 90
Node 5
Total data: 332
Node 6
Total data: 145
-0.3
Node 7
Total data: 187
0.8
Pwheel < 2Pwheel < 2Pwheel < 2 Pwheel ≥ 2Pwheel ≥ 2Pwheel ≥ 2
[54 , 66]
[81 , 99]

30. Implementation
30

31. Implementation
!31
S-TALIRO
https://github.com/SNTSVV/EPIcuRus

32. Evaluation
32

33. Research Questions
• RQ1: Which test case generation policy learns assumptions
most effectively and efficiently?
• RQ2: Can EPIcuRus generate assumptions for real world
Simulink models within a practical time limit?
!33

34. RQ1: Effectiveness and Efficiency
• RQ1: Which test case generation policy learns assumptions
most effectively and efficiently?
!34

35. RQ1: Effectiveness and Efficiency
11 case studies
• Developed by a company in the defence and aerospace sector
• Represent different types of CPS Simulink models
• Each model has a list of (textual) functional requirements
!35

36. RQ1: Effectiveness and Efficiency
• 92 requirements
• 18 satisfy the prerequisites
• 74 violate the prerequisites
• Evaluated the UR, ART, IFBT-UR, IFBT-ART test case generation
policies
!36

37. RQ1: Effectiveness and Efficiency
• For each policy, we run EPIcuRus
• We consider input signals with one (IP), two (IP’) and three (IP”) control points
• we measured among 50 experiment runs:
• V-SAFE: the percentage of runs, in which a v-safe assumption was computed
• AVG_TIME: the average execution time
• INF_IDX: the number of times an assumption was more informative than
another
!37

38. RQ1: Effectiveness and Efficiency
!38
200 300 400 500 600 700 800 900
AVG_TIME (s)
54
56
58
60
62
64
V_SAFE(%)
978
991
1054
1030 UR
ART
IFBT-UR
IFBT-ART

39. RQ1: Effectiveness and Efficiency
!39
Conclusion1: Among the four test case
generation policies we compared, IFBT-UR learns
the most v-safe assumptions in less time
200 300 400 500 600 700 800 900
AVG_TIME (s)
54
56
58
60
62
64
V_SAFE(%)
978
991
1054
1030 UR
ART
IFBT-UR
IFBT-ART

40. RQ1: Effectiveness and Efficiency
!40
Conclusion2: The assumptions learned by IFBT-
UR are more informative than those learned by
other test generation policies
200 300 400 500 600 700 800 900
AVG_TIME (s)
54
56
58
60
62
64
V_SAFE(%)
978
991
1054
1030 UR
ART
IFBT-UR
IFBT-ART

41. RQ2: Usefulness
RQ2: Can EPIcuRus generate assumptions for real world
Simulink models within a practical time limit?
!41

42. • We select the best performing test case generation policy: IFBT-UR
• We considered four models and 18 requirements.
• Among 50 experiment runs:
• We compute the percentage of requirements for which a v-safe
assumption is computed
• We examine the usefulness, the structure and the length of all
the computed assumptions
!42
RQ2: Usefulness

43. RQ2: Usefulness
!43
• EPIcuRus computed a v-safe assumption, within one
hour, for ≈78% of the requirements
• Across all 50 runs, which take around four hours per
requirement, EPIcuRus computed a v-safe assumption
for all the 18 requirements
• EPIcuRus learnt non-vacuous and short assumptions

44. Conclusions
44

45. Conclusions
• EPIcuRus: infer assumptions for software components
• It is applicable to complex signal-based modelling notations
• Combines search-based software testing, Machine Learning
and model checking
• IFBT: A test case generation technique to guide the search
through the most informative features and areas in the
search space
!45

46. Conclusion
• We were able to compute v-safe assumptions
• The computed assumptions are short and non-vacuous
• Assumptions are computed based on a large set of
requirements
!46

47. .lusoftware verification & validation
VVS
Mining Assumptions for Software
Components using Machine Learning
Khouloud Gaaloul
Claudio Menghi
Shiva Nejati
Lionel C. Briand
QRA Corp, Canada
David Wolfe
University of Luxembourg,
Luxembourg
University of Ottawa,
Canada
University of Luxembourg,
Luxembourg
University of Ottawa,
Canada
University of Luxembourg,
Luxembourg
University of Luxembourg,
Luxembourg