SlideShare a Scribd company logo
1 of 59
Download to read offline
Making Model-Driven Verification Practical 
and Scalable: 
Experiences and Lessons Learned 
Lionel Briand 
IEEE Fellow, FNR PEARL Chair 
Interdisciplinary Centre for ICT Security, Reliability, and Trust (SnT) 
University of Luxembourg, Luxembourg 
SAM, Valencia, 2014
SnT Software Verification and Validation Lab 
• SnT centre, Est. 2009: Interdisciplinary, 
ICT security-reliability-trust 
• 230 scientists and Ph.D. candidates, 20 
industry partners 
• SVV Lab: Established January 2012, 
www.svv.lu 
• 25 scientists (Research scientists, 
associates, and PhD candidates) 
• Industry-relevant research on system 
dependability: security, safety, reliability 
• Six partners: Cetrel, CTIE, Delphi, SES, 
IEE, Hitec … 
2
An Effective, Collaborative Model of Research 
and Innovation 
Innova3on 
& 
Development 
Schneiderman, 2013 
Basic 
Research 
Applied 
Research 
• Basic and applied research take place in a rich context 
• Basic Research is also driven by problems raised by applied 
research, which is itself fed by innovation and development 
• Publishable research results and focused practical solutions that 
serve an existing market. 3
Collaboration in Practice 
• Well-defined problems in context 
• Realistic evaluation 
• Long term industrial collaborations 
4 
Problem 
Identification 
Problem 
Formulation 
Realistic 
Validation 
Candidate 
Solution(s) 
State of the 
Art Review 
Initial 
Validation 
Training 
Industry 
Partners 
Research 
Groups 
1 
2 
3 
4 
5 
7 
Solution 
Release 
8 
6
Motivations 
• The term “verification” is used in its wider sense: Defect 
detection and removal 
• One important application of models is to drive and automate 
verification 
• In practice, despite significant advances in model-based testing, 
this is not commonly part of practice 
• Decades of research have not yet significantly and widely 
impacted practice 
5
6 
Applicability? 
Scalability?
Definitions 
• Applicable: Can a technology be efficiently and 
effectively applied by engineers in realistic 
conditions? 
– realistic ≠ universal 
– includes usability 
• Scalable: Can a technology be applied on large 
artifacts (e.g., models, data sets, input spaces) and 
still provide useful support within reasonable effort, 
CPU and memory resources? 
7
Outline 
• Project examples, with industry collaborations 
• Lessons learned regarding developing applicable and 
scalable solutions (our research paradigm) 
• Meant to be an interactive talk – I am also here to 
learn 
8
Some Past Projects (< 5 years) 
9 
Company Domain Objective Notation Automation 
Cisco Video conference Robustness testing UML profile Search, model 
transformation 
Kongsberg Maritime Oil & Gas CPU usage UML+MARTE Constraint Solving 
WesternGeco Marine seismic 
acquisition 
Functional testing UML profile + MARTE Search, constraint 
solving 
SES Satellite Functional and 
robustness testing, 
requirements QA 
UML profile Search, Model 
mutation, NLP 
Delphi Automotive systems Testing safety 
+performance 
Matlab/Simulink Search, machine 
learning, statistics 
CTIE Legal & financial Legal Requirements 
testing 
UML Profile Model transformation, 
constraint checking 
HITEC Crisis Support systems Security, Access 
Control 
UML Profile Constraint verification, 
machine learning, 
Search 
CTIE eGovernment Conformance testing UML Profile, BPMN, 
OCL extension 
Domain specific 
language, Constraint 
checking 
IEE Automotive, sensor 
systems 
Functional and 
Robustness testing, 
traceaibility and 
certification 
UML profile, Use Case 
Modeling extension, 
Matlab/Simulink 
NLP, Constraint solving
Testing Closed-Loop Controllers 
References: 
10 
• R. Matinnejad et al., “MiL Testing of Highly Configurable Continuous Controllers: 
Scalable Search Using Surrogate Models”, IEEE/ACM ASE 2014 
• R. Matinnejad et al., “Search-Based Automated Testing of Continuous Controllers: 
Framework, Tool Support, and Case Studies”, Information and Software Technology 
(2014)
Dynamic continuous controllers are present in 
many embedded systems 
11
Development Process (Delphi) 
12 
Hardware-in-the-Loop 
Stage 
Model-in-the-Loop 
Stage 
Simulink Modeling 
Generic 
Functional 
Model 
MiL Testing 
Software-in-the-Loop 
Stage 
Code Generation 
and Integration 
Software Running 
on ECU 
SiL Testing 
Software 
Release 
HiL Testing
Controllers at MIL 
13 
Inputs: Time-dependent variables 
Plant Model 
+ 
+ 
⌃ 
+ 
+ 
- 
e(t) 
actual(t) 
desired(t) 
⌃ 
KP e(t) 
KD 
de(t) 
dt 
KI 
R 
e(t) dt 
P 
I 
D 
output(t) 
Configuration Parameters
Inputs, Outputs, Test Objectives 
14 
Initial Desired 
(ID) 
Desired ValueI (input) 
Actual Value (output) 
Final Desired 
(FD) 
Responsiveness 
Smoothness 
T/2 T 
time 
Stability
Process and Technology 
15 
HeatMap 
Diagram 
1. Exploration 
List of 
Critical 
Domain Regions 
Expert 
Worst-Case 
Scenarios 
Objective 
Functions 
based on 
Requirements 
+ 
Controller-plant 
model 
2. Single-State 
Search 
https://sites.google.com/site/cocotesttool/ 
Initial Desired (ID) 
Final Desired (FD) 
Worst Case(s)?
Process and Technology (2) 
16 
HeatMap 
Diagram 
1. Exploration 
List of 
Critical 
Domain Regions 
Expert 
Objective 
Functions 
based on 
Requirements 
+ 
Controller-plant 
model 
(a) Liveness (b) Smoothness
Process and Technology (3) 
17 
List of 
Critical 
Domain Regions 
Expert 
Worst-Case 
Scenarios 
2. Single-State 
Search 
time 
Desired Value 
Actual Value 
0 1 2 
1.0 
0.9 
0.8 
0.7 
0.6 
0.5 
0.4 
0.3 
0.2 
0.1 
0.0 
Initial Desired 
Final Desired
Challenges, Solutions 
• Achieving scalability with configuration parameters: 
– Simulink simulations are expensive 
– Sensitivity analysis to eliminate irrelevant 
parameters 
– Machine learning (Regression trees) to partition 
the space automatically and identify high-risk 
areas 
– Surrogate modeling (statistical and machine 
learning prediction) to predict properties and avoid 
simulation, when possible 
18
Results 
• Automotive controllers on Electronics Control Units 
• Our approach enabled our partner to identify worst-case 
scenarios that were much worse than known 
and expected scenarios, entirely automatically 
19
Fault Localisation in Simulink Models 
Reference: 
20 
• Bing Liu et al., “Kanvoo: Fault Localization in Simulink Models”, submitted
Context and Problem 
21 
• Simulink models 
– are complex 
• hundreds of blocks and lines 
• many hierarchy levels 
• continuous functions 
– might be faulty 
• output signals do not match 
• wrong connection of lines 
• wrong operators in blocks 
• Debugging Simulink models is 
– difficult 
– time-consuming 
– but yet crucial 
• Automated techniques to support debugging?
Context and Problem (2) 
22 
• Simulink models 
– are complex 
• hundreds of blocks and lines 
• many hierarchy levels 
• continuous functions 
– might be faulty 
• output signals do not match 
• wrong connection of lines 
• wrong operators in blocks 
• Debugging Simulink models is 
– difficult 
– time-consuming 
– but yet crucial 
• Automated techniques to support debugging?
Solution Overview 
23 
Specifica+on% 
Test%Case%Genera+on% 
Test%Suite% ?% Test%Oracle% 
Test%Case%Execu+on% 
Slicing% 
Ranking% 
Coverage%Reports% 
PASS/FAIL% 
Results% 
Simulink%Model% 
Model%Slices% 
0.95% 0.71% 0.62% 0.43% 
Ranked%Blocks% 
Any test strategy 
Provided by Matlab tool 
One slice for each 
test case and output 
For each test case 
and output, or overall
Evaluation and Challenges 
• Good accuracy overall: 5-6% blocks must be inspected on 
average to detect faults 
• But less accurate predictions for certain faults: Low observability 
• Possible Solution: Augment test oracle (observability) 
– Use subsystems outputs 
– Iterate at deeper levels of hierarchy 
– Tradeoff: cost of test oracle vs. debugging effort 
– 2.3% blocks on average 
• 5-6%: still too many blocks for certain models 
• Information requirements to help further filtering blocks? 
24
Modeling and Verifying Legal 
Requirements 
Reference: 
25 
• G. Soltana et al., “ UML for Modeling Procedural Legal Rule”, IEEE/ACM MODELS 
2014 
• M. Adedjouma et al., “Automated Detection and Resolution of Legal Cross 
References”, RE 2014
Context and Problem 
26 
• CTIE: Government computer centre in Luxembourg 
• Large government (information) systems 
• Implement legal requirements, must comply with the 
law 
• The law usually leaves room for interpretation and 
changes on a regular basis, many cross-references 
• Involves many stakeholders, IT specialists but also 
legal experts, etc.
Article Example 
Art. 105bis […]The commuting expenses deduction (FD) is 
defined as a function over the distance between the principal 
town of the municipality on whose territory the taxpayer's home 
is located and the place of taxpayer’s work. The distance is 
measured in units of distance expressing the kilometric distance 
between [principal] towns. A ministerial regulation provides 
these distances. 
The amount of the deduction is calculated as follows: 
If the distance exceeds 4 units but is less than 30 units, the 
deduction is € 99 per unit of distance. 
The first 4 units does not trigger any deduction and the 
deduction for a distance exceeding 30 units is limited to € 
2,574.
Project Objectives 
28 
Objective Benefits 
Specification of legal requirements 
• including rationale and traceability 
to the text of law 
• 
Make interpretation of the law explicit 
• Improve communication 
• Prerequisite for automation 
Checking consistency of legal 
requirements 
• Prevent errors in the interpretation of 
the law to propagate 
Automated test strategies for checking 
system compliance to legal requirements 
• Provide effective and scalable ways to 
Run-time verification mechanisms to verify compliance 
check compliance with legal 
requirements 
Analyzing the impact of changes in the 
law 
• Decrease costs and risks associated 
with change 
• Make change more predictable
Solution Overview 
29 
Test cases 
Actual 
software 
system 
Traces to 
Traces to 
Analyzable 
interpretation 
Generates of the law 
Results match? 
Impact of legal 
changes 
Simulates
Research Steps 
30 
2. Build UML 
profile 
3. Model 
Transformation 
to enable V&V 
• What information 
content should we 
expect? 
• What are the 
complexity factors? 
• Explicit means for 
capturing information 
requirements 
• Basis for modeling 
methodology 
• Target: Legal experts 
and IT specialists 
• Target existing 
automation techniques 
• Solvers for testing 
• MATLAB for simulation 
1. Conduct 
grounded 
theory study
Example 
31 
Art. 105bis […]The commuting 
expenses deduction (FD) is 
defined as a function over the 
distance between the principal 
town of the municipality on whose 
territory the taxpayer's home is 
located and the place of 
taxpayer’s work. The distance is 
measured in units of distance 
expressing the kilometric distance 
between [principal] towns. A 
ministerial regulation provides 
these distances. 
Interpretation + Traces
Example 
32 
The amount of the deduction is 
calculated as follows: 
If the distance exceeds 4 units but is 
less than 30 units, the deduction is € 
99 per unit of distance. 
The first 4 units does not trigger any 
deduction and the deduction for a 
distance exceeding 30 units is limited 
to € 2,574. 
Interpretation + Traces
Challenges and Results 
• Profile must lead to models that are: 
– understandable by both IT specialists and legal experts 
– precise enough to enable model transformation and support 
our objectives 
– tutorials, many modeling sessions with legal experts 
• In theory, though such legal requirements can be captured by 
OCL constraints alone, this is not applicable 
• That is why we resorted to customized activity modeling, 
carefully combined with a simple subset of OCL 
• Many traces to law articles, dependencies among articles: 
automated detection (NLP) of cross-references 33
Run-Time Verification of 
Business Processes 
References: 
34 
• W. Dou et al., “OCLR: a More Expressive, Pattern-based Temporal Extension of 
OCL”, ECMFA 2014 
• W. Dou et al., “Revisiting Model-Driven Engineering for Run-Time Verification of 
Business Processes”, IEEE/ACM SAM 2014 
• W. Dou et al., “A Model-Driven Approach to Offline Trace Checking of Temporal 
Properties with OCL”, submitted
Context and Problem 
• CTIE: Government Computing Centre of Luxembourg 
• E-government systems mostly implemented as business 
processes 
• CTIE models these business processes 
• Business models have temporal properties that must be 
checked 
– Temporal logics not applicable 
– Limited tool support (scalability) 
• Goal: Efficient, scalable, and practical off-line and run-time 
verification 35
Solution Overview 
36
Solution Overview 
37 
• We identified patterns based on 
analyzing many properties of real 
business process models 
• Properties must be defined based on 
business process models (BPMN) 
according to modeling methodology 
at CTIE (applicability) 
• The goal was to achieve usability 
• Early adoption by our partner
Solution Overview 
38 
• Want to transform the checking of 
temporal constraints into checking 
regular constraints on trace 
conceptual model 
• OCL engines (Eclipse) are our target, 
to rely on mature technology 
(scalability) 
• Defined extension of OCL to facilitate 
translation 
• Target: IT specialists, BPM analysts
Scalability Analysis 
• Analyzed 47 properties in Identity Card Management System 
• “Once a card request is approved, the applicant is notified within 
three days; this notification has to occur before the production of 
the card is started.” 
• Scalability: Check time as a function of trace size … 
39 
100 
200 
300 
400 
500 
600 
700 
800 
900 
1,000 
100 
80 
60 
40 
20 
0 
Trace Size (k) 
Average Check Time (s) 
P7 P8 P9 
(a) globally / P7–P9 
100 
200 
300 
400 
500 
600 
700 
800 
900 
1,000 
4,500 
4,000 
3,500 
3,000 
2,500 
2,000 
1,500 
1,000 
500 
0 
Trace Size (k) 
Average Check Time (s) 
P10 P11 P12 
(b) globally / P10–P12 
10 
20 
30 
40 
50 
60 
70 
80 
90 
100 
30 
25 
20 
15 
10 
5 
0 
Boundary Position (k) 
Average Check Time (s) 
P17 P18 
P19 P20 
(c) before / P17–P20 
Fig. 3: Average check time of properties with globally and before scopes
Schedulability Analysis and Stress 
Testing 
References: 
40 
• S. Nejati, S. Di Alesio, M. Sabetzadeh, and L. Briand, “Modeling and analysis of cpu 
usage in safety-critical embedded systems to support stress testing,” in IEEE/ACM 
MODELS 2012. 
• S. Di Alesio, S. Nejati, L. Briand. A. Gotlieb, “Stress Testing of Task Deadlines: A 
Constraint Programming Approach”, ISSRE 2013, San Jose, USA 
• S. Di Alesio, S. Nejati, L. Briand. A. Gotlieb, “Worst-Case Scheduling of Software 
Tasks – A Constraint Optimization Model to Support Performance Testing, Constraint 
Programming (CP), 2014
Problem 
• Real-time, concurrent systems (RTCS) have concurrent 
interdependent tasks which have to finish before their deadlines 
• Some task properties depend on the environment, some are 
design choices 
• Tasks can trigger other tasks, and can share computational 
resources with other tasks 
• Schedulability analysis encompasses techniques that try to 
predict whether all (critical) tasks are schedulable, i.e., meet 
their deadlines 
• Stress testing runs carefully selected test cases that have a high 
probability of leading to deadline misses 
• Testing in RTCS is typically expensive, e.g., hardware in the 
loop 
41
Arrival Times Determine Deadline Misses 
42 
0 
1 
2 
3 
4 
5 
6 
7 
8 
9 
j0, j1 , j2 arrive at at0 , at1 , at2 and must 
finish before dl0 , dl1 , dl2 
at1 
J1 can miss its deadline dl1 depending on 
when at2 occurs! 
0 
1 
2 
3 
4 
5 
6 
7 
8 
9 
j0 j1 j2 j0 j1 j2 
at0 
dl0 
at1 dl2 
dl1 
at2 
T 
T 
at0 
dl0 dl1 
at2 
dl2
Context 
43 
Drivers 
(Software-Hardware Interface) 
Control Modules Alarm Devices 
(Hardware) 
Real-Time Operating System 
Multicore Architecture 
Monitor gas leaks and fire in oil 
extraction platforms
Challenges and Solutions 
• Ranges for arrival times form a very large input space 
• Task interdependencies and properties constrain 
what parts of the space are feasible 
• We re-expressed the problem as a constraint 
optimisation problem 
• Constraint programming 
44
Constraint Optimization 
45 
Constraint Optimization Problem 
Static Properties of Tasks 
(Constants) 
Dynamic Properties of Tasks 
(Variables) 
OS Scheduler Behaviour 
Performance Requirement 
(Objective Function) 
(Constraints)
Process and Technologies 
46 
UML Modeling (e.g., 
MARTE) 
Constraint Optimization 
System Design Design Model (Time 
Optimization Problem 
(Find arrival times that maximize the 
chance of deadline misses) 
System Platform 
Solutions 
(Task arrival times likely to 
lead to deadline misses) 
Deadline Misses 
Analysis 
and Concurrency 
Information) 
INPUT 
OUTPUT 
Stress Test Cases 
Constraint 
Programming 
(CP)
Challenges and Solutions (2) 
• Scalability problem: Constraint programming (e.g., 
IBM CPLEX) cannot handle such large input spaces 
(CPU, memory) 
• Solution: Combine metaheuristic search and 
constraint programming 
– metaheuristic search identifies high risk regions in 
the input space 
– constraint programming finds provably worst-case 
schedules within these (limited) regions 
47
Process and Technologies 
48 
UML Modeling (e.g., 
MARTE) 
Constraint Optimization 
System Design Design Model (Time 
Optimization Problem 
(Find arrival times that maximize the 
chance of deadline misses) 
System Platform 
Solutions 
(Task arrival times likely to 
lead to deadline misses) 
Deadline Misses 
Analysis 
and Concurrency 
Information) 
INPUT 
OUTPUT 
Genetic 
Algorithms 
(GA) 
Stress Test Cases 
Constraint 
Programming 
(CP)
Applicable? Scalable? 
49
Scalability examples 
• This is the most common challenge in practice 
• Testing closed-loop controllers 
– Large input and configuration space 
– Smart search optimization heuristics (machine learning) 
• Fault localization 
– Large number of blocks and lines in Simulink models 
– Even a small percentage of blocks to inspect can be 
impractical 
– Additional information to support decision making? 
Incremental fault localisation? 
• Schedulability analysis and stress testing 
– Constraint programming cannot scale by itself 
– Must be carefully combined with genetic algorithms 
50
Scalability examples (2) 
• Verifying legal requirements 
– Traceability to the law is complex 
– Many provisions and articles 
– Many dependencies within the law 
– Natural Language Processing: Cross references, support for 
identifying missing modeling concepts 
• Run-time Verification of Business Processes 
– Traces can be large and properties complex to verify 
– Transformation of temporal properties into regular OCL 
properties, defined on a trace conceptual model 
– Incremental verification at regular time intervals 
– Heuristics to identify subtraces to verify 
51
Scalability: Lessons Learned 
• Scalability must be part of the problem definition and solution 
from the start, not a refinement or an after-thought 
• It often involves heuristics, e.g., meta-heuristic search, NLP, 
machine learning, statistics 
• Scalability often leads to solutions that offer “best answers” 
within time constraints, not guarantees 
• Solutions to scalability are multi-disciplinary 
• Scalability analysis should be a component of every research 
project – otherwise it is unlikely to be adopted in practice 
• How many papers in MODELS or SAM do include even a 
minimal form of scalability analysis? 
52
Applicability 
• Definition? 
• Usability: Can the target user population efficiently apply it? 
• Assumptions: Are working assumptions realistic, e.g., realistic 
information requirements? 
• Integration into the development process, e.g., are required 
inputs available in the right form and level of precision? 
53
Applicability examples 
• Testing closed-loop controllers 
– Working assumption: availability of sufficiently precise plant 
(environment) models 
– Means to visualize relevant properties in the search space 
(inputs, configuration), to get an overview and focus search 
on high-risk areas 
• Schedulability analysis and stress testing 
– Availability of tasks architecture models 
– Precise WCET analysis 
– Applicability requires to assess risk based on near-deadline 
misses 
54
Applicability examples (2) 
• Fault localization: 
– Trade-off between # of model outputs considered versus cost of 
test oracles 
– Better understanding of the mental process and information 
requirements for fault localization 
• Run-time verification of business process models 
– Temporal logic not usable by analysts 
– Language closer to natural language, directly tied to business 
process model 
– Easy transition to industry strength constraint checker 
• Verifying legal requirements 
– Modeling notation must be shared by IT specialists and legal 
experts 
– One common representation for many applications, with traces 
to the law to handle changes 
– Multiple model transformation targets 
55
Applicability: Lessons Learned 
• Make working assumptions explicit: Determine the 
context of applicability 
• Make sure those working assumptions are at least 
realistic in some industrial domain and context 
• Assumptions don’t need to be universally true – they 
rarely are anyway 
• Run usability studies – do it for real! 
56
Conclusions 
• In most research endeavors, applicability and scalability are an after-thought, 
a secondary consideration, when at all considered 
• Implicit assumptions are often made, often unrealistic in any context 
• Problem definition in a vacuum 
• Not adapted to research in an engineering discipline 
• Leads to limited impact 
• Research in model-based V&V is necessarily multi-disciplinary 
• User studies are required and far too rare 
• In engineering research, there is no substitute to reality 
57
Acknowledgements 
PhD. Students: 
• Marwa Shousha 
• Reza Matinnejad 
• Stefano Di Alesio 
• Wei Dou 
• Ghanem Soltana 
• Bing Liu 
Scientists: 
• Shiva Nejati 
• Mehrdad Sabetzadeh 
• Domenico Bianculli 
• Arnaud Gotlieb 
• Yvan Labiche 
58
Making Model-Driven Verification Practical 
and Scalable: Experiences and Lessons 
Learned 
Lionel Briand 
IEEE Fellow, FNR PEARL Chair 
Interdisciplinary Centre for ICT Security, Reliability, and Trust (SnT) 
University of Luxembourg, Luxembourg 
SAM, Valencia, 2014 
SVV lab: svv.lu 
SnT: www.securityandtrust.lu

More Related Content

What's hot

Temporal EMF: A temporal metamodeling platform
Temporal EMF: A temporal metamodeling platformTemporal EMF: A temporal metamodeling platform
Temporal EMF: A temporal metamodeling platformJordi Cabot
 
Towards a UML and IFML mapping to GraphQL
Towards a UML and IFML mapping to GraphQLTowards a UML and IFML mapping to GraphQL
Towards a UML and IFML mapping to GraphQLJordi Cabot
 
Is there a future for Model Transformation Languages?
Is there a future for Model Transformation Languages?Is there a future for Model Transformation Languages?
Is there a future for Model Transformation Languages?Jordi Cabot
 
Web technologies: Model Driven Engineering
Web technologies: Model Driven EngineeringWeb technologies: Model Driven Engineering
Web technologies: Model Driven EngineeringPiero Fraternali
 
Lightweight Model-Driven Engineering
Lightweight Model-Driven EngineeringLightweight Model-Driven Engineering
Lightweight Model-Driven EngineeringJordi Cabot
 
All Researchers Should Become Entrepreneurs
All Researchers Should Become EntrepreneursAll Researchers Should Become Entrepreneurs
All Researchers Should Become EntrepreneursJordi Cabot
 
A UML profile for OData Web APIs
A UML profile for OData Web APIsA UML profile for OData Web APIs
A UML profile for OData Web APIsJordi Cabot
 
Model driven software engineering in practice book - chapter 7 - Developing y...
Model driven software engineering in practice book - chapter 7 - Developing y...Model driven software engineering in practice book - chapter 7 - Developing y...
Model driven software engineering in practice book - chapter 7 - Developing y...Marco Brambilla
 
Model-Driven Software Engineering in Practice - Chapter 1 - Introduction
Model-Driven Software Engineering in Practice - Chapter 1 - IntroductionModel-Driven Software Engineering in Practice - Chapter 1 - Introduction
Model-Driven Software Engineering in Practice - Chapter 1 - IntroductionMarco Brambilla
 
Towards the intelligent generation of software modeling components
Towards the intelligent generation of software modeling componentsTowards the intelligent generation of software modeling components
Towards the intelligent generation of software modeling componentsLola Burgueño
 
Lessons learned from building a commercial bot development platform
Lessons learned from building a commercial bot development platformLessons learned from building a commercial bot development platform
Lessons learned from building a commercial bot development platformJordi Cabot
 
Model-Driven Software Engineering in Practice - Chapter 4 - Model-Driven Arch...
Model-Driven Software Engineering in Practice - Chapter 4 - Model-Driven Arch...Model-Driven Software Engineering in Practice - Chapter 4 - Model-Driven Arch...
Model-Driven Software Engineering in Practice - Chapter 4 - Model-Driven Arch...Jordi Cabot
 
CG2010 Tailored Code Generators
CG2010 Tailored Code GeneratorsCG2010 Tailored Code Generators
CG2010 Tailored Code GeneratorsPedro J. Molina
 
Developing Open Source MDE Tools / Eclipse Stories and Lessons Learned - OSS4...
Developing Open Source MDE Tools / Eclipse Stories and Lessons Learned - OSS4...Developing Open Source MDE Tools / Eclipse Stories and Lessons Learned - OSS4...
Developing Open Source MDE Tools / Eclipse Stories and Lessons Learned - OSS4...Hugo Bruneliere
 
Towards Smart Modeling (Environments)
Towards Smart Modeling (Environments)Towards Smart Modeling (Environments)
Towards Smart Modeling (Environments)Benoit Combemale
 
Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 ...
Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 ...Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 ...
Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 ...Jordi Cabot
 
Domain-specific Modeling and Code Generation for Cross-platform Mobile and Io...
Domain-specific Modeling and Code Generation for Cross-platform Mobile and Io...Domain-specific Modeling and Code Generation for Cross-platform Mobile and Io...
Domain-specific Modeling and Code Generation for Cross-platform Mobile and Io...Università degli Studi dell'Aquila
 
Model-Driven Software Engineering in Practice - Chapter 2 - MDSE Principles
Model-Driven Software Engineering in Practice - Chapter 2 - MDSE PrinciplesModel-Driven Software Engineering in Practice - Chapter 2 - MDSE Principles
Model-Driven Software Engineering in Practice - Chapter 2 - MDSE PrinciplesMarco Brambilla
 

What's hot (20)

Temporal EMF: A temporal metamodeling platform
Temporal EMF: A temporal metamodeling platformTemporal EMF: A temporal metamodeling platform
Temporal EMF: A temporal metamodeling platform
 
Towards a UML and IFML mapping to GraphQL
Towards a UML and IFML mapping to GraphQLTowards a UML and IFML mapping to GraphQL
Towards a UML and IFML mapping to GraphQL
 
Is there a future for Model Transformation Languages?
Is there a future for Model Transformation Languages?Is there a future for Model Transformation Languages?
Is there a future for Model Transformation Languages?
 
Web technologies: Model Driven Engineering
Web technologies: Model Driven EngineeringWeb technologies: Model Driven Engineering
Web technologies: Model Driven Engineering
 
Lightweight Model-Driven Engineering
Lightweight Model-Driven EngineeringLightweight Model-Driven Engineering
Lightweight Model-Driven Engineering
 
All Researchers Should Become Entrepreneurs
All Researchers Should Become EntrepreneursAll Researchers Should Become Entrepreneurs
All Researchers Should Become Entrepreneurs
 
A UML profile for OData Web APIs
A UML profile for OData Web APIsA UML profile for OData Web APIs
A UML profile for OData Web APIs
 
Model driven software engineering in practice book - chapter 7 - Developing y...
Model driven software engineering in practice book - chapter 7 - Developing y...Model driven software engineering in practice book - chapter 7 - Developing y...
Model driven software engineering in practice book - chapter 7 - Developing y...
 
Model-Driven Software Engineering in Practice - Chapter 1 - Introduction
Model-Driven Software Engineering in Practice - Chapter 1 - IntroductionModel-Driven Software Engineering in Practice - Chapter 1 - Introduction
Model-Driven Software Engineering in Practice - Chapter 1 - Introduction
 
Towards the intelligent generation of software modeling components
Towards the intelligent generation of software modeling componentsTowards the intelligent generation of software modeling components
Towards the intelligent generation of software modeling components
 
Lessons learned from building a commercial bot development platform
Lessons learned from building a commercial bot development platformLessons learned from building a commercial bot development platform
Lessons learned from building a commercial bot development platform
 
Model-Driven Software Engineering in Practice - Chapter 4 - Model-Driven Arch...
Model-Driven Software Engineering in Practice - Chapter 4 - Model-Driven Arch...Model-Driven Software Engineering in Practice - Chapter 4 - Model-Driven Arch...
Model-Driven Software Engineering in Practice - Chapter 4 - Model-Driven Arch...
 
CG2010 Tailored Code Generators
CG2010 Tailored Code GeneratorsCG2010 Tailored Code Generators
CG2010 Tailored Code Generators
 
Introducing MDSD
Introducing MDSDIntroducing MDSD
Introducing MDSD
 
Developing Open Source MDE Tools / Eclipse Stories and Lessons Learned - OSS4...
Developing Open Source MDE Tools / Eclipse Stories and Lessons Learned - OSS4...Developing Open Source MDE Tools / Eclipse Stories and Lessons Learned - OSS4...
Developing Open Source MDE Tools / Eclipse Stories and Lessons Learned - OSS4...
 
Towards Smart Modeling (Environments)
Towards Smart Modeling (Environments)Towards Smart Modeling (Environments)
Towards Smart Modeling (Environments)
 
Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 ...
Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 ...Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 ...
Explicating and Reasoning with Model Uncertainty by Marsha Chechik (ECMFA'14 ...
 
Domain-specific Modeling and Code Generation for Cross-platform Mobile and Io...
Domain-specific Modeling and Code Generation for Cross-platform Mobile and Io...Domain-specific Modeling and Code Generation for Cross-platform Mobile and Io...
Domain-specific Modeling and Code Generation for Cross-platform Mobile and Io...
 
Model-Driven Software Engineering in Practice - Chapter 2 - MDSE Principles
Model-Driven Software Engineering in Practice - Chapter 2 - MDSE PrinciplesModel-Driven Software Engineering in Practice - Chapter 2 - MDSE Principles
Model-Driven Software Engineering in Practice - Chapter 2 - MDSE Principles
 
01 mde principles
01 mde principles01 mde principles
01 mde principles
 

Similar to Making Model-Driven Verification Practical and Scalable: Experiences and Lessons Learned

Bart Knaack - The Truth About Model-Based Quality Improvements
Bart Knaack - The Truth About Model-Based Quality ImprovementsBart Knaack - The Truth About Model-Based Quality Improvements
Bart Knaack - The Truth About Model-Based Quality ImprovementsTEST Huddle
 
Keynote SBST 2014 - Search-Based Testing
Keynote SBST 2014 - Search-Based TestingKeynote SBST 2014 - Search-Based Testing
Keynote SBST 2014 - Search-Based TestingLionel Briand
 
From Model-based to Model and Simulation-based Systems Architectures
From Model-based to Model and Simulation-based Systems ArchitecturesFrom Model-based to Model and Simulation-based Systems Architectures
From Model-based to Model and Simulation-based Systems ArchitecturesObeo
 
Automated Testing of Autonomous Driving Assistance Systems
Automated Testing of Autonomous Driving Assistance SystemsAutomated Testing of Autonomous Driving Assistance Systems
Automated Testing of Autonomous Driving Assistance SystemsLionel Briand
 
ES2022-Minh-Nguyen-ShapingTestsIntoModelsForAutomatedTCGeneration.pdf
ES2022-Minh-Nguyen-ShapingTestsIntoModelsForAutomatedTCGeneration.pdfES2022-Minh-Nguyen-ShapingTestsIntoModelsForAutomatedTCGeneration.pdf
ES2022-Minh-Nguyen-ShapingTestsIntoModelsForAutomatedTCGeneration.pdfMinh Nguyen
 
Software Engineering Research: Leading a Double-Agent Life.
Software Engineering Research: Leading a Double-Agent Life.Software Engineering Research: Leading a Double-Agent Life.
Software Engineering Research: Leading a Double-Agent Life.Lionel Briand
 
Enabling Automated Software Testing with Artificial Intelligence
Enabling Automated Software Testing with Artificial IntelligenceEnabling Automated Software Testing with Artificial Intelligence
Enabling Automated Software Testing with Artificial IntelligenceLionel Briand
 
Incremental Queries and Transformations for Engineering Critical Systems
Incremental Queries and Transformations for Engineering Critical SystemsIncremental Queries and Transformations for Engineering Critical Systems
Incremental Queries and Transformations for Engineering Critical SystemsÁkos Horváth
 
DSD-INT 2014 - OpenMI Symposium - Federated modelling of Critical Infrastruct...
DSD-INT 2014 - OpenMI Symposium - Federated modelling of Critical Infrastruct...DSD-INT 2014 - OpenMI Symposium - Federated modelling of Critical Infrastruct...
DSD-INT 2014 - OpenMI Symposium - Federated modelling of Critical Infrastruct...Deltares
 
Survey on Software Defect Prediction
Survey on Software Defect PredictionSurvey on Software Defect Prediction
Survey on Software Defect PredictionSung Kim
 
20220914-MBT-Experiences-SB1-final.pptx
20220914-MBT-Experiences-SB1-final.pptx20220914-MBT-Experiences-SB1-final.pptx
20220914-MBT-Experiences-SB1-final.pptxMinh Nguyen
 
Scalable and Cost-Effective Model-Based Software Verification and Testing
Scalable and Cost-Effective Model-Based Software Verification and TestingScalable and Cost-Effective Model-Based Software Verification and Testing
Scalable and Cost-Effective Model-Based Software Verification and TestingLionel Briand
 
Testing Machine Learning-enabled Systems: A Personal Perspective
Testing Machine Learning-enabled Systems: A Personal PerspectiveTesting Machine Learning-enabled Systems: A Personal Perspective
Testing Machine Learning-enabled Systems: A Personal PerspectiveLionel Briand
 
Traceability Beyond Source Code: An Elusive Target?
Traceability Beyond Source Code: An Elusive Target?Traceability Beyond Source Code: An Elusive Target?
Traceability Beyond Source Code: An Elusive Target?Lionel Briand
 
Emerging standards and support organizations within engineering simulation
Emerging standards and support organizations within engineering simulation Emerging standards and support organizations within engineering simulation
Emerging standards and support organizations within engineering simulation Modelon
 
Automated Discovery of Performance Regressions in Enterprise Applications
Automated Discovery of Performance Regressions in Enterprise ApplicationsAutomated Discovery of Performance Regressions in Enterprise Applications
Automated Discovery of Performance Regressions in Enterprise ApplicationsSAIL_QU
 
Modelon Modelica executable requirements Ansys Conference 2016
Modelon Modelica executable requirements Ansys Conference 2016Modelon Modelica executable requirements Ansys Conference 2016
Modelon Modelica executable requirements Ansys Conference 2016Modelon
 
How to Guarantee Continuous Value from your Test Automation
How to Guarantee Continuous Value from your Test AutomationHow to Guarantee Continuous Value from your Test Automation
How to Guarantee Continuous Value from your Test AutomationPerfecto by Perforce
 
Scalable Software Testing and Verification of Non-Functional Properties throu...
Scalable Software Testing and Verification of Non-Functional Properties throu...Scalable Software Testing and Verification of Non-Functional Properties throu...
Scalable Software Testing and Verification of Non-Functional Properties throu...Lionel Briand
 

Similar to Making Model-Driven Verification Practical and Scalable: Experiences and Lessons Learned (20)

Bart Knaack - The Truth About Model-Based Quality Improvements
Bart Knaack - The Truth About Model-Based Quality ImprovementsBart Knaack - The Truth About Model-Based Quality Improvements
Bart Knaack - The Truth About Model-Based Quality Improvements
 
Keynote SBST 2014 - Search-Based Testing
Keynote SBST 2014 - Search-Based TestingKeynote SBST 2014 - Search-Based Testing
Keynote SBST 2014 - Search-Based Testing
 
From Model-based to Model and Simulation-based Systems Architectures
From Model-based to Model and Simulation-based Systems ArchitecturesFrom Model-based to Model and Simulation-based Systems Architectures
From Model-based to Model and Simulation-based Systems Architectures
 
Automated Testing of Autonomous Driving Assistance Systems
Automated Testing of Autonomous Driving Assistance SystemsAutomated Testing of Autonomous Driving Assistance Systems
Automated Testing of Autonomous Driving Assistance Systems
 
ES2022-Minh-Nguyen-ShapingTestsIntoModelsForAutomatedTCGeneration.pdf
ES2022-Minh-Nguyen-ShapingTestsIntoModelsForAutomatedTCGeneration.pdfES2022-Minh-Nguyen-ShapingTestsIntoModelsForAutomatedTCGeneration.pdf
ES2022-Minh-Nguyen-ShapingTestsIntoModelsForAutomatedTCGeneration.pdf
 
Software Engineering Research: Leading a Double-Agent Life.
Software Engineering Research: Leading a Double-Agent Life.Software Engineering Research: Leading a Double-Agent Life.
Software Engineering Research: Leading a Double-Agent Life.
 
Enabling Automated Software Testing with Artificial Intelligence
Enabling Automated Software Testing with Artificial IntelligenceEnabling Automated Software Testing with Artificial Intelligence
Enabling Automated Software Testing with Artificial Intelligence
 
Incremental Queries and Transformations for Engineering Critical Systems
Incremental Queries and Transformations for Engineering Critical SystemsIncremental Queries and Transformations for Engineering Critical Systems
Incremental Queries and Transformations for Engineering Critical Systems
 
DSD-INT 2014 - OpenMI Symposium - Federated modelling of Critical Infrastruct...
DSD-INT 2014 - OpenMI Symposium - Federated modelling of Critical Infrastruct...DSD-INT 2014 - OpenMI Symposium - Federated modelling of Critical Infrastruct...
DSD-INT 2014 - OpenMI Symposium - Federated modelling of Critical Infrastruct...
 
ppt2.pptx
ppt2.pptxppt2.pptx
ppt2.pptx
 
Survey on Software Defect Prediction
Survey on Software Defect PredictionSurvey on Software Defect Prediction
Survey on Software Defect Prediction
 
20220914-MBT-Experiences-SB1-final.pptx
20220914-MBT-Experiences-SB1-final.pptx20220914-MBT-Experiences-SB1-final.pptx
20220914-MBT-Experiences-SB1-final.pptx
 
Scalable and Cost-Effective Model-Based Software Verification and Testing
Scalable and Cost-Effective Model-Based Software Verification and TestingScalable and Cost-Effective Model-Based Software Verification and Testing
Scalable and Cost-Effective Model-Based Software Verification and Testing
 
Testing Machine Learning-enabled Systems: A Personal Perspective
Testing Machine Learning-enabled Systems: A Personal PerspectiveTesting Machine Learning-enabled Systems: A Personal Perspective
Testing Machine Learning-enabled Systems: A Personal Perspective
 
Traceability Beyond Source Code: An Elusive Target?
Traceability Beyond Source Code: An Elusive Target?Traceability Beyond Source Code: An Elusive Target?
Traceability Beyond Source Code: An Elusive Target?
 
Emerging standards and support organizations within engineering simulation
Emerging standards and support organizations within engineering simulation Emerging standards and support organizations within engineering simulation
Emerging standards and support organizations within engineering simulation
 
Automated Discovery of Performance Regressions in Enterprise Applications
Automated Discovery of Performance Regressions in Enterprise ApplicationsAutomated Discovery of Performance Regressions in Enterprise Applications
Automated Discovery of Performance Regressions in Enterprise Applications
 
Modelon Modelica executable requirements Ansys Conference 2016
Modelon Modelica executable requirements Ansys Conference 2016Modelon Modelica executable requirements Ansys Conference 2016
Modelon Modelica executable requirements Ansys Conference 2016
 
How to Guarantee Continuous Value from your Test Automation
How to Guarantee Continuous Value from your Test AutomationHow to Guarantee Continuous Value from your Test Automation
How to Guarantee Continuous Value from your Test Automation
 
Scalable Software Testing and Verification of Non-Functional Properties throu...
Scalable Software Testing and Verification of Non-Functional Properties throu...Scalable Software Testing and Verification of Non-Functional Properties throu...
Scalable Software Testing and Verification of Non-Functional Properties throu...
 

More from Lionel Briand

Precise and Complete Requirements? An Elusive Goal
Precise and Complete Requirements? An Elusive GoalPrecise and Complete Requirements? An Elusive Goal
Precise and Complete Requirements? An Elusive GoalLionel Briand
 
Large Language Models for Test Case Evolution and Repair
Large Language Models for Test Case Evolution and RepairLarge Language Models for Test Case Evolution and Repair
Large Language Models for Test Case Evolution and RepairLionel Briand
 
Metamorphic Testing for Web System Security
Metamorphic Testing for Web System SecurityMetamorphic Testing for Web System Security
Metamorphic Testing for Web System SecurityLionel Briand
 
Simulator-based Explanation and Debugging of Hazard-triggering Events in DNN-...
Simulator-based Explanation and Debugging of Hazard-triggering Events in DNN-...Simulator-based Explanation and Debugging of Hazard-triggering Events in DNN-...
Simulator-based Explanation and Debugging of Hazard-triggering Events in DNN-...Lionel Briand
 
Fuzzing for CPS Mutation Testing
Fuzzing for CPS Mutation TestingFuzzing for CPS Mutation Testing
Fuzzing for CPS Mutation TestingLionel Briand
 
Data-driven Mutation Analysis for Cyber-Physical Systems
Data-driven Mutation Analysis for Cyber-Physical SystemsData-driven Mutation Analysis for Cyber-Physical Systems
Data-driven Mutation Analysis for Cyber-Physical SystemsLionel Briand
 
Many-Objective Reinforcement Learning for Online Testing of DNN-Enabled Systems
Many-Objective Reinforcement Learning for Online Testing of DNN-Enabled SystemsMany-Objective Reinforcement Learning for Online Testing of DNN-Enabled Systems
Many-Objective Reinforcement Learning for Online Testing of DNN-Enabled SystemsLionel Briand
 
ATM: Black-box Test Case Minimization based on Test Code Similarity and Evolu...
ATM: Black-box Test Case Minimization based on Test Code Similarity and Evolu...ATM: Black-box Test Case Minimization based on Test Code Similarity and Evolu...
ATM: Black-box Test Case Minimization based on Test Code Similarity and Evolu...Lionel Briand
 
Black-box Safety Analysis and Retraining of DNNs based on Feature Extraction ...
Black-box Safety Analysis and Retraining of DNNs based on Feature Extraction ...Black-box Safety Analysis and Retraining of DNNs based on Feature Extraction ...
Black-box Safety Analysis and Retraining of DNNs based on Feature Extraction ...Lionel Briand
 
PRINS: Scalable Model Inference for Component-based System Logs
PRINS: Scalable Model Inference for Component-based System LogsPRINS: Scalable Model Inference for Component-based System Logs
PRINS: Scalable Model Inference for Component-based System LogsLionel Briand
 
Revisiting the Notion of Diversity in Software Testing
Revisiting the Notion of Diversity in Software TestingRevisiting the Notion of Diversity in Software Testing
Revisiting the Notion of Diversity in Software TestingLionel Briand
 
Applications of Search-based Software Testing to Trustworthy Artificial Intel...
Applications of Search-based Software Testing to Trustworthy Artificial Intel...Applications of Search-based Software Testing to Trustworthy Artificial Intel...
Applications of Search-based Software Testing to Trustworthy Artificial Intel...Lionel Briand
 
Autonomous Systems: How to Address the Dilemma between Autonomy and Safety
Autonomous Systems: How to Address the Dilemma between Autonomy and SafetyAutonomous Systems: How to Address the Dilemma between Autonomy and Safety
Autonomous Systems: How to Address the Dilemma between Autonomy and SafetyLionel Briand
 
Mathematicians, Social Scientists, or Engineers? The Split Minds of Software ...
Mathematicians, Social Scientists, or Engineers? The Split Minds of Software ...Mathematicians, Social Scientists, or Engineers? The Split Minds of Software ...
Mathematicians, Social Scientists, or Engineers? The Split Minds of Software ...Lionel Briand
 
Reinforcement Learning for Test Case Prioritization
Reinforcement Learning for Test Case PrioritizationReinforcement Learning for Test Case Prioritization
Reinforcement Learning for Test Case PrioritizationLionel Briand
 
Mutation Analysis for Cyber-Physical Systems: Scalable Solutions and Results ...
Mutation Analysis for Cyber-Physical Systems: Scalable Solutions and Results ...Mutation Analysis for Cyber-Physical Systems: Scalable Solutions and Results ...
Mutation Analysis for Cyber-Physical Systems: Scalable Solutions and Results ...Lionel Briand
 
On Systematically Building a Controlled Natural Language for Functional Requi...
On Systematically Building a Controlled Natural Language for Functional Requi...On Systematically Building a Controlled Natural Language for Functional Requi...
On Systematically Building a Controlled Natural Language for Functional Requi...Lionel Briand
 
Efficient Online Testing for DNN-Enabled Systems using Surrogate-Assisted and...
Efficient Online Testing for DNN-Enabled Systems using Surrogate-Assisted and...Efficient Online Testing for DNN-Enabled Systems using Surrogate-Assisted and...
Efficient Online Testing for DNN-Enabled Systems using Surrogate-Assisted and...Lionel Briand
 
Guidelines for Assessing the Accuracy of Log Message Template Identification ...
Guidelines for Assessing the Accuracy of Log Message Template Identification ...Guidelines for Assessing the Accuracy of Log Message Template Identification ...
Guidelines for Assessing the Accuracy of Log Message Template Identification ...Lionel Briand
 
A Theoretical Framework for Understanding the Relationship between Log Parsin...
A Theoretical Framework for Understanding the Relationship between Log Parsin...A Theoretical Framework for Understanding the Relationship between Log Parsin...
A Theoretical Framework for Understanding the Relationship between Log Parsin...Lionel Briand
 

More from Lionel Briand (20)

Precise and Complete Requirements? An Elusive Goal
Precise and Complete Requirements? An Elusive GoalPrecise and Complete Requirements? An Elusive Goal
Precise and Complete Requirements? An Elusive Goal
 
Large Language Models for Test Case Evolution and Repair
Large Language Models for Test Case Evolution and RepairLarge Language Models for Test Case Evolution and Repair
Large Language Models for Test Case Evolution and Repair
 
Metamorphic Testing for Web System Security
Metamorphic Testing for Web System SecurityMetamorphic Testing for Web System Security
Metamorphic Testing for Web System Security
 
Simulator-based Explanation and Debugging of Hazard-triggering Events in DNN-...
Simulator-based Explanation and Debugging of Hazard-triggering Events in DNN-...Simulator-based Explanation and Debugging of Hazard-triggering Events in DNN-...
Simulator-based Explanation and Debugging of Hazard-triggering Events in DNN-...
 
Fuzzing for CPS Mutation Testing
Fuzzing for CPS Mutation TestingFuzzing for CPS Mutation Testing
Fuzzing for CPS Mutation Testing
 
Data-driven Mutation Analysis for Cyber-Physical Systems
Data-driven Mutation Analysis for Cyber-Physical SystemsData-driven Mutation Analysis for Cyber-Physical Systems
Data-driven Mutation Analysis for Cyber-Physical Systems
 
Many-Objective Reinforcement Learning for Online Testing of DNN-Enabled Systems
Many-Objective Reinforcement Learning for Online Testing of DNN-Enabled SystemsMany-Objective Reinforcement Learning for Online Testing of DNN-Enabled Systems
Many-Objective Reinforcement Learning for Online Testing of DNN-Enabled Systems
 
ATM: Black-box Test Case Minimization based on Test Code Similarity and Evolu...
ATM: Black-box Test Case Minimization based on Test Code Similarity and Evolu...ATM: Black-box Test Case Minimization based on Test Code Similarity and Evolu...
ATM: Black-box Test Case Minimization based on Test Code Similarity and Evolu...
 
Black-box Safety Analysis and Retraining of DNNs based on Feature Extraction ...
Black-box Safety Analysis and Retraining of DNNs based on Feature Extraction ...Black-box Safety Analysis and Retraining of DNNs based on Feature Extraction ...
Black-box Safety Analysis and Retraining of DNNs based on Feature Extraction ...
 
PRINS: Scalable Model Inference for Component-based System Logs
PRINS: Scalable Model Inference for Component-based System LogsPRINS: Scalable Model Inference for Component-based System Logs
PRINS: Scalable Model Inference for Component-based System Logs
 
Revisiting the Notion of Diversity in Software Testing
Revisiting the Notion of Diversity in Software TestingRevisiting the Notion of Diversity in Software Testing
Revisiting the Notion of Diversity in Software Testing
 
Applications of Search-based Software Testing to Trustworthy Artificial Intel...
Applications of Search-based Software Testing to Trustworthy Artificial Intel...Applications of Search-based Software Testing to Trustworthy Artificial Intel...
Applications of Search-based Software Testing to Trustworthy Artificial Intel...
 
Autonomous Systems: How to Address the Dilemma between Autonomy and Safety
Autonomous Systems: How to Address the Dilemma between Autonomy and SafetyAutonomous Systems: How to Address the Dilemma between Autonomy and Safety
Autonomous Systems: How to Address the Dilemma between Autonomy and Safety
 
Mathematicians, Social Scientists, or Engineers? The Split Minds of Software ...
Mathematicians, Social Scientists, or Engineers? The Split Minds of Software ...Mathematicians, Social Scientists, or Engineers? The Split Minds of Software ...
Mathematicians, Social Scientists, or Engineers? The Split Minds of Software ...
 
Reinforcement Learning for Test Case Prioritization
Reinforcement Learning for Test Case PrioritizationReinforcement Learning for Test Case Prioritization
Reinforcement Learning for Test Case Prioritization
 
Mutation Analysis for Cyber-Physical Systems: Scalable Solutions and Results ...
Mutation Analysis for Cyber-Physical Systems: Scalable Solutions and Results ...Mutation Analysis for Cyber-Physical Systems: Scalable Solutions and Results ...
Mutation Analysis for Cyber-Physical Systems: Scalable Solutions and Results ...
 
On Systematically Building a Controlled Natural Language for Functional Requi...
On Systematically Building a Controlled Natural Language for Functional Requi...On Systematically Building a Controlled Natural Language for Functional Requi...
On Systematically Building a Controlled Natural Language for Functional Requi...
 
Efficient Online Testing for DNN-Enabled Systems using Surrogate-Assisted and...
Efficient Online Testing for DNN-Enabled Systems using Surrogate-Assisted and...Efficient Online Testing for DNN-Enabled Systems using Surrogate-Assisted and...
Efficient Online Testing for DNN-Enabled Systems using Surrogate-Assisted and...
 
Guidelines for Assessing the Accuracy of Log Message Template Identification ...
Guidelines for Assessing the Accuracy of Log Message Template Identification ...Guidelines for Assessing the Accuracy of Log Message Template Identification ...
Guidelines for Assessing the Accuracy of Log Message Template Identification ...
 
A Theoretical Framework for Understanding the Relationship between Log Parsin...
A Theoretical Framework for Understanding the Relationship between Log Parsin...A Theoretical Framework for Understanding the Relationship between Log Parsin...
A Theoretical Framework for Understanding the Relationship between Log Parsin...
 

Recently uploaded

Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...
Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...
Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...drm1699
 
OpenChain Webinar: AboutCode and Beyond - End-to-End SCA
OpenChain Webinar: AboutCode and Beyond - End-to-End SCAOpenChain Webinar: AboutCode and Beyond - End-to-End SCA
OpenChain Webinar: AboutCode and Beyond - End-to-End SCAShane Coughlan
 
BusinessGPT - Security and Governance for Generative AI
BusinessGPT  - Security and Governance for Generative AIBusinessGPT  - Security and Governance for Generative AI
BusinessGPT - Security and Governance for Generative AIAGATSoftware
 
What Goes Wrong with Language Definitions and How to Improve the Situation
What Goes Wrong with Language Definitions and How to Improve the SituationWhat Goes Wrong with Language Definitions and How to Improve the Situation
What Goes Wrong with Language Definitions and How to Improve the SituationJuha-Pekka Tolvanen
 
Weeding your micro service landscape.pdf
Weeding your micro service landscape.pdfWeeding your micro service landscape.pdf
Weeding your micro service landscape.pdftimtebeek1
 
[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse
[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse
[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypseTomasz Kowalczewski
 
The mythical technical debt. (Brooke, please, forgive me)
The mythical technical debt. (Brooke, please, forgive me)The mythical technical debt. (Brooke, please, forgive me)
The mythical technical debt. (Brooke, please, forgive me)Roberto Bettazzoni
 
GraphSummit Milan - Neo4j: The Art of the Possible with Graph
GraphSummit Milan - Neo4j: The Art of the Possible with GraphGraphSummit Milan - Neo4j: The Art of the Possible with Graph
GraphSummit Milan - Neo4j: The Art of the Possible with GraphNeo4j
 
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale Ibrida
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale IbridaUNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale Ibrida
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale IbridaNeo4j
 
GraphSummit Milan - Visione e roadmap del prodotto Neo4j
GraphSummit Milan - Visione e roadmap del prodotto Neo4jGraphSummit Milan - Visione e roadmap del prodotto Neo4j
GraphSummit Milan - Visione e roadmap del prodotto Neo4jNeo4j
 
Community is Just as Important as Code by Andrea Goulet
Community is Just as Important as Code by Andrea GouletCommunity is Just as Important as Code by Andrea Goulet
Community is Just as Important as Code by Andrea GouletAndrea Goulet
 
Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...
Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...
Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...Abortion Clinic
 
AzureNativeQumulo_HPC_Cloud_Native_Benchmarks.pdf
AzureNativeQumulo_HPC_Cloud_Native_Benchmarks.pdfAzureNativeQumulo_HPC_Cloud_Native_Benchmarks.pdf
AzureNativeQumulo_HPC_Cloud_Native_Benchmarks.pdfryanfarris8
 
Workshop - Architecting Innovative Graph Applications- GraphSummit Milan
Workshop -  Architecting Innovative Graph Applications- GraphSummit MilanWorkshop -  Architecting Innovative Graph Applications- GraphSummit Milan
Workshop - Architecting Innovative Graph Applications- GraphSummit MilanNeo4j
 
[GRCPP] Introduction to concepts (C++20)
[GRCPP] Introduction to concepts (C++20)[GRCPP] Introduction to concepts (C++20)
[GRCPP] Introduction to concepts (C++20)Dimitrios Platis
 
Evolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI EraEvolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI Eraconfluent
 
Automate your OpenSIPS config tests - OpenSIPS Summit 2024
Automate your OpenSIPS config tests - OpenSIPS Summit 2024Automate your OpenSIPS config tests - OpenSIPS Summit 2024
Automate your OpenSIPS config tests - OpenSIPS Summit 2024Andreas Granig
 

Recently uploaded (20)

Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...
Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...
Abortion Pills For Sale WhatsApp[[+27737758557]] In Birch Acres, Abortion Pil...
 
OpenChain Webinar: AboutCode and Beyond - End-to-End SCA
OpenChain Webinar: AboutCode and Beyond - End-to-End SCAOpenChain Webinar: AboutCode and Beyond - End-to-End SCA
OpenChain Webinar: AboutCode and Beyond - End-to-End SCA
 
BusinessGPT - Security and Governance for Generative AI
BusinessGPT  - Security and Governance for Generative AIBusinessGPT  - Security and Governance for Generative AI
BusinessGPT - Security and Governance for Generative AI
 
Abortion Clinic In Stanger ](+27832195400*)[ 🏥 Safe Abortion Pills In Stanger...
Abortion Clinic In Stanger ](+27832195400*)[ 🏥 Safe Abortion Pills In Stanger...Abortion Clinic In Stanger ](+27832195400*)[ 🏥 Safe Abortion Pills In Stanger...
Abortion Clinic In Stanger ](+27832195400*)[ 🏥 Safe Abortion Pills In Stanger...
 
What Goes Wrong with Language Definitions and How to Improve the Situation
What Goes Wrong with Language Definitions and How to Improve the SituationWhat Goes Wrong with Language Definitions and How to Improve the Situation
What Goes Wrong with Language Definitions and How to Improve the Situation
 
Weeding your micro service landscape.pdf
Weeding your micro service landscape.pdfWeeding your micro service landscape.pdf
Weeding your micro service landscape.pdf
 
[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse
[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse
[GeeCON2024] How I learned to stop worrying and love the dark silicon apocalypse
 
The mythical technical debt. (Brooke, please, forgive me)
The mythical technical debt. (Brooke, please, forgive me)The mythical technical debt. (Brooke, please, forgive me)
The mythical technical debt. (Brooke, please, forgive me)
 
GraphSummit Milan - Neo4j: The Art of the Possible with Graph
GraphSummit Milan - Neo4j: The Art of the Possible with GraphGraphSummit Milan - Neo4j: The Art of the Possible with Graph
GraphSummit Milan - Neo4j: The Art of the Possible with Graph
 
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale Ibrida
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale IbridaUNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale Ibrida
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale Ibrida
 
GraphSummit Milan - Visione e roadmap del prodotto Neo4j
GraphSummit Milan - Visione e roadmap del prodotto Neo4jGraphSummit Milan - Visione e roadmap del prodotto Neo4j
GraphSummit Milan - Visione e roadmap del prodotto Neo4j
 
Community is Just as Important as Code by Andrea Goulet
Community is Just as Important as Code by Andrea GouletCommunity is Just as Important as Code by Andrea Goulet
Community is Just as Important as Code by Andrea Goulet
 
Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...
Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...
Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...
 
AzureNativeQumulo_HPC_Cloud_Native_Benchmarks.pdf
AzureNativeQumulo_HPC_Cloud_Native_Benchmarks.pdfAzureNativeQumulo_HPC_Cloud_Native_Benchmarks.pdf
AzureNativeQumulo_HPC_Cloud_Native_Benchmarks.pdf
 
Abortion Clinic In Pongola ](+27832195400*)[ 🏥 Safe Abortion Pills In Pongola...
Abortion Clinic In Pongola ](+27832195400*)[ 🏥 Safe Abortion Pills In Pongola...Abortion Clinic In Pongola ](+27832195400*)[ 🏥 Safe Abortion Pills In Pongola...
Abortion Clinic In Pongola ](+27832195400*)[ 🏥 Safe Abortion Pills In Pongola...
 
Workshop - Architecting Innovative Graph Applications- GraphSummit Milan
Workshop -  Architecting Innovative Graph Applications- GraphSummit MilanWorkshop -  Architecting Innovative Graph Applications- GraphSummit Milan
Workshop - Architecting Innovative Graph Applications- GraphSummit Milan
 
[GRCPP] Introduction to concepts (C++20)
[GRCPP] Introduction to concepts (C++20)[GRCPP] Introduction to concepts (C++20)
[GRCPP] Introduction to concepts (C++20)
 
Evolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI EraEvolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI Era
 
Automate your OpenSIPS config tests - OpenSIPS Summit 2024
Automate your OpenSIPS config tests - OpenSIPS Summit 2024Automate your OpenSIPS config tests - OpenSIPS Summit 2024
Automate your OpenSIPS config tests - OpenSIPS Summit 2024
 
Abortion Pill Prices Polokwane ](+27832195400*)[ 🏥 Women's Abortion Clinic in...
Abortion Pill Prices Polokwane ](+27832195400*)[ 🏥 Women's Abortion Clinic in...Abortion Pill Prices Polokwane ](+27832195400*)[ 🏥 Women's Abortion Clinic in...
Abortion Pill Prices Polokwane ](+27832195400*)[ 🏥 Women's Abortion Clinic in...
 

Making Model-Driven Verification Practical and Scalable: Experiences and Lessons Learned

  • 1. Making Model-Driven Verification Practical and Scalable: Experiences and Lessons Learned Lionel Briand IEEE Fellow, FNR PEARL Chair Interdisciplinary Centre for ICT Security, Reliability, and Trust (SnT) University of Luxembourg, Luxembourg SAM, Valencia, 2014
  • 2. SnT Software Verification and Validation Lab • SnT centre, Est. 2009: Interdisciplinary, ICT security-reliability-trust • 230 scientists and Ph.D. candidates, 20 industry partners • SVV Lab: Established January 2012, www.svv.lu • 25 scientists (Research scientists, associates, and PhD candidates) • Industry-relevant research on system dependability: security, safety, reliability • Six partners: Cetrel, CTIE, Delphi, SES, IEE, Hitec … 2
  • 3. An Effective, Collaborative Model of Research and Innovation Innova3on & Development Schneiderman, 2013 Basic Research Applied Research • Basic and applied research take place in a rich context • Basic Research is also driven by problems raised by applied research, which is itself fed by innovation and development • Publishable research results and focused practical solutions that serve an existing market. 3
  • 4. Collaboration in Practice • Well-defined problems in context • Realistic evaluation • Long term industrial collaborations 4 Problem Identification Problem Formulation Realistic Validation Candidate Solution(s) State of the Art Review Initial Validation Training Industry Partners Research Groups 1 2 3 4 5 7 Solution Release 8 6
  • 5. Motivations • The term “verification” is used in its wider sense: Defect detection and removal • One important application of models is to drive and automate verification • In practice, despite significant advances in model-based testing, this is not commonly part of practice • Decades of research have not yet significantly and widely impacted practice 5
  • 7. Definitions • Applicable: Can a technology be efficiently and effectively applied by engineers in realistic conditions? – realistic ≠ universal – includes usability • Scalable: Can a technology be applied on large artifacts (e.g., models, data sets, input spaces) and still provide useful support within reasonable effort, CPU and memory resources? 7
  • 8. Outline • Project examples, with industry collaborations • Lessons learned regarding developing applicable and scalable solutions (our research paradigm) • Meant to be an interactive talk – I am also here to learn 8
  • 9. Some Past Projects (< 5 years) 9 Company Domain Objective Notation Automation Cisco Video conference Robustness testing UML profile Search, model transformation Kongsberg Maritime Oil & Gas CPU usage UML+MARTE Constraint Solving WesternGeco Marine seismic acquisition Functional testing UML profile + MARTE Search, constraint solving SES Satellite Functional and robustness testing, requirements QA UML profile Search, Model mutation, NLP Delphi Automotive systems Testing safety +performance Matlab/Simulink Search, machine learning, statistics CTIE Legal & financial Legal Requirements testing UML Profile Model transformation, constraint checking HITEC Crisis Support systems Security, Access Control UML Profile Constraint verification, machine learning, Search CTIE eGovernment Conformance testing UML Profile, BPMN, OCL extension Domain specific language, Constraint checking IEE Automotive, sensor systems Functional and Robustness testing, traceaibility and certification UML profile, Use Case Modeling extension, Matlab/Simulink NLP, Constraint solving
  • 10. Testing Closed-Loop Controllers References: 10 • R. Matinnejad et al., “MiL Testing of Highly Configurable Continuous Controllers: Scalable Search Using Surrogate Models”, IEEE/ACM ASE 2014 • R. Matinnejad et al., “Search-Based Automated Testing of Continuous Controllers: Framework, Tool Support, and Case Studies”, Information and Software Technology (2014)
  • 11. Dynamic continuous controllers are present in many embedded systems 11
  • 12. Development Process (Delphi) 12 Hardware-in-the-Loop Stage Model-in-the-Loop Stage Simulink Modeling Generic Functional Model MiL Testing Software-in-the-Loop Stage Code Generation and Integration Software Running on ECU SiL Testing Software Release HiL Testing
  • 13. Controllers at MIL 13 Inputs: Time-dependent variables Plant Model + + ⌃ + + - e(t) actual(t) desired(t) ⌃ KP e(t) KD de(t) dt KI R e(t) dt P I D output(t) Configuration Parameters
  • 14. Inputs, Outputs, Test Objectives 14 Initial Desired (ID) Desired ValueI (input) Actual Value (output) Final Desired (FD) Responsiveness Smoothness T/2 T time Stability
  • 15. Process and Technology 15 HeatMap Diagram 1. Exploration List of Critical Domain Regions Expert Worst-Case Scenarios Objective Functions based on Requirements + Controller-plant model 2. Single-State Search https://sites.google.com/site/cocotesttool/ Initial Desired (ID) Final Desired (FD) Worst Case(s)?
  • 16. Process and Technology (2) 16 HeatMap Diagram 1. Exploration List of Critical Domain Regions Expert Objective Functions based on Requirements + Controller-plant model (a) Liveness (b) Smoothness
  • 17. Process and Technology (3) 17 List of Critical Domain Regions Expert Worst-Case Scenarios 2. Single-State Search time Desired Value Actual Value 0 1 2 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0.0 Initial Desired Final Desired
  • 18. Challenges, Solutions • Achieving scalability with configuration parameters: – Simulink simulations are expensive – Sensitivity analysis to eliminate irrelevant parameters – Machine learning (Regression trees) to partition the space automatically and identify high-risk areas – Surrogate modeling (statistical and machine learning prediction) to predict properties and avoid simulation, when possible 18
  • 19. Results • Automotive controllers on Electronics Control Units • Our approach enabled our partner to identify worst-case scenarios that were much worse than known and expected scenarios, entirely automatically 19
  • 20. Fault Localisation in Simulink Models Reference: 20 • Bing Liu et al., “Kanvoo: Fault Localization in Simulink Models”, submitted
  • 21. Context and Problem 21 • Simulink models – are complex • hundreds of blocks and lines • many hierarchy levels • continuous functions – might be faulty • output signals do not match • wrong connection of lines • wrong operators in blocks • Debugging Simulink models is – difficult – time-consuming – but yet crucial • Automated techniques to support debugging?
  • 22. Context and Problem (2) 22 • Simulink models – are complex • hundreds of blocks and lines • many hierarchy levels • continuous functions – might be faulty • output signals do not match • wrong connection of lines • wrong operators in blocks • Debugging Simulink models is – difficult – time-consuming – but yet crucial • Automated techniques to support debugging?
  • 23. Solution Overview 23 Specifica+on% Test%Case%Genera+on% Test%Suite% ?% Test%Oracle% Test%Case%Execu+on% Slicing% Ranking% Coverage%Reports% PASS/FAIL% Results% Simulink%Model% Model%Slices% 0.95% 0.71% 0.62% 0.43% Ranked%Blocks% Any test strategy Provided by Matlab tool One slice for each test case and output For each test case and output, or overall
  • 24. Evaluation and Challenges • Good accuracy overall: 5-6% blocks must be inspected on average to detect faults • But less accurate predictions for certain faults: Low observability • Possible Solution: Augment test oracle (observability) – Use subsystems outputs – Iterate at deeper levels of hierarchy – Tradeoff: cost of test oracle vs. debugging effort – 2.3% blocks on average • 5-6%: still too many blocks for certain models • Information requirements to help further filtering blocks? 24
  • 25. Modeling and Verifying Legal Requirements Reference: 25 • G. Soltana et al., “ UML for Modeling Procedural Legal Rule”, IEEE/ACM MODELS 2014 • M. Adedjouma et al., “Automated Detection and Resolution of Legal Cross References”, RE 2014
  • 26. Context and Problem 26 • CTIE: Government computer centre in Luxembourg • Large government (information) systems • Implement legal requirements, must comply with the law • The law usually leaves room for interpretation and changes on a regular basis, many cross-references • Involves many stakeholders, IT specialists but also legal experts, etc.
  • 27. Article Example Art. 105bis […]The commuting expenses deduction (FD) is defined as a function over the distance between the principal town of the municipality on whose territory the taxpayer's home is located and the place of taxpayer’s work. The distance is measured in units of distance expressing the kilometric distance between [principal] towns. A ministerial regulation provides these distances. The amount of the deduction is calculated as follows: If the distance exceeds 4 units but is less than 30 units, the deduction is € 99 per unit of distance. The first 4 units does not trigger any deduction and the deduction for a distance exceeding 30 units is limited to € 2,574.
  • 28. Project Objectives 28 Objective Benefits Specification of legal requirements • including rationale and traceability to the text of law • Make interpretation of the law explicit • Improve communication • Prerequisite for automation Checking consistency of legal requirements • Prevent errors in the interpretation of the law to propagate Automated test strategies for checking system compliance to legal requirements • Provide effective and scalable ways to Run-time verification mechanisms to verify compliance check compliance with legal requirements Analyzing the impact of changes in the law • Decrease costs and risks associated with change • Make change more predictable
  • 29. Solution Overview 29 Test cases Actual software system Traces to Traces to Analyzable interpretation Generates of the law Results match? Impact of legal changes Simulates
  • 30. Research Steps 30 2. Build UML profile 3. Model Transformation to enable V&V • What information content should we expect? • What are the complexity factors? • Explicit means for capturing information requirements • Basis for modeling methodology • Target: Legal experts and IT specialists • Target existing automation techniques • Solvers for testing • MATLAB for simulation 1. Conduct grounded theory study
  • 31. Example 31 Art. 105bis […]The commuting expenses deduction (FD) is defined as a function over the distance between the principal town of the municipality on whose territory the taxpayer's home is located and the place of taxpayer’s work. The distance is measured in units of distance expressing the kilometric distance between [principal] towns. A ministerial regulation provides these distances. Interpretation + Traces
  • 32. Example 32 The amount of the deduction is calculated as follows: If the distance exceeds 4 units but is less than 30 units, the deduction is € 99 per unit of distance. The first 4 units does not trigger any deduction and the deduction for a distance exceeding 30 units is limited to € 2,574. Interpretation + Traces
  • 33. Challenges and Results • Profile must lead to models that are: – understandable by both IT specialists and legal experts – precise enough to enable model transformation and support our objectives – tutorials, many modeling sessions with legal experts • In theory, though such legal requirements can be captured by OCL constraints alone, this is not applicable • That is why we resorted to customized activity modeling, carefully combined with a simple subset of OCL • Many traces to law articles, dependencies among articles: automated detection (NLP) of cross-references 33
  • 34. Run-Time Verification of Business Processes References: 34 • W. Dou et al., “OCLR: a More Expressive, Pattern-based Temporal Extension of OCL”, ECMFA 2014 • W. Dou et al., “Revisiting Model-Driven Engineering for Run-Time Verification of Business Processes”, IEEE/ACM SAM 2014 • W. Dou et al., “A Model-Driven Approach to Offline Trace Checking of Temporal Properties with OCL”, submitted
  • 35. Context and Problem • CTIE: Government Computing Centre of Luxembourg • E-government systems mostly implemented as business processes • CTIE models these business processes • Business models have temporal properties that must be checked – Temporal logics not applicable – Limited tool support (scalability) • Goal: Efficient, scalable, and practical off-line and run-time verification 35
  • 37. Solution Overview 37 • We identified patterns based on analyzing many properties of real business process models • Properties must be defined based on business process models (BPMN) according to modeling methodology at CTIE (applicability) • The goal was to achieve usability • Early adoption by our partner
  • 38. Solution Overview 38 • Want to transform the checking of temporal constraints into checking regular constraints on trace conceptual model • OCL engines (Eclipse) are our target, to rely on mature technology (scalability) • Defined extension of OCL to facilitate translation • Target: IT specialists, BPM analysts
  • 39. Scalability Analysis • Analyzed 47 properties in Identity Card Management System • “Once a card request is approved, the applicant is notified within three days; this notification has to occur before the production of the card is started.” • Scalability: Check time as a function of trace size … 39 100 200 300 400 500 600 700 800 900 1,000 100 80 60 40 20 0 Trace Size (k) Average Check Time (s) P7 P8 P9 (a) globally / P7–P9 100 200 300 400 500 600 700 800 900 1,000 4,500 4,000 3,500 3,000 2,500 2,000 1,500 1,000 500 0 Trace Size (k) Average Check Time (s) P10 P11 P12 (b) globally / P10–P12 10 20 30 40 50 60 70 80 90 100 30 25 20 15 10 5 0 Boundary Position (k) Average Check Time (s) P17 P18 P19 P20 (c) before / P17–P20 Fig. 3: Average check time of properties with globally and before scopes
  • 40. Schedulability Analysis and Stress Testing References: 40 • S. Nejati, S. Di Alesio, M. Sabetzadeh, and L. Briand, “Modeling and analysis of cpu usage in safety-critical embedded systems to support stress testing,” in IEEE/ACM MODELS 2012. • S. Di Alesio, S. Nejati, L. Briand. A. Gotlieb, “Stress Testing of Task Deadlines: A Constraint Programming Approach”, ISSRE 2013, San Jose, USA • S. Di Alesio, S. Nejati, L. Briand. A. Gotlieb, “Worst-Case Scheduling of Software Tasks – A Constraint Optimization Model to Support Performance Testing, Constraint Programming (CP), 2014
  • 41. Problem • Real-time, concurrent systems (RTCS) have concurrent interdependent tasks which have to finish before their deadlines • Some task properties depend on the environment, some are design choices • Tasks can trigger other tasks, and can share computational resources with other tasks • Schedulability analysis encompasses techniques that try to predict whether all (critical) tasks are schedulable, i.e., meet their deadlines • Stress testing runs carefully selected test cases that have a high probability of leading to deadline misses • Testing in RTCS is typically expensive, e.g., hardware in the loop 41
  • 42. Arrival Times Determine Deadline Misses 42 0 1 2 3 4 5 6 7 8 9 j0, j1 , j2 arrive at at0 , at1 , at2 and must finish before dl0 , dl1 , dl2 at1 J1 can miss its deadline dl1 depending on when at2 occurs! 0 1 2 3 4 5 6 7 8 9 j0 j1 j2 j0 j1 j2 at0 dl0 at1 dl2 dl1 at2 T T at0 dl0 dl1 at2 dl2
  • 43. Context 43 Drivers (Software-Hardware Interface) Control Modules Alarm Devices (Hardware) Real-Time Operating System Multicore Architecture Monitor gas leaks and fire in oil extraction platforms
  • 44. Challenges and Solutions • Ranges for arrival times form a very large input space • Task interdependencies and properties constrain what parts of the space are feasible • We re-expressed the problem as a constraint optimisation problem • Constraint programming 44
  • 45. Constraint Optimization 45 Constraint Optimization Problem Static Properties of Tasks (Constants) Dynamic Properties of Tasks (Variables) OS Scheduler Behaviour Performance Requirement (Objective Function) (Constraints)
  • 46. Process and Technologies 46 UML Modeling (e.g., MARTE) Constraint Optimization System Design Design Model (Time Optimization Problem (Find arrival times that maximize the chance of deadline misses) System Platform Solutions (Task arrival times likely to lead to deadline misses) Deadline Misses Analysis and Concurrency Information) INPUT OUTPUT Stress Test Cases Constraint Programming (CP)
  • 47. Challenges and Solutions (2) • Scalability problem: Constraint programming (e.g., IBM CPLEX) cannot handle such large input spaces (CPU, memory) • Solution: Combine metaheuristic search and constraint programming – metaheuristic search identifies high risk regions in the input space – constraint programming finds provably worst-case schedules within these (limited) regions 47
  • 48. Process and Technologies 48 UML Modeling (e.g., MARTE) Constraint Optimization System Design Design Model (Time Optimization Problem (Find arrival times that maximize the chance of deadline misses) System Platform Solutions (Task arrival times likely to lead to deadline misses) Deadline Misses Analysis and Concurrency Information) INPUT OUTPUT Genetic Algorithms (GA) Stress Test Cases Constraint Programming (CP)
  • 50. Scalability examples • This is the most common challenge in practice • Testing closed-loop controllers – Large input and configuration space – Smart search optimization heuristics (machine learning) • Fault localization – Large number of blocks and lines in Simulink models – Even a small percentage of blocks to inspect can be impractical – Additional information to support decision making? Incremental fault localisation? • Schedulability analysis and stress testing – Constraint programming cannot scale by itself – Must be carefully combined with genetic algorithms 50
  • 51. Scalability examples (2) • Verifying legal requirements – Traceability to the law is complex – Many provisions and articles – Many dependencies within the law – Natural Language Processing: Cross references, support for identifying missing modeling concepts • Run-time Verification of Business Processes – Traces can be large and properties complex to verify – Transformation of temporal properties into regular OCL properties, defined on a trace conceptual model – Incremental verification at regular time intervals – Heuristics to identify subtraces to verify 51
  • 52. Scalability: Lessons Learned • Scalability must be part of the problem definition and solution from the start, not a refinement or an after-thought • It often involves heuristics, e.g., meta-heuristic search, NLP, machine learning, statistics • Scalability often leads to solutions that offer “best answers” within time constraints, not guarantees • Solutions to scalability are multi-disciplinary • Scalability analysis should be a component of every research project – otherwise it is unlikely to be adopted in practice • How many papers in MODELS or SAM do include even a minimal form of scalability analysis? 52
  • 53. Applicability • Definition? • Usability: Can the target user population efficiently apply it? • Assumptions: Are working assumptions realistic, e.g., realistic information requirements? • Integration into the development process, e.g., are required inputs available in the right form and level of precision? 53
  • 54. Applicability examples • Testing closed-loop controllers – Working assumption: availability of sufficiently precise plant (environment) models – Means to visualize relevant properties in the search space (inputs, configuration), to get an overview and focus search on high-risk areas • Schedulability analysis and stress testing – Availability of tasks architecture models – Precise WCET analysis – Applicability requires to assess risk based on near-deadline misses 54
  • 55. Applicability examples (2) • Fault localization: – Trade-off between # of model outputs considered versus cost of test oracles – Better understanding of the mental process and information requirements for fault localization • Run-time verification of business process models – Temporal logic not usable by analysts – Language closer to natural language, directly tied to business process model – Easy transition to industry strength constraint checker • Verifying legal requirements – Modeling notation must be shared by IT specialists and legal experts – One common representation for many applications, with traces to the law to handle changes – Multiple model transformation targets 55
  • 56. Applicability: Lessons Learned • Make working assumptions explicit: Determine the context of applicability • Make sure those working assumptions are at least realistic in some industrial domain and context • Assumptions don’t need to be universally true – they rarely are anyway • Run usability studies – do it for real! 56
  • 57. Conclusions • In most research endeavors, applicability and scalability are an after-thought, a secondary consideration, when at all considered • Implicit assumptions are often made, often unrealistic in any context • Problem definition in a vacuum • Not adapted to research in an engineering discipline • Leads to limited impact • Research in model-based V&V is necessarily multi-disciplinary • User studies are required and far too rare • In engineering research, there is no substitute to reality 57
  • 58. Acknowledgements PhD. Students: • Marwa Shousha • Reza Matinnejad • Stefano Di Alesio • Wei Dou • Ghanem Soltana • Bing Liu Scientists: • Shiva Nejati • Mehrdad Sabetzadeh • Domenico Bianculli • Arnaud Gotlieb • Yvan Labiche 58
  • 59. Making Model-Driven Verification Practical and Scalable: Experiences and Lessons Learned Lionel Briand IEEE Fellow, FNR PEARL Chair Interdisciplinary Centre for ICT Security, Reliability, and Trust (SnT) University of Luxembourg, Luxembourg SAM, Valencia, 2014 SVV lab: svv.lu SnT: www.securityandtrust.lu