Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Research Questions for Validation
and Verification in the Context of
Model-Based Engineering
Catherine Dubois
Michalis Fam...
Introduction
Abstraction techniques are one of the promising paths for
the future advances in the field of verification.
–...
About
• This paper is the result of the working group on V & V at
Dagstuhl-Seminar 13182 held in May 2013.
• Dagstuhl: sci...
4
Our goal:
Identify main areas in the synergy between MBE and
V&V where we need to focus research. What is the
status, wh...
Disclaimer
• No claim of completeness or exhaustiveness
• Represents the informed opinions of the authors
• May have misse...
Thematic Categories
1. Gap between Models and V&V Formalisms
2. Need to Refine Existing Methodologies
3. Design-time vs. R...
Models and V&V Formalisms (context)
Model Property
Model' Property'
Designer Level
(can be Domain-
Specific)
Verification
...
Models and V&V Formalisms
(questions)
• How to express properties at the level of models in a way
understandable to client...
Refining Existing Methodologies
(context)
• Integrating V&V in development can support early V&V
similar to how debugging ...
Refining Existing Methodologies
(questions)
• How do we integrate V&V in the overall development
and modeling process?
– O...
Design-time vs. Runtime (context)
• Models are specified at design time
• During execution these models are instantiated
•...
Design-time vs. Runtime (questions)
 During the V&V phase, how do we obtain an initial model
instantiation?
 How do we o...
Properties (context)
• Model and model transformation properties relevant in V&V:
– consistency, reachability, dependence,...
Properties (questions)
• What are the benefits and trade-offs between expressing
properties on more abstract modeling leve...
Model Transformations (context)
• Core component of MBE
• Many applications:
– Maintaining inter-model consistency
– Seman...
Model Transformations (questions)
• What verification techniques are meaningful for verifying
model transformations?
• How...
Informal vs. Formal vs. Incomplete
Modeling (context)
• During V&V: switch on or off particular model elements (in class d...
Informal vs. Formal vs. Incomplete
Modeling (questions)
• How do we leverage informal assumptions found in sketches for
ex...
Comparison and Benchmarking
(context)
• Benchmarking can boost research:
– Fair comparison of tools
– Impartial benchmark ...
Comparison and Benchmarking
(questions)
• How to compare existing V&V tools w.r.t.
functionality, coverage, scalability,
e...
Domain-Specific Languages
(Context)
• Most verification tools have hard to use input languages,
alien to normal developers...
Domain-Specific Languages
(Questions)
• How to define DSLs close to domain concepts but still allow
generation of meaningf...
Thematic Categories
1. Gap between Models and
V&V Formalisms
2. Need to Refine Existing
Methodologies
3. Design-time vs. R...
Conclusion
• Report-back from working group on MBE and V&V at
Dagstuhl in May 2013.
• Main areas in the synergy of MBE and...
Questions?
25
Upcoming SlideShare
Loading in …5
×

Research Questions for Validation and Verification in the Context of Model-Based Engineering

783 views

Published on

Abstract. In model-based engineering (MBE), the abstraction power
of models is used to deal with the ever increasing complexity of modern
software systems. As models play a central role in MBE-based develop-
ment processes, for the adoption of MBE in practical projects it becomes
indispensable to introduce rigorous methods for ensuring the correctness
of the models. Consequently, much effort has been spent on developing
and applying validation and verification (V&V) techniques for models.
However, there are still many open challenges.
In this paper, we shortly review the status quo of V&V techniques in
MBE and derive a catalogue of open questions whose answers would
contribute to successfully putting MBE into practice.


Catherine Dubois, ENSIIE, France
Michalis Famelis, University of Toronto, Canada
Martin Gogolla, Database Systems Group, University of Bremen, Germany
Leonel Nobrega, University of Madeira, Portugal
Ileana Ober, University of Toulouse, France
Martina Seidl, Johannes Kepler University Linz, Austria
Markus Völter, Völter Ingenieurbüro, Germany

Published in: Technology, Design
  • Be the first to comment

  • Be the first to like this

Research Questions for Validation and Verification in the Context of Model-Based Engineering

  1. 1. Research Questions for Validation and Verification in the Context of Model-Based Engineering Catherine Dubois Michalis Famelis Martin Gogolla Leonel Nobrega Ileana Ober Martina Seidl Markus Voelter ENSIIE, Evry, France University of Toronto, University of Bremen, University of Madeira, Funchal, University of Toulouse, Johannes Kepler University Linz, Austria Voelter Ingenieurburo, Heidenheim, Germany October 1st , 2013 MoDeVVa 2013, Miami, USA
  2. 2. Introduction Abstraction techniques are one of the promising paths for the future advances in the field of verification. – Clarke, Emerson, Sifakis: Turing Lecture 2008 V&V crucial for MBE – Uncover hidden properties and errors – Verify transformations – Ensure quality, etc. 2
  3. 3. About • This paper is the result of the working group on V & V at Dagstuhl-Seminar 13182 held in May 2013. • Dagstuhl: scientific retreat in western Germany • Seminar topic: Meta-Modeling Model- Based Engineering Tools • Three break-outs: Informal Modeling, Compositionality, Modeling and V&V 3
  4. 4. 4 Our goal: Identify main areas in the synergy between MBE and V&V where we need to focus research. What is the status, what are the research questions? • Two-day collaborative brainstorming workshop. • Culminated in a plenary presentation …and this report
  5. 5. Disclaimer • No claim of completeness or exhaustiveness • Represents the informed opinions of the authors • May have missed existing answers to some questions • Is this the right level of granularity / level of detail? 5
  6. 6. Thematic Categories 1. Gap between Models and V&V Formalisms 2. Need to Refine Existing Methodologies 3. Design-time vs. Runtime 4. Properties 5. Model Transformations 6. Informal vs. Formal vs. Incomplete Modeling 7. Comparison and Benchmarking 8. Domain-Specific Languages 6
  7. 7. Models and V&V Formalisms (context) Model Property Model' Property' Designer Level (can be Domain- Specific) Verification formalism level Verification engine Verification feed-back 7 Transformations and traceability mechanisms But behavioral semantics often leads to non-bijective correspondences between design time and runtime artifacts. V&V tool configuration
  8. 8. Models and V&V Formalisms (questions) • How to express properties at the level of models in a way understandable to clients? • How to formulate models and properties in a single language transparent to clients? • How to report the V&V results and diagnostics in an appropriate form to clients? • How to bridge the gap between formally expressed and verified properties on one side and client attention on the other side? • Can modeling language extensions help in making explicit the “needs” of V&V machines?
  9. 9. Refining Existing Methodologies (context) • Integrating V&V in development can support early V&V similar to how debugging is offered by IDEs. • Generic methodologies identify points where V&V can be used. • A Model-based development methodology would need to allow variations based on application domain, nature of project, etc. • Goal: better identify which V&V activities are meaningful at the various phases of design; take full advantage of V&V engines. 9
  10. 10. Refining Existing Methodologies (questions) • How do we integrate V&V in the overall development and modeling process? – On the technical level of tool exchange? – On the methodological level of using the right technique at the right time for the right task? • When are techniques like animation, execution, symbolic evaluation, testing, simulation, proving or test case generation used most efficiently during development? – For which model and model transformation properties can they be employed? 10
  11. 11. Design-time vs. Runtime (context) • Models are specified at design time • During execution these models are instantiated • The dynamic nature of structure during execution makes it difficult to understand and represent runtime information • Existing modeling environments offer limited support for precisely specifying instantiation and snapshots 11
  12. 12. Design-time vs. Runtime (questions)  During the V&V phase, how do we obtain an initial model instantiation?  How do we obtain large and meaningful instantiations?  How do we connect design time and runtime artifacts?  How do we deal with the issue of scalability in the context of V&V ?  How do we handle time and space concerns w.r.t. design time and runtime artifacts?  How do we automatically or semi-automatically manage the V&V machine configuration? 12
  13. 13. Properties (context) • Model and model transformation properties relevant in V&V: – consistency, reachability, dependence, minimality, conformance, safety, liveness, deadlock freeness, termination, confluence, correctness • Confusion caused by: – Different kinds according to the nature of the model (static/ dynamic), its level of abstraction, etc. – Many tools and techniques (potentially complementary) Main challenge: What kind of property to verify on which model at what stage with what kind of technique? 13
  14. 14. Properties (questions) • What are the benefits and trade-offs between expressing properties on more abstract modeling levels in contrast to expressing them on more concrete levels? • How do we find the right techniques for uncovering static and dynamic model properties? • Which techniques are appropriate for uncovering static modeling language inherent properties, which for static model-specific properties? • Which techniques are appropriate for uncovering dynamic generic properties, which for dynamic model-specific properties? 14
  15. 15. Model Transformations (context) • Core component of MBE • Many applications: – Maintaining inter-model consistency – Semantics definition of (domain-specific) modelling languages – … • Challenge: Verification of model transformations – Proving correctness, termination, confluence – What are the differences to “normal” code? – Is the higher abstraction level beneficial to V&V? 15
  16. 16. Model Transformations (questions) • What verification techniques are meaningful for verifying model transformations? • How do we analyse properties like confluence and termination? • How do we analyse correctness of model transformations w.r.t. a transformation contract? • How do we infer a transformation contract from a model transformation? 16
  17. 17. Informal vs. Formal vs. Incomplete Modeling (context) • During V&V: switch on or off particular model elements (in class diagrams, e.g., multiplicities); configure constraints by negating, deactivating or activating them (in class diagrams, e.g., class invariants) • Different types of granularity (a) all model elements may be relaxed (b) only a manual model element selection can be considered for relaxation (c) a semi- automatic element selection for relaxation may be offered • Ultimate vision: sliders on the user interface to gradually go from a strict, formal model through various intermediate levels to a totally relaxed and informal model; fewer formal model parts activated means more informality in the model • Minimal formal frame for test case construction must be preserved: e.g., for class diagrams of central classes and associations and for state charts of central states and transitions • For a completely informal model no formal scenario (no test cases) can be formulated 17
  18. 18. Informal vs. Formal vs. Incomplete Modeling (questions) • How do we leverage informal assumptions found in sketches for exploratory V&V? • Are informal sketches close enough to V&V at all? • What are appropriate relaxation mechanisms for different degrees of formality? • How do we handle incomplete or partial models w.r.t. V&V? • How do we deactivate and activate model units? • How do we handle the exploration of model properties and alternatives? 18
  19. 19. Comparison and Benchmarking (context) • Benchmarking can boost research: – Fair comparison of tools – Impartial benchmark selection which covers the spectrum of interesting test cases – Clear documentation of outcomes; reproducibility – Publicity makes it easy to identify progress, problems • However, in MBE: – No common standards – No community platform for benchmarks – Not clear what metrics are relevant for measuring “improvement” 19
  20. 20. Comparison and Benchmarking (questions) • How to compare existing V&V tools w.r.t. functionality, coverage, scalability, expressiveness, executing system (i.e., for models at runtime)? • Which criteria are appropriate for comparison? • Can we globally compare fairly at all? – Broad and diverse spectrum of V&V machines: B, Coq, HOL/Isabelle, SAT, SMT, CSP solvers, Relational logic, enumerative techniques 20
  21. 21. Domain-Specific Languages (Context) • Most verification tools have hard to use input languages, alien to normal developers. – Hence, verification tools are often not used. • MBE approaches become more and more mainstream. • Models can simplify analysis and verification, because of the higher degree of domain semantics they express. • Potential to exploit the two approaches synergistically – From the high-level models, we can automate the generation of the input to the verification tools. 21
  22. 22. Domain-Specific Languages (Questions) • How to define DSLs close to domain concepts but still allow generation of meaningful input for V&V tools? • V&V tools need the specification of properties. – How to express them at the domain level in a user-friendly way? – Can the property specifications be integrated with the same DSL and/or model used for describing the to-be-verified system without creating self-fulfilling prophecies? • How to bring V&V feedback back to the domain level and express it in terms of the DSL-level input? • Can incremental languages extensions help with making programs expressed in general-purpose languages more checkable? For example, the semantics of a specific extension construct may enable the generation of very rich inputs to the verification tool. 22
  23. 23. Thematic Categories 1. Gap between Models and V&V Formalisms 2. Need to Refine Existing Methodologies 3. Design-time vs. Runtime 4. Properties 5. Model Transformations 6. Informal vs. Formal vs. Incomplete Modeling 7. Comparison and Benchmarking 8. Domain-Specific Languages 23
  24. 24. Conclusion • Report-back from working group on MBE and V&V at Dagstuhl in May 2013. • Main areas in the synergy of MBE and V&V. • For each: described status and identified research questions. • Our hope: Spark discussions and debate, help focus research in MBE and V&V. 24
  25. 25. Questions? 25

×