Uncertainty
Management
With Partial
Models
M. Famelis
Introduction
Modeling
Uncertainty
Uncertainty Management With Partial Models
Managing
Uncertainty
Completed Work
Current Work Michalis Famelis
Future Work
Evalutation
University of Toronto
Conclusion
October 2nd, 2012,
MODELS’12 Doctoral Symposium
1 / 33

Uncertainty
Management
With Partial
Models
Hi, I’m Michalis Famelis
M. Famelis
Introduction
• 3rd year PhD student.
Modeling
Uncertainty
Managing • ...“a couple years to finish”
Uncertainty
Completed Work
Current Work
Future Work
Evalutation • Supervisor: Marsha Chechik
Conclusion
• MSc: Model Management with Relation
Types, CS UofT 2010.
• Undergrad: Model analysis and
transformation. ECE NTUA 2008.
2 / 33

Uncertainty
Management
With Partial
Models
Uncertainty in SE
M. Famelis
Introduction
Modeling “The reality of today’s software systems requires us to consider
Uncertainty
uncertainty as a first-class concern in the design, implementation,
Managing
Uncertainty and deployment of those systems.” [Garlan, 2010]
Completed Work
Current Work
Future Work
Evalutation
• Self-adaptive systems
Conclusion
[Esfahani et al., 2011, Cheng and Garlan, 2007]
• Probabilistic systems [Hinton et al., 2006]
• Imperfect information in requirements
[Noppen et al., 2007]
• Requirements clarifications [Knauss et al., 2012]
• Risk management [Islam and Houmb, 2010]
3 / 33

Uncertainty
Management
With Partial
Models
Design Uncertainty
M. Famelis
Introduction Uncertainty about design decisions – the contents of a model.
Modeling
Uncertainty
Managing
Ultimately, we need to capture and support design uncertainty.
Uncertainty
Completed Work
Current Work
Future Work
Enable MDSE with design uncertainty:
Evalutation • Create representations
Conclusion
• Check properties and give feedback
• Do uncertainty-removing refinement
• Enable transformations
• Create tooling support
• Create methodological support
4 / 33

Uncertainty
Management
With Partial
Models
What is Design Uncertainty
M. Famelis Example: a simple class diagram.
Introduction
Modeling What does the modeler know?
Uncertainty
Managing
Uncertainty
Completed Work
Current Work
Future Work
Evalutation
Conclusion
5 / 33

Uncertainty
Management
With Partial
Models
What is Design Uncertainty
M. Famelis Example: a simple class diagram.
Introduction
Modeling What does the modeler not know?
Uncertainty
Managing
Uncertainty
Completed Work
Current Work
Future Work
Evalutation
Conclusion
5 / 33

Uncertainty
Management
With Partial
Models
What is Design Uncertainty
M. Famelis Example: a simple class diagram.
Introduction
Modeling What does the modeler not know?
Uncertainty
Managing
Uncertainty
Completed Work
Current Work
Future Work
Evalutation
Conclusion
5 / 33

Uncertainty
Management
With Partial
Models
What is Design Uncertainty
M. Famelis Example: a simple class diagram.
Introduction
Modeling What does the modeler not know?
Uncertainty
Managing
Uncertainty
Completed Work
Current Work
Future Work
Evalutation
Conclusion
5 / 33

Uncertainty
Management
With Partial
Models
What is Design Uncertainty
M. Famelis Example: a simple class diagram.
Introduction
Modeling What does the modeler not know?
Uncertainty
Managing
Uncertainty
Completed Work
Current Work
Future Work
Evalutation
Conclusion
5 / 33

Uncertainty
Management
With Partial
Models
Dealing with Uncertainty: Options
M. Famelis
Classical:
Introduction
Modeling
Uncertainty 1 Wait until uncertainty is lifted.
Managing Inefficient; under-utilization of resources.
Uncertainty
Completed Work
Current Work
Future Work 2 Plan for all possible solutions.
Evalutation Intractable: they may be too many.
Conclusion
3 Make a decision. Now.
Artificial; risk undo or premature commitments.
We propose:
4 “Live” with uncertainty, defer resolution until more
information becomes available.
6 / 33

Uncertainty
Management
With Partial
Models
Design Uncertainty
M. Famelis
Uncertainty pervasive in MDSE.
Introduction
Modeling But also:
Uncertainty
Uncertainty about design decisions – the contents of a model.
Managing
Uncertainty
Completed Work Ultimately, we need to capture and support design uncertainty.
Current Work
Future Work
Evalutation
Enable MDSE with design uncertainty:
Conclusion • Create representations
• Check properties and give feedback
• Do uncertainty-removing refinement
• Enable transformations
• Create tooling support
• Create methodological support
7 / 33

Uncertainty
Management
With Partial
Models
Modeling Formalisms to Represent
M. Famelis Uncertainty
Introduction Modal
Modeling
Uncertainty
Behavioral • Restricted to one model type only.
Managing Specification
Uncertainty • Only optional elements.
Completed Work
Current Work
e.g.: • Over-approximation of set:
Future Work MTSs [Larsen’88]
Information loss
Evalutation DMTSs [Larsen’90]
Conclusion
Variability • Different conceptual domain (SPLs).
Modeling • Not rich enough. Eg:
– No variable meta-attributes.
e.g.:
(e.g. variable container class)
FTSs [Classen’10]
– No open world option.
TVL [Boucher’10]
Clafer [Bak’10] • More heavyweight (maintain an SPL
for ephemeral uncertainty?)
8 / 33

Uncertainty
Management
With Partial
Models
Uncertainty: a Set of Possible
M. Famelis Refinements
Introduction
Modeling
Uncertainty
Managing
Uncertainty
Completed Work
Current Work
Future Work
Evalutation
If we remove all uncertainty, we have a concrete refinement.
Conclusion
9 / 33

Uncertainty
Management
With Partial
Models
Modeling Uncertainty with Partial
M. Famelis Models
Introduction
Explicating uncertainty in a partial model.
Modeling
Uncertainty
Managing
Uncertainty
Completed Work
Current Work
Future Work
Evalutation
Conclusion
10 / 33

Uncertainty
Management
With Partial
Models
Modeling Uncertainty with Partial
M. Famelis Models
Introduction
Explicating uncertainty in a partial model.
Modeling
Uncertainty
Managing
Uncertainty
Completed Work
Current Work
Future Work
Evalutation
Conclusion
• May: Element is optional.
10 / 33

Uncertainty
Management
With Partial
Models
Modeling Uncertainty with Partial
M. Famelis Models
Introduction
Explicating uncertainty in a partial model.
Modeling
Uncertainty
Managing
Uncertainty
Completed Work
Current Work
Future Work
Evalutation
Conclusion
• May: Element is optional.
• Abs: Element can be multiplied to many copies.
10 / 33

Uncertainty
Management
With Partial
Models
Modeling Uncertainty with Partial
M. Famelis Models
Introduction
Explicating uncertainty in a partial model.
Modeling
Uncertainty
Managing
Uncertainty
Completed Work
Current Work
Future Work
Evalutation
Conclusion
• May: Element is optional.
• Abs: Element can be multiplied to many copies.
• Var: Element can be merged with others.
10 / 33

Uncertainty
Management
With Partial
Models
Modeling Uncertainty with Partial
M. Famelis Models
Introduction
Explicating uncertainty in a partial model.
Modeling
Uncertainty
Managing
Uncertainty
Completed Work
Current Work
Future Work
Evalutation
Conclusion
• May: Element is optional.
• Abs: Element can be multiplied to many copies.
• Var: Element can be merged with others.
• OW: Model is incomplete.
10 / 33

Uncertainty
Management
With Partial
Models
Modeling Uncertainty with Partial
M. Famelis Models
Introduction
Explicating uncertainty in a partial model.
Modeling
Uncertainty
Managing
Uncertainty
Completed Work
Current Work
Future Work
Evalutation
Conclusion
• May: Element is optional.
• Abs: Element can be multiplied to many copies.
• Var: Element can be merged with others.
• OW: Model is incomplete.
10 / 33

Uncertainty
Management
With Partial
Models
“Extended“ Partial Models
M. Famelis
Additional constraints to correlate points of uncertainty.
Introduction
e.g. May Model: variant presented in [ICSE’12]:
Modeling
Uncertainty
Managing
Uncertainty
Completed Work
Current Work
Future Work
Evalutation
Conclusion
11 / 33

Uncertainty
Management
With Partial
Models
Semantics of Partial Models
M. Famelis
Introduction
Modeling
Uncertainty
Managing
Uncertainty
Completed Work
Current Work
Future Work
Evalutation
Conclusion
12 / 33

Uncertainty
Management
With Partial
Models
Semantics of Partial Models
M. Famelis
Introduction
Modeling
Uncertainty
Managing
Uncertainty
Completed Work
Current Work
Future Work
Evalutation
Conclusion
12 / 33

Uncertainty
Management
With Partial
Models
Semantics of Partial Models
M. Famelis
Introduction
Modeling
Uncertainty
Managing
Uncertainty
Completed Work
Current Work
Future Work
Evalutation
Conclusion
12 / 33

Uncertainty
Management
With Partial
Models
Compared to Related Formalisms
M. Famelis
Introduction
Partial models vs Modal behavioral specification formalisms
Modeling
Uncertainty • Language/metamodel independent.
Managing
Uncertainty • Many kinds of partiality.
Completed Work
Current Work • Exact representation of set.
Future Work
Evalutation
Conclusion
Partial models vs Variability modeling formalisms
• Specific to uncertainty.
(c.f. uncertainty-removing refinement).
• Rich formalism.
• Ultimately disposable.
13 / 33

Uncertainty
Management
With Partial
Models
Status
M. Famelis
Uncertainty pervasive in MDSE.
Introduction
Modeling But also:
Uncertainty
Uncertainty about design decisions – the contents of a model.
Managing
Uncertainty
Completed Work Ultimately, we need to capture and support design uncertainty.
Current Work
Future Work
Evalutation
Enable MDSE with design uncertainty:
Conclusion • Create representations
• Check properties and give feedback
• Do uncertainty-removing refinement
• Enable transformations
• Create tooling support
• Create methodological support
14 / 33

Uncertainty
Management
With Partial
Models
Completed Work I
M. Famelis
Introduction
Explicating uncertainty
Modeling
Uncertainty • May models (M-elements + May formula) [ICSE’12]
Managing
Uncertainty • MAVO models + FOL semantics [FASE’12]
Completed Work
Current Work • Relational-algebraic encoding of MAVO [MoDeVVa’12]
Future Work
Evalutation
Conclusion
Checking properties
• Verifying May models with SAT [ICSE’12]
• Verifying MAVO models with Alloy [FASE’12]
• Verifying MAV- models with Alloy, CSP, SMT and ASP
[MoDeVVa’12]
15 / 33

Uncertainty
Management
With Partial
Models
Completed Work II
M. Famelis
Introduction
Modeling
Uncertainty-removing refinement
Uncertainty
• Semantics and verification [FASE’12]
Managing
Uncertainty • Property-driven refinement [ICSE’12]
Completed Work
Current Work
Future Work
Evalutation
Conclusion Tooling
• Python and MathSAT [ICSE’12]
• Alloy [FASE’12]
• Translations from annotated Ecore to RA and
from RA to SMT, CSP, ASP, Alloy [MoDeVVa’12]
16 / 33

Uncertainty
Management
With Partial
Models
Status
M. Famelis
Uncertainty pervasive in MDSE.
Introduction
Modeling But also:
Uncertainty
Uncertainty about design decisions – the contents of a model.
Managing
Uncertainty
Completed Work Ultimately, we need to capture and support design uncertainty.
Current Work
Future Work
Evalutation
Enable MDSE with design uncertainty:
Conclusion • Create representations
• Check properties and give feedback
• Do uncertainty-removing refinement
• Enable transformations
• Create tooling support
• Create methodological support
17 / 33

Uncertainty
Management
With Partial
Models
Lifting Transformations [MiSE’12]
M. Famelis May models + Graph transformations
Introduction
Modeling
Uncertainty
Managing
Uncertainty
Completed Work
Current Work
Future Work
Evalutation
Conclusion
Q1: How do we transform M directly to N?
– Lifted semantics of transformations, using logic.
(“transfer predicates”)
Q2: Are the concretizations of N exactly the models n1 . . . nk ?
– Check equivalence of encodings using SAT.
(“correctness criterion”)
18 / 33

Uncertainty
Management
With Partial
Models
Transformations: Next Steps
M. Famelis
Introduction • Compositionally test Correctness Criterion.
Modeling – Reuse Alloy-based verification method from [VOLT’12]?
Uncertainty
Managing
Uncertainty
Completed Work
• Systematically create Transfer Predicates using FOL.
Current Work
Future Work
– Predicates for atomic operations as building blocks.
Evalutation – May be hard to simplify resulting formulas.
Conclusion
• Handle expanding/contracting model vocabularies.
– E.g. adding a new element vs. concretizing.
• Reuse results from Category Theory? [Ehrig et al., 2006]
– Generalize to full MAVO.
– What are the right morphisms?
19 / 33

Uncertainty
Management
With Partial
Models
Tool support
M. Famelis
Full MAVO support for Ecore-based model editors.
Introduction
• As an EMF Profile? [Langer et al., 2011]
Modeling
Uncertainty • Via RAMification of metamodels? [K¨hne et al., 2009]
u
Managing – Would solve issues with concrete syntax
Uncertainty
Completed Work (e.g. when the base model is not well-formed)
Current Work
Future Work
Evalutation
Model management support
Conclusion
• Integration with the Model Management Tool Framework.
[Salay et al., 2007]
• Inherit macro-management tools for partial models
– name-based matching
– structural merging
– etc...
• “Push-button” integration of reasoning.
20 / 33

Uncertainty
Management
With Partial
Models
Status
M. Famelis
Uncertainty pervasive in MDSE.
Introduction
Modeling But also:
Uncertainty
Uncertainty about design decisions – the contents of a model.
Managing
Uncertainty
Completed Work Ultimately, we need to capture and support design uncertainty.
Current Work
Future Work
Evalutation
Enable MDSE with design uncertainty:
Conclusion • Create representations
• Check properties and give feedback
• Do uncertainty-removing refinement
• Enable transformations
• Create tooling support
• Create methodological support
21 / 33

Uncertainty
Management
With Partial
Models
Guidelines for Encoding
M. Famelis Uncertainty
Introduction
Modeling
Uncertainty
Managing
Uncertainty
Completed Work
Current Work
Future Work
Evalutation
Conclusion
“Placement” “Alternatives”
“Bottom-up” patterns of uncertainty.
• Help guide efficient/correct encoding.
• Support for choice of suitable reasoners.
22 / 33

Uncertainty
Management
With Partial
Models
Towards a Methodology for Design
M. Famelis Uncertainty
Introduction
Modeling
Uncertainty
Managing
We know how to do reasoning, refinement, transformations
Uncertainty under uncertainty.
Completed Work
Current Work
Future Work
Evalutation
We need to help identify what sorts of reasoning, refinement,
Conclusion etc. are necessary in each SE setting.
“Top-down” patterns of uncertainty:
• How does it appear in SE?
• What are its sources?
• What are its characteristic properties?
• What are refinements/transformations are sensible?
23 / 33

Uncertainty
Management
With Partial
Models
A Half Baked Example:
M. Famelis “Disagreement Pattern”
Introduction
Potential sources:
Modeling
Uncertainty • Conflicting stakeholder requirements.
Managing
Uncertainty
• Merge conflicts after commits.
Completed Work
Current Work • View-updates.
Future Work
Evalutation Characteristic properties:
Conclusion
• Is the conflict reconciliable?
• Have we reached agreement?
Sensible refinement and transformation checks:
• Does a refactoring expose/conceal the source of
disagreement?
• Is a series of refinements a negotiation?
(“are they making progress removing uncertainty?”)
24 / 33

Uncertainty
Management
With Partial
Models
Outline
M. Famelis
Introduction
“....enable MDSE with design uncertainty:”
Modeling
Uncertainty • Create representations
Managing
Uncertainty • Check properties and give feedback
Completed Work
Current Work
Future Work • Do uncertainty-removing refinement
Evalutation
• Enable transformations
Conclusion
• Create tooling support
• Create methodological support
How do we go about evaluating all this?
25 / 33

Uncertainty
Management
With Partial
Models
Evaluation of Individual Aspects
M. Famelis
Introduction
Experiments using randomly generated inputs
Modeling • To validate feasibility and Scalability of Property Checking
Uncertainty
[ICSE’12][MoDeVVa’12]
Managing
Uncertainty • Experimental driver in MMTF for rapid experiment setups.
Completed Work
Current Work
Future Work
Evalutation Case Studies
Conclusion
• Triangulated results of random experimentation in [ICSE’12].
• Realistic bugfix of real world bug in UMLet
Theorem Proving using Alloy
• Used to verify partial model refinements in [VOLT’12]
• Technique could be adapted for proving transformations
26 / 33

Uncertainty
Management
With Partial
Models
Evaluation of the Entire Approach
M. Famelis
Introduction
Modeling
Uncertainty
Plan for a Power Window Case Study
Managing • Comprehensive case study from the Automotive domain.
Uncertainty
Completed Work • Management of uncertainty throughout the MDE lifecycle.
Current Work
Future Work Requirements to implementation via transformations.
Evalutation
Conclusion
However:
• Is such a case study enough?
• Best method for evaluation not clear.
• How to evaluate “green-field” research?
27 / 33

Uncertainty
Management
With Partial
Models
Summary
M. Famelis
Uncertainty pervasive in MDSE.
Introduction
Modeling Not well studied:
Uncertainty
Uncertainty about design decisions – the contents of a model.
Managing
Uncertainty
Completed Work Ultimately, we need to capture and support design uncertainty.
Current Work
Future Work
Evalutation
Enable MDSE with design uncertainty:
Conclusion • Create representations
• Check properties and give feedback
• Do uncertainty-removing refinement
• Enable transformations
• Create tooling support
• Create methodological support
28 / 33

Questions?

Uncertainty
Management
With Partial
Models
Bibliography I
M. Famelis
Bak, K., Czarnecki, K., and Wasowski, A. (2011).
Introduction Feature and meta-models in clafer: Mixed, specialized, and coupled.
In SLE’10, pages 102–122.
Modeling
Uncertainty Boucher, Q., Classen, A., Faber, P., and Heymans, P. (2010).
Introducing TVL, a text-based feature modelling language.
Managing
In VaMoS’10, pages 159–162.
Uncertainty
Completed Work Cheng, S.-W. and Garlan, D. (2007).
Current Work
Handling uncertainty in autonomic systems.
Future Work
In ASE’07.
Evalutation
Classen, A., Heymans, P., Schobbens, P.-Y., Legay, A., and Raskin, J.-F. (2010).
Conclusion Model checking lots of systems: efficient verification of temporal properties in software product lines.
In ICSE’10, pages 335–344.
Ehrig, H., Ehrig, K., Prange, U., and Taentzer, G. (2006).
Fundamentals of Algebraic Graph Transformation.
EATCS. Springer.
Esfahani, N., Kouroshfar, E., and Malek, S. (2011).
Taming uncertainty in self-adaptive software.
In ESEC/FSE ’11, pages 234–244.
Famelis, M., Ben-David, S., Chechik, M., and Salay, R. (2011).
“Partial Models: A Position Paper”.
In Proceedings of MoDeVVa’11, pages 1–6.
30 / 33

Uncertainty
Management
With Partial
Models
Bibliography II
M. Famelis
Famelis, M., Chechik, M., and Salay, R. (2012a).
Introduction “Partial Models: Towards Modeling and Reasoning with Uncertainty”.
In Proceedings of ICSE’12.
Modeling
Uncertainty Famelis, M., Salay, R., and Chechik, M. (2012b).
The semantics of partial model transformations.
Managing In MISE at ICSE’12, pages 64 –69.
Uncertainty
Completed Work Garlan, D. (2010).
Current Work Software engineering in an uncertain world.
Future Work In FoSER ’10, pages 125–128.
Evalutation Hinton, A., Kwiatkowska, M., Norman, G., and Parker, D. (2006).
Conclusion Prism: A tool for automatic verification of probabilistic systems.
In Hermanns, H. and Palsberg, J., editors, Tools and Algorithms for the Construction and Analysis of
Systems, volume 3920 of Lecture Notes in Computer Science, pages 441–444.
Islam, S. and Houmb, S. H. (2010).
Integrating risk management activities into requirements engineering.
In RCIS’10, pages 299–310.
Knauss, E., Damian, D., Poo-Caamano, G., and Cleland-Huang, J. (2012).
Detecting and classifying patterns of requirements clarifications.
In RE’12.
K¨hne, T., Mezei, G., Syriani, E., Vangheluwe, H., and Wimmer, M. (2009).
u
Explicit transformation modeling.
In MODELS’09, pages 240–255.
31 / 33

Uncertainty
Management
With Partial
Models
Bibliography III
M. Famelis
Langer, P., Wieland, K., Wimmer, M., and Cabot, J. (2011).
From uml profiles to emf profiles and beyond.
Introduction In Objects, Models, Components, Patterns, volume 6705, pages 52–67.
Modeling Larsen, K. G. and Thomsen, B. (1988).
Uncertainty
“A Modal Process Logic”.
Managing In Proceedings of LICS’88, pages 203–210.
Uncertainty
Larsen, K. G. and Xinxin, L. (1990).
Completed Work
Current Work “Equation Solving Using Modal Transition Systems”.
Future Work In Proc. of LICS’90, pages 108–117.
Evalutation Noppen, J., van den Broek, P., and Akşit, M. (2007).
Software development with imperfect information.
Conclusion Soft Comput., 12(1):3–28.
Saadatpanah, P., Famelis, M., Gorzny, J., Robinson, N., Chechik, M., and Salay, R. (2012).
Comparing the effectiveness of reasoning formalisms for partial models.
In MoDeVVa’12.
Salay, R., Chechik, M., Easterbrook, S., Diskin, Z., McCormick, P., Nejati, S., Sabetzadeh, M., and
Viriyakattiyaporn, P. (2007).
An eclipse-based tool framework for software model management.
In Proceedings of the 2007 OOPSLA workshop on eclipse technology eXchange, eclipse ’07, pages
55–59.
Salay, R., Chechik, M., and Gorzny, J. (2012a).
“Towards a Methodology for Verifying Partial Model Refinements”.
In Proceedings of VOLT’12.
32 / 33

Uncertainty
Management
With Partial
Models
Bibliography IV
M. Famelis
Introduction
Modeling
Uncertainty
Managing
Uncertainty
Completed Work
Current Work
Future Work Salay, R., Famelis, M., and Chechik, M. (2012b).
“Language Independent Refinement using Partial Modeling”.
Evalutation In Proceedings of FASE’12.
Conclusion
33 / 33