- 1. Comparing the Eﬀectiveness of Reasoning Formalisms for Partial Models Saadatpanah, Famelis, Gorzny, Comparing the Eﬀectiveness of Reasoning Robinson, Chechik, Salay Formalisms for Partial Models Introduction Designer Uncertainty Pooya Saadatpanah, Michalis Famelis, Jan Gorzny, Modeling Nathan Robinson, Marsha Chechik, Rick Salay Uncertainty Property Checking University of Toronto Process Veriﬁcation Technologies September 30th, 2012 Experiments MoDeVVa’12 Results Conclusion 1 / 32
- 2. Comparing the Eﬀectiveness of Reasoning Formalisms for Uncertainty in software modeling Partial Models Saadatpanah, Famelis, Gorzny, Robinson, Chechik, Salay • Uncertainty: pervasive in MDE Introduction Designer Uncertainty • Models with uncertainty: Modeling • Represent choice among many possibilities Uncertainty • Can be reﬁned to many diﬀerent classical models Property Checking Process Veriﬁcation Technologies • Our goal: Experiments Handle models with uncertainty in MDE Results without having to remove it [MoDeVVa’11]. Conclusion 2 / 32
- 3. Comparing the Eﬀectiveness of Reasoning Formalisms for Existing Work Partial Models Saadatpanah, Famelis, Gorzny, Robinson, Chechik, Salay Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Veriﬁcation Technologies Experiments Results Conclusion 3 / 32
- 4. Comparing the Eﬀectiveness of Reasoning Formalisms for In This Paper Partial Models Saadatpanah, Famelis, Gorzny, Robinson, Chechik, Salay Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Veriﬁcation Technologies Experiments Results Conclusion 4 / 32
- 5. Comparing the Eﬀectiveness of Reasoning Formalisms for Outline Partial Models Saadatpanah, Famelis, Introduction Gorzny, Robinson, Chechik, Salay What is uncertainty? Introduction Designer How to represent uncertainty with partial models (MAVO). Uncertainty Modeling Uncertainty Process for checking properties Property Checking Process Alternative veriﬁcation technologies Veriﬁcation Technologies Experiments Experiments Results Results Conclusion Conclusion 5 / 32
- 6. Comparing the Eﬀectiveness of Reasoning Formalisms for Introduction to Uncertainty Partial Models Saadatpanah, Famelis, Gorzny, Robinson, Chechik, Salay What the designer knows. Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Veriﬁcation Technologies Experiments Results Conclusion 6 / 32
- 7. Comparing the Eﬀectiveness of Reasoning Formalisms for Introduction to Uncertainty Partial Models Saadatpanah, Famelis, Gorzny, Robinson, Chechik, Salay What the designer does not know. Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Veriﬁcation Technologies Experiments Results Conclusion 6 / 32
- 8. Comparing the Eﬀectiveness of Reasoning Formalisms for Introduction to Uncertainty Partial Models Saadatpanah, Famelis, Gorzny, Robinson, Chechik, Salay What the designer does not know. Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Veriﬁcation Technologies Experiments Results Conclusion 6 / 32
- 9. Comparing the Eﬀectiveness of Reasoning Formalisms for Introduction to Uncertainty Partial Models Saadatpanah, Famelis, Gorzny, Robinson, Chechik, Salay What the designer does not know. Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Veriﬁcation Technologies Experiments Results Conclusion 6 / 32
- 10. Comparing the Eﬀectiveness of Reasoning Formalisms for Uncertainty: a Set of Possible Partial Models Saadatpanah, Reﬁnements. Famelis, Gorzny, Robinson, Chechik, Salay Introduction Designer Uncertainty Modeling Uncertainty If we remove all uncertainty, we have a concrete reﬁnement. Property Checking Process Veriﬁcation Technologies Experiments Results Conclusion 7 / 32
- 11. Comparing the Eﬀectiveness of Reasoning Formalisms for Modeling Uncertainty with Partial Partial Models Saadatpanah, Models Famelis, Gorzny, Robinson, Explicating uncertainty in a partial model. Chechik, Salay Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Veriﬁcation Technologies In a reﬁnement, a May element is optional. Experiments In a reﬁnement, a Set element can be multiplied to many copies. Results In a reﬁnement, a Variable element can be uniﬁed with some other. Conclusion In a reﬁnement, an Open world model can be expanded with some other elements. Partial models: Syntactic annotations of the points of uncertainty. 8 / 32
- 12. Comparing the Eﬀectiveness of Reasoning Formalisms for Modeling Uncertainty with Partial Partial Models Saadatpanah, Models Famelis, Gorzny, Robinson, Explicating uncertainty in a partial model. Chechik, Salay Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Veriﬁcation Technologies In a reﬁnement, a May element is optional. Experiments In a reﬁnement, a Set element can be multiplied to many copies. Results In a reﬁnement, a Variable element can be uniﬁed with some other. Conclusion In a reﬁnement, an Open world model can be expanded with some other elements. Partial models: Syntactic annotations of the points of uncertainty. 8 / 32
- 13. Comparing the Eﬀectiveness of Reasoning Formalisms for Modeling Uncertainty with Partial Partial Models Saadatpanah, Models Famelis, Gorzny, Robinson, Explicating uncertainty in a partial model. Chechik, Salay Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Veriﬁcation Technologies In a reﬁnement, a May element is optional. Experiments In a reﬁnement, a Set element can be multiplied to many copies. Results In a reﬁnement, a Variable element can be uniﬁed with some other. Conclusion In a reﬁnement, an Open world model can be expanded with some other elements. Partial models: Syntactic annotations of the points of uncertainty. 8 / 32
- 14. Comparing the Eﬀectiveness of Reasoning Formalisms for Modeling Uncertainty with Partial Partial Models Saadatpanah, Models Famelis, Gorzny, Robinson, Explicating uncertainty in a partial model. Chechik, Salay Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Veriﬁcation Technologies In a reﬁnement, a May element is optional. Experiments In a reﬁnement, a Set element can be multiplied to many copies. Results In a reﬁnement, a Variable element can be uniﬁed with some other. Conclusion In a reﬁnement, an Open world model can be expanded with some other elements. Partial models: Syntactic annotations of the points of uncertainty. 8 / 32
- 15. Comparing the Eﬀectiveness of Reasoning Formalisms for Modeling Uncertainty with Partial Partial Models Saadatpanah, Models Famelis, Gorzny, Robinson, Explicating uncertainty in a partial model. Chechik, Salay Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Veriﬁcation Technologies In a reﬁnement, a May element is optional. Experiments In a reﬁnement, a Set element can be multiplied to many copies. Results In a reﬁnement, a Variable element can be uniﬁed with some other. Conclusion In a reﬁnement, an Open world model can be expanded with some other elements. Partial models: Syntactic annotations of the points of uncertainty. 8 / 32
- 16. Comparing the Eﬀectiveness of Reasoning Formalisms for Outline Partial Models Saadatpanah, Famelis, Introduction Gorzny, Robinson, Chechik, Salay What is uncertainty? Introduction Designer How to represent uncertainty with partial models (MAVO). Uncertainty Modeling Uncertainty Process for checking properties Property Checking Process Alternative veriﬁcation technologies Veriﬁcation Technologies Experiments Experiments Results Results Conclusion Conclusion 9 / 32
- 17. Comparing the Eﬀectiveness of Reasoning Formalisms for Property Checking in Partial Partial Models Saadatpanah, Models Famelis, Gorzny, Robinson, Chechik, Salay Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Veriﬁcation Technologies Experiments Results Conclusion 10 / 32
- 18. Comparing the Eﬀectiveness of Reasoning Formalisms for Property Checking in Partial Partial Models Saadatpanah, Models Famelis, Gorzny, Robinson, Chechik, Salay Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Veriﬁcation Technologies Experiments Results Conclusion 10 / 32
- 19. Comparing the Eﬀectiveness of Reasoning Formalisms for Property Checking in Partial Partial Models Saadatpanah, Models Famelis, Gorzny, Robinson, Chechik, Salay Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Veriﬁcation Technologies Experiments Results Conclusion 10 / 32
- 20. Comparing the Eﬀectiveness of Reasoning Formalisms for Property Checking in Partial Partial Models Saadatpanah, Models Famelis, Gorzny, Robinson, Chechik, Salay Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Veriﬁcation Technologies Experiments Results Conclusion 10 / 32
- 21. Comparing the Eﬀectiveness of Reasoning Formalisms for Veriﬁcation Technologies I Partial Models Saadatpanah, Famelis, Gorzny, Robinson, • Alloy Chechik, Salay • Lightweight formal methods Introduction • Model ﬁnder based on SAT Designer • First order logic speciﬁcations expressed in relational logic Uncertainty • Grounded to CNF representation Modeling • Finds counter examples Uncertainty Property Checking Process • Constraint Satisfaction Problem (Minizinc/Flatzinc) Veriﬁcation Technologies • Assign value to variables to satisfy all constraints Experiments • Constraint modeling language Results • Easily translatable to the form required by other CSP Conclusion solvers 11 / 32
- 22. Comparing the Eﬀectiveness of Reasoning Formalisms for Veriﬁcation Technologies II Partial Models Saadatpanah, Famelis, Gorzny, Robinson, Chechik, Salay • Satisﬁability Modulo Theory (Z3) • Constraint satisfaction search with richer theories Introduction • Theorem prover Designer Uncertainty • Check the satisﬁability logical formulas Modeling Uncertainty Property Checking Process • Answer Set Programming (Clingo=Gringo+Clasp) Veriﬁcation • Answer set solvers Technologies • Conﬂict-driven nogood learning Experiments • Normal logic programs Results Conclusion 12 / 32
- 23. Comparing the Eﬀectiveness of Reasoning Formalisms for Outline Partial Models Saadatpanah, Famelis, Introduction Gorzny, Robinson, Chechik, Salay What is uncertainty? Introduction Designer How to represent uncertainty with partial models (MAVO). Uncertainty Modeling Uncertainty Process for checking properties Property Checking Process Alternative veriﬁcation technologies Veriﬁcation Technologies Experiments Experiments Results Results Conclusion Conclusion 13 / 32
- 24. Comparing the Eﬀectiveness of Reasoning Formalisms for Experimental Setup Partial Models Saadatpanah, Famelis, Gorzny, Robinson, Chechik, Salay Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Veriﬁcation Technologies Experiments Results Conclusion 14 / 32
- 25. Comparing the Eﬀectiveness of Reasoning Formalisms for Experimental Setup Partial Models Saadatpanah, Famelis, Gorzny, Robinson, Chechik, Salay Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Veriﬁcation Technologies Experiments Results Conclusion 15 / 32
- 26. Comparing the Eﬀectiveness of Reasoning Formalisms for Random Input Generation Partial Models Saadatpanah, Famelis, Gorzny, Robinson, • Meta-model : directed graphs Chechik, Salay • Minimal meta-model Introduction • A few constraints Designer • Most diﬃcult one for solvers Uncertainty Modeling Uncertainty • Randomly decorated with MAVO annotations. Property Checking Process • Parameters are based on real case studies. Veriﬁcation • Graph density Technologies • Percentage of MAVO annotated elements Experiments • Percentages of M-, S- and V-annotated elements Results Conclusion • 3 Model Size : Small, Medium, Large, X-Large 16 / 32
- 27. Comparing the Eﬀectiveness of Reasoning Formalisms for Experimental Setup Partial Models Saadatpanah, Famelis, Gorzny, Robinson, Chechik, Salay Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Veriﬁcation Technologies Experiments Results Conclusion 17 / 32
- 28. Comparing the Eﬀectiveness of Reasoning Formalisms for Relational Encoding I Partial Models Saadatpanah, Famelis, Gorzny, Robinson, Chechik, Salay • Relational Algebra used in DBMSs Introduction Designer • Directly translatable into the diﬀerent formalisms Uncertainty Modeling Uncertainty • Intermediate representation Property Checking • FOL semantics of MAVO Process • Reasoning formalisms Veriﬁcation Technologies Experiments • Meaningful comparison Results • Most eﬃcient encoding in each formalism : impossible! Conclusion • Solution : common encoding 18 / 32
- 29. Comparing the Eﬀectiveness of Reasoning Formalisms for Relational Encoding II Partial Models Saadatpanah, Famelis, Gorzny, Robinson, Chechik, Salay (Very) high level overview. Introduction • The metamodel encoded Designer as a schema. Uncertainty • Partial model FOL Modeling Uncertainty semantics encoded as Property Checking constraints over the Process schema. Veriﬁcation Technologies • Creating a concrete Experiments reﬁnement populates the Results database. Conclusion 19 / 32
- 30. Comparing the Eﬀectiveness of Reasoning Formalisms for Experimental Setup Partial Models Saadatpanah, Famelis, Gorzny, Robinson, Chechik, Salay Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Veriﬁcation Technologies Experiments Results Conclusion 20 / 32
- 31. Comparing the Eﬀectiveness of Reasoning Formalisms for Translation To Formalisms Partial Models • Alloy Saadatpanah, Famelis, • Relations : Alloy signatures Gorzny, Robinson, • Instances : Atoms Chechik, Salay • MAVO constraints : quantiﬁed predicates over signatures • Bound is required Introduction Designer • CSP Uncertainty • Relations : Finite set of Integers Modeling • Instances : Integers Uncertainty • MAVO constraints : cardinality and intersection of sets Property Checking • Bound is required Process • SMT Veriﬁcation • Relations : Uninterpreted boolean functions Technologies • Instances : Abstract values Experiments • MAVO constraints : Quantiﬁed logic over truth table of Results functions Conclusion • ASP • Program rules for both instances and relations • Bound is required 21 / 32
- 32. Comparing the Eﬀectiveness of Reasoning Formalisms for Experimental Setup Partial Models Saadatpanah, Famelis, Gorzny, Robinson, Chechik, Salay Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Veriﬁcation Technologies Experiments Results Conclusion 22 / 32
- 33. Comparing the Eﬀectiveness of Reasoning Formalisms for Properties Checked Partial Models Saadatpanah, Famelis, Gorzny, • Inspired by real metamodel constraints. Robinson, Chechik, Salay • No transitive closure, since it is expensive to check. Introduction Designer Uncertainty P1: There exists a node with a self-loop. Modeling Uncertainty P2: All nodes have outgoing edges. Property Checking P3: All nodes have outgoing or incoming edges. Process Veriﬁcation P4: For all pairs of nodes n1 , n2 there exists at most one edge e Technologies e such that n1 → n2 Experiments Results P5: For every pair of nodes n1 , n2 , n1 = n2 there exist two edges e1 e2 Conclusion e1 , e2 such that n1 → n2 and n2 → n1 . 23 / 32
- 34. Comparing the Eﬀectiveness of Reasoning Formalisms for Experimental Setup Partial Models Saadatpanah, Famelis, Gorzny, Robinson, Chechik, Salay Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Veriﬁcation Technologies Experiments Results Conclusion 24 / 32
- 35. Comparing the Eﬀectiveness of Reasoning Formalisms for Experimental Parameters Partial Models Saadatpanah, • Bound (2, 4, 6) Famelis, Gorzny, • Solvers (except SMT) use bound for grounding expressions Robinson, Chechik, Salay to atoms. • How many times can an ’S’-annotated element be Introduction replicated in a reﬁnement. Designer Uncertainty • Repetitions Modeling Uncertainty • 5 times Property Checking • Cutoﬀ time/memory Process • less than 10 minutes Veriﬁcation Technologies • less than 5 gigabyte Experiments • otherwise timeout Results • What we measure Conclusion • How long does it take for each solver to return an answer • A score out of 1200 • if timeout : zero! 25 / 32
- 36. Comparing the Eﬀectiveness of Reasoning Formalisms for Findings Partial Models Saadatpanah, Famelis, Gorzny, Robinson, Chechik, Salay Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Veriﬁcation Technologies Experiments Results Conclusion 26 / 32
- 37. Comparing the Eﬀectiveness of Reasoning Formalisms for Findings Partial Models Saadatpanah, Famelis, Gorzny, Robinson, Chechik, Salay Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Veriﬁcation Technologies Experiments Results Conclusion 26 / 32
- 38. Comparing the Eﬀectiveness of Reasoning Formalisms for Findings Partial Models Saadatpanah, Famelis, Gorzny, Robinson, Chechik, Salay Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Veriﬁcation Technologies Experiments Results Conclusion 26 / 32
- 39. Comparing the Eﬀectiveness of Reasoning Formalisms for SMT the champion? Partial Models Saadatpanah, Famelis, Gorzny, Robinson, Chechik, Salay • Unaﬀected by bounds! Introduction • Works at higher level of abstraction Designer (Theory of uninterpreted functions.) Uncertainty Modeling • Unaﬀected by expensive grounding phase. Uncertainty Property Checking Process Veriﬁcation Caveat: Technologies Experiments • SMT can theoretically return “I don’t know”. Results • (However: we didn’t observe that.’) Conclusion 27 / 32
- 40. Comparing the Eﬀectiveness of Reasoning Formalisms for Threats to Validity Partial Models Saadatpanah, Famelis, Gorzny, Randomly generated graphs. Robinson, Chechik, Salay • Tuned the generator with realistic graph properties. Introduction • Values of graph properties from case studies. Designer Uncertainty Modeling Uncertainty Fairness of comparisons. Property • Common encoding to level the ﬁeld. Checking Process Veriﬁcation Technologies Choice of speciﬁc reasoning engines Experiments Results • When available: winners of competitions. Conclusion • CSP: most convenient input language. 28 / 32
- 41. Comparing the Eﬀectiveness of Reasoning Formalisms for Summary Partial Models Saadatpanah, Famelis, Gorzny, Robinson, Chechik, Salay Research Question What is the most eﬃcient formalism for verifying models Introduction Designer containing uncertainty? Uncertainty Modeling Uncertainty Property Checking Process Veriﬁcation Technologies Experiments Results Conclusion 29 / 32
- 42. Comparing the Eﬀectiveness of Reasoning Formalisms for Summary Partial Models Saadatpanah, Famelis, Gorzny, Robinson, Chechik, Salay Introduction Designer Uncertainty Modeling Uncertainty Property Checking Process Veriﬁcation Other results: Technologies • Framework for running experiments (now full tool support). Experiments Results • Random generator for arbitrary type graphs. Conclusion • Relational encoding. • Translations of the RA encoding to diﬀerent formalisms. 29 / 32
- 43. Comparing the Eﬀectiveness of Reasoning Formalisms for Future Work Partial Models Saadatpanah, Famelis, Gorzny, Robinson, Chechik, Salay • Implement symmetry breaking in the SMT encoding. Introduction Designer Uncertainty Modeling Uncertainty • Experiment with properties that require transitive closure. Property Checking Process Veriﬁcation Technologies Experiments • Experiment with partial models containing OW. Results Conclusion 30 / 32
- 44. Questions?
- 45. Comparing the Eﬀectiveness of Reasoning Formalisms for Bibliography I Partial Models Saadatpanah, Famelis, Gorzny, Robinson, Chechik, Salay Introduction Famelis, M., Ben-David, S., Chechik, M., and Salay, R. (2011). “Partial Models: A Position Paper”. Designer In Proceedings of MoDeVVa’11, pages 1–6. Uncertainty Famelis, M., Chechik, M., and Salay, R. (2012). Modeling Uncertainty “Partial Models: Towards Modeling and Reasoning with Uncertainty”. In Proceedings of ICSE’12. Property Checking Salay, R., Famelis, M., and Chechik, M. (2012). Process “Language Independent Reﬁnement using Partial Modeling”. In Proceedings of FASE’12. Veriﬁcation Technologies Experiments Results Conclusion 32 / 32