Comparing the Effectiveness of Reasoning Formalisms for Partial Models

  • 329 views
Uploaded on

Pooya Saadatpanah, Michalis Famelis, Jan Gorzny, Nathan Robinson, Marsha Chechik, Rick Salay …

Pooya Saadatpanah, Michalis Famelis, Jan Gorzny, Nathan Robinson, Marsha Chechik, Rick Salay

University of Toronto

September 30th, 2012

MoDeVVa’12

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
329
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
1
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Comparing the Effectiveness of ReasoningFormalisms forPartial ModelsSaadatpanah, Famelis, Gorzny, Comparing the Effectiveness of Reasoning Robinson,Chechik, Salay Formalisms for Partial ModelsIntroductionDesignerUncertainty Pooya Saadatpanah, Michalis Famelis, Jan Gorzny,Modeling Nathan Robinson, Marsha Chechik, Rick SalayUncertaintyPropertyChecking University of TorontoProcessVerificationTechnologies September 30th, 2012Experiments MoDeVVa’12ResultsConclusion 1 / 32
  • 2. Comparing the Effectiveness of ReasoningFormalisms for Uncertainty in software modelingPartial ModelsSaadatpanah, Famelis, Gorzny, Robinson,Chechik, Salay • Uncertainty: pervasive in MDEIntroductionDesignerUncertainty • Models with uncertainty:Modeling • Represent choice among many possibilitiesUncertainty • Can be refined to many different classical modelsPropertyCheckingProcessVerificationTechnologies • Our goal:Experiments Handle models with uncertainty in MDEResults without having to remove it [MoDeVVa’11].Conclusion 2 / 32
  • 3. Comparing the Effectiveness of ReasoningFormalisms for Existing WorkPartial ModelsSaadatpanah, Famelis, Gorzny, Robinson,Chechik, SalayIntroductionDesignerUncertaintyModelingUncertaintyPropertyCheckingProcessVerificationTechnologiesExperimentsResultsConclusion 3 / 32
  • 4. Comparing the Effectiveness of ReasoningFormalisms for In This PaperPartial ModelsSaadatpanah, Famelis, Gorzny, Robinson,Chechik, SalayIntroductionDesignerUncertaintyModelingUncertaintyPropertyCheckingProcessVerificationTechnologiesExperimentsResultsConclusion 4 / 32
  • 5. Comparing the Effectiveness of ReasoningFormalisms for OutlinePartial ModelsSaadatpanah, Famelis, Introduction Gorzny, Robinson,Chechik, Salay What is uncertainty?IntroductionDesigner How to represent uncertainty with partial models (MAVO).UncertaintyModelingUncertainty Process for checking propertiesPropertyCheckingProcess Alternative verification technologiesVerificationTechnologies ExperimentsExperimentsResults ResultsConclusion Conclusion 5 / 32
  • 6. Comparing the Effectiveness of ReasoningFormalisms for Introduction to UncertaintyPartial ModelsSaadatpanah, Famelis, Gorzny, Robinson,Chechik, Salay What the designer knows.IntroductionDesignerUncertaintyModelingUncertaintyPropertyCheckingProcessVerificationTechnologiesExperimentsResultsConclusion 6 / 32
  • 7. Comparing the Effectiveness of ReasoningFormalisms for Introduction to UncertaintyPartial ModelsSaadatpanah, Famelis, Gorzny, Robinson,Chechik, Salay What the designer does not know.IntroductionDesignerUncertaintyModelingUncertaintyPropertyCheckingProcessVerificationTechnologiesExperimentsResultsConclusion 6 / 32
  • 8. Comparing the Effectiveness of ReasoningFormalisms for Introduction to UncertaintyPartial ModelsSaadatpanah, Famelis, Gorzny, Robinson,Chechik, Salay What the designer does not know.IntroductionDesignerUncertaintyModelingUncertaintyPropertyCheckingProcessVerificationTechnologiesExperimentsResultsConclusion 6 / 32
  • 9. Comparing the Effectiveness of ReasoningFormalisms for Introduction to UncertaintyPartial ModelsSaadatpanah, Famelis, Gorzny, Robinson,Chechik, Salay What the designer does not know.IntroductionDesignerUncertaintyModelingUncertaintyPropertyCheckingProcessVerificationTechnologiesExperimentsResultsConclusion 6 / 32
  • 10. Comparing the Effectiveness of ReasoningFormalisms for Uncertainty: a Set of PossiblePartial ModelsSaadatpanah, Refinements. Famelis, Gorzny, Robinson,Chechik, SalayIntroductionDesignerUncertaintyModelingUncertainty If we remove all uncertainty, we have a concrete refinement.PropertyCheckingProcessVerificationTechnologiesExperimentsResultsConclusion 7 / 32
  • 11. Comparing the Effectiveness of ReasoningFormalisms for Modeling Uncertainty with PartialPartial ModelsSaadatpanah, Models Famelis, Gorzny, Robinson, Explicating uncertainty in a partial model.Chechik, SalayIntroductionDesignerUncertaintyModelingUncertaintyPropertyCheckingProcessVerificationTechnologies In a refinement, a May element is optional.Experiments In a refinement, a Set element can be multiplied to many copies.Results In a refinement, a Variable element can be unified with some other.Conclusion In a refinement, an Open world model can be expanded with some other elements. Partial models: Syntactic annotations of the points of uncertainty. 8 / 32
  • 12. Comparing the Effectiveness of ReasoningFormalisms for Modeling Uncertainty with PartialPartial ModelsSaadatpanah, Models Famelis, Gorzny, Robinson, Explicating uncertainty in a partial model.Chechik, SalayIntroductionDesignerUncertaintyModelingUncertaintyPropertyCheckingProcessVerificationTechnologies In a refinement, a May element is optional.Experiments In a refinement, a Set element can be multiplied to many copies.Results In a refinement, a Variable element can be unified with some other.Conclusion In a refinement, an Open world model can be expanded with some other elements. Partial models: Syntactic annotations of the points of uncertainty. 8 / 32
  • 13. Comparing the Effectiveness of ReasoningFormalisms for Modeling Uncertainty with PartialPartial ModelsSaadatpanah, Models Famelis, Gorzny, Robinson, Explicating uncertainty in a partial model.Chechik, SalayIntroductionDesignerUncertaintyModelingUncertaintyPropertyCheckingProcessVerificationTechnologies In a refinement, a May element is optional.Experiments In a refinement, a Set element can be multiplied to many copies.Results In a refinement, a Variable element can be unified with some other.Conclusion In a refinement, an Open world model can be expanded with some other elements. Partial models: Syntactic annotations of the points of uncertainty. 8 / 32
  • 14. Comparing the Effectiveness of ReasoningFormalisms for Modeling Uncertainty with PartialPartial ModelsSaadatpanah, Models Famelis, Gorzny, Robinson, Explicating uncertainty in a partial model.Chechik, SalayIntroductionDesignerUncertaintyModelingUncertaintyPropertyCheckingProcessVerificationTechnologies In a refinement, a May element is optional.Experiments In a refinement, a Set element can be multiplied to many copies.Results In a refinement, a Variable element can be unified with some other.Conclusion In a refinement, an Open world model can be expanded with some other elements. Partial models: Syntactic annotations of the points of uncertainty. 8 / 32
  • 15. Comparing the Effectiveness of ReasoningFormalisms for Modeling Uncertainty with PartialPartial ModelsSaadatpanah, Models Famelis, Gorzny, Robinson, Explicating uncertainty in a partial model.Chechik, SalayIntroductionDesignerUncertaintyModelingUncertaintyPropertyCheckingProcessVerificationTechnologies In a refinement, a May element is optional.Experiments In a refinement, a Set element can be multiplied to many copies.Results In a refinement, a Variable element can be unified with some other.Conclusion In a refinement, an Open world model can be expanded with some other elements. Partial models: Syntactic annotations of the points of uncertainty. 8 / 32
  • 16. Comparing the Effectiveness of ReasoningFormalisms for OutlinePartial ModelsSaadatpanah, Famelis, Introduction Gorzny, Robinson,Chechik, Salay What is uncertainty?IntroductionDesigner How to represent uncertainty with partial models (MAVO).UncertaintyModelingUncertainty Process for checking propertiesPropertyCheckingProcess Alternative verification technologiesVerificationTechnologies ExperimentsExperimentsResults ResultsConclusion Conclusion 9 / 32
  • 17. Comparing the Effectiveness of ReasoningFormalisms for Property Checking in PartialPartial ModelsSaadatpanah, Models Famelis, Gorzny, Robinson,Chechik, SalayIntroductionDesignerUncertaintyModelingUncertaintyPropertyCheckingProcessVerificationTechnologiesExperimentsResultsConclusion 10 / 32
  • 18. Comparing the Effectiveness of ReasoningFormalisms for Property Checking in PartialPartial ModelsSaadatpanah, Models Famelis, Gorzny, Robinson,Chechik, SalayIntroductionDesignerUncertaintyModelingUncertaintyPropertyCheckingProcessVerificationTechnologiesExperimentsResultsConclusion 10 / 32
  • 19. Comparing the Effectiveness of ReasoningFormalisms for Property Checking in PartialPartial ModelsSaadatpanah, Models Famelis, Gorzny, Robinson,Chechik, SalayIntroductionDesignerUncertaintyModelingUncertaintyPropertyCheckingProcessVerificationTechnologiesExperimentsResultsConclusion 10 / 32
  • 20. Comparing the Effectiveness of ReasoningFormalisms for Property Checking in PartialPartial ModelsSaadatpanah, Models Famelis, Gorzny, Robinson,Chechik, SalayIntroductionDesignerUncertaintyModelingUncertaintyPropertyCheckingProcessVerificationTechnologiesExperimentsResultsConclusion 10 / 32
  • 21. Comparing the Effectiveness of ReasoningFormalisms for Verification Technologies IPartial ModelsSaadatpanah, Famelis, Gorzny, Robinson, • AlloyChechik, Salay • Lightweight formal methodsIntroduction • Model finder based on SATDesigner • First order logic specifications expressed in relational logicUncertainty • Grounded to CNF representationModeling • Finds counter examplesUncertaintyPropertyCheckingProcess • Constraint Satisfaction Problem (Minizinc/Flatzinc)VerificationTechnologies • Assign value to variables to satisfy all constraintsExperiments • Constraint modeling languageResults • Easily translatable to the form required by other CSPConclusion solvers 11 / 32
  • 22. Comparing the Effectiveness of ReasoningFormalisms for Verification Technologies IIPartial ModelsSaadatpanah, Famelis, Gorzny, Robinson,Chechik, Salay • Satisfiability Modulo Theory (Z3) • Constraint satisfaction search with richer theoriesIntroduction • Theorem proverDesignerUncertainty • Check the satisfiability logical formulasModelingUncertaintyPropertyCheckingProcess • Answer Set Programming (Clingo=Gringo+Clasp)Verification • Answer set solversTechnologies • Conflict-driven nogood learningExperiments • Normal logic programsResultsConclusion 12 / 32
  • 23. Comparing the Effectiveness of ReasoningFormalisms for OutlinePartial ModelsSaadatpanah, Famelis, Introduction Gorzny, Robinson,Chechik, Salay What is uncertainty?IntroductionDesigner How to represent uncertainty with partial models (MAVO).UncertaintyModelingUncertainty Process for checking propertiesPropertyCheckingProcess Alternative verification technologiesVerificationTechnologies ExperimentsExperimentsResults ResultsConclusion Conclusion 13 / 32
  • 24. Comparing the Effectiveness of ReasoningFormalisms for Experimental SetupPartial ModelsSaadatpanah, Famelis, Gorzny, Robinson,Chechik, SalayIntroductionDesignerUncertaintyModelingUncertaintyPropertyCheckingProcessVerificationTechnologiesExperimentsResultsConclusion 14 / 32
  • 25. Comparing the Effectiveness of ReasoningFormalisms for Experimental SetupPartial ModelsSaadatpanah, Famelis, Gorzny, Robinson,Chechik, SalayIntroductionDesignerUncertaintyModelingUncertaintyPropertyCheckingProcessVerificationTechnologiesExperimentsResultsConclusion 15 / 32
  • 26. Comparing the Effectiveness of ReasoningFormalisms for Random Input GenerationPartial ModelsSaadatpanah, Famelis, Gorzny, Robinson, • Meta-model : directed graphsChechik, Salay • Minimal meta-modelIntroduction • A few constraintsDesigner • Most difficult one for solversUncertaintyModelingUncertainty • Randomly decorated with MAVO annotations.PropertyCheckingProcess • Parameters are based on real case studies.Verification • Graph densityTechnologies • Percentage of MAVO annotated elementsExperiments • Percentages of M-, S- and V-annotated elementsResultsConclusion • 3 Model Size : Small, Medium, Large, X-Large 16 / 32
  • 27. Comparing the Effectiveness of ReasoningFormalisms for Experimental SetupPartial ModelsSaadatpanah, Famelis, Gorzny, Robinson,Chechik, SalayIntroductionDesignerUncertaintyModelingUncertaintyPropertyCheckingProcessVerificationTechnologiesExperimentsResultsConclusion 17 / 32
  • 28. Comparing the Effectiveness of ReasoningFormalisms for Relational Encoding IPartial ModelsSaadatpanah, Famelis, Gorzny, Robinson,Chechik, Salay • Relational Algebra used in DBMSsIntroductionDesigner • Directly translatable into the different formalismsUncertaintyModelingUncertainty • Intermediate representationPropertyChecking • FOL semantics of MAVOProcess • Reasoning formalismsVerificationTechnologiesExperiments • Meaningful comparisonResults • Most efficient encoding in each formalism : impossible!Conclusion • Solution : common encoding 18 / 32
  • 29. Comparing the Effectiveness of ReasoningFormalisms for Relational Encoding IIPartial ModelsSaadatpanah, Famelis, Gorzny, Robinson,Chechik, Salay (Very) high level overview.Introduction • The metamodel encodedDesigner as a schema.Uncertainty • Partial model FOLModelingUncertainty semantics encoded asPropertyChecking constraints over theProcess schema.VerificationTechnologies • Creating a concreteExperiments refinement populates theResults database.Conclusion 19 / 32
  • 30. Comparing the Effectiveness of ReasoningFormalisms for Experimental SetupPartial ModelsSaadatpanah, Famelis, Gorzny, Robinson,Chechik, SalayIntroductionDesignerUncertaintyModelingUncertaintyPropertyCheckingProcessVerificationTechnologiesExperimentsResultsConclusion 20 / 32
  • 31. Comparing the Effectiveness of ReasoningFormalisms for Translation To FormalismsPartial Models • AlloySaadatpanah, Famelis, • Relations : Alloy signatures Gorzny, Robinson, • Instances : AtomsChechik, Salay • MAVO constraints : quantified predicates over signatures • Bound is requiredIntroductionDesigner • CSPUncertainty • Relations : Finite set of IntegersModeling • Instances : IntegersUncertainty • MAVO constraints : cardinality and intersection of setsPropertyChecking • Bound is requiredProcess • SMTVerification • Relations : Uninterpreted boolean functionsTechnologies • Instances : Abstract valuesExperiments • MAVO constraints : Quantified logic over truth table ofResults functionsConclusion • ASP • Program rules for both instances and relations • Bound is required 21 / 32
  • 32. Comparing the Effectiveness of ReasoningFormalisms for Experimental SetupPartial ModelsSaadatpanah, Famelis, Gorzny, Robinson,Chechik, SalayIntroductionDesignerUncertaintyModelingUncertaintyPropertyCheckingProcessVerificationTechnologiesExperimentsResultsConclusion 22 / 32
  • 33. Comparing the Effectiveness of ReasoningFormalisms for Properties CheckedPartial ModelsSaadatpanah, Famelis, Gorzny, • Inspired by real metamodel constraints. Robinson,Chechik, Salay • No transitive closure, since it is expensive to check.IntroductionDesignerUncertainty P1: There exists a node with a self-loop.ModelingUncertainty P2: All nodes have outgoing edges.PropertyChecking P3: All nodes have outgoing or incoming edges.ProcessVerification P4: For all pairs of nodes n1 , n2 there exists at most one edge eTechnologies e such that n1 → n2ExperimentsResults P5: For every pair of nodes n1 , n2 , n1 = n2 there exist two edges e1 e2Conclusion e1 , e2 such that n1 → n2 and n2 → n1 . 23 / 32
  • 34. Comparing the Effectiveness of ReasoningFormalisms for Experimental SetupPartial ModelsSaadatpanah, Famelis, Gorzny, Robinson,Chechik, SalayIntroductionDesignerUncertaintyModelingUncertaintyPropertyCheckingProcessVerificationTechnologiesExperimentsResultsConclusion 24 / 32
  • 35. Comparing the Effectiveness of ReasoningFormalisms for Experimental ParametersPartial ModelsSaadatpanah, • Bound (2, 4, 6) Famelis, Gorzny, • Solvers (except SMT) use bound for grounding expressions Robinson,Chechik, Salay to atoms. • How many times can an ’S’-annotated element beIntroduction replicated in a refinement.DesignerUncertainty • RepetitionsModelingUncertainty • 5 timesPropertyChecking • Cutoff time/memoryProcess • less than 10 minutesVerificationTechnologies • less than 5 gigabyteExperiments • otherwise timeoutResults • What we measureConclusion • How long does it take for each solver to return an answer • A score out of 1200 • if timeout : zero! 25 / 32
  • 36. Comparing the Effectiveness of ReasoningFormalisms for FindingsPartial ModelsSaadatpanah, Famelis, Gorzny, Robinson,Chechik, SalayIntroductionDesignerUncertaintyModelingUncertaintyPropertyCheckingProcessVerificationTechnologiesExperimentsResultsConclusion 26 / 32
  • 37. Comparing the Effectiveness of ReasoningFormalisms for FindingsPartial ModelsSaadatpanah, Famelis, Gorzny, Robinson,Chechik, SalayIntroductionDesignerUncertaintyModelingUncertaintyPropertyCheckingProcessVerificationTechnologiesExperimentsResultsConclusion 26 / 32
  • 38. Comparing the Effectiveness of ReasoningFormalisms for FindingsPartial ModelsSaadatpanah, Famelis, Gorzny, Robinson,Chechik, SalayIntroductionDesignerUncertaintyModelingUncertaintyPropertyCheckingProcessVerificationTechnologiesExperimentsResultsConclusion 26 / 32
  • 39. Comparing the Effectiveness of ReasoningFormalisms for SMT the champion?Partial ModelsSaadatpanah, Famelis, Gorzny, Robinson,Chechik, Salay • Unaffected by bounds!Introduction • Works at higher level of abstractionDesigner (Theory of uninterpreted functions.)UncertaintyModeling • Unaffected by expensive grounding phase.UncertaintyPropertyCheckingProcessVerification Caveat:TechnologiesExperiments • SMT can theoretically return “I don’t know”.Results • (However: we didn’t observe that.’)Conclusion 27 / 32
  • 40. Comparing the Effectiveness of ReasoningFormalisms for Threats to ValidityPartial ModelsSaadatpanah, Famelis, Gorzny, Randomly generated graphs. Robinson,Chechik, Salay • Tuned the generator with realistic graph properties.Introduction • Values of graph properties from case studies.DesignerUncertaintyModelingUncertainty Fairness of comparisons.Property • Common encoding to level the field.CheckingProcessVerificationTechnologies Choice of specific reasoning enginesExperimentsResults • When available: winners of competitions.Conclusion • CSP: most convenient input language. 28 / 32
  • 41. Comparing the Effectiveness of ReasoningFormalisms for SummaryPartial ModelsSaadatpanah, Famelis, Gorzny, Robinson,Chechik, Salay Research Question What is the most efficient formalism for verifying modelsIntroductionDesigner containing uncertainty?UncertaintyModelingUncertaintyPropertyCheckingProcessVerificationTechnologiesExperimentsResultsConclusion 29 / 32
  • 42. Comparing the Effectiveness of ReasoningFormalisms for SummaryPartial ModelsSaadatpanah, Famelis, Gorzny, Robinson,Chechik, SalayIntroductionDesignerUncertaintyModelingUncertaintyPropertyCheckingProcessVerification Other results:Technologies • Framework for running experiments (now full tool support).ExperimentsResults • Random generator for arbitrary type graphs.Conclusion • Relational encoding. • Translations of the RA encoding to different formalisms. 29 / 32
  • 43. Comparing the Effectiveness of ReasoningFormalisms for Future WorkPartial ModelsSaadatpanah, Famelis, Gorzny, Robinson,Chechik, Salay • Implement symmetry breaking in the SMT encoding.IntroductionDesignerUncertaintyModelingUncertainty • Experiment with properties that require transitive closure.PropertyCheckingProcessVerificationTechnologiesExperiments • Experiment with partial models containing OW.ResultsConclusion 30 / 32
  • 44. Questions?
  • 45. Comparing the Effectiveness of ReasoningFormalisms for Bibliography IPartial ModelsSaadatpanah, Famelis, Gorzny, Robinson,Chechik, SalayIntroduction Famelis, M., Ben-David, S., Chechik, M., and Salay, R. (2011). “Partial Models: A Position Paper”.Designer In Proceedings of MoDeVVa’11, pages 1–6.Uncertainty Famelis, M., Chechik, M., and Salay, R. (2012).ModelingUncertainty “Partial Models: Towards Modeling and Reasoning with Uncertainty”. In Proceedings of ICSE’12.PropertyChecking Salay, R., Famelis, M., and Chechik, M. (2012).Process “Language Independent Refinement using Partial Modeling”. In Proceedings of FASE’12.VerificationTechnologiesExperimentsResultsConclusion 32 / 32