MAV-Vis: A Notation for Model Uncertainty

1. MAV-Vis: ANotation forModelUncertaintyM.Famelis,S.SantosaIntroductionDesignUncertaintyMAV-VisUser StudyConclusionMAV-Vis: A Notation for Model UncertaintyMichalis Famelis and Stephanie SantosaUniversity of TorontoMay 18th, 2013Models in Software Engineering Workshop at ICSE1 / 27

2. MAV-Vis: ANotation forModelUncertaintyM.Famelis,S.SantosaIntroductionDesignUncertaintyMAV-VisUser StudyConclusionDesign Uncertainty“The reality of today’s software systems requires us to consideruncertainty as a ﬁrst-class concern in the design, implementation,and deployment of those systems.” [Garlan, 2010]Our focus:Uncertainty about design decisions – the contents of a model.Our agenda:Enable MBSE with design uncertainty using Partial models.2 / 27

3. MAV-Vis: ANotation forModelUncertaintyM.Famelis,S.SantosaIntroductionDesignUncertaintyMAV-VisUser StudyConclusionDesign Uncertainty“The reality of today’s software systems requires us to consideruncertainty as a ﬁrst-class concern in the design, implementation,and deployment of those systems.” [Garlan, 2010]Our focus:Uncertainty about design decisions – the contents of a model.Our agenda:Enable MBSE with design uncertainty using Partial models.2 / 27

4. MAV-Vis: ANotation forModelUncertaintyM.Famelis,S.SantosaIntroductionDesignUncertaintyMAV-VisUser StudyConclusionUsable Notation for UncertaintyPartial models: effective for automated reasoning.However:Existing notation (“MAV-Text”) developed ad-hoc.Is it effective for human communication?Need to maximize cognitive effectiveness.“Physics of Notations” [Moody, 2009]Principles for designing graphical languages.What we did:• Developed a new notation: MAV-Vis• Evaluated our implementation with a user study:“Does MAV-Vis improve cognitive effectiveness?”Speed, ease, accuracy for reading and writing.3 / 27

5. 1 Introduction2 Design Uncertainty3 Designing MAV-Vis4 User Study5 Conclusion

6. MAV-Vis: ANotation forModelUncertaintyM.Famelis,S.SantosaIntroductionDesignUncertaintyMAV-VisUser StudyConclusionWhat is Design UncertaintyExample: a simple class diagram.What does the modeler know?5 / 27

7. MAV-Vis: ANotation forModelUncertaintyM.Famelis,S.SantosaIntroductionDesignUncertaintyMAV-VisUser StudyConclusionWhat is Design UncertaintyExample: a simple class diagram.What does the modeler not know?5 / 27

11. MAV-Vis: ANotation forModelUncertaintyM.Famelis,S.SantosaIntroductionDesignUncertaintyMAV-VisUser StudyConclusionUncertainty: a Set of PossibleReﬁnementsIf we remove all uncertainty, we have a concrete reﬁnement.6 / 27

12. MAV-Vis: ANotation forModelUncertaintyM.Famelis,S.SantosaIntroductionDesignUncertaintyMAV-VisUser StudyConclusionModeling Uncertainty with PartialModelsExplicating uncertainty in a partial model.7 / 27

13. MAV-Vis: ANotation forModelUncertaintyM.Famelis,S.SantosaIntroductionDesignUncertaintyMAV-VisUser StudyConclusionModeling Uncertainty with PartialModelsExplicating uncertainty in a partial model.• May: Element is optional.7 / 27

14. MAV-Vis: ANotation forModelUncertaintyM.Famelis,S.SantosaIntroductionDesignUncertaintyMAV-VisUser StudyConclusionModeling Uncertainty with PartialModelsExplicating uncertainty in a partial model.• May: Element is optional.• Abs: Element can be multiplied to many copies.7 / 27

15. MAV-Vis: ANotation forModelUncertaintyM.Famelis,S.SantosaIntroductionDesignUncertaintyMAV-VisUser StudyConclusionModeling Uncertainty with PartialModelsExplicating uncertainty in a partial model.• May: Element is optional.• Abs: Element can be multiplied to many copies.• Var: Element can be merged with others.7 / 27

16. MAV-Vis: ANotation forModelUncertaintyM.Famelis,S.SantosaIntroductionDesignUncertaintyMAV-VisUser StudyConclusionModeling Uncertainty with PartialModelsExplicating uncertainty in a partial model.• May: Element is optional.• Abs: Element can be multiplied to many copies.• Var: Element can be merged with others.• OW: Model is incomplete.7 / 27

17. MAV-Vis: ANotation forModelUncertaintyM.Famelis,S.SantosaIntroductionDesignUncertaintyMAV-VisUser StudyConclusionModeling Uncertainty with PartialModelsExplicating uncertainty in a partial model.• May: Element is optional.• Abs: Element can be multiplied to many copies.• Var: Element can be merged with others.• OW: Model is incomplete.7 / 27

18. MAV-Vis: ANotation forModelUncertaintyM.Famelis,S.SantosaIntroductionDesignUncertaintyMAV-VisUser StudyConclusion“Extended” Partial ModelsExpressing dependencies between points of uncertainty.e.g. May Model: variant presented in [ICSE’12]alternative reﬁnements and relations between them.8 / 27

19. MAV-Vis: ANotation forModelUncertaintyM.Famelis,S.SantosaIntroductionDesignUncertaintyMAV-VisUser StudyConclusionExisting Notation: MAV-Text9 / 27

20. MAV-Vis: ANotation forModelUncertaintyM.Famelis,S.SantosaIntroductionDesignUncertaintyMAV-VisUser StudyConclusionAssessing MAV-TextMAV-Text: Textual annotations + Propositional formulaAssessment (based on [Moody, 2009]):Pros:• Graphically economical• 1:1 symbol-concept correspondenceCons:• Hard to visually distinguish symbols• Does not intuitively suggest meaning• Must annotate each element individually• Does not take advantage of graphics• Dependencies “hidden” in formula• Must know how to read propositional formulas10 / 27

22. MAV-Vis: ANotation forModelUncertaintyM.Famelis,S.SantosaIntroductionDesignUncertaintyMAV-VisUser StudyConclusionIntroducing MAV-Vis12 / 27

23. MAV-Vis: ANotation forModelUncertaintyM.Famelis,S.SantosaIntroductionDesignUncertaintyMAV-VisUser StudyConclusionIntroducing MAV-VisRepresenting Var12 / 27

24. MAV-Vis: ANotation forModelUncertaintyM.Famelis,S.SantosaIntroductionDesignUncertaintyMAV-VisUser StudyConclusionIntroducing MAV-VisRepresenting Abs12 / 27

25. MAV-Vis: ANotation forModelUncertaintyM.Famelis,S.SantosaIntroductionDesignUncertaintyMAV-VisUser StudyConclusionIntroducing MAV-VisRepresenting May: a color for each PoU12 / 27

26. MAV-Vis: ANotation forModelUncertaintyM.Famelis,S.SantosaIntroductionDesignUncertaintyMAV-VisUser StudyConclusionIntroducing MAV-VisRepresenting May: identify alternative12 / 27

27. MAV-Vis: ANotation forModelUncertaintyM.Famelis,S.SantosaIntroductionDesignUncertaintyMAV-VisUser StudyConclusionIntroducing MAV-VisRepresenting May: grouping elements in alternatives12 / 27

28. MAV-Vis: ANotation forModelUncertaintyM.Famelis,S.SantosaIntroductionDesignUncertaintyMAV-VisUser StudyConclusionIntroducing MAV-VisRepresenting May: the other alternative12 / 27

29. MAV-Vis: ANotation forModelUncertaintyM.Famelis,S.SantosaIntroductionDesignUncertaintyMAV-VisUser StudyConclusionIntroducing MAV-VisRepresenting May: numbers for diﬀerent alternatives12 / 27

30. MAV-Vis: ANotation forModelUncertaintyM.Famelis,S.SantosaIntroductionDesignUncertaintyMAV-VisUser StudyConclusionIntroducing MAV-VisRepresenting May: alternative with many parts12 / 27

31. MAV-Vis: ANotation forModelUncertaintyM.Famelis,S.SantosaIntroductionDesignUncertaintyMAV-VisUser StudyConclusionIntroducing MAV-VisRepresenting May: a diﬀerent PoU12 / 27

32. MAV-Vis: ANotation forModelUncertaintyM.Famelis,S.SantosaIntroductionDesignUncertaintyMAV-VisUser StudyConclusionIntroducing MAV-VisRepresenting May: expressing PoU dependencies12 / 27

36. MAV-Vis: ANotation forModelUncertaintyM.Famelis,S.SantosaIntroductionDesignUncertaintyMAV-VisUser StudyConclusionImprovements in MAV-VisBased on the principles in [Moody, 2009]:• 1:1 symbol-concept correspondence• Diﬀerent retinal vars for each symbol (shape, texture)• Notation (more) suggestive of concepts• Relationships are visualized• Grouping of annotations (not per-element)• Visually expressive (using shape, color, texture, size)• Dual coding with text and color• No need to know propositional logic (per se)• Relatively economical, cognitively manageable13 / 27

37. MAV-Vis: ANotation forModelUncertaintyM.Famelis,S.SantosaIntroductionDesignUncertaintyMAV-VisUser StudyConclusionMAV-Vis LimitationsPortability Annotation language → cannot guarranteesymbols won’t conﬂict!Implemented for Class Diagrams, E-R Diagrams.Porting to other notations not automatable.BUT: can use with any abstract syntax (MOF)Less powerful than propositional logic (of course)But dependency sub-language can be extended.No OW OW annotates entire model.(Megamodeling?)No tooling Out of scope here: focus on ideal notation.14 / 27

39. MAV-Vis: ANotation forModelUncertaintyM.Famelis,S.SantosaIntroductionDesignUncertaintyMAV-VisUser StudyConclusionOverviewMotivation Evaluate MAV-Vis as an implementation of thetheory in [Moody, 2009].Goal User study to confirm or refute the hypothesis:“MAV-Vis improves cognitive effectiveness forreading and writing compared to MAV-Text”Cognitive Effectiveness:Ease, Speed, AccuracyParticipants 12 unpaid particip., with Bach. in CS or higherAverage experience in MAVO: 2.2/5(3 participants were experts in MAVO,already familiar with MAV-Text)16 / 27

40. MAV-Vis: ANotation forModelUncertaintyM.Famelis,S.SantosaIntroductionDesignUncertaintyMAV-VisUser StudyConclusionSetupProcedure:• Tutorial• Freeform excercise• [ Reading, Writing ]x2• QuestionnaireDesign:• Within subjects to allow comparison and minimizeselection bias• 2x2 Latin square to control for:• Order of syntaxes (MAV-Vis, MAV-Text)• Modeling scenario(“Hotel Admin” in UML, “School Personnel” in E-R)Measurements:Ease Questionnaire responsesSpeed Task completion timeAccuracy Error counts and comprehension scores17 / 27

41. MAV-Vis: ANotation forModelUncertaintyM.Famelis,S.SantosaIntroductionDesignUncertaintyMAV-VisUser StudyConclusionResultsEase:• MAV-Vis considered more intuitive, easier to remember,efficient to read• MAV-Text and MAV-Vis almost tied for writing efficiency• MAV-Vis gathered more preferencesSpeed:• MAV-Text took 2:08min longer for reading.• MAV-Vis was 13sec slower for writing.Accuracy:• No difference in reading Abs, Var.• Noticeable difference for reading May(on average 1.4 fewer errors)• On average 0.7 more errors writing with MAV-Vis.(most errors were about coloring PoUs)18 / 27

42. MAV-Vis: ANotation forModelUncertaintyM.Famelis,S.SantosaIntroductionDesignUncertaintyMAV-VisUser StudyConclusionResult Tables19 / 27

43. MAV-Vis: ANotation forModelUncertaintyM.Famelis,S.SantosaIntroductionDesignUncertaintyMAV-VisUser StudyConclusionThreats to Validity• Small sample size: no statisticsResults to be understood as preliminary evidence• Prior exposure to MAVO• Familiarity with propositional logic• Confusion regarding uncertainty concepts (MAVO)(But both syntaxes aﬀected equally)• Selection bias from imbalanced knowledge of UML/E-R(Reported by 1 subject)20 / 27

44. MAV-Vis: ANotation forModelUncertaintyM.Famelis,S.SantosaIntroductionDesignUncertaintyMAV-VisUser StudyConclusionDiscussionOverall:MAV-Vis more eﬃcientbut more writing errors.Most writing errors:PoU colors.(PoU not a formal concept)However:Not necessarily universalsolution!Cognitive FitLearning Style and ExpertiseFreeform: dashes, pilesquestion marks, ellipses21 / 27

46. MAV-Vis: ANotation forModelUncertaintyM.Famelis,S.SantosaIntroductionDesignUncertaintyMAV-VisUser StudyConclusionSummaryPartial models:Formalism for management of design uncertainty.Existing work:Automated reasoning vs human communicationAd-hoc notation (MAV-Text)What we did:Developed MAV-Vis, using [Moody, 2009]Performed user study to evaluate our implementationCognitive Eﬀectiveness for reading and writing:Ease, Speed, AccuracyResults:MAV-Vis more eﬃcient but more writing errors.23 / 27

47. MAV-Vis: ANotation forModelUncertaintyM.Famelis,S.SantosaIntroductionDesignUncertaintyMAV-VisUser StudyConclusionNext StepsLarger picture:Notation only one aspect.Eﬀective methodological support for Design Uncertainty.Patterns of uncertainty? Sources?How do users already cope with Design Uncertainty?Focused on notation design, not tooling.Future:Tooling integration and impact on cognitive eﬀectiveness“MAVO-isation” of arbitrary languagesDependency sublanguage:What are reasonable extensions?Relation to patterns.Impact on tooling.24 / 27

48. Questions?

49. MAV-Vis: ANotation forModelUncertaintyM.Famelis,S.SantosaIntroductionDesignUncertaintyMAV-VisUser StudyConclusionBibliography IFamelis, M., Ben-David, S., Chechik, M., and Salay, R. (2011).“Partial Models: A Position Paper”.In Proceedings of MoDeVVa’11, pages 1–6.Famelis, M., Chechik, M., and Salay, R. (2012a).“Partial Models: Towards Modeling and Reasoning with Uncertainty”.In Proceedings of ICSE’12.Famelis, M., Salay, R., and Chechik, M. (2012b).The semantics of partial model transformations.In MISE at ICSE’12, pages 64 –69.Garlan, D. (2010).Software engineering in an uncertain world.In FoSER ’10, pages 125–128.Moody, D. (2009).The “Physics” of Notations: Toward a Scientific Basis for Constructing Visual Notations in SoftwareEngineering.TSE, 35(6):756–779.Saadatpanah, P., Famelis, M., Gorzny, J., Robinson, N., Chechik, M., and Salay, R. (2012).Comparing the effectiveness of reasoning formalisms for partial models.In MoDeVVa’12.Salay, R., Chechik, M., and Gorzny, J. (2012a).“Towards a Methodology for Verifying Partial Model Refinements”.In Proceedings of VOLT’12.26 / 27

50. MAV-Vis: ANotation forModelUncertaintyM.Famelis,S.SantosaIntroductionDesignUncertaintyMAV-VisUser StudyConclusionBibliography IISalay, R., Famelis, M., and Chechik, M. (2012b).“Language Independent Reﬁnement using Partial Modeling”.In Proceedings of FASE’12.27 / 27