Your SlideShare is downloading. ×
0
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
ICPC12a.ppt
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

ICPC12a.ppt

86

Published on

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
86
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Professional status vs. ExpertiseZ´phyrin Soh et al. eProblem andMotivations Professional Status and Expertise for UMLProblemMotivations Class Diagram Comprehension:Related WorkExpertise Studies An Empirical StudyUML Class DiagramComprehensionEmpirical StudyStudy Design Z´phyrin Soh, Zohreh Sharafi, Bertrand Van den Plas, eResultsRQ1: Status Gerardo Cepeda Porras, Yann-Ga¨l Gu´h´neuc and e e eRQ2: ExpertiseRQ3: Status vs. Expertise Giuliano AntoniolRQ4: Question PrecisionConclusion and Department of Computer and Software EngineeringFuture Work ´ Ecole Polytechnique de Montr´al, Qu´bec, Canada e eConclusionThreats to Validity andFuture Work June 13, 2012 Pattern Trace Identification, Detection, and Enhancement in Java SOftware Cost-effective Change and Evolution Research Lab
  • 2. Professional status vs. Expertise OutlineZ´phyrin Soh et al. e Problem and MotivationsProblem andMotivations ProblemProblemMotivations MotivationsRelated Work Related WorkExpertise StudiesUML Class Diagram Expertise StudiesComprehension UML Class Diagram ComprehensionEmpirical StudyStudy Design Empirical StudyResults Study DesignRQ1: StatusRQ2: ExpertiseRQ3: Status vs. Expertise ResultsRQ4: Question Precision RQ1: StatusConclusion andFuture Work RQ2: ExpertiseConclusion RQ3: Status vs. ExpertiseThreats to Validity andFuture Work RQ4: Question Precision Conclusion and Future Work Conclusion Threats to Validity and Future Work 2 / 24
  • 3. Professional status vs. Expertise Problem and MotivationsZ´phyrin Soh et al. e Problem (1/1)Problem andMotivations What is experience?ProblemMotivations To manage subject/programmer experience:Related WorkExpertise Studies Years and education as main criteria [1]UML Class DiagramComprehension Authors sometime combine many criteriaEmpirical StudyStudy DesignResultsRQ1: StatusRQ2: ExpertiseRQ3: Status vs. ExpertiseRQ4: Question PrecisionConclusion andFuture WorkConclusionThreats to Validity andFuture Work [1] J. Feigenspan et al., Measuring Programming Experience, ICPC 2012, pp. 3 / 24 73-82.
  • 4. Professional status vs. Expertise Problem and MotivationsZ´phyrin Soh et al. e Motivations (1/1)Problem andMotivationsProblemMotivationsRelated Work MotivationsExpertise StudiesUML Class Diagram Consider two following cases:ComprehensionEmpirical Study A student who used UML for 4 years during her studyStudy Design A professional with 3 years of experience with UMLResultsRQ1: Status Who is the best at understanding of UML classRQ2: ExpertiseRQ3: Status vs. Expertise diagrams?RQ4: Question PrecisionConclusion andFuture WorkConclusionThreats to Validity andFuture Work 4 / 24
  • 5. Professional status vs. Expertise Problem and MotivationsZ´phyrin Soh et al. e Motivations (1/1)Problem andMotivationsProblemMotivationsRelated Work MotivationsExpertise StudiesUML Class Diagram Consider two following cases:ComprehensionEmpirical Study A student who used UML for 4 years during her studyStudy Design A professional with 3 years of experience with UMLResultsRQ1: Status Who is the best at understanding of UML classRQ2: ExpertiseRQ3: Status vs. Expertise diagrams?RQ4: Question Precision Project managers when recruiting a new softwareConclusion andFuture Work designer by prioritized the important “factor”ConclusionThreats to Validity andFuture Work Future designers to know “where” to acquire the competitive skills by considering the important “factor” 4 / 24
  • 6. Professional status vs. Expertise Related WorkZ´phyrin Soh et al. e Expertise Studies (1/3)Problem andMotivations Previous work on expertiseProblemMotivations Novices spend less time than experts analysts [2]Related WorkExpertise StudiesUML Class DiagramComprehensionEmpirical StudyStudy DesignResultsRQ1: StatusRQ2: ExpertiseRQ3: Status vs. ExpertiseRQ4: Question PrecisionConclusion andFuture WorkConclusionThreats to Validity andFuture Work [2] K. D. Schenk, N. P. Vitalari, and K. S. Davis, Differences between novice and expert systems analysts: what do we know and what do we do?, Journal of Management Information System, vol. 15, pp. 9-50, June 1998 5 / 24
  • 7. Professional status vs. Expertise Related WorkZ´phyrin Soh et al. e Expertise Studies (1/3)Problem andMotivations Previous work on expertiseProblemMotivations Novices spend less time than experts analysts [2]Related WorkExpertise Studies Graduate students are faster than junior ones andUML Class DiagramComprehension intermediate professionals [3]Empirical StudyStudy DesignResultsRQ1: StatusRQ2: ExpertiseRQ3: Status vs. ExpertiseRQ4: Question PrecisionConclusion andFuture WorkConclusionThreats to Validity andFuture Work [3] E. Arisholm and D. I. K. Sjøberg, Evaluating the effect of a delegated versus centralized control style on the maintainability of object-oriented software,IEEE Transactions on Software Engineering, vol. 30, no. 8, pp. 5 / 24 521-534, aug. 2004
  • 8. Professional status vs. Expertise Related WorkZ´phyrin Soh et al. e Expertise Studies (1/3)Problem andMotivations Previous work on expertiseProblemMotivations Novices spend less time than experts analysts [2]Related WorkExpertise Studies Graduate students are faster than junior ones andUML Class DiagramComprehension intermediate professionals [3]Empirical StudyStudy Design Experts are better for abstract questions and novices areResults better for concrete questions [4]RQ1: StatusRQ2: ExpertiseRQ3: Status vs. ExpertiseRQ4: Question PrecisionConclusion andFuture WorkConclusionThreats to Validity andFuture Work [4] B. Adelson, When novices surpass experts: The difficulty of a task may increase with expertise,Journal of Experimental Psychology: Learning, Memory, and Cognition, vol. 10, pp. 483-495, Jul. 1984 5 / 24
  • 9. Professional status vs. Expertise Related WorkZ´phyrin Soh et al. e Expertise Studies (1/3)Problem andMotivations Previous work on expertiseProblemMotivations Novices spend less time than experts analysts [2]Related WorkExpertise Studies Graduate students are faster than junior ones andUML Class DiagramComprehension intermediate professionals [3]Empirical StudyStudy Design Experts are better for abstract questions and novices areResults better for concrete questions [4]RQ1: StatusRQ2: Expertise Experts and novices have different program model forRQ3: Status vs. ExpertiseRQ4: Question Precision documentation task, no difference for reuse task [5]Conclusion andFuture WorkConclusionThreats to Validity andFuture Work [5] J.-M. Burkhardt, F. D´tienne, and S. Wiedenbeck, Object-oriented e program comprehension: Effect of expertise, task and phase, Empirical Software Engineering, vol. 7, no. 2, pp. 115-156, 2002 5 / 24
  • 10. Professional status vs. Expertise Related WorkZ´phyrin Soh et al. e Expertise Studies (2/3)Problem andMotivationsProblemMotivationsRelated WorkExpertise Studies ComparisonUML Class DiagramComprehension To compare our work with previous work, we consider:Empirical StudyStudy DesignResultsRQ1: Status ⇒ ObjectRQ2: ExpertiseRQ3: Status vs. ExpertiseRQ4: Question Precision ⇒ Kind of task/questionConclusion andFuture WorkConclusion ⇒ Subject categorisation criterionThreats to Validity andFuture Work 6 / 24
  • 11. Professional status vs. Expertise Related WorkZ´phyrin Soh et al. e Expertise Studies (2/3)Problem andMotivations ComparisonProblemMotivationsRelated Work Ref.Expertise StudiesUML Class Diagram years of experienceComprehension requirements [2] textual description analysis rating scale ofEmpirical Study supervisorsStudy Design students andResults [3] Java program change task professionalsRQ1: StatusRQ2: Expertise undergraduate program + abstract +RQ3: Status vs. ExpertiseRQ4: Question Precision [4] flowcharts concrete question students fellow teachersConclusion andFuture Work students and ex- documentationConclusion [5] database program and reuse perts (nominationThreats to Validity andFuture Work by colleagues, ...) Legend: Same to our work | Different to our work [2] Schenk et al. (1998) [3] Arisholm and Sjøberg (2004) [4] Adelson (1984) 6 / 24 [5] Burkhardt et al. (2002)
  • 12. Professional status vs. Expertise Related WorkZ´phyrin Soh et al. e Expertise Studies (3/3)Problem andMotivations LimitationsProblemMotivations Previous work:Related WorkExpertise Studies Did not precisely distinguish years of experience andUML Class DiagramComprehension professionalism:Empirical Study Inexperienced students as novices [5]Study Design Senior professionals with less years of programmingResults experience than graduate students [3]RQ1: StatusRQ2: ExpertiseRQ3: Status vs. ExpertiseRQ4: Question PrecisionConclusion andFuture WorkConclusionThreats to Validity andFuture Work [5] Burkhardt et al. (2002) 7 / 24 [3] Arisholm and Sjøberg (2004)
  • 13. Professional status vs. Expertise Related WorkZ´phyrin Soh et al. e Expertise Studies (3/3)Problem andMotivations LimitationsProblemMotivations Previous work:Related WorkExpertise Studies Did not precisely distinguish years of experience andUML Class DiagramComprehension professionalism:Empirical Study Inexperienced students as novices [5]Study Design Senior professionals with less years of programmingResults experience than graduate students [3]RQ1: StatusRQ2: Expertise We distinguish the years of experience fromRQ3: Status vs. ExpertiseRQ4: Question Precision professionalismConclusion andFuture WorkConclusionThreats to Validity andFuture Work [5] Burkhardt et al. (2002) 7 / 24 [3] Arisholm and Sjøberg (2004)
  • 14. Professional status vs. Expertise Related WorkZ´phyrin Soh et al. e Expertise Studies (3/3)Problem andMotivations LimitationsProblemMotivations Previous work:Related WorkExpertise Studies Did not precisely distinguish years of experience andUML Class DiagramComprehension professionalism:Empirical Study Inexperienced students as novices [5]Study Design Senior professionals with less years of programmingResults experience than graduate students [3]RQ1: StatusRQ2: Expertise We distinguish the years of experience fromRQ3: Status vs. ExpertiseRQ4: Question Precision professionalismConclusion andFuture Work Studied the source code or textual descriptions ofConclusionThreats to Validity andFuture Work requirements [5] Burkhardt et al. (2002) 7 / 24 [3] Arisholm and Sjøberg (2004)
  • 15. Professional status vs. Expertise Related WorkZ´phyrin Soh et al. e Expertise Studies (3/3)Problem andMotivations LimitationsProblemMotivations Previous work:Related WorkExpertise Studies Did not precisely distinguish years of experience andUML Class DiagramComprehension professionalism:Empirical Study Inexperienced students as novices [5]Study Design Senior professionals with less years of programmingResults experience than graduate students [3]RQ1: StatusRQ2: Expertise We distinguish the years of experience fromRQ3: Status vs. ExpertiseRQ4: Question Precision professionalismConclusion andFuture Work Studied the source code or textual descriptions ofConclusionThreats to Validity andFuture Work requirements We use the UML class diagram [5] Burkhardt et al. (2002) 7 / 24 [3] Arisholm and Sjøberg (2004)
  • 16. Professional status vs. Expertise Related WorkZ´phyrin Soh et al. e UML Class Diagram Comprehension (1/2)Problem andMotivations UML and eye-trackingProblemMotivations Stereotype, color, and layout facilitate class diagramRelated Work exploration and comprehension [6]Expertise StudiesUML Class DiagramComprehension Multi-cluster (by requirement) and three-cluster (byEmpirical Study stereotype) layout positively affect the comprehension ofStudy Design class diagrams [7]ResultsRQ1: StatusRQ2: ExpertiseRQ3: Status vs. ExpertiseRQ4: Question PrecisionConclusion andFuture WorkConclusionThreats to Validity andFuture Work [6] S. Yusuf, H. Kagdi, and J. I. Maletic, Assessing the comprehension of UML diagrams via eye tracking, ICPC’07 [7] B. Sharif and J. I. Maletic, An empirical study on the comprehension of stereotyped UML class diagram layouts, ICPC’09 8 / 24
  • 17. Professional status vs. Expertise Related WorkZ´phyrin Soh et al. e UML Class Diagram Comprehension (1/2)Problem andMotivations UML and eye-trackingProblemMotivations Stereotype, color, and layout facilitate class diagramRelated Work exploration and comprehension [6]Expertise StudiesUML Class DiagramComprehension Multi-cluster (by requirement) and three-cluster (byEmpirical Study stereotype) layout positively affect the comprehension ofStudy Design class diagrams [7]ResultsRQ1: StatusRQ2: Expertise Canonical representation of the Visitor pattern in classRQ3: Status vs. Expertise diagram reduce the effort of maintenance task [8]RQ4: Question PrecisionConclusion and The representations of design patterns affect theFuture WorkConclusion identification of their participants and their roles [9]Threats to Validity andFuture Work [8] S. Jeanmart, Y.-G. Gu´h´neuc, H. Sahraoui, and N. Habra, Impact of the e e visitor pattern on program comprehension and maintenance, ESEM’09, Oct 2009, pp. 69-78 [9] G. Cepeda Porras and Y.-G. Gu´h´neuc, An empirical study on the e e efficiency of different design pattern representations in UML class diagrams, 8 / 24 Empirical Software Engineering, vol. 15, no. 5, pp. 493-522, 2010
  • 18. Professional status vs. Expertise Related WorkZ´phyrin Soh et al. e UML Class Diagram Comprehension (2/2)Problem andMotivationsProblemMotivationsRelated Work Subjects’ categoriesExpertise StudiesUML Class Diagram Previous work used subject’s proficiency as categorisationComprehension criterion:Empirical StudyStudy Design Subjects’ performance in task realizationResultsRQ1: Status Subjects’ grade in the course they were enrolledRQ2: ExpertiseRQ3: Status vs. ExpertiseRQ4: Question PrecisionConclusion andFuture WorkConclusionThreats to Validity andFuture Work 9 / 24
  • 19. Professional status vs. Expertise Related WorkZ´phyrin Soh et al. e UML Class Diagram Comprehension (2/2)Problem andMotivationsProblemMotivationsRelated Work Subjects’ categoriesExpertise StudiesUML Class Diagram Previous work used subject’s proficiency as categorisationComprehension criterion:Empirical StudyStudy Design Subjects’ performance in task realizationResultsRQ1: Status Subjects’ grade in the course they were enrolledRQ2: ExpertiseRQ3: Status vs. ExpertiseRQ4: Question Precision MotivationsConclusion andFuture Work No previous work that uses the maintenance task onConclusionThreats to Validity and UML class diagrams and eye-tracking system to studyFuture Work separately the professional status and the expertise Combine expertise studies and UML eye-tracking studies 9 / 24
  • 20. Professional status vs. Expertise Empirical StudyZ´phyrin Soh et al. e Study Design (1/8)Problem andMotivationsProblemMotivationsRelated Work Research QuestionsExpertise StudiesUML Class Diagram RQ1: What is the relation between a designer’sComprehensionEmpirical Study professional status and her class diagramStudy Design comprehension?ResultsRQ1: StatusRQ2: ExpertiseRQ3: Status vs. ExpertiseRQ4: Question PrecisionConclusion andFuture WorkConclusionThreats to Validity andFuture Work 10 / 24
  • 21. Professional status vs. Expertise Empirical StudyZ´phyrin Soh et al. e Study Design (1/8)Problem andMotivationsProblemMotivationsRelated Work Research QuestionsExpertise StudiesUML Class Diagram RQ1: What is the relation between a designer’sComprehensionEmpirical Study professional status and her class diagramStudy Design comprehension?ResultsRQ1: Status RQ2: What is the relation between a designer’sRQ2: ExpertiseRQ3: Status vs. Expertise expertise and her class diagram comprehension?RQ4: Question PrecisionConclusion andFuture WorkConclusionThreats to Validity andFuture Work 10 / 24
  • 22. Professional status vs. Expertise Empirical StudyZ´phyrin Soh et al. e Study Design (1/8)Problem andMotivationsProblemMotivationsRelated Work Research QuestionsExpertise StudiesUML Class Diagram RQ1: What is the relation between a designer’sComprehensionEmpirical Study professional status and her class diagramStudy Design comprehension?ResultsRQ1: Status RQ2: What is the relation between a designer’sRQ2: ExpertiseRQ3: Status vs. Expertise expertise and her class diagram comprehension?RQ4: Question Precision RQ3: What is the most important factor betweenConclusion andFuture Work expertise and professional status?ConclusionThreats to Validity andFuture Work 10 / 24
  • 23. Professional status vs. Expertise Empirical StudyZ´phyrin Soh et al. e Study Design (1/8)Problem andMotivationsProblemMotivationsRelated Work Research QuestionsExpertise StudiesUML Class Diagram RQ1: What is the relation between a designer’sComprehensionEmpirical Study professional status and her class diagramStudy Design comprehension?ResultsRQ1: Status RQ2: What is the relation between a designer’sRQ2: ExpertiseRQ3: Status vs. Expertise expertise and her class diagram comprehension?RQ4: Question Precision RQ3: What is the most important factor betweenConclusion andFuture Work expertise and professional status?ConclusionThreats to Validity andFuture Work RQ4: What is the effect of the question precision on the comprehension of a UML class diagram? 10 / 24
  • 24. Professional status vs. Expertise Empirical StudyZ´phyrin Soh et al. e Study Design (2/8)Problem andMotivationsProblemMotivationsRelated Work Objects and TasksExpertise StudiesUML Class DiagramComprehension = ArgoUML, JUnit, and QuickUMLEmpirical StudyStudy Design Number of Average AverageResults classes/ number of number ofRQ1: Status Interfaces attributes per methods perRQ2: Expertise Class/Interface Class/InterfaceRQ3: Status vs. ExpertiseRQ4: Question Precision ArgoUML 10 0.4 8.6Conclusion and JUnit 14 0.57 6.14Future WorkConclusion QuickUML 16 1.75 3.87Threats to Validity andFuture Work = : one maintenance task per object 11 / 24
  • 25. Professional status vs. Expertise Empirical StudyZ´phyrin Soh et al. e Study Design (3/8)Problem andMotivationsProblem Independent variablesMotivationsRelated Work = Professional status + ExpertiseExpertise StudiesUML Class DiagramComprehensionEmpirical StudyStudy DesignResultsRQ1: StatusRQ2: ExpertiseRQ3: Status vs. ExpertiseRQ4: Question PrecisionConclusion andFuture WorkConclusionThreats to Validity andFuture Work 12 / 24
  • 26. Professional status vs. Expertise Empirical StudyZ´phyrin Soh et al. e Study Design (3/8)Problem andMotivationsProblem Independent variablesMotivationsRelated Work = Professional status + ExpertiseExpertise StudiesUML Class DiagramComprehension Professional statusEmpirical Study = practitioners (9)Study Design = students (12)Results (in industry)RQ1: StatusRQ2: ExpertiseRQ3: Status vs. ExpertiseRQ4: Question PrecisionConclusion andFuture WorkConclusionThreats to Validity andFuture Work 12 / 24
  • 27. Professional status vs. Expertise Empirical StudyZ´phyrin Soh et al. e Study Design (3/8)Problem andMotivationsProblem Independent variablesMotivationsRelated Work = Professional status + ExpertiseExpertise StudiesUML Class DiagramComprehension Professional statusEmpirical Study = practitioners (9)Study Design = students (12)Results (in industry)RQ1: StatusRQ2: ExpertiseRQ3: Status vs. Expertise Expertise: We used the number of years of experienceRQ4: Question Precision to categorise experts and novices.Conclusion and Pair-wise Wilcoxon comparison (+ BonferroniFuture WorkConclusion correction)Threats to Validity andFuture Work Categorization with the highlest Cliff’s δ value = experts (12): {3, 4, 5} years of experience = novices (9): {1, 2} years of experience 12 / 24
  • 28. Professional status vs. Expertise Empirical StudyZ´phyrin Soh et al. e Study Design (4/8)Problem andMotivationsProblemMotivationsRelated WorkExpertise StudiesUML Class DiagramComprehension Mitigating variableEmpirical Study Question precision: The level of details in the formulationStudy Design of the question:ResultsRQ1: Status Precise: state the kind of operation to performRQ2: ExpertiseRQ3: Status vs. Expertise (add/remove) and the kind of target elementRQ4: Question PrecisionConclusion and (class/method/attribute)Future WorkConclusion Not precise: no operation or target elementThreats to Validity andFuture Work 13 / 24
  • 29. Professional status vs. Expertise Empirical StudyZ´phyrin Soh et al. e Study Design (5/8)Problem andMotivations Dependent variablesProblemMotivations Accuracy, Time spentRelated WorkExpertise Studies Search effort = convex hull & spatial density [10]UML Class DiagramComprehension Overall effort = AFD [9] and NRRF [8]Empirical StudyStudy Design Question comprehension effort = NDQA and NFQAResultsRQ1: StatusRQ2: ExpertiseRQ3: Status vs. ExpertiseRQ4: Question Precision AFD: Average Fixation Duration NRRF: Normalized Rate of Relevant FixationsConclusion andFuture Work NDQA: Normalized Duration in Question AreaConclusion NFQA: Normalized Fixations in Question AreaThreats to Validity andFuture Work [8] Jeanmart et al. (2009) [9] Cepeda Porras and Gu´h´neuc (2010) e e [10] J. H. Goldberg and X. P. Kotval, Computer interface evaluation using eye movements: methods and constructs, Journal of Industrial Ergonomics, vol. 14 / 24 24, no. 6, pp. 631-645, 1999
  • 30. Professional status vs. Expertise Empirical StudyZ´phyrin Soh et al. e Study Design (6/8)Problem andMotivationsProblemMotivationsRelated WorkExpertise StudiesUML Class DiagramComprehensionEmpirical Study Convex hull areaStudy Design Smaller convex set of fixationsResultsRQ1: Status containing all subject’s fixationsRQ2: ExpertiseRQ3: Status vs. Expertise Smaller convex hull ⇒ close fixationsRQ4: Question PrecisionConclusion and ⇒ less search effortFuture WorkConclusionThreats to Validity andFuture Work 15 / 24
  • 31. Professional status vs. Expertise Empirical StudyZ´phyrin Soh et al. e Study Design (7/8)Problem andMotivations Spatial densityProblemMotivations Number of visited cells / total number of cellsRelated WorkExpertise Studies less visits ⇒ less search effortUML Class DiagramComprehensionEmpirical StudyStudy DesignResultsRQ1: StatusRQ2: ExpertiseRQ3: Status vs. ExpertiseRQ4: Question PrecisionConclusion andFuture WorkConclusionThreats to Validity andFuture Work In TAUPE [11], cell’s size = 64x64px [11] B. D. Smet, L. Lempereur, Z. Sharafi, Y.-G. Gu´h´neuc, G. Antoniol, and e e N. Habra, Taupe: Visualising and analysing eye-tracking data, Science of 16 / 24 Computer Programming, 2011
  • 32. Professional status vs. Expertise Empirical StudyZ´phyrin Soh et al. e Study Design (8/8) Overall effort: Fixations’ duration and relevanceProblem andMotivationsProblemMotivationsRelated WorkExpertise StudiesUML Class DiagramComprehensionEmpirical StudyStudy DesignResultsRQ1: StatusRQ2: ExpertiseRQ3: Status vs. ExpertiseRQ4: Question PrecisionConclusion andFuture WorkConclusionThreats to Validity andFuture Work 17 / 24
  • 33. Professional status vs. Expertise Empirical StudyZ´phyrin Soh et al. e Study Design (8/8) Question Comprehension Effort: Fixations’ count and durationProblem andMotivationsProblemMotivationsRelated WorkExpertise StudiesUML Class DiagramComprehensionEmpirical StudyStudy DesignResultsRQ1: StatusRQ2: ExpertiseRQ3: Status vs. ExpertiseRQ4: Question PrecisionConclusion andFuture WorkConclusionThreats to Validity andFuture Work 17 / 24
  • 34. Professional status vs. Expertise Results What is the relation between aZ´phyrin Soh et al. e RQ1: Status (1/1) designer’s professional status and her class diagram comprehension?Problem andMotivationsProblemMotivationsRelated WorkExpertise StudiesUML Class DiagramComprehensionEmpirical StudyStudy DesignResultsRQ1: StatusRQ2: ExpertiseRQ3: Status vs. ExpertiseRQ4: Question PrecisionConclusion andFuture WorkConclusionThreats to Validity andFuture Work Practitioners are more accurate than students 18 / 24
  • 35. Professional status vs. Expertise Results What is the relation between aZ´phyrin Soh et al. e RQ1: Status (1/1) designer’s professional status and her class diagram comprehension?Problem andMotivationsProblemMotivationsRelated WorkExpertise StudiesUML Class DiagramComprehensionEmpirical StudyStudy DesignResultsRQ1: StatusRQ2: ExpertiseRQ3: Status vs. ExpertiseRQ4: Question PrecisionConclusion andFuture WorkConclusionThreats to Validity andFuture Work Practitioners are more accurate than students Students spent around 35% less time than practitioners 18 / 24
  • 36. Professional status vs. Expertise Results What is the relation between aZ´phyrin Soh et al. e RQ1: Status (1/1) designer’s professional status and her class diagram comprehension?Problem andMotivationsProblemMotivationsRelated WorkExpertise StudiesUML Class DiagramComprehensionEmpirical StudyStudy DesignResultsRQ1: StatusRQ2: ExpertiseRQ3: Status vs. ExpertiseRQ4: Question PrecisionConclusion andFuture WorkConclusionThreats to Validity andFuture Work Practitioners are more accurate than students Students spent around 35% less time than practitioners No significant difference for other dependent variables 18 / 24
  • 37. Professional status vs. Expertise Results What is the relation between aZ´phyrin Soh et al. e RQ1: Status (1/1) designer’s professional status and her class diagram comprehension?Problem andMotivations 100 100Problem q q qq q q q q qMotivations 80 80Related WorkExpertise Studies q q q q q q q qUML Class Diagram Accuracy (%) Accuracy (%) 60 60ComprehensionEmpirical Study 40 40Study Design q qResults 20 20RQ1: StatusRQ2: Expertise q q 0 0RQ3: Status vs. ExpertiseRQ4: Question Precision 100 300 500 700 150 250 350Conclusion and Time spent (s) Time spent (s)Future Work (a) Accuracy − Time tradeoff for Practitioners (b) Accuracy − Time tradeoff for StudentsConclusionThreats to Validity andFuture Work Practitioners are more accurate than students Students spent around 35% less time than practitioners No significant difference for other dependent variables Students could be more accurate if spending more time 18 / 24
  • 38. Professional status vs. Expertise Results What is the relation between aZ´phyrin Soh et al. e RQ2: Expertise (1/1) designer’s expertise and her class diagram comprehension?Problem andMotivationsProblemMotivationsRelated WorkExpertise StudiesUML Class DiagramComprehensionEmpirical StudyStudy DesignResultsRQ1: StatusRQ2: ExpertiseRQ3: Status vs. ExpertiseRQ4: Question PrecisionConclusion andFuture WorkConclusionThreats to Validity and Experts are more accurate than novicesFuture Work 19 / 24
  • 39. Professional status vs. Expertise Results What is the relation between aZ´phyrin Soh et al. e RQ2: Expertise (1/1) designer’s expertise and her class diagram comprehension?Problem andMotivationsProblemMotivationsRelated WorkExpertise StudiesUML Class DiagramComprehensionEmpirical StudyStudy DesignResultsRQ1: StatusRQ2: ExpertiseRQ3: Status vs. ExpertiseRQ4: Question PrecisionConclusion andFuture WorkConclusionThreats to Validity and Experts are more accurate than novicesFuture Work Novices spent around 33% less time than experts 19 / 24
  • 40. Professional status vs. Expertise Results What is the relation between aZ´phyrin Soh et al. e RQ2: Expertise (1/1) designer’s expertise and her class diagram comprehension?Problem andMotivationsProblemMotivationsRelated WorkExpertise StudiesUML Class DiagramComprehensionEmpirical StudyStudy DesignResultsRQ1: StatusRQ2: ExpertiseRQ3: Status vs. ExpertiseRQ4: Question PrecisionConclusion andFuture WorkConclusionThreats to Validity and Experts are more accurate than novicesFuture Work Novices spent around 33% less time than experts Experts have a more efficient ability to search relevant elements than novices 19 / 24
  • 41. Professional status vs. Expertise Results What is the relation between aZ´phyrin Soh et al. e RQ2: Expertise (1/1) designer’s expertise and her class diagram comprehension?Problem andMotivationsProblemMotivationsRelated WorkExpertise StudiesUML Class DiagramComprehensionEmpirical StudyStudy DesignResultsRQ1: StatusRQ2: ExpertiseRQ3: Status vs. ExpertiseRQ4: Question PrecisionConclusion andFuture WorkConclusionThreats to Validity and Experts are more accurate than novicesFuture Work Novices spent around 33% less time than experts Experts have a more efficient ability to search relevant elements than novices No significant difference for other dependent variables 19 / 24
  • 42. Professional status vs. Expertise Results What is the relation between aZ´phyrin Soh et al. e RQ2: Expertise (1/1) designer’s expertise and her class diagram comprehension?Problem andMotivations 100 100Problem qq qq qq q qMotivations 80 80Related WorkExpertise Studies q q q q q q q qUML Class Diagram Accuracy (%) Accuracy (%) 60 60ComprehensionEmpirical Study 40 40Study Design q qResults 20 20RQ1: StatusRQ2: Expertise q q 0 0RQ3: Status vs. ExpertiseRQ4: Question Precision 100 300 500 700 150 250 350Conclusion and Time spent (s) Time spent (s)Future Work (a) Accuracy − Time tradeoff for Experts (a) Accuracy − Time tradeoff for NovicesConclusionThreats to Validity and Experts are more accurate than novicesFuture Work Novices spent around 33% less time than experts Experts have a more efficient ability to search relevant elements than novices No significant difference for other dependent variables 19 / 24 Novices could be more accurate if spending more time
  • 43. Professional status vs. Expertise Results What is the most important factorZ´phyrin Soh et al. e RQ3: Status vs. Expertise (1/1) between expertise and professional status?Problem andMotivationsProblemMotivationsRelated WorkExpertise StudiesUML Class DiagramComprehensionEmpirical StudyStudy DesignResults Experts are more accurate than practitionersRQ1: StatusRQ2: ExpertiseRQ3: Status vs. ExpertiseRQ4: Question PrecisionConclusion andFuture WorkConclusionThreats to Validity andFuture Work 20 / 24
  • 44. Professional status vs. Expertise Results What is the most important factorZ´phyrin Soh et al. e RQ3: Status vs. Expertise (1/1) between expertise and professional status?Problem andMotivationsProblemMotivationsRelated WorkExpertise StudiesUML Class DiagramComprehensionEmpirical StudyStudy DesignResults Experts are more accurate than practitionersRQ1: StatusRQ2: ExpertiseRQ3: Status vs. Expertise Experts spent around 7% less time than practitionersRQ4: Question PrecisionConclusion andFuture WorkConclusionThreats to Validity andFuture Work 20 / 24
  • 45. Professional status vs. Expertise Results What is the most important factorZ´phyrin Soh et al. e RQ3: Status vs. Expertise (1/1) between expertise and professional status?Problem andMotivationsProblemMotivationsRelated WorkExpertise StudiesUML Class DiagramComprehensionEmpirical StudyStudy DesignResults Experts are more accurate than practitionersRQ1: StatusRQ2: ExpertiseRQ3: Status vs. Expertise Experts spent around 7% less time than practitionersRQ4: Question Precision When considering expert subjectsConclusion andFuture WorkConclusionThreats to Validity andFuture Work 20 / 24
  • 46. Professional status vs. Expertise Results What is the most important factorZ´phyrin Soh et al. e RQ3: Status vs. Expertise (1/1) between expertise and professional status?Problem andMotivationsProblemMotivationsRelated WorkExpertise StudiesUML Class DiagramComprehensionEmpirical StudyStudy DesignResultsRQ1: StatusRQ2: Expertise Experts are more accurate than practitionersRQ3: Status vs. ExpertiseRQ4: Question Precision Experts spent around 7% less time than practitionersConclusion and When considering expert subjectsFuture WorkConclusion Experienced students are more accurate thanThreats to Validity andFuture Work experienced practitioners 20 / 24
  • 47. Professional status vs. Expertise Results What is the most important factorZ´phyrin Soh et al. e RQ3: Status vs. Expertise (1/1) between expertise and professional status?Problem andMotivationsProblemMotivationsRelated WorkExpertise StudiesUML Class DiagramComprehensionEmpirical StudyStudy DesignResultsRQ1: StatusRQ2: Expertise Experts are more accurate than practitionersRQ3: Status vs. ExpertiseRQ4: Question Precision Experts spent around 7% less time than practitionersConclusion andFuture Work When considering expert subjectsConclusion Experienced students are more accurate thanThreats to Validity andFuture Work experienced practitioners Experienced students spent around 37% less time than experienced practitioners 20 / 24
  • 48. Professional status vs. Expertise Results What is the most important factorZ´phyrin Soh et al. e RQ3: Status vs. Expertise (1/1) between expertise and professional status?Problem andMotivationsProblemMotivationsRelated WorkExpertise StudiesUML Class DiagramComprehensionEmpirical StudyStudy DesignResultsRQ1: StatusRQ2: Expertise Experts are more accurate than practitionersRQ3: Status vs. ExpertiseRQ4: Question Precision Experts spent around 7% less time than practitionersConclusion andFuture Work When considering expert subjectsConclusion Experienced students are more accurate thanThreats to Validity andFuture Work experienced practitioners Experienced students spent around 37% less time than experienced practitioners The effects of expertise on accuracy and time depend on the status 20 / 24
  • 49. Professional status vs. Expertise ResultsZ´phyrin Soh et al. e RQ4: Question Precision (1/1)Problem andMotivationsProblemMotivationsRelated WorkExpertise StudiesUML Class DiagramComprehension Question PrecisionEmpirical Study What is the effect of the question precision on theStudy Design comprehension of a UML class diagram?ResultsRQ1: Status The accuracy of students benefits from precise questionRQ2: ExpertiseRQ3: Status vs. Expertise descriptionRQ4: Question PrecisionConclusion and The accuracy of novices benefits from precise questionFuture WorkConclusion descriptionThreats to Validity andFuture Work 21 / 24
  • 50. Professional status vs. Expertise Conclusion and Future WorkZ´phyrin Soh et al. e Conclusion (1/1)Problem andMotivationsProblem StatusMotivationsRelated WorkExpertise StudiesUML Class DiagramComprehensionEmpirical StudyStudy DesignResultsRQ1: StatusRQ2: ExpertiseRQ3: Status vs. ExpertiseRQ4: Question PrecisionConclusion andFuture WorkConclusionThreats to Validity andFuture Work 22 / 24
  • 51. Professional status vs. Expertise Conclusion and Future WorkZ´phyrin Soh et al. e Conclusion (1/1)Problem andMotivationsProblem Status ExpertiseMotivationsRelated WorkExpertise StudiesUML Class DiagramComprehensionEmpirical StudyStudy DesignResultsRQ1: StatusRQ2: ExpertiseRQ3: Status vs. ExpertiseRQ4: Question PrecisionConclusion andFuture WorkConclusionThreats to Validity andFuture Work 22 / 24
  • 52. Professional status vs. Expertise Conclusion and Future WorkZ´phyrin Soh et al. e Conclusion (1/1)Problem andMotivationsProblem Status ExpertiseMotivationsRelated WorkExpertise StudiesUML Class DiagramComprehensionEmpirical StudyStudy DesignResultsRQ1: StatusRQ2: ExpertiseRQ3: Status vs. ExpertiseRQ4: Question PrecisionConclusion andFuture WorkConclusionThreats to Validity andFuture Work Experts vs. Practitioners 22 / 24
  • 53. Professional status vs. Expertise Conclusion and Future WorkZ´phyrin Soh et al. e Threats to Validity and Future Work (1/1)Problem andMotivationsProblem Threats to Validity and Future WorkMotivations Construct validity: We did not use all combination ofRelated WorkExpertise Studies treatments for each systemUML Class DiagramComprehensionEmpirical StudyStudy DesignResultsRQ1: StatusRQ2: ExpertiseRQ3: Status vs. ExpertiseRQ4: Question PrecisionConclusion andFuture WorkConclusionThreats to Validity andFuture Work 23 / 24
  • 54. Professional status vs. Expertise Conclusion and Future WorkZ´phyrin Soh et al. e Threats to Validity and Future Work (1/1)Problem andMotivationsProblem Threats to Validity and Future WorkMotivations Construct validity: We did not use all combination ofRelated WorkExpertise Studies treatments for each systemUML Class DiagramComprehension Conclusion validity: Practitioners from the sameEmpirical StudyStudy Design company + difficulty to find inexperienced practitionersResults (only one)RQ1: StatusRQ2: Expertise ⇒ Practitioners from other companyRQ3: Status vs. ExpertiseRQ4: Question PrecisionConclusion andFuture WorkConclusionThreats to Validity andFuture Work 23 / 24
  • 55. Professional status vs. Expertise Conclusion and Future WorkZ´phyrin Soh et al. e Threats to Validity and Future Work (1/1)Problem andMotivationsProblem Threats to Validity and Future WorkMotivations Construct validity: We did not use all combination ofRelated WorkExpertise Studies treatments for each systemUML Class DiagramComprehension Conclusion validity: Practitioners from the sameEmpirical StudyStudy Design company + difficulty to find inexperienced practitionersResults (only one)RQ1: StatusRQ2: Expertise ⇒ Practitioners from other companyRQ3: Status vs. ExpertiseRQ4: Question Precision Internal validity: We did not limit the time (fatigueConclusion and biais)Future WorkConclusion ⇒ Limit the experiment time to investigate how muchThreats to Validity andFuture Work time affect the subject’s accuracy 23 / 24
  • 56. Professional status vs. Expertise Conclusion and Future WorkZ´phyrin Soh et al. e Threats to Validity and Future Work (1/1)Problem andMotivationsProblem Threats to Validity and Future WorkMotivations Construct validity: We did not use all combination ofRelated WorkExpertise Studies treatments for each systemUML Class DiagramComprehension Conclusion validity: Practitioners from the sameEmpirical StudyStudy Design company + difficulty to find inexperienced practitionersResults (only one)RQ1: StatusRQ2: Expertise ⇒ Practitioners from other companyRQ3: Status vs. ExpertiseRQ4: Question Precision Internal validity: We did not limit the time (fatigueConclusion and biais)Future WorkConclusion ⇒ Limit the experiment time to investigate how muchThreats to Validity andFuture Work time affect the subject’s accuracy External validity: Only three systems and small range of years of experience ⇒ Use other systems 23 / 24
  • 57. Professional status vs. ExpertiseZ´phyrin Soh et al. e Thanks for your attention!Problem andMotivationsProblem Status ExpertiseMotivationsRelated WorkExpertise StudiesUML Class DiagramComprehensionEmpirical StudyStudy DesignResultsRQ1: StatusRQ2: Expertise The accuracy of students and novicesRQ3: Status vs. ExpertiseRQ4: Question Precision benefits from precise question descriptionsConclusion andFuture WorkConclusionThreats to Validity andFuture Work Experts vs. Practitioners 24 / 24

×