Professional Status and Expertise for UML Class Diagram Comprehension: An Empirical Study

1. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Professional Status and Expertise for UML
Class Diagram Comprehension:
An Empirical Study
Z´ephyrin Soh, Zohreh Sharafi, Bertrand Van den Plas,
Gerardo Cepeda Porras, Yann-Ga¨el Gu´eh´eneuc and
Giuliano Antoniol
Department of Computer and Software Engineering
´Ecole Polytechnique de Montr´eal, Qu´ebec, Canada
June 13, 2012
Pattern Trace Identification, Detection, and Enhancement in Java
SOftware Cost-effective Change and Evolution Research Lab

2. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Outline
Problem and Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and Future Work
Conclusion
Threats to Validity and Future Work
2 / 24

3. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Problem and Motivations
Problem (1/1)
What is experience?
To manage subject/programmer experience:
Years and education as main criteria [1]
Authors sometime combine many criteria
[1] J. Feigenspan et al., Measuring Programming Experience, ICPC 2012, pp.
73-82.3 / 24

4. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Problem and Motivations
Motivations (1/1)
Motivations
Consider two following cases:
A student who used UML for 4 years during her study
A professional with 3 years of experience with UML
Who is the best at understanding of UML class
diagrams?
4 / 24

5. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Problem and Motivations
Motivations (1/1)
Motivations
Consider two following cases:
A student who used UML for 4 years during her study
A professional with 3 years of experience with UML
Who is the best at understanding of UML class
diagrams?
Project managers when recruiting a new software
designer by prioritized the important“factor”
Future designers to know“where”to acquire the
competitive skills by considering the important“factor”
4 / 24

6. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Related Work
Expertise Studies (1/3)
Previous work on expertise
Novices spend less time than experts analysts [2]
[2] K. D. Schenk, N. P. Vitalari, and K. S. Davis, Differences between novice
and expert systems analysts: what do we know and what do we do?, Journal
of Management Information System, vol. 15, pp. 9-50, June 1998
5 / 24

7. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Related Work
Expertise Studies (1/3)
Previous work on expertise
Novices spend less time than experts analysts [2]
Graduate students are faster than junior ones and
intermediate professionals [3]
[3] E. Arisholm and D. I. K. Sjøberg, Evaluating the effect of a delegated
versus centralized control style on the maintainability of object-oriented
software,IEEE Transactions on Software Engineering, vol. 30, no. 8, pp.
521-534, aug. 20045 / 24

8. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Related Work
Expertise Studies (1/3)
Previous work on expertise
Novices spend less time than experts analysts [2]
Graduate students are faster than junior ones and
intermediate professionals [3]
Experts are better for abstract questions and novices are
better for concrete questions [4]
[4] B. Adelson, When novices surpass experts: The difficulty of a task may
increase with expertise,Journal of Experimental Psychology: Learning,
Memory, and Cognition, vol. 10, pp. 483-495, Jul. 1984
5 / 24

9. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Related Work
Expertise Studies (1/3)
Previous work on expertise
Novices spend less time than experts analysts [2]
Graduate students are faster than junior ones and
intermediate professionals [3]
Experts are better for abstract questions and novices are
better for concrete questions [4]
Experts and novices have different program model for
documentation task, no difference for reuse task [5]
[5] J.-M. Burkhardt, F. D´etienne, and S. Wiedenbeck, Object-oriented
program comprehension: Effect of expertise, task and phase, Empirical
Software Engineering, vol. 7, no. 2, pp. 115-156, 2002
5 / 24

10. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Related Work
Expertise Studies (2/3)
Comparison
To compare our work with previous work, we consider:
⇒ Object
⇒ Kind of task/question
⇒ Subject categorisation criterion
6 / 24

11. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Related Work
Expertise Studies (2/3)
Comparison
Ref.
[2] textual description
requirements
analysis
years of experience
rating scale of
supervisors
[3] Java program change task
students and
professionals
[4] program +
flowcharts
abstract +
concrete question
undergraduate
students
fellow teachers
[5] database program documentation
and reuse
students and ex-
perts (nomination
by colleagues, ...)
Legend: Same to our work | Different to our work
[2] Schenk et al. (1998)
[3] Arisholm and Sjøberg (2004)
[4] Adelson (1984)
[5] Burkhardt et al. (2002)6 / 24

12. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Related Work
Expertise Studies (3/3)
Limitations
Previous work:
Did not precisely distinguish years of experience and
professionalism:
Inexperienced students as novices [5]
Senior professionals with less years of programming
experience than graduate students [3]
[5] Burkhardt et al. (2002)
[3] Arisholm and Sjøberg (2004)7 / 24

13. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Related Work
Expertise Studies (3/3)
Limitations
Previous work:
Did not precisely distinguish years of experience and
professionalism:
Inexperienced students as novices [5]
Senior professionals with less years of programming
experience than graduate students [3]
We distinguish the years of experience from
professionalism
[5] Burkhardt et al. (2002)
[3] Arisholm and Sjøberg (2004)7 / 24

14. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Related Work
Expertise Studies (3/3)
Limitations
Previous work:
Did not precisely distinguish years of experience and
professionalism:
Inexperienced students as novices [5]
Senior professionals with less years of programming
experience than graduate students [3]
We distinguish the years of experience from
professionalism
Studied the source code or textual descriptions of
requirements
[5] Burkhardt et al. (2002)
[3] Arisholm and Sjøberg (2004)7 / 24

15. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Related Work
Expertise Studies (3/3)
Limitations
Previous work:
Did not precisely distinguish years of experience and
professionalism:
Inexperienced students as novices [5]
Senior professionals with less years of programming
experience than graduate students [3]
We distinguish the years of experience from
professionalism
Studied the source code or textual descriptions of
requirements
We use the UML class diagram
[5] Burkhardt et al. (2002)
[3] Arisholm and Sjøberg (2004)7 / 24

16. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Related Work
UML Class Diagram Comprehension (1/2)
UML and eye-tracking
Stereotype, color, and layout facilitate class diagram
exploration and comprehension [6]
Multi-cluster (by requirement) and three-cluster (by
stereotype) layout positively affect the comprehension of
class diagrams [7]
[6] S. Yusuf, H. Kagdi, and J. I. Maletic, Assessing the comprehension of UML
diagrams via eye tracking, ICPC’07
[7] B. Sharif and J. I. Maletic, An empirical study on the comprehension of
stereotyped UML class diagram layouts, ICPC’09
8 / 24

17. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Related Work
UML Class Diagram Comprehension (1/2)
UML and eye-tracking
Stereotype, color, and layout facilitate class diagram
exploration and comprehension [6]
Multi-cluster (by requirement) and three-cluster (by
stereotype) layout positively affect the comprehension of
class diagrams [7]
Canonical representation of the Visitor pattern in class
diagram reduce the effort of maintenance task [8]
The representations of design patterns affect the
identification of their participants and their roles [9]
[8] S. Jeanmart, Y.-G. Gu´eh´eneuc, H. Sahraoui, and N. Habra, Impact of the
visitor pattern on program comprehension and maintenance, ESEM’09, Oct
2009, pp. 69-78
[9] G. Cepeda Porras and Y.-G. Gu´eh´eneuc, An empirical study on the
efficiency of different design pattern representations in UML class diagrams,
Empirical Software Engineering, vol. 15, no. 5, pp. 493-522, 20108 / 24

18. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Related Work
UML Class Diagram Comprehension (2/2)
Subjects’ categories
Previous work used subject’s proficiency as categorisation
criterion:
Subjects’ performance in task realization
Subjects’ grade in the course they were enrolled
9 / 24

19. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Related Work
UML Class Diagram Comprehension (2/2)
Subjects’ categories
Previous work used subject’s proficiency as categorisation
criterion:
Subjects’ performance in task realization
Subjects’ grade in the course they were enrolled
Motivations
No previous work that uses the maintenance task on
UML class diagrams and eye-tracking system to study
separately the professional status and the expertise
Combine expertise studies and UML eye-tracking studies
9 / 24

20. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Empirical Study
Study Design (1/8)
Research Questions
RQ1: What is the relation between a designer’s
professional status and her class diagram
comprehension?
10 / 24

21. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Empirical Study
Study Design (1/8)
Research Questions
RQ1: What is the relation between a designer’s
professional status and her class diagram
comprehension?
RQ2: What is the relation between a designer’s
expertise and her class diagram comprehension?
10 / 24

22. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Empirical Study
Study Design (1/8)
Research Questions
RQ1: What is the relation between a designer’s
professional status and her class diagram
comprehension?
RQ2: What is the relation between a designer’s
expertise and her class diagram comprehension?
RQ3: What is the most important factor between
expertise and professional status?
10 / 24

23. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Empirical Study
Study Design (1/8)
Research Questions
RQ1: What is the relation between a designer’s
professional status and her class diagram
comprehension?
RQ2: What is the relation between a designer’s
expertise and her class diagram comprehension?
RQ3: What is the most important factor between
expertise and professional status?
RQ4: What is the effect of the question precision on
the comprehension of a UML class diagram?
10 / 24

24. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Empirical Study
Study Design (2/8)
Objects and Tasks
= ArgoUML, JUnit, and QuickUML
Number of
classes/
Interfaces
Average
number of
attributes per
Class/Interface
Average
number of
methods per
Class/Interface
ArgoUML 10 0.4 8.6
JUnit 14 0.57 6.14
QuickUML 16 1.75 3.87
= : one maintenance task per object
11 / 24

25. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Empirical Study
Study Design (3/8)
Independent variables
= Professional status + Expertise
12 / 24

26. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Empirical Study
Study Design (3/8)
Independent variables
= Professional status + Expertise
Professional status
= practitioners (9)
(in industry)
= students (12)
12 / 24

27. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Empirical Study
Study Design (3/8)
Independent variables
= Professional status + Expertise
Professional status
= practitioners (9)
(in industry)
= students (12)
Expertise: We used the number of years of experience
to categorise experts and novices.
Pair-wise Wilcoxon comparison (+ Bonferroni
correction)
Categorization with the highlest Cliff’s δ value
= experts (12): {3, 4, 5} years of experience
= novices (9): {1, 2} years of experience
12 / 24

28. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Empirical Study
Study Design (4/8)
Mitigating variable
Question precision: The level of details in the formulation
of the question:
Precise: state the kind of operation to perform
(add/remove) and the kind of target element
(class/method/attribute)
Not precise: no operation or target element
13 / 24

29. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Empirical Study
Study Design (5/8)
Dependent variables
Accuracy, Time spent
Search effort = convex hull & spatial density [10]
Overall effort = AFD [9] and NRRF [8]
Question comprehension effort = NDQA and NFQA
AFD: Average Fixation Duration
NRRF: Normalized Rate of Relevant Fixations
NDQA: Normalized Duration in Question Area
NFQA: Normalized Fixations in Question Area
[8] Jeanmart et al. (2009)
[9] Cepeda Porras and Gu´eh´eneuc (2010)
[10] J. H. Goldberg and X. P. Kotval, Computer interface evaluation using eye
movements: methods and constructs, Journal of Industrial Ergonomics, vol.
24, no. 6, pp. 631-645, 199914 / 24

30. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Empirical Study
Study Design (6/8)
15 / 24
Convex hull area
Smaller convex set of fixations
containing all subject’s fixations
Smaller convex hull ⇒ close fixations
⇒ less search effort

31. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Empirical Study
Study Design (7/8)
Spatial density
Number of visited cells / total number of cells
less visits ⇒ less search effort
In TAUPE [11], cell’s size = 64x64px
[11] B. D. Smet, L. Lempereur, Z. Sharafi, Y.-G. Gu´eh´eneuc, G. Antoniol, and
N. Habra, Taupe: Visualising and analysing eye-tracking data, Science of
Computer Programming, 201116 / 24

32. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Empirical Study
Study Design (8/8)
17 / 24
Overall effort: Fixations’ duration and relevance

33. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Empirical Study
Study Design (8/8)
17 / 24
Question Comprehension Effort: Fixations’ count and duration

34. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Results
RQ1: Status (1/1)
Practitioners are more accurate than students
18 / 24
What is the relation between a
designer’s professional status and her
class diagram comprehension?

35. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Results
RQ1: Status (1/1)
Practitioners are more accurate than students
Students spent around 35% less time than
practitioners
18 / 24
What is the relation between a
designer’s professional status and her
class diagram comprehension?

36. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Results
RQ1: Status (1/1)
Practitioners are more accurate than students
Students spent around 35% less time than
practitioners
No significant difference for other dependent variables
18 / 24
What is the relation between a
designer’s professional status and her
class diagram comprehension?

37. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Results
RQ1: Status (1/1)
q
q qq
q qq
q
q
100 300 500 700
020406080100
(a) Accuracy − Time tradeoff for Practitioners
Time spent (s)
Accuracy(%)
q q
q
q
q
q
q
q
qq
qq
150 250 350
020406080100
(b) Accuracy − Time tradeoff for Students
Time spent (s)
Accuracy(%)
Practitioners are more accurate than students
Students spent around 35% less time than
practitioners
No significant difference for other dependent variables
Students could be more accurate if spending more time
18 / 24
What is the relation between a
designer’s professional status and her
class diagram comprehension?

38. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Results
RQ2: Expertise (1/1)
Experts are more accurate than novices
19 / 24
What is the relation between a
designer’s expertise and her class
diagram comprehension?

39. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Results
RQ2: Expertise (1/1)
Experts are more accurate than novices
Novices spent around 33% less time than experts
19 / 24
What is the relation between a
designer’s expertise and her class
diagram comprehension?

40. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Results
RQ2: Expertise (1/1)
Experts are more accurate than novices
Novices spent around 33% less time than experts
Experts have a more efficient ability to search
relevant elements than novices
19 / 24
What is the relation between a
designer’s expertise and her class
diagram comprehension?

41. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Results
RQ2: Expertise (1/1)
Experts are more accurate than novices
Novices spent around 33% less time than experts
Experts have a more efficient ability to search
relevant elements than novices
No significant difference for other dependent variables
19 / 24
What is the relation between a
designer’s expertise and her class
diagram comprehension?

42. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Results
RQ2: Expertise (1/1)
q
q qq
q qqqq qq q
100 300 500 700
020406080100
(a) Accuracy − Time tradeoff for Experts
Time spent (s)
Accuracy(%)
q
q q
q
q q
q
qq
150 250 350
020406080100
(a) Accuracy − Time tradeoff for Novices
Time spent (s)
Accuracy(%)
Experts are more accurate than novices
Novices spent around 33% less time than experts
Experts have a more efficient ability to search
relevant elements than novices
No significant difference for other dependent variables
Novices could be more accurate if spending more time19 / 24
What is the relation between a
designer’s expertise and her class
diagram comprehension?

43. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Results
RQ3: Status vs. Expertise (1/1)
Experts are more accurate than practitioners
20 / 24
What is the most important factor
between expertise and professional
status?

44. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Results
RQ3: Status vs. Expertise (1/1)
Experts are more accurate than practitioners
Experts spent around 7% less time than practitioners
20 / 24
What is the most important factor
between expertise and professional
status?

45. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Results
RQ3: Status vs. Expertise (1/1)
Experts are more accurate than practitioners
Experts spent around 7% less time than practitioners
When considering expert subjects
20 / 24
What is the most important factor
between expertise and professional
status?

46. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Results
RQ3: Status vs. Expertise (1/1)
Experts are more accurate than practitioners
Experts spent around 7% less time than practitioners
When considering expert subjects
Experienced students are more accurate than
experienced practitioners
20 / 24
What is the most important factor
between expertise and professional
status?

47. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Results
RQ3: Status vs. Expertise (1/1)
Experts are more accurate than practitioners
Experts spent around 7% less time than practitioners
When considering expert subjects
Experienced students are more accurate than
experienced practitioners
Experienced students spent around 37% less time than
experienced practitioners
20 / 24
What is the most important factor
between expertise and professional
status?

48. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Results
RQ3: Status vs. Expertise (1/1)
Experts are more accurate than practitioners
Experts spent around 7% less time than practitioners
When considering expert subjects
Experienced students are more accurate than
experienced practitioners
Experienced students spent around 37% less time than
experienced practitioners
The effects of expertise on accuracy and time depend
on the status
20 / 24
What is the most important factor
between expertise and professional
status?

49. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Results
RQ4: Question Precision (1/1)
Question Precision
What is the effect of the question precision on the
comprehension of a UML class diagram?
The accuracy of students benefits from precise question
description
The accuracy of novices benefits from precise question
description
21 / 24

50. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Conclusion and Future Work
Conclusion (1/1)
22 / 24
Status

51. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Conclusion and Future Work
Conclusion (1/1)
22 / 24
Status Expertise

52. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Conclusion and Future Work
Conclusion (1/1)
22 / 24
Status Expertise
Experts vs. Practitioners

53. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Conclusion and Future Work
Threats to Validity and Future Work (1/1)
Threats to Validity and Future Work
Construct validity: We did not use all combination of
treatments for each system
23 / 24

54. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Conclusion and Future Work
Threats to Validity and Future Work (1/1)
Threats to Validity and Future Work
Construct validity: We did not use all combination of
treatments for each system
Conclusion validity: Practitioners from the same
company + difficulty to find inexperienced practitioners
(only one)
⇒ Practitioners from other company
23 / 24

55. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Conclusion and Future Work
Threats to Validity and Future Work (1/1)
Threats to Validity and Future Work
Construct validity: We did not use all combination of
treatments for each system
Conclusion validity: Practitioners from the same
company + difficulty to find inexperienced practitioners
(only one)
⇒ Practitioners from other company
Internal validity: We did not limit the time (fatigue
biais)
⇒ Limit the experiment time to investigate how much
time affect the subject’s accuracy
23 / 24

56. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
Conclusion and Future Work
Threats to Validity and Future Work (1/1)
Threats to Validity and Future Work
Construct validity: We did not use all combination of
treatments for each system
Conclusion validity: Practitioners from the same
company + difficulty to find inexperienced practitioners
(only one)
⇒ Practitioners from other company
Internal validity: We did not limit the time (fatigue
biais)
⇒ Limit the experiment time to investigate how much
time affect the subject’s accuracy
External validity: Only three systems and small range
of years of experience
⇒ Use other systems
23 / 24

57. Professional status
vs. Expertise
Z´ephyrin Soh et al.
Problem and
Motivations
Problem
Motivations
Related Work
Expertise Studies
UML Class Diagram
Comprehension
Empirical Study
Study Design
Results
RQ1: Status
RQ2: Expertise
RQ3: Status vs. Expertise
RQ4: Question Precision
Conclusion and
Future Work
Conclusion
Threats to Validity and
Future Work
24 / 24
Thanks for your attention!
Status Expertise
Experts vs. Practitioners
The accuracy of students and novices
benefits from precise question descriptions