1. Program
Exploration
Z´phyrin Soh et al.
e
Introduction
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
On the Effect of Program Exploration on
Maintenance Tasks
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
Z´phyrin Soh, Foutse Khomh, Yann-Ga¨l Gu´h´neuc,
e
e
e e
Giuliano Antoniol, Bram Adams
Department of Computer and Software Engineering
´
Ecole Polytechnique de Montr´al, Qu´bec, Canada
e
e
October 14, 2013
Pattern Trace Identification, Detection, and Enhancement in Java
SOftware Cost-effective Change and Evolution Research Lab

2. Program
Exploration
Outline
Z´phyrin Soh et al.
e
Introduction
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
Introduction
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
User Study
Empirical Study
Conclusion and Future Work
2 / 23

3. Program
Exploration
Z´phyrin Soh et al.
e
Introduction
Context and Example (1/3)
Introduction
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
3 / 23
Program Exploration
Developers interact with program entities, e.g., open
files, edit methods, etc.
Developers explore the program, i.e., move from
program entities to program entities.
Types of program entities.

4. Program
Exploration
Z´phyrin Soh et al.
e
Introduction
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
4 / 23
Introduction
Context and Example (2/3)

5. Program
Exploration
Z´phyrin Soh et al.
e
Introduction
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
4 / 23
Introduction
Context and Example (2/3)

6. Program
Exploration
Z´phyrin Soh et al.
e
Introduction
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
4 / 23
Introduction
Context and Example (2/3)

7. Program
Exploration
Z´phyrin Soh et al.
e
Introduction
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
4 / 23
Containment
Introduction principle:
the exploration graph
Context and Example (2/3)
Aggregate
at class level

8. Program
Exploration
Z´phyrin Soh et al.
e
Introduction
Context and Example (2/3)
Introduction
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
4 / 23
Exploration graph aggregated at class level

9. Program
Exploration
Z´phyrin Soh et al.
e
Introduction
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
5 / 23
Introduction
Context and Example (3/3)

10. Program
Exploration
Z´phyrin Soh et al.
e
Introduction
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
5 / 23
Introduction
Context and Example (3/3)

11. Program
Exploration
Introduction
Z´phyrin Soh et al.
e
Context and Example (3/3)
A graph
represents a program exploration
Introduction
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
(a) Graph 1
(b) Graph 2
Is there any difference between exploration graphs?
How to find the difference?
How does the difference affect the maintenance tasks?
5 / 23

12. Program
Exploration
Z´phyrin Soh et al.
e
Introduction
Why do we care? (1/3)
Introduction
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Research on Program Exploration
Developers spend on average 35% of their time
navigating within and between source files [1]
Developers perform on average 19.31 navigation actions
between two modifications [2]
Conclusion
Threats to Validity
and Future Work
[1] Ko et al., An exploratory study of how developers seek, relate, and collect
relevant information during software maintenance tasks. TSE 2006
[2] D. R¨thlisberger, SmartGroups: Focusing on Task-Relevant Source
o
Artifacts in IDEs, ICPC 2011
6 / 23

13. Program
Exploration
Z´phyrin Soh et al.
e
Introduction
Why do we care? (2/3)
Introduction
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
Reasons of Program Exploration
Developers may have several different ways to explore a
program e.g.,
Looking for relevant entities ⇒ Some developers (may)
not know where to look at.
Validate previous changes ⇒ Developers (may) know
the target entity to look at.
⇒ In some cases, developers may perform more back and
forth navigations (i.e., revisitation) on program entities
7 / 23

14. Program
Exploration
Z´phyrin Soh et al.
e
Introduction
Why do we care? (3/3)
Introduction
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
Revisitation-based Exploration
Some recommendation systems (e.g., Mylyn [3],
NavClus [4]) are based on the revisitation of program
entities
We wonder if revisitation is “good“ or “bad“
⇒ Knowing that is helpful to improve revisitation-based
approach for recommendation systems.
[3] Kersten and Murphy, Mylar: a degree-of-interest model for IDEs, AOSD
2005
[4] Lee and Kang, Clustering and recommending collections of code relevant
to tasks, ICSM 2011
8 / 23

15. Program
Exploration
Z´phyrin Soh et al.
e
User Study
Research Question
Introduction
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
9 / 23
Exploration Strategies
In the revisitation-based exploration, we focus on two
extreme cases:
Referenced Exploration (RE): A developer revisits one
(or a set of) entity(ies) already visited
⇒ referenced entities
Unreferenced Exploration (UE): There is no set of
referenced entities
⇒ Almost the same revisitation of program entity(ies)

16. Program
Exploration
Z´phyrin Soh et al.
e
User Study
Research Question
Introduction
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
Exploration Strategies
In the revisitation-based exploration, we focus on two
extreme cases:
Referenced Exploration (RE): A developer revisits one
(or a set of) entity(ies) already visited
⇒ referenced entities
Unreferenced Exploration (UE): There is no set of
referenced entities
⇒ Almost the same revisitation of program entity(ies)
Do developers follow a referenced exploration when
performing maintenance tasks?
9 / 23

17. Program
Exploration
Z´phyrin Soh et al.
e
Introduction
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
10 / 23
User Study
Approach (1/5)
Data Collection
Mylyn’s IHs appear as attachment (XML file with name
“mylyn-context.zip”) to a bug

18. Program
Exploration
Z´phyrin Soh et al.
e
Introduction
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
User Study
Approach (1/5)
Data Collection
Mylyn’s IHs appear as attachment (XML file with name
“mylyn-context.zip”) to a bug
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
Top four Eclipse projects with more IHs (from 2,601 bugs)
ECF Mylyn PDE Platform Total
# IHs
26
1,273
131
275 1,705
10 / 23

19. Program
Exploration
Z´phyrin Soh et al.
e
Introduction
Context and Example
Why do we care?
User Study
Approach (2/5)
How do we build an oracle?
User Study
Research Question
Approach
Results
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
11 / 23
Random sample of IHs
(proportional among projects)

20. Program
Exploration
Z´phyrin Soh et al.
e
Introduction
Context and Example
Why do we care?
User Study
Approach (2/5)
How do we build an oracle?
User Study
Research Question
Approach
Results
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
11 / 23
Random sample of IHs
(proportional among projects)

21. Program
Exploration
Z´phyrin Soh et al.
e
User Study
Approach (2/5)
Introduction
How do we build an oracle?
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
Random sample of IHs
(proportional among projects)
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
9 subjects
Java experience
- mean : 4.16 yrs
-sd : 2.80 yrs
11 / 23

22. Program
Exploration
Z´phyrin Soh et al.
e
User Study
Approach (2/5)
Introduction
How do we build an oracle?
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
Random sample of IHs
(proportional among projects)
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
9 subjects
Java experience
- mean : 4.16 yrs
-sd : 2.80 yrs
11 / 23

23. Program
Exploration
Z´phyrin Soh et al.
e
User Study
Approach (2/5)
Introduction
How do we build an oracle?
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
Random sample of IHs
(proportional among projects)
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
9 subjects
Java experience
- mean : 4.16 yrs
-sd : 2.80 yrs
RE
11 / 23
UE
D

24. Program
Exploration
Z´phyrin Soh et al.
e
User Study
Approach (2/5)
Introduction
How do we build an oracle?
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
Random sample of IHs
(proportional among projects)
Empirical Study
Research Questions
RQ1
RQ2
Discussions
How much the subjects agreed?
Fliess' Kappa interater agreement coeficient
Conclusion and
Future Work
- Overall: fair agreement (0.36)
- RE: almost moderate agreement (0.38)
- UE: almost moderate agreement (0.39)
- Doubt: No agreement (-0.009)
Conclusion
Threats to Validity
and Future Work
9 subjects
Java experience
- mean : 4.16 yrs
-sd : 2.80 yrs
RE
11 / 23
UE
D

25. Program
Exploration
Z´phyrin Soh et al.
e
User Study
Approach (3/5)
Introduction
Context and Example
Why do we care?
How do we aggregate the results?
User Study
Research Question
Approach
Results
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
RE
12 / 23
RE
UE
UE
D

26. Program
Exploration
Z´phyrin Soh et al.
e
User Study
Approach (3/5)
Introduction
Context and Example
Why do we care?
How do we aggregate the results?
User Study
Research Question
Approach
Results
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
RE
RE
UE
>= 2/3 subjects
RE
12 / 23
UE
D

27. Program
Exploration
Z´phyrin Soh et al.
e
User Study
Approach (3/5)
Introduction
Context and Example
Why do we care?
How do we aggregate the results?
User Study
Research Question
Approach
Results
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
RE
RE
UE
UE
>= 2/3 subjects
UE
12 / 23
D

28. Program
Exploration
Z´phyrin Soh et al.
e
User Study
Approach (3/5)
Introduction
Context and Example
Why do we care?
How do we aggregate the results?
User Study
Research Question
Approach
Results
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
RE
RE
UE
< 2/3 subjects
< 2/3 subjects
D
12 / 23
UE
D

29. Program
Exploration
Z´phyrin Soh et al.
e
Introduction
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
13 / 23
User Study
Approach (4/5)
Identification Technique
Subjects count the number of nodes and the number of in/out arrows in the graphs
⇒ look at the distribution/inequality of revisits across graphs

30. Program
Exploration
Z´phyrin Soh et al.
e
Introduction
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
User Study
Approach (4/5)
Identification Technique
Subjects count the number of nodes and the number of in/out arrows in the graphs
⇒ look at the distribution/inequality of revisits across graphs
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
Gini Inequality Index [5]
We measure how much classes are (in)equally revisited
Used in econometrics to measure the inequality of
income among a population
Population = classes
Income of a class = NumRevisits
Gini(IH) =
1
2n2 µ
n
n
i=1 j=1
| NumRevisit(eni ) − NumRevisit(enj ) |
[5] K. Xu, How has the literature on Gini’s index evolved in the past 80 years?,
Technical report, Department of Economics, Dalhouse University, 2004
13 / 23

31. Program
Exploration
Z´phyrin Soh et al.
e
User Study
Approach (5/5)
Introduction
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
14 / 23
Inequality Index Threshold
Define a threshold to know if classes are (in)equally revisited
If Gini(IH) < Threshold
⇒ Classes are almost equally revisited
⇒ UE: Unreferenced Exploration
If Gini(IH) ≥ Threshold
⇒ Some classes are more revisited than others
⇒ RE: Referenced Exploration

32. Program
Exploration
Z´phyrin Soh et al.
e
User Study
Approach (5/5)
Introduction
Inequality Index Threshold
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
0
Perfect
equality
14 / 23
1
Maximum
inequality

33. Program
Exploration
Z´phyrin Soh et al.
e
User Study
Approach (5/5)
Introduction
Inequality Index Threshold
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Threshold ?
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
0
Perfect
equality
14 / 23
1
Maximum
inequality

34. Program
Exploration
Z´phyrin Soh et al.
e
Introduction
Context and Example
Why do we care?
User Study
Approach (5/5)
Inequality Index Threshold
User Study
Research Question
Approach
Results
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
14 / 23
RE

35. Program
Exploration
Z´phyrin Soh et al.
e
User Study
Approach (5/5)
Introduction
Inequality Index Threshold
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
RE
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
0.28
Conclusion
Threats to Validity
and Future Work
0
Perfect
equality
14 / 23
1
Maximum
inequality

36. Program
Exploration
Z´phyrin Soh et al.
e
User Study
Approach (5/5)
Introduction
Inequality Index Threshold
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
RE
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
0.28
Conclusion
Threats to Validity
and Future Work
0
Perfect
equality
14 / 23
0.2
0.3
- Threshold = 0.2 ==> RE
- Threshold = 0.3 ==> UE
1
Maximum
inequality

37. Program
Exploration
Z´phyrin Soh et al.
e
User Study
Approach (5/5)
Introduction
Inequality Index Threshold
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
RE
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
0.4
Conclusion
Threats to Validity
and Future Work
0
Perfect
equality
1
Maximum
inequality
Use F-Measure to maximize both precision and recall
14 / 23

38. Program
Exploration
Z´phyrin Soh et al.
e
User Study
Approach (5/5)
Introduction
Inequality Index Threshold
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
RE
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
0.4
Conclusion
Threats to Validity
and Future Work
0
Perfect
equality
1
UE
RE
Maximum
inequality
Use F-Measure to maximize both precision and recall
14 / 23

39. Program
Exploration
Z´phyrin Soh et al.
e
Introduction
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
15 / 23
User Study
Results (1/2)
Percentage RE vs. UE
RE: 28.03% of Interaction Histories
UE: 71.96% of Interaction Histories
⇒ Developers follow mostly the unreferenced
exploration (UE) when performing a maintenance task.

40. Program
Exploration
Z´phyrin Soh et al.
e
Introduction
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
User Study
Results (1/2)
Percentage RE vs. UE
RE: 28.03% of Interaction Histories
UE: 71.96% of Interaction Histories
⇒ Developers follow mostly the unreferenced
exploration (UE) when performing a maintenance task.
Methodical developers do not reinvestigate methods as
frequently as opportunistic developers [6]
RE = opportunistic developers?
UE = methodical developers?
⇒ We need more investigations
[6] Robillard et al., How effective developers investigate source code: An
exploratory study, TSE 2004
15 / 23

41. Program
Exploration
Z´phyrin Soh et al.
e
Introduction
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
16 / 23
User Study
Results (2/2)
Confounding Factors
Architecture of the system

42. Program
Exploration
Z´phyrin Soh et al.
e
Introduction
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
16 / 23
User Study
Results (2/2)
Confounding Factors
Architecture of the system
Number of common classes (NCC): The more A and B
have CC, the more they used the same part of the
system

43. Program
Exploration
Z´phyrin Soh et al.
e
Introduction
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
16 / 23
User Study
Results (2/2)
Confounding Factors
Architecture of the system
Number of common classes (NCC): The more A and B
have CC, the more they used the same part of the
system
Compute NCC for each pair of IH.
NCC pairs with same ES vs. NCC pairs with different
ES ⇒ No difference.

44. Program
Exploration
Z´phyrin Soh et al.
e
Introduction
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
16 / 23
User Study
Results (2/2)
Confounding Factors
Architecture of the system
Number of common classes (NCC): The more A and B
have CC, the more they used the same part of the
system
Compute NCC for each pair of IH.
NCC pairs with same ES vs. NCC pairs with different
ES ⇒ No difference.
Task interruption: The interruption time seems to push
developers to be concentrate on a set of (referenced)
entities (RE)

45. Program
Exploration
Z´phyrin Soh et al.
e
Introduction
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
User Study
Results (2/2)
Confounding Factors
Architecture of the system
Number of common classes (NCC): The more A and B
have CC, the more they used the same part of the
system
Compute NCC for each pair of IH.
NCC pairs with same ES vs. NCC pairs with different
ES ⇒ No difference.
Task interruption: The interruption time seems to push
developers to be concentrate on a set of (referenced)
entities (RE)
Type of the task: Bug severity because developers may
have more back and forth navigation when fixing severe
bugs wrt. less severe bugs
Inequality Indexes are not significantly different for
different types of bugs.
16 / 23

46. Program
Exploration
Z´phyrin Soh et al.
e
Empirical Study
Research Questions
Introduction
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
Research Questions
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Does any difference exist in maintenance time between RE and UE?
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
17 / 23
Does any difference exist in edit/navigation ratio
between RE and UE?

47. Program
Exploration
Z´phyrin Soh et al.
e
Introduction
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
18 / 23
Empirical Study
RQ1
Approach
Overall Time spend =
Unpaired Wilcoxon test
Duration(entity )
#entity
entity ∈IH

48. Program
Exploration
Z´phyrin Soh et al.
e
Introduction
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
Empirical Study
RQ1
Approach
Overall Time spend =
Duration(entity )
#entity
entity ∈IH
Unpaired Wilcoxon test
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
18 / 23
Results
The UE is on average 12.30% less time consuming
than the RE.

49. Program
Exploration
Z´phyrin Soh et al.
e
Introduction
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
19 / 23
Empirical Study
RQ2
Approach
Edit/navigation ratio =
Unpaired Wilcoxon test
NumEdit(IH)
NumEvent(IH)

50. Program
Exploration
Z´phyrin Soh et al.
e
Introduction
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
19 / 23
Empirical Study
RQ2
Approach
Edit/navigation ratio =
NumEdit(IH)
NumEvent(IH)
Unpaired Wilcoxon test
Results
An UE requires less edit/navigation ratio than a
RE.

51. Program
Exploration
Z´phyrin Soh et al.
e
Introduction
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
20 / 23
Empirical Study
Discussions
Is there a good strategy?
RE (more) ⇒ The back and forth actions on refUE (less) erenced entities are time consuming
⇒ More edit actions compare to navRE (more)
igation actions. Wrong edits/modif.
UE (less)
then come back to revert/cancel?

52. Program
Exploration
Z´phyrin Soh et al.
e
Introduction
Context and Example
Why do we care?
Empirical Study
Discussions
Is there a good strategy?
RE (more) ⇒ The back and forth actions on refUE (less) erenced entities are time consuming
User Study
Research Question
Approach
Results
Empirical Study
⇒ More edit actions compare to navRE (more)
igation actions. Wrong edits/modif.
UE (less)
then come back to revert/cancel?
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
RE
F1
F2
F1
F3
F1
F4
F1
F2
F3
F1
F4
F5
UE
20 / 23
4/8 > 3/6 (RE better)
F2
3/6
F5
4/8

53. Program
Exploration
Z´phyrin Soh et al.
e
Introduction
Context and Example
Why do we care?
Empirical Study
Discussions
Is there a good strategy?
RE (more) ⇒ The back and forth actions on refUE (less) erenced entities are time consuming
User Study
Research Question
Approach
Results
Empirical Study
⇒ More edit actions compare to navRE (more)
igation actions. Wrong edits/modif.
UE (less)
then come back to revert/cancel?
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
RE
F1
F2
F1
F3
F1
F4
F1
F2
F3
F1
F4
F5
UE
20 / 23
4/8 > 3/6 (RE better)
F2
3/6
F5
4/8
2/8
2/6
2/8 < 2/6 (UE better)

54. Program
Exploration
Z´phyrin Soh et al.
e
Conclusion and Future Work
Conclusion
Introduction
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
RE
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
21 / 23
0.4
0
Perfect
equality
1
UE
RE
Maximum
inequality
Use F-Measure to maximize both precision and recall

55. Program
Exploration
Z´phyrin Soh et al.
e
Conclusion and Future Work
Conclusion
Introduction
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
RE
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
21 / 23
0.4
0
Perfect
equality
1
UE
RE
Maximum
inequality
Use F-Measure to maximize both precision and recall

56. Program
Exploration
Z´phyrin Soh et al.
e
Conclusion and Future Work
Conclusion
Introduction
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
RE
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
21 / 23
0.4
0
Perfect
equality
1
UE
RE
Maximum
inequality
Use F-Measure to maximize both precision and recall

57. Program
Exploration
Z´phyrin Soh et al.
e
Conclusion and Future Work
Threats to Validity and Future Work
Introduction
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
22 / 23
Threats to Validity and Future Work
Construct validity: An IH can be a part of developers’
work

58. Program
Exploration
Z´phyrin Soh et al.
e
Conclusion and Future Work
Threats to Validity and Future Work
Introduction
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
22 / 23
Threats to Validity and Future Work
Construct validity: An IH can be a part of developers’
work
Conclusion validity: The time recorded can be
different to the “real” time spent
⇒ Perform an experiment to collect a data

59. Program
Exploration
Z´phyrin Soh et al.
e
Conclusion and Future Work
Threats to Validity and Future Work
Introduction
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
22 / 23
Threats to Validity and Future Work
Construct validity: An IH can be a part of developers’
work
Conclusion validity: The time recorded can be
different to the “real” time spent
⇒ Perform an experiment to collect a data
Internal validity: We use only Mylyn’s IH
⇒ Use the IH from other tools

60. Program
Exploration
Z´phyrin Soh et al.
e
Conclusion and Future Work
Threats to Validity and Future Work
Introduction
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
Threats to Validity and Future Work
Construct validity: An IH can be a part of developers’
work
Conclusion validity: The time recorded can be
different to the “real” time spent
⇒ Perform an experiment to collect a data
Internal validity: We use only Mylyn’s IH
⇒ Use the IH from other tools
External validity: Our subject projets are Eclipse-based
projects
⇒ Use other systems
22 / 23

61. Program
Exploration
Z´phyrin Soh et al.
e
Thanks for your attention!
Introduction
Context and Example
Why do we care?
User Study
Research Question
Approach
Results
RE
Empirical Study
Research Questions
RQ1
RQ2
Discussions
Conclusion and
Future Work
Conclusion
Threats to Validity
and Future Work
23 / 23
0.4
0
Perfect
equality
1
UE
RE
Maximum
inequality
Use F-Measure to maximize both precision and recall