Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
An Empirical Investigation on
Documentation Usage Patterns in
Maintenance Tasks
Gabriele Gerardo Massimiliano Rocco Sebast...
Context and Motivations
• Software Development
Case Study
• User Study with 33 Partecipants
Results
•Documentation usage p...
Software Development
Maintenance tasks are generally
facilitated when software
documentation is available
Past Work
The use of UML diagram
significantly improves the
functional correctness of changes
E. Arisholm et al. TSE 2006
Past Work
Help to reduce the time
needed to understand ho
perform maintenance tasks
by approximately 20 %
Eirik Tryggeseth...
How such documentation is
browsed by developers to perform
maintenance activities?
Empirical Study
Goal: observe the software documents
browsed by Developers during maintenance
tasks
Quality focus: underst...
• RQ1: How much time did participants spend on
different kinds of artifacts?
• RQ2: How do participants navigate
different...
Context
• Object: software artifacts from SMOS, a school
automation system developed by graduate
students at the Universit...
Study
Procedure
Maintenance tasks
Bug Fixing:
Add a new feature:
Improve existing features:
Example: Maintenance task
Monitoring Maintenance Activity
Monitoring Maintenance Activity
TIMESTAMP
Data Collection
XML log
Files
Data Collection
XML log
Files
0
0,1
0,2
0,3
0,4
0,5
0,6
0,7
0,8
0,9
1
Recall
Precision
F-Measure
Recall
Precision
F-Measure
Recall
Precision
F-Measure
U...
0
0,1
0,2
0,3
0,4
0,5
0,6
0,7
0,8
0,9
1
Recall
Precision
F-Measure
Recall
Precision
F-Measure
Recall
Precision
F-Measure
U...
0
0,1
0,2
0,3
0,4
0,5
0,6
0,7
0,8
0,9
1
Recall
Precision
F-Measure
Recall
Precision
F-Measure
Recall
Precision
F-Measure
U...
RQ1
How much time did participants
spend on different kinds of artifacts?
Javadoc
Use Case
Class Diagram
Sequence Diagram
Source Code
ALL
Graduates
Undergraduates
15%
Use of each artifacts in over...
RQ1: How much time did participants
spend on different kinds of artifacts?
72%
13%10%3%2%
72%
13%10%3%2%
RQ1: How much time did participants
spend on different kinds of artifacts?
Undergraduates students used
Sou...
72%
13%10%3%2%
RQ1: How much time did participants
spend on different kinds of artifacts?
Graduate
Students
Graduate stude...
Perceived Importance of
Software Artifacts
1
3
4
4
2
3
4
5
4
2
0 2 4 6
Use Case
Class Diagram
Sequence Diagram
Source Code...
Perceived Importance of
Software Artifacts
1
3
4
4
2
3
4
5
4
2
0 1 2 3 4 5 6
Use Case
Class Diagram
Sequence Diagram
Sourc...
Perceived Importance of
Software Artifacts
1
3
4
4
2
3
4
5
4
2
0 1 2 3 4 5 6
Use Case
Class Diagram
Sequence Diagram
Sourc...
Summary RQ1
Summary RQ1
1) Participants spent more time to analyze low-level artifacts
as compared to high-level artifacts
2) Particip...
Summary RQ1
1) Participants spent more time to analyze low-level artifacts
as compared to high-level artifacts
2) Particip...
Summary RQ1
1) Participants spent more time to analyze low-level artifacts
as compared to high-level artifacts
2) Particip...
RQ2
How do participants navigate different kinds
of artifacts to identify code to be changed
during the evolution task?
3%
Starting Point:
Graduate vs Undergraduate students
12%
17%
25%
42%
Starting Point:
Graduate vs Undergraduate
Undergraduate
Graduate
58%
32%
24%
26%
9%
22%
3%
17%
4% 3%
18%
8%
2%
2%
1%
1%
3%
0%
1%
0%
4%
12%
16%
12%
10%
4%
4%
1%
2%
1%
1%
5%
S
D
(SD)+
(US)+
U(SD)+
(DS)+
J
U
S(US)+
SU(SD)+
Oth...
18%
8%
2%
2%
1%
1%
3%
0%
1%
0%
4%
12%
16%
12%
10%
4%
4%
1%
2%
1%
1%
5%
0% 2% 4% 6% 8% 10% 12% 14% 16% 18% 20%
S
D
(SD)+
(U...
18%
8%
2%
2%
1%
1%
3%
0%
1%
0%
4%
12%
16%
12%
10%
4%
4%
1%
2%
1%
1%
5%
0% 2% 4% 6% 8% 10% 12% 14% 16% 18% 20%
S
D
(SD)+
(U...
Source
Code
Sequence
Diagram
Javadoc
Transition Graph between kinds of
Software Artifacts
Source
Code
Sequence
Diagram
Javadoc
1) From Source Code
participants in most
cases “go back” to
Sequence and Class
Diagra...
0%
20%
11%
30%
16%
61%
41%
70%
12%
15%
60%
24%
58%
54%
0% 10% 20% 30% 40% 50% 60% 70% 80%
USDC
USD
USC
UDC
SDC
SC
DC
Gradu...
Summary RQ2
• 1) Participants tend to start the assigned task from source
code or from design documents , i.e., class and ...
…Conclusions…
…Conclusions…
…Conclusions…
…Conclusions…
…Conclusions…
…Future Directions…
An Empirical Investigation on Documentation Usage Patterns in Maintenance Tasks
Upcoming SlideShare
Loading in …5
×

An Empirical Investigation on Documentation Usage Patterns in Maintenance Tasks

668 views

Published on

When developers perform a software maintenance
task, they need to identify artifacts—e.g., classes or more specifically
methods—that need to be modified. To this aim, they
can browse various kind of artifacts, for example use case
descriptions, UML diagrams, or source code.
This paper reports the results of a study—conducted with 33
participants— aimed at investigating (i) to what extent developers
use different kinds of documentation when identifying artifacts
to be changed, and (ii) whether they follow specific navigation
patterns among different kinds of artifacts.
Results indicate that, although developers spent a conspicuous
proportion of the available time by focusing on source code,
they browse back and forth between source code and either
static (class) or dynamic (sequence) diagrams. Less frequently,
developers—especially more experienced ones—follow an “integrated”
approach by using different kinds of artifacts.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

An Empirical Investigation on Documentation Usage Patterns in Maintenance Tasks

  1. 1. An Empirical Investigation on Documentation Usage Patterns in Maintenance Tasks Gabriele Gerardo Massimiliano Rocco Sebastiano Bavota Canfora Di Penta Oliveto Panichella
  2. 2. Context and Motivations • Software Development Case Study • User Study with 33 Partecipants Results •Documentation usage pattern during maintenance evolution actvity
  3. 3. Software Development Maintenance tasks are generally facilitated when software documentation is available
  4. 4. Past Work The use of UML diagram significantly improves the functional correctness of changes E. Arisholm et al. TSE 2006
  5. 5. Past Work Help to reduce the time needed to understand ho perform maintenance tasks by approximately 20 % Eirik Tryggeseth EMSE 1997
  6. 6. How such documentation is browsed by developers to perform maintenance activities?
  7. 7. Empirical Study Goal: observe the software documents browsed by Developers during maintenance tasks Quality focus: understand the documentation usage pattern during maintenance activities Perspective: researchers interested to develop tool to improve the program comprehension
  8. 8. • RQ1: How much time did participants spend on different kinds of artifacts? • RQ2: How do participants navigate different kinds of artifacts to identify code to be changed during the evolution task? Research Questions
  9. 9. Context • Object: software artifacts from SMOS, a school automation system developed by graduate students at the University of Salerno (Italy). • Subjects: 33 partecipants 11 Bachelor Students 18 Master Students 4 PhD Students
  10. 10. Study Procedure
  11. 11. Maintenance tasks Bug Fixing: Add a new feature: Improve existing features:
  12. 12. Example: Maintenance task
  13. 13. Monitoring Maintenance Activity
  14. 14. Monitoring Maintenance Activity TIMESTAMP
  15. 15. Data Collection XML log Files
  16. 16. Data Collection XML log Files
  17. 17. 0 0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 1 Recall Precision F-Measure Recall Precision F-Measure Recall Precision F-Measure Undergraduates Graduates ALL Subjects Performances
  18. 18. 0 0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 1 Recall Precision F-Measure Recall Precision F-Measure Recall Precision F-Measure Undergraduates Graduates ALL Subjects Performances Average 0.75 FM
  19. 19. 0 0,1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 1 Recall Precision F-Measure Recall Precision F-Measure Recall Precision F-Measure Undergraduates Graduates ALL Subjects Performances Average 0.75 FM + 5% FM for Graduate Students
  20. 20. RQ1 How much time did participants spend on different kinds of artifacts?
  21. 21. Javadoc Use Case Class Diagram Sequence Diagram Source Code ALL Graduates Undergraduates 15% Use of each artifacts in overall tasks 60% 33% 100% 72%
  22. 22. RQ1: How much time did participants spend on different kinds of artifacts? 72% 13%10%3%2%
  23. 23. 72% 13%10%3%2% RQ1: How much time did participants spend on different kinds of artifacts? Undergraduates students used Source Code and Javadoc significantly more than Graduate students Undergraduate Students
  24. 24. 72% 13%10%3%2% RQ1: How much time did participants spend on different kinds of artifacts? Graduate Students Graduate students used Class Diagrams significantly more than Undergraduates
  25. 25. Perceived Importance of Software Artifacts 1 3 4 4 2 3 4 5 4 2 0 2 4 6 Use Case Class Diagram Sequence Diagram Source Code Javadoc Graduates Undergraduates
  26. 26. Perceived Importance of Software Artifacts 1 3 4 4 2 3 4 5 4 2 0 1 2 3 4 5 6 Use Case Class Diagram Sequence Diagram Source Code Javadoc Graduates Undergraduates Sequence diagram represents a fair compromise between use cases (too abstracts) and class diagrams (providing useless details about an entire subsystem)..
  27. 27. Perceived Importance of Software Artifacts 1 3 4 4 2 3 4 5 4 2 0 1 2 3 4 5 6 Use Case Class Diagram Sequence Diagram Source Code Javadoc Graduates Undergraduates Sequence diagram represents a fair compromise between use cases (too abstracts) and class diagrams (providing useless details about an entire subsystem).. With the other sources of documentation available Javadoc became useless to identify impacted components
  28. 28. Summary RQ1
  29. 29. Summary RQ1 1) Participants spent more time to analyze low-level artifacts as compared to high-level artifacts 2) Participants consider the most important sources of information sequence and class diagrams 3) Undergraduate students spent a significantly higher proportion of time on source code than graduate students that, instead, spent more time on class diagrams
  30. 30. Summary RQ1 1) Participants spent more time to analyze low-level artifacts as compared to high-level artifacts 2) Participants consider the most important sources of information sequence and class diagrams 3) Undergraduate students spent a significantly higher proportion of time on source code than graduate students that, instead, spent more time on class diagrams
  31. 31. Summary RQ1 1) Participants spent more time to analyze low-level artifacts as compared to high-level artifacts 2) Participants consider the most important sources of information sequence and class diagrams 3) Undergraduate students spent a significantly higher proportion of time on source code than graduate students that, instead, spent more time on class diagrams
  32. 32. RQ2 How do participants navigate different kinds of artifacts to identify code to be changed during the evolution task?
  33. 33. 3% Starting Point: Graduate vs Undergraduate students 12% 17% 25% 42%
  34. 34. Starting Point: Graduate vs Undergraduate Undergraduate Graduate 58% 32% 24% 26% 9% 22% 3% 17% 4% 3%
  35. 35. 18% 8% 2% 2% 1% 1% 3% 0% 1% 0% 4% 12% 16% 12% 10% 4% 4% 1% 2% 1% 1% 5% S D (SD)+ (US)+ U(SD)+ (DS)+ J U S(US)+ SU(SD)+ Other Graduate Undegraduate S= Sequence Diagram D= Class Diagram U= Use Case J= Javadoc Most frequents patterns before reaching source code
  36. 36. 18% 8% 2% 2% 1% 1% 3% 0% 1% 0% 4% 12% 16% 12% 10% 4% 4% 1% 2% 1% 1% 5% 0% 2% 4% 6% 8% 10% 12% 14% 16% 18% 20% S D (SD)+ (US)+ U(SD)+ (DS)+ J U S(US)+ SU(SD)+ Other Graduate Undegraduate S= Sequence Diagram D= Class Diagram U= Use Case J= Javadoc Most frequents patterns before reaching source code
  37. 37. 18% 8% 2% 2% 1% 1% 3% 0% 1% 0% 4% 12% 16% 12% 10% 4% 4% 1% 2% 1% 1% 5% 0% 2% 4% 6% 8% 10% 12% 14% 16% 18% 20% S D (SD)+ (US)+ U(SD)+ (DS)+ J U S(US)+ SU(SD)+ Other Graduate Undegraduate More experienced participants use a more “Integrated approach” S= Sequence Diagram D= Class Diagram U= Use Case J= Javadoc Most frequents patterns before reaching source code
  38. 38. Source Code Sequence Diagram Javadoc Transition Graph between kinds of Software Artifacts
  39. 39. Source Code Sequence Diagram Javadoc 1) From Source Code participants in most cases “go back” to Sequence and Class Diagrams 2) From Sequence and Class Diagrams participants in most cases “go back” to Source Code 3) Starting from a Use Case, participants go ahead reading Sequence Diagrams. Only after, they reading and writing Source Code Transition Graph between kinds of Software Artifacts
  40. 40. 0% 20% 11% 30% 16% 61% 41% 70% 12% 15% 60% 24% 58% 54% 0% 10% 20% 30% 40% 50% 60% 70% 80% USDC USD USC UDC SDC SC DC Graduates Undergraduates 1) Developers never used Javadoc Most Frequents Patterns 2) Patterns end always with Source Code S= Sequence Diagram D= Class Diagram U= Use Case J= Javadoc C= Source Code
  41. 41. Summary RQ2 • 1) Participants tend to start the assigned task from source code or from design documents , i.e., class and sequence diagrams • 2) More experienced participants tend to follow a more integrated approach than less experienced Ones • 3) During tasks, participants tend to go back and forth repeatedly between source code and to design diagrams (sequence and class diagrams).
  42. 42. …Conclusions…
  43. 43. …Conclusions…
  44. 44. …Conclusions…
  45. 45. …Conclusions…
  46. 46. …Conclusions…
  47. 47. …Future Directions…

×