1. The document summarizes a family of experiments on how documenting design pattern instances impacts source code comprehensibility. 2. The experiments measured the correctness, time, confidence and difficulty participants had in answering comprehension questions about code with and without pattern documentation. 3. The results showed that documenting pattern instances improved correctness of understanding for experienced participants like professionals and PhD students, but not for less experienced participants like undergraduate students. There was no significant difference between textual vs graphical documentation.

1. Documenting design pattern instances:
A Family of experiments on
Source code comprehensibility
Gabriella Dodero (uniBZ)
Joint work with G.Scanniello (uniBAS), C.Gravino, M.Risi,
G.Tortora (uniSA)
RDSS, December 2014

2. Summary:
1. A research question
2. A family of experiments
3. Our results

3. A research question
● Consider a software maintenance task, typically done by
someone who is NOT an author of such code.
● The system under maintenance contains several
patterns
● Is there a difference in such a task, if code is
DOCUMENTED showing use of pattern instances?
● That is, not documenting patterns w.r.t. documenting
wherever patterns have been used
● Is there a difference between ways of documenting
patterns?
● That is, textual w.r.t. graphical documentation

4. A family of experiments-1
● Make an experiment about software maintenance, i.e.
give a set of „software comprehension tasks“
● Repeat the experiment with different populations
(developers with different experience)
● What is measured:
– Correctness of understanding
– Time
– Confidence
– Perceived difficulty

5. A family of experiments - 2
Choose a piece of
software
● A chunk of JHotDraw v5.1
(1326 LOC, non trivial!)
Choose a notation for
documenting patterns
● Textual descriptions reported
in the code as comments
● Graphic documentation like
class diagrams

6. A family of experiments - 3
● „Family“ means replicating an original experiment
under controlled conditions
● In this case: using populations with different
experience as developers
Participants Where How many
Practitioners Companies 25
2nd year Msc U.Salerno 25
3rd year Bsc U.Basilicata 23
2-3 yr PhD unibz 15

7. A family of experiments - 4
● In each experiment,
● 1/3 of participants were given textual documentation,
● 1/3 were given graphical documentation, and
● 1/3 had no documentation on pattern instances
● Hypotheses:
(1) There is no difference in Correctness nor Time
between people having documentation and not having
documentation
(2)There is no difference between people with different
experience

8. A family of experiments - 5
● What was asked: 14 questions like this one:
● A post assessment questionnaire on the experience
● Typical duration was 2-3 hours

9. Results - 1
● Hypothesis (1) is rejected for Correctness of
understanding, by professionals and PhD
students only
– Time is not affected by documentation
● But there is no significant difference between
those who had textual documentation or
graphical documentation
● Hypothesis (2) is rejected for Experience, with
respect to Correctness of Understanding and
Time

10. Results - 2
● Correctness of understanding is significantly greater when
pattern instances are correctly identified
● 30% improvement on Correctness of understanding
● Correctness of understanding is greater also for Msc
students, when patterns are documented, but this is not
statistically significant
● Experienced participants (professionals, PhD) benefit
more than non-experienced participants (BSc, Msc
students) from the documentation on patterns
● Experienced participants with textually documented
patterns were the fastest to do the task

11. Results - 3
● Lessons for practitioners:
● Documenting instances improves code
comprehensibility, for maintainers with experience
● Reading such documentation does not slow down
comprehension tasks
● Comprehensibility improves when pattern instances are
correctly identified, when developers have at least a
Bsc degree
● Lessons for researchers:
● Bsc or Msc students may be too inexperienced
compared to professionals, to gather reliable results
by performing experiments on them!!!

12. Thank You!
Questions?
Gdodero @ unibz.it