Downloaded 15 times
- 1 / 99
- 2 / 99
- 3 / 99
- 4 / 99
- 5 / 99
- 6 / 99
- 7 / 99
- 8 / 99
- 9 / 99
- 10 / 99
- 11 / 99
- 12 / 99
- 13 / 99
- 14 / 99
- 15 / 99
- 16 / 99
- 17 / 99
- 18 / 99
- 19 / 99
- 20 / 99
![Software Systems as Cities: A Controlled Experiment
Richard Wettel and Michele Lanza
REVEAL @ Faculty of Informatics
University of Lugano
{richard.wettel,michele.lanza}@usi.ch
Romain Robbes
PLEIAD @ DCC
University of Chile
rrobbes@dcc.uchile.cl
ABSTRACT
Software visualization is a popular program comprehension tech-
nique used in the context of software maintenance, reverse engineer-
ing, and software evolution analysis. While there is a broad range
of software visualization approaches, only few have been empiri-
cally evaluated. This is detrimental to the acceptance of software
visualization in both the academic and the industrial world.
We present a controlled experiment for the empirical evaluation
of a 3D software visualization approach based on a city metaphor
and implemented in a tool called CodeCity. The goal is to provide
experimental evidence of the viability of our approach in the context
of program comprehension by having subjects perform tasks related
to program comprehension. We designed our experiment based on
lessons extracted from the current body of research. We conducted
the experiment in four locations across three countries, involving
41 participants from both academia and industry. The experiment
shows that CodeCity leads to a statistically significant increase in
terms of task correctness and decrease in task completion time. We
detail the experiment we performed, discuss its results and reflect
on the many lessons learned.
It has earned a reputation as an effective program comprehension
technique, and is widely used in the context of software maintenance,
reverse engineering and reengineering [2].
The last decade has witnessed a proliferation of software visual-
ization approaches, aimed at supporting a broad range of software
engineering activities. As a consequence, there is a growing need for
objective assessments of visualization approaches to demonstrate
their effectiveness. Unfortunately, only few software visualization
approaches have been empirically validated so far, which is detri-
mental to the development of the field [2].
One of the reasons behind the shortage of empirical evaluation lies
in the difficulty of performing a controlled experiment for software
visualization. On the one hand, the variety of software visualization
approaches makes it almost impossible to reuse the design of an
experiment from the current body of research. On the other hand, or-
ganizing and conducting a proper controlled experiment is, in itself,
a difficult endeavor, which can fail in many ways: data which does
not support a true hypothesis, conclusions which are not statistically
significant due to insufficient data points, etc.
We present a controlled experiment for the empirical evaluation
of a software visualization approach based on a city metaphor [16].
The aim is to show that our approach, implemented in a tool called](https://image.slidesharecdn.com/andreamocci-beautifuldesignbeautifulcoding-150609144224-lva1-app6891/85/Andrea-Mocci-Beautiful-Design-Beautiful-Coding-at-I-T-A-K-E-Unconference-2015-26-320.jpg)
![Software Systems as Cities: A Controlled Experiment
Richard Wettel and Michele Lanza
REVEAL @ Faculty of Informatics
University of Lugano
{richard.wettel,michele.lanza}@usi.ch
Romain Robbes
PLEIAD @ DCC
University of Chile
rrobbes@dcc.uchile.cl
ABSTRACT
Software visualization is a popular program comprehension tech-
nique used in the context of software maintenance, reverse engineer-
ing, and software evolution analysis. While there is a broad range
of software visualization approaches, only few have been empiri-
cally evaluated. This is detrimental to the acceptance of software
visualization in both the academic and the industrial world.
We present a controlled experiment for the empirical evaluation
of a 3D software visualization approach based on a city metaphor
and implemented in a tool called CodeCity. The goal is to provide
experimental evidence of the viability of our approach in the context
of program comprehension by having subjects perform tasks related
to program comprehension. We designed our experiment based on
lessons extracted from the current body of research. We conducted
the experiment in four locations across three countries, involving
41 participants from both academia and industry. The experiment
shows that CodeCity leads to a statistically significant increase in
terms of task correctness and decrease in task completion time. We
detail the experiment we performed, discuss its results and reflect
on the many lessons learned.
It has earned a reputation as an effective program comprehension
technique, and is widely used in the context of software maintenance,
reverse engineering and reengineering [2].
The last decade has witnessed a proliferation of software visual-
ization approaches, aimed at supporting a broad range of software
engineering activities. As a consequence, there is a growing need for
objective assessments of visualization approaches to demonstrate
their effectiveness. Unfortunately, only few software visualization
approaches have been empirically validated so far, which is detri-
mental to the development of the field [2].
One of the reasons behind the shortage of empirical evaluation lies
in the difficulty of performing a controlled experiment for software
visualization. On the one hand, the variety of software visualization
approaches makes it almost impossible to reuse the design of an
experiment from the current body of research. On the other hand, or-
ganizing and conducting a proper controlled experiment is, in itself,
a difficult endeavor, which can fail in many ways: data which does
not support a true hypothesis, conclusions which are not statistically
significant due to insufficient data points, etc.
We present a controlled experiment for the empirical evaluation
of a software visualization approach based on a city metaphor [16].
The aim is to show that our approach, implemented in a tool called
solutions to the same task shows that the systems are quite different.
Experimenter effect. One of the experimenters is also the author
of the approach and of the tool. This may have influenced any
subjective aspect of the experiment. One instance of this threat
is that the task solutions may not have been graded correctly. To
mitigate this threat, the three authors built a model of the answers
and a grading scheme and then reached consensus. Moreover, the
grading was performed in a similar manner and two of the three
experimenters graded the solutions blinded, i.e., without knowing
the treatments (e.g., tool) used to obtain the solutions. Even if we
tried to mitigate this threat extensively, we cannot exclude all the
possible influences of this factor on the results of the experiment.
11. CONCLUSION
We presented a controlled experiment aimed at evaluating our
software visualization approach based on a city metaphor. We
designed our experiment from a list of desiderata built during an
extensive study of the current body of research. We recruited large
samples of our target population, with subjects from both academia
and industry with a broad range of experience.
The main result of the experiment is that our approach leads
to an improvement, in both correctness (+24%) and completion
time (-12%), over the state-of-the-practice exploration tools. This
result is statistically significant. We believe this is due to both the
visualization as such, as well as the metaphor used by CodeCity, but
we can not measure the exact contribution of each factor.
Apart from an aggregated analysis, we performed a detailed anal-](https://image.slidesharecdn.com/andreamocci-beautifuldesignbeautifulcoding-150609144224-lva1-app6891/85/Andrea-Mocci-Beautiful-Design-Beautiful-Coding-at-I-T-A-K-E-Unconference-2015-27-320.jpg)
![Color Legend
UI View
Open
Activate
Minimize / ExpandResize / Move
Close Minimized backbone
Activity Timeline
Navigation
Inspection
Edit
Understanding
Vertical Lines
Explicit pause (sub-session)
Implicit pause (idle)
Commit
10:20 20:12
3:00 6:00 18:35 21:00 23:00 45:43 48:00 51:00 54:00 56:33
Session
Start
Duration of
pause between two
sub-sessions
[d.hh:mm:ss]
Explicit pause
(sub-sessions)
Implicit pause
(idle time) Minimization
Commit
Session
End
UI View
Activity
Timeline](https://image.slidesharecdn.com/andreamocci-beautifuldesignbeautifulcoding-150609144224-lva1-app6891/85/Andrea-Mocci-Beautiful-Design-Beautiful-Coding-at-I-T-A-K-E-Unconference-2015-53-320.jpg)
![I Know What You Did Last Summer
An Investigation of How Developers Spend Their Time
Roberto Minelli, Andrea Mocci and Michele Lanza
REVEAL @ Faculty of Informatics — University of Lugano, Switzerland
Abstract—Developing software is a complex mental activity,
requiring extensive technical knowledge and abstraction capa-
bilities. The tangible part of development is the use of tools to
read, inspect, edit, and manipulate source code, usually through
an IDE (integrated development environment). Common claims
about software development include that program comprehension
takes up half of the time of a developer, or that certain UI
(user interface) paradigms of IDEs offer insufficient support to
developers. Such claims are often based on anecdotal evidence,
throwing up the question of whether they can be corroborated
on more solid grounds.
We present an in-depth analysis of how developers spend their
time, based on a fine-grained IDE interaction dataset consisting
of ca. 740 development sessions by 18 developers, amounting to
200 hours of development time and 5 million of IDE events. We
propose an inference model of development activities to precisely
measure the time spent in editing, navigating and searching for
artifacts, interacting with the UI of the IDE, and performing
corollary activities, such as inspection and debugging. We report
several interesting findings which in part confirm and reinforce
some common claims, but also disconfirm other beliefs about
software development.
I. INTRODUCTION
Software development is a complex activity that requires
both technical knowledge and extensive abstraction capabili-
ties [1]. Even if the outcome of software development is code,
the development process is far from being just code writing
[2]. In fact, software systems are so large and complex [3]
While interacting with IDEs, developers generate a large
amount of data [13], [14]. These interactions happen at differ-
ent levels of abstraction: they can be conceptual events, like
adding a method to a class, or low level events like pressing
keys or moving the mouse to navigate between entities. While
researchers believe that this data is valuable [14], [15], most
of these interactions are not captured.
Our aim is to record interaction data and measure the time
effectively devoted to different activities. With this data, we
provide insights on the distribution of development activities
and we quantitatively answer questions like:
1) What is the effective time spent in program comprehen-
sion/understanding? What about the other activities like
editing, inspecting and navigating through code entities?
2) How much time do developers spend in fiddling with the
UI of an IDE?
3) What is the impact of the fragmentation of the develop-
ment flow?
We present an in-depth analysis on how developers spend
their time. We collected fine-grained interaction data using
DFLOW [16], our non-intrusive profiler for the PHARO IDE1
.
Our dataset consists of 740 development sessions of 18 devel-
opers. DFLOW collected about 200 hours of development time,
amounting to more than 5 million of IDE events. We propose](https://image.slidesharecdn.com/andreamocci-beautifuldesignbeautifulcoding-150609144224-lva1-app6891/85/Andrea-Mocci-Beautiful-Design-Beautiful-Coding-at-I-T-A-K-E-Unconference-2015-85-320.jpg)
Skip to main content
Uploaded byMozaic Works
Andrea Mocci: Beautiful Design, Beautiful Coding at I T.A.K.E. Unconference 2015
The document describes a software visualization approach called CodeCity that represents software systems as 3D cities. It presents results from a controlled experiment evaluating CodeCity on program comprehension tasks. The experiment involved 41 participants from academia and industry and found that CodeCity led to a statistically significant increase in task correctness and decrease in completion time compared to traditional tools. The document discusses lessons learned from designing and conducting the experiment and concludes that the city metaphor visualization provided benefits for program comprehension.
