Interaction-Aware Development Environments

Interaction-Aware
Development Environments
Research Advisor
Prof. Michele Lanza
Dissertation Committee
Prof. Serge Demeyer
Prof. Radu Marinescu
Prof. Matthias Hauswirth
Prof. Cesare Pautasso
Research Co-Advisor
Dr. Andrea Mocci
R E V E A L
@robertominelli
Roberto Minelli
Un iversità
della
Svizzera
italian a
Software
Institute

Interaction-Aware

Interaction-Aware
record mine leverage

Interaction-Aware
record mine leverage
understand support

IDE interaction data
Interactions with the IDE

Interactions with the IDE
support software engineering activities
evolve IDEs according to user needs
How Are Java Software Developers Using the Eclipse IDE? 
G. C. Murphy, M. Kersten, L. Findlater. IEEE Software 2006
Categorization of Concerns: A Categorical Program Comprehension Model 
T. Frey, M. Gelhausen, G. Saake. PLATEAU 2011
IDE interaction data

“Interaction-Aware Development Environments
enable novel and in-depth analyses of the behavior
of software developers and set the ground to provide
developers with effective and actionable support
for their activities inside the IDE.”
—Roberto Minelli, 2017
Our Thesis

workspace
spotter
inspector
debugger
code
browser
finder

senders
browser
implementors 
browser

Pharo Object Model
classes
packages

Pharo Object Model
classes protocols
packages

Pharo Object Model
classes protocols
packages selectors
(methods)

Pharo Object Model
source
code
classes protocols
packages selectors
(methods)

Recording, Modeling, and  
Interpreting Interaction Data
2

2
Modeling and Recording IDE Interactions

Modeling IDE Interactions
interaction
event

meta 
event
interaction
event
• Selecting a method or a class in the code browser
• Stepping in a debugger
• Adding or removing a method from a class

meta 
event
user input
event
interaction
event
• Pressing a mouse button
• Pressing a keystroke on the keyboard
• Moving the mouse

meta 
event
user input
event
user interface
event
interaction
event
• Opening or closing a window
• Resizing or moving a window

DFlow
IDE Interactions
DFlow: Recording IDE Interactions

DFlow

DFlow
filter
Interesting Interactions
Irrelevant Interactions

DFlow
Recorder
Visualizer
…
propagate
pub/sub
filter
Interesting Interactions
Irrelevant Interactions
Propagated Interactions

PositionEntities
User Input EventMeta Event User Interface Event
Navigation Event Inspect Event Edit Event
Attributes
Window Event
Attributes
Mouse Event
Key Combination
Keystroke Event
Mouse Moved Mouse Button Mouse Wheel
Direction
Window Collapsed Window ExpandedWindow Moved Window Resized
Window Activated Window Closed Window LabelledWindow Opened
Button IDStart Point
End Point
Old Label
New Label
Initial Position
Initial Extent
Old Position
New Position
Old Size
New Size
Event

sessions
development time
events
windows
1,631
>770h
~9.5M
>40k
developers17

2
Interpreting and Analyzing the Data
Modeling and Recording IDE Interactions
Measuring Navigation Efﬁciency in the IDE 
R. Minelli, A. Mocci, M. Lanza. IWESEP 2016
Quantifying Program Comprehension with Interaction Data 
R. Minelli, A. Mocci, M. Lanza, T. Kobayashi. QSIC 2014
I Know What You Did Last Summer 
R. Minelli, A. Mocci, M. Lanza. ICPC 2015

program
understanding

program
understanding
development 
activities

program
understanding
development 
activities
navigation 
efficiency

Development 
Activities
Navigation 
Efficiency
program
understanding
development 
activities
navigation 
efficiency

“Program comprehension takes more than half the time
spent on software maintenance”
program
understanding
“Maintenance accounts for 55 to 95% of the total costs
of a software system”
“Program comprehension occupies half of the time of
developers”
Principles of Software Engineering and Design 
M. Zelkowitz, A. Shaw, J. Gannon. Prentice Hall. 1979
Leveraging Legacy System Dollars for E-Business  
L. Erlikh. IT Professional. 2000
Application Program Maintenance Study: Report to Our Respondents  
R. K. Fjeldstad, W. T. Hamlen. GUIDE 48. 1983

Modeling Interaction Histories
t1
NR NR NW NR
t2 t3 t6
NR
t4
NR
t5
t1
N N E N I N
t2 t3 t4 t5 t6

t1
NR NR NW NR
t2 t3 t6
NR
t4
NR
t5
t1
N N E N I N
t2 t3 t4 t5 t6
events are instantaneous!

t1
N N E N I N
t2 t3 t4 t5 t6
UnderstandNavigation Edit Inspect
t1
NR NR NW NR
t2 t3 t6
NR
t4
NR
t5

t1
N N N N
t2 t4 t7
N
t5
N
t6
E
t1
N N E N I N
t2 t3 t4 t5 t6
t3
UnderstandNavigation Edit Inspect

program
understanding
development 
activities
navigation 
efficiency
time54-88%
56-94% time

program
understanding
development 
activities
navigation 
efficiency
time54-88%
56-94% time
“Program comprehension occupies half of the time of
developers”
Application Program Maintenance Study: Report to Our Respondents  
R. K. Fjeldstad, W. T. Hamlen. GUIDE 48. 1983
more than

+ low-level eventsmeta events
Reconstructing Development Activities

events
sprees
t
t
method
saved
search
starts
search
ends
out/in
the IDE
window
activated
window
activated
active window
>RT >RT >RT DOI
MS1 MS2 MS3 MS4 MS5KS1 KS2 KS3 KS4
activities t
A1 A2 A3 A4 A5 A6 A7
RT = 1s
Mouse Keyboard
Window Meta
Events
Mouse Keyboard
Activity
Sprees and Activities
Workspace Code Browser
Finder
Windows

events
sprees
t
t
method
saved
search
starts
search
ends
out/in
the IDE
window
activated
window
activated
active window
>RT >RT >RT DOI
MS1 MS2 MS3 MS4 MS5KS1 KS2 KS3 KS4
activities t
A1 A2 A3 A4 A5 A6 A7
RT = 1s
Mouse Keyboard
Window Meta
Events
Mouse Keyboard
Activity
Finder
Windows
5 M
175k
32 k

active window
activities t
A1 A2 A3 A4 A5 A6 A7
Mouse Keyboard
Window Meta
Events
Mouse Keyboard
Activity
Finder
Windows
method
saved
search
starts
search
ends
out/in
the IDE
window
activated
window
activated

activities t
A1 A2 A3 A4 A5 A6 A7
Mouse Keyboard
Window Meta
Events
Mouse Keyboard
Activity
Finder
Windows
active window
method
saved
search
starts
search
ends
out/in
the IDE
window
activated
window
activated
editing
time

activities t
A1 A2 A3 A4 A5 A6 A7
understanding
time(basic)
Mouse Keyboard
Window Meta
Events
Mouse Keyboard
Activity
Finder
Windows
active window
method
saved
search
starts
search
ends
out/in
the IDE
window
activated
window
activated
editing
time

understanding
time
editing
time

understanding
time
editing
time
user interface
time

understanding
time
editing
time
user interface
time
navigation
time

understanding
time
editing
time
user interface
time
navigation
time
outside IDE
time

8%
4%
14%
5%
69%
How Do Developers Spend Their Time?
70%
5%
14%
4%
8%

The Impact of Work Fragmentation

time spent
outside the IDE
number of times
outside the IDE

time spent
outside the IDE
number of times
outside the IDE
understanding
time
user interface
time
vs.

vs.
time spent
outside the IDE
number of times
outside the IDE
understanding
time
user interface
time

The more fragmented is the workflow of developers…
…the more time they need to rearrange  
the UI of the IDE and to understand code

development 
activities
program
understanding

program
understanding
development 
activities
navigation 
efficiency

=
ideal navigation effort
real navigation effort
Modeling Navigation Efficiency

=
recorded IDE interactions

=
estimation
recorded IDE interactions

Estimating the Ideal Navigation Effort

navigation cost

navigation cost involved entities

Navigation Cost

Navigation Cost
optimistic
unitary cost

Navigation Cost
pessimistic
maximum cost

Navigation Cost
realistic
Δ-cost or UI-aware

Involved Entities

Involved Entities
edited

Involved Entities
t
edited

Involved Entities
t
sequence
edited

Involved Entities
t
sequence
set
edited

Involved Entities
t
sequence
set
sorted set
edited

involved
entities
navigation
cost
optimistic
pessimistic
realistic
sequence
set
sorted set
unitary cost
maximum cost
Δ-cost
UI-aware

Navigation Efficiency
≈ 5-39%

Navigation Efficiency
D1 D2 D3 D4 D5 D6
0.00.10.20.30.40.5
≈ 5-39%

Developers perform 
more navigations than needed

Developers perform 
more navigations than needed
≈ 13-17x
1
navigation efficiency
- 1

Visual Analytics of  
Development Sessions
3

3
Visualizing User Interface Interactions
Visualizing Developer Interactions 
R. Minelli, A. Mocci, M. Lanza, L. Baracchi. VISSOFT 2014

Track of Windows
main window
short window(s)

UI View
Open
Activate
Resize/Move
Close
Minimize/Expand
Minimized Backbone
Track of Windows

3:00 6:00 18:35 21:00 23:00 45:43 48:00 51:00 54:00 56:33
10:20 20:12
Activity Timeline
Navigation
Inspection
Edit
Understanding
Vertical Lines
Explicit Pause
Implicit Pause
Commit
UI View
Open
Activate
Resize/Move
Close
Minimize/Expand
Minimized Backbone
Visualizing UI Interactions

storytelling
The Inspection Valley
Implement First, Verify Later
Home Sweet Home
Curing the Window Plague

“[…] developers using a modern IDE are forced to
open views on numerous source artifacts to reveal
[these] hidden relationships, leading to a crowded
workspace with many opened windows or tabs.”
Curing the Window Plague
Autumn Leaves: Curing the Window Plague in IDEs 
D. Röthlisberger, O. Nierstrasz, S. Ducasse. WCRE 2009

classification
dominant track
single-track
multi-track
fragmented

classification
dominant track
track flow
single-track
sequential
multi-track
ping-pong
fragmented

not classifiedfragmented
multi-track
single-track
ping-pong
not classified
sequential
dominant track
track flow

3
Visualizing User Interface Interactions
Visualizing the Evolution of Working Sets
Visualizing the Evolution of Working Sets 
R. Minelli, A. Mocci, M. Lanza. VISSOFT 2016

A group of program entities which a developer has
interacted with during a particular period of time.
Method
Class
What is a Working Set?

Method
Class
current

Method
Class
current past

Nodes
shape
1 event
10+ events
edited
color&
stroke
recent
old
size
Edges
color&
stroke
1 occurrence
10+ occurrences
past
working
set
current
working
set
start
end
Visualizing the Evolution of Working Sets
Method
Class

snapshots / session~80
snapshots72,613

snapshots / session
working set (ws) size
current ws size
~80
10
7
snapshots72,613
past ws size3

Visual Patterns
snapshot evolutionary

Visual Patterns
Past: To Edit or Not To Edit
U Can’t Touch This
The Guiding Star
Stay Focused, Stay Foolish!
Moving in Circlessnapshot

U Can’t Touch This
1 event
10+ events
edited
color&
stroke

Visual Patterns
The Past Awakens
Multi-Part Session
Thirst for Knowledge
The Working Funnel
evolutionary

The Past Awakens
part 1 part 2 part 3
no past WS past WS increases the past awakens
(i.e., past WS decreases)

Multi-Part Session
part 1 part 2 part 4part 3
1st task exploration 2nd task wrapping up

The Working Funnel
part 1 part 2 part 4part 3 part 5

The Working Funnel
part 1 part 2 part 3 part 4

Supporting Developers  
with Interaction Data
4

4
The Plague Doctor
The Plague Doctor: A Promising Cure for the Window Plague 
R. Minelli, A. Mocci, M. Lanza. ICPC 2015 (ERA)

The Window Plague

AutumnLeaves automatically selects windows unlikely
for future use to be closed or grayed out while
important ones are displayed more prominently.
Autumn Leaves

The Plague Doctor
young old
age
pinned
extra
low
importance
high

Models & Strategies
models strategies

Models & Strategies
models strategies
entities windows

Models & Strategies
models strategies
closingweightingentities windows

models strategies
closingweightingentities windows
fixed update top # windows
below threshold
Models & Strategies

4
The Plague Doctor
Taming the UI of the IDE
Taming the IDE with Fine-grained Interaction Data 
R. Minelli, A, Mocci, R. Robbes, M. Lanza. ICPC 2016

screen
space
Space Occupancy Metrics

screen
space
free
space

screen
space
free
space
no
overlapping

screen
space
free
space
no
overlappinglow overlap
(depth: 2)

screen
space
free
space
no
(depth: 2)
high overlap
(depth: 4)

no
(depth: 2)
high overlap
(depth: 4)
free
space
focus
space
screen
space

+ +
+
needed
space+ + =

space overlapping
occupied
free
focus
needed
space
depth
weighted
20.95%
2.76
69.05%
48.22%
51.78%
32.66%
96.83%

Chaos Levels
comfy ok mess hell

16.98% 21.11% 10.88%51.4%
Chaos Levels
comfy ok mess hell
11%
21%
17%
51%

Developers spend more than 30% of their time  
in highly chaotic IDE configurations…

so
what?

Chaos, UI, and Understanding
8%
4%
14%
5%
69% 70%
5%
14%
4%
8%

vs.
11%
21%
17%
51%

vs.
11%
21%
17%
51%
not statistically significant
statistically significant
comfy
ok
-0.34
-0.04

vs.
11%
21%
17%
51%
comfy
ok
-0.34
-0.04
mess
hell
0.16
0.42

vs.
11%
21%
17%
51%
comfy
ok
-0.34
-0.04
mess
hell
0.16
0.42
comfy
ok
-0.27
0.05
mess
hell
0.11
0.26

…and this impacts both  
UI and understanding time!

Windows, Focus, and Source Code

code browser debugger send / implement

source code
code browser debugger send / implement

Taming Strategies
= +
elision layout

comfy ok mess hell
11%
21%
17%
51%
16.98% 21.11% 10.88%51.4%
Chaos Levels

comfy ok mess hell
11%
21%
17%
51%
3%
13%
16%
69%
-1.35% -8.08% -8.30%+17.73%
Chaos LevelsTamed (simulation)

The Eye
recommender
systems
crowdsourced  
mental models
adaptive 
user interfaces
Our Vision
adaptive 
user interfaces

Interaction-based Recommenders
“[…] tools that provide information items that are
estimated to be valuable for a software engineering
task in a given context.”
Recommendation Systems for Software Engineering 
M. P. Robillard, R. J. Walker, T. Zimmermann.. IEEE Software 2010

“How to establish the context is a general
challenge for recommendation systems.”

fine-grained IDE interactions
“How to establish the context is a general
challenge for recommendation systems.”

Supporting Software Developers with a Holistic Recommender System 
L. Ponzanelli, S. Scalabrino, G. Bavota, A. Mocci, R. Oliveto, M. Di Penta, M. Lanza. ICSE 2017

Live & Adaptive Views
live
coevolve with the
system and always
depict its current state

live
low high
interactions
no
coevolve with the
system and always

live adaptive
coevolve with the
system and always
adapt their “shape”
depending on the
context, the history,
and the task at hand

adaptive
adapt their “shape”
depending on the
context, the history,
and the task at hand
layout, colors, etc.
live
coevolve with the
system and always

Adaptive User Interfaces
“Although developers are known to spend much of
their development time reading and analyzing code,
mainstream IDEs do not do a good job of supporting
program comprehension.
IDEs are basically glorified text editors.”
—Oscar Nierstrasz
The Death of Object-Oriented Programming 
O. Nierstrasz. FASE 2016

Moldable Tools
Inauguraldissertationder Philosophisch-naturwissenschaftlichen Fakultätder Universität Bern
vorgelegt von
Andrei Chiş
von Rumänien
Leiter der Arbeit:
Prof. Dr. Oscar Nierstrasz
Institut für Informatik
Adaptive User Interfaces
Moldable Tools 
A. Chiş. PhD Dissertation. 2016

The Eye
recommender
systems
crowdsourced  
mental models
adaptive 
user interfaces
adaptive 
user interfaces
Our Vision

Crowdsourced Mental Models
)=f( , , , ,

5
Epilogue
Our Vision
Conclusions

What is interaction data?
The Pharo IDE
Part 1: Prologue

Modeling and recording IDE interactions
Interpreting and analyzing interaction histories
Part 2: Interpreting

Visualizing UI interactions
Visualizing the evolution of working sets
Visualizing other aspects of development
Part 3: Visualizing

The Plague Doctor
Measuring and taming the chaos in the IDE
Part 4: Supporting

FeedBaG++Corporate Research

MSRFeedBaG++Corporate Research SANER

Visualizing the Evolution of Working Sets 
R. Minelli, A. Mocci, M. Lanza. VISSOFT 2016
Taming the IDE with Fine-grained Interaction Data 
R. Minelli, A, Mocci, R. Robbes, M. Lanza. ICPC 2016
Blended, Not Stirred: Multi-Concern Visualization of Large Software Systems 
T. dal Sasso, R. Minelli, A. Mocci, M. Lanza. VISSOFT 2015
Free Hugs — Praising Developers For Their Actions 
R. Minelli, A. Mocci, M. Lanza. ICSE 2015 (NIER)
Towards Self-Adaptive IDEs 
R. Minelli. ICSME 2014 (DocSym)
Quantifying Program Comprehension with Interaction Data 
R. Minelli, A. Mocci, M. Lanza, T. Kobayashi. QSIC 2014
Visual Storytelling of Development Sessions 
R. Minelli, L. Baracchi, A. Mocci, M. Lanza. ICSME 2014 (ERA)
Conference
Visualizing the Workflow of Developers 
R. Minelli, M. Lanza. VISSOFT 2013
Software Analytics for Mobile Applications - Insights & Lessons Learned 
R. Minelli, M. Lanza. CSMR 2013
Measuring Navigation Efficiency in the IDE 
R. Minelli, A. Mocci, M. Lanza. IWESEP 2016
SODA: The Stack Overflow Dataset Almanac 
N. Latorre, R. Minelli, A, Mocci, L. Ponzanelli, M. Lanza. MUD 2015
DFlow - Towards the Understanding of the Workflow of Developers 
R. Minelli, M. Lanza. SATToSE 2013
UrbanIt: Visualizing Repositories Everywhere 
A. Ciani, R. Minelli, A. Mocci, M. Lanza. ICSME 2015
SAMOA - A Visual Software Analytics Platform for Mobile Applications 
R. Minelli, M. Lanza. ICSM 2013
Workshop
Tool Demo

“The journey is more important
than the end or the start.”

Interaction-Aware Development Environments

More Related Content

Similar to Interaction-Aware Development Environments

More from Roberto Minelli

Recently uploaded

Interaction-Aware Development Environments