Software that requires maintenance and evolution presumably has value that causes the producers of the software—individuals and organizations—to invest in these activities. Given that there is almost always more that any given software package or product can provide, software producers should be motivated in enabling maintenance and evolution activities and should be interested in the software engineering research efforts that are undertaken to address identified pain points. Yet, despite efforts by providers of research results (software engineering researchers) and interest by recipients (software producing individuals and organizations), a gap remains and too few research results make their way into use. In this keynote talk from ICSME 2021, I focus on research results that take the form of software tools for software producers and explore what this gap is and how the gap might be bridged. This exploration aims to provide some practical tips for how to orient research to create usable and useful software tools.

1. Creating
Usable and Useful
Software Tools
Gail C. Murphy
University of British Columbia
@gail_murphy

2. 2
Useful
Usable

3. Useful
Usable

4. - X
Too often a gap exists between
developers’ needs and tools

5. - X
Attention to usability and
usefulness can bridge the gap

6. Where we will end up
Usability + Usefulness Large-scale use
opportunity
requires
Understanding/study
of context of use
Impactful research
opportunities

7. Identify a problem Propose a tool
Consider usage
context
Build tool
Study tool in
context of use
Deploy License
Iterate
Roadmap

8. Propose a tool
Study tool in
context of use
Deploy License
Iterate
Part 1: Case study of refactoring tools
Based on the
Ph.D. work
of Anna
Eilertsen
Identify a problem
Build tool
Consider usage
context

9. Propose a tool
Study tool in
context of use
Deploy License
9
Iterate
Part 2: Re
fl
ections on tool research & development
Based on joint
work with
Dr. Mik Kersten
Identify a problem
Consider usage
context
Build tool

10. Part I: Case study of refactoring tools

11. 11
Identify the problem (for refactoring)
“Maintenance tends to degrade the structure of software,
ultimately making maintenance more costly. At times, then, it is
worthwhile to manipulate the structure of a system
to make changes easier. However, it is shown that
manual restructuring is an error-prone and expensive activity.”
W.G. Griswold
Program Restructuring as an Aid to Software Maintenance, 1991

12. 12
Propose a tool (refactoring)
Refactoring catalog
(Fowler)
Refactoring tools available in IDEs
1992
1991 2000
Proof-of-concept tools
(Griswold, Opdyke, Brandt)
Examples
Rename method / class
Extract class
Change method signature
Inline

13. 13
Identify a problem Propose a tool
Consider usage
context
Study tool in

14. fi
eld study of
refactoring challenges
and bene
fi
ts
“Developers said they
do 86% of refactoring
manually”
[MPB12] [KZN12] [SSS15]
Challenges to
and solutions for
refactoring adoption
“59% of the
respondents were
dissatis
fi
ed with the
quality of available
refactoring tools”
[STV16]
55% of developers
“answered that
refactoring was
performed manually
without tool support”
Why we refactor?
Confessions of GitHub
contributors

15. [MPB12] [KZN12] [SSG15] [STV16]
Refactoring tools are under-utilized
15

16. 16
Awareness
[MPB09, V+12,
STV16]
Inability to select
code
[MB08]
Predictability [V+12]
Trust
[MBP09, L+15,
STV16]
And many other
reasons…
Why are refactoring tools
under-utilized?

17. Identify a problem Propose a tool
Study tool in
context of use
Deploy License
Iterate
Consider usage
context
Build tool

18. 18
Furda meras
Consider usage context…

19. 19
Furda meras
Consider usage context…
Can study tool invocation
Can help improve usability
e.g., [MB08]
Modi
fi
cations may still limit overall use
(and usefulness)

20. 20
Furda meras
Consider usage context…
Need to study tool use in likely tasks
Can lead to new requirements for tools
Consider interaction of usability and
usefulness

21. 21
A study of refactorings during
software change tasks
Organize test
methods
1
Removing
redundant
methods
2
Overloaded
method
reduction
3
Tasks
[Eilertsen and Murphy, JSEP 2021]
15 male and 2 female
Average 10 years of experience
Participants
System Replica of part of Apache Commons-Lang
78k lines of Java, 1335 JUnit tests

22. A study of refactorings during
software change tasks
Organize test
methods
1
Removing
redundant
methods
2
Overloaded
method
reduction
3
Tasks
Applicable
refactorings
move-metho
d
extract-class
inline-metho
d
safe-delet
e
inline-metho
d
remove-paramete
r
Observation and interviews, up to 1.5 hours
32 hours of video, transcribed to 13k words
Method
Data
[Eilertsen and Murphy, JSEP 2021]

23. 23
A study of refactorings during
software change tasks
VSC-diffs Investigate how code was changed and verify change
structural-navigation
Understand code and locate code that needs to
change
refactoring-tool Change code and organize change
test-suite
Understand code, understand impact of changes,
verify a change is correct
debug-tool Understand what code does
text-search
Find references, callers and declarations of methods
and parameters
compiler-output
Errors and warnings used to navigate code,
understand impact of changes and verify changes
copy/cut/paste Used to edit code
Tools used
[Eilertsen and Murphy, JSEP 2021]

24. 24
A study of refactorings during
software change tasks
Local
Changes guided by local and immediate
feedback, such as from the compiler
Structure
Changes guided by code structure, such as
callers and references
Execute Changes guided by executing the test suite
Strategy
[Eilertsen and Murphy, JSEP 2021]

25. A study of refactorings during
software change tasks
Participants
Tasks
Local
Structure
Execute
Strategies were distributed participants and tasks
Individual participants sometimes used different strategies
[Eilertsen and Murphy, JSEP 2021]

26. A study of refactorings during
software change tasks
Refactoring tool use dropped off from considering the tool to actually using the tool
[Eilertsen and Murphy, JSEP 2021]

27. 27
A study of refactorings during
software change tasks
[Eilertsen and Murphy, JSEP 2021]

28. 28
A study of refactorings during
software change tasks
[Eilertsen and Murphy, JSEP 2021]

29. 29
Themes
Dif
fi
culties controlling impact Loss of knowledge of code
[Eilertsen and Murphy, JSEP 2021]

30. 30
The extent to which a system, product or
service can be used by speci
fi
ed users to
achieve speci
fi
ed goals with e
ff
ectiveness,
e
ffi
ciency and satisfaction in a speci
fi
ed
context of use.
Usability
Software developers employ refactoring tools
to help prepare or complete functional changes
to a software system. Software developers
seek these tools to be reasonable to locate and
apply for a desired intent (e
ff
ective), reasonable
in terms of time and e
ff
ort required to use the
tool (e
ffi
cient), and to add value to the
development process (satisfying).
Proposal for refactoring tools:
ISO 9241-11
[Eilertsen and Murphy, SANER 2021]

31. 31
Analysis of study transcripts
for usability
[Eilertsen and Murphy, SANER 2021]

32. 32
Analysis of study transcripts
for usability
Code Description
E
ff
ective User experiences tool as successfully performing a desired result
E
ffi
cient User experiences the tool as being fast or increasing productivity
Satisfaction User experiences having their needs or wants met by the use of the tool
Trust User experiences the tools as trustworthy, safe or reliable
Predictable
User understands up-front what will happen by using the tool or indicates
lack of surprise when using tool
[Eilertsen and Murphy, SANER 2021]

33. 33
Analysis of study transcripts
for usability
Participants commented frequently on
the predictability of refactoring tools
[Eilertsen and Murphy, SANER 2021]

34. 34
Considering a broader
population
Survey about tools
using in change tasks
46 respondents

35. 35
Tool usability profiles
Copy-Cut-Paste Textual Search Compiler Output
Simple Refactoring
Tools
Developers do not think these
tools need to be satisfying
Simple refactoring tools (e.g., rename)
should be satisfying,
compiler should be trustworthy
[Eilertsen Ph.D. Thesis, to appear]

36. 36
Tool usability profiles
Simple Refactoring
Tools
[Eilertsen Ph.D. Thesis, to appear]
Complex Refactoring
Tools
Note scales are slightly different on the top plots

37. 37
What did we learn about
usefulness and usability of
refactoring tools?
Re
fi
ned Theory of Usability for Refactoring Tools
Software developers employ refactoring tools to help prepare
or complete functional changes to a software system. Software
developers seek these tools to be reasonable to locate, and
for the tools to help them assess the e
ffi
ciency of the tool, in
terms of the costs and bene
fi
ts of the tool, before its use. To
enable e
ff
ective use in multiple situations, software developers
seek to guide how a tool changes the source code for a
system; this ability to tailor how a tool works can improve the
e
ffi
ciency of the tool for the developer. Software developers
also seek refactoring tools to explain their impact to source
code so that the software developer can understand the
e
ff
ectiveness of the tool. Software developers also expect
tools to communicate clearly and directly in terms that match
how software developers perceive refactoring operations.
These characteristics in a refactoring tool increase the
satisfaction of the software developer using a refactoring tool.
Controlling Impact Retaining Knowledge

38. Identify a problem Propose a tool
Study tool in
context of use
Deploy License
Iterate
Consider usage
context
Build tool

39. 39
Stepwise refactoring tools
Designed with attention to
usability themes learned
from usage of tools in
context of tasks
Assessed against common
problems with existing tools
from study of refactoring use
Eilertsen & Murphy, ICSME NIER 2021

40. Identify a problem Propose a tool
Study tool in
context of use
Deploy License
40
Iterate
Consider usage
context
Build tool
Opportunities to
continue study
of usage contexts
and new tool design
s
in context of use

41. Part 2: Re fl
ections on tool research & development

42. Propose a tool
Study tool in
context of use
Deploy License
42
Iterate
Based on joint
work with
Dr. Mik Kersten
Identify a problem
Consider usage
context
Build tool
Case study of Mylar/Mylyn/Tasktop

43. 43
Timeline
MYLAR MYLYN TASKTOP
Mylar - task-focused interface for
Eclipse IDE
Mylyn - rename of Mylar
Tasktop Technologies Inc. - created to
commercialize task-focused interface
2004 - Mylar created &
fi
rst industrial trial
2005 - Mylar open-sourced & large
fi
eld
study
2007 - Mylar renamed Mylyn
2007 - Tasktop created: individual & enterprise
sales (developer version)
2011 - Pivot to server version
2021 - Value stream management tools

44. 44
2004 - Mylar created &
fi
rst industrial trial
2005 - Mylar open-sourced & large
fi
eld
study
2007 - Mylar renamed Mylyn
2007 - Tasktop created: individual & enterprise
sales (developer version)
2011 - Pivot to server version
2021 - Value stream management tools
Study
tool in
context
of use
Deploy
(open
source)
Deploy
(open &
closed
source)

45. Propose a tool
Study tool in
context of use
Deploy License
45
Iterate
Continual attention to context of use
Identify a problem
Consider usage
context
Build tool

46. Summary

47. Special thanks to…
ANNA MARIA EILERTSEN DR. MIK KERSTEN
Organizers of
Fantastic students too numerous to name
Wonderful colleagues at Tasktop
NSERC for the support to discover

48. We have given software developers
a lot of tools…
Attending to tool usefulness and
usability can help bridge the gap to
more use

49. We need to consider and study
more than the moment a tool is used
We need to study the contexts in which
the tool is used

50. Propose a tool
Deploy License
50
Iterate
Usage context is key
Identify a problem
Consider usage
context
Build tool
Study tool in
context of use

51. 51
Lots of questions
to address
?
?
?
?
Can we de
fi
ne
and/or classify
usage contexts?
How can we make
tools
fi
t more
usage contexts?
What is the
“right” number
of tools?
Can we auto-
collect useful data
about usage
contexts?

52. Creating Usable and Useful
Software Tools
Usability + Usefulness Large-scale use
opportunity
requires
Understanding/study
of context of use
Impactful research
opportunities
@gail_murphy

53. 53
References
[KZN12] Kim, M., Zimmermann, T. and N. Nagappan. A
fi
eld study of refactoring challenges and bene ts. In Proc. Of FSE, 2012, 50:1-50:11.
[L+15] Leppänen, M., Mäkinen, S., Lahtinen, S., Sievi-Korte, O., Tuovinen, A.-P. and Männistö, T. Refactoring-a-shot in the dark?
IEEE Software, 2015, 32(6): 62-70.
[MB08] Murphy-Hill, E. and Black, A. Breaking the barriers to successful refactoring: Observations and tools for extract method.
In Proc. of ICSE, 2008, 421-430.
[MPB09] Murphy-Hill, E., Parnin, C. and Black, A.P. How we refactor, and how we know it. In Proc. Of ICSE, 2009, 287-297.
[MPB12] Murphy-Hill, E., Parnin, C. and Black, A.P. How we refactor, and how we know it. IEEE TSE, 2012, 38(1): 5-18.
[SSS15] Sharma, T., Suryanarayana, G. And Samarthyam, G. Challenges to and solutions for refactoring adoption: An industrial
perspective. IEEE Software, 32(6): 44-51.
[STV16] Silva, D., Tsantalis, N. and Valente, M.T. Why we refactor? Confessions of GitHub contributors. In Proc. Of FSE, 2016, 858-870.
[V+12] Vakilian, M., Chen, N., Negara, S., Rajkumar, B.A., Bailey, B.P. and Johnson, R.E. Use, disuse and misuse of automated
refactorings. In Proc. Of ICSE, 2012, 233-243.
Eilertsen, A.M. and Murphy, G.C., JSEP 2021. A study of refactorings during software change tasks.
Eilertsen, A.M. and Murphy, G.C., SANER 2021. The usability (or not) of refactoring tools.
Eilertsen, A.M. and Murphy, G.C., ICSME NIER 2021. Stepwise refactoring tools.
Eilertsen, A.M. Improving the usability of refactoring tools for software change tasks. Ph.D. thesis, University of Bergen, to appear.
Griswold 1991. Program restructuring as an aid to software maintenance, Ph.D. thesis, University of Washington.