Impactful SE Research:

Some Do’s

and More Don’ts
Gail C. Murphy

University of British Columbia

Tasktop Technologies Inc.
@gail_murphy
“software is eating
the world”
Marc Andreesen
IMPACT
3
Multiple Paths to Impact
€ Economic
Ý Social
Ħ Policy
á People
! Academic
Do
•understand and use the
real problem as a
beacon
•be open to
opportunities
•continually listen
•collaborate
•adapt
•be generous
•approach problems
with integrity
Don’t
•focus only on home runs
•put the research ahead
of the people
•think innovation only
happens in academia
•think scale doesn’t
matter
•think open source is
enough
•think the message
doesn’t matter
Do
•understand and use the
real problem as a
beacon
•be open to
opportunities
•continually listen
•collaborate
•adapt
•be generous
•approach problems
with integrity
Don’t
•focus only on home runs
•put the research ahead
of the people
•think innovation only
happens in academia
•think scale doesn’t
matter
•think open source is
enough
•think the message
doesn’t matter
Who Should

Fix This
Bug?
7
Mylar
Story
Hipikat
Four Stories
PhD: 

Reflexion

Models
Who Should

Fix This
Bug?
8
Mylar
Story
Hipikat
Reflexion
Models
A story about
academic and
social impact

and openness
to opportunity
PhD: 

Reflexion

Models
A bit of background
B.Sc.
Computing
Science
"♀
Software
Developer
(5 years)
!
Grad
School
(U. Washington)
Ph.D.: Reflexion Models
Murphy, Notkin & Sullivan, FSE 1995. SIGSOFT Retrospective Impact Award.
Goals
•Lightweight
• Apply to a large system in < 30 minutes

•Flexible
• Multiple kinds of source, relationships between ‘modules’

•Human-centric
• Provide information to human in human’s terms
I got lost
12
Reflexion Models for Excel
File
Sheet
Graph
Reflexion Models for Excel
Graph
Sheet
File
0
36734
912
1210
...
...
Reflexion Models: Take-aways
•Focus on the problem
• Understanding the implementation

•Notkin guidance
• “Your Ph.D. doesn’t have to be a home run”
• “It is the start of your career and life’s work not the end”

•Openness to opportunity
• Be ready to pitch
• Not the perfect case study but lots of good aspects
Ý Social! Academic
Who Should

Fix This
Bug?
16
Mylar
Story
PhD:

Reflexion

Models
Hipikat
Hipikat
A story about

academic impact
and the need
for scale
Hipikat
• Context:
• The best information about a development is in the heads of the
developers’

• Hypothesis:
• Pertinent artifacts to a task can be recommended from a latent group
memory formed from digital assets recorded as part of the development

• Prototype:
• Built a latent group memory automatically from repository information,
mailing lists, etc.
• Evaluation:
• Wizard-of-oz, controlled lab studies, simulation studies
Çubranić and Murphy, ICSE 2003. Ten year ICSE impact award.
Hipikat in Action
Hipikat in Action
Building Hipikat
• Collaboration with IBM when Eclipse was in development

• Likely first academic plug-in built for Eclipse

• IBM personnel participated in lab studies, increasing validity

• Lots of hiccups to overcome
• New development environment and architecture
• Constant upgrading at IBM to be on the latest
Hipikat: Take-aways
• Collaboration as an enabler
• Joint collaboration with industry partner

• Scale matters
• Eclipse presented itself as an opportunity we acted upon

• Messages matter
• Hipikat is still first in Google search
• The water cooler description always did and still resonates
• Research can push policy boundaries
• What was ethical to use from open source development?
! Academic á PeopleĦ Policy
Who Should

Fix This
Bug?
22
Mylar
Story
PhD:

Reflexion

Models
Hipikat
Who Should
Fix This Bug?
A story about

academic impact
and first base
can be enough
Who Should Fix This Bug?
• Problem:
• Lots of issues filed in open source
• How to reduce the effort and costs of triaging the bugs

• Proposed solution:
• Use supervised machine learning to learn triage results, like who
should fix a particular bug, which component the bug should be filed
against, etc.

• Evaluation:
• Applied against a few open source systems
• My favourite review comment: “academically honest” because we
explained where it didn’t work
Anvik, Murphy and Hiew, ICSE 2006. Ten-year ICSE impact award.
A problem before its time
• We cared about how hard it would be to apply the machine
learning approach to a new system

• Semi-automated configuration of machine learning approach
was important to us for approach to be usable

• At least 1/2 of John Anvik’s thesis was this semi-automated
configuration

• Hard to publish because the problem wasn’t understood (and we
didn’t explain well enough)

• We persevered
Who Should Fix This Bug: Take-aways
• Messages matter
• The title captured attention

• Integrity is important
• Outlining context where something works and where it does not is
valuable

• “First base” can be enough
• We opened a new area more than solving a problem completely

• Sometimes impact is not where you expect
• Would I have predicted this as the most-cited paper from the group?
! Academic á People
Who Should

Fix This
Bug?
26
Mylar
Story
PhD:

Reflexion

Models
Hipikat
The Mylar/
Tasktop
A story about

academic and

industrial
impact
TIMELINE
2004 Mylar
Information Overload
Mylar



See only what you’re working
27
TIMELINE
28
2007 Tasktop
Mik Kersten
Rob Elves
3 Co-founders
Gail Murphy
Shawn 

Minto
Leo Dos
Santos
2 Staff
2004 Mylar
TIMELINE
29
2017 Tasktop
2004 Mylar
43 of the Fortune 100
Mylar Highlights: 2004-2007
2004
2004
2005
2005
2005-06
2007
Mylar invention & innovation
1st industrial trial at IBM Toronto
1st academic publication
EclipseCon/Field Study/Eclipse project
Company discussions; biz competitions
Mik’s Ph.D. Friday; incorporate Monday
IBM

Trial
Publication
Murphy and Kersten, AOSD 2006. Ten-year Modularity impact award.
Mylar Highlights: 2004-2007
2004
2004
2005
2005
2005-06
2007
Mylar invention & innovation
1st industrial trial at IBM Toronto
1st academic publication
EclipseCon/Field Study/Eclipse project
Company discussions; biz competitions
Mik’s Ph.D. Friday; incorporate Monday
IBM

Trial
Publication
Robustness
&
community
building
Murphy and Kersten, AOSD 2006. Ten-year Modularity impact award.
Mylar Highlights: 2004-2007
2004
2004
2005
2005
2005-06
2007
Mylar invention & innovation
1st industrial trial at IBM Toronto
1st academic publication
EclipseCon/Field Study/Eclipse project
Company discussions; biz competitions
Mik’s Ph.D. Friday; incorporated Monday
IBM

Trial
Publication
Murphy and Kersten, AOSD 2006. Ten-year Modularity impact award.
2007
2008
2009
2011
2014
2015-17
Mylar becomes Eclipse Mylyn
Tasktop Dev
Tasktop Dev - Enterprise sales
Tasktop Sync
$11M Series A funding
Tasktop Data & Gateway capabilities
Eclipse Mylyn
Various innovations
Mylyn/Tasktop Highlights: 2007-2016
2007
2008
2009
2011
2014
2015-16
Mylar becomes Eclipse Mylyn
Tasktop Dev
Tasktop Dev - Enterprise sales
Tasktop Sync
$11M Series A funding
Tasktop Data & Gateway capabilities
Eclipse Mylyn
Various innovations
Mylyn/Tasktop Highlights: 2007-2016
2007
2008
2009
2011
2014
2015-16
Mylar becomes Eclipse Mylyn
Tasktop Dev
Tasktop Dev - Enterprise sales
Tasktop Sync
$11M Series A funding
Tasktop Data & Gateway capabilities
Eclipse Mylyn
Various innovations
Mylyn/Tasktop Highlights: 2007-2016
2007
2008
2009
2011
2014
2015-16
Mylar becomes Eclipse Mylyn
Tasktop Dev
Tasktop Dev - Enterprise sales
Tasktop Sync
$11M Series A funding
Tasktop Data & Gateway capabilities
Eclipse Mylyn
Various innovationsVarious
inventions
Mylyn/Tasktop Highlights: 2007-2016
2007
2008
2009
2011
2014
2015-16
Mylar becomes Eclipse Mylyn
Tasktop Dev
Tasktop Dev - Enterprise sales
Tasktop Sync
$11M Series A funding
Tasktop Data & Gateway capabilities
Eclipse Mylyn
Various innovationsVarious
inventions
Mylyn/Tasktop Highlights: 2007-2016
2007
2008
2009
2011
2014
2015-16
Mylar becomes Eclipse Mylyn
Tasktop Dev
Tasktop Dev - Enterprise sales
Tasktop Sync
$11M Series A funding
Tasktop Data & Gateway capabilities
Eclipse Mylyn
Various innovationsVarious
inventions
Continual
innovation &
incremental
validation
Mylyn/Tasktop Highlights: 2007-2016
2017: $11M Series B Funding

2017: Tasktop Integration Hub
Robustness
&
community
building
Continual
innovation &
incremental
validation
Mylar/Tasktop: Take-aways
•Collaboration
• With Mik Kersten, with open source community, with
Tasktopians, with industry partners, with customers …

•Adaptability
• Pivot to Tasktop Sync

•Open source takes work
• Tasktop supports Mylyn community
!Academic € Economic á People
Who Should

Fix This
Bug?
41
Mylar
Story
Hipikat
PhD: 

Reflexion

Models
42
Thanks to…
Mylar + Early Tasktop:
Tasktopians circa 2013
Mik Kersten, Rob Elves, Shawn Minto, Davor Cubranic, Taivo Evard, Nathan Hapke, 

Wesley Coelho, Meghan Allen, Leo Dos Santos, Steffen Pingel
Collaborators:

David Notkin, Kevin Sullivan,

Davor Çubranić, John

Anvik, Lyndon Hiew
Who Should

Fix This
Bug?
43
Mylar
Story
Hipikat
PhD: 

Reflexion

Models
Academic
Social
People
Academic

People


Academic
{Policy}
People
Academic

Social

Economic
People
Impacts
Multiple Paths to Impact
€ Economic
Ý Social
Ħ Policy
á People
! Academic
Do
•understand and use
the real problem as a
beacon
•be open to
opportunities
•continually listen
•collaborate
•adapt
•be generous
•be a great academic
Don’t
•focus only on home runs
•put the research ahead
of the people
•think innovation only
happens in academia
•think scale doesn’t
matter
•think open source is
enough
•think the message
doesn’t matter
At the start of the talk…
All in terms of “Do”s
•understand and use
the real problem as a
beacon
•be open to
opportunities
•continually listen
•collaborate
•adapt
•be generous
•be a great academic
•focus on the problem at
hand
•see the value in people in
your network
•follow innovations in
industry
•consider whether scale
matters
•realize open source
requires effort
•spend time on the
message
Impact can take many forms



Gail C. Murphy

University of British Columbia

Tasktop Technologies Inc. @gail_murphy
Focus on the problem
Be open to possibilities
Think outside the box
Do research with integrity

Impactful SE Research: Some Do's and More Don'ts

  • 1.
    Impactful SE Research:
 SomeDo’s
 and More Don’ts Gail C. Murphy
 University of British Columbia
 Tasktop Technologies Inc. @gail_murphy
  • 2.
    “software is eating theworld” Marc Andreesen
  • 3.
  • 4.
    Multiple Paths toImpact € Economic Ý Social Ħ Policy á People ! Academic
  • 5.
    Do •understand and usethe real problem as a beacon •be open to opportunities •continually listen •collaborate •adapt •be generous •approach problems with integrity Don’t •focus only on home runs •put the research ahead of the people •think innovation only happens in academia •think scale doesn’t matter •think open source is enough •think the message doesn’t matter
  • 6.
    Do •understand and usethe real problem as a beacon •be open to opportunities •continually listen •collaborate •adapt •be generous •approach problems with integrity Don’t •focus only on home runs •put the research ahead of the people •think innovation only happens in academia •think scale doesn’t matter •think open source is enough •think the message doesn’t matter
  • 7.
    Who Should
 Fix This Bug? 7 Mylar Story Hipikat FourStories PhD: 
 Reflexion
 Models
  • 8.
    Who Should
 Fix This Bug? 8 Mylar Story Hipikat Reflexion Models Astory about academic and social impact
 and openness to opportunity PhD: 
 Reflexion
 Models
  • 9.
    A bit ofbackground B.Sc. Computing Science "♀ Software Developer (5 years) ! Grad School (U. Washington)
  • 10.
    Ph.D.: Reflexion Models Murphy,Notkin & Sullivan, FSE 1995. SIGSOFT Retrospective Impact Award.
  • 11.
    Goals •Lightweight • Apply toa large system in < 30 minutes
 •Flexible • Multiple kinds of source, relationships between ‘modules’
 •Human-centric • Provide information to human in human’s terms
  • 12.
  • 13.
    Reflexion Models forExcel File Sheet Graph
  • 14.
    Reflexion Models forExcel Graph Sheet File 0 36734 912 1210 ... ...
  • 15.
    Reflexion Models: Take-aways •Focuson the problem • Understanding the implementation
 •Notkin guidance • “Your Ph.D. doesn’t have to be a home run” • “It is the start of your career and life’s work not the end”
 •Openness to opportunity • Be ready to pitch • Not the perfect case study but lots of good aspects Ý Social! Academic
  • 16.
  • 17.
    Hipikat • Context: • Thebest information about a development is in the heads of the developers’
 • Hypothesis: • Pertinent artifacts to a task can be recommended from a latent group memory formed from digital assets recorded as part of the development
 • Prototype: • Built a latent group memory automatically from repository information, mailing lists, etc. • Evaluation: • Wizard-of-oz, controlled lab studies, simulation studies Çubranić and Murphy, ICSE 2003. Ten year ICSE impact award.
  • 18.
  • 19.
  • 20.
    Building Hipikat • Collaborationwith IBM when Eclipse was in development
 • Likely first academic plug-in built for Eclipse
 • IBM personnel participated in lab studies, increasing validity
 • Lots of hiccups to overcome • New development environment and architecture • Constant upgrading at IBM to be on the latest
  • 21.
    Hipikat: Take-aways • Collaborationas an enabler • Joint collaboration with industry partner
 • Scale matters • Eclipse presented itself as an opportunity we acted upon
 • Messages matter • Hipikat is still first in Google search • The water cooler description always did and still resonates • Research can push policy boundaries • What was ethical to use from open source development? ! Academic á PeopleĦ Policy
  • 22.
    Who Should
 Fix This Bug? 22 Mylar Story PhD:
 Reflexion
 Models Hipikat WhoShould Fix This Bug? A story about
 academic impact and first base can be enough
  • 23.
    Who Should FixThis Bug? • Problem: • Lots of issues filed in open source • How to reduce the effort and costs of triaging the bugs
 • Proposed solution: • Use supervised machine learning to learn triage results, like who should fix a particular bug, which component the bug should be filed against, etc.
 • Evaluation: • Applied against a few open source systems • My favourite review comment: “academically honest” because we explained where it didn’t work Anvik, Murphy and Hiew, ICSE 2006. Ten-year ICSE impact award.
  • 24.
    A problem beforeits time • We cared about how hard it would be to apply the machine learning approach to a new system
 • Semi-automated configuration of machine learning approach was important to us for approach to be usable
 • At least 1/2 of John Anvik’s thesis was this semi-automated configuration
 • Hard to publish because the problem wasn’t understood (and we didn’t explain well enough)
 • We persevered
  • 25.
    Who Should FixThis Bug: Take-aways • Messages matter • The title captured attention
 • Integrity is important • Outlining context where something works and where it does not is valuable
 • “First base” can be enough • We opened a new area more than solving a problem completely
 • Sometimes impact is not where you expect • Would I have predicted this as the most-cited paper from the group? ! Academic á People
  • 26.
    Who Should
 Fix This Bug? 26 Mylar Story PhD:
 Reflexion
 Models Hipikat TheMylar/ Tasktop A story about
 academic and
 industrial impact
  • 27.
  • 28.
    TIMELINE 28 2007 Tasktop Mik Kersten RobElves 3 Co-founders Gail Murphy Shawn 
 Minto Leo Dos Santos 2 Staff 2004 Mylar
  • 29.
  • 30.
    Mylar Highlights: 2004-2007 2004 2004 2005 2005 2005-06 2007 Mylarinvention & innovation 1st industrial trial at IBM Toronto 1st academic publication EclipseCon/Field Study/Eclipse project Company discussions; biz competitions Mik’s Ph.D. Friday; incorporate Monday IBM
 Trial Publication Murphy and Kersten, AOSD 2006. Ten-year Modularity impact award.
  • 31.
    Mylar Highlights: 2004-2007 2004 2004 2005 2005 2005-06 2007 Mylarinvention & innovation 1st industrial trial at IBM Toronto 1st academic publication EclipseCon/Field Study/Eclipse project Company discussions; biz competitions Mik’s Ph.D. Friday; incorporate Monday IBM
 Trial Publication Robustness & community building Murphy and Kersten, AOSD 2006. Ten-year Modularity impact award.
  • 32.
    Mylar Highlights: 2004-2007 2004 2004 2005 2005 2005-06 2007 Mylarinvention & innovation 1st industrial trial at IBM Toronto 1st academic publication EclipseCon/Field Study/Eclipse project Company discussions; biz competitions Mik’s Ph.D. Friday; incorporated Monday IBM
 Trial Publication Murphy and Kersten, AOSD 2006. Ten-year Modularity impact award.
  • 33.
    2007 2008 2009 2011 2014 2015-17 Mylar becomes EclipseMylyn Tasktop Dev Tasktop Dev - Enterprise sales Tasktop Sync $11M Series A funding Tasktop Data & Gateway capabilities Eclipse Mylyn Various innovations Mylyn/Tasktop Highlights: 2007-2016
  • 34.
    2007 2008 2009 2011 2014 2015-16 Mylar becomes EclipseMylyn Tasktop Dev Tasktop Dev - Enterprise sales Tasktop Sync $11M Series A funding Tasktop Data & Gateway capabilities Eclipse Mylyn Various innovations Mylyn/Tasktop Highlights: 2007-2016
  • 35.
    2007 2008 2009 2011 2014 2015-16 Mylar becomes EclipseMylyn Tasktop Dev Tasktop Dev - Enterprise sales Tasktop Sync $11M Series A funding Tasktop Data & Gateway capabilities Eclipse Mylyn Various innovations Mylyn/Tasktop Highlights: 2007-2016
  • 36.
    2007 2008 2009 2011 2014 2015-16 Mylar becomes EclipseMylyn Tasktop Dev Tasktop Dev - Enterprise sales Tasktop Sync $11M Series A funding Tasktop Data & Gateway capabilities Eclipse Mylyn Various innovationsVarious inventions Mylyn/Tasktop Highlights: 2007-2016
  • 37.
    2007 2008 2009 2011 2014 2015-16 Mylar becomes EclipseMylyn Tasktop Dev Tasktop Dev - Enterprise sales Tasktop Sync $11M Series A funding Tasktop Data & Gateway capabilities Eclipse Mylyn Various innovationsVarious inventions Mylyn/Tasktop Highlights: 2007-2016
  • 38.
    2007 2008 2009 2011 2014 2015-16 Mylar becomes EclipseMylyn Tasktop Dev Tasktop Dev - Enterprise sales Tasktop Sync $11M Series A funding Tasktop Data & Gateway capabilities Eclipse Mylyn Various innovationsVarious inventions Continual innovation & incremental validation Mylyn/Tasktop Highlights: 2007-2016 2017: $11M Series B Funding
 2017: Tasktop Integration Hub
  • 39.
  • 40.
    Mylar/Tasktop: Take-aways •Collaboration • WithMik Kersten, with open source community, with Tasktopians, with industry partners, with customers …
 •Adaptability • Pivot to Tasktop Sync
 •Open source takes work • Tasktop supports Mylyn community !Academic € Economic á People
  • 41.
  • 42.
    42 Thanks to… Mylar +Early Tasktop: Tasktopians circa 2013 Mik Kersten, Rob Elves, Shawn Minto, Davor Cubranic, Taivo Evard, Nathan Hapke, 
 Wesley Coelho, Meghan Allen, Leo Dos Santos, Steffen Pingel Collaborators:
 David Notkin, Kevin Sullivan,
 Davor Çubranić, John
 Anvik, Lyndon Hiew
  • 43.
    Who Should
 Fix This Bug? 43 Mylar Story Hipikat PhD:
 Reflexion
 Models Academic Social People Academic
 People 
 Academic {Policy} People Academic
 Social
 Economic People Impacts
  • 44.
    Multiple Paths toImpact € Economic Ý Social Ħ Policy á People ! Academic
  • 45.
    Do •understand and use thereal problem as a beacon •be open to opportunities •continually listen •collaborate •adapt •be generous •be a great academic Don’t •focus only on home runs •put the research ahead of the people •think innovation only happens in academia •think scale doesn’t matter •think open source is enough •think the message doesn’t matter At the start of the talk…
  • 46.
    All in termsof “Do”s •understand and use the real problem as a beacon •be open to opportunities •continually listen •collaborate •adapt •be generous •be a great academic •focus on the problem at hand •see the value in people in your network •follow innovations in industry •consider whether scale matters •realize open source requires effort •spend time on the message
  • 47.
    Impact can takemany forms
 
 Gail C. Murphy
 University of British Columbia
 Tasktop Technologies Inc. @gail_murphy Focus on the problem Be open to possibilities Think outside the box Do research with integrity