Expert Recommendation
with Usage Expertise
David Ma David Schuler
University of Calgary Saarland University
Tom Zimmermann Jonathan Sillito
Microsoft Research University of Calgary
1

Stuck...
AutoBuildJob(Workspace workspace) {
super(Messages.events_building_0);
setRule(workspace.getRoot());
setPriority(BUILD);
isAutoBuilding = workspace.isAutoBuilding();
preferences.addPropertyChangeListener(this);
}
2

Who knows how
this code works?
AutoBuildJob(Workspace workspace) {
super(Messages.events_building_0);
setRule(workspace.getRoot());
setPriority(BUILD);
isAutoBuilding = workspace.isAutoBuilding();
preferences.addPropertyChangeListener(this);
}
3

Who knows how
this code works?
Who has modified
this code before?
4

AutoBuildJob(Workspace workspace) {
super(Messages.events_building_0);
setRule(workspace.getRoot());
setPriority(BUILD);
isAutoBuilding = workspace.isAutoBuilding();
preferences.addPropertyChangeListener(this);
}
5

AutoBuildJob(Workspace workspace) {
super(Messages.events_building_0);
setRule(workspace.getRoot());
setPriority(BUILD);
isAutoBuilding = workspace.isAutoBuilding();
preferences.addPropertyChangeListener(this);
}
Anna changes AutoBuildJob.
She knows how this method works.
6

Implementation
Expertise
AutoBuildJob(Workspace workspace) {
super(Messages.events_building_0);
setRule(workspace.getRoot());
setPriority(BUILD);
isAutoBuilding = workspace.isAutoBuilding();
preferences.addPropertyChangeListener(this);
}
Anna changes AutoBuildJob.
She knows how this method works.
7

Implementation
Expertise
Projects unlikely to share code
This concept can’t transfer
expertise across projects
8

Developers moving between
multiple projects or that are
new to projects
Implementation
Expertise
New projects
don’t have data
Some projects don’t have
enough data or is of poor
quality
9

10

Usage
Expertise
AutoBuildJob(Workspace workspace) {
super(Messages.events_building_0);
setRule(workspace.getRoot());
setPriority(BUILD);
isAutoBuilding = workspace.isAutoBuilding();
preferences.addPropertyChangeListener(this);
}
Matt adds a call to setPriority().
Matt knows how to use setPriority()
without knowing how it was coded.
11

Usage
Expertise and Context
AutoBuildJob(Workspace workspace) {
super(Messages.events_building_0);
setRule(workspace.getRoot());
setPriority(BUILD);
isAutoBuilding = workspace.isAutoBuilding();
preferences.addPropertyChangeListener(this);
}
Matt adds a call to setPriority().
Matt knows how to use setPriority()
and the surrounding method calls.
12

Why
Usage Expertise?
Projects unlikely to share code
Projects likely to share calls to libraries
13

How Do We Make
Recommendations?
14

Expertise Profiles
Database of Profiles
Mine developer
history for
Bob’s Profile
Changed Methods
and
Added Method Calls
Dan’s Profile
Alice’s Profile
15

Looking for Experts
Who has Database of Profiles
Implementation
Expertise?
AutoBuildJob(Workspace workspace) { Bob’s Profile
super(Messages.events_building_0);
setRule(workspace.getRoot());
setPriority(BUILD);
isAutoBuilding = workspace.isAutoBuilding();
preferences.addPropertyChangeListener(this);
}
Dan’s Profile
Alice’s Profile
Who has
Usage Expertise?
16

Ranking Implementation Expertise
(How previous research has done it)
Change
Change Recency
Frequency
17

Ranking Usage Expertise
Frequency of Breadth of
Usage Usage
Frequency of Breadth of
Usage Relative Usage Relative
to Everyone Else to Everyone Else
18

Research Questions
Can Usage Expertise correctly
1 recommend experts within projects as
often as Implementation Expertise?
Can Usage Expertise + Context correctly
2 recommend experts within projects as
often as Implementation Expertise?
Is it possible to recommend developers
3 across projects?
19

Experiment 1 of 2
Recommending Within Projects
accuracy of implementation-based recommendations
vs. accuracy usage-based recommendations
20

Training
Database of Profiles
Bob’s Profile
CVS history Dan’s Profile
Alice’s Profile
21

Testing
Database of Changeset Ranked by Joe
Profiles impl. expertise
Matt
Changed Bob (expert)
+
Methods
Ranked by
Added usage expertise
Method
Kim
Calls Bob (expert)
Joe
Evaluation
Did we recommend the actual expert?
22

Experiment 2 of 2
Recommending Across Projects
Determining if usage-based recommendations
are possible across projects
23

Training
Database of Profiles
Eclipse CVS history Bob’s Profile
Jack’s Profile
AspectJ CVS history Dan’s Profile
Alice’s Profile
Kate’s Profile
24

Testing
Database of Changeset
Profiles
Changed Ranked by
usage expertise
+
Methods
Alice (Eclipse)
Added Jake (Eclipse)
Method Kim (AspectJ)
Calls
Evaluation
Did we recommend devs. from other projects?
25

Results
(Within Projects)
Impl-based recommendations are
correct ~80% of time time while usage
based are correct ~50% of the time.
26

Can Usage Expertise correctly
1 recommend experts within projects as
often as Implementation Expertise?
No
27

Results
(Within Projects)
Usage Expertise + Context correctly
recommends experts just as often as
Implementation Expertise
28

Can Usage Expertise + Context correctly
2 recommend experts within projects as
often as Implementation Expertise?
YES
29

Results
(Across Projects)
Our usage expertise heuristics
recommend developers 25-75% of
the time from other projects.
30

Is it possible to recommend developers
3 across projects?
YES
31

Is it possible to recommend developers
3 across projects?
YES
BUT
32

Open Questions
How precise are recommendations
made across projects?
Can we recommend for projects with
no or little history?
What if we combine both
implementation and usage expertise?
33

Research Questions
Can Usage Expertise correctly
recommend experts within projects as
often as Implementation Expertise?
No
Can Usage Expertise + Context correctly
recommend experts within projects as
often as Implementation Expertise?
YES
Is it possible to recommend developers
across projects? YES
34