Your SlideShare is downloading. ×
Expert Recommendation
 with    Usage Expertise

             David Ma                 David Schuler
        University of ...
Stuck...

AutoBuildJob(Workspace workspace) {
	 super(Messages.events_building_0);
	 setRule(workspace.getRoot());
	 setPr...
Who knows how
this code works?
AutoBuildJob(Workspace workspace) {
	 super(Messages.events_building_0);
	 setRule(workspac...
Who knows how
this code works?



      Who has modified
      this code before?


             4
AutoBuildJob(Workspace workspace) {
	 super(Messages.events_building_0);
	 setRule(workspace.getRoot());
	 setPriority(BUI...
AutoBuildJob(Workspace workspace) {
	 super(Messages.events_building_0);
	 setRule(workspace.getRoot());
	 setPriority(BUI...
Implementation
                                                 Expertise
AutoBuildJob(Workspace workspace) {
	 super(Mess...
Implementation
                              Expertise


   Projects unlikely to share code



This concept can’t transfer...
Developers moving between
multiple projects or that are
new to projects

                  Implementation
                ...
10
Usage
                                                 Expertise
AutoBuildJob(Workspace workspace) {
	 super(Messages.even...
Usage
                          Expertise and Context
AutoBuildJob(Workspace workspace) {
	 super(Messages.events_building...
Why
                          Usage Expertise?


Projects unlikely to share code

  Projects likely to share calls to libr...
How Do We Make
Recommendations?




       14
Expertise Profiles
                             Database of Profiles



Mine developer
history for
                        ...
Looking for Experts
               Who has                                Database of Profiles

               Implementati...
Ranking Implementation Expertise

   (How previous research has done it)


         Change
                         Change...
Ranking Usage Expertise


   Frequency of             Breadth of
      Usage                   Usage




   Frequency of  ...
Research Questions
         Can Usage Expertise correctly
1    recommend experts within projects as
      often as Impleme...
Experiment 1 of 2

Recommending Within Projects
 accuracy of implementation-based recommendations
 vs. accuracy usage-base...
Training
                            Database of Profiles




                                   Bob’s Profile




      CVS...
Testing

 Database of       Changeset     Ranked by               Joe
   Profiles                     impl. expertise
     ...
Experiment 2 of 2

Recommending Across Projects
  Determining if usage-based recommendations
  are possible across project...
Training
                                  Database of Profiles




    Eclipse CVS history                         Bob’s P...
Testing

 Database of       Changeset
   Profiles
                   Changed       Ranked by
                              ...
Results
             (Within Projects)

  Impl-based recommendations are
correct ~80% of time time while usage
 based are ...
Can Usage Expertise correctly
1   recommend experts within projects as
     often as Implementation Expertise?




       ...
Results
             (Within Projects)

Usage Expertise + Context correctly
recommends experts just as often as
     Imple...
Can Usage Expertise + Context correctly
2    recommend experts within projects as
      often as Implementation Expertise?...
Results
            (Across Projects)

  Our usage expertise heuristics
recommend developers 25-75% of
   the time from ot...
Is it possible to recommend developers
3                across projects?




           YES
                  31
Is it possible to recommend developers
3                across projects?




          YES
          BUT
                 ...
Open Questions

 How precise are recommendations
 made across projects?

 Can we recommend for projects with
 no or little...
Research Questions
Can Usage Expertise correctly
recommend experts within projects as
often as Implementation Expertise?
 ...
Upcoming SlideShare
Loading in...5
×

Expert Recommendation with Usage Expertise

2,228

Published on

Presented at ICSM 2009

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
2,228
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Expert Recommendation with Usage Expertise"

  1. 1. Expert Recommendation with Usage Expertise David Ma David Schuler University of Calgary Saarland University Tom Zimmermann Jonathan Sillito Microsoft Research University of Calgary 1
  2. 2. Stuck... AutoBuildJob(Workspace workspace) { super(Messages.events_building_0); setRule(workspace.getRoot()); setPriority(BUILD); isAutoBuilding = workspace.isAutoBuilding(); preferences.addPropertyChangeListener(this); } 2
  3. 3. Who knows how this code works? AutoBuildJob(Workspace workspace) { super(Messages.events_building_0); setRule(workspace.getRoot()); setPriority(BUILD); isAutoBuilding = workspace.isAutoBuilding(); preferences.addPropertyChangeListener(this); } 3
  4. 4. Who knows how this code works? Who has modified this code before? 4
  5. 5. AutoBuildJob(Workspace workspace) { super(Messages.events_building_0); setRule(workspace.getRoot()); setPriority(BUILD); isAutoBuilding = workspace.isAutoBuilding(); preferences.addPropertyChangeListener(this); } 5
  6. 6. AutoBuildJob(Workspace workspace) { super(Messages.events_building_0); setRule(workspace.getRoot()); setPriority(BUILD); isAutoBuilding = workspace.isAutoBuilding(); preferences.addPropertyChangeListener(this); } Anna changes AutoBuildJob. She knows how this method works. 6
  7. 7. Implementation Expertise AutoBuildJob(Workspace workspace) { super(Messages.events_building_0); setRule(workspace.getRoot()); setPriority(BUILD); isAutoBuilding = workspace.isAutoBuilding(); preferences.addPropertyChangeListener(this); } Anna changes AutoBuildJob. She knows how this method works. 7
  8. 8. Implementation Expertise Projects unlikely to share code This concept can’t transfer expertise across projects 8
  9. 9. Developers moving between multiple projects or that are new to projects Implementation Expertise New projects don’t have data Some projects don’t have enough data or is of poor quality 9
  10. 10. 10
  11. 11. Usage Expertise AutoBuildJob(Workspace workspace) { super(Messages.events_building_0); setRule(workspace.getRoot()); setPriority(BUILD); isAutoBuilding = workspace.isAutoBuilding(); preferences.addPropertyChangeListener(this); } Matt adds a call to setPriority(). Matt knows how to use setPriority() without knowing how it was coded. 11
  12. 12. Usage Expertise and Context AutoBuildJob(Workspace workspace) { super(Messages.events_building_0); setRule(workspace.getRoot()); setPriority(BUILD); isAutoBuilding = workspace.isAutoBuilding(); preferences.addPropertyChangeListener(this); } Matt adds a call to setPriority(). Matt knows how to use setPriority() and the surrounding method calls. 12
  13. 13. Why Usage Expertise? Projects unlikely to share code Projects likely to share calls to libraries 13
  14. 14. How Do We Make Recommendations? 14
  15. 15. Expertise Profiles Database of Profiles Mine developer history for Bob’s Profile Changed Methods and Added Method Calls Dan’s Profile Alice’s Profile 15
  16. 16. Looking for Experts Who has Database of Profiles Implementation Expertise? AutoBuildJob(Workspace workspace) { Bob’s Profile super(Messages.events_building_0); setRule(workspace.getRoot()); setPriority(BUILD); isAutoBuilding = workspace.isAutoBuilding(); preferences.addPropertyChangeListener(this); } Dan’s Profile Alice’s Profile Who has Usage Expertise? 16
  17. 17. Ranking Implementation Expertise (How previous research has done it) Change Change Recency Frequency 17
  18. 18. Ranking Usage Expertise Frequency of Breadth of Usage Usage Frequency of Breadth of Usage Relative Usage Relative to Everyone Else to Everyone Else 18
  19. 19. Research Questions Can Usage Expertise correctly 1 recommend experts within projects as often as Implementation Expertise? Can Usage Expertise + Context correctly 2 recommend experts within projects as often as Implementation Expertise? Is it possible to recommend developers 3 across projects? 19
  20. 20. Experiment 1 of 2 Recommending Within Projects accuracy of implementation-based recommendations vs. accuracy usage-based recommendations 20
  21. 21. Training Database of Profiles Bob’s Profile CVS history Dan’s Profile Alice’s Profile 21
  22. 22. Testing Database of Changeset Ranked by Joe Profiles impl. expertise Matt Changed Bob (expert) + Methods Ranked by Added usage expertise Method Kim Calls Bob (expert) Joe Evaluation Did we recommend the actual expert? 22
  23. 23. Experiment 2 of 2 Recommending Across Projects Determining if usage-based recommendations are possible across projects 23
  24. 24. Training Database of Profiles Eclipse CVS history Bob’s Profile Jack’s Profile AspectJ CVS history Dan’s Profile Alice’s Profile Kate’s Profile 24
  25. 25. Testing Database of Changeset Profiles Changed Ranked by usage expertise + Methods Alice (Eclipse) Added Jake (Eclipse) Method Kim (AspectJ) Calls Evaluation Did we recommend devs. from other projects? 25
  26. 26. Results (Within Projects) Impl-based recommendations are correct ~80% of time time while usage based are correct ~50% of the time. 26
  27. 27. Can Usage Expertise correctly 1 recommend experts within projects as often as Implementation Expertise? No 27
  28. 28. Results (Within Projects) Usage Expertise + Context correctly recommends experts just as often as Implementation Expertise 28
  29. 29. Can Usage Expertise + Context correctly 2 recommend experts within projects as often as Implementation Expertise? YES 29
  30. 30. Results (Across Projects) Our usage expertise heuristics recommend developers 25-75% of the time from other projects. 30
  31. 31. Is it possible to recommend developers 3 across projects? YES 31
  32. 32. Is it possible to recommend developers 3 across projects? YES BUT 32
  33. 33. Open Questions How precise are recommendations made across projects? Can we recommend for projects with no or little history? What if we combine both implementation and usage expertise? 33
  34. 34. Research Questions Can Usage Expertise correctly recommend experts within projects as often as Implementation Expertise? No Can Usage Expertise + Context correctly recommend experts within projects as often as Implementation Expertise? YES Is it possible to recommend developers across projects? YES 34

×