Expert Recommendation with Usage Expertise

2,364 views
2,306 views

Published on

Presented at ICSM 2009

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
2,364
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Expert Recommendation with Usage Expertise

  1. 1. Expert Recommendation with Usage Expertise David Ma David Schuler University of Calgary Saarland University Tom Zimmermann Jonathan Sillito Microsoft Research University of Calgary 1
  2. 2. Stuck... AutoBuildJob(Workspace workspace) { super(Messages.events_building_0); setRule(workspace.getRoot()); setPriority(BUILD); isAutoBuilding = workspace.isAutoBuilding(); preferences.addPropertyChangeListener(this); } 2
  3. 3. Who knows how this code works? AutoBuildJob(Workspace workspace) { super(Messages.events_building_0); setRule(workspace.getRoot()); setPriority(BUILD); isAutoBuilding = workspace.isAutoBuilding(); preferences.addPropertyChangeListener(this); } 3
  4. 4. Who knows how this code works? Who has modified this code before? 4
  5. 5. AutoBuildJob(Workspace workspace) { super(Messages.events_building_0); setRule(workspace.getRoot()); setPriority(BUILD); isAutoBuilding = workspace.isAutoBuilding(); preferences.addPropertyChangeListener(this); } 5
  6. 6. AutoBuildJob(Workspace workspace) { super(Messages.events_building_0); setRule(workspace.getRoot()); setPriority(BUILD); isAutoBuilding = workspace.isAutoBuilding(); preferences.addPropertyChangeListener(this); } Anna changes AutoBuildJob. She knows how this method works. 6
  7. 7. Implementation Expertise AutoBuildJob(Workspace workspace) { super(Messages.events_building_0); setRule(workspace.getRoot()); setPriority(BUILD); isAutoBuilding = workspace.isAutoBuilding(); preferences.addPropertyChangeListener(this); } Anna changes AutoBuildJob. She knows how this method works. 7
  8. 8. Implementation Expertise Projects unlikely to share code This concept can’t transfer expertise across projects 8
  9. 9. Developers moving between multiple projects or that are new to projects Implementation Expertise New projects don’t have data Some projects don’t have enough data or is of poor quality 9
  10. 10. 10
  11. 11. Usage Expertise AutoBuildJob(Workspace workspace) { super(Messages.events_building_0); setRule(workspace.getRoot()); setPriority(BUILD); isAutoBuilding = workspace.isAutoBuilding(); preferences.addPropertyChangeListener(this); } Matt adds a call to setPriority(). Matt knows how to use setPriority() without knowing how it was coded. 11
  12. 12. Usage Expertise and Context AutoBuildJob(Workspace workspace) { super(Messages.events_building_0); setRule(workspace.getRoot()); setPriority(BUILD); isAutoBuilding = workspace.isAutoBuilding(); preferences.addPropertyChangeListener(this); } Matt adds a call to setPriority(). Matt knows how to use setPriority() and the surrounding method calls. 12
  13. 13. Why Usage Expertise? Projects unlikely to share code Projects likely to share calls to libraries 13
  14. 14. How Do We Make Recommendations? 14
  15. 15. Expertise Profiles Database of Profiles Mine developer history for Bob’s Profile Changed Methods and Added Method Calls Dan’s Profile Alice’s Profile 15
  16. 16. Looking for Experts Who has Database of Profiles Implementation Expertise? AutoBuildJob(Workspace workspace) { Bob’s Profile super(Messages.events_building_0); setRule(workspace.getRoot()); setPriority(BUILD); isAutoBuilding = workspace.isAutoBuilding(); preferences.addPropertyChangeListener(this); } Dan’s Profile Alice’s Profile Who has Usage Expertise? 16
  17. 17. Ranking Implementation Expertise (How previous research has done it) Change Change Recency Frequency 17
  18. 18. Ranking Usage Expertise Frequency of Breadth of Usage Usage Frequency of Breadth of Usage Relative Usage Relative to Everyone Else to Everyone Else 18
  19. 19. Research Questions Can Usage Expertise correctly 1 recommend experts within projects as often as Implementation Expertise? Can Usage Expertise + Context correctly 2 recommend experts within projects as often as Implementation Expertise? Is it possible to recommend developers 3 across projects? 19
  20. 20. Experiment 1 of 2 Recommending Within Projects accuracy of implementation-based recommendations vs. accuracy usage-based recommendations 20
  21. 21. Training Database of Profiles Bob’s Profile CVS history Dan’s Profile Alice’s Profile 21
  22. 22. Testing Database of Changeset Ranked by Joe Profiles impl. expertise Matt Changed Bob (expert) + Methods Ranked by Added usage expertise Method Kim Calls Bob (expert) Joe Evaluation Did we recommend the actual expert? 22
  23. 23. Experiment 2 of 2 Recommending Across Projects Determining if usage-based recommendations are possible across projects 23
  24. 24. Training Database of Profiles Eclipse CVS history Bob’s Profile Jack’s Profile AspectJ CVS history Dan’s Profile Alice’s Profile Kate’s Profile 24
  25. 25. Testing Database of Changeset Profiles Changed Ranked by usage expertise + Methods Alice (Eclipse) Added Jake (Eclipse) Method Kim (AspectJ) Calls Evaluation Did we recommend devs. from other projects? 25
  26. 26. Results (Within Projects) Impl-based recommendations are correct ~80% of time time while usage based are correct ~50% of the time. 26
  27. 27. Can Usage Expertise correctly 1 recommend experts within projects as often as Implementation Expertise? No 27
  28. 28. Results (Within Projects) Usage Expertise + Context correctly recommends experts just as often as Implementation Expertise 28
  29. 29. Can Usage Expertise + Context correctly 2 recommend experts within projects as often as Implementation Expertise? YES 29
  30. 30. Results (Across Projects) Our usage expertise heuristics recommend developers 25-75% of the time from other projects. 30
  31. 31. Is it possible to recommend developers 3 across projects? YES 31
  32. 32. Is it possible to recommend developers 3 across projects? YES BUT 32
  33. 33. Open Questions How precise are recommendations made across projects? Can we recommend for projects with no or little history? What if we combine both implementation and usage expertise? 33
  34. 34. Research Questions Can Usage Expertise correctly recommend experts within projects as often as Implementation Expertise? No Can Usage Expertise + Context correctly recommend experts within projects as often as Implementation Expertise? YES Is it possible to recommend developers across projects? YES 34

×