CosTriage is a cost-aware algorithm for bug triage in bug reporting systems. It improves upon previous approaches by considering both accuracy and cost (expected bug fix time) when assigning bugs to developers. It uses a topic modeling technique to categorize bugs into types to address the sparseness problem in past bug fix history data. It then constructs developer profiles based on estimated bug fix times for different bug types. A hybrid recommender system merges scores from a classifier and the cost profiles to recommend bug assignments that optimize both accuracy and cost, improving expected fix time by up to 30% without sacrificing much accuracy.

1. AAAI 2011
CosTriage: A Cost-Aware Algorithm for
Information & Database Systems Lab
Bug Reporting Systems
Jin-woo Park1, Mu-Woong Lee1, Jinhan Kim1, Seung-won Hwang1, Sunghun Kim2
POSTECH, Korea, Republic of1
HKUST, Hong Kong2

2. Bug reporting systems
 Bugs!!
 More than 300 bug reports per day in Mozilla (a big software
project)
Information & Database Systems Lab
 Bug Solving
 One of the important issues in a software development process
 Bug reports are posted, discussed, and assigned to developers
 Open sources projects
 Apache
 Eclipse
 Linux kernel
 Mozilla

3. Bug reporting systems
 Bug reports
 Has Bug ID, title, description, status, and other meta data
 Assigned to developers
 Fixed by developers
Information & Database Systems Lab
 Challenges
 Bug triage
 Duplicate bug detection
 …
bug ID
title (summary)
status
bug fix history (time)
other data
description

4. Bug reporting systems
 Bug Triage
 Assigning a new bug report to a suitable developer
 Bottleneck of bug fixing process
 Labor intensive
Information & Database Systems Lab
 Miss-assignment can lead to slow bug fix
Bottleneck of
the bug fixing
process
assign
Open
Source Bug
Project Reports
Triager
Developers
Can be
automated!!!

5. Preliminary (recommendation)
 Recommender algorithms
 Content-based recommendation (CBR)
 Predicting user’s interests based on item features
 Machine learning methods
Information & Database Systems Lab
 Over-specialization problem
 Collaborative filtering recommendation (CF)
 Predicting user’s interests based on affinity’s interests for items
 User neighborhood
 Sparsity problem
 Hybrid recommendation
 Content-boosted collaborative filtering (CBCF)
 Combining an existing CBR with a CF
 Better performance than either approach alone

6. Preliminary (recommendation)
 Recommender algorithms
 Content-based recommendation (CBR)
 Predict user’s interests based on item features
 Title, Status, Description, …
Information & Database Systems Lab
 Learn features using machine learning methods
 Recommend similar items
 Over-specialization problem
Developers
Bug
Report 4 Title Bug 1 Bug 2 Bug 3 Bug 4
Dev 1 10 9 1 ?
Feature word1 word2 word3 Dev 2 6 5 10 ?
Count 1 1 4 Dev 3 8 7 7 ?
Rating score table

7. Preliminary (recommendation)
 Recommender algorithms
 Content-based recommendation (CBR)
 Predict user’s interests based on item features
 Title, Status, Description, …
Information & Database Systems Lab
 Learn features using machine learning methods
 Recommend similar items
 Over-specialization problem
Developers
Bug
Report 4 Title Bug 1 Bug 2 Bug 3 Bug 4
Dev 1 10 9 1 9
Feature word1 word2 word3 Dev 2 6 5 10 5
Count 1 1 4 Dev 3 8 7 7 7
Rating score table

8. Preliminary (recommendation)
 Recommender algorithms
 Collaborative filtering recommendation (CF)
 Predicting user’s interests based on affinity’s interests for items
 User neighborhood
Information & Database Systems Lab
 Sparsity problem
Bug
Reports
Bug 1 Bug 2 Bug 3
Developer 1 10 5 10
Developer 2 5 10 6
Developer 3 7 7 7
Developer 4 9 5 ?

9. Preliminary (recommendation)
 Recommender algorithms
 Collaborative filtering recommendation (CF)
 Predicting user’s interests based on affinity’s interests for items
 User neighborhood
Information & Database Systems Lab
 Sparsity problem
Bug
Reports
Bug 1 Bug 2 Bug 3
Developer 1 10 5 10
Developer 2 5 10 6
Developer 3 7 7 7
Developer 4 9 5 ?

10. Preliminary (recommendation)
 Recommender algorithms
 Collaborative filtering recommendation (CF)
 Predicting user’s interests based on affinity’s interests for items
 User neighborhood
Information & Database Systems Lab
 Sparsity problem
Bug
Reports
Bug 1 Bug 2 Bug 3
Developer 1 10 5 10
Developer 2 5 10 6
Developer 3 7 7 7
Developer 4 9 5 9

11. Preliminary (recommendation)
 Recommender algorithms
 Collaborative filtering recommendation (CF)
 Predicting user’s interests based on affinity’s interests for items
 User neighborhood
Information & Database Systems Lab
 Sparsity problem
Unsuitable for triaging because:
Bug
 No one solved new bug!!
Report 4
Bug 1 Bug 2 Bug 3 Bug 4
Developer 1 10 5 10 ?
Developer 2 5 10 6 ?
Developer 3 7 7 7 ?
Developer 4 9 5 9 ?

12. Preliminary (recommendation)
 Recommender algorithms
 Collaborative filtering recommendation (CF)
 Predicting user’s interests based on affinity’s interests for items
 User neighborhood
Information & Database Systems Lab
 Sparsity problem
Unsuitable for triaging because:
Bug
 Rating is extremely sparse!!
Report 4
Bug 1 Bug 2 Bug 3 Bug 4
Developer 1 10 5 ? ?
Developer 2 ? ? ? ?
Developer 3 ? ? 10 ?
Developer 4 ? ? ? 3

13. Preliminary (recommendation)
 Recommender algorithms
 Hybrid recommendation
 Content-boosted collaborative filtering (CBCF)
 Combining an existing CBR with a CF
Information & Database Systems Lab
 Better performance than either approach alone
Two Phases
Bug
- CBR phase Feature word1 word2 word3
Report 5
- CF Phase Count 3 2 4
Bug 1 Bug 2 Bug 3 Bug 4 Bug 5
Dev 1 10 ? ? ? ?
Dev 2 ? 8 3 ? ?
Dev 3 ? ? ? 7 ?

14. Preliminary (recommendation)
 Recommender algorithms
 Hybrid recommendation
 Content-boosted collaborative filtering (CBCF)
 Combining an existing CBR with a CF
Information & Database Systems Lab
 Better performance than either approach alone
CBR phase
Bug
Report 5
Bug 1 Bug 2 Bug 3 Bug 4 Bug 5
Dev 1 10 10 10 10 10
Dev 2 3 8 3 8 3
Dev 3 7 7 7 7 7

15. Preliminary (recommendation)
 Recommender algorithms
 Hybrid recommendation
 Content-boosted collaborative filtering (CBCF)
 Combining an existing CBR with a CF
Information & Database Systems Lab
 Improving CBR/CF by combining the complementary strength of CBR
and CF
CF phase
Bug
Report 5
Bug 1 Bug 2 Bug 3 Bug 4 Bug 5
Dev 1 10 9 7 9 8
Dev 2 5 8 3 8 5
Dev 3 7 8 5 7 6
Existing recommendation approaches are not suitable!

16. Preliminary (Bug triage)
 PureCBR [Anvik06]
 Construct multi-class classifier using a SVM classifier
 Bug reports B are converted into pair <F(B), D> for training
 Bug Report History  Input Data
Information & Database Systems Lab
 Developers  Classes
F(B) is the feature vector indicating the counts of the keyword w of description of B
D is the developer who fixed B (class)
Bug Fix
History
training
Assign a new bug to Dev 1
New Bug
Report
Classifier’s scores

17. Preliminary (Bug triage)
 PureCBR [Anvik06]
 Good performances
 Problem
 This approach only considers accuracy
Information & Database Systems Lab
 Many bugs  One super-developer
 Over-specialization problem
 Are developer happy?
 We consider developer’s cost (e.g., interests, bug fix time, and
expertise)
 We assume that faster bug fixing time has higher developer cost
Bug Bug
Report 1 Report 2
50 days 2 days

18. Goal
 Goal
 Find efficient bug-developer matching
 Optimizing not only accuracy but also cost
 Use modified CBCF approach
Information & Database Systems Lab
 Constructing developer profiles for cost
 Challenge
 Enhancing CBCF approach for sparse data
 Extreme sparseness of the past bug fix history data
 A bug fixed by a developer
 Need to reduce sparseness for enhancing quality of CBCF
Bug fix time from bug fix history

19. Overview
Cost
Developer profiles Cost score
Information & Database Systems Lab
Recommended
New Bug Aggregation
Developer
Report Accuracy
Bug classifier Accuracy score
<SVM>
 Merging classifier’s scores and developer’s cost scores.
 The accuracy scores are obtained using PureCBR [Anvik06]
 The developer cost scores are obtained from “de-sparsified” bug
fix history.
 Two scores are then merged for prediction
Assign a new bug to Dev 1
+ =
Accuracy scores Cost scores Hybrid scores

20. CosTriage (Cost Estimation)
 CosTriage: A Cost-aware Triage Algorithm for bug reporting
system
 Challenge to estimate the developer cost?
 How to reduce the sparseness problem?
Information & Database Systems Lab
 Using a Topic Modeling
Bug fix time from bug fix history
Categorization bugs to reduce the sparseness

21. CosTriage
 Categorizing bugs
 Topic Modeling
 Latent Dirichlet Allocation (LDA) [BleiNg03]
 Each topic is represented as a bug type
Information & Database Systems Lab
 The topic distribution of reports determine bug types
 We adopt the divergence measure proposed in [Arun, R. PAKDD ‘10]
 Finding the natural number of topics (# bug types)
t is the natural number of bug types

22. CosTriage
 Developer profiles modeling
 Developer Profiles
 N-dimensional feature vector
 The element of developer profiles, Pu[i], denotes the developer cost
Information & Database Systems Lab
for ith-type bugs
T denotes the number of bug types
 Developer Cost
 The average time to fix ith type bugs

23. CosTriage
 Predicting missing values in profiles
 Using CF for developer profiles
 Similarity measure:
Information & Database Systems Lab
k=1

24. CosTriage
 Obtaining developer’s cost for a new bug report
Bug type = 1
Information & Database Systems Lab
New Bug
Report
Developer cost for a new bug

25. Merging
Cost
Developer profiles Cost score
<Bug types>
Information & Database Systems Lab
Recommended
Bug Aggregation
Reports Developer
Accuracy
Bug classifier
<SVM> Accuracy score
 Merging classifier’s scores and developer’s cost scores.
+ =
Accuracy scores Cost scores (CosTriage) Hybrid scores
[Anvik06]

26. Experiments
 Subject Systems
 97,910 valid bug reports
 255 active developers
 From four open source projects
Information & Database Systems Lab
 Approaches
 PureCBR: State of the art CBR-based approach
 CBCF: Original CBCF
 CosTraige: Our approach

27. Experiments
 Two research questions
Q1. How much can our approach improve cost (bug fix time) without
sacrificing bug assignment accuracy?
Q2. What are the trade-offs between accuracy and cost (bug fix
Information & Database Systems Lab
time)?
 Evaluation measures
W is the set of bug reports predicted correctly.
N is the number of bug reports in the test set.
 The real fix time is unknown, we only use the fix time for correctly
matched bugs.

28. Experiments
 Relative errors of expected bug fix time
Information & Database Systems Lab
 Improvement of bug fix time (Q1)
 CosTriage improves the costs efficiently up to 30% without
seriously compromising accuracy

29. Experiments
 Trade-off between accuracy and bug fix time (Q2)
Information & Database Systems Lab

30. Conclusion
 We proposed a new bug triaging technique
 Optimize not only accuracy but also cost
 Solve data sparseness problem by using topic modeling
Information & Database Systems Lab
 Experiments using four real bug report corpora
 Improve the cost without heavy losses of accuracy

31. Q&A
Information & Database Systems Lab
Thank you!
Do you have any questions?

32. Contact
Information & Database Systems Lab
Jin-woo Park
jwpark85@postech.ac.kr

33. Back up
 We adopt the divergence measure proposed in [Arun10]
 Finding the natural number of topics (# bug types)
Information & Database Systems Lab
t is the natural number of bug types
Mozilla Apache

34. Back up - Bug features
 Bug features
 Keywords of title and description
 Other meta data
Information & Database Systems Lab
Title : Traditional Memory Rendering refactoring request
New Bug Description : Request additional refactoring so we can
…
Report Traditional Rendering.
Remove stopwords
Traditional Memory Rendering refactoring request Request
refactoring … Traditional Rendering