1. Pragmatic Prioritization of Software
Quality Assurance Efforts
Emad Shihab
Queen’s University
1

2. Motivation
Software ^ has bugs
2
How to allocate quality assurance
efforts for legacy and new code?
Q:
managers ^ have limited resources

3. One Answer: Defect Prediction
Many recent studies focused on defect
prediction
3
….However adoption in practice remains low

4. An Industrial Example
How to prioritize the creation of unit
tests for a large legacy system
4

5. Criteria for Pragmatic QA Approaches
Focused Granularity
Unit of
prioritization
must be focused
(e.g., function or
method)
5
Current Approach Desired Approach
File, Folder or
subsystem level
(Thousands of
LOC)

6. Criteria for Pragmatic QA Approaches
Responsive Feedback
Continuous
reflection of
system changes.
6
Current Approach Desired Approach
Predict for next
release

7. Criteria for Pragmatic QA Approaches
Consider Effort
Consider effort in
recommendations
(e.g., use size)
7
Current Approach Desired Approach
No consideration
for effort
In addition, need to evaluate the generality of the approaches
to understand implications of findings

8. Proposed Pragmatic SQA Approaches
8
Unit test creation
How to allocate quality assurance
efforts for legacy code?
Q:
A:
Change risk
How to allocate quality assurance
efforts for new code?
Q:
A:

9. Unit Test Creation
Approach
1. Search modification record comments for keywords,
bug identifiers and modified files
2. Extract source code of modified file(s) and compare
to previous version to identify changed functions
3. Combine data from 1 and 2 to calculate
function-level heuristics
9

10. Unit Test Creation
Heuristics
Most Frequently Modified (MFM)
Most Recently Modified (MRM)
Most Frequently Fixed (MFF)
Most Recently Fixed (MRF)
Largest Fixed (LF)
Largest Modified (LM)
Change Risk (CR)
Size Risk (SR)
Random
Modification
Fix
Size
Risk
Random
10

11. Unit Test Creation
Heuristic Performance
Based on a heuristic, generate list of 10
functions to write unit tests for
Use size of function to measure effort
required to write unit test
11

12. Unit Test Creation
Usefulness: Was writing the unit test useful?
Time to write unit test
A
B
C
6 bug fixes
2 bug fixes
0 bug fixes
Usefulness = 2/3 = 66.67% 12

13. Unit Test Creation
Usefulness of Heuristics
27.7
43.1
48.8
55
56.9
80
83.8
84.7
87
8.5
9.9
12.6
17.4
16
28.1
32.3
32.9
44.7
0 10 20 30 40 50 60 70 80 90 100
Random
Most Recently Modified (MRM)
Size Risk (SR)
Change Risk (CR)
Most Recently Fixed (MRF)
Most Frequently Modified (MFM)
Most Frequently Fixed (MFF)
Largest Modified (LM)
Largest Fixed (LF)
Open Source
Commercial
13

14. Revisiting the Pragmatic QA Criteria
Unit Test Creation
Unit Test Creation
Function level
recommendations
Heuristic lists are
continuously updated
Size of function used to
consider effort required
Focused Granularity
Consider Effort
Responsive Feedback
Evaluate Generality: Studied Open Source and Commercial systems

15. Change Risk
Proposed Approach
1. Mine changes and extract change metrics
2. Identify bug-inducing changes
3. Predict risky changes
15

16. Revisiting the Pragmatic QA Criteria
Change Risk
Change Risk
Map changes to modified
functions
Flag risky changes while
they are fresh in
developer’s minds
Size of change used to
consider effort
Focused Granularity
Consider Effort
Responsive Feedback
Evaluate Generality: Study on Open Source and Commercial systems

17. Thank you
I am grateful for this opportunity!

18. Backups

19. Revisiting the Pragmatic SQA Criteria
Unit Test Creation Change Risk
Focused Granularity
Estimate Effort
Evaluate Generality
Timely Feedback

20. Pragmatic SQA Criteria
What we have What we want
File level Unit of prioritization
must be focused (e.g.,
function)
Act on results in
future release
Must be able to act on
results of approach
quickly
No effort provided Provide an estimate of
the effort required for
the task
Focused Granularity
Estimate Effort
Timely Feedback
Evaluate
Generality
To help
understand
implications
of findings

21. Changes Risk
New code
Identify risky changes for closer examination before they
are integrated into the code base
21

22. Unit Test Creation
Implemented code
Prioritize the creation of unit tests for
implemented code
22

23. Revisiting the Pragmatic-aware
Criteria
Focused Granularity Timely feedback Estimate Effort Evaluate Generality
23

24. Avoid the most bugs effectively!
Write unit tests for functions with best
Return on Investment (ROI)
How can we avoid the most
bugs given limited resources?
24

25. Defect Prediction... An Example
Uses code metrics and pre-release defects to
predict files with post-release defects in the
Eclipse project.
25

26. Motivation
Software has^ bugs and
managers have ^ limited resources
26
How to allocate quality assurance
efforts for implemented and new
code?
Q:
A: Re-active and pro-active SQA

27. Proposed Approach
Extracting Historical Data
1. Extract change data to calculate metrics (e.g., churn)
2. Use SZZ to identify and predict bug-inducing changes
3.
27

28. Answer … Defect Prediction!
Use code metrics and process metrics
to predict which files are most likely to
contain defects in future releases
28

29. Defect Prediction... The Response
“The results are interesting, but of limited use to us ...”
... Why?
29

30. Defect Prediction... Lessons Learned
1. Identifying files is too coarse of a granularity
2. Having to wait till the next release is too late
3. Need an estimate of effort required
4. Why should we believe this will work for us?
30

31. Criteria for Pragmatic SQA
Approaches
Timely Feedback
Must be able to act on
results of approach quickly
31

32. Criteria for Pragmatic SQA
Approaches
Estimate Effort
Provide an estimate of
the effort required for
the task
32

33. Criteria for Pragmatic SQA
Approaches
Evaluate Generality
Evaluate on multiple
projects/domains to clearly
understand implications of
the findings
33