presentation of a research paper

1. Improving the ROI of Software Quality
Assurance Activities: An Empirical Study
PRESENTED BY KHUSH BAKHAT

2. Return On Investment
Investment source
Investor benefit

3. IEEE’s Definition of Software Quality Assurance
 “A planned and systematic pattern of all actions necessary
to provide adequate confidence that an item or product
conforms to established technical requirements and a set of
activities designed to evaluate the process by which the
products are developed or manufactured ”

4. Quality Assurance Activities

5. Context
Series Of
Incremental
Projects In
China
3 Types Of
QA
Activities
Empirical
Findings Of
Effort
Distribution
Pattern

6. Results
The result of the study gives implications on:
How to identify which type of QA activity is insufficient
How to balance the effort allocation and planning for future projects
How to improve the weak part of each QA activity and finally improve
the return on investment (ROI) of QA activities and the whole process
effectiveness under the specific organization context

7. Related Work
Related Work

8. Background and Data Source of the Study
The empirical study is based on the process
data from developing a series of incremental
R&D software projects within a medium-
sized software organization in China

9. Project Quality Goal
Every incremental version of this project shares the
same quality goal before releasing:
Rate of requirements coverage by test cases is
100%
All planned test cases executed and passed
No non-trivial defects are detected during at
least one day of continuous testing
Satisfy the quality goal of 0.2defects/KLOC
when it is released

10. Objective
Empirical assessment
• Effort distribution patterns of QA activities
Develop understanding
• QA activities during software life cycle
Provide guidelines and process implication
• Planning and balancing the effort distribution
• Maximize the ROI of QA activities

11. Defects of specification, missing
portions, developer blind spots are
easier to be found by review
Defects of numerical approximations
and program dynamics are harder to
be found by review
Reviewers can’t tell how fast and
reliable the system will be
how user-friendly the system is by
just reviewing
Testing serves as a hands-on
experience of the actual and
operational system that review can’t
achieve by imaging
It is good at finding numerical
approximations and program
dynamics defects but harder to find
defects of specification, missing
portions, and developer blind spots
REVIEW
TESTING

12. Process Audit
Process audit has an indirect impact on the software
product
Adopt and tailor the most effective method that comes
from the best industrial practices to the specific context of
the project

13. Return on Investment
Investments to the software development life cycle is an important and
challenging issue for project planning
Balance the effort allocation among QA Activities to improve resource
allocation and maximize the return on investment (ROI) of the entire quality
assurance

14. Methodology
This empirical study follows the Goal/Question/Metric
(GQM) methodology

15. GQM Model
The GQM model used in this study is shown as follows:
• Goal: Improve the ROI of QA activities
 Question 1(Q1): What is the baseline of QA activities effort
allocation under the specific organization context?
 Metric1(M1):Statistic description: Mean, Median, Standard
Deviation of Effort Distribution Percentage

16. Question 1 and Metric

17. Goal/Question/Metric (GQM)
Question 2(Q2): How to identify which QA activities might be insufficient
while others might be overdone?
 Metric2(M2):Regression Diagnostic Model
Metric3(M3):Cost of Finding and Removing Defects in Each Phase
 Metric4(M4):Distribution of Types of Defects Found by Testing

18. Question 2 and Metric 2, 3, 4
Metric 2: Regression Diagnostic Model

19. Testing Is Inversely Proportional To Review & Audit

20. Multivariable Linear Regression Analysis

21. Process Implication
For this case study, testing is the most labor-intensive activities of
the QA activities
For this software organization, improving the process audit activity
might be more cost-effective

22. Metric 3
Compare the requirement, design and testing phase’s cost of finding and
removing defects

23. Process Implication
Findings from this analysis support that process audit and
reviewing are more effective than testing

24. Metric 4: Distribution of Types of
Defects Found by Testing

25. Goal/Question/Metric (GQM)
Question 3(Q3): How to improve reviewing if it is
insufficient?
 Metric3(M5): Filter Effectiveness

26. Question 3 and Metric 5
If put the extra effort on the weakest part would be more cost-
effective
The definition the Filter Effectiveness (FE) of each phase:

27. Process Implication

28. GQM
Question 4(Q4): How to improve process audit if it is insufficient?
 Metric4(M6): NCs (Not Compatibles) Distribution
Question 5(Q5): How to improve testing if it is insufficient?
 Metric5(M7):Value-based Software Testing: Feature Prioritization

30. Periodical Reports
Main items of periodical reports include task completion ratio,
planned and actual effort, problems or risks they encounter.
 Periodical reports are usually used by project manager to monitor
developers’ task progress, identify risks and issues.

31. Task Estimation
Overestimating the task effort might lead to project delay, shorten
the time for testing and lead to low quality product,
 Underestimates might lead to resource waste
For estimation its better to adopt systematic and professional effort
estimation methods or tools, such as COCOMO tool

32. Process Implication
Analysis help organization to maintain their process maturity on a higher level
Continuous process improvement by putting more effort on process audit and
improving the process of Project Planning and Project Monitoring and Control
In future it would benefit for process improvement and achieve a higher ROI.

33. Threats to Validity
• Oversimplified assumptions of Regression Model
The assumption of regression equation is that the utility function of each QA
activity is linear
The utility function of QA activities is not simply linear in real life, usually it
comes out as an “S” shape production function

34. Data Acquisition for other organizations
In this case study, analysis based on a series of maintenance and evolution
projects with the same process maturity level and quality goals.
This assumption limits its adoption by only those organizations that collect and
have the required process data

35. Data Quantity and Soundness of Statistical Analysis
In this case study, the data is very limited with missing which is a threat to
analysis’s validity
Study assume that all releases are the same
 In reality, different releases may involve work of different nature, requiring
different types of QA activities, and different effort distribution patterns

36. Data Variance Analysis
In this case study, the author’s deal with averages without further
analysis for its variance, e.g.:
Baseline of required QA effort percentage in table 1
 Cost of finding and removing defects in table 4
 Filter effectiveness of each phase by reviewing in table 5

37. Conclusion
The results from this study support that early review is more effective than late
testing
This study will effective to prevent the defect introduction; these three types of
QA activities complement each other to achieve the required quality goal

39. Future Work
Future work would include adopting value-based principles to each
QA activity to improve its effectiveness
Expand the basis of this approach to find a way to balance the effort
distribution of QA activities to maximize its ROI.