Paper: Estimating Software Maintenance Effort from Use Cases: an Industrial Case Study Authors:Yan Ku, Jing Du, Ye Yang, Qing Wang Session: Industry Tracking 5: Metrics and Estimation

1. Estimating Software Maintenance
Effort from Use Cases: an
Industrial Case Study
Yan Ku, Jing Du, Ye Yang, Qing Wang
Institute of Software, Chinese Academy of Sciences
2011-09-29
*This work is supported by the National Natural Science Foundation of China under Grant Nos. 90718042, 60873072, 60903050 and
61073044; the National Hi-Tech Research and Development Plan of China under Grant Nos. 2007AA010303, 2007AA01Z186 and
2007AA01Z179.

2. Agenda
Motivation
Industrial Setting
Methodology
Case Study
Discussion
Tool Implementation
2

3. Motivation
An anecdote in software engineering domain*:
Elephant: Discipline
Monkey: Agile
Elephant & Monkey: Practical
*Barry W. Boehm, Richard Turner, Balancing Agility and Discipline: A Guide for the Perplexed, Aug, 2003
*Picture Source: www.image.baidu.com 3

4. Motivation
“Speaking of the problems we are facing during software lifecycle,
hmmm, there is no way a short list. … I hate to, but I have to say that
effort estimation especially for maintenance project, if not totally
impossible, is really a big challenge
-----A Project Manager
Formal estimation models: time-consuming, requiring
enough information and data---- the elephant
Expert Judgment: expert-dependent, easily lead to
thumb-up---- the monkey
*Picture Source: http://www.zcool.com.cn 4

5. Industrial Setting
The industrial research is inspired by the estimation
dilemma a mentioned before.
The problem occurred in developing the leading product
of a medium-size software enterprise with CMMI ML4
in China.
The product, named QONE, is a commercial software
process management tool.
It has contributed to the process improvement for more
than 300 small and medium-sized software companies
and organizations in China.
5

6. Industrial Setting (Con)
Since 2004, QONE has released several major versions as well
as branches for special customized ones in succession.
Several of the evolving versions are maintenance projects.
Expert estimates were mainly used in the past effort estimation
of QONE. Versions Begin Date End Date
The estimation results are not so stable due2004-11-15
v1 2004-10-8 to the objectivity
and other issue. v2 2005-7-11 2005-11-30
The actual effortv3 other data including use case documents
and 2006-1-16 2007-3-30
have been accumulated by QONE itself. 2007-10-31
v4 2007-5-28
v5 2007-12-10 2008-7-31
v6 2008-3-20 2008-8-21
v7 2008-9-1 2009-3-20
6

7. Methodology
Goal: Achieving the balance of simplicity, early-
estimating and accuracy in one effort estimate .
Methodology Principles:
Apply use cases as the size metric and introduce
requirement elaboration factors to make the estimate in
Maintenance task type is not distinguished
advance
due to the difficulty for effort classification
Introduce adjusted factors as few as possible in order to
reduce the complexity
Take advantage of the history data to help improve the
estimation accuracy.
7

8. Modeling Process
Get the lowest level Use the same
requirements data unit
8

9. Count Data
Use case: number of use cases
newUC: new-added
modUC: modified
reuUC: reused without modified
delUC: delete
9

10. Modeling Process
10

11. Count Data & Construct Model
Weight
Wmod/ Wreu / Wdel :effort ratio of
modified/reused/deleted use case to a new-added one
Sizeadjusted = newUC + Wmod * modUC
+Wreu * reuUC + Wdel * delUC
Effort = A * (Sizeadjusted) B
Where
Effort is the maintenance effort;
Sizeadjusted is the adjusted product size;
A is the multiplicative calibration constant;
B is the exponential calibration constant 11

12. Modeling Process
12

13. Validation
Metric Definition
MRE = |predictive effort – actual effort| / actual effort
Referred Measures
MMRE: mean magnitude relative erro
MdMRE: median magnitude relative error
PRED25: the % of the data points with RE<=0.25
PRED30: the % of the data points with RE<=0.30
13

14. Prediction Process
Data collection
Prediction
To apply in different phases during lifecycle,
elaboration factors are referred to estimate size input.
Sizere-adjusted = EF * Sizeadjusted,where EF is the
elaboration factor between higher and lower-level
requirements
1 n NUC i whereNRi is the number of higher-
EF = ∑
n i=1 NR i
level requirements and NUCi is the number of lower-leverl
requirements in ith data point
*Picture Source: Alistair Cockburn, Writing Effective Use Cases 14

15. Case Study
Historical data are the projects mentioned in
slice 6.
Requirements are described in three level:
capability goals (CG, least detailed)
capability requirements (CR)
use cases (UC, most detailed)
15

16. Number Number of Number of Number of Actual Effort
Versions Requirements Levels
of new modified reused deleted (person/hour)
Capability goals(CG) 0 5 13 0
v1 Capability requirements(CR) 0 7 35 0 2284.5
Use cases(UC) 3 10 216 0
Capability goals(CG) 0 11 7 0
v2 Capability requirements(CR) 0 16 26 0 3941
Use cases(UC) 7 22 207 0
Capability goals(CG) 4 15 3 0
v3 Capability requirements(CR) 12 30 12 0 30945
Use cases(UC) 86 94 134 8
Capability goals(CG) 1 13 6 3
v4 Capability requirements(CR) 3 33 17 4 10340.1
Use cases(UC) 50 61 229 17
Capability goals(CG) 1 12 7 1
v5 Capability requirements(CR) 1 18 33 2 7477.5
Use cases(UC) 12 31 301 15
Capability goals(CG) 1 8 12 0
v6 Capability requirements(CR) 2 20 32 0 14903.6
Use cases(UC) 37 30 311 3
Capability goals(CG) 0 3 18 0
v7 Capability requirements(CR) 3 3 51 0 7166
16
Use cases(UC) 15 8 366 4

17. Case Study
Weight:
Wmod = 0.2, Wreu = 0.05, Wdel = 0
Sizeadjusted = newUC + 0.2 * modUC +0.05 * reuUC
Parameters Values
A 96.9396
B 1.192651
P-value 0.000481
R2 0.928219
Adjusted R2 0.913862
Effort = 96.9396 * (Sizeadjusted) 1.192651
17

18. Validation Result
Leave-one-out cross validation is applied.
Elaboration factors are referred from A. A. Malik’s research*
since the data used there is the subset of our dataset.
Metrics CG CR UC
MMRE 36.87% 26.18% 26.94%
UC yields the best
MdMRE 27.09% 24.13% 20.01% result
PRED25 0.4286 0.5714 0.7143
PRED30 0.5714 0.7143 0.8571
*A. A. Malik, B. W. Boehm, Y. Ku, and Y. Yang, “Comparative Analysis of Requirements Elaboration of an Industrial Product,”
Proceedings of the 2nd International Conference on Software Technology and Engineering(ICSTE 2010), Oct. 2010, pp. 46-50 18

19. Validation Result (Cont)
Versions Adjusted Size Actual Effort Predictive Effort MRE (%)
v1 15.8 2284.5 2916.8925 27.68
v2 21.75 3941 3767.9859 4.39
v3 111.5 30945 23276.7727 24.78
v4 73.65 10340.1 19263.9780 86.30
v5 33.25 7477.5 6136.3455 17.94
v6 58.55 14903.6 11921.1786 20.01
v7 34.9 7166 6628.5125 7.50
1. COTS has been used
2. Increased productivity through
requirement management
19

20. Method Comparison
Analogy
Database used for analogy is from China Software
Benchmarking Standard Group (CSBSG)
999 software project data from 140 organizations distributed
in 15 regions across China
COCOMO2000
Methods MMRE MDMRE PRED25 PRED30
Analogy 33.09% 31.59% 0.2857 0.2857
COCOMO 32.47% 15.47% 0.5714 0.5714
Use case 26.94% 20.01% 0.7143 0.8571
20

21. Discussion
Lessons Learned:
Use Case Metrics
Requirement Elaboration Factors
Advantages of use-case based estimation
Linear vs. Exponential relationship between effort and use case
Threats to Validity
Internal threats:
Weak outlier tolerance
Complexity of use cases
External threats:
Use case weight
21

22. Tool Implementation
22

23. Than You ！
Our Lab: http://itechs.iscas.ac.cn