Local Bias and its Impacts on the Performance of Parametric Estimation Models Ye Yang, Lang Xie, Zhimin He (ISCAS) Qi Li, Vu Nguyen, Barry Boehm (USC) Ricardo Valerdi (MIT/Univ. of Arizona) Sep. 21, 2011 Promise 2011, Banff, Canada
Conduct representative local calibration to produce A’ and B’.
Calculate local bias and compare among groups.
CII 2000 Subset After2000 Subset Subset 1 … A, B A 1 ’ , B 1 ’ A 2 ’ , B 2 ’ A n ’ , B n ’ local_bias 1 local_bias 2 local_bias n CII 2010 Dataset Subset 2 Subset n Group by Organization_ID Default Constants: A, B
Basic performance indicators: MMRE (mean MRE), stdMRE (the variance of MRE)
Average MMRE, Range of MMRE, Average stdMRE, and Range of stdMRE are used to assess the performance of an estimation model.
Average MMRE Range of MMRE Average stdMRE Range of stdMRE Repeat the above steps for 2000 times 2000 (MMRE, stdMRE) pairs Spliting data set into training set and test set Tuning model parameters on training set Evaluating model performance on test set MMRE, stdMRE
Spearman correlation coefficients between local bias and model performance:
At the significant level of p-value less than 0.05, the range of stdMRE is significantly positive correlated with local bias and local_bias*num. Both the average stdMRE and the average MMRE are significantly positive correlated with local_bias*num.
Range of stdMRE reflects the uncertainty of model performance. Hence, the bigger the local bias is, the weaker the performance is.