The document discusses a study that analyzed the effect of highlighting error categories in COSMIC functional size measurement (FSM) training on the accuracy of FSM. The study analyzed FSM assignments from 2010-2013, where the first two years did not highlight errors and the second two years did. There was a sharp decrease in many common error categories, such as those related to listing FPs and naming, in the later years when errors were highlighted, indicating highlighting errors improves FSM accuracy. However, some core measurement rules saw little change. The study concludes highlighting frequent errors in training significantly increases FSM accuracy.

2. The Effect of Highlighting Error Categories in FSM
Training on the Accuracy of Measurement
ALI MERT ERTUGRUL*, GOKCEN YILMAZ*, MURAT SALMANOGLU*, ONUR DEMIRORS*
* Department of Information Systems, Middle East Technical University – Ankara /
TURKEY
October 6-8, 2014
International Workshop on Software Measurement and International Conference on
Software Process and Product Measurement – IWSM / MENSURA 2014
Rotterdam, the NETHERLANDS

3. OUTLINE
• Introduction to FSM
• Related Work on FSM accuracy
• Motivation
• Research Methodology and Implementation
• Results & Discussion
• Conclusion & Future Work
07/10/2014 3

4. Functional Size Measurement
• Functional Size Measurement (FSM) is a technique
for measuring software in terms of the functionality
it delivers.
• IFPUG, MkII, COSMIC, etc.
• FSM accuracy is crucial since based on the size of
the software, the cost and effort estimation are
done.
07/10/2014 4

5. Variance in FSM
• Two main reasons:
• Different assumptions and interpretations
• Misunderstanding of measurement standards
• Solution: COSMIC Training
Turetken et al. 2008
Ungan et al. 2009
07/10/2014 5

6. FSM Training on FSM Accuracy
• Educational framework for the teaching and
learning of software measurement topics at the
undergraduate level.
• Guidelines to teachers and instructors for
promoting the learning of software measurement
topics.
• Besides FSM, productivity, customer satisfaction,
etc.
Villavicencio et al. 2013
07/10/2014 6

7. FSM Training on FSM Accuracy
• Conducted two experiments:
• Standard COSMIC training
• Standard COSMIC training with practical cases
• The latter increased the reliability and accuracy of
FSM more.
• Errors were not the prior concern in practical
examples.
Ungan et al. 2010
07/10/2014 7

8. Definition of Error Categories
• Errors made during COSMIC FSM are categorized.
• Proposed tool detects the measurement errors
automatically.
• Does not aim to increase FSM accuracy via training.
Yilmaz et al. 2013
07/10/2014 8

9. Motivation
• COSMIC FSM accuracy increases with training.
• Frequent error categories are defined.
• We argued, whether highlighting the error
categories during FSM training increases the
accuracy of FSM.
07/10/2014 9

10. Error Categories
(EC)
Name
EC1 Duplicate Functional Process (FP)
EC2 Lack of List FP before Update FP
EC3 Lack of List FP before Delete FP
EC4 Lack of Retrieve FP before Update FP
EC5
Lack of Data Movement (DM) type Write (W) in Add, Delete and
Update FPs
EC6 Redundant DM type W in List FPs
EC7 Multiple occurrences of the same DM within the same FP
EC8 Each FP should be composed of at least 2 DMs
EC9 Each FP should contain at least 1 Write (W) / Exit (X) DM
EC10 Each FP should contain at least 1 Entry (E) DM
EC11 List FP might be included in Update/Delete FPs
EC12 Create/Delete/Update operations might be combined
EC13 Data Group (DG) Duplication
EC14
User interface components and System users are considered as DG /
Object Of Interest (OOI)
EC15 OOIs are named wrong
07/10/2014 10

11. Data Acquisition
• We analyzed COSMIC FSM assignment data:
• Obtained from two courses (SM502 and IS529)
• Four consequent years (2010, 2011, 2012 and 2013)
• Given by the same instructor for four years and two
courses
• Similar MIS projects
• 44 measurement results (10, 11, 11 and 12)
07/10/2014 11

12. Highlighting Error Categories
• Measurer teams were:
• Unexperienced about COSMIC FSM before the given
courses.
• Given COSMIC FSM training without being warned about
Error Categories in 2010 and 2011.
• Given COSMIC FSM training in which Error Categories
were highlighted with detailed examples in 2012 and
2013.
07/10/2014 12

13. Expert Review Process
• Reference keys were prepared for each project.
• Two experienced measurers evaluated the results.
• An iterative review process was followed.
07/10/2014 13

14. Total and Average # of Errors 2010 – 2013
EC
Total number of errors for
years
Average number of errors
for years
2010 2011 2012 2013 2010 2011 2012 2013
EC1 60 18 19 14 6 1,6 1,8 1,2
EC2 17 57 5 1 1,7 5,2 0,5 0,1
EC3 23 69 6 5 2,3 6,3 0,5 0,4
EC4 20 20 6 10 2 1,8 0,5 0,8
EC5 26 11 3 6 2,6 1 0,3 0,5
EC6 3 0 0 0 0,3 0 0 0
EC7 19 81 4 6 1,9 7,4 0,4 0,5
EC8 0 0 0 1 0 0 0 0,1
EC9 19 2 1 0 1,9 0,2 0,1 0
EC10 7 4 2 2 0,7 0,4 0,2 0,2
EC11 25 38 4 9 2,5 3,5 0,4 0,8
EC12 3 1 0 7 0,3 0,1 0 0,6
EC13 N/A N/A N/A N/A N/A N/A N/A N/A
EC14 330 63 13 7 33 5,7 1,2 0,6
EC15 66 33 7 10 6,6 3 0,6 0,8
07/10/2014 14

15. 07/10/2014 15

16. Results & Discussion
• Majority of the ECs has a significant decrease in
total and average number of errors.
• Related to List FP:
• EC2: Lack of List FP before Update FP
• EC3: Lack of List FP before Delete FP
• EC11: List FP might be included in Update/Delete FPs
• Related to Naming:
• EC14: User interface components and System users
are considered as DG / Object Of Interest (OOI)
• EC15: OOIs are named wrong
• Also, EC5 and EC7
07/10/2014 16

17. Results & Discussion
• Some of the ECs do not have a significant decrease
in total and average number of errors.
• Core measurement rules:
• EC8: Each FP should be composed of at least 2 DMs
• EC9: Each FP should contain at least 1 Write (W) /
Exit (X) DM
• EC10: Each FP should contain at least 1 Entry (E) DM
• Naturally correct:
• EC6: Redundant DM type W in List FPs
• EC12: Create/Delete/Update operations might be
combined
• Analysis on the errors does not contain EC13.
• Students were not responsible for EC13 during
measurements
07/10/2014 17

18. Conclusion
• Frequent errors during COSMIC FSM were analyzed.
• In the first two years COSMIC FSM training did not
include Error Categories.
• Error categories were highlighted in detail during
COSMIC FSM training in the second two years.
• Sharp decrease was observed in ECs in the second two
years.
• Proposed study shows that highlighting error categories
in the course curriculum increases the FSM accuracy
remarkably.
07/10/2014 18

19. Future Work
• Increase sample size
• Apply enhanced statistical analysis (e.g. variance)
• Combine with error prevention and identification
methods and tools
• Categorize errors to be prevented via training or
tools
07/10/2014 19

20. References
• O. Turetken, O. O. Top, B. Ozkan, and O. Demirors, “The Impact of Individual Assumptions on
Functional Size Measurement,” IWSM/Metrikon/Mensura, volume 5338 of Lecture Notes in
Computer Science, pp. 155–169, 2008.
• E. Ungan, O. Demirörs, and Ö. Ö. Top, “An Experimental Study on the Reliability of COSMIC
Measurement Results.”, IWSM '09 /Mensura '09 Proceedings of the International Conferences
on Software Process and Product Measurement, pp. 321-336, 2009
• O. O. Top, O. Demirors, and B. Ozkan, “Reliability of COSMIC Functional Size Measurement
Results: A Multiple Case Study on Industry Cases,” 35th Euromicro Conf. Softw. Eng. Adv. Appl.,
pp. 327–334, 2009.
• E. Ungan, Ö. Ö. Top, B. Özkan, and O. Demirörs, “Evaluation of Reliability Improvements for
COSMIC Size Measurement Results,” IWSM/MetriKon/Mensura, 2010.
• The Common Software Measurement International Consortium (COSMIC): 2011 Guideline for
Assuring the Accuracy of Measurements, Version 1.0.
• M. Villavicencio and A. Abran, “Towards the Development of a Framework for Education in
Software Measurement,” 2013 Jt. Conf. 23rd Int. Work. Softw. Meas. 8th Int. Conf. Softw.
Process Prod. Meas., pp. 113–119, Oct. 2013.
• G. Yilmaz, S. Tunalilar, and O. Demirors, “Towards the Development of a Defect Detection Tool
for COSMIC Functional Size Measurement,” 2013 Jt. Conf. 23rd Int. Work. Softw. Meas. 8th Int.
Conf. Softw. Process Prod. Meas., pp. 9–16, Oct. 2013.
07/10/2014 20

21. Thank You!
07/10/2014 21