Progress report at UK COSMIC SIG meeting 8, plus results of measurements in Agile environments showing superioriity of CFP over Story Points

1. GENERAL PROGRESS IN USE OF COSMIC
SIZING, INCLUDING IN AGILE PROJECTS
Charles Symons
UK COSMIC SIG Meeting, June 2018
Copyright © COSMIC 2018. All rights reserved

2. Agenda
2
 General progress
 Experience of using COSMIC in Agile
environments

3. The COSMIC method is now frozen at v4.0.2
3
 The method is now mature and stable.
 Measurement Practices Committee priorities:
 align Guidelines, Case Studies, Certification exams, etc.,
with v4.0.2
 improve ‘accessibility’ of the method documentation
 Other activities:
 develop training modules that can e.g. be shown on Youtube
 improve ‘cosmic-sizing’ website for accessibility and pro-
active use

4. There is great interest in COSMIC size
measurement automation
4
 100+ research papers now published.
 Automated CFP sizing from:
 Text using natural language processing, AI, etc. (emerging)
 Designs (in production)
 In UML (Poland, government contracts)
 In Matlab Statemate (Renault et al)
 Code (demonstrated to good accuracy)
 Static and dynamic analysis of Java code
 Motivations: measurement speed & accuracy; requirements
quality control; cost estimation

5. There is great interest in COSMIC in China
5
 Much COSMIC documentation now translated
into Chinese
 IWSM 2018 in Beijing devoted to COSMIC
 SIG established by Tencent QQ, has 200
members
 6+ workshops on COSMIC/year
 Data on 5 projects to ISBSG
 Work to start on COSMIC size automation

6. The big challenge: gaining acceptance in the
Agile community
6
 Agilists don’t like having their performance
measured and compared
 ‘Velocity’ measures do not measure velocity
 The ‘No Estimate’ movement
 Yet growing interest in ‘Agile-at-scale’ is a
sign of accepting real-world constraints

7. COSMIC, IFPUG and Nesma have collaborated
to raise awareness of FSM amongst Agilists
7
We prefer Facts to Stories
(Managing Agile activities using standardised measures)
I F P U G
May 2018
Contents:
• Benefits of Agile processes
• So what’s wrong with Agile measurement?
• The solution: use standard software functional
sizing methods to manage ‘Agile-at-scale’
• Outline description of FSM methods
• When to use FSM methods in Agile
• Estimating in Agile environments
• Software sizing in outsourcing contracts
• Introducing a standard software size
measurement method
• Summary: Functional size measures versus Story
Points

8. Agenda
8
 General progress
 Experience of using COSMIC in Agile
environments

9. Background
 Organizations experienced in Agile methods are
starting to realise the limitations of Story Points
 Very difficult to compare performance and track
progress across Teams
 No help for early effort estimation, or for
organizational learning
 Reports are now coming in on the use of COSMIC
Function Points instead of Story Points
9

10. Case: Canadian supplier of security and
surveillance software systems
 A customer request for new or changed function is called a ‘task’
 Uses Scrum method; iterations last 3 – 6 weeks
 Teams estimate tasks within each iteration in Story Points, and
convert directly to effort in work-hours (this is not considered good
Agile practice)
 Study* involved measurements on 24 tasks in nine iterations
 Each task estimated in SP, converted to effort
 Actual effort recorded
 Each task also measured in CFP
‘Effort Estimation with Story Points and COSMIC Function Points - An Industry Case Study’, Christophe Commeyne, Alain
Abran, Rachida Djouab. ‘Software Measurement News’. Vol 21, No. 1, 2016. Obtainable from www.cosmic-sizing.org

11. A best-fit straight line would be a poor
predictor of effort from SP sizes
0
20
40
60
80
100
120
140
160
180
200
0 20 40 60 80 100 120 140 160 180 200
ActualEffort(hours)
Estimated Effort (Hours)
Effort = 0.47 x Story Points + 17.6 hours and R2 = 0.33)
Notice the wide spread and the 17.6 hours ‘overhead’ for zero CFP

12. The Effort vs CFP size graph (24 tasks)
shows a good fit, but there are two outliers
0
20
40
60
80
100
120
140
160
180
200
0 10 20 30 40 50 60 70 80
ActualEffort(Hours)
Functional Size in CFP
Effort = 1.84 x CFP + 6.11 hours and R2 = 0.782
The two projects with low effort/CFP were found to involve significant software re-use,
so were rejected as outliers

13. Now we have a good effort vs CFP correlation
(22 tasks), usable for predicting effort
0
20
40
60
80
100
120
140
160
180
200
0 10 20 30 40 50 60 70 80
ActualEffor(Hours)
Functional Size in CFP
Y = 2.35 x CFP - 0.08hrs and R2 = 0.977)

14. Large Turkish supplier of security software*
14
 Agile/SCRUM method
 Web portal project for one Team (6 developers, 2 testers)
 Ten 3-week Sprints analysed
 Planning meeting for each Sprint estimates Story Points
and allocates Stories to Sprints
 CFP sizes measured retrospectively from ‘mature’
documentation in JIRA
 Measurement effort averaged 4.1 hours/Sprint (= 25
CFP/hour)
* ‘Effort estimation for Agile software development. Comparative case studies using COSMIC Function Points and Story
Points’. Murat Salmanoglu, Tuna Hacaloglu, Onur Demirors. Ankara, Turkey. IWSM/Mensura Conference, Gothenburg 2017

15. Completed CFP correlate much better with
Actual Effort than do Story Points
15
y = 3.9322x + 345.31
R² = 0.6648
0
200
400
600
800
1000
1200
1400
0 50 100 150 200
Effort(Work-hours)
Story Points
Case 1: SP vs Actual Effort
y = 9.0669x - 37.783
R² = 0.8576
0
200
400
600
800
1000
1200
1400
0 20 40 60 80 100 120 140
Effort(Work-hours)
COSMIC Function Points
Case 1: CFP vs Actual Effort
CFP vs Actual effort has much better R2 and much lower intercept for CFP = 0

16. Large Turkish software organization mainly
supplying the telecoms industry*
16
 500 Developers using Agile approaches
 Study of 10 Change Request ‘Projects’ for one
specific development team
 Story Points estimated by experts and converted
directly to ‘Predicted effort’
 CFP sizes measured retrospectively from the same
‘not mature’ CR documents + other information
 Measurement effort averaged 1 day/project (~ 9
CFP/WH)

17. Completed CFP correlate better with Actual
Effort than does Predicted Effort (≡ SP)
17
y = 1.0414x + 50.031
R² = 0.9093
0
200
400
600
800
1000
1200
1400
0 200 400 600 800 1000 1200 1400
ActualEffort(WH)
Predicted Effort (WH)
Case 2: Predicted vs Actual Effort
y = 5.968x + 3.8385
R² = 0.9528
0
200
400
600
800
1000
1200
1400
1600
0 50 100 150 200 250
ActualEffort
COSMIC Function Points
Case 2: CFP vs Actual Effort
CFP vs Actual effort has better R2 and much lower intercept for CFP = 0

18. Large Turkish software developer, supplying
mainly to finance and banking industry*
18
 SCRUM methodology
 Requirements documentation ‘lacking’
 Story Points are directly converted to estimated
effort, but no predicted effort data was available
 CFP measured retrospectively
 Results shown here are for 6 projects that used the
same C# technology

19. Completed CFP correlate much better with
Actual Effort than do Story Points
19
y = 5.6693x + 100.75
R² = 0.5647
0
100
200
300
400
500
600
0 10 20 30 40 50 60 70
ActuaolEffort(WH)
Story Points
Case 3.1: SP vs Actual Effort
y = 2.3693x - 34.877
R² = 0.9264
0
100
200
300
400
500
600
0 50 100 150 200 250
ActualEffort(WH)
COSMIC Function Points
Case 3.1: CFP vs Actual Effort
CFP vs Actual effort has much better R2 and much better intercept for CFP = 0

20. A User view of ‘COSMIC for Agile’
 “We have found that adopting this approach provides us
with excellent predictability and comparability across
projects, teams, time and technologies.”
 The reality of achieving predictable project performance has
driven me to investigate many methods of prediction.
COSMIC is the method that lets me sleep at night.”
Denis Krizanovic, Aon Australia, August 2014
Copyright: COSMIC 2017
20

21. Conclusion: CFP sizes correlate very well with
effort – much better than Story Points
21
 Correlations of post-calculated CFP sizes with actual
effort versus SP/effort correlations:
 higher R-squared (better)
 intercepts for zero CFP much closer to zero effort
(more realistic)
 See the original papers for other interesting results

22. The productivity figures of the four datasets vary
significantly; they should not be compared
22
‘Product Delivery Rate’ figures of the four datasets vary from 2.35 to 9.1
work-hours/CFP.
The following factors almost certainly influence performance:
 Different levels of decomposition of the software
 Different activities included in effort figures
 Different work mixes (new requirements, change requests)
 Varying requirements documentation quality (and no measures of
product quality)
 Varying amounts of functional or code re-use
 Different application domains, technologies, work practices
 (Maybe) different skill-levels, hence different real productivity

23. Conclusion: CFP can beneficially replace SP,
with no other changes to Agile practices
 In practice, a subjective measure
of relative effort
 Meaningful only within a project
team
 Poor for estimating total project
effort
 No guidance on how to deal with
Non-Functional Requts.
 An objective, ISO Standard
measure of functional size
 Sizes meaningful across projects
and teams
 Good for estimating at all levels
(US, Sprint, Release, System)
 Method advises how to deal with
NFR
Story Points COSMIC Function Points
Copyright: COSMIC 2017
23
Getting Agile teams to accept measurement is the biggest challenge