The document summarizes feedback from a survey on using COSMIC (a method for measuring functional size of software) in Agile projects. It finds that while COSMIC can objectively measure size at any level, in practice it is difficult to use at the user story level due to variable requirements quality. However, COSMIC is useful for overall project sizing, estimating, quality control of requirements, and measuring progress. While Agile adoption is increasing, there is a lack of standardization and published information on performance measurement across projects.

1. Survey of experience of using COSMIC in
Agile projects
- interim findings
Presented at UK COSMIC Special Interest Group
10th April 2014
Charles Symons

2. The Guideline on using COSMIC in Agile projects
makes several claims
Story Points are a subjective size measure
COSMIC sizes are an objective measure that:
• can be measured at any level of an Agile
project
• may be aggregated from User Stories
upwards to any level
• hence can be used to estimate at any level
• hence can be used to track progress in line
with Agile processes, e.g. of the backlog, of
earned value, etc.
Project
Release Component
Sprint/
Iteration
User Story
(Sub-) Task
(N. B. The
number of
levels in the
hierarchy, and
terminology
vary)
10/04/2014 2

3. Feedback from a brief survey of how COSMIC is
used in Agile projects in practice
• Canada
• UQAM Masters student study of four projects
• UQAM Masters student studying 15 projects: aim to build an estimation model (WIP)
• India, Italy
• General industry experience
• Netherlands
• Fairly extensive experience in Sogeti Nederland
• Russia
• SRG (small software house) experience
10/04/2014 3

4. Use COSMIC to measure at the Sprint level?
Difficult:
• ‘Wrong decomposition of requirements’ (Canada)
• Some User Stories may have part of a FP, others map to 4 x FP’s (Canada)
• Some US may comprise a new FUR plus a CR and be implemented over
multiple sprints. Overall, # US > # FP’s (Canada)
• Hard to use for small sprints. We measure only large ‘tasks’ (= several
sprints) and the whole application. We don’t measure every sprint. Small
tasks are measured with ‘proxies’, e.g. ‘add simple web form = n CFP’
(Russia)
• We measure at sprint level for progress control (Netherlands)
10/04/2014 4

5. How are Agile projects estimated? At what level?
• The strength of using USP is in the team discussions and use of ‘Poker
Planning’ – it’s not the same as traditional expert judgement
• Good correlation of CFP and effort expected at the Project level
• Now researching correlation of USP and CFP (Canada)
• We are mostly required to estimate whole projects before the
requirements are known in detail. We estimate using NESMA
approximate sizing. Most projects use Agile processes (Netherlands)
• We have a satisfactory correlation of size with effort for the whole
application and for large tasks (Russia)
10/04/2014 5

6. COSMIC can help with QC of requirements
Study 1 (and a 2011 study)
• Objective measurement of the quality of User Stories showed improvement with
each iteration, as a result of CFP measurement
Total projets Itération 1 Itération 2 Itération 3
% a 0 7 63
% b 0 72 37
% c 18 21 0
% d 82 0 0
% e 0 0 0
Study 2 (UQAM)
• No data models, poor naming consistency, etc leads to poor US that can’t easily
be measured
• having US that are measurable objectively, using methods such as COSMIC, could
increase the accuracy of future estimations
10/04/2014 6

7. COSMIC measurements are used to control progress
UQAM Study 1
Sogeti experience: ‘We also use QSM Control to control the project progress
and for this COSMIC is much more powerful than NESMA (≡IFPUG FPA), as it
gives a more accurate picture of the actual functionality being ready at a
certain point in time, resulting in more accurate forecasts.’
10/04/2014 7

8. Some general industry observations
• Companies are starting to use Agile, but there is not yet much
published information (Italy)
• Lack of standardization of ‘story points’ is a concern when carrying
out performance measurement across projects. COSMIC is being
studied/piloted. (India)
• Use of traditional estimation methods at bid/proposal stage, and use
of User Story or Work Breakdown Structure at the iteration stage,
seem most common (Indian BSPIN report, 2012)
• UK Government Digital Services experience is very positive for small
web projects, but no real performance measurements (?)
10/04/2014 8

9. Use of COSMIC with Agile projects: Conclusions
• The claims for the applicability of COSMIC are theoretically true
• In practice, given current Agile process maturity, it is difficult to use COSMIC at
the US level for sizing and estimating, due to:
• the variable quality of requirements
• some US or tasks include re-factoring of earlier US
• difficulties of dealing with NFR, etc
• Agile Project teams would benefit from being trained in COSMIC sizing. The
‘Planning Poker’ process is also important
• COSMIC can be used with advantage for Agile projects for:
• whole project sizing & estimating
• QC of requirements
• progress control
10/04/2014 9