Presentation about how to use objective metrics as a basis for agile cost estimation and monitoring

1. How to use objective metrics as a basis for
agile cost estimation and monitoring?
Nesma Network event IT Cost Management:
How to adapt software metrics in the Agile world to become more predictable and transparent?
May 31, 2023
Harold van Heeringen, Principal Consultant

2. 2
© IDC |
• Principal Consultant at IDC Metri and Global Practice Lead IT
Intelligence services
• 25+ years experience in the IT industry
• Immediate Past President International Software Benchmarking
Standards Group (ISBSG), Board member Nesma.
• Experienced in helping application development teams to measure,
benchmark and improve productivity and product quality
• Expertise: fact-based AD and AMS decision-making
• Software Quality and Risk Assessments and improvements plans
• Agile Team Performance Assessments
• Agile Value Management
• Software Cost Estimation
• Portfolio Quality and Risk Assessments
• Software Sizing: Nesma, IFPUG and COSMIC FSM
• Vendor bid support
• Married, 3 kids, living in Veendam (the North of the Netherlands),
chess player, zwifter, fitness.
Who am I?
hvanheeringen@idc.com

3. 3
© IDC |
Topics addressed today
Introduction IDC Metri IT Intelligence
Challenges in ‘Agile’ Software Cost Estimation
Using Sizing for Software Estimation
Objective metrics to understand team performance
Understand how the team spends their effort
Create reference points
Monitor and manage the projects using facts
10 Key Take aways

4. 4
© IDC |

5. 5
© IDC |
IT Intelligence
IT Intelligence exists of several elements;
• Quality of software; Optimizing the (bespoke) development chain.
• Team Performance Optimization; team performance against the right quality levels
• Benchmark against the industry; your journey reflected to similar market situations
• Application Portfolio Insights; scan your landscape to get insight in risks/health
issues, dependencies between applications, technology used and
transition/transformation readiness (such as cloud).
• Cost estimation; estimate the size and costs of change, counter estimate to find
estimation erosion.
• Business value driven development; bring together cost allocation, cloud
economics, architectural principles and portfolio metrics to prioritize application
development based on business value.
• Team sourcing; how to structure your teams based on best practices and rate card
benchmark
• Compliancy with market standards such as ISO

6. 6
© IDC |
Our 7 IT Intelligence propositions
Value, Benefit and Cost
Management
IT Intelligence Assessment Team Performance
Optimization
Software Quality and Risk
Assessment
Intelligent Requirements
Management
Software Estimation
and Monitoring
Application Landscape
Assessment

7. 7
© IDC |
IDC Research survey

8. 8
© IDC |
Project Success rates

9. 9
© IDC |
The Agile Challenge
01 02 04
03
Agile development is a challenge to
manage and measure, especially
compared to traditional
development models like Waterfall
Management is often uncertain
when functionality will be delivered
and at what cost, its quality and
risks inherent in the application
Key challenges experienced by
business owners of agile products
are:
• When will it be done
• How much will it cost
• Is the quality of the work
produced sufficient
• How do I measure it and have
actionable KPIs
• How can I communicate project
success and value effectively to
my business peers
• Is it stable
• Does it meet the
organizational/customer
requirements
• Will it be reliable and perform
• Can it scale
• Is it secure
• Due to the nature of agile (Story
Points to assess progress), it is
difficult to quantify agile
• If something is hard to quantify, it
is hard to evaluate and
manage.
• Still, Senior Management need to
make decisions and
demonstrate value delivered for
budget spent.
• Software Cost Estimation is a
necessity, also for agile
projects!

10. 10
© IDC |
A realistic estimate is one of the most important conditions for a successful project.
The estimate is the basis for:
• Business case
• Planning
• Proposal (outsourcing: fixed price / date)
• Financial result of the project… and the organization
• Claiming and releasing of resources
• Alignment between IT and business / customer
• Progress reports / dashboards and corrective actions
• The feeling of the team and the stakeholders
• Demonstrating Value for Money to stakeholders
Without a realistic estimate, the project is likely to fail!
Realistic estimation is a Key Success Factor

11. 11
© IDC |
Software Cost Estimation is hard?!

12. 12
© IDC |
#NoEstimates? Easy if it’s not your money that you are
spending!

13. 13
© IDC |
Software Cost Estimation maturity model
Source: Galorath
Human Judgment
Biased
Parametric
Unbiased

14. 14
© IDC |
Sizing allows estimation and benchmarking using data
System or project
Size
Effort
Cost
Estimating &
Benchmarking
Level 0 and 1
Level 3, 4 and 5
1000 FP → 6700 – 7800 – 9400 effort hours
Metric PDR (h/FP)
N 151
Min 2,1
P10 5
P25 6,7
Median 7,8
P75 9,4
P90 12,6
Max 31,9
Avg 11,8

15. 15
© IDC |
Agile Project Sizing
High Level FPA Approximate FPA Approximate FPA
5000 FP (+/- 30%) 500 FP (+/- 15%)
12 FP
15 FP
8 FP
11 FP
16 FP
11 FP
19 FP
11 FP
12 FP

16. 16
© IDC |
Company starts to develop a new ERP system in .Net.
Indicative FPA measurement: 5000 FP +/- 30%
ISBSG repository data
Team: 7 fte, 560 effort hours per sprint (2 weeks)
Blended rate: $100
Simple High-level Example
Metric PDR (h/FP)
N 151
Min 2,1
P10 5
P25 6,7
Median 7,8
P75 9,4
P90 12,6
Max 31,9
Avg 11,8
Size (FP) PDR (h/FP)Effort (hours) Cost ($) Sprints (#)
5000 6,7 33.500 3.350.000
$ 60
5000 7,8 39.000 3.900.000
$ 70
5000 9,4 47.000 4.700.000
$ 84

17. 17
© IDC |
Predictability is important

18. 18
© IDC |
Measuring Team Performance
-80%
-60%
-40%
-20%
0%
20%
40%
60%
80%
100%
Team Metrics Product Metrics
High Performing Team
Industry Average
Low Performing Team

19. 19
© IDC |
Story Points won’t help managing the value creation
Capacity
per
sprint
(e.g.,
1000
hours)
Time
Sprint 1
Scheduled
Visible Business Value
Yes: Functional Size (FP)
Yes: Story Points
Unplanned, no visible Business Value
No: Functional Size (FP)
Yes: Story Points
New functionality
Tasks
80 Story Points
80 FP new
Story Points realized:
Functionality delivered:

20. 20
© IDC |
Story Points won’t help managing the value creation
Capacity
per
sprint
(e.g.,
1000
hours)
Time
Sprint 1 Sprint 2-5
Scheduled
Visible Business Value
Yes: Functional Size (FP)
Yes: Story Points
Unplanned, no visible Business Value
No: Functional Size (FP)
Yes: Story Points
New functionality
Modified
functionality
Code refactoring
Tasks
Tasks
New functionality
80 Story Points
80 FP new
80 Story Points
50 FP new
20 FP rework
Story Points realized:
Functionality delivered:

21. 21
© IDC |
Story Points won’t help managing the value creation
Capacity
per
sprint
(e.g.,
1000
hours)
Time
Sprint 1 Sprint 2-5 Sprint 6-10
Scheduled
Visible Business Value
Yes: Functional Size (FP)
Yes: Story Points
Unplanned, no visible Business Value
No: Functional Size (FP)
Yes: Story Points
New functionality
Modified
functionality
Code refactoring
Removed
functionality
Fix bugs
Code refactoring
Tasks
Tasks
Tasks
New functionality
Modified
functionality
New functionality
80 Story Points
80 FP new
80 Story Points
50 FP new
20 FP rework
Story Points realized:
Functionality delivered:
80 Story Points
20 FP new
40 FP rework

22. 22
© IDC |
Story Points won’t help managing the value creation
Capacity
per
sprint
(e.g.,
1000
hours)
Tijd
Sprint 1 Sprint 2-5 Sprint 6-10 Sprint 11- n
Scheduled
Visible Business Value
Yes: Functional Size (FP)
Yes: Story Points
Unplanned, no visible Business Value
No: Functional Size (FP)
Yes: Story Points
New functionality
Modified
functionality
Code refactoring
Removed
functionality
Fix bugs
Code refactoring
Fix bugs
Code refactoring
3rd line support
Tasks
Tasks
Tasks
Tasks
New functionality
Modified
functionality
New functionality
Removed
functionality
Changed
functionality
New functionality
80 Story Points
80 FP new
80 Story Points
50 FP new
20 FP rework
Story Points realized:
Functionality delivered:
80 Story Points
20 FP new
40 FP rework
80 Story Points
10 FP new
30 FP rework
Not managing the functionality
delivered means value delivery
may become at risk!

23. 23
© IDC |
When you understand the performance of the team(s), you can manage the project and forecast how much
functionality will be ready at which moment in time and the cost of this.
Productivity (PDR)= Effort hours spent per FP delivered, e.g., 10 hours per Function Point
Cost Efficiency = Cost spent per FP delivered, e.g., € 750 per Function Point
Delivery Speed = FP delivered per Month, e.g., 150 FP per month
Process Quality = Defects found per 1000 FP, e.g., 85 defects per 1000 FP delivered
Value Delivered = FP per € 1000 spent, e.g., 4,5 FP per € 1000 spent
What do you need to do?
• Register effort per ‘colored block’ to understand where the effort is spent! This is often not done in ‘agile’.
• Analyze the blocks: How big the is the green part compared to the other parts?
• Register volumes: defects. Also not often done in ‘agile’.
• Measure Product Quality as well!
Key metrics to measure and benchmark

24. 24
© IDC |
Team Performance dashboard

25. 25
© IDC |
Team Benchmark dashboard

26. 26
© IDC |
Effort registration is key for project control
If you want to control your project, you need to understand:
• Total functionality that needs to be delivered
• Functional size ready after each sprint/release
• Effort spent - Productivity of the team(s)
• Best case scenario based on best productivity
• Worst case scenario
• %Rework per sprint
• #Defects found during sprints and on the backlog,
• Team effort spent on not-productive work
• This means understanding where the effort of the team
members is spent!
I.
Incident- en probleem-
management
IV.
Strategic
portfolio change
III.
Small (business)
change
CHANGE &
INNOVATION
MAINTENANCE & RUN
CURRENT
FUTURE
32%
0%
10%
20%
49%
20%
30%
40%
II.
Application
health

27. 27
© IDC |
Monitoring ‘green blocks’ ready

28. 28
© IDC |
1. Software Cost Estimation seems hard, why is this?
2. Software cost estimation is not a one-time activity.
3. There are objective Agile team metrics that can be compared to industry
averages.
4. (Nesma) sizing is a crucial step!
5. In all software cost estimations, there is an implied Productivity
distribution as there is output and input involved.
6. Story point metrics are misleading when used for estimation.
7. Effort registration and data collection/analysis are important prerequisites
for successful management.
8. Functional Size Measurement creates reference points.
9. Measuring progress based on functional size helps managing the project.
10. Especially in agile, beware of scope creep due to rework and/or
requirements change. These will result in extra story points but may not in
extra function points.
10 key take aways

29. IDC.com linkedin.com/company/idc twitter.com/idc blogs.idc.com
© IDC
Harold van Heeringen
hvanheeringen@idc.com