Independent, data-driven cost estimating is now standard practice at NATO. Software intensive projects are estimated using function point analysis and SEER tools. Full lifecycle analysis and independent cost reviews help manage risks. Estimates support contractor negotiations and tracking actual performance. While process improvements continue, NATO's estimating maturity has grown significantly since early reliance on subjective expert opinions led to cost and schedule overruns.

1. A Structured Approach to Estimating
Software Intensive Projects
René Berghuijs and Shaun Irvin
10 March 2015
10/03/2015 NATO UNCLASSIFIED 1

2. • NATO is a military, political Alliance
headquartered in Brussels, Belgium
• Core business: peace and security
• The NATO Communications & Information
Agency is NATO’s acquisition organization, with
entities in Brussels, The Hague and Mons (BEL)
• In today’s context NATO is a user organization
11/03/2015 NATO UNCLASSIFIED 2
Introduction

3. 11/03/2015 NATO UNCLASSIFIED 3
Software Estimating Progression
Rene’s
NATO
Experience
Early Days
of NATO
Software
Estimating
NATO
Reform &
Software
Intensive
Task Force
NATO
Software
Estimating
Today

4. 11/03/2015 NATO UNCLASSIFIED 4
Rene’s NATO estimating experience
• Hired in 2005 to do independent SW cost estimating
for changes and block upgrades to NATO’s AIR C2
program
– Previous experience as HW cost estimator for production
plants of (petro) chemicals
• Familiarization with SW cost estimating through:
– Literature (Barry Boehm, Capers Jones)
– Training on COTS products
– Joining cost society (ISPA)

5. • NATO Theatre Missile Defense program, based on the
ACCS program (nowadays integrated Air & Missile
Defense)
• Ingredients of the cost estimate:
– Requirements driven
– Function Point Analysis
– COTS estimating program (calibrated)
– Validation
• Outcome: an investment cost estimate used in the
funding request to the nations
– Screened by WGNTEs, no comments on cost
– Authorized by the then Investment Committee
11/03/2015 NATO UNCLASSIFIED 5
First big estimating project

6. Lack of Analytical Expertise
• Budget already set
• Type-B Cost Estimate (TBCE) based on technical expert level of
effort (LOE) input
• Speculation at lower levels led to large error margins
• Estimates not defendable or repeatable
• “Catch-22” with NATO Office of Resources (NOR) and
requirements
Cost/Schedule Overruns
• Expert opinion ALWAYS led to underestimation
• No BOEs or baselines to track contractor progress or support
during negotiations
11/03/2015 NATO UNCLASSIFIED 6
NATO Early Days
Software Estimating

7. 11/03/2015 NATO UNCLASSIFIED 7
LOE Estimating
MAIN GROUP ACTIVITY WORK Labor Duration Total Unit Cost Subtotal Total Cost Total Labor Group Cost
Project Manager 1.0 24.0 24.0 20,000€ 480,000€
Technical Management 1.0 24.0 24.0 20,000€ 480,000€
Configuration Management 0.2 24.0 4.8 18,000€ 86,400€
Quality Management 0.2 24.0 4.8 18,000€ 86,400€
Test Management 0.2 24.0 4.8 18,000€ 86,400€
Contract Management 0.2 24.0 4.8 18,000€ 86,400€
Finance Management 0.2 24.0 4.8 18,000€ 86,400€
Project Admin Office 0.2 24.0 4.8 15,000€ 72,000€
IT System Admin 0.2 24.0 4.8 18,000€ 86,400€
Other Staff Contribution 0.2 24.0 4.8 10,000€ 48,000€
System Analysis (SME) 4.0 2.0 8.0 20,000€ 160,000€
SSS 2.0 1.0 2.0 16,000€ 32,000€
System Requirements Review (SRR) 4.0 0.2 0.8 18,000€ 14,400€
System Design 2.0 1.0 2.0 20,000€ 40,000€
TA, ICD 4.0 1.0 4.0 18,000€ 72,000€
SSDD 2.0 1.0 2.0 18,000€ 36,000€
Critical Design Review (CDR) 6.0 0.2 1.2 18,000€ 21,600€
Software Requirements Analysis 8.0 4.0 32.0 20,000€ 640,000€
SRS, IRS 4.0 2.0 8.0 16,000€ 128,000€
Software Design 8.0 4.0 32.0 18,000€ 576,000€
SDD, IDD 4.0 2.0 8.0 16,000€ 128,000€
Software Implementation (coding) 18.0 20.0 360.0 16,000€ 5,760,000€ 360.0 5,760,000€
Software Unit Testing 10.0 6.0 60.0 16,000€ 960,000€ 60.0 960,000€
Specification, Integration and Test 2.0 1.0 2.0 18,000€ 36,000€
Procurement 2.0 0.5 1.0 18,000€ 18,000€
Test Plan Preparation 2.0 0.5 1.0 20,000€ 20,000€
Test Procedures 4.0 2.0 8.0 16,000€ 128,000€
Test Readiness Review (TRR) 10.0 0.1 1.0 18,000€ 18,000€
Pilot System Installation and Test 6.0 2.0 12.0 16,000€ 192,000€
Factory Acceptance Test - FAT 15.0 1.0 15.0 18,000€ 270,000€
System Acceptance Test - SAT 10.0 1.0 10.0 18,000€ 180,000€
808,000€
Development Systems Engineering
376,000€ 20.0 376,000€
Software Development
Detailed in Sheet:
WP1 Software Dev.
8,192,000€
80.0 1,472,000€
Hardware Development
54,000€ 3.0 54,000€
System Integration
and Testing
808,000€ 47.0
1,392,000€
Support Functions
14.4 206,400€
WP 1: XXXX Implementation
Management Project Management
1,598,400€
72.0

8. Lack of Analytical Expertise
• Budget already set
• Type-B Cost Estimate (TBCE) based on technical expert level of
effort (LOE) input
• Speculation at lower levels led to large error margins
• Estimates not defendable or repeatable
• “Catch-22” with NATO Office of Resources (NOR) and
requirements
Cost/Schedule Overruns
• Expert opinion ALWAYS led to underestimation
• No BOEs or baselines to track contractor progress or support
during negotiations
11/03/2015 NATO UNCLASSIFIED 8
NATO Early Days
Software Estimating

9. • NCI Agency was established 2012 as the outcome of NATO’s
Agency Reform – merger of NC3A, NACMA etc.
• Founding Principles:
• Customer Funding
• Full Project Lifecycle
• Best Practices
• Service Lines and Program Offices
• Centralized Cost Capability:
• Four Cost Professionals co-located in Brussels
• Common tools (SEER suite)
• Using shared historical data
• Setting up new policies and processes
11/03/2015 NATO UNCLASSIFIED 9
Birth of NCI Agency

10. • Established in Dec 2013 to analyze cost and
schedule overruns in NATO software projects
• Goal: To identify specific process improvements
for better project management
11/03/2015 NATO UNCLASSIFIED 10
Software Intensive Project
Task Force

11. 11/03/2015 NATO UNCLASSIFIED 11
Cost/Schedule Overruns
• Initial analysis of 27 recently implemented SIPs
• 5 cancelled or failed with little or no capability
• 10 completed projects with severe cost/schedule impacts (>4 years
late or >30% over budget
• 10 projects still being implemented (average 3 years late)
• Only 2 completed less than 2 years late and less than 15% over
budget

12. • Independent, data-driven, lifecycle cost estimating is
truly in NATO’s DNA:
– Software Intensive Projects are now estimated using
Function Point Analysis and SEER tools
– SIPs are recognized as being more risky than other
projects; contingencies are now project specific
– Full Life-cycle analysis supports O&M Phase
• Estimate baselines are used in the analysis of
contractor ECPs through:
– Audited Labour rates (basic rates, overhead and general /
administrative)
– Tracking contractor’s performance across life-cycle
(productivity, quality, schedule)
11/03/2015 NATO UNCLASSIFIED 12
Fast Forward to Today…

13. 11/03/2015 NATO UNCLASSIFIED 13
Function Point Counting

14. • Independent, data-driven, lifecycle cost estimating is
truly in NATO’s DNA:
– Software Intensive Projects are now estimated using
Function Point Analysis and SEER tools
– SIPs are recognized as being more risky than other
projects; contingencies are now project specific
– Full Life-cycle analysis supports O&M Phase
• Estimate baselines are used in the analysis of
contractor ECPs through:
– Audited Labour rates (basic rates, overhead and general /
administrative)
– Tracking contractor’s performance across life-cycle
(productivity, quality, schedule)
11/03/2015 NATO UNCLASSIFIED 14
Fast Forward to Today…

15. 11/03/2015 NATO UNCLASSIFIED 15
SEER Outputs
1.11 Program - CTE Missions Application
Activity Proj Mgr Analyst Design Program Data Test/QA CM/RM QC Lead Total
Sys Reqs 10 44 12 0 5 10 2 2 85
SW Reqs 30 115 35 15 15 30 5 5 249
Pre Design 169 153 629 184 123 215 31 31 1,533
Det Design 93 84 346 101 68 118 17 17 844
Code 273 117 234 2,148 234 586 156 156 3,905
Int & Test 47 12 23 227 47 169 29 29 583
Prog Test 31 8 16 153 31 114 20 20 393
Sys I&T 148 37 74 352 19 1,092 93 37 1,851
Development Total 801 570 1,369 3,180 541 2,334 352 296 9,443
Maint 857 214 429 4,072 107 4,286 536 214 10,716
Life Cycle Total 1,658 784 1,797 7,252 648 6,620 888 511 20,159

16. • Independent, data-driven, lifecycle cost estimating is
truly in NATO’s DNA:
– Software Intensive Projects are now estimated using
Function Point Analysis and SEER tools
– SIPs are recognized as being more risky than other
projects; contingencies are now project specific
– Full Life-cycle analysis supports O&M Phase
• Estimate baselines are used in the analysis of
contractor ECPs through:
– Audited Labour rates (basic rates, overhead and general /
administrative)
– Tracking contractor’s performance across life-cycle
(productivity, quality, schedule)
11/03/2015 NATO UNCLASSIFIED 16
Fast Forward to Today…

17. 11/03/2015 NATO UNCLASSIFIED 17
Risk Analysis & Uncertainty
Risk #01 JTS interim system continues to evolve, making it a moving
target and endangering the user acceptance of the N-JTS.
Probability High
Description As N-JTS delivery is only possible after a lengthy NSIP
procurement (approximately 3-4 years after 1st
stage
authorization), JTS interim system might be asked to implement
emerging NATO requirements, making it a “moving target” for
the N-JTS. As a result, the user community may not be
motivated to embrace the N-JTS, as it will eventually be
delivered with less functionality than the interim system. On the
other hand, incorporating these charges can delay the project
and add additional costs.
Impact Medium
Scope
Impact
Yes
Schedule
Impact
Up to 6
months
Cost Impact Up to €
500,000
Mitigation 1 Limit changes to JTS to MMR, governed by the IPMT. During Execution
2 Foreseen additional budget/time to incorporate necessary
changes.
Not included in current
proposal
3 Introducing early releases will allow to re-prioritise some of
the solution features to ensure that those features that are
mandatory are included
Included in current
proposal
• Schedule and Cost impact
factored into Contingency plan
• Higher confidence level used in
estimates

18. • Independent, data-driven, lifecycle cost estimating is
truly in NATO’s DNA:
– Software Intensive Projects are now estimated using
Function Point Analysis and SEER tools
– SIPs are recognized as being more risky than other
projects; contingencies are now project specific
– Full Life-cycle analysis supports O&M Phase
• Estimate baselines are used in the analysis of
contractor ECPs through:
– Audited Labour rates (basic rates, overhead and general /
administrative)
– Tracking contractor’s performance across life-cycle
(productivity, quality, schedule)
11/03/2015 NATO UNCLASSIFIED 18
Fast Forward to Today…

19. 0
2
4
6
8
10
12
2018 2019 2020 2021 2022 2023 Fiscal Year
Effort Months
Corrective Adaptive Perfective Enhancements
TMD 1 (Step 4) (Effort)
11/03/2015 NATO UNCLASSIFIED 19
Software Maintenance

20. • Independent, data-driven, lifecycle cost estimating is
truly in NATO’s DNA:
– Software Intensive Projects are now estimated using
Function Point Analysis and SEER tools
– SIPs are recognized as being more risky than other
projects; contingencies are now project specific
– Full Life-cycle analysis supports O&M Phase
• Estimate baselines are used in the analysis of
contractor ECPs through:
– Audited Labour rates (basic rates, overhead and general /
administrative)
– Tracking contractor’s performance across life-cycle
(productivity, quality, schedule)
11/03/2015 NATO UNCLASSIFIED 20
Fast Forward to Today…

21. • Better negotiation techniques through SEER
• Investment Committee has more confidence in our estimates
• Challenges:
– Better defined initial requirements are necessary
– Closing the feedback loop (actual vs. planned)
– To be at maturity level 2 by 2016 and level 3 by 2017
11/03/2015 NATO UNCLASSIFIED 21
Today…
Hours Price Travel Hours Price Travel Hours Price Travel Hours Price Travel
PM 3,672 501,230€ 125,580€ 3,030 413,597€ 19,000€ 3,030 413,597€ 19,000€ 3,030 413,597€ 19,000€
SE 14,263 1,946,840€ 683,622€ 5,233 714,275€ 97,500€ 6,047 825,434€ 97,500€ 8,443 1,152,514€ 97,500€
SW Dev 23,074 3,149,615€ 323,054€ 10,720 1,463,236€ 125,000€ 12,581 1,717,248€ 125,000€ 18,056 2,464,668€ 125,000€
QA 1,720 234,781€ -€ 414 56,459€ -€ 509 69,458€ -€ 789 107,710€ -€
CM/DM 664 90,636€ -€ 491 66,960€ -€ 604 82,379€ -€ 936 127,747€ -€
ILS 1,390 189,736€ -€ 550 75,075€ -€ 550 75,075€ -€ 550 75,075€ -€
44,783 6,112,838€ 1,132,256€ 20,437 2,789,603€ 241,500€ 23,320 3,183,192€ 241,500€ 31,804 4,341,311€ 241,500€
Area
SEER Low
3,031,103€
3,679,535€
Agreed Price: €4,100,000 (57% savings) !!!
Proposed SEER HighSEER Med
TOTAL: 4,582,811€3,424,692€
7,245,094€

22. 11/03/2015 NATO UNCLASSIFIED 22
Repeat Customer

23. • Better negotiation techniques through SEER
• Investment Committee has more confidence in our estimates
• Challenges:
– Better defined initial requirements are necessary
– Closing the feedback loop (actual vs. planned)
– To be at maturity level 2 by 2016 and level 3 by 2017
11/03/2015 NATO UNCLASSIFIED 23
Today…
Hours Price Travel Hours Price Travel Hours Price Travel Hours Price Travel
PM 3,672 501,230€ 125,580€ 3,030 413,597€ 19,000€ 3,030 413,597€ 19,000€ 3,030 413,597€ 19,000€
SE 14,263 1,946,840€ 683,622€ 5,233 714,275€ 97,500€ 6,047 825,434€ 97,500€ 8,443 1,152,514€ 97,500€
SW Dev 23,074 3,149,615€ 323,054€ 10,720 1,463,236€ 125,000€ 12,581 1,717,248€ 125,000€ 18,056 2,464,668€ 125,000€
QA 1,720 234,781€ -€ 414 56,459€ -€ 509 69,458€ -€ 789 107,710€ -€
CM/DM 664 90,636€ -€ 491 66,960€ -€ 604 82,379€ -€ 936 127,747€ -€
ILS 1,390 189,736€ -€ 550 75,075€ -€ 550 75,075€ -€ 550 75,075€ -€
44,783 6,112,838€ 1,132,256€ 20,437 2,789,603€ 241,500€ 23,320 3,183,192€ 241,500€ 31,804 4,341,311€ 241,500€
Area
SEER Low
3,031,103€
3,679,535€
Agreed Price: €4,100,000 (57% savings) !!!
Proposed SEER HighSEER Med
TOTAL: 4,582,811€3,424,692€
7,245,094€

24. 11/03/2015 NATO UNCLASSIFIED 24
Mature Requirements

25. • Better negotiation techniques through SEER
• Investment Committee has more confidence in our estimates
• Challenges:
– Better defined initial requirements are necessary
– Closing the feedback loop (actual vs. planned)
– To be at maturity level 2 by 2016 and level 3 by 2017
11/03/2015 NATO UNCLASSIFIED 25
Today…
Hours Price Travel Hours Price Travel Hours Price Travel Hours Price Travel
PM 3,672 501,230€ 125,580€ 3,030 413,597€ 19,000€ 3,030 413,597€ 19,000€ 3,030 413,597€ 19,000€
SE 14,263 1,946,840€ 683,622€ 5,233 714,275€ 97,500€ 6,047 825,434€ 97,500€ 8,443 1,152,514€ 97,500€
SW Dev 23,074 3,149,615€ 323,054€ 10,720 1,463,236€ 125,000€ 12,581 1,717,248€ 125,000€ 18,056 2,464,668€ 125,000€
QA 1,720 234,781€ -€ 414 56,459€ -€ 509 69,458€ -€ 789 107,710€ -€
CM/DM 664 90,636€ -€ 491 66,960€ -€ 604 82,379€ -€ 936 127,747€ -€
ILS 1,390 189,736€ -€ 550 75,075€ -€ 550 75,075€ -€ 550 75,075€ -€
44,783 6,112,838€ 1,132,256€ 20,437 2,789,603€ 241,500€ 23,320 3,183,192€ 241,500€ 31,804 4,341,311€ 241,500€
Area
SEER Low
3,031,103€
3,679,535€
Agreed Price: €4,100,000 (57% savings) !!!
Proposed SEER HighSEER Med
TOTAL: 4,582,811€3,424,692€
7,245,094€

26. 11/03/2015 NATO UNCLASSIFIED 26
Questions?