This document summarizes key points from a presentation on software estimating and its impact on successful missions. It discusses how software is a critical and increasingly complex component of systems that is prone to failures and costly overruns if not estimated properly. Estimation processes and using metrics can help program managers reduce risks. The document provides examples of software failures and their high costs, and emphasizes that software estimates should be treated as high risk and reviewed at each program review. It also outlines best practices for software estimation like using a 10 step process and accounting for size, complexity, productivity and risks.

1. NASA PM Challenge 2011
What Program Managers Need to Know About
Software Estimating and its Impact On
Successful Missions
Dan Galorath
galorath@galorath.com
Used with permission

2. Key Points
Software is Software Software
critical to estimating estimation &
systems metrics can
and is processes are
provide
getting core to program
more reducing managers
complex software risk significant
information
© 2010 Galorath Incorporated

3. Software Is A Critical Component of Military &
Aerospace Systems Failures Ariane 5
• Ariane 5 video
• Note: $500 Million payload
• Failure due to reused software from (Ariane 4)
• With incompatible assumptions for exception condition
that was not required
• “the culture within the Ariane program…” only
addressed “random hardware failures… which can
quite rationally be handled by a backup system”
• “the view had been taken that software should be
considered correct until it is shown to be at fault”!
© 2010 Galorath Incorporated

4. Software Is Getting More Complex Source: (Impact of
Weapon System Complexity on Systems Acquisition by Robert A. Dietrick, Major, USAF)
Why should we care: Software is already difficult,
costly, and error prone. More complexity means more
problems
© 2010 Galorath Incorporated

5. Software Is Providing an Increasing
Percentage of Functionality
• Software development has been problematic for military
aircraft developers
– Consuming about 40 percent of the USAF’s RDT&E budget
– Average software schedule overrun was 222 percent of the
original estimate
– Overruns often justified by stating weapon systems
performance requirements are evermore demanding
• And thus cause greater reliance on software
• Software provided about 10 percent of an F-4’s
functionality in the 1960s but provides over 80
percent of an F-22’s
Why should we care: “unrealistic cost estimates lead to
unrealistic budgets and un-executable programs”
Meier Study… DAU
© 2010 Galorath Incorporated

6. Software Failures Are Costly and
Dangerous
• C130J: December 23, 2004, internal memo by Deputy Secretary of Defense
called for the cancellation of the U.S. Air Force’s C-130J transport aircraft to cut
its losses on the overpriced, unneeded, and problem-plagued C-130J.
Estimated cancellation cost were $1.78 Billion – in 2004 dollars!
• Satellite: The loss of a classified satellite after only 7 seconds on orbit
prompted the review of software and processors that has caused the most
recent delay and a potential $1 billion overrun in Lockheed Martin's Space-
Based Infrared System (SBIRS). Strategic Command is worried about
potential gaps in coverage for some mission areas in part because
satellites are being delivered later than planned.
• Patriot: Iraqi Scud missile slipped through Patriot missile defenses a year ago
and hit U.S. Army barracks in Dhahran, Saudi Arabia, killing 28 servicemen.
• NASA Mars Climate Orbiter: Mathematical mismatch that was not
caught until after the $125-million spacecraft, a key part of NASA's Mars
exploration program, was sent crashing too low and too fast into the
Martian atmosphere.
© 2010 Galorath Incorporated

7. Software Is A Key Risk Item In
Weapons Systems
• GAO identified 42 programs at risk for cost & schedule
1. Military requirements changes
2. Software development challenges
3. Workforce issues
• Navy Mobile User Objective Satellite Communication
System delays to the Joint Tactical Radio
System, a set of software-defined radios causes
advanced MUOS capabilities to be drastically
underused… GAO
• National Institute of Standards and Technology (NIST)
– Software defects cost nearly $60 Billion Annually
– 80% of development costs involve identifying and
correcting defects
Why should we care: Software, not Hardware or
technology readiness levels were called out
© 2010 Galorath Incorporated

8. ESTIMATION & PLANNING:
An Estimate Defined
• An estimate is the most knowledgeable statement you can
make at a particular point in time regarding:
– Effort / Cost
– Schedule
– Staffing
– Risk
– Reliability
• Estimates more precise with progress
• A WELL FORMED ESTIMATE IS A
DISTRIBUTION
8
© 2010 Galorath Incorporated

9. Estimation Methods 1 of 2
Model
Description Advantages Limitations
Category
No Basis or substantiation
Guessing Quick Can obtain any
Off the cuff estimates No Process
(Widely Used) answer desired
Usually Wrong
Analogy Compare project with past Estimates are based on Truly similar projects must
(Widely Used) similar projects. actual experience. exist.
Experts tend to be biased;
Expert Little or no historical
Consult with one or more knowledge level is sometimes
Judgment data is needed; good for
experts. questionable; may not be
(Widely Used) new or unique projects.
consistent.
A hierarchical
Provides an estimate
decomposition of the Need valid requirements.
Top Down linked to requirements
system into progressively Difficult to track architecture;
Estimation and allows common
smaller components is engineering bias may lead to
(Widely Used) libraries to size lower
used to estimate the size underestimation.
level components.
of a software component.
9
© 2010 Galorath Incorporated

10. Estimation Methods 2 of 2
Model Category Description Advantages Limitations
Uses expert judgment
to determine how Easy to get under
Design To Cost much functionality can stakeholder Little or no engineering basis.
be provided for given number
budget.
Equation with one or Simple relationships may not tell the
Simple CER’s more unknowns that Some basis in whole story
(Widely Used) provides cost / data Historical data may not tell the whole
schedule estimate story
Models are
usually fast and
easy to use, and Models can be inaccurate if not
Perform overall
useful early in a properly calibrated and validated;
Comprehensive estimate using design
program; they are historical data may not be relevant
Parametric Models parameters and
also objective to new programs; optimism in
(Widely Used) mathematical
and repeatable. parameters may lead to
algorithms.
Easy tradeoffs underestimation.
can provide
better plans
Why should we care: The methods of estimation provide
insights into the likelihood of its viability
10
© 2010 Galorath Incorporated

11. Initial Cost Estimation Problems (Software
Program Managers Network)
• Many programs that have been evaluated tend to initially estimate using a
very optimistic method.
– Low bid to win contract
– Naïve level of effort estimation
– Human optimism (HBR & Rich Hartley USAF Undersec Acquisition)
– Often wrong since they are not based on a thorough analysis of requirements. The formula
Cost = Size x Complexity/Productivity
• Has three unknowns upfront: size, complexity, and productivity.
• Methods & estimating tools determine size given system requirements
• Organizational databases of productivity on comparable size and
complexity projects can be used
– Or other bottom-up estimation techniques
“In any event, all initial software cost
estimates should be considered as potential
high risk and should be reviewed at each
program review” SPMN
© 2010 Galorath Incorporated

12. 10 Step Software Estimation
Process
Consistent Processes = Reliable
Estimates
1. Establish 10. Track Project
Estimate Scope Throughout
Development
9. Document Estimate
and Lessons
2. Establish Technical Learned
Baseline, Ground
Rules, Assumptions
8. Generate a
Project Plan
3. Collect Data
7. Quantify Risks and
Risk Analysis
4. Estimate and Validate
Software Size 6. Review, Verify
and Validate
Estimate
5. Prepare
Baseline
13 Estimates
© 2010 Galorath Incorporated

13. Balancing Resources & Schedule
Is A Science
For a given Size, Complexity and Technology
Minimum Time Work Expands
To Fill Time
To Complete (Effort Increases
(Effort Increases
due to lack of pressure )
to Reduce Schedule)
Effort Increase
Minimum Time due to Longer
Effort Months
Schedule
Optimal Effort
(Lower Effort
for Longer
Schedule)
Why should we care: These impact both
Calendar Time
staffing (effort) schedule
© 2010 Galorath Incorporated

14. Avoid “Death Marches” and Failed
Projects By Applying “Brooks Law”
12
Staff Level (FTE people)
10 Optimal
Staffing
8 Unaccomplished
Work Level
6 Cost Staffing
Overrun
4
Schedule
2
Slip
Planned Actual
Delivery Delivery
0
1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46
Elapsed Calendar Time (months)
Effective Staffing Staffing Beyond Plan Overstaffed Understaffed
© 2010 Galorath Incorporated

15. Size is a Very important Software
Metric
• Software Size is the main driver of software development cost,
effort and schedule -- use the best available estimate of size
range!
– Knowing size allows analysts to determine effort (cost)
– Better size estimates = better cost estimates
40
Development Schedule
35
30
25
Months
20
15 1000
Development Effort Months
10 900
5 800
700
0
600
20000 40000 60000 80000
500 100000
Size (SLOC) 400
300
200
100
0
20000 40000 60000 80000 100000
Size (SLOC)
© 2010 Galorath Incorporated

16. Proper Definition is Important
Actual Project Example:
Size Estimate at Start of Development = 1.6M Lines
Actual Size = 736,000 SLOC
46% Comments 23%
Blanks 25%
Debug Code 6%
SLOC 46%
23%
6%
25%
Why should we care: Poor sizing is a common
reason for poor software estimates
© 2010 Galorath Incorporated

17. Sizing Pitfalls
Sizing Mistake Consequence
Wrong sizing metric chosen for level Large variance in estimates
of detail desired
Not enough time/effort spent on Unbelievable estimates – results
software sizing in general don’t match the program and are too
optimistic or pessimistic
No clear definition of size Inconsistent estimates – results
don’t pass the sanity check,
unreliable output, blame the model
Size growth not considered OR Inaccurate estimates – results are
size estimates reduced to too optimistic, programs will overrun
achieve desired cost cost / schedule estimates
Why should we care: Bad sizing yields bad estimates
© 2010 Galorath Incorporated

18. Reuse: Watch Out For Low Cost
Assumptions on “Heritage”
• Reuse or Heritage: applying existing software to a new
mission (or additional innovation in its current
mission)
• Effort to reuse software is routinely under estimated
Test
Implementation Design
Why should we care: Heritage is often underestimated
and causes major schedule / cost overruns
© 2010 Galorath Incorporated

19. Software Design For Reuse
(Lower Cost Heritage)
• Designed for reuse
– Reusability requirements during original development
– Significant extra coding and documentation effort
during original development to insure reusability
– Minor to moderate work (mostly retest) required when
reused IF NOT MODIFIED
• Not designed for reuse
– Developed without extra effort to ensure reusability
– Sources include any legacy code not specifically
designed for reuse
• Other projects & programs
• Prior builds of the current program
– Moderate to major rework required
• Redesign Reuse OFTEN optimistic… look at
• Reimplementation risk
• Retest © 2010 Galorath Incorporated

20. Rework Causes Software Project
Issues
• Rework: Doing the same work over again because it
was incorrect the first time
– Prototyping is not rework
– Refactoring (tuning to make software better) is not
rework
• Between 40% and 50% of the total effort on software
projects is spent on rework.. Barry Boehm
• Initiatives to reduce rework can save significantly
Why should we care: Unplanned rework can drive cost up
dramatically AND reducing rework can be a boon to
projects
© 2010 Galorath Incorporated

21. COTS or GOTS Hoped For Reuse In
Embedded Systems
• Developmental
Software: Customization
– Functionality COTS Software
developed Developmental
specifically for the Software Glue
project at hand
– May include
“COTS Cognition”
customization of
COTS
• COTS Software:
• “Glue” Code: – Purchased functionality
– Code written to bind • Direct Cost component of COTS
COTS to integration
developmental
software • COTS Cognition:
– Development effort – Required functionality within the COTS
must be captured software that must be understood
• Effort component of COTS integration
Why should we care: COTS & GOTS promises are often not
accurate meaning additional development cost & Schedule
© 2010 Galorath Incorporated

22. Software Implemented Security and Safety
Requirements Add Significant Cost & Schedule
Why should we care: Software implemented security and
safety requirements can drive costs thru the roof
© 2010 Galorath Incorporated

23. Understand Project Total Ownership
Costs Up Front
Most Projects Spend Low During Maintenance
Staff Vs Maintenance Rigor
3500
staff hours per month
3000
2500 develop
2000 Rigor vhi+
1500 Rigor nom
1000
Rigor vlo
500
0
1 7 13 19 25 31 37 43 49 55 61 67 73 79 85
Time
Why should we care: Maintenance is costly and
usually not well considered in estimation & planning
© 2010 Galorath Incorporated

24. Example Sophisticated Parametric
Model Can Provide Significant Insight
© 2010 Galorath Incorporated

25. Example Sophisticated Parametric
Model Risk Analysis
© 2010 Galorath Incorporated

26. Example Sophisticated Parametric
Model Results 3
© 2010 Galorath Incorporated

27. Example Benchmark From an
Estimation Model.. Why Are We
So Expensive?
© 2010 Galorath Incorporated

28. Understand Project Risks Include Them In Planning
Decisions (Example SEER-SEM Outputs)
Probability
Effort Probability
Probability
Schedule Probability 99%
Example Application 1
Example Application 1 90%
99%
90% 80%
80% 70%
70% 60%
60% 50%
50% 40%
40% 30%
30% 20%
20% 10%
10% 1%
0 1800 3600 5400 7200 9000
1%
0 4 8 12 16 20 Effort (person-hours)
Time (calendar months)
Defects Probability
Probability Example Application 1
99%
90%
80%
70%
60%
50%
40%
30%
20%
10%
1%
0 12 24 36 48 60
Defects (count)
Why should we care: Many programs plan at a 70 or
80% probability to reduce risk
2010 Galorath Incorporated
© 30

29. Total Cost Growth for Two Space Programs
David Graham, NASA
Development Growth Causes
Requirements
Generation & Translation
8% Budget/Funding
11% 25%
Cost Estimation
Underestimation of Risk
11%
11% Schedule Slips (Govt &
Contractor)
9%
Price Increases
25%
Other
Quantitative Framework
5 “The Success Triangle of Cost, Schedule, and Performance: A Blueprint for Development of Large-Scale
Systems in an Increasingly Complex Environment” - (Booz|Allen|Hamilton, 2003)
© 2010 Galorath Incorporated

30. GAO-093SP Cost Estimating and Assessment
Guide of March 2009
Why should we care: Estimates have uncertainty.
We need to know probability of estimate or our
project could fail
© 2010 Galorath Incorporated 32

31. Software Has Additional Characteristics That
Should Be Considered
Track defect
discovery and
removal rates
against expected
rates
Heath and Status Indicator
shows status and trends from
the previous snapshot
Thresholds are user definable
Increased defect
reporting rate
shows a
worsening trend
33
Why should we care: EVM can be misleading in
software… performance should be considered
© 2010 Galorath Incorporated

32. Software Maintenance -
Background
• Maintenance typically 50% to 75% of the total software
workload:
Dependent on maintenance rigor & operational “life
expectancy”
• IEEE and others generally identify four categories of
Software Maintenance efforts
– Correct - maintenance is a change made in order to remove
a fault – 20%
– Adapt - maintenance is a change made in order to become
suited to a different condition – 25%
– Perfect - maintenance is a change made in order to
improve – 5%
– Enhance - maintenance is a change made to forestall or
reverse a deterioration – 50%
– Plus Innovation which is generally block changes
© 2010 Galorath Incorporated

33. Software Project Management
Challenges
• “No good deed will go unpunished.” Unreasonable
expectations on the next project are supported by
heroic behavior on the previous project
• The most important business decisions about a
software project are made at the time of minimum
knowledge and maximum uncertainty.
• Adding and/or changing software means more time
and/or more effort
• When a project is in trouble ask for more time rather
than more people. Deferring functionality common
approach to asking for more time
• Increasing concurrency is usually not a solution (e.g.
designing several releases concurrently)
© 2010 Galorath Incorporated A1-35

34. Summary
• Software projects are manageable
• Basis of management is a viable estimate
• The 10 step process provides a repeatable
approach to estimation process
• You can help ensure useful results by adopting a
process that is standardized and repeatable
• Measurement is a key part of the estimation
process
• Beware of using simple measurements to drive
estimates
36
© 2010 Galorath Incorporated