Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • What Program Managers Need to Know About Software Estimating and its Impact On Successful Missions Abstract 250Program managers sometimes give software secondary status next to more tangible items in a space system development.  Yet software is responsible for many program delays and failures. For example, ESA’s Arian- 5 disaster was completely due to a software issue. The purpose of this presentation is to bring up the significance of software and appropriate software estimating in the development of successful systems.  It covers cost and schedule estimation and how that impacts mission success along with numerous “why should we care” points for program managers to increase awareness of software and software issues that should be managed.   Review From A Recent 2 Hour Presentation:   “Thanks for your excellent seminar.  I just finished reviewing the students’ evaluation of our course content and I want to pass along that your seminar on SW cost estimating received TOP marks!  One student wrote, “While I don’t fully understand software and IT and I now know that I better pay more attention to it or else watch my program tank.””
  • Galorath.dan

    1. 1. NASA PM Challenge 2011What Program Managers Need to Know AboutSoftware Estimating and its Impact OnSuccessful MissionsDan Used with permission
    2. 2. Key PointsSoftware is Software Software critical to estimating estimation & systems metrics can and is processes are provide getting core to program more reducing managers complex software risk significant information © 2010 Galorath Incorporated
    3. 3. Software Is A Critical Component of Military &Aerospace Systems Failures Ariane 5• Ariane 5 video• Note: $500 Million payload• Failure due to reused software from (Ariane 4)• With incompatible assumptions for exception condition that was not required• “the culture within the Ariane program…” only addressed “random hardware failures… which can quite rationally be handled by a backup system”• “the view had been taken that software should be considered correct until it is shown to be at fault”! © 2010 Galorath Incorporated
    4. 4. Software Is Getting More Complex Source: (Impact ofWeapon System Complexity on Systems Acquisition by Robert A. Dietrick, Major, USAF) Why should we care: Software is already difficult, costly, and error prone. More complexity means more problems © 2010 Galorath Incorporated
    5. 5. Software Is Providing an IncreasingPercentage of Functionality• Software development has been problematic for military aircraft developers – Consuming about 40 percent of the USAF’s RDT&E budget – Average software schedule overrun was 222 percent of the original estimate – Overruns often justified by stating weapon systems performance requirements are evermore demanding • And thus cause greater reliance on software• Software provided about 10 percent of an F-4’s functionality in the 1960s but provides over 80 percent of an F-22’s Why should we care: “unrealistic cost estimates lead to unrealistic budgets and un-executable programs” Meier Study… DAU © 2010 Galorath Incorporated
    6. 6. Software Failures Are Costly andDangerous• C130J: December 23, 2004, internal memo by Deputy Secretary of Defense called for the cancellation of the U.S. Air Force’s C-130J transport aircraft to cut its losses on the overpriced, unneeded, and problem-plagued C-130J. Estimated cancellation cost were $1.78 Billion – in 2004 dollars!• Satellite: The loss of a classified satellite after only 7 seconds on orbit prompted the review of software and processors that has caused the most recent delay and a potential $1 billion overrun in Lockheed Martins Space- Based Infrared System (SBIRS). Strategic Command is worried about potential gaps in coverage for some mission areas in part because satellites are being delivered later than planned.• Patriot: Iraqi Scud missile slipped through Patriot missile defenses a year ago and hit U.S. Army barracks in Dhahran, Saudi Arabia, killing 28 servicemen.• NASA Mars Climate Orbiter: Mathematical mismatch that was not caught until after the $125-million spacecraft, a key part of NASAs Mars exploration program, was sent crashing too low and too fast into the Martian atmosphere. © 2010 Galorath Incorporated
    7. 7. Software Is A Key Risk Item InWeapons Systems• GAO identified 42 programs at risk for cost & schedule 1. Military requirements changes 2. Software development challenges 3. Workforce issues• Navy Mobile User Objective Satellite Communication System delays to the Joint Tactical Radio System, a set of software-defined radios causes advanced MUOS capabilities to be drastically underused… GAO• National Institute of Standards and Technology (NIST) – Software defects cost nearly $60 Billion Annually – 80% of development costs involve identifying and correcting defects Why should we care: Software, not Hardware or technology readiness levels were called out © 2010 Galorath Incorporated
    8. 8. ESTIMATION & PLANNING: An Estimate Defined• An estimate is the most knowledgeable statement you can make at a particular point in time regarding: – Effort / Cost – Schedule – Staffing – Risk – Reliability• Estimates more precise with progress• A WELL FORMED ESTIMATE IS A DISTRIBUTION 8 © 2010 Galorath Incorporated
    9. 9. Estimation Methods 1 of 2 Model Description Advantages Limitations Category No Basis or substantiationGuessing Quick Can obtain any Off the cuff estimates No Process(Widely Used) answer desired Usually WrongAnalogy Compare project with past Estimates are based on Truly similar projects must(Widely Used) similar projects. actual experience. exist. Experts tend to be biased;Expert Little or no historical Consult with one or more knowledge level is sometimesJudgment data is needed; good for experts. questionable; may not be(Widely Used) new or unique projects. consistent. A hierarchical Provides an estimate decomposition of the Need valid requirements.Top Down linked to requirements system into progressively Difficult to track architecture;Estimation and allows common smaller components is engineering bias may lead to(Widely Used) libraries to size lower used to estimate the size underestimation. level components. of a software component. 9 © 2010 Galorath Incorporated
    10. 10. Estimation Methods 2 of 2 Model Category Description Advantages Limitations Uses expert judgment to determine how Easy to get under Design To Cost much functionality can stakeholder Little or no engineering basis. be provided for given number budget. Equation with one or Simple relationships may not tell the Simple CER’s more unknowns that Some basis in whole story (Widely Used) provides cost / data Historical data may not tell the whole schedule estimate story Models are usually fast and easy to use, and Models can be inaccurate if not Perform overall useful early in a properly calibrated and validated; Comprehensive estimate using design program; they are historical data may not be relevant Parametric Models parameters and also objective to new programs; optimism in (Widely Used) mathematical and repeatable. parameters may lead to algorithms. Easy tradeoffs underestimation. can provide better plansWhy should we care: The methods of estimation provide insights into the likelihood of its viability 10 © 2010 Galorath Incorporated
    11. 11. Initial Cost Estimation Problems (SoftwareProgram Managers Network)• Many programs that have been evaluated tend to initially estimate using a very optimistic method. – Low bid to win contract – Naïve level of effort estimation – Human optimism (HBR & Rich Hartley USAF Undersec Acquisition) – Often wrong since they are not based on a thorough analysis of requirements. The formula Cost = Size x Complexity/Productivity• Has three unknowns upfront: size, complexity, and productivity.• Methods & estimating tools determine size given system requirements• Organizational databases of productivity on comparable size and complexity projects can be used – Or other bottom-up estimation techniques “In any event, all initial software cost estimates should be considered as potential high risk and should be reviewed at each program review” SPMN © 2010 Galorath Incorporated
    12. 12. 10 Step Software EstimationProcessConsistent Processes = ReliableEstimates1. Establish 10. Track Project Estimate Scope Throughout Development 9. Document Estimate and Lessons 2. Establish Technical Learned Baseline, Ground Rules, Assumptions 8. Generate a Project Plan 3. Collect Data 7. Quantify Risks and Risk Analysis 4. Estimate and Validate Software Size 6. Review, Verify and Validate Estimate 5. Prepare Baseline 13 Estimates © 2010 Galorath Incorporated
    13. 13. Balancing Resources & Schedule Is A Science For a given Size, Complexity and Technology Minimum Time Work Expands To Fill Time To Complete (Effort Increases (Effort Increases due to lack of pressure ) to Reduce Schedule) Effort Increase Minimum Time due to LongerEffort Months Schedule Optimal Effort (Lower Effort for Longer Schedule) Why should we care: These impact both Calendar Time staffing (effort) schedule © 2010 Galorath Incorporated
    14. 14. Avoid “Death Marches” and FailedProjects By Applying “Brooks Law” 12Staff Level (FTE people) 10 Optimal Staffing 8 Unaccomplished Work Level 6 Cost Staffing Overrun 4 Schedule 2 Slip Planned Actual Delivery Delivery 0 1 4 7 10 13 16 19 22 25 28 31 34 37 40 43 46 Elapsed Calendar Time (months) Effective Staffing Staffing Beyond Plan Overstaffed Understaffed © 2010 Galorath Incorporated
    15. 15. Size is a Very important Software Metric • Software Size is the main driver of software development cost, effort and schedule -- use the best available estimate of size range! – Knowing size allows analysts to determine effort (cost) – Better size estimates = better cost estimates 40Development Schedule 35 30 25 Months 20 15 1000 Development Effort Months 10 900 5 800 700 0 600 20000 40000 60000 80000 500 100000 Size (SLOC) 400 300 200 100 0 20000 40000 60000 80000 100000 Size (SLOC) © 2010 Galorath Incorporated
    16. 16. Proper Definition is Important Actual Project Example:Size Estimate at Start of Development = 1.6M Lines Actual Size = 736,000 SLOC 46% Comments 23% Blanks 25% Debug Code 6% SLOC 46% 23% 6% 25% Why should we care: Poor sizing is a common reason for poor software estimates © 2010 Galorath Incorporated
    17. 17. Sizing Pitfalls Sizing Mistake ConsequenceWrong sizing metric chosen for level Large variance in estimatesof detail desiredNot enough time/effort spent on Unbelievable estimates – resultssoftware sizing in general don’t match the program and are too optimistic or pessimisticNo clear definition of size Inconsistent estimates – results don’t pass the sanity check, unreliable output, blame the modelSize growth not considered OR Inaccurate estimates – results aresize estimates reduced to too optimistic, programs will overrunachieve desired cost cost / schedule estimatesWhy should we care: Bad sizing yields bad estimates © 2010 Galorath Incorporated
    18. 18. Reuse: Watch Out For Low CostAssumptions on “Heritage”• Reuse or Heritage: applying existing software to a new mission (or additional innovation in its current mission)• Effort to reuse software is routinely under estimated TestImplementation DesignWhy should we care: Heritage is often underestimated and causes major schedule / cost overruns © 2010 Galorath Incorporated
    19. 19. Software Design For Reuse (Lower Cost Heritage)• Designed for reuse – Reusability requirements during original development – Significant extra coding and documentation effort during original development to insure reusability – Minor to moderate work (mostly retest) required when reused IF NOT MODIFIED• Not designed for reuse – Developed without extra effort to ensure reusability – Sources include any legacy code not specifically designed for reuse • Other projects & programs • Prior builds of the current program – Moderate to major rework required • Redesign Reuse OFTEN optimistic… look at • Reimplementation risk • Retest © 2010 Galorath Incorporated
    20. 20. Rework Causes Software ProjectIssues• Rework: Doing the same work over again because it was incorrect the first time – Prototyping is not rework – Refactoring (tuning to make software better) is not rework• Between 40% and 50% of the total effort on software projects is spent on rework.. Barry Boehm• Initiatives to reduce rework can save significantlyWhy should we care: Unplanned rework can drive cost up dramatically AND reducing rework can be a boon to projects © 2010 Galorath Incorporated
    21. 21. COTS or GOTS Hoped For Reuse In Embedded Systems• Developmental Software: Customization – Functionality COTS Software developed Developmental specifically for the Software Glue project at hand – May include “COTS Cognition” customization of COTS • COTS Software:• “Glue” Code: – Purchased functionality – Code written to bind • Direct Cost component of COTS COTS to integration developmental software • COTS Cognition: – Development effort – Required functionality within the COTS must be captured software that must be understood • Effort component of COTS integration Why should we care: COTS & GOTS promises are often not accurate meaning additional development cost & Schedule © 2010 Galorath Incorporated
    22. 22. Software Implemented Security and SafetyRequirements Add Significant Cost & ScheduleWhy should we care: Software implemented security and safety requirements can drive costs thru the roof © 2010 Galorath Incorporated
    23. 23. Understand Project Total OwnershipCosts Up FrontMost Projects Spend Low During Maintenance Staff Vs Maintenance Rigor 3500 staff hours per month 3000 2500 develop 2000 Rigor vhi+ 1500 Rigor nom 1000 Rigor vlo 500 0 1 7 13 19 25 31 37 43 49 55 61 67 73 79 85 Time Why should we care: Maintenance is costly and usually not well considered in estimation & planning © 2010 Galorath Incorporated
    24. 24. Example Sophisticated ParametricModel Can Provide Significant Insight © 2010 Galorath Incorporated
    25. 25. Example Sophisticated ParametricModel Risk Analysis © 2010 Galorath Incorporated
    26. 26. Example Sophisticated ParametricModel Results 3 © 2010 Galorath Incorporated
    27. 27. Example Benchmark From anEstimation Model.. Why Are WeSo Expensive? © 2010 Galorath Incorporated
    28. 28. Understand Project Risks Include Them In Planning Decisions (Example SEER-SEM Outputs) Probability Effort ProbabilityProbability Schedule Probability 99% Example Application 1 Example Application 1 90% 99% 90% 80% 80% 70% 70% 60% 60% 50% 50% 40% 40% 30% 30% 20% 20% 10% 10% 1% 0 1800 3600 5400 7200 9000 1% 0 4 8 12 16 20 Effort (person-hours) Time (calendar months) Defects Probability Probability Example Application 1 99% 90% 80% 70% 60% 50% 40% 30% 20% 10% 1% 0 12 24 36 48 60 Defects (count) Why should we care: Many programs plan at a 70 or 80% probability to reduce risk 2010 Galorath Incorporated © 30
    29. 29. Total Cost Growth for Two Space Programs David Graham, NASA Development Growth Causes Requirements Generation & Translation 8% Budget/Funding 11% 25% Cost Estimation Underestimation of Risk 11% 11% Schedule Slips (Govt & Contractor) 9% Price Increases 25% Other Quantitative Framework5 “The Success Triangle of Cost, Schedule, and Performance: A Blueprint for Development of Large-Scale Systems in an Increasingly Complex Environment” - (Booz|Allen|Hamilton, 2003) © 2010 Galorath Incorporated
    30. 30. GAO-093SP Cost Estimating and AssessmentGuide of March 2009 Why should we care: Estimates have uncertainty. We need to know probability of estimate or our project could fail © 2010 Galorath Incorporated 32
    31. 31. Software Has Additional Characteristics That Should Be Considered Track defect discovery and removal rates against expected rates Heath and Status Indicatorshows status and trends from the previous snapshotThresholds are user definable Increased defect reporting rate shows a worsening trend 33 Why should we care: EVM can be misleading in software… performance should be considered © 2010 Galorath Incorporated
    32. 32. Software Maintenance - Background• Maintenance typically 50% to 75% of the total software workload: Dependent on maintenance rigor & operational “life expectancy”• IEEE and others generally identify four categories of Software Maintenance efforts – Correct - maintenance is a change made in order to remove a fault – 20% – Adapt - maintenance is a change made in order to become suited to a different condition – 25% – Perfect - maintenance is a change made in order to improve – 5% – Enhance - maintenance is a change made to forestall or reverse a deterioration – 50% – Plus Innovation which is generally block changes © 2010 Galorath Incorporated
    33. 33. Software Project ManagementChallenges• “No good deed will go unpunished.” Unreasonable expectations on the next project are supported by heroic behavior on the previous project• The most important business decisions about a software project are made at the time of minimum knowledge and maximum uncertainty.• Adding and/or changing software means more time and/or more effort• When a project is in trouble ask for more time rather than more people. Deferring functionality common approach to asking for more time• Increasing concurrency is usually not a solution (e.g. designing several releases concurrently) © 2010 Galorath Incorporated A1-35
    34. 34. Summary• Software projects are manageable• Basis of management is a viable estimate• The 10 step process provides a repeatable approach to estimation process• You can help ensure useful results by adopting a process that is standardized and repeatable• Measurement is a key part of the estimation process• Beware of using simple measurements to drive estimates 36 © 2010 Galorath Incorporated