Graham dave


Published on

Published in: Business, Economy & Finance
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Graham dave

  1. 1. Using Cost-Risk toConnect Cost Estimating and EVM Presentation to PM Challenge 2007 Down to Business Track David R. Graham NASA HQ’s PA&E/Cost Analysis Division 202-358-1002
  2. 2. Outline• NASA Cost Initiatives: Emphasis on identifying, quantifying and managing cost-risk• OMB’s Capital Planning Guide• Point Estimates vs Range Estimates• Cost-risk assessment and analysis – Cost estimating relationship risk – Cost model input parameter/driver risk – Key system/subsystem/WBS element characteristic risk – Correlation• Connecting Cost Estimating and EVM• Risk Management Metric 1
  3. 3. NASA Cost Initiatives• Cost Initiatives motivated by congressional interest (negative 2004 GAO NASA Cost Estimating Report) and President’s Commission Report on Space Exploration• Initiatives to improve Agency cost estimating are documented in the new NASA Cost Estimating Handbook ( which is tied closely to NPR 7120.5D• The initiatives include: – The use of cost risk analysis to quantify uncertainty – Better cost data collection using a Cost Analysis Data Requirement (CADRe) – A corporate data base of CADRes – the One NASA Cost Engineering (ONCE) database 2
  4. 4. June 2006 Capital Programming Guide Supplement A-11, Part 7 OMB Focus on Risk Without the knowledge of the risks involved managers at alllevels—Agency, Office of Management and Budget (OMB) andCongress — cannot make the best decisions for the allocation ofresources among the competing investments Project managers when developing the cost, schedule andperformance goals on developmental projects with significant riskmust, therefore, provide the agency Executive Review Committeewith risk adjusted, most likely cost, schedule andperformance goals 3
  5. 5. June 2006 Capital Programming Guide Supplement A-11, Part 7Risks ……for each WBS element should be identified,analyzed, and quantified in terms of potential cost to theprogram Risk identification involves analyzing program areas and critical technical elements to identify and document the associated risk Risk analysis involves examining each risk issue to determine the probability of the risk occurring Risk quantification results in the cost, schedule and technical consequences if the risk occurs 4
  6. 6. June 2006 Capital Programming Guide Supplement A-11, Part 7 Program (Project for NASA) Risk-Adjusted Budget (PRB) The cost of the risk occurrence is added to the BAC and theresult of this analysis is a risk adjusted budget The programs (investments) risk-adjusted budget establishesthe baseline for reporting to OMB on program performance The appropriate agency official must ensure the PRB is justifiedbased on risk, and that the agency will fund the program atthat level 5
  7. 7. June 2006 Capital Programming Guide Supplement A-11, Part 7 Risk management begins with evaluating the WBS for cost, schedule and technical risk Program budget, expected outcomes and cost/schedule performance measurements are integrated with risk management 6
  8. 8. Point vs Range Estimates• Being precise about point estimates is next to impossible• However, range estimating is eminently possible• Cost-risk quantification enables the process of developing range estimating• Projects expected to do range estimating in their CADRe Part C LCCE 7
  9. 9. Cost-Risk Assessment & Analysis• Assessment 1. Cost model uncertainty (parametric estimating only) • “Scatter” in data base used to derive estimating equation 2. Input parameter uncertainty (parametric estimating only) • Spread in estimating input parameter values 3. Risk-Driven Key Element Characteristic (KEC) uncertainty • Level of TRL, New Design, Schedule, etc., effects on – Key Engineering Parameter Performance – Key Management Characteristics – Key System Engineering Characteristics • Evaluated using techniques such as the Relative Risk Weighting (RRW) process 4. Correlation uncertainty • Correlation in movement of WBS element costs• Analysis 8 – Convolve all distributions for “S”-curve (CDF)
  10. 10. Cost-Risk Analysis: ConvolutionCORRELATED WBS ELEMENT COST SUMMARY COST DISTRIBUTIONS:DISTRIBUTIONS (note: not necessarily BELL CURVEall triangles): PROBABILITY DENSITY RPE(Power Subsystem ASIC) CER, = + Parameter Input, and KEC’s 100 ∑RPE COST “S”-CURVE Cost-Risk Distributions on 85 RPE + • Lower Level WBS Elements CONFIDENCE 70 • LEVEL 50 + EPS 20 9 RPE ∑RPE COST
  11. 11. Connecting Cost Estimating and EVM (Integrated Cost-Risk)Early id, assess & cost Assess Dataimpacts of medium- and For Model & Build/Updatehigh-risk WBS elements Database Updates Reqmts-Function-in cost estimate (CADRe & ONCE) WBS Matrix (CADRe)Cost estimators Compile End-of-Contractcommunicate WBS Cost-Risk Dataelement risk to project for Evaluation Develop Refmanagers for focused & Analysis Point Cost & 11 Schedulecost-risk management Estimate Update LCCEPost-cost estimate (CADRe) & Medium &tracking of identified Cost / Identify & AssessWBS element risk using Sched-Risk High-Risks WBS Element Risk AssessmentEVM systemKDP CADRe update, 5collection and archiving Do EVM, Developof technical data, cost “S”-curve, Quantify Risk CPR & schedule risk intodata and cost-risk data critical path CADRe Cost /Schedule Costfor cost model analyses, RFP Impactsimprovement etc. DRDsDraft NPR 8000.4Bdescribes this integrated 10cost-risk management
  12. 12. EVM Working Group Draft Contract Performance Report (CPR) DRD Template ( value performance measurement data for government-identified mediumand high-risk WBS elements (see list below), if available and appropriate, shallbe reported on Formats 1 & 2 of the monthly CPR until such time as bothgovernment project management and the contractor agree that they no longerrepresent medium or high risks Power Subsystem ASIC; Solar Power Converter; Pointing & Control System Laser Amplifier/Transmitter; Laser Transmit Antenna; Microwave Receive Antenna; Laser Receive Antenna; Tracking & Control System; Laser Conditioning Receiver; Laser Rectifier/Converter; Flywheel Storage SystemThis reporting on medium and high-risk WBS elements shall be at a level that isadequately sensitive to performance measurement indicators to ensure earliestidentification of cost and schedule problems caused by the source risks (e.g.,level 5, 6, or 7 or just above control account level)Narrative variance analysis is not required for this level of medium and high-riskWBS elementsThe contractor shall identify all known medium and high-risk WBS elementsspecific to his design, if not provided in the list above, and report theirperformance measurement on CPR Formats 1 & 2
  13. 13. Traditional CPR Level 3 Reporting 10% Variance ReportingNormally, if a 10% variance threshold is Power Subsystem breached at Level 3 (a $5M problem) a WBS (CWBS Level 3) Space Solar Power drill-down to lower WBS levels SSP Project (CWBS Level 1) NORMAL REPORTING LEVEL PROJECT commences to locate the source of the ($50M Budget) risk for subsequent scrutiny eventually Bus Flywhe el developing to the high-risk ASICs in the Optical Rectifier <$1M Level 6 CA (CWBS Power Payload AKM Drill-down Subsystem Level 4) from level 3 OBS Drill-Down EPS ADACS TT&C thru level 6 to Level Level 7 ASICs only AFTER OBS Electronics Earth Sensor Sun Sensor Optical Electronics Optical Electronics problem Assembly Level 5 Level 4 WBS Drill-Down Manufacturing Control Control Control Elec Assy Account Account Account Level Control Account (Level 6) Hardware Electronics Control VP/GM Engineering Engineering Assembly Account ASICs Work Packages Software Control Control BCWS Test Engineering Electrical Account Account BCWP ACWP BAC Planning EAC Packages 12
  14. 14. Med/High Risk Reporting 3 High-Risk No-Threshold Variance ReportingA reasonably low enough reporting level is identified, e.g., Level 5, which must Power Subsystem WBS Space Solar Power (CWBS Level 3) SSP Project (CWBS Level 1) be sensitive enough to indicate the NORMAL REPORTING LEVEL PROJECT Level 6 CA cost and schedule problems ($50M Budget) Report up Bus Flywhe where the high-risk ASIC effort is Optical Rectifier el from level 5 located (CWBS Power Payload AKM to capture Subsystem Level 4) level 7 ASICs OBS Reporting EPS ADACS TT&C BEFORE big Level problem Optical OBS Electronics Earth Sensor Sun Sensor Optical Electronics Electronics WBS Reporting Assembly Level 5 Level 4 ($5M Budget) Level Manufacturing Control Control Control Elec Assy Account Account Account Control Account (Level 6) Hardware Electronics Control VP/GM Engineering Engineering Assembly Account ASICs Work Packages Software Control Control BCWS Test Engineering Electrical Account Account BCWP ACWP BAC Planning EAC Packages 133Until risk is no longer a threat or is retired
  15. 15. HIGH RISK CONTROL ACCOUNT with ASIC MITIGATION PLAN WORK PACKAGE EV Techniques 0/100, 50/50, Units Complete, % Complete, Milestones CONTROL ACCT. TITLE: Optical Frequency Demodulator CONTROL ACCOUNT MANAGER: Joe Hamaker BUDGET: $10,000 TIER I MILESTONE CA Start CA COMP WP# WORK DESCRIPTION EV METHOD MONTH 1 MONTH 2 MONTH 3 MONTH 4 MONTH 5 MONTH 6 TOTAL BAC BCWS 1,500 1,500 1 Procure Casing 0/100 BCWP 1,500High-risk BCWS 500 500 1,000WBS element 2 Optical Freq Receiver 50/50 BCWP 500 500 BCWS 600 600 600 600 600 3,000 3 OPT-RF ASICs units complete BCWP 600 600 - 1,200 600 BCWS 50 50 50 150 ASIC MITIGATION 4 PLAN milestone 1 2 3 BCWP 50 - 50 50 BCWS 500 500 500 1,500 5 Integration % completeRisk handling BCWP - 300 1,200work package TOTAL CONTROL ACCOUNT PLAN BCWS 600 650 1,150 1,650 2,600 500 7,150 BCWP 600 650 500 1,250 2,950 1,200 7,150 Schedule Variance month 0 0 -650 -400 350 700 cumulative 0 0 -650 -1,050 -700 0 Actual Costs 700 1,700 1,300 2,300 5,200 2,100 13,300 Cost Variance month -100 -1,050 -800 -1,050 -2,250 -900 cumulative -100 -1,150 -1,950 -3,000 -5,250 -6,150 14 14
  16. 16. Risk Management Metric Measuring the effect of cost-risk management using S-curves 15
  17. 17. S-Curve Improvement Metric Beginning-of-Phase A S-Curve (Starting Point for Risk Management) 100 Phase A 70Confidence Level 50 25 Cost Delta between initial point cost and 70% CL 16
  18. 18. S-Curve Improvement Metric (Ideal) Beginning-of-Phase B S-Curve (RM occurred from Phase A to Phase B) 100 Phase B Phase A 70Confidence Level 50 Potential result from integrated cost-risk using EVM & CADRe 25 New delta between initial Cost point cost and 70% CL 17
  19. 19. S-Curve Improvement Metric (Ideal) Beginning-of-Phase C S-Curve (RM occurred from Phase B to Phase C) 100 Phase C Phase B Phase A 70Confidence Level 50 Target cost probably 25 will increase some, but is now at 70% CL Cost Initial Point Cost Phase C Point Cost 18
  20. 20. S-Curve Metric (Realistic?) Phase B S-Curve (RM from Phase A to Phase B) 100 Phase B S-curve right shift (unknown, unknown- driven) Phase A 70Confidence Level 50 25 19 Cost New delta between initial point cost and 70% CL
  21. 21. S-Curve Improvement Metric (Realistic?) Phase B S-Curve (RM from Phase A to Phase B) 100 S-curve Phase C Phase B right shift 70 Phase AConfidence Level 50 Target cost increases 25 more in the realistic case but still is lower than original 70% Cost New delta between initial point cost and 70% CL 20
  22. 22. Why Do Cost-Risk Analysis?“….these uncertainties exist whetheror not we do the analysis”Statement on risk analysis made by Dr.George Apostolakis, MIT Professor of RiskManagement & expert in nuclear powerprobability risk analysis (PRA), speaking atDec 05 NASA Risk ManagementConference, Orlando, FL. 21
  23. 23. BACKUPS 22
  24. 24. Cost Model and Input Parameter Uncertainty Cost Quantification COMBINED COST MODELING AND CER TECHNICAL RISK Cost ==aa++bXc c Cost bX COST MODELING UNCERTAINTY CostEstimate Historical data point $ Cost estimating relationship TECHNICAL RISK Standard percent error bounds Cost Driver (Weight) Input 23 variable
  25. 25. Key WBS Element Characteristic Uncertainty• Key Engineering Performance Parameters1 (KEPPs) for new electronic component for a S/C • Dynamic load resistance • Operating voltage • Power regulation • ASIC • Radiation resistance • Emissivity • Component mass • Operating temperature range • Operating efficiency • etc. 1 24 The Technology Puzzle: Quantitative Methods for Developing Advanced Aerospace Technology; Liam Sarsfield (RAND)
  26. 26. Key WBS Element Characteristic Uncertainty• Key Management Characteristics (KMCs) • Experience of personnel • Risk management effort levels • Earned Value Management implementation level • Management structure (IPT, functional, matrix, etc.) • etc. 25
  27. 27. Key WBS Element Characteristic Uncertainty• Key System Engineering Characteristics (KSECs) • Level of system engineering expertise • Percentage of system engineering performed early • Tools used for requirement/function allocation • Logistics considerations • Planning, monitoring, measuring, B/C studies, etc. • Percentage of system engineering performed during effort • etc. 26
  28. 28. Correlation • Dr. Stephen Book (MCR) plotted the theoretical underestimation of percent total cost standard deviation (y-axis) when correlation (x-axis) is assumed to be zero rather than its true value, ρ. – In cost estimates we would underestimate % SD ~60%-80% if we ignored correlation and it was actually 0.2 100 n = 1000 n = 100 80 n = 30 Percent Underestimated n = 10 % Underestimated 60 Theoretical Total 40 Cost Standard Deviation 20From: 1999 Cost 27 0Risk Analysis Seminar, 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1Manhattan Beach, CA Actual Correlation