DEVELOPMENT OF UNCERTAINTY FACTORS FROM        HISTORICAL NASA PROJECTS                             Dr. Tahani Amer, OE/IP...
Project Background          • Uncertainty analysis should be performed to capture the            program risks.          •...
Data Collection Methodology              • NASA Fiscal Year Budget Estimates              • Cost Analysis Data Requirement...
Data Collection Model                                             Budget Report                 CADRe Report              ...
Source of Data    FY 2009 Budget    February 4, 2008                                                  Amer DENG           ...
Data Sampling Criteria              • Difficulty in collecting accurate data – limited dataset              • Criteria    ...
Summary of NASA Missions Investigated                                                                                     ...
Initial & Final Cost for 50 Completed                           Projects and Cost Growth PercentageNASA Program Management...
60 NASA Missions - Historical DataNASA Program Management Challenge 2012   This briefing is for status only and does not r...
Mission Cost Growth HistogramNASA Program Management Challenge 2012   This briefing is for status only and does not repres...
NASA & DOD Distribution                        14                        12                        10    Number of Project...
60 NASA Missions                            14                            12                            10        Number o...
Reasons for Cost Growth            • The major categories for cost growth:                   – Overly optimistic and unrea...
Uncertainly Factors from non-NASA Missions                   Aerospace                                                    ...
NASA Data & Air Force UFNASA Program Management Challenge 2012        This briefing is for status only and does not repres...
NASA Data & Aerospace UFNASA Program Management Challenge 2012        This briefing is for status only and does not repres...
NASA Data & BAH UFNASA Program Management Challenge 2012     This briefing is for status only and does not represent compl...
Development of Distribution-54NASA Program Management Challenge 2012    This briefing is for status only and does not repr...
Development of Distribution-44                                  0.12                                  0.11                ...
More Test Runs                       •    All data including Project A & B (A- Data), 54 missions.                       •...
Selected Distribution    • The lognormal distribution is the most suitable distribution that      the cost estimator can u...
Recommended UFs                                                               Uncertainty              Level of Risk      ...
UFs w/Conservative High Risk LevelNASA Program Management Challenge 2012   This briefing is for status only and does not r...
UFs w/Aggressive Moderate Risk LevelNASA Program Management Challenge 2012   This briefing is for status only and does not...
Guideline to UFs              • Cost Analyst Judgment              • Mission Complexity: Number of Instruments,           ...
Comparison of UFsNASA Program Management Challenge 2012    This briefing is for status only and does not represent complet...
UFs ------Test Case           – Project A---               • Development Cost as of 2006 ~ $969 M               • Developm...
UFs Test Case                         970             1.2   1.4          1.8           2.2           2.6            2.8   ...
Conclusions            • This research provides an important contribution to the              discipline of cost estimatin...
Point Taken             • The greatest benefit of this research is providing a               proactive approach to make a ...
Future Work               • Increase the number of historical missions captured in                 the database.          ...
BackupNASA Program Management Challenge 2012   This briefing is for status only and does not represent complete release of...
Upcoming SlideShare
Loading in …5
×

Amer v2

15,111
-1

Published on

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
15,111
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
19
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Amer v2

  1. 1. DEVELOPMENT OF UNCERTAINTY FACTORS FROM HISTORICAL NASA PROJECTS Dr. Tahani Amer, OE/IPAO NASA-HQ Dr. Will Jarvis, OE/IPAO NASA-HQ Dr. Ariel Pinto, Old Dominion UniversityNASA Program Management Challenge 2012 This briefing is for status only and does not represent complete release of usage.
  2. 2. Project Background • Uncertainty analysis should be performed to capture the program risks. • NASA has been using available uncertainty factors from Aerospace, Air Force, and Booz Allen Hamilton. • NASA has no insight into the development of these factors, which can lead to unrealistic risks in most NASA projects. • SRB analysis requires cost growth factors that are transparent and based on recent NASA experience. • Better estimation and prediction of missions cost is needed early on.NASA Program Management Challenge 2012 This briefing is for status only and does not represent complete release of usage. 2
  3. 3. Data Collection Methodology • NASA Fiscal Year Budget Estimates • Cost Analysis Data Requirement (CADRe) • Government Accountability Office Reports • NASA Personal- IPAO/CAD/SMDNASA Program Management Challenge 2012 This briefing is for status only and does not represent complete release of usage. 3
  4. 4. Data Collection Model Budget Report CADRe Report Analysis- Recheck Data Set Assess Assess Assess Analysis- Recheck GAO ReportNASA Program Management Challenge 2012 This briefing is for status only and does not represent complete release of usage. 4
  5. 5. Source of Data FY 2009 Budget February 4, 2008 Amer DENG 5NASA Program Management Challenge 2012 This briefing is for status only and does not represent complete release of usage. 5
  6. 6. Data Sampling Criteria • Difficulty in collecting accurate data – limited dataset • Criteria – Science Mission Projects – Developmental Phase – Only cost data – Use CARDe data as final for completed Missions – For active missions: used in search of credible earliest available and latest available estimates. Errors probably exist because some projects are still in development and continue to evolveNASA Program Management Challenge 2012 This briefing is for status only and does not represent complete release of usage. 6
  7. 7. Summary of NASA Missions Investigated Active Completed Missions Missions NEAR MER HESSI EOS-Aqua AIM JUNO LUNAR MRO GALEX EOS-Aura DAWN AQUARIUS PROSPECTOR FAST SWIFT LANDSAT-7 PHOENIX LDCM GENESIS SWAS GRACE TRMM GLAST NPP MESSENGER TRACE CLOUDSAT TIMED KEPLER GPM MARS WIRE CALIPSO GRAVITY SDO MMS PATHFINDER DS-1 PROBE B WISE JWST ACE STARDUST EO-1 THEMIS NEW MSL FUSE CONTOUR SIRTF HETE-II HORIZONS GLORY IMAGE STEREO GRAIL DEEP IMPACT SORCE LRO MAP MGS ICESAT OCO MCO/MPLNASA Program Management Challenge 2012 This briefing is for status only and does not represent complete release of usage. 7
  8. 8. Initial & Final Cost for 50 Completed Projects and Cost Growth PercentageNASA Program Management Challenge 2012 This briefing is for status only and does not represent complete release of usage. 8
  9. 9. 60 NASA Missions - Historical DataNASA Program Management Challenge 2012 This briefing is for status only and does not represent complete release of usage. 9
  10. 10. Mission Cost Growth HistogramNASA Program Management Challenge 2012 This briefing is for status only and does not represent complete release of usage. 10
  11. 11. NASA & DOD Distribution 14 12 10 Number of Project 8 6 4 2 0 -21-0% 0 -10% 10-20% 20-30% 30-40% 40-50% 50-60% 60-70% >70% Permission from McCrillis, DODNASA Program Management Challenge 2012 This briefing is for status only and does not represent complete release of usage. 11
  12. 12. 60 NASA Missions 14 12 10 Number of Project 8 6 4 2 0 - 21-0% 0 -10% 10-20% 20-30% 30-40% 40-50% 50-60% 60-70% >70%NASA Program Management Challenge 2012 This briefing is for status only and does not represent complete release of usage. 12
  13. 13. Reasons for Cost Growth • The major categories for cost growth: – Overly optimistic and unrealistic initial cost estimates – Project instability and funding issues – Problems with development of instruments and other spacecraft technology – Launch service issues – “Stuff” HappensNASA Program Management Challenge 2012 This briefing is for status only and does not represent complete release of usage. 13
  14. 14. Uncertainly Factors from non-NASA Missions Aerospace Air Force Level Optimistic Most Likely Pessimistic Low 0.95 1 1.1 Low Plus 0.96 1 1.23 Moderate 0.97 1 1.36 Moderate Plus 0.98 1 1.49 High 0.98 1 1.61 High Plus 0.99 1 1.74 Very High 1 1 1.87 Very High Plus 1 1 2 Booz Allen HamiltonNASA Program Management Challenge 2012 This briefing is for status only and does not represent complete release of usage. 14
  15. 15. NASA Data & Air Force UFNASA Program Management Challenge 2012 This briefing is for status only and does not represent complete release of usage. 15
  16. 16. NASA Data & Aerospace UFNASA Program Management Challenge 2012 This briefing is for status only and does not represent complete release of usage. 16
  17. 17. NASA Data & BAH UFNASA Program Management Challenge 2012 This briefing is for status only and does not represent complete release of usage. 17
  18. 18. Development of Distribution-54NASA Program Management Challenge 2012 This briefing is for status only and does not represent complete release of usage. 18
  19. 19. Development of Distribution-44 0.12 0.11 0.1 0.09 0.08 0.07 PDF 0.06 0.05 0.04 0.03 0.02 0.01 0 0 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 Cost Growth (%) Lognormal µ= 3.03 F(X) = σ= 1.08NASA Program Management Challenge 2012 This briefing is for status only and does not represent complete release of usage. 19
  20. 20. More Test Runs • All data including Project A & B (A- Data), 54 missions. • All data excluding Project A & B (B- Data), 52 missions. • All completed missions (C-Data), 44 missions. • All data less than 100% cost growth (D- Data). 0.12 Type of Data µ =Mean σ= Std 0.11 Dev. 0.1 0.09 A - Data 3.01 1.18 0.08 20.3 % 0.07 B - Data 2.92= 1.11 PDF 0.06 18.5 % 0.05 C - Data 3.03 1.08 0.04 0.03 Mean=37.08 0.02 Median= 20.7 0.01 Std. Dev.= 55.14 0 0 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 D-Data 2.85 1.07 Cost Growth (%) 17.3% October 7, 2011 LognormalNASA Program Management Challenge 2012 This briefing is for status only and does not represent complete release of usage. 20
  21. 21. Selected Distribution • The lognormal distribution is the most suitable distribution that the cost estimator can use to perform an uncertainty analysis for the NASA data. • The quality of the fit of two tests: – The Kolmogorov-test 0.032 – The Anderson-Darling test 0.028 0.024 0.02 PDF 0.016 0.012 0.008 0.004 0 0 50 100 Cost Growth (%) Lognormal (1.08; 3.03)NASA Program Management Challenge 2012 This briefing is for status only and does not represent complete release of usage. 21
  22. 22. Recommended UFs Uncertainty Level of Risk Factors No- Risk Adjusted Conservative Aggressive Moderate (10-30%) 1 1.1 1.3 High (30-75%) 1 1.3 1.75 Very High (>75%) 1 1.75 2.5NASA Program Management Challenge 2012 This briefing is for status only and does not represent complete release of usage. 22
  23. 23. UFs w/Conservative High Risk LevelNASA Program Management Challenge 2012 This briefing is for status only and does not represent complete release of usage. 23
  24. 24. UFs w/Aggressive Moderate Risk LevelNASA Program Management Challenge 2012 This briefing is for status only and does not represent complete release of usage. 24
  25. 25. Guideline to UFs • Cost Analyst Judgment • Mission Complexity: Number of Instruments, Technical Development • Heritage Level • Partners Performance • Mission Visibility: International & Commercial • Similarity of MissionsNASA Program Management Challenge 2012 This briefing is for status only and does not represent complete release of usage. 25
  26. 26. Comparison of UFsNASA Program Management Challenge 2012 This briefing is for status only and does not represent complete release of usage. 26
  27. 27. UFs ------Test Case – Project A--- • Development Cost as of 2006 ~ $969 M • Development Cost as of 2011 ~ $1.8 B • Apply NUFs: – Make the following assumptions: » High Risk Mission (10 instruments, new Technology, Planetary) » Aggressive Risk » Select a range from NUFs, Table 20, 1.75-2.5 • Expected development estimate range: 1.69- $2.42 B • Other factors estimates: – AF (1.61- 1.74) $1.56B- $1.69 B, which is lower than the current actual development cost. – Aerospace (2.05-2.5) $1.99- $2.52 B, which is much higher then expected estimate from the low range. Note: LCC is @$2.5B. – BAH (1.9-2.10) $1.84- $2.03 B, which is might be closer to the expected development cost . Note: see section 3.3 for these factors interpolation from BAH table.NASA Program Management Challenge 2012 This briefing is for status only and does not represent complete release of usage. 27
  28. 28. UFs Test Case 970 1.2 1.4 1.8 2.2 2.6 2.8 Current EST. $1.69- $2.42B . UF $1.56B- $1.69B AF $1.99- $2.52B AS $1.84- $2.03B BAHNASA Program Management Challenge 2012 This briefing is for status only and does not represent complete release of usage. 28
  29. 29. Conclusions • This research provides an important contribution to the discipline of cost estimating. • It has developed a solid database of actual cost growth history and adds some statistical rigor to the derivation of cost growth factors based on the data. • Thoughtful execution and effective monitoring of the NUFs will improve the cost estimating. • Additionally, it is expected that this work will be referenced often for independent cost estimates, correction of advocacy-bias in project-generated estimates, and other programmatic work.NASA Program Management Challenge 2012 This briefing is for status only and does not represent complete release of usage. 29
  30. 30. Point Taken • The greatest benefit of this research is providing a proactive approach to make a quick cost estimating as in back of envelope by: – Program Managers – Decision Makers – Congressional Staffer – Cost analysts – Awareness and reference to Standard Review BoardNASA Program Management Challenge 2012 This briefing is for status only and does not represent complete release of usage. 30
  31. 31. Future Work • Increase the number of historical missions captured in the database. • Validate UFs with more available actual cost data. • Test the UFs for missions in various life-cycle phases. • Develop a more adaptable tool for NUFs. • Accept the UFs by the cost analysis’ community for implementation. • Develop Agency standard.NASA Program Management Challenge 2012 This briefing is for status only and does not represent complete release of usage. 31
  32. 32. BackupNASA Program Management Challenge 2012 This briefing is for status only and does not represent complete release of usage. 32
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×