Life Cycle Review - Programmatic    Analysis Lessons Learned    NASA PM Challenge – February 2012      Richard Greathouse,...
Overview•   Programmatic analysis lessons learned in preparing    for a Life-Cycle Review (LCR) will be presented.       ...
Programmatic Analysis                        Criteria                        Rating   RationaleAdequacy of management appr...
4
Data Deliveries•   Documents (Requested of Projects) focusing on:        Project Management Plan, Risk Management Plan, S...
Alignment with and Contributing to Agency Needs, Goals, and    Objectives, Including the   Adequacy of Requirements       ...
Needs, Goals, Objectives and          Requirements Flow-Down•   Mission success criteria identified.•   Clear and consiste...
Adequacy of Management      Approach                         8
Management Approach•   An assessment of management processes including acquisition strategy,    budgeting, tracking, repor...
Management Approach Lessons              Learned• Important to meet with project’s programmatic  team as part of the SRB r...
Alignment of Technical Approach,    as Defined by NPR 7123.1  Entrance and Success Criteria                               ...
Technical Approach•   What are the technical drivers for cost, schedule,    and risk?•   Are technical margins adequate?• ...
Technical Approach –                   Lessons Learned•   Allowing the SRB to participate and have insight    into the ful...
Adequacy of the Integrated Cost  and Schedule Estimate andFunding Strategy in Accordance       with NPD 1000.5            ...
Analysis & Methodology• SRB will assess the project’s cost and schedule estimate.• SRB will evaluate risks and derive impa...
Analysis & Methodology -               Lessons Learned•   Early, frequent, and active communication and    dialogue betwee...
Basis of Estimate•   Where does the project get this information?•   Does it explain “why” the estimate is what it is? (Is...
Basis of Estimate – Cost                    Lessons Learned• The BOE should document the estimating methodology used to  p...
Basis of Estimate Evaluation                                                                                        Define...
Basis of Estimate – Schedule                Lessons Learned (1)•   If 3-point estimates are provided, then the BOE should ...
Basis of Estimate – Schedule                Lessons Learned (2)•   Estimates should not be padded and should reflect requi...
Basis of Estimate – Schedule            Lessons Learned (3)• A good BOE should not only document the basis for the activit...
Estimating - Live Demo Note – these people are not actors; they are professionals!                                        ...
Schedule Analysis -                    Lessons Learned•   Schedules may pass the health checks but still have flaws;    fo...
Reserve Analysis –                    Lessons Learned•   Different NASA centers and organizations apply and    manage rese...
Range Estimate /           Joint Confidence Level (JCL)                 Lessons Learned•   Importance of analysis schedule...
Adequacy/Availability ofResources other than Budget                              27
Resources•   Analysis includes an assessment of workforce    planning, skill mix, retention, obsolescence, and    ramping ...
Resources - Lessons Learned•   A ramp-up or ramp-down of workforce needs to be    consistent with the work flow (i.e. the ...
Adequacy of Risk ManagementApproach and Risk Identification and Mitigation per NPR 8000.4                                 ...
Risk Management Approach•   Assessment of BOEs for any cost or schedule mitigation    plans for the identified risks.•   S...
Risk Management Approach –                  Lessons Learned•   The Risk Management Plan should accurately    reflect the p...
Uncertainty•   Planned schedule activity durations are    estimations; completion dates appear as definite    dates with a...
Uncertainty Lessons Learned•   Uncertainty values are applied to WBS levels within a P/p•   In many cases, project trends ...
Risk•   Known risks, identified by Program, Project or SRB    are normally called “discrete risks” with    corresponding r...
Risk - Lessons Learned•   Projects should be able to quantitatively    evaluate their risks, by identifying the following ...
SRB Risk Worksheet                                                      Likelihood (% chance of occurring)                ...
Parting Words•   It takes one woman nine months to have a baby. It         •   The sooner you begin coding the later you f...
Upcoming SlideShare
Loading in …5
×

Ipao great houseetc-programmatic analysis ll

15,474 views

Published on

Published in: Business, Technology
  • Be the first to comment

  • Be the first to like this

Ipao great houseetc-programmatic analysis ll

  1. 1. Life Cycle Review - Programmatic Analysis Lessons Learned NASA PM Challenge – February 2012 Richard Greathouse, NASA IPAO Heidemarie Borchardt, PMP, NASA IPAO Michele King, Reed Integration Kelly Moses, PE, Reed Integration Robin Smith, Reed Integration
  2. 2. Overview• Programmatic analysis lessons learned in preparing for a Life-Cycle Review (LCR) will be presented.  Standing Review Board (SRB) Timeline  Data Deliveries• Topics that will be covered include the SRB independent analysis process and methodology:  Basis of Estimates (BOE)  Schedule Analysis  Reserve Analysis  Range Estimates and Joint Confidence Levels (JCL)  Resource Analysis  Uncertainty and Risk 2
  3. 3. Programmatic Analysis Criteria Rating RationaleAdequacy of management approach.Alignment with and contribution to Agency needs,goals, and objectives, including the adequacy ofrequirements flow-down.Adequacy of technical approach, as defined by NPR7123.1 entrance and success criteria.Adequacy of the integrated cost and schedule estimateand funding strategy in accordance with NPR 1000/5.Adequacy/availability of resources other than budget.Adequacy of risk management approach and riskidentification and mitigation per NPR 8000.4. 3
  4. 4. 4
  5. 5. Data Deliveries• Documents (Requested of Projects) focusing on:  Project Management Plan, Risk Management Plan, Schedule Management Plan  Work Breakdown Structure (WBS) and WBS Dictionary  Resources Plan (staffing, workforce, facilities)  Project management status reports and EVM Data• Master Project Schedules and P/p Integrated Master Schedule (IMS)  Program runs STAT health check and schedule should be green for delivery  Identification of Critical Path and Schedule Margin/Reserve with basis  Removal of constraints from schedule for purposes of performing SRA  Provide basis for any constraints not removed from schedule  Identification of risk mitigation activities in the schedule  Qualitative uncertainty ratings by WBS (Ref. schedule risk consequence chart)• Risk List  Likelihood and Consequence Probabilities for Risks  Mapping of Risks to Specific Schedule Activities 5
  6. 6. Alignment with and Contributing to Agency Needs, Goals, and Objectives, Including the Adequacy of Requirements Flow-Down 6
  7. 7. Needs, Goals, Objectives and Requirements Flow-Down• Mission success criteria identified.• Clear and consistent requirements.• Requirements flow-down.• Dependencies on other program requirements within NASA or outside. 7
  8. 8. Adequacy of Management Approach 8
  9. 9. Management Approach• An assessment of management processes including acquisition strategy, budgeting, tracking, reporting and control. Analysis will provide insight into trends, problems issues and risks so that mitigation strategies can be developed.• Analysis includes an assessment of the acquisition plans, including major contracts, scope, and contract types.• Cost and schedule baseline and Earned Value Management (EVM) analysis.• Reporting process (per Program Plan): monthly status reports.• Adequacy of staffing, training, communications, information technology and resources.• Review of management processes: Program Plan, Program Commitment Agreement, Work Breakdown Structure (WBS) and WBS Dictionary, Schedule Management Plan. 9
  10. 10. Management Approach Lessons Learned• Important to meet with project’s programmatic team as part of the SRB review process  Risk Manager  Scheduler  Business Manager• Open communication facilitates the review process. SRB experts can provide valuable feedback to P/p prior to LCR. 10
  11. 11. Alignment of Technical Approach, as Defined by NPR 7123.1 Entrance and Success Criteria 11
  12. 12. Technical Approach• What are the technical drivers for cost, schedule, and risk?• Are technical margins adequate?• What is the Technology Readiness Level (TRL) and are there issues in meeting the TRL?• What are the heritage assumptions, rework, and sparing plans which can affect design, manufacturing, and test cycles?• Are test plans / testing adequate? 12
  13. 13. Technical Approach – Lessons Learned• Allowing the SRB to participate and have insight into the full set of instrument and subsystem reviews leading to the Site Review:  Minimizes the prospect of a steep learning curve at the Site Review.  Allows time to discuss any findings among the SRB prior to the Site review.  Provides the SRB with insight into the issues across the project and aids in identification of systemic issues. 13
  14. 14. Adequacy of the Integrated Cost and Schedule Estimate andFunding Strategy in Accordance with NPD 1000.5 14
  15. 15. Analysis & Methodology• SRB will assess the project’s cost and schedule estimate.• SRB will evaluate risks and derive impacts against cost and schedule.• The SRB will employ its expertise to evaluate Program/project (P/p) risks and will require the P/p cost/schedule models to respond to their inputs for risk and uncertainty.• SRB will evaluate the reserve and lien posture of the P/p.• SRB will use the Independent Programmatic Assessment (IPA) to derive impacts of uncertainties and risks to cost and schedule. 15
  16. 16. Analysis & Methodology - Lessons Learned• Early, frequent, and active communication and dialogue between the SRB and the project, consistent with the SRB timeline, facilitates the understanding of the project’s analysis and methodology regarding its integrated cost and schedule estimate. 16
  17. 17. Basis of Estimate• Where does the project get this information?• Does it explain “why” the estimate is what it is? (Is a basis provided? Is it traceable?)• Is the explanation credible? (Is it based on expert opinion, heritage, calculations, etc?)• Is it realistic? Is it based on the complete scope of work?• If it’s cost, is it time-phased? 17
  18. 18. Basis of Estimate – Cost Lessons Learned• The BOE should document the estimating methodology used to prepare the cost and schedule estimates for each WBS element. The content of the BOE depends on the estimating method used. The four most common methods are:  Time and Material Estimates – the BOE should describe in detail how the effort and the duration were determined.  Parametric Estimates – the BOE should describe any cost or schedule models used and how they were applied.  Analogy Estimates – The BOE should describe the analogous system(s) and explain how and why it was used, explain any scaling or adjustments made to the data for the analogy. The BOE should document any historic schedule plan/actual data from analogous missions or projects that were used in the estimate.  Engineering Judgment – The BOE should describe the thought process and justification for an Engineering Judgment type of estimate. 18
  19. 19. Basis of Estimate Evaluation Defined WBS WBS WBS Formal Scope/ Schedule Basis TimeNumber Level Element BOE Traceable Baseline Durations Estimating Methodology Provided Complete Accurate Realistic Phasing Risk Methodology123456 1 Project ----- ----- ----- ----- Summation ----- ----- ----- ----- ----- -----123456 2 Project Management ----- ----- ----- ----- Summation ----- ----- ----- ----- ----- -----123456.01.01 Project Management 3 G G Y R Time & Materials G G Y R G R123456.01.02 Business Management 3 Time & Materials123456.01.03 Risk Management 3 Time & Materials123456.01.04 Procurement Management 3 Time & Materials123456.01.05 Facilities Management 3 Time & Materials123456 2 Systems Engineering ----- ----- ----- ----- Summation ----- ----- ----- ----- ----- -----123456.02.01 Systems Engineering Management 3 Time & Materials123456.02.02 System Requirements 3 Time & Materials123456.02.03 System Interface & Configuration 3 Engineering Build-Up123456.02.05 System Verification and Validation 3 Time & Materials123456.02.06 Trade Studies 3 Time & Materials123456.02.07 Systems Risk Management Plan 3 Time & Materials123456 2 Safety and Mission Assurance ----- ----- ----- ----- Summation ----- ----- ----- ----- ----- -----123456.03.01 Safety & Mission Assurance Management 3 Time & Materials123456.03.02 System Safety 3 Time & Materials123456.03.03 System Reliability 3 Time & Materials123456.03.04 Quality Assurance 3 Time & Materials123456.03.05 Environmental Safety 3 Time & Materials123456 2 Science and Technology ----- ----- ----- ----- Summation ----- ----- ----- ----- ----- -----123456.04.01 Science and Technology Management 3 Analogy123456.04.02 Science Requirements 3 Analogy123456.04.03 Science Development 3 Analogy123456 2 Payloads ----- ----- ----- ----- Summation ----- ----- ----- ----- ----- -----123456.05.01 Payloads Management 3 Time & Materials123456.05.02 Payloads Requirements 3 Time & Materials123456.05.03 Payload #1 Development 3 ----- ----- ----- ----- Summation ----- ----- ----- ----- ----- -----123456.05.03.01Payload #1 Subsystem #1 Development 4 Analogy123456.05.03.02Payload #1 Subsystem #2 Development 4 Analogy123456.05.04 Payload #2 Development 3 ----- ----- ----- ----- Summation ----- ----- ----- ----- ----- -----123456.05.04.01Payload #2 Subsystem #1 Development 4 Parametric CER123456.05.04.02Payload #2 Subsystem #2 Development 4 ----- ----- ----- ----- Summation ----- ----- ----- ----- ----- -----123456.05.04.02.01 5 Payload #2 Subsystem #2 Component #1 Development Parametric CER123456.05.04.02.02 5 Payload #2 Subsystem #2 Component #2 Development ----- ----- ----- ----- Summation ----- ----- ----- ----- ----- -----123456.05.04.02.02.01 #2 Subsystem #2 Component #2 Part #1 Development 6 Payload Engineering Build-Up123456.05.04.02.02.02 #2 Subsystem #2 Component #2 Part #2 Development 6 Payload Engineering Build-Up123456.05.05 Payload Systems Integration & Testing 3 Analogy123456 2 Spacecraft ----- ----- ----- ----- Summation ----- ----- ----- ----- ----- -----123456.06.01 Spacecraft Management 3 Factor123456.06.02 Propulsion 3 Analogy123456.06.03 Architecture 3 Parametric CER123456.06.04 Avionics 3 Parametric CER123456.06.05 Thermal Control 3 Parametric CER123456.06.06 Software 3 Parametric CER123456 2 Mission Operations ----- ----- ----- ----- Summation ----- ----- ----- ----- ----- ----- 19
  20. 20. Basis of Estimate – Schedule Lessons Learned (1)• If 3-point estimates are provided, then the BOE should include the rationale for estimating the 3 points.• The BOE should discuss whether or not the task duration is a function of the level of effort or resources applied to the task, e.g. if working overtime or multiple shifts will accelerate the task.• Activity owner should make the duration estimate for the task.• Durations should be estimated for most likely time values and should be representative of timecard accounting periods or less than 2 months. If durations are not realistic, deliverables and critical path will be incorrect.• LOE or hammock tasks should be clearly identified and justified. 20
  21. 21. Basis of Estimate – Schedule Lessons Learned (2)• Estimates should not be padded and should reflect required effort to complete the activity, resources available.• Activity durations and cost estimates should map to the corresponding WBS; FTEs/WYEs requirements support activity and cost estimate.• Schedule estimates may be based upon historical data, parametric analysis, and subject matter experts.• Document the basis of the estimate: “Documenting the basis for the duration facilitates communication of expectations between activity owners and senior decision makers, and aids in the estimating of future analogous activity durations.” – GAO 21
  22. 22. Basis of Estimate – Schedule Lessons Learned (3)• A good BOE should not only document the basis for the activity duration, but also: Mandatory logic Constrained dates Leads and Lags Calendars Resources• Scheduler documents BOE rationale in MS Project text field.• Durations of other missions can be used as part of the BOE. 22
  23. 23. Estimating - Live Demo Note – these people are not actors; they are professionals! 23
  24. 24. Schedule Analysis - Lessons Learned• Schedules may pass the health checks but still have flaws; focusing solely on schedule mechanics will not fix a poor quality schedule.• Schedules that fail the health checks should be corrected prior to performing integrated cost and schedule analysis (Note: If the true critical path cannot be identified in the schedule before the range estimate or JCL activity, then the resulting critical path will not be credible).• Analysis schedules should be developed to a level that will incorporate any identified risks that the project is carrying.• Schedule trends for milestones, deliverables, and interfaces need to be analyzed and monitored. 24
  25. 25. Reserve Analysis – Lessons Learned• Different NASA centers and organizations apply and manage reserve differently.• It is important to understand the basis for the reserves and how the P/p is managing reserve.• Margin or reserve tasks should also be clearly identified and the method used to estimate margin should be explained.• References:  Schedule Management Handbook  JPL Rules  GSFC Gold Rules 25
  26. 26. Range Estimate / Joint Confidence Level (JCL) Lessons Learned• Importance of analysis schedule reflecting true critical path, logical linkages and enough detail to map risks and cost elements.• Importance of working with cost, schedule and risk analysts.• Tools should not affect the process.• Understanding and communicating results. 26
  27. 27. Adequacy/Availability ofResources other than Budget 27
  28. 28. Resources• Analysis includes an assessment of workforce planning, skill mix, retention, obsolescence, and ramping issues.• Assessment of availability of facilities, availability, costs and supporting infrastructure.• Analysis of any mission partner’s potential impact to cost and schedule. 28
  29. 29. Resources - Lessons Learned• A ramp-up or ramp-down of workforce needs to be consistent with the work flow (i.e. the project schedule/project plan).• Projects often rely on the same facilities for testing. Schedule slips in one project can cause delays for other projects. In many cases, problems may be avoided if work-arounds are identified early. 29
  30. 30. Adequacy of Risk ManagementApproach and Risk Identification and Mitigation per NPR 8000.4 30
  31. 31. Risk Management Approach• Assessment of BOEs for any cost or schedule mitigation plans for the identified risks.• SRB supported assessment of the risks which may include new risks that are identified by the SRB with associated cost and schedule impacts.• SRB supported assessment of the risks which may included adjustments to assumptions made by the P/p with associated cost and scheduled impacts (TRL and heritage).• Assessment of the reserve strategy and the current state of the reserves in relation to where P/p are in life cycle.• Assessment of whether risk has been accounted for in P/p cost, schedule or JCL process. 31
  32. 32. Risk Management Approach – Lessons Learned• The Risk Management Plan should accurately reflect the process that is in place, including:  How the P/p/subsystem levels coordinate and interact.  Definitions of the 5x5 scoring criteria.  Processes for identifying, tracking, mitigating, controlling, and documenting risks.• Risks need to be mapped to the appropriate schedule task(s). 32
  33. 33. Uncertainty• Planned schedule activity durations are estimations; completion dates appear as definite dates with associated definitive costs.  Schedule finish dates and costs are not facts, but projections with associated uncertainty.  Generic risks captured in the project’s risk list, such as “Instrument Schedule,” cannot typically be mapped to specific work activities in the schedule; these types of risks are actually “uncertainty” and should be captured as such when performing a range estimate or JCL. 33
  34. 34. Uncertainty Lessons Learned• Uncertainty values are applied to WBS levels within a P/p• In many cases, project trends (performance over time) can be a good indication of the uncertainty associated with a particular element in a project, especially if discrete risks cannot be identified. Schedule - Uncertainty Distributions• Understanding the P/p’s Risk Level Low 10% Mode 50% High 90% 0.984 1.000 1.070 uncertainty gives the SRB Low Low+ 0.954 1.000 1.240 members a basis for Medium 0.924 1.000 1.390 assessing uncertainty Medium+ 0.893 1.000 1.550 durations. High 0.863 1.000 1.710 High+ 0.832 1.000 2.020 Very High 0.802 1.000 2.350 Very High+ 0.771 1.000 3.370 Extra High 0.741 1.000 4.384 34
  35. 35. Risk• Known risks, identified by Program, Project or SRB are normally called “discrete risks” with corresponding risk mitigation activities.  Mitigation may require funds and schedule and the SRB needs to understand the impact on schedule/cost reserves.  Discrete risks should mapped to the schedule activities they affect and be assigned associated probability and impact values for cost and schedule. 35
  36. 36. Risk - Lessons Learned• Projects should be able to quantitatively evaluate their risks, by identifying the following (reference SRB risk chart):  Probability of Risk Occurrence  Best Case Schedule (Work Days)  Most Likely Schedule (Work Days)  Worst Case Schedule (Work Days)  Best Case Cost ($)  Most Likely Cost ($)  Worst Case Schedule ($)• *Note: A basis of estimate should be provided with each of the above quantities. 36
  37. 37. SRB Risk Worksheet Likelihood (% chance of occurring) Consequence (Duration Beyond Already Scheduled [days])Risk TypeIdentifiedby Project ml min max Risk Associated Specific Activities (Cost, Rank Risk ID Title Statement L % likelihood rationale C min rationale (most ml rationale max rationale (optimisitic) (pessimistic) Owner WBS # ImpactedSchedule, likely) orTechnical) 1 2 3 4 5 6 7 8 9 10 11 12 13Risk TypeIdentified by SRB ml SRB- min max Risk Associated Specific Activities (Cost, Title Statement L % likelihood rationale C min rationale (most ml rationale max rationale Risk ID (optimisitic) (pessimistic) Owner WBS # ImpactedSchedule, likely) orTechnical) SRB-1 SRB-2 SRB-3 SRB-4 SRB-5 SRB-6 SRB-7 SRB-8 Understanding the project’s risk assessment gives the SRB members a basis for their assessment. 37
  38. 38. Parting Words• It takes one woman nine months to have a baby. It • The sooner you begin coding the later you finish. cannot be done in one month by impregnating nine • Anything that can be changed will be changed until women (although it is more fun trying). * Spanish there is no time left to change anything. Proverb • Change is inevitable – except from vending• The same work under the same conditions will be machines. Robert C. Gallagher estimated differently by ten different estimators or by • The person who says it will take the longest and cost one estimator at ten different times. the most is the only one with a clue how to do the• Any project can be estimated accurately (once its job. completed). • If youre 6 months late on a milestone due next week but nevertheless really believe you can make it,• A change freeze is like the abominable snowman: it youre a project manager. is a myth and would anyway melt when heat is • If you don’t know where you are going any road will applied. take you there. Lewis Carroll (Alice in Wonderland)• Right answers to wrong questions are just as wrong • If you fail to plan you are planning to fail. as wrong answers to right questions. • If you dont attack the risks, the risks will attack you.• What you dont know hurts you. • A little risk management saves a lot of fan cleaning.• Theres never enough time to do it right first time but • The sooner you get behind schedule, the more time theres always enough time to go back and do it you have to make it up. again. • A badly planned project will take three times longer• Estimators do it in groups - bottom up and top down. than expected - a well planned project only twice as long as expected.• Good estimators arent modest: if its huge they say so. MHR• If project content is allowed to change freely the rate of change will exceed the rate of progress.Reference: http://www.project-training-k.freeserve.co.uk/ 38

×