Programmatic analysts are either SRB members or consultants to the SRB.Analysts take SRB inputs for uncertainty and risk to perform analysis.
Icons at bottom explained:We know it is sportyThings start shifting and slipping, always happensWe get to the point where we are trying to slow down time and fit more things inIt is nearly panic time and we are trying to STOP the flow and the bleeding!Timeline is shot to hell at this point!WE are FREAKING out, up late at night, stressed and have a direct line of caffeine into our veins because we don’t have time to pick up a cup and drink!
Late delivery of requested data products cause anxiety and frustration. It is difficult for the SRB is form initial observations on cost and schedule prior to the Site Review if the programmatic analysis is not within 95% of completion by the time of the review.SRB needs time to become familiar with P/p.
Examples we’ve seen:Using the Center’s Risk Management Plan template without tailoring it to the Project does not a Risk Management Plan make.Can “project management” be a strength if the project is late on schedule and over on cost?
Everyone puts on lipstick; Rich puts on chapstick.
Examples we’ve seen:Holding a half-day working session with the entire SRB to discuss the impact of the technical risks on the project’s programmatics provides a high level of understanding and concurrence on the JCL results.Points to get across: We understand that we are not the only “review team” that the project has to deal with. That is why the data we request include standard products as required by 7120.5.
The NASA Office of Inspector General (OIG) issued a report in November 2011, which identified project management as one of NASA’s management challenges for the year. The report recaps major OIG and GAO findings from the recent past and urges more robust cost and schedule estimating and tracking. IPAO conducts forensics analysis on projects and findings support the GAO findings as well.Note that GAO is coming out with a revised Cost Estimation Handbook this month – lots of information on schedule and probabilistic risk analysis.
Examples we’ve seen: Projects often have some form of a BoE for cost, but not for schedule.According to Kelley Cyr:The BOE should document the estimating methodology used to prepare the cost and schedule estimates for each WBS element. The content of the BOE depends on the estimating method used. The four most common methods are: Time and material estimates - the BOE should describe in detail how the effort and the duration were determined. Parametric estimates - the BOE should describe any cost or schedule models used and how they were applied. Analogy Estimates - The BOE should describe the analogous system(s) and explain how and why it was used, explain any scaling or adjustments made to the data for the analogy. The BOE should document any historic schedule plan/actual data from analogous missions or projects that were used in the estimate.Engineering Judgment - The BOE should describe the thought process and justification for an Engineering Judgment type of estimate.If 3-point estimates are provided, then the BOE should include the rationale for estimating the 3 points. LOE or hammock tasks should be clearly identified and justified. Margin or reserve tasks should also be clearly identified and the method used to estimate margin should be explained. The BOE should discuss whether or not the task duration is a function of the level of effort or resources applied to the task, e.g. if working overtime or multiple shifts will accelerate the task. In addition to documenting the rationale for the task duration estimates, I think that a good schedule BOE should also document the rationale for the schedule logic, sequencing, soft and hard constraints, and leads/lags.
Rob Strain calls in the JWST Business Analyst and asks “What is the life cycle cost estimate of JWST?” The JWST Business Analyst replies “8.8B.” The Administrator asks “$8B exactly?” The Business Analyst looks at the Administrator incredulously and says “Yes, $8.8B, exactly.” Then the Administrator calls in Ron Larson and asks the same question. “What is the life cycle cost estimate of JWST?” Ron says, “On average, $9B – give or take fifteen percent, but on average, $9B.” Then the Administrator calls in the JWST SRB Chair and poses the same question. “What is the life cycle cost estimate of JWST?” The JWST SRB Chair gets up, locks the door, closes the shade, sits down next to the interviewer and says, “What do you want it to equal?”KM-I vote, as we act this out, Rich is the Administrator, an Angel is the Business Analyst, Another Angel plays the Cost analyst, and Heide and/or one/two other angels play the part of an SRB Chair/Bubbas. We need a fake door for this scene.BADGES
When updates and changes are made to the schedule, it is important to rerun the health checks and critical path analysis.
Examples we’ve seen:Projects sometimes include risks like “Instrument Schedule” as a risk to the overall project. This risk is generic and cannot be adequately mapped to specific work in the schedule. These types of risks are actually “uncertainty” and should be captured as such when doing a range estimate or JCL.
Sample risk chart attached.
Ipao great houseetc-programmatic analysis ll
Life Cycle Review - Programmatic Analysis Lessons Learned NASA PM Challenge – February 2012 Richard Greathouse, NASA IPAO Heidemarie Borchardt, PMP, NASA IPAO Michele King, Reed Integration Kelly Moses, PE, Reed Integration Robin Smith, Reed Integration
Overview• Programmatic analysis lessons learned in preparing for a Life-Cycle Review (LCR) will be presented. Standing Review Board (SRB) Timeline Data Deliveries• Topics that will be covered include the SRB independent analysis process and methodology: Basis of Estimates (BOE) Schedule Analysis Reserve Analysis Range Estimates and Joint Confidence Levels (JCL) Resource Analysis Uncertainty and Risk 2
Programmatic Analysis Criteria Rating RationaleAdequacy of management approach.Alignment with and contribution to Agency needs,goals, and objectives, including the adequacy ofrequirements flow-down.Adequacy of technical approach, as defined by NPR7123.1 entrance and success criteria.Adequacy of the integrated cost and schedule estimateand funding strategy in accordance with NPR 1000/5.Adequacy/availability of resources other than budget.Adequacy of risk management approach and riskidentification and mitigation per NPR 8000.4. 3
Data Deliveries• Documents (Requested of Projects) focusing on: Project Management Plan, Risk Management Plan, Schedule Management Plan Work Breakdown Structure (WBS) and WBS Dictionary Resources Plan (staffing, workforce, facilities) Project management status reports and EVM Data• Master Project Schedules and P/p Integrated Master Schedule (IMS) Program runs STAT health check and schedule should be green for delivery Identification of Critical Path and Schedule Margin/Reserve with basis Removal of constraints from schedule for purposes of performing SRA Provide basis for any constraints not removed from schedule Identification of risk mitigation activities in the schedule Qualitative uncertainty ratings by WBS (Ref. schedule risk consequence chart)• Risk List Likelihood and Consequence Probabilities for Risks Mapping of Risks to Specific Schedule Activities 5
Alignment with and Contributing to Agency Needs, Goals, and Objectives, Including the Adequacy of Requirements Flow-Down 6
Needs, Goals, Objectives and Requirements Flow-Down• Mission success criteria identified.• Clear and consistent requirements.• Requirements flow-down.• Dependencies on other program requirements within NASA or outside. 7
Management Approach• An assessment of management processes including acquisition strategy, budgeting, tracking, reporting and control. Analysis will provide insight into trends, problems issues and risks so that mitigation strategies can be developed.• Analysis includes an assessment of the acquisition plans, including major contracts, scope, and contract types.• Cost and schedule baseline and Earned Value Management (EVM) analysis.• Reporting process (per Program Plan): monthly status reports.• Adequacy of staffing, training, communications, information technology and resources.• Review of management processes: Program Plan, Program Commitment Agreement, Work Breakdown Structure (WBS) and WBS Dictionary, Schedule Management Plan. 9
Management Approach Lessons Learned• Important to meet with project’s programmatic team as part of the SRB review process Risk Manager Scheduler Business Manager• Open communication facilitates the review process. SRB experts can provide valuable feedback to P/p prior to LCR. 10
Alignment of Technical Approach, as Defined by NPR 7123.1 Entrance and Success Criteria 11
Technical Approach• What are the technical drivers for cost, schedule, and risk?• Are technical margins adequate?• What is the Technology Readiness Level (TRL) and are there issues in meeting the TRL?• What are the heritage assumptions, rework, and sparing plans which can affect design, manufacturing, and test cycles?• Are test plans / testing adequate? 12
Technical Approach – Lessons Learned• Allowing the SRB to participate and have insight into the full set of instrument and subsystem reviews leading to the Site Review: Minimizes the prospect of a steep learning curve at the Site Review. Allows time to discuss any findings among the SRB prior to the Site review. Provides the SRB with insight into the issues across the project and aids in identification of systemic issues. 13
Adequacy of the Integrated Cost and Schedule Estimate andFunding Strategy in Accordance with NPD 1000.5 14
Analysis & Methodology• SRB will assess the project’s cost and schedule estimate.• SRB will evaluate risks and derive impacts against cost and schedule.• The SRB will employ its expertise to evaluate Program/project (P/p) risks and will require the P/p cost/schedule models to respond to their inputs for risk and uncertainty.• SRB will evaluate the reserve and lien posture of the P/p.• SRB will use the Independent Programmatic Assessment (IPA) to derive impacts of uncertainties and risks to cost and schedule. 15
Analysis & Methodology - Lessons Learned• Early, frequent, and active communication and dialogue between the SRB and the project, consistent with the SRB timeline, facilitates the understanding of the project’s analysis and methodology regarding its integrated cost and schedule estimate. 16
Basis of Estimate• Where does the project get this information?• Does it explain “why” the estimate is what it is? (Is a basis provided? Is it traceable?)• Is the explanation credible? (Is it based on expert opinion, heritage, calculations, etc?)• Is it realistic? Is it based on the complete scope of work?• If it’s cost, is it time-phased? 17
Basis of Estimate – Cost Lessons Learned• The BOE should document the estimating methodology used to prepare the cost and schedule estimates for each WBS element. The content of the BOE depends on the estimating method used. The four most common methods are: Time and Material Estimates – the BOE should describe in detail how the effort and the duration were determined. Parametric Estimates – the BOE should describe any cost or schedule models used and how they were applied. Analogy Estimates – The BOE should describe the analogous system(s) and explain how and why it was used, explain any scaling or adjustments made to the data for the analogy. The BOE should document any historic schedule plan/actual data from analogous missions or projects that were used in the estimate. Engineering Judgment – The BOE should describe the thought process and justification for an Engineering Judgment type of estimate. 18
Basis of Estimate Evaluation Defined WBS WBS WBS Formal Scope/ Schedule Basis TimeNumber Level Element BOE Traceable Baseline Durations Estimating Methodology Provided Complete Accurate Realistic Phasing Risk Methodology123456 1 Project ----- ----- ----- ----- Summation ----- ----- ----- ----- ----- -----123456 2 Project Management ----- ----- ----- ----- Summation ----- ----- ----- ----- ----- -----123456.01.01 Project Management 3 G G Y R Time & Materials G G Y R G R123456.01.02 Business Management 3 Time & Materials123456.01.03 Risk Management 3 Time & Materials123456.01.04 Procurement Management 3 Time & Materials123456.01.05 Facilities Management 3 Time & Materials123456 2 Systems Engineering ----- ----- ----- ----- Summation ----- ----- ----- ----- ----- -----123456.02.01 Systems Engineering Management 3 Time & Materials123456.02.02 System Requirements 3 Time & Materials123456.02.03 System Interface & Configuration 3 Engineering Build-Up123456.02.05 System Verification and Validation 3 Time & Materials123456.02.06 Trade Studies 3 Time & Materials123456.02.07 Systems Risk Management Plan 3 Time & Materials123456 2 Safety and Mission Assurance ----- ----- ----- ----- Summation ----- ----- ----- ----- ----- -----123456.03.01 Safety & Mission Assurance Management 3 Time & Materials123456.03.02 System Safety 3 Time & Materials123456.03.03 System Reliability 3 Time & Materials123456.03.04 Quality Assurance 3 Time & Materials123456.03.05 Environmental Safety 3 Time & Materials123456 2 Science and Technology ----- ----- ----- ----- Summation ----- ----- ----- ----- ----- -----123456.04.01 Science and Technology Management 3 Analogy123456.04.02 Science Requirements 3 Analogy123456.04.03 Science Development 3 Analogy123456 2 Payloads ----- ----- ----- ----- Summation ----- ----- ----- ----- ----- -----123456.05.01 Payloads Management 3 Time & Materials123456.05.02 Payloads Requirements 3 Time & Materials123456.05.03 Payload #1 Development 3 ----- ----- ----- ----- Summation ----- ----- ----- ----- ----- -----123456.05.03.01Payload #1 Subsystem #1 Development 4 Analogy123456.05.03.02Payload #1 Subsystem #2 Development 4 Analogy123456.05.04 Payload #2 Development 3 ----- ----- ----- ----- Summation ----- ----- ----- ----- ----- -----123456.05.04.01Payload #2 Subsystem #1 Development 4 Parametric CER123456.05.04.02Payload #2 Subsystem #2 Development 4 ----- ----- ----- ----- Summation ----- ----- ----- ----- ----- -----123456.05.04.02.01 5 Payload #2 Subsystem #2 Component #1 Development Parametric CER123456.05.04.02.02 5 Payload #2 Subsystem #2 Component #2 Development ----- ----- ----- ----- Summation ----- ----- ----- ----- ----- -----123456.05.04.02.02.01 #2 Subsystem #2 Component #2 Part #1 Development 6 Payload Engineering Build-Up123456.05.04.02.02.02 #2 Subsystem #2 Component #2 Part #2 Development 6 Payload Engineering Build-Up123456.05.05 Payload Systems Integration & Testing 3 Analogy123456 2 Spacecraft ----- ----- ----- ----- Summation ----- ----- ----- ----- ----- -----123456.06.01 Spacecraft Management 3 Factor123456.06.02 Propulsion 3 Analogy123456.06.03 Architecture 3 Parametric CER123456.06.04 Avionics 3 Parametric CER123456.06.05 Thermal Control 3 Parametric CER123456.06.06 Software 3 Parametric CER123456 2 Mission Operations ----- ----- ----- ----- Summation ----- ----- ----- ----- ----- ----- 19
Basis of Estimate – Schedule Lessons Learned (1)• If 3-point estimates are provided, then the BOE should include the rationale for estimating the 3 points.• The BOE should discuss whether or not the task duration is a function of the level of effort or resources applied to the task, e.g. if working overtime or multiple shifts will accelerate the task.• Activity owner should make the duration estimate for the task.• Durations should be estimated for most likely time values and should be representative of timecard accounting periods or less than 2 months. If durations are not realistic, deliverables and critical path will be incorrect.• LOE or hammock tasks should be clearly identified and justified. 20
Basis of Estimate – Schedule Lessons Learned (2)• Estimates should not be padded and should reflect required effort to complete the activity, resources available.• Activity durations and cost estimates should map to the corresponding WBS; FTEs/WYEs requirements support activity and cost estimate.• Schedule estimates may be based upon historical data, parametric analysis, and subject matter experts.• Document the basis of the estimate: “Documenting the basis for the duration facilitates communication of expectations between activity owners and senior decision makers, and aids in the estimating of future analogous activity durations.” – GAO 21
Basis of Estimate – Schedule Lessons Learned (3)• A good BOE should not only document the basis for the activity duration, but also: Mandatory logic Constrained dates Leads and Lags Calendars Resources• Scheduler documents BOE rationale in MS Project text field.• Durations of other missions can be used as part of the BOE. 22
Estimating - Live Demo Note – these people are not actors; they are professionals! 23
Schedule Analysis - Lessons Learned• Schedules may pass the health checks but still have flaws; focusing solely on schedule mechanics will not fix a poor quality schedule.• Schedules that fail the health checks should be corrected prior to performing integrated cost and schedule analysis (Note: If the true critical path cannot be identified in the schedule before the range estimate or JCL activity, then the resulting critical path will not be credible).• Analysis schedules should be developed to a level that will incorporate any identified risks that the project is carrying.• Schedule trends for milestones, deliverables, and interfaces need to be analyzed and monitored. 24
Reserve Analysis – Lessons Learned• Different NASA centers and organizations apply and manage reserve differently.• It is important to understand the basis for the reserves and how the P/p is managing reserve.• Margin or reserve tasks should also be clearly identified and the method used to estimate margin should be explained.• References: Schedule Management Handbook JPL Rules GSFC Gold Rules 25
Range Estimate / Joint Confidence Level (JCL) Lessons Learned• Importance of analysis schedule reflecting true critical path, logical linkages and enough detail to map risks and cost elements.• Importance of working with cost, schedule and risk analysts.• Tools should not affect the process.• Understanding and communicating results. 26
Adequacy/Availability ofResources other than Budget 27
Resources• Analysis includes an assessment of workforce planning, skill mix, retention, obsolescence, and ramping issues.• Assessment of availability of facilities, availability, costs and supporting infrastructure.• Analysis of any mission partner’s potential impact to cost and schedule. 28
Resources - Lessons Learned• A ramp-up or ramp-down of workforce needs to be consistent with the work flow (i.e. the project schedule/project plan).• Projects often rely on the same facilities for testing. Schedule slips in one project can cause delays for other projects. In many cases, problems may be avoided if work-arounds are identified early. 29
Adequacy of Risk ManagementApproach and Risk Identification and Mitigation per NPR 8000.4 30
Risk Management Approach• Assessment of BOEs for any cost or schedule mitigation plans for the identified risks.• SRB supported assessment of the risks which may include new risks that are identified by the SRB with associated cost and schedule impacts.• SRB supported assessment of the risks which may included adjustments to assumptions made by the P/p with associated cost and scheduled impacts (TRL and heritage).• Assessment of the reserve strategy and the current state of the reserves in relation to where P/p are in life cycle.• Assessment of whether risk has been accounted for in P/p cost, schedule or JCL process. 31
Risk Management Approach – Lessons Learned• The Risk Management Plan should accurately reflect the process that is in place, including: How the P/p/subsystem levels coordinate and interact. Definitions of the 5x5 scoring criteria. Processes for identifying, tracking, mitigating, controlling, and documenting risks.• Risks need to be mapped to the appropriate schedule task(s). 32
Uncertainty• Planned schedule activity durations are estimations; completion dates appear as definite dates with associated definitive costs. Schedule finish dates and costs are not facts, but projections with associated uncertainty. Generic risks captured in the project’s risk list, such as “Instrument Schedule,” cannot typically be mapped to specific work activities in the schedule; these types of risks are actually “uncertainty” and should be captured as such when performing a range estimate or JCL. 33
Uncertainty Lessons Learned• Uncertainty values are applied to WBS levels within a P/p• In many cases, project trends (performance over time) can be a good indication of the uncertainty associated with a particular element in a project, especially if discrete risks cannot be identified. Schedule - Uncertainty Distributions• Understanding the P/p’s Risk Level Low 10% Mode 50% High 90% 0.984 1.000 1.070 uncertainty gives the SRB Low Low+ 0.954 1.000 1.240 members a basis for Medium 0.924 1.000 1.390 assessing uncertainty Medium+ 0.893 1.000 1.550 durations. High 0.863 1.000 1.710 High+ 0.832 1.000 2.020 Very High 0.802 1.000 2.350 Very High+ 0.771 1.000 3.370 Extra High 0.741 1.000 4.384 34
Risk• Known risks, identified by Program, Project or SRB are normally called “discrete risks” with corresponding risk mitigation activities. Mitigation may require funds and schedule and the SRB needs to understand the impact on schedule/cost reserves. Discrete risks should mapped to the schedule activities they affect and be assigned associated probability and impact values for cost and schedule. 35
Risk - Lessons Learned• Projects should be able to quantitatively evaluate their risks, by identifying the following (reference SRB risk chart): Probability of Risk Occurrence Best Case Schedule (Work Days) Most Likely Schedule (Work Days) Worst Case Schedule (Work Days) Best Case Cost ($) Most Likely Cost ($) Worst Case Schedule ($)• *Note: A basis of estimate should be provided with each of the above quantities. 36
SRB Risk Worksheet Likelihood (% chance of occurring) Consequence (Duration Beyond Already Scheduled [days])Risk TypeIdentifiedby Project ml min max Risk Associated Specific Activities (Cost, Rank Risk ID Title Statement L % likelihood rationale C min rationale (most ml rationale max rationale (optimisitic) (pessimistic) Owner WBS # ImpactedSchedule, likely) orTechnical) 1 2 3 4 5 6 7 8 9 10 11 12 13Risk TypeIdentified by SRB ml SRB- min max Risk Associated Specific Activities (Cost, Title Statement L % likelihood rationale C min rationale (most ml rationale max rationale Risk ID (optimisitic) (pessimistic) Owner WBS # ImpactedSchedule, likely) orTechnical) SRB-1 SRB-2 SRB-3 SRB-4 SRB-5 SRB-6 SRB-7 SRB-8 Understanding the project’s risk assessment gives the SRB members a basis for their assessment. 37
Parting Words• It takes one woman nine months to have a baby. It • The sooner you begin coding the later you finish. cannot be done in one month by impregnating nine • Anything that can be changed will be changed until women (although it is more fun trying). * Spanish there is no time left to change anything. Proverb • Change is inevitable – except from vending• The same work under the same conditions will be machines. Robert C. Gallagher estimated differently by ten different estimators or by • The person who says it will take the longest and cost one estimator at ten different times. the most is the only one with a clue how to do the• Any project can be estimated accurately (once its job. completed). • If youre 6 months late on a milestone due next week but nevertheless really believe you can make it,• A change freeze is like the abominable snowman: it youre a project manager. is a myth and would anyway melt when heat is • If you don’t know where you are going any road will applied. take you there. Lewis Carroll (Alice in Wonderland)• Right answers to wrong questions are just as wrong • If you fail to plan you are planning to fail. as wrong answers to right questions. • If you dont attack the risks, the risks will attack you.• What you dont know hurts you. • A little risk management saves a lot of fan cleaning.• Theres never enough time to do it right first time but • The sooner you get behind schedule, the more time theres always enough time to go back and do it you have to make it up. again. • A badly planned project will take three times longer• Estimators do it in groups - bottom up and top down. than expected - a well planned project only twice as long as expected.• Good estimators arent modest: if its huge they say so. MHR• If project content is allowed to change freely the rate of change will exceed the rate of progress.Reference: http://www.project-training-k.freeserve.co.uk/ 38