You Can’t Grade Your Own Homework Dr. James Ortiz, Deputy DirectorIndependent Program Assessment Office Sixth Annual NASA Program Management Challenge 2009 February 25, 2009 Page 1
GoalThe NASA independent review process helps ensure thehighest probability of mission success to Programs andprojects. Page 2
Topics addressed• What is an independent life-cycle review ?• Why do we do independent reviews?• Who is involved?• What is the Standing Review Board (SRB) ?• How are independent reviews done?• Why can’t you grade your own homework?• Summary Page 3
What is an Independent Life - Cycle Review ?• A review of Programs and projects at each life-cycle milestone performed by competent individuals who are not dependent or affiliated with the Program or project and who do not have an organizational or personal interest or stake in the results of the review• The review objectively assesses: – Adequacy and credibility of the technical approach (requirements, architecture, design) – Schedule – Resources – Cost – Risk – Management approach – Compliance with Agency policy (NPR 7120.5, NPR 7123.1) – Readiness to proceed to the next phase• The results of the independent review are advisory to the Program/project and to the decision authority Page 4
Why Do We Do Independent Reviews?1. Agency wants Program/projects to receive independent assurance that they are on-track2. NASA senior management wants: – Independent validation at key decision points of the Program/project’s readiness to proceed into the next phase of its life-cycle – Externally-imposed impediments to Program/project success are being removed3. Agency needs to provide external stakeholders assurance we can deliver to our commitments4. Significant additional benefit is that preparation for the review milestone allows for a holistic examination by the Program/project and the review team Page 5
Who Is Involved?• Per the Agency governance structure, the independent life- cycle review process is implemented as a collaborative effort between: – Agency Senior Management – Center Management – Technical Authorities (TA) – Program/Project Management Decision Authority Technical Authority Associate Administrator Center NASA AA MDAA NASA CE PA&E Director Programs Approve Approve Approve Approve Establish Category 1 SRB, Approve Approve Concur Approve Approve Projects Develop ToR. Approve Category 2 Approve Approve Approve* Chairperso Projects n, RM, and Other Board Members Category 3 Approve Approve Projects Page 6
What is the SRB? (NPR 7120.5 & SRB Handbook)• Independent life-cycle reviews are conducted by a Standing Review Board (SRB) – The SRB has a single chairperson and a NASA review manager – The SRB remains intact, with the goal of having the same core membership, for the duration of the Program or project – Board members must be independent of the programmatic and technical authority chain of command for the Program/project – Board members are free from organizational and personal conflict of interests – The main attributes of SRB members are currency, competence and independence• SRBs provide the Agency a non-advocate, objective, and competent assessment of the Program/project as it advances through its key decision points (KDP) Page 7
How Is It Done? (IPAO process) I N P U T S P/p Documentation Agency Review Agency/Directorate/Centers Cost & Schedule SRB Briefings to Minutes/Decisions/ActionsSchedule Documentation & Documentation Program/Project Review Chair Requirements Program/ProjectNomination Briefings PLANNING PREPARING REVIEWING REPORTING CLOSING-OUT Verbal Pre-Briefings to P/p Review Closeout Monitor Agency Initiate Contact and Determine Receive CADRe Inputs Write one-page summary CustomerBaseline Program/ Review Budget SRB Kick-Off Meetings Write the Report FeedbackProject Review Identify and obtain approval of Attend Reviews Prepare the Summary Develop Lessons Determine IPAO SRB Chair/RM/SRB members Attend Site Visit Briefings LearnedReview Budget Determine funding mechanism Develop ICE/ICA/ISA Present the Briefings: Process Review Review Assignments Develop the TOR - Program/ project and Improvement Develop Schedule/Logistics - CMC Administrative of the Review - MD PMC Close-out Develop the Cost Plan - APMC (if required) Team Recognition Develop the Schedule Plan P R O D U C T S Individual Review Budget ICE/ICA/ISA One Page Summary Review Summary RM/CA/SA Assignments ToR, Cost Plan, Schedule Plan SRB Findings SRB Report Contracts Close-out by Chair and Team Nomination RFA RRD Report COTR and Guidance Letters Recommendations SRB Briefings Review Records Review Schedule RRD Report Contract Task Statement Process improvements R&R for SRB Page 9
Recent improvements• The following diagram of the SRB “engagement” timeline approved by the Agency Program Management Council (APMC) resulted from process improvement activities jointly performed between IPAO, Mission Directorates, Programs, and the Office of the Chief Engineer (OCE). Path to Orion PDR DAC SRB observer role 11/08 5/09 “one pager” by Project, includes Plan to KDP S/S Reviews Internal 6/09 8/09 Project PDR 8/09 RFAs Findings SRB assessment of entry criteria to SRB Site “One pager” + 1 day (joint activity) Project/Program Review ‐ Quick status PMR Rev 1 ‐ Only big issues, not concerns/observations 11/08 3/09 4/09 8/09 ‐ Initial ICE SRB l‐‐‐‐l PDR/NAR KDP 1 – 6 l ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐ ‐l mos 5‐10 days 30 days Integrated ‐ Tech ‐ Cost “Pre‐work” ‐ Sched Briefings: P/p, CMC, DPMC, APMC ‐ Risk Model Development Joint with IPAO/Project Independent Life Cycle Review (ILCR) ICA/ICE/ISA reconciliations Page 10
Life-Cycle Review Success Criteria1. Alignment with and contributing to Agency needs, goals, and objectives, and the adequacy of requirements flow-down from those2. Adequacy of technical approach as defined by NPR 7123.1 entrance and success criteria3. Adequacy of schedule4. Adequacy of estimated costs (total and by fiscal year), including Independent Cost Analyses (ICAs) and Independent Cost Estimates (ICEs), against approved budget resources5. Adequacy/availability of resources other than budget6. Adequacy of risk management approach and risk identification/mitigation7. Adequacy of management approach Page 11
Summary: Why You Can’t Grade Your Own Homework?• The independent life-cycle review supports the NASA Administrator’s Charge: “You can not grade your own homework regardless of the position you hold as part of the Program or project”• The independent life-cycle review process is an integral part of the Agency’s check and balances built into the NASA governance structure and complements the programmatic and technical lines of command and authority• It is implemented by Standing Review Boards (SRBs) staffed with members that are current, competent and independent• Its processes are continuously assessed for improvement by IPAO and its stakeholders• The independent life-cycle review process helps ensure the highest probability of success of the Agency’s Program and projects Page 12
Terminology• Program – A strategic investment by a Mission Directorate or Mission Support Office that has a defined architecture and/or technical approach, requirements, funding level, and a management structure that initiates and directs one or more projects. A Program defines a strategic direction that the Agency has identified as critical.• Project – A specific investment identified in a Program Plan having defined requirements, a life- cycle cost, a beginning, and an end. A project yields new or revised products that directly address NASA’s strategic needs.• Standing Review Board (SRB) – The entity responsible for conducting independent reviews of the Program/project per the life-cycle requirements. The SRB is advisory and is chartered to objectively assess the material presented by the Program/project at a specific review.• Decision Authority (DA) – The Agency’s responsible individual who authorizes the transition of a program/project to the next life-cycle phase.• Key Decision Point (KDP) – The event at which the Decision Authority determines the readiness of a Program/project to progress to the next phase of the life cycle (or to the next KDP). Page 14
Governing Documentation NPD 1000.0 NPD 1000.3 NPD 1000.5 Engineering NPD Program Mgt. OSMA NPD Health and Mission Support Office NPDs NPD Medical NPDNPR 7123.1 and Other NPR 7120.5 and OSMA NPRs NID 1240-41 and Support Org Engineering NPRs other PM NPRs OCHMO NPRs NPRs Health & Engineering Program/Project SMA MSO Functional Medical Requirements Mgmt Requirements Requirements Requirements Requirements Mission Directorate Center Engineering & Programmatic Management Requirements Policies and Practices Program Plans Project Plans Page 17
A particular slide catching your eye?
Clipping is a handy way to collect important slides you want to go back to later.