The document provides an overview of NASA's policy for independent program reviews: 1. NASA policy mandates independent reviews at key decision points to validate programs' readiness and identify risks. This includes NPD 1000.0 requiring checks and balances like independent reviews. 2. The Standing Review Board process in NPR 7120.5 standardizes independent reviews across NASA. Reviews assess technical and programmatic status at life cycle milestones. 3. The SRB Handbook provides guidance for Standing Review Boards to apply review criteria consistently across programs in accordance with NASA's technical and program management requirements. It outlines the roles, processes, and expected work products for the reviews.

1. Connect the Dots
Dr. James Ortiz, Director
Independent Program Assessment Office
Seventh Annual NASA
Program Management Challenge 2010
February 10, 2010
Used with Permission
Page 1

2. Introduction
• The Independent Program Assessment Office (IPAO) manages
the independent review of the Agency's Programs and projects
at life-cycle milestones to ensure the highest probability of
mission success.
•This presentation provides an overview of the Agency’s policy
context for the Agency Independent Life Cycle Review process
(The “Why”, the “What”, and the “How”).
Page 2

3. Outline
• Connecting the dots on the Agency policy context
– The “Why”
» NPD 1000.0; NPD 1000.5
– The “What”
» NPR 7120.5D NID; NPR 7123.1
– The “How”
» SRB Handbook
» IPAO System Operating Procedures
• Summary
Page 3

4. Agency Policy Context for
Independent Review
Page 4

5. Policy Flow-down
NPD 1000.0
NPD 1000.3
NPD 1000.5
NPD 7120.4 NPD 8700.1 NPD 8900.5A Mission Support Office
NPR 7120.5D Support Organization
NPR 7123.1A OSMA Directives OCHMO Directives
and NID Directives
1 2 3 4 5
NASA -SP 2009 -10 -015 - HQ Center Engineering Mission Directorate
NASA -SP -6105
SRB HANDBOOK Requirements Requirements
SYSTEMS ENGINEERING
HANDBOOK
7 6
IPAO 1000
IPAO MANAGEMENT
PLAN LEGEND:
DIRECT
1: ENGINEERING REQUIREMENTS
2: PROGRAM/PROJECT
IPAO 2000 MANAGEMENT REQUIREMENTS
STANDARD OPERATING INDIRECT
3. SAFETY AND MISSION
PROCEDURE ASSURANCE REQUIREMENTS
4. HEALTH AND MEDICAL REQTS
5. MSO FUNCTIONAL REQTS
6. MD REQUIREMENTS
7. CENTER REQUIREMENTS
Page 5

6. The “Why”
Page 6

7. The Governance and Strategic
Management Handbook (NPD 1000.0A)
“NASA governance and strategic management provide the discipline and rigor
to enable success of NASA’s Mission- ….
Mission success delivers on our commitment to be good stewards of the
resources entrusted to us by the taxpayer. To enable Agency-wide
accomplishments, NASA’s governance framework is founded on the following
tenets:
• …..
NASA’s governance principles that provide this framework are:
• Lean governance.
• Clear roles, responsibility and decision making.
• Strategic acquisition.
• Checks and balances. ……
There are many process-related checks and balances built into NASA’s way of
doing business. They range from peer reviews conducted at the lowest level to
oversight reviews conducted by the Agency’s Program Management Council.
Three checks and balances of particular importance at the program or project
level are: the independent life-cycle review process, the process for tailoring
a specific prescribed requirement, and the Dissenting Opinion process.”
Page 7

8. Why Independent Review
The Governance and Strategic Management
Handbook (NPD 1000.0A)
The independent life-cycle review process provides a comprehensive
review of programs and projects at each life-cycle milestone by competent,
independent individuals. The purpose of these reviews is to provide:
1. Agency wants P/p to receive independent assurance that they are on-
track
2. NASA senior management wants:
– Independent validation at key decision points of the
Program/project’s readiness to proceed into the next phase of its
life-cycle
– Externally-imposed impediments to Program/project success to
be removed
3. Agency needs to give external stakeholders assurance we can deliver
to our commitments
4. Significant additional benefit is that preparation for the review
milestone allows for a holistic examination by the Program/project and
the review team
Page 8

9. Policy for NASA Acquisition
NPD 1000.5
• NPD 1000.5A calling for the Joint Confidence Level (JCL)
analysis became effective in January 2009
• Policy is directed to ensure appropriate level of funding is
provided and maintained to increase probability of success of
the Agency’s Programs and Projects
• Requirements
– (a) Programs are to be baselined or rebaselined and budgeted at a confidence level
of 70% or the level approved by the decision authority of the responsible Agency-
level management council. For a 70% confidence level, this is the point on the joint
cost and schedule probability distribution where there is a 70% probability that the
project will be completed at or lower than the estimated amount and at or before the
projected schedule. The basis for a confidence level less than 70% is to be formally
documented.
– (b) Projects are to be baselined or rebaselined and budgeted at a
confidence level consistent with the program’s confidence level.
– The program's or project's proposed cost and schedule baseline are to be assessed
by an independent review team. The program or project is to present and justify its
resulting cost and schedule to the decision authority of the responsible Agency-level
management council. The independent review team is to discuss with the decision
authority its key concerns with the plans and baselines proposed by the program or
9
project.
Page 9

10. The “What”
Page 10

11. NPR 7120.5D NID
Independent Review Requirements
• Provides the basis for how NASA execute its space
flight programs and projects.
• Provided a standardized life cycle review process that
is built around Key Decision Points (KDP).
• Formalized the Technical Authority, Dissenting
Opinion Resolution, and Waiver processes.
• Defines the Independent review policy requirements
 Standardizes the Standing Review Board concept across NASA
 Combines technical (7123) and programmatic reviews into one Board
the Standing Review Board (SRB)
 One independent review board for each program or project
 Conducts all 7120.5-required independent reviews for that
program or project throughout its life cycle
 Implemented by the IPAO for all Programs, Category 1 Projects
and Category 2 projects with a life-cycle cost > 11
$250M Page 11

12. Project Life Cycle (NPR 7120.5)
Page 12

13. Project Life Cycle (NPR 7120.5)
Page 13

14. NPR 7123.1 Systems Engineering
Technical Review Requirements
• NPR 7123.1A – Systems Engineering Processes and Requirements
 Articulates and establishes the requirements on the implementing
organization for performing, supporting, and evaluating systems
engineering
 This systems approach is applied to all elements of a system and all
hierarchical levels of a system over the complete project life cycle
• 5.2.1.3 A technical review is an evaluation of the project, or
element thereof, by a knowledgeable group for the purposes of:
 Assessing status of and progress toward accomplishing planned activities.
 Validating technical tradeoffs explored and design solutions proposed.
 Identifying technical weaknesses or marginal design and potential
problems (risks) and recommending improvements and corrective actions.
 Making judgments on activities’ readiness for follow-on events, incl
additional future evaluation milestones to improve likelihood of successful
outcome.
 Making assessments and recommendations to project team, Center, and
Agency management.
 Providing historical record that can be referenced of decisions that were
made during these formal reviews.
• Entrance and Success criteria for each life-cycle milestone are
contained in Appendix G.
Page 14

15. The “How”
Page 15

16. The SRB Handbook
The How
• The SRB is written to provide guidance to the NASA Program and
project (P/p) community and the Standing Review Boards (SRBs)
regarding expectations, processes, products, timelines, and working
interfaces with Mission Directorates, P/p, Centers, other review
organizations, and the Management Councils.
• The SRB handbook was published by PA&E/IPAO with concurrence
from OCE
IR-16
Page 16

17. SRB Handbook Contents
• The SRB Handbook consist of six sections
– Section 1: Context for Independent Lifecycle Review (ILCR) Process. Objectives,
intent, and governance of the Standing Review Boards (SRB).
– Section 2: High level principles that govern the SRBs. Independence and issue
resolution. Points to Annex with Agency policy for “Review Board Composition,
Balance and Conflict of Interest”.
– Section 3: Scope and expectations of ILCR. Provides guidelines (roadmaps) for
application of review criteria in NPR 7120.5 as a function of the lifecycle stage of P/p,
the types of mission (Robotics versus Human Spaceflight), and type of Program (
tightly coupled or loosely coupled).
– Section 4: SRB Initiation process. Roles and responsibilities for Chair/RM nomination,
team composition and staffing, and Terms of Reference (TOR).
– Section 5: SRB Products. Technical and programmatic assessments, findings and
recommendation, briefings and reports.
– Section 6: Notional review timeline. Describes the review phases for a typical review.
– Appendices: Agency Independence Policy, PIR guidance
IR-17
Page 17

18. Example of Section 3
(SRB engagement roadmap)
NASA Life Cycle Formulation Implementation
Phases Pre-Systems Acquisition Systems Acquisition Operations
Pre-Phase A Phase A Phase B Phase C Phase D Phase E
Reviews MCR
1
SRR SDR PDR CDR PRR
1
SIR SAR ORR FRR
2
LRR
2
PFAR
2
(PNAR) (NAR)
Key Decision Points KDP A KDP B KDP C KDP D KDP E
SRB Participation Case-by- Full Board Full Board Full Board Full Board Chair and Full Board Chair and Chair and Chair and Chair and Chair and
case Member Member Member Member Member Member
3 3 3 3 3 3
Subset Subset Subset Subset Subset Subset
Sample Assessments
Preliminary Functional and Flowdown to Flowdown to Any Changes, Production Integration System Phase E, Flight Launch Flight
Performance Functional Functional Flowdown to processes , Plan and meets Descope operations system and performance
Baseline ; Elements; Elements, Functional certified Procedures acceptance plans are certified spacecraft/ reporting
Requirements Requirements System Descope Elements, design criteria and to proceed payloads
Traceability Requirements Plans Descope has been readiness
Report; SEMP Document Plans verified and for launch
validated
Key Preliminary Mission Baseline: Design meets Design Previous Technical All waivers Hardware Launch Performance
Technolgies, Approach, TRL, System Systems performance, documentati component, data and and system and report and
TRL, MOEs, Technology Architecture, Engineeering TRL on, subsystem package, anomalies software spacecraft/ Anomaly
MOPs Devlpt. TRL, TDMAP, Mgt Plan, Production and system Certification have been systems are payloads resolution
5 Maturity SEMP, SDP Design meets Plans tests have package closed; configured readiness
Technical
Assessment performance been verified Operational for flight for launch
Plan (TDMAP), to support procedures
Software integration and
Devlpt. Plan contingency
(SDP) planning
ROM Cost ROM Preliminary: Baseline: Performance Production Performance Remaining Performance Performance
Schedule Integrated BOEs IMS BOEs IMS against plans, Plans against liens or open against against
Estimates Cost Schedule Cost/Budget Cost/Budget EVM, plans, items and plans, plans,
Integrated cost and
6
Estimates UFE, CADRe UFE, CADRe UFE , LLIL EVM, plans for EVM, EVM,
schedule WBS JCL , Detail UFE, closure UFE UFE
ICE schedules, JCL
ICE
Preliminary: Preliminary: Preliminary: Baseline: Facilities Facilities Facilities Facilities Facilities Facilities
Resources Facilities Facilities Facilities Facilities Workforce Workforce Workforce Workforce Workforce Workforce
Workforce Workforce Workforce Workforce
Preliminary Preliminary Preliminary: Baseline: Risk List Risk List Risk List Risk List Risk List Risk List
Risks Risks Risk Mgt Plan Risk Mgt Plan Mitigation Mitigation Mitigation Mitigation Mitigation Mitigation
Risk
Risk list Risk list
Mitigation Mitigation
Preliminary: Preliminary: Preliminary: Baseline: Performance Production Performance Performance Phase E Performance
Management Approach Plans Project Plans Project Plans against plans Engineering against plans against Mgt Plan against
and Planning Plans plans
Reporting Venues
Project x x x x x x x x
4
Program x x x x x x x x
4
CMC x x x x x x x x
DPMC x x x
4
x
4
x x x
4
x
APMC Cat 1 Cat 1 Cat 1 Cat 1
Notes:
1. PRRs are only needed when multiple flight system copies are being developed; timing is discretionary.
2. LRRs, FRRs, PFARs are performed by the Mission Management Team (MMT); the SRB Chair (and/or Designee) attend as non-voting observers.
3. Chair determines which members should attend, including themselves.
4. The SRB chair and RM will make themselves available to support these interim briefings.
5. Compliance with NPR 7123.1 review entrance and success criteria will be assessed.
6. Compliance with NPR 1000.5. IR-18
Page 18

19. Example from Section 4
(SRB Formulation )
SRB Initiation Schedule prepared monthly for the APMC
BPR
Cat 1 & Cat 3 &
Programs Cat 2 Projects Cat 2 Projects
>$250M <$250M
MDAA CD CD
Nominates Chair Nominates Chair Nominates Chair
Reiterate
selection steps
with new Chair
IPAO RM IPAO RM Center RM
nomination
Assignment Assignment assignment
No
MDAA/DA/TA/PA&E*
Nominations vettedvetted all Approval Authorities (MDAA, DA, TA, PA&E) and cleared for OCI/PCI.
Nominations with with all Approval Authorities (MDAA, DA, TA, PA&E)
Approval/Concurrence ?
Selection of the SRB Chair and RM
Yes
Reiterate as
Chair/RM develop Chair/RM develop ToR necessary to
SRB Membership List and review with DA and TA obtain Approvals
No
Nominations vetted with all Approval Authorities MDAA/DA/TA/PA&E*
Approval/Concurrence ?
and cleared for OCI/PCI.
SRB Staffing and ToR Preparation
Participating Organizations Yes
Mission Directorate Host Center Proceed with SRB Reviews
IPAO SRB
* For Programs, Cat.1 and Cat 2 (LCC>$250M) Projects only
Joint Effort
IR-19
Page 19

20. Handbook Status
• Initial version of Standing Review Board (SRB) Handbook (HB) developed
October 2007 by Agency team (IPAO lead and representative from each
Center).
• SRB HB on hold pending review of Federal Advisory Committee Act (FACA)
regulations and independence policy requirements.
• SRB HB on hold for completion of NPR 7120.5D NID (~February 2009).
• NID-aligned version of the SRB Handbook released for PPMB review on Sept
25, 2009 ( 2 week review).
• Baseline version of SRB HB published ( December 2009)
• Next revision to be published with NPR 7120.5E
IR-20
Page 20

21. IPAO
The NASA organization responsible for the agency
independent life cycle review process
• Enables independent review of programs and projects to
ensure highest probability of mission success.
• Ensures objectivity, quality, integrity and consistency of
independent review process per NPRs 7120.5D and 7123.1A.
 Works with SRB Chair and Convening Authorities
 Reviews SRB products to ensure they meet Agency
expectations
• Provides Review Manager, cost, schedule and other
programmatic analysis SRB members
• Provides advice and recommendations to Agency
on program and project policies
 Book Manager for Agency’s SRB Handbook
Page 21

22. IPAO Standard Operating Procedure
Detail processes
• The Standard Operating Procedure (SOP) documents the Independent
Program Assessment Office (IPAO) process for conducting
Independent Life-Cycle Reviews (ILCRs).
• The SOP is a guide for the IPAO personnel when conducting ILCRs of
the Agency’s programs and projects.
• The purpose of this document is to ensure ILCR quality and efficiency
through the development of common definitions and processes.
Page 22

23. How is it done?
(IPAO Process)
I N P U T S
 P/p Documentation
 Agency Review  Agency/Directorate/Centers  Cost & Schedule  SRB Briefings to  Minutes/Decisions/Actions
Schedule Documentation & Documentation Program/Project
 Review Chair Requirements  Program/Project
Nomination Briefings
PLANNING PREPARING REVIEWING REPORTING CLOSING-OUT
Continuous 2- Months 1-3 Months 2-3 Months 1- Month
 Monitor Agency  Initiate Contact and Determine  Receive CADRe Inputs  Verbal Pre-Briefings to P/p  Review Closeout
Baseline Program/ Review Budget  SRB Kick-Off Meetings  Write one-page summary  Customer
Project Review  Identify and obtain approval of  Attend Reviews  Write the Report Feedback
 Determine IPAO SRB Chair/RM/SRB members  Attend Site Visit  Prepare the Summary  Develop Lessons
Review Budget  Determine funding mechanism  Develop ICE/ICA/ISA Briefings Learned
 Review Assignments  Develop the TOR  Present the Briefings:  Process Review
 Develop Schedule/Logistics - Program/ project and Improvement
of the Review - CMC  Administrative
 Develop the Cost Plan - MD PMC Close-out
 Develop the Schedule Plan - APMC (if required)  Team Recognition
P R O D U C T S
 Individual Review Budget  ICE/ICA/ISA  One Page Summary  Review Summary
 RM/CA/SA Assignments ToR, Cost Plan, Schedule Plan  SRB Findings  SRB Report  Contracts Close-out by
Chair and Team Nomination  RFA  RRD Report COTR
and Guidance Letters  Recommendations SRB Briefings  Review Records
 Review Schedule
 RRD Report
 Contract Task Statement
 Process improvements
 R&R for SRB
Page 23

24. SOP:Sample SRB Engagement
Timeline
SRB KO: PDR: 2/11/10 Site Visit: 5/11/10 - 5/21/10 KDP: 6/11/10 - 6/21/10
10/14/09
PDR KO:12/1/09
24 Page 24

25. Summary
• The independent life cycle review process is an integral part of the
Agency’s check and balances built into the NASA governance
structure and complements the programmatic and technical lines of
command and authority.
• The independent lifecycle review process is encoded as part of NASA
policy direction; its requirements are stipulated in policy requirements;
and guidance to reviews team and implementing personnel is provided
in handbooks and operating procedures.
• Its processes are continuously assessed for improvement by IPAO and
its stakeholders.
• The independent lifecycle review process helps ensure the highest
probability of success of the Agency’s program and projects.
Page 25