This document outlines improvements made to NASA's independent program review process in fiscal year 2010. It discusses the reviews completed, including 8 program and 20 project reviews. Process improvements included quick look reports, increased coordination, readiness assessments, and streamlining of review documentation. The roles and responsibilities of review board members are covered, including ensuring member competency, currency, and independence. Coordination between review boards and NASA mission directorates and centers is also summarized.

1. Headquarters
Independent Program & Cost Evaluation
State-of-the-Art
in
Independent Review Execution
Tahani Amer and Kaiser Adeni
Review Managers
Independent Program Assessment Office
8th Annual 2011 NASA Program Management Challenge
February 10, 2011

2. Outline
Headquarters
Independent Program and Cost Evaluation
• Look Back at FY10 Reviews
• Review Process Improvements
• SRB Balance & Structure
• SRB Members Roles & Responsibilities
• SRB Coordination (between Mission Directorates,
Centers, and Programs/projects (P/p))
• SRB Products
• SRB – HB updates
• IPAO State-of-the-Art Activities
• Look Forward
2

3. Look Back at FY10 Reviews
Headquarters
Independent Program and Cost Evaluation
• 8 Programs and 20 project review activities executed in FY10:
 Preliminary Design Review: MAVEN
 Critical Design Reviews: GRAIL, RBSP, GPM, TDRS, LDCM, MMS,
OCO-II
 System Integration Reviews: MSL, GRAIL and Juno
 ORR/FRR/PLAR: WISE
 Program Approval Review: RPS and LQP
 Program Implementation Review: SCaN, ESMP, D/NF
 Special Reviews: MSL, CxP, Aquarius, SOFIA
 8 Non-IPAO reviews: 2- GOES-R, ASP, AvSP, ATP, FAP, ENAS, CAS
3

4. FY10 Completed Reviews
Headquarters
Independent Program and Cost Evaluation
4

5. FY10 Completed Reviews
Headquarters
Independent Program and Cost Evaluation
5

6. FY10 IPAO Completed Reviews
IPAO supported 8 non-IPAO review activities in FY10
Headquarters
Independent Program and Cost Evaluation
6

7. Review Process Improvements
Headquarters
Independent Program and Cost Evaluation
• SRB process improvements:
• Quick Look Reports (one-pagers);
• 30-day reporting requirement;
• Increased coordination for programmatic assessments;
• Readiness-to-proceed assessments;
• Alternate opinions for non-consensus boards;
• Key Decision Point (KDP) Decision Memo improvements;
• Deferral of program reviews approved at APMC;
• Institutionalized electronic signature of SRB approval letters;
• Streamlining of ToR content as defined in draft NPR
7120.5E;
7

8. Review Process Improvements
Headquarters
Independent Program and Cost Evaluation
• SRB process improvements (continued):
• Better coordination of SRB team nomination process; early
stakeholders involvement in the process to ensure proper
team balance of competency/currency/independence;
• Final reports are posted on APMC electronic repository
(https://nx.arc.nasa.gov/nx/dsweb/View/Collection-93608); notification at
APMC when reports are posted;
• Strengthening and improving the rigor and integration of the
technical and programmatic assessments;
• Working with SMD and the Centers to formulate principles on
conducting joint Program/project reviews when there is a
significant external partner involved. Efforts are evolving
w/ExoMars and JPSS.
8

9. SRB Balance & Structure
Headquarters
Independent Program and Cost Evaluation
• SRB members should be competent, current, and
independent from the management or advocacy chain of
the P/p, with membership balanced between the host
Center and other organizations to ensure the needs of
the convening authorities are met.
• Although balance of each SRB member is important,
ultimately the goal is to have the SRB, as a whole,
balanced.
• More inclusive set of discussion with Center and TA
• It is not a numerical formula, but it goals to meet
Agency’s goals
9

10. SRB Balance & Structure
Headquarters
Independent Program and Cost Evaluation
 Competency
 Relevant experience and expertise with the technical, specific
technologies and programmatic domain of the P/p under review.
 Currency
 Addresses recent/current experience and expertise in
programmatics or the technical domain(s) of the project or program
under review.
 Understanding of current Agency governance, project management
and systems engineering policies, procedures and methods,
specifically NPR 7120.5, NPR 7123.1, NPD 1000.5, specific Center
practices and procedures and the SRB Handbook.
 Independence
 Not in the programmatic chain of command of the program or
project and have no conflicts of interest either personally,
institutionally or organizationally.
10

11. SRB Members Roles & Responsibilities
Headquarters
Independent Program and Cost Evaluation
• A member of an SRB is an “agent” of the convening authorities:
 More emphasis on programmatic risk assessment and analysis;
 More emphasis on the individual member independent report (IMIR);
 Advise the SRB Chair and the Review Manager (RM) on areas that
require particular attention by the SRB per their area of expertise;
 Support the SRB “Quick-Look Report” summaries and briefings to the
Convening Authorities (CA), and provide expert opinion to SRB
preliminary and final reports.
11

12. SRB Coordination
Headquarters
Independent Program and Cost Evaluation
• Coordination with the Mission Directorates:
 Early and frequent coordination is conducted with the Mission Directorate Program
Executive (PE) to:
 Establish SRB membership
 Chair with support from RM establishes the discipline areas to be covered
 PE/MDs provide a list of potential candidates. CAs approve or provide alternate
nominations
 RM leads the vetting effort for Organizational and Personal Conflict of Interest
(OCI/PCI) in coordination with the LaRC Legal Office and Contracting Officer(s)
 Finalize Terms of Reference (ToR) content
 Content is reviewed and agreed-to prior to final approval by the CA
 Identify any additional Mission Directorate review criteria
 Establish timing of the Site Visit
 Establish post-review out brief schedule
 CMC/DPMC briefing
 APMC briefing (Cat 1 and some Cat 2 Programs/projects)
 APMC special topics for Cat 2 project life cycle reviews
 Pause and Learn (PAL)
12

13. SRB Coordination
Headquarters
Independent Program and Cost Evaluation
• Coordination with the Center
 Coordination with the Center is conducted to:
 Coordinate and develop a list of SRB candidates from the Center based on the
discipline areas established by the Chair
 Facilitate the review of the ToR content and ensure that the Center specific
requirements are being met
 Golden Rules
 JPL Rules
 Facilitate Center CA approvals of ToRs, team nominations, etc.
 Establish dates for post-review CMC briefing
 Establish a Community of Practice (CoP) at each Center (in progress)
 Knowledge sharing of best practices and expectations for independent assessment
 Review manifest coordination and approval
 Identify Center personnel to participate in SRBs
 Assist with nomination of high potential candidates for RM assignment with IPAO
 Request potential Center Review Managers as detailees to IPAO
 Develop a CoP for the Agency after completion on the discussion with Centers
13

14. SRB Coordination
Headquarters
Independent Program and Cost Evaluation
• Coordination with Program/project:
 Program/project pre-planning coordination begins ~120 days prior to Site Visit to:
 Establish Site Visit review requirements
 Discipline areas to be covered
 Data drop timeline for cost and schedule products
 Establish agreement on Terms of Reference (ToR) review criteria
 Single ToR for LCR (0.5E)
 Readiness to Proceed Review
 Establish timing of the Site Visit
 Establish post-review out brief schedule
 Quick- Look Report (One Pagers)
 P/p briefing
 CMC briefing
 DPMC briefing
 APMC briefing (Cat 1 and some Cat 2 P/p)
14

15. Headquarters
SRB Products
Independent Program and Cost Evaluation
SRB products produced for each Site Visit:
 Briefings
 One-pager (Quick Look Report)
 P/p Briefing
 CMC Briefing
 DPMC
 APMC Briefing (if a Cat 1 Project)
ToRs, team nomination letters, alternate opinions
Vetting Package
 OCI/PCI mitigation plans/annual vetting
 SRB member resumes/bios
 SRB approval letters
 Reports
Final SRB summary report to include each SRB individual reports
to DA. https://nx.arc.nasa.gov/nx/dsweb/View/Collection-93608
IPAO Review Record
RRD for the Agency
15

16. Headquarters
SRB –HB Update
Independent Program and Cost Evaluation
 Guidance to the P/p and SRB members
 The SRB-HB is posted on the
http://nodis.hq.nasa.gov/policy_letters/NM_7120-81_.pdf
 Updates to comply with NPR 7120.5E
16

17. IPAO State-of-the-Art Activities
Headquarters
Independent Program and Cost Evaluation
• JCL assessments and improved programmatic analysis:
 Using Defense Contract Management Agency (DCMA) and Government
Accounting Office (GAO) criterion as standard evaluation criteria for
schedules
 Implementation of standard analysis timeline allowing for time to work
with the projects to improve cost and schedule concerns
 Analysis processes have been defined and documented in Standard
Operating Procedure Instructions (SOPI)
• Better risk assessments:
 Using state-of-the-art tools for integrated risk analysis
 Establishment of Schedule Analysis Working Group (SAWG) to develop a
CoP for programmatic risk analysis
• Forensics Study:
 Analysis of the SRB findings
 Identify trends and systematic issues
 Provide recommendation to improve PM at NASA
17

18. IPAO State-of-the-Art Activities
Headquarters
Independent Program and Cost Evaluation
• Lessons Learned (LL) after each review
 IPAO members share the LL with RMs and PAG Analysts
 Database of LL
• ILCR - Customer/CA/SRB member surveys
 Developed ILCR surveys & Approved by CA
 Implementing the survey: RBSP
• Training: RMs & Chairs
• Outreach Effort: Articles, PM Track, Visits to Centers, PAL
• Developmental Program: Detail opportunity to support
Agency Project Management
18

19. Look Forward
Headquarters
Independent Program and Cost Evaluation
• Implementation of the NPR 7120.5E
• Update the SRB Hand Book
• Continue coordination with MDs, Centers, and P/p
• Communities of Practice
• SRB Balance
• Independent Reviews involving external partners
• Independent Reviews and Technology P/p
19

20. Headquarters
Independent Program and Cost Evaluation
Back-up Slides
20

21. One Step PDR
Life Cycle Review Overview
Headquarters
Independent Program and Cost Evaluation
KDP-C
KDP-B
(2)
Quick Look Report
PDR Readiness
Assessment (2) PDR-LCR
(30- 90 days) (30 Days)
-Required prior to LCR Technical Baseline with C/S/R
-Report to DA for life cycle reviews and Integrated Assessment of
preceding KDP B&C and during any Technical and Programmatic
major replan or rebaseline (3) Baseline
Not To Scale
Programmatic Data Drops to
SRB (includes JCL Model) P/p Center MD
Deliveries start at 100 days
before site review
Brief Brief Brief
(2)
CheckPoint if needed. Periodic SRB Involvement as Appropriate FOOTNOTES:
1. A One Step Review may be used for any LCR.
2. Appendix I provides information on the readiness assessment,
quick-look reports and checkpoints associated with life cycle
reviews.
3. For all other life cycle reviews, report to Chief Engineer if
significant unresolvable disagreements. 21
Page 21

22. Two Step PDR
Life Cycle Review Overview
Headquarters
Independent Program and Cost Evaluation
KDP-C
KDP-B PDR LCR
(2)
Quick Look Report (2)
Quick Look Report
PDR Readiness
Independent Integrated
Assessment (2) PDR
PDR Assessment
(30-90 days) (1-6 months) (30 Days)
-Required prior to LCR Technical Baseline Integrated
-Report to DA for life cycle reviews with Cost, Assessment of
preceding KDP B&C and during any Schedule, and Risk Technical and
major replan or rebaseline (3) Information Programmatic
Baseline
Not To Scale
Programmatic Data Drops to Resolve Tech
SRB (includes JCL Model) Issues/Risks, Update P/p Center MD
Cost/Schedule Baseline
Brief Brief Brief
(2)
CheckPoint if needed. Periodic SRB Involvement as Appropriate
FOOTNOTES:
1. A Two Step Review may be used for any LCR
2. Appendix I provides information on the readiness assessment,
quick-look reports and checkpoints associated with life cycle
reviews
3. For all other life cycle reviews report to Chief Engineer if 22
significant unresolvable disagreements 22 Page