NASA BaselinePerformance Review (BPR) Steven Noneman, NASA HQ Robert Woods, Perot Systems Office of Program Analysis and Evaluation Office of the Chief Engineer Program Management Challenge, Galveston, TX February 2010 Used with Permission
Presentation Purpose• Describe NASA’s Baseline Performance Review (BPR) to the greater NASA community at PM Challenge 2010. Characterize the BPR objectives, scope, process, and products Summarize NASA’s technique for monitoring Agency performance against its baseline plans 2
BPR History and Context• In the spring of 2006, the Office of the Chief Engineer (OCE) was given the action by the Associate Administrator to develop a State of the Agency (SoA) process that provides the Agency Project Management Council (APMC) an independent top level view of the status of the Agency’s programs and projects.• The first SoA presentation, to the Agency PMC, was by the Science Mission Directorate (SMD) in August of 2006. The process has matured and continued to meet the objectives outlined by the Associate Administrator, the Associate Deputy Administrator, the Chief Engineer (CE) and the various stakeholders. The SoA evolved to a distinct monthly meeting in December 2007 and was officially entitled the Baseline Performance Review (BPR).• The BPR instituted a modified “balanced score card stop-light chart” process based on metrics encompassed in a core set of functional area questions and supporting raw data.• Participants from the projects, programs, Centers, Mission Directorates (MDs) and Mission Support Offices respond monthly to each core area question based on provided thresholds and key metrics for use in internal and external reporting.• All Mission Directorates are reporting at the BPR meeting the findings of the independent process.
BPR Objectives• Provide NASA Senior Leadership comprehensive, integrated, and objective information that describes the “performance-to-plan” of the Agency’s programs, projects, and institutional capabilities. A single place to see the full execution context of the Agency• Assure open cross-functional communications among NASA’s organizations to enhance Agency performance.• Identify and analyze performance trends and cross- cutting or systemic issues and risks• Serve the Agency’s decision making Councils as their ”execution arm” Within the NASA Governance model, the BPR is distinct from the decision-making Strategic Management, Program Management, and Operations Management Councils BPR is “action-oriented” to improve performance and inform Agency decision authorities of issues needing attention. 5
BPR Scope and Participants• The BPR is co-chaired by the NASA Associate Administrator and the Associate Deputy Administrator.• Membership includes the heads of NASA Staff Offices, Mission Directorates, Mission Support Offices, and its ten Field Centers.• Mission Support functions (e.g., finance, workforce, acquisition, infrastructure, IT, etc.) report monthly and Small Business and Diversity and Equal opportunity report quarterly.• Mission Directorates (covering all program areas) report monthly with one (and sometimes 2) “highlighted” in greater detail.• The Office of the Chief Engineer (OCE) leads a team of “independent assessors” from OCE, Program Analysis and Evaluation (PA&E), and Office of Safety Mission Assurance that analyze programs’ and projects’ performance. “Green/Yellow/Red” criteria were established to assess of technical, cost, schedule, and programmatic performance are provided monthly for all Mission Directorates.• PA&E provides the BPR Coordinator who manages the review by setting agendas, tracking actions, and assisting the co-chairs, OCE, and BPR participants. 6
BPR Detailed Process Flow 3 MD issue charts 1 Mission MDs Directorates GYR rankings for 2 Mission Support projects andOCE sends out Offices programs from the OCE (OCE,notifications to consolidates OSMA, and Assessor Group OCE 3users who answer NASA Centers input from PA & E) generatesquestions stakeholders Assessor BPR 1- 5 Group package 4 Programs 5 Technical Projects X-Cutting, OCE PCA Status reps. for MDs Non- Package Technical briefing to X-Cutting CE for 3 OCE feedback and approval Action items on BPR PA & E process BPR reps. for OCE Package is Package MDs Provides available presented Exe. Sum. for Read- at the BPR only viewing from site
Example Outcomes/Results• Cross-cutting issues Across missions, faulty material was discovered resulting in a mitigation process for the materials use saving time and resources.• Organizational communication issues Turned a conflict between projects on thermal protection systems into a collaboration between the major development projects who previously were competing for shared resources.• Mission support issues Workforce planning resolution was expedited when a project was confronted with changes resulting in 500 people at a Center needing reassignment because the project was delayed 2 years.• Senior leaders as an “Execution Board of Directors” Leadership asked and Centers provided additional workforce during transition to the NASA Support and Services Center because invoices were not getting paid on time, which had been costing projects late fees.• Mission specific issues Timely communication resulted in a Center director de-conflicting a vendors schedule between 2 NASA projects from different Centers.
BPR Project Questions O f Mission Directorate fi c e o Program Project f GUIDANCE FOR R&T P r QUESTIONS THRESHOLDS PROGRAMS/ o c PROJECTS u r e m e n t COMMENTS G R * What is the status and level G=Yes or <10%, of maturity of the research Are your top level requirements(such as 10%<Y<=20% objectives? requirements/objectives not * Have you baselined the KPPs) or objectives stable and state of the art/knowledge and stable or established/ Unknowns and lack of established/baselined? baselined, R>20% are moving off this clarity with Lunar requirements not stable or accordingly toward those requirements specific to established/ baselined objectives? X Y X technology development. G=Yes, Y<=20% Are you on track to meet your top level requirements/ objectives not requirements/ objectives? on track, R>20% requirements/ objectives not on track X G X Has requirement feasabilityDoes the project/program have plans that been established and have meet the needs/ objectives of the G=Yes, Y=minor weakness requirements flowed down community customer? identified, R=major weakness with traceability from customer identified needs? X G X G=Yes, Y=minor weaknessAre the results of research/ technology being identified, R=major weakness widely disseminated? identified X G X G= On track to complete all What is the readiness to support the next deliverables with no delay, R= Review? Not on track for deliverables or delay necessary X NA X G= All actions complete, Y>= What is the status of actions from the last 50% of actions complete, R <= review? 50% of actions complete X NA X 9
BPR Project Questions Currently, more than 1 risk is G= All "green" or equivalent, Y= * Is the risk posture or your researchWhat is your overall risk status/ posture (i.e. risk plan/ 5 red. Any "yellow" but no "red" or portfolio balanced and have you x 5 matrix)? equivalent, R= Any "red" or identified the top risks to the success of equivalent your research objectives? X R X G= Can be resolved withinAre risks manageable and appropriate ( i.e. risk plan/ 5 available resources, Y= Cant be x 5 matrix)? resolved by project but can with other resources, R= Only resolveable by Agency resources X G X Are technologies sufficiently mature (i.e. TRL or equivalent) at this point, or expected to be mature in G=yes, Y=one is no, R=more than time, per program/ project plans? one no X G X Will all the externally reported performance goals G=all req met, Y= 1 not met, R= (Annual Performance Goals)be met? more than 1 not met X G X Have the externally reported performance goals changed within the current year? G=no, R=yes X G XAre the externally reported performance goals per your plan? G=yes, R=no X G X G= no, Y= has changed once, R= Has the life-cycle budget (LCB)/ top line budget changed more than one time in changed since last reported? same phase or last 6 months X G X G= no, Y= has changed once, R= Has the current year funding changed since last changed more than one time in reported? same phase or last 6 months X G X Overrun 0<=G<5% LCB, Are costs over/ under running your phasing plan? 5%LCB<=Y<10%LCB, 10%LCB<=R X G X Is there a significant need to reallocate resources between Centers in order to accomplish technical tasks? G=no, R=yes X G X Is the current year funding per the approved baseline 90<=G<100%, 80%<=Y<90%, plan? R<80% X G XWhat is the difference between your current estimate at Difference of 0<=G<5%, * Are the cost estimates for obtaining completion and budget at completion? 5%<=Y<10%, 10%<=R your research objectives still valid? X G X Are mechanisms in place and functioning properly toensure that NASA receives external funding and sends out funding to others in proper time? G=no, R=yes X G XFor projects/ programs over $250M over 5 years, is the project/ program cost per baseline plan and beneath NA congressional limits (15% of Baseline plan)? 0<=G<5%, 5%<=Y<10%, 10%<=R X G X 10
BPR Project QuestionsAre major milestones (per Integrated Master Schedule G=Next major milestone slippage less * Are the schedule estimates for or equivalent) being met? or equal to 45 days, Y=Greater than 45 obtaining your major research but less than 90, R= Greater than 90 objectives milestones still valid? X G X * For those Outcomes/APGs that are Are all GPRA milestones on schedule? also key milestones are they still on G=Yes, Y= 30-60 days behind, R= track for completion per the annual greater than 60 days behind performance plan? X G XHas the schedule been significantly impacted since last G=No change, Y=Change but no reported? schedule slippage, R=Change but with schedule slippage X G XFor projects/ programs over $250M over 5 years, is the project/ program under congressional limits for key G=Next major milestone slippage less NA milestone slippage (less than 6 months)? or equal to 45 days, Y=Greater than 45 but less than 90, R= Greater than 90 X G XAre all external agreements in place and executing per plan? G = yes , R = no X G XAre key partners delivering their commitments on time? G = yes , R = no X G X Have key partners changed their commitments since last reported? G = no , R = yes X G XAre critical contracts in place and executing per plan? G = yes , R = no X G X Are undefinitized contract actions within guidelines? G = yes , R = no X G X X Are existing facilities available (or do results of required infrastructure assessment/business caseanalysis show existing facilities are available to support G=Yes, Y=No - scheduling issue. the activities)? R=No - need to refurb or build X G X Are all critical staff and skills identified and available 90% Critical staff<=G<100%Critical per plan? staff, 80%<=Y<90%, R<80% critical staff X G XAre all support staff and skills identified and available 90% Critical staff<=G<100%Critical per plan? staff, 80%<=Y<90%, R<80% critical staff X G X G=Y, Y= 1 required document notAre required documents per Agency and Center policy complete, R> required document not complete (i.e, FAD, PCA, Program/Project Plan)? complete X G X 11
Sample BPR Agenda10:00 Opening Remarks, Actions Associate Administrator, Coordinator10:05 Recovery Implementation Status Recovery Act Implementation Team10:15 Mission Support Reports - Finance Office of the Chief Financial Officer - Workforce Office of Human Capital Management - Acquisition Office of Procurement - Infrastructure Office of Infrastructure11:30 Highlighted Mission Support: Small Business Programs* Office of Small Business Programs12:00 Lunch12:15 Agency Aggregate Assessments OSMA Safety Assessment: NASA Mishap Posture Office of Safety and Mission Assurance Cross Cutting Non-Technical Issues Program Analysis and Evaluation - Special Topic: 2nd Quarter OMB Cost/Schedule Report Cross Cutting Technical Issues Office of the Chief Engineer - Special Topic: Flight S/W Complexity Jet Propulsion Lab Documents Compliance Office of the Chief Engineer Life Cycle Milestone Schedule Office of the Chief Engineer Center Summaries (Programs/Projects and Institutional) Center Representatives2:45 Roll-up of MD Programs/Projects Office of the Chief Engineer - JWST Project On-going Status/Recovery Plans Science Mission Directorate3:30 Highlighted Mission Directorate Office of the Chief Engineer Space Ops Mission Directorate (SOMD) Assessment* Space Operations Mission Directorate Launch Services Program Status/Recovery Plan Space Operations Mission Directorate4:45 Action Summary Coordinator5:00 Adjourn *Specific topic is rotated Associate Administrator 12
Samples of BPR Analysis Content• Finance• Workforce• Acquisition• Small Business• Infrastructure• Safety and Mishaps• Crosscutting Issues• Field Center Summaries• Independent Assessments of Programs and Projects (“Stoplight” ratings) 13
Undefinitized Contract Action (UCA) Status February 27 – March 27, 2009 16 14 1 $1.3B 12Number of UCAs 4 $27M 10 8 6 $836M 4 8 9 2 1 $35M 2 1 $3.5M 4 0 1 $700K 1 1 $749K 10 1 2 3 4 5 6 7 8 9 er er er er er er er er er er t t t t t t t t t en en en en en en en en en t en C C C C C C C C C C 1-180 Days 180-360 Days >360 Days
FY08 Federal Agency Prime Small Business Dollars Overview Top Seven Agencies w/o DOD General Services Administration Department of Justice Department of Homeland Security Department of Veterans SB Dollars Affairs Total Dollars ObligatedNational Aeronautics & Space Administration Health and Human Services Department of Energy $0 $5,000 $10,000 $15,000 $20,000 $25,000 In Million $ Total Dollars Small Business Dollars
BPR: I&A Functional Areas ARC DFRC GRC GSFC JPL JSC KSC LaRC MSFC SSC RemarksFacilities JSC damage from Ike could impact JWST Project.. Chamber A MOU G G G G G Y G G G G and Management Plan not complete. Concern about communication Design & Construction between Eng Div and Facil Div at JSC Facility repair backlog is 8.6% of facility value with increasing Y Y Y Y Y R Y Y Y Y trend.JSC damage from Ike could impact SOMD Operations Facility Maintenance DFRC, KSC, and SSC were changed to yellow since they have not G Y G Y G Y Y G Y Y submitted revised Master Plans. GSFC presented new Master Plan, Planning/Real Estate but has not submitted document for formal approval.Logistics Delay in replacement of critical civil service personnel at GRC and KSC. Also at KSC, a concern existed as retiring civil servants positions were not backfilled. Situation improved by increased use of G G Y G G G G Y G Y Support Services Contractors. Sinlge point failure for multiple logistics functions at LaRC and SSC. Anticipated hire at SSC Supply currently on hold. Delay in replacement of critical civil service personnel at GRC and KSC. GRC expects to fill vacancy by end of 2008. At KSC, a concern existed as retiring civil servants positions were not backfilled. G G Y G NA G G Y G Y Situation improved by increased use of Support Services Contractors. Single point failure for multiple log functions at LaRC and SSC. Anticipated hire at SSC currently on hold. GSFC filled HQ Equipment SEMO position. CFO management control findings related to property management are being addressed, however center IPO staffing concerns to support transition workload and FAR update present an on-going staffing Y Y Y Y Y Y Y Y Y Y issue. JSC in process of filling Contract Property Administrator position. JSC also submitted request to fill the following: 2 positions for Property Disposal, 1 position for Shuttle Clerical (Property) and 1 position for Packing/Shipping in Transportation. Contract Property Agency risk for not meeting EO 13423 (alternate fuel): discussions with MSFC to resolve center contribution ongoing. Single point failure for multiple log functions at SSC, anticipated hire currently on hold. G G G Y G G G Y Y Y GSFC loss of CNG capability for approx. 13 vehicles. KSC added additional contractor to assist with transportation functions. At LaRC, TO/SEMO is one person, he will be out in Jan. (medical leave) no Transportation replacement appointed. Increase in excess property workload due to Shuttle and Hubble G G G Y G Y Y G Y Y disposition at GSFC, MSFC, KSC, SSC & JSC. Single point failure Property Disposal for multiple log functions at SSC.
NASA and Contractor Safety Statistics Dashboards February Mishap OCOJan 2008 – Mar 2008 Jan 2009 - Mar 2009 Jan 2008 – Mar 2008 Jan 2009 - Mar 2009 Jan 2008 - Mar 2008 Jan 2009 - Mar 2009 March Mishaps: • No Type A Mishaps • One Type B Mishap: DRC- (Deluge System Release in Shuttle Hangar - 3/13/09) • One Potential Type B Mishap: DRC- (Global Hawk Electronics Failure 3/12/09) – currently a Type D Data does not include 4/2/09 Type B Mishap: Water Drill mishap resulting in multiple amputations Close Call Incidents Show An Increase From Previous Reporting Period (Jan – Mar 2008).
False Material Certification Status as of: Date x, 200x• Issue Description: Company xyz, manufacturer of a material used in flight hardware, was indicted on federal charges of issuing false material certifications for supplies they sold. Material of uncertain pedigree may have been used by NASA projects. Mission Directorate Impact MD 1 MD 2 MD 3 MD 4 • Cleared: LRO, LCROSS, Ares • Suspect material found in • Suspect material found• No use of Suspect I-X, J2-X, Orion PA-1, HRP Kepler spacecraft in a on the Taurus and Delta material determined as ISS Experiments & ETDP IV vehicles used by critical area wherein failure yet • No other project could be catastrophic. OCO, Glory, and NRO L-• Supplier traceability assessments required • Suspect material also found 26. investigation still in the AIA instrument on • NRO mission cleared. underway SDO. • SSP & ISS cleared. • Other projects are • LSP - All vehicles now investigating usage. cleared.Information on Agency Actions and Status (Estimated Completion) POC: Ken Ledbetter • The Kepler project procured material from Company xyz who obtained some of the material from Company xxx Inc, including the material for the Telescope Spider Hub Assembly. Company xyz had 46 different material procurements. The Kepler project performed tests on residual material at xyz and confirmed that it was sufficient to satisfy their mission application. • GIDEP indicates earliest occurrence was no later than 2002 with the latest false certification in 7/03 for a purchase order in 11/04 by Company xyz. • SDO project also found suspect material in the AIA instrument. AIA has now been cleared. • Centers responding with evaluations on a project-by-project basis. • Top Level Description/ Status: Material found to have uncertain pedigree due to false certifications. Extent of usage in NASA projects and resulting impact, are under investigation.
Agency Programs: PCA Status Highlight indicates change from previous report MD Formulation ImplementationSMD Astrophysics Planetary Science Astrophysics Physics of the Cosmos G Discovery (8/19/08) Y SOFIA (7/26/07) Cosmic Origins Y Mars Exploration (1/26/07) G Hubble Space Telescope (2/27/09) G New Frontiers (3/18/08) G Fermi (formerly GLAST) (8/18/08) Planetary Science Heliophysics G James Webb Space Telescope (3/17/09) G Lunar Quest Explorer (2/2/09) G ExoPlanet Exploration (4/14/09) G Living with a Star (8/25/08) Earth Science G New Millennium (10/6/08) G Earth Systematic Missions (4/14/08) Y Solar Terrestrial Probes (12/21/07) G ESSP (4/25/08)ESMD Constellation (in G Human Research (5/21/07) Rev A in Rev Lunar Precursor Robotic (Waived requirement) development; expect Technology Development (10/21/07) G approval at KDP-II)ARMD G Airspace Systems (9/14/2007) G Fundamental Aeronautics (5/21/07) G Aviation Safety (9/17/2007) G Aeronautics Test (5/21/07)SOMD G ISS (8/01/08) G SSP (1/26/07) G LSP (1/28/08) G SCaN (10/16/08) G RPT (1/23/09) G TDRS K/L (4/18/07) CHSP (waived requirement)PCA Status Legend Dates are last signed PCA G = Signed PCA with no update required Y = MD has identified for update R = No signed PCA BLACK = NAFebruary 2010 21
Agency Review ScheduleFebruary 2010 For NASA Internal Use Only
NASA Monthly Baseline Performance Review (BPR) GSFC Center Summary Input Highlights from 11/17/2009 Programs/Projects Technical Authority/Mission Support/Institutional Summary Performance Institutional Risk to Mission ESMD: ESP Budget/Finance: LRO – Supported LCROSS Lunar Impact, data being analyzed. Grants reconciliation and transition: The ORR for the Phase 2 transition Transition to SSMO will occur in December. to NSSC is scheduled for November 30th; will complete GSFC transition SMD: activities. HST- SM4 SMOV activities complete. Workforce: – Working possible options to increase FY10 reserves. OTE Early Career Hiring: FY09 - 66 on-board; FY10 – 10 with an additional CDR complete. 10 offers accepted and negotiating dates. GLORY- Successful CMC and ESD rebaseline reviews conducted. LV actions being worked prior to DPMC. Acquisition: GPM – Re-planned road to KDP-C. Working toward Mission CDR Aerospace Corp. Support: Consolidated Agency contracting approach planned for December 14-17th. being pursued. GSFC had restricted authorization to use the existing GSFC LDCM contract for critical tasks ending in December. “Authorization” to seek services on the Airforce contract granted but is not expected to save NPP - Systemic issues with systems engineering and mission resources and requires unnecessary actions. assurance escapes on IPO-provided instruments reduce confidence in Instrument life expectancy to be less than the 5-year design life. SCNS contract protest activities continue. NENS contract has been MMS – S Band antenna height change complete. Swedish Univ, KTH extended through April 2010. The delayed transition from NENS to SCNS ability to deliver Spin Plane Probe Electric Field Instruments. continues to impact the contractor’s ability to retain experienced employees to perform the work. RBSP–Schedule delays are being worked. • Agency-level protests denied on August 11th and the Corrective Action SDO – LRD is 2/3/2010. AIA Hopa Circuit anomaly. Plan (CAP) implementation was initiated on August 17th. SAM – Power Distribution Box is integrated onto the flight suite. • A protest was submitted on August 21st to GAO with regard to the CAP. GOES-R – ITT performance on ABI. CAP implementation permitted to continue. GAO decision due NLT November 30th. NOAA Climate Sensors • Congressional request for information with regard to the CAP was SOMD: submitted October 22nd. Ground Network Space Network- See Acquisition, SCNS. TDRS – Continue to Work with BSS regarding requirements flow-down to subcontractors. SGSS – RFP development activities continue. ELC – Successful SMSR October 23rd. Decks #1 and #2 were Launched November 16th.February 2010 23
SMD Overview November 2009 Status: – 2 project rated Red overall: NPP, SOFIA – 4 projects rated Yellow overall: JWST, MSL, Juno, Glory – SOFIA changed from Yellow to Red due to eroding schedule, cost issues & delay in rebaseline APMC. – Juno progress degraded from Green to Yellow; Glory returned to Yellow pending replan DPMC. – Replan DPMCs pending on Glory, Aquarius, & MSL. – MRO in safemode due to computer resets; Restarts unsuccessful for Last IceSat laser failure Technical Schedule Planetary: MSL actuators. MRO safemode implications on Astrophysics: developing missions using heritage avionics (e.g. Juno, Earth Sci: Glory Taurus XL return to flight schedule. Potential GRAIL, MAVEN) further NPP launch delays from instrument rework. All: Continued difficulties developing instruments; Implications from usage of Western Titanium. Cost Programmatic Planetary: Funding of MSL slip severely impacts division All: Access to Space continues to be an issue: Delta II Astrophysics: Overall division budget issues, esp JWST. replacement; Taurus XL failure, & impact on Glory. Very crowded launch manifest in fall 2011. Watch list Items: Uncertainty of SMAP launch date for project planningFebruary 2010 For NASA Internal Use Only 24
ESMD Projects Status Roll Up Continued2009 Month A B C D COMMENTS S O N S O N S O N S O N Y Y G Successful Launch!! October 28, 2009 at 11:30 am ET Post-Flight Data Review working group will meet twice a week to review progress of data collection and reductionAres I-X The first stage solid rocket motor recovered; hardware disposition is in progress; data recorder removed and shipped to Lockheed Martin. Lockheed Martin who will provide best and final set of all flight data from a combination of data from telemetry and the on-board recorder Lunar Insufficient funds in PMR09 budget to meet HLR in 2020. Surface Constrained to “study only” after FY2010. SystemsFebruary 2010 25
PSD Budget Reporting Matrix (11/16/09) External Internal Project MPAR OMB Qtrly (Baseline) IBPD (FY10)*1 (4QFY09) PCA/PRLA/other*2 Development $968.6 $1,631.0 $1,719.9 $1657 ₃ MSL LCC $1,642.2 $2,299.3 $2,394.3 $2,318.5 LRD Sep 2009 Oct 2011 Nov-11 Oct-11 Development $427.0 $427.0 $427.0 $407.7 GRAIL LCC $496.2 $465.6M* $496.2 $468.2 LRD Sep 2011 Sep 2011 Sep 2011 Sep 2011 Development $742.3 $742.3 $742.3 $695.3 JUNO LCC $1,107.0 $1,091.9M* $1,107.0 $1,050.0 LRD Aug 2011 Aug 2011 Aug 2011 Aug-11*1 - IBPD is updated through N2 automatically and as of 4-15-9 N2 has not been updated to reflectthe FY09 Op Plan Changes nor the GRAIL and Juno KDP-C decisions.*2 - The PCA (Program Commitment Agreement) is located in Science Works under RequirementsManagement system, Planetary Science, Program, Key Document PCA, and KPD*3 - MSL project assumes an approved $13.4M adjustment from Phase E to Phase C/D, which isNOT included in the $1657M internal development number. 26
Oct Nov JWST YY YStatus: In Phase C Y Technical: G Schedule: • NIRCam off-axis image quality • WATCH: Low near-term reserves pose a • NIRSpec microshutter separation - decision threat to LRD. to change coating R Cost: G Programmatic:• FY10 reserves negative & FY11 reservesonly 0.9%• Project requested $20M addition to FY10reserves. HQ told project to continue to fundJSC facility mod, with decision on extra $20Min 4-6 weeks.
Summary• NASA Senior Management monitors Agency-Wide performance monthly through its baseline Performance Review (BPR) Process Each Program area is assessed quarterly on a rotating basis• The automated tool is coming online at MSFC• BPR continues to evolve and improve
BPR Actions and Status7.23.2009 BPR Meeting• 2009-07-026: PA&E – Revisit the FY09 Q3 Contract reporting chart to clarify the way the information is being conveyed. (E.g. Contract base value apparent misalignment with LCC.) Due: October 2009 BPR Status: Propose to CLOSE. PA&E has met with ESMD to identify how to add clarity to this reporting. ESMD will provide additional information that clarifies the contract values. Interim information was presented in September BPR by PA&E. ESMD has provided data from the Ares I and Orion. Language has been added to the contract chart to explain what contracts are and are not included in contract reporting to OMB.
Green Criteria A Category for a Project is Green if: There are minimal programmatic issues and adequate margin (budget reserve,schedule slack, technical margin), or capability (for operational missions), tocomplete the Project on plan. A Project is Green overall if: All 4 categories are Green, or One or more are Yellow and the issues are collectively not a threat/impact to theproject by virtue of their being solvable within the projects planned resources.(Margin in one category may provide capability to resolve a lack of margin in anothercategory.) A Program is Green if: The constituent projects are Green overall, or are Yellow with realistic, defined plansto return to green and all other program elements are on budget and schedule.
Yellow CriteriaA Category for a Project is Yellow if:The remaining margin within the category appears inadequate to meet Projectrequirements or external commitments, however, mitigation plans are in place or indevelopment to recover margin and achieve commitments within the availableresources of the Project. (For operational projects in the technical category "margin"may be interpreted as operational capabilities.)A Project is Yellow overall if:Any one category is Red, the overall is at least Yellow, orOne or more categories are Yellow and the issues are collectively a threat to theprojects ability to meet their commitments within the projects planned resources.A Program is Yellow if:Any of its constituent projects are Red overall, or if one or more are Yellow and theissues are collectively a threat to the Programs ability to meet its commitmentswithin planned resources.
Red CriteriaA Category for a Project is Red if:There is minimal margin or negative margin within the category such that theProject is estimated not to meet its requirements or commitments without significantimpact to other categories. There is high risk that the Project will require externalresources or relief from performance or programmatic requirements.A Project is Red overall if:One or more categories are Red, and it is estimated that the Project will not havesufficient resources to meet its commitments.A Program is Red if:Most of its constituent projects are Red overall, or if one or more are Red and theissues are collectively a threat to the Programs ability to meet its commitments.
Assessment Rubric for Space Flight Projects and Multi-Project Programs Cost Schedule Technical ProgrammaticCost estimates remain consistent with cost Schedule progress remains consistent with Technical progress remains consistent with Programmatic performance remains consistentcommitments: schedule commitments: technical commitments: with program/project requirements:-- Formulation: Current estimated -- Formulation: The current estimated -- Formulation: The project/program is -- Formulation: An approved FAD is in place andformulation costs are consistent with the authority-to-proceed to implementation date is making planned progress in developing planned progress is being made in completingprogram FAD. at or before the FAD. technical design, reliability, safety, and risk programmatic arrangements necessary to enter-- Development: Current estimated LCC at -- Development: Current estimated operational mitigation requirements necessary to development as planned.completion is at or below the LCC readiness date (ORD) is at or before the ORD proceed to implementation as planned. -- Development: Top-level objectives andcommitment. For multi-project programs, the commitment. For multi-project programs, the -- Development: Technical requirements programmatic requirements in the plan are stablecurrent cost estimate by year for the current estimated schedule for planned start are stable and progress and risk and programmatic capabilities are beingprogram’s projects, including reserve, is dates for future projects is consistent with the management is consistent with plan while developed/ managed as planned. Plans arewithin the 5 year cost run-out in the program program plan. maintaining safety. For multi-project consistent with 7120.5 or necessary waivers areplan. The program is consistently executing -- Operations: Planned operations, including programs, any lack of planned technical in place. A multi-project program is effectivelyits projects within their cost cap or maintenance, are proceeding on schedule. progress at the project level is not resulting managing its projects, with new projectscommitment. and there are minimal issues and adequate in an increased risk to the program’s beginning per the Program Plan.-- Operations: Annual operational costs are schedule margin to complete the Program or technical commitments and objectives. -- Operations. Operational requirements areat or below planned costs for these Project on plan: -- Operations: Operational systems are stable consistent with plan, and operations areoperations --Funded schedule reserves & phasing are delivering the planned level of services and being managed as planned. Retirement/ transitionand there are minimal issues and adequate consistent with plan and are being drawn down reliability, with a general trend towards plans are being developed/ implementedcost margin (budget reserve) to complete at a rate consistent with plan and emergent improved safety, and efficiency. consistent with the timing for these events.the Program or Project on plan: schedule risks/threats. For multi-project and there are minimal issues and adequate and there are minimal issues and adequate-Lifecycle cost reserves are consistent with programs, any faster-than-planned project technical margin to complete the Program programmatic flexibility to complete the Programplan and are being drawn down at a rate draw-down of schedule reserve is not or Project on plan: or Project on plan:consistent with plan and emergent cost impinging on the ability of the program to meet -- Technical reserves are being drawn down --Key contract actions are being undertaken andrisks/threats. For multi-project programs, its overall schedule commitments. are consistent with plan; technical completed as planned. The multi-projectreserves are being managed consistent with --Key interim milestones (KDP as well as WBS difficulties/risks encountered are typical for program’s acquisition strategy (both contractedthe annual reserves planned for each milestones) being met. this type of project/program and are being and in-house) is resulting in the timely start ofproject. -- For multi-project programs, any delay in mitigated or controlled as required to make projects and delivery of project elements.-- The project/program will not require project schedule is not jeopardizing inputs planned technical progress. --Key critical workforce, access to key facilities,additional funds to complete the current required for another project or resulting in an -- Are consistent with the requirements of EVM, and management systems are in placefiscal year with required carryover. increased risk to the program’s schedule safety, reliability & quality assurance, and consistent with plan.-- Phasing of cost reserves are consistent commitments. incorporate good systems engineering & --Key partnerships are in place and operating aswith emergent risks or threats. -- External partners are providing key integration practice. planned; external requirements are stable; and-- Current year budget level and timing of deliverables on schedule. -- External partners are making planned there are no other factors external to thefunds distribution are consistent with plan or technical progress and external events project/program (e.g., new Agency requirements)commitments are not threatened as a result (e.g., import approvals) are not altering which are significantly disrupting plannedof any or delay in current year funding. technical requirements. project/program management.-- External partners are providing key -- For operational projects, the ground andresources as planned. in-flight hardware is operating satisfactorily on prime or backup units and mission requirements are being met.
Assessment Rubric for Space Flight Projects and Multi-Project Programs (continued) Cost Schedule Technical ProgrammaticIn general, the program or project follows In general, the program or project follows the In general, the program or project follows In general, programmatic performance remains the green assessment guidelines, green assessment guidelines, but the green assessment guidelines, consistent with program/project but cost commitments are now schedule commitments are now but technical commitments are now requirements for a green assessment, but threatened. threatened. threatened: remaining programmatic flexibility may beRemaining cost margin appears inadequate Remaining schedule margin appears Remaining technical margins appear inadequate to meet requirements due to to meet requirements due to one or inadequate to meet requirements due inadequate to meet requirements one or more of the following programmatic more of the following deviations from to one or more of the following due to one or more of the following deviations from plan however, mitigation plan, however, mitigation plans are deviations from plan, however, technical deviations from plan plans are in place or in development to in place or in development to mitigation plans are in place or in however, mitigation plans are in recover margin and achieve commitments recover margin and achieve development to recover margin and place or in development to recover within the planned resources: commitments within the planned achieve commitments within the margin and achieve commitments -- Key contract actions are not being undertaken resources of the Program or Project: planned resources: within the planned resources: or completed as planned.-- Lifecycle cost reserves (including those -- Schedule reserves are being drawn down -- Technical reserves are being drawn down -- A multi-project program’s acquisition strategy held at the program level) are being faster than planned or as required for at a rate inconsistent with plan; or (both contracted and in-house) is not drawn down faster than planned or emergent schedule risks/threats. technical difficulties/risks resulting in the timely start of projects and required for emergent cost -- Key interim milestones (KDP and/or WBS encountered exceed those which delivery of project elements. risks/threats. milestones) have slipped. are typical for this type of -- Key critical workforce, access to key facilities,-- Phasing of cost reserves are inconsistent -- For multi-project programs, delay in project project/program or are not being EVM, and management systems are not with emergent risks or threats. schedules is resulting in an increased mitigated or controlled as required consistent with plan..-- Current year cost growth is exceeding risk to the program’s schedule to make planned technical progress. -- Key partnerships are not in place or operating planned current year cost reserves commitments. -- Are not consistent in some key areas with as planned; external requirements are not and will require a re-phasing of the -- External partners are delayed in providing the requirements of safety, reliability stable; or there are other factors external annual budget in order to complete key deliverables or external factors are & quality assurance, and the to the project/program (e.g., new Agency the fiscal year with required delaying completion of a key milestone. incorporation of good systems requirements) which are significantly carryover. engineering & integration practice. disrupting planned project/program -- A reduction in current year budget or -- External partners are not making planned management. slow- down in funds distribution will technical progress or external require the project/program to events (e.g., import approvals) are rephrase future year annual costs. altering technical requirements.-- External partner changes or factors -- For operational projects, ground or flight external to the project/program are capabilities have been curtailed, but placing additional pressure on plans are in work to recover the project/program costs. capability or develop a workaround-- For operating missions, in flight anomalies such that mission requirements will are draining cost reserves. still be met.
Assessment Rubric for Space Flight Projects and Multi-Project Programs (continued) Cost Schedule Technical ProgrammaticThe Program or Project is estimated not to The Program or Project is estimated not to The Program or Project is estimated not to The Program or Project is estimated not to be meet its cost commitments without meet its schedule commitments without meet its technical requirements able to meet its requirements due to significant impact to other significant impact to other categories. without significant impact to other programmatic issues affecting the other categories. The key end-of-phase KDP milestones categories:. categories.Depending on phase, the projects are unlikely to be met. A project in formulation is not making the A formulation project has failed to make the formulation costs, or development For multi-project programs, the planned start- technical progress required to enter programmatic progress required to be costs, or annual operating costs up of future projects is delayed beyond development. confirmed for development. are exceeding the planned budget program commitments or objectives. For a project in development, technical A development project lacks an approved plan or beyond all planned reserves. For operations missions, the current failures or delays preclude the has failed to implement that planFor multi-project programs, total annual operations schedule is insufficient to project from being able to meet its consistent with meeting its commitments. costs for the program’s projects meet project commitments or the technical requirements and The multi-project program lacks an exceed the program’s 5 year mission failed prior to the planned end- commitments. approved PCA or project plan, or has planned run-out, including reserves. of-operations date. For multi-project There is minimal or negative technical failed to implement the projects require toThere is minimal or negative cost margin programs: The current operations margin such that there is high risk meet its commitments. such that there is high risk that the schedule is insufficient to meet that the Program or Project will There is limited or no programmatic flexibility such Program or Project will require program commitments or the failure of require additional technical that there is high risk that the Program or additional funding or relief from one or missions prior to the planned resources or relief from performance Project will require relief from performance performance or programmatic end date has resulted in the project not or programmatic requirements due or programmatic requirements due to one requirements due to one or more of being able to meet its commitments or to one or more of the following or more of the following deviations from the following deviations from plan for objectives. technical deviations from plan for plan for which the project/ program has a realistic recovery strategy has not There is minimal or negative schedule margin which the project or program has not identified a realistic strategy for been identified: is such that there is high risk that the not identified a realistic strategy for recovery:-- Lifecycle cost reserves (including those Program or Project will require recovery: --Key contract actions are not being undertaken held at the program level) are being additional schedule or relief from -- Technical reserves are being drawn down or completed as required. The multi- drawn down at an unsustainable performance or programmatic at an unsustainable rate or project program’s acquisition strategy rate and remaining reserves are requirements due to one or more of the emergent technical difficulties, risks, (both contracted and in-house) is not insufficient given emergent following deviations from plan for which or threats are not being resolved as resulting in the timely start of projects and risks/threats. a realistic strategy for recovery has not required to meet the technical delivery of project elements.-- Phasing of cost reserves are inconsistent been identified: commitments. --Key critical workforce, availability of key with emergent risks or threats. -- Schedule reserves are being drawn down at -- Are not consistent with the requirements facilities, EVM, and management systems-- Current year cost growth exceeds an unsustainable rate and remaining of safety, reliability & quality are missing or inconsistent with meeting planned costs and cannot be reserves are insufficient given assurance, and the incorporation of the project’s commitments. accommodated by rephrasing emergent risks/threats. good systems engineering & --Key planned partnerships did not materialize, annual costs. -- A slip of one or more key interim milestones integration practice. quit or not to delivering consistent with-- A reduction or delay in current year (KDP and/or WBS milestones) is -- Lack of technical progress by external project/program commitments; unstable or funding is inconsistent with inconsistent with the planned ORD. partners or external events (e.g., newly emergent external requirements are maintaining the overall LCC For programs, delay in project import approvals) are inconsistent not implementable; or other external commitment. schedules is inconsistent with with meeting the project/program developments are inconsistent with the-- A loss of key external partners or program’s schedule commitments. technical requirements, project/program commitments. resources is inconsistent with -- An external partner delays or delays caused commitments, or objectives, or keeping the project or program by other external factors are additional/different external cannot within planned costs. inconsistent with the planned ORD. be met within plan.-- For operational projects, in-flight For multi-project programs, delays -- For operational missions, in-flight anomalies have drained all cost caused by external factors are anomalies have used up reserves and the project will need inconsistent with the program’s redundancy and made it unlikely more funds to complete the prime schedule commitments. that the mission will achieve its mission. mission requirements.
BPR Process Organizational Structure As of June 3, 2008 Associate Administrator And Associate Deputy Administrator Chief Engineer Division Chief for Engineering and Program & Project Management BPR Team LeadFebruary 2010 For NASA Internal Use Only 38
BPR Process Organizational Structure As of June 3, 2008 BPR Executive Secretariat Team Lead PA&E Technical OSMA OCE Admin Center OCE MD P&P Team Lead Cross-Cutting Team Lead Support POC’s Assessment Team Lead Institutional Non- Special Topic / Program Mission Mission OSMA PA&E Assessment Technical Root Cause & Directorate Centers Support Assessor Assessor Team Lead Cross Cutting Analysis Lead Projects (*) Offices (One representative from each MD) MSO Representatives I&A IT OCFO OHCM OPII PROC (*) One representative from eachFebruary 2010 For NASA Internal Use Only 39
Primary BPR Deliverables• Monthly meeting BPR presentation package• Executive summary and pre-brief• Meeting Summary Notes and Post-brief• Executive Summary for the Administrator and Deputy Administrator• Meeting Actions List and tracking 40