1. Lessons Learned Implementing EVM in the EVA Office Project Management Challenge 2011 Long Beach, CA February 9-10, 2011 Les Sorge/SGT, Inc. Kristen C. Kehrer/NASA KSC Used with permission

2. Presentation Agenda Background Lessons Learned Management Reserve Budget versus Funding Risk Management Impact to EAC & MR Work Authorization Traceability & Scope Identical Structures and Titles for WBS and IMS WBS/Charge Numbers and EVM Control Accounts/Work Packages Institutional or Project OBS Data Integration with Multiple Tools Integration of Prime Contractor Data (1 control account or multiple) Setting and Validating the PMB Questions & Answers Acronyms and Reference Documents 2

3. Background Constellation (CxP) EVA Systems Project Office (ESPO) was selected as the first of two pilot projects for the NASA Earned Value Management (EVM) Capability Project The goal of the pilot project was to capitalize off of the EVM implementation efforts in EVA to develop an Agency EVM capability, test the Agency EVM capability processes and tools for integrated in-house and contractor EVM, and to incorporate identified changes and lessons learned into the documentation for Agency wide streamlined EVM processes and training The EVMS Capability team identified 33 gaps and 6 other areas of concern during the initial gap analysis of the ESPO pilot project in March, 2010 Closure plans were developed and documented with a plan to track and coordinate them to ensure proper vetting across affected organizations The closure plans were to be implemented and tested through monthly reporting cycles – did the fix work? The pilot project was prematurely terminated in June, 2010 because of changes/uncertainty within the Constellation Program and the ESPO Lessons learned from the pilot project were documented as part of the EVM Capability Project Two issue papers were written and presented to the EVM Capability Project Steering Committee for consideration Inconsistency Between Plan and Performance Business Rules and Actual Costs 3

4. Lessons Learned 4

5. Management Reserve Driving Event CxP provided direction for no Management Reserve (MR) for the projects Misunderstanding of financial reserves (Unallocated Future Expense – UFE) vs budget reserves (Management Reserve - MR) Lesson Learned – Allow projects to budget for and report MR to manage project risks and in-scope changes and use the ANSI/EIA-748 standard term of MR 5 Budget Process Funding Process NASA EVM Project CONGRESS & OMB Program Office Agency Baseline Commitment Management Baseline - PROGRAM AUW Management Reserve UFE Project Budget Base UFE Undistributed Budget UFE AUTHORIZED PMB AUTHORIZED Distributed Budget Key: Held Back Funds/Funding Commitment Agreement Held Back Funds/Funding Management Agreement Not Strictly “Budget”, but should be inviolate Budget Graphic modified from Chambers/Gibby 07/2010

6. Budget versus Funding Driving Event Misunderstanding the difference between budget and funding for EVM by all levels of management No distinction between EVM budget and funding driving replans of the EVM Performance Measurement Baseline (PMB) when funding changes occur EVM not fully integrated with current NASA financial processes and software tools Lesson Learned – To support EVM, NASA culture must differentiate between budget and funding and not replan the PMB (eliminate variances and trends) for every funding change. The validity and accuracy of the performance measurement and forecast data rely on a stable PMB 6 NASA EVM Project EAC The funding limit (obligation) represents the maximum funds available to accomplish the work VAC > 15% Notification to Program / Mission Directorate Program Office VAC BAC Management Baseline - PROGRAM Funding Plan Actuals/ETC/EAC Time Now The BCWS & BAC represent the work Annual PPBE (Funding Process) UFE The ACWP, ETC, & EAC represent the funds (i.e. money) required for that work Funding Limits Obligations AUW Management Reserve AUTHORIZED Undistributed Budget Initial Project Budget Base (BAC with MR = EAC) The BCWP represents the work accomplished Distributed Budget Define Project Budget (Budget Process) 6

7. Risk Management Impact to EAC & MR Driving Event Project risks captured in Integrated Risk Management Assessment (IRMA) tool but no process defined to utilize risks to develop initial project MR No process defined to include risks in an EAC update or how it relates to uncertainty Lesson Learned – Projects should develop processes to tie risk assessments to the development and justification of MR and EAC updates 7 Uncertainty (Funding Process) Risk (Budget Process) Sample Process Identify Initial Risks & Risk Mitigation Plans Generate a risk factor multiplier (0 to 1) Develop rule of thumb for unknown risks Define MR $ Calculate Uncertainty Update Risks & Risk Mitigation Plans Update EAC with Risk Updates Update JCL for KDP gate reviews Known Risks (known unknowns) Joint Confidence Level (JCL) Risk Register Cost S-curve Schedule S-curve Unknown Risks (unknown unknowns) ??? Rules of thumb based on historical projects MR $ = [Risk Impact $ x Risk Factor] + rule of thumb for unknown risks 70% JCL = Prob[Cost] × Prob[Schedule | Cost]

8. Work Authorization Traceability & Scope Driving Event The authorization document (PAD) did not authorize all of the ESPO work scope, i.e. lunar content, even though work was being performed/actuals collected on that work ESPO had to maintain a separate Deltek Cobra project with this scope Authorizing documents are not kept current with updates Limited scope description to reference for in-scope vs. out-of-scope discussions Lesson Learned Create authorization document(s) that cover the entire scope of work and use incremental authorization so all actual costs trace to an authorizing document Maintain all levels of authorizing documents to provide traceability Develop more detailed Statements of Work (SOW) between NASA Programs and Projects 8 Funding Process Budget/Funding Process NASA AA MDAA Program Mgr Project Mgr CAM Center Org Program PCA FAD - Formulation Authorization Document PCA - Program Commitment Agreement PAD - Project Authorization Document WAD - Work Authorization Document ITA - Internal Task Agreement * - Not currently considered an authorization document Program FAD Project FAD PAD WAD JSC ITA*

9. Identical Structures and Titles for WBS and IMS Driving Event The WBS element nomenclature and hierarchy is not consistently used within the project software toolset Lesson Learned – Use only the project approved WBS numbering scheme and nomenclature for integration and easy traceability of all project cost, schedule, and technical content If CAMs prefer the schedule outline to match the technical WBS (e.g. without the 6 digit project code and numbering hierarchy), then create a separate data field for the financial WBS 9 Deltek Cobra NASA MdM / SAP Primavera

11. WP should have durations of four months or less

12. During a replan, a WP should be closed with s=p to date, with the remaining work planned in a new WP CxP Reporting Level Control Account Work Packages / Charge Codes Planning Packages Control Account

13. Institutional or Project OBS Driving Event The Organizational Breakdown Structure (OBS) for the project Responsibility Assignment Matrix (RAM) was changed from the NASA center institutional organizations to the project organizational structure (i.e. integrated product teams) NASA operates with two organizational breakdown structures as a system of checks and balances Lesson Learned – The project needs to define the RAM OBS that best reflects the organizational authority and how work scope is subdivided between the orgs Questions to ask: Who is ultimately responsible for the work and has the budget authority? Who does the planning of the schedule and budget for that scope of work? To what level is control required or appropriate? Do the institutional organizations just provide extra bodies or do they provide a “finished product”? Should other NASA centers be listed as separate organizations and therefore, separate control accounts within the RAM? 11 Institutional (functional) Org Structure Project Org Structure Project Coordination Project Coordination Staff members are grouped into departments or product teams Staff members are grouped by functions or specialties such as Engineering, Safety & Mission Assurance, Public Affairs, etc.

15. Unique Task IDs

16. IMP Codes

17. WBS

18. EVM Percent Complete

19. File Formats

20. File Types (.xls, .xml, etc)

21. Etc.Interface Requirements Deltek wInsight Deltek wInsight

22. Integration of Prime Contractor Data(1 control account or multiple) Driving Event ESPO defined multiple control accounts (and therefore multiple WBS/charge numbers) for the prime contractor Each charge number required obligations which sometimes created “cost over obligations” on a charge number even though the total cost was within the total obligations Lesson Learned – Projects should develop contractor data integration plans and weigh the pros/cons of the different options 13 Example of Single Control Account Multiple Control Accounts Pros / Cons 731384.06.04 EVA HW Development 731384.06.04 EVA HW Development 731384.06.04.01 Suit Element 731384.06.04.01 Suit Element .31 Govt CA Govt CA WP / PP WP / PP .32 Prime CA 731384.06.04.03 Vehicle Interface Element WP / PP 731384.06.04.03 Vehicle Interface Element Govt CA WP / PP .31 Govt CA 731384.06.04.05 CSSS Contract (Prime CA) WP / PP .32 Prime CA WP / PP

23. Setting and Validating the PMB Driving Event Prime contractor operated for a year under letter contract without a Performance Measurement Baseline (PMB) Constant, significant changes to the Constellation Program/Projects funding did not allow for a stable baseline so as to set a PMB A PMB was developed for an Integrated Baseline Review (IBR), but the announcement to cancel the Constellation Program and associated reduction in funding changed the focus of the IBR to an FY10 review Prolonged proposal submittal and authorization process encumbered the change process Lesson Learned Clearly define expectations and requirements to initialize the PMB and begin EVM reporting to include letter contracts Utilize existing NASA resources such as the IBR Handbook, the EVM Handbook, and the Schedule Handbook to define appropriate EVM requirements Projects and contractors should strive for timely contract authorization and definitization of the base contract and subsequent contract modifications Given the fluid nature of funding, develop “right-sized” products Schedules and budgets that are not overwhelming to update Use of planning packages and summary level planning packages Utilization of rolling wave planning 14

24. Questions & Answers 15

25. Acronyms and Reference Documents Acronyms AUW – Authorized Unpriced Work CA – Control Account CAM – Control Account Manager CPR – Cost Performance Report CSSS – Constellation Space Suit System CxP – Constellation Program DRD – Data Requirements Document EAC – Estimate at Completion ESPO – EVA Systems Project Office ETC – Estimate to Completion EVA – Extravehicular Activity EVM – Earned Value Management EVMS – Earned Value Management System IBR – Integrated Baseline Review IMS – Integrated Master Schedule IPT – Integrated Product Team IRMA – Integrated Risk Management Application ITA - Internal Task Agreement JCL - Joint Confidence Level JSC – Johnson Space Center MR – Management Reserve OBS – Organizational Breakdown Structure PAD – Project Authorization Document PMB – Performance Measurement Baseline PP – Planning Package PPBE - Planning, Programming, Budgeting, and Execution RAM – Responsibility Assignment Matrix SAP – Systems Application and Products UFE - Unallocated Future Expense WBS – Work Breakdown Structure WP – Work Package Reference Documents NPR 7120.5D – NASA Program and Project Management Processes and Requirements CxP 72179 Anx01 – CxP ESPO EVM Implementation Plan ANSI/EIA-748 – Earned Value Management Systems 16