1. Overview of GAO Quicklook
PM Challenge
February 2012
Katie Gallagher and Tracy Osborne
OCFO/SID
2. Outline
• Background
– Why is this important?
• GAO’s Approach
– What projects are involved?
– Why does GAO request technical data?
• What should I expect if my project is in
Quicklook?
• What does NASA’s data say?
• Agency Messages
1/22/2012 2
3. Background
• The GAO’s Assessment of Large-Scale [NASA] Projects is
Congressionally-mandated
• GAO sends its findings to Congress in an annual “QuickLook
Book” (QLB) Report focusing on changes in project cost and
schedule and providing GAO’s explanations for these changes
• GAO is frequently called upon to testify about the results.
• Engagement objective: “Identify and gauge the progress and
potential risks associated with selected NASA acquisitions.”
(i.e., projects)
• GAO uses data from QuickLook Book (QLB) in other audits, such
as the ongoing High Risk audit, and vice versa.
1/22/2012 3
4. Why the Stakeholder Interest?
• NASA’s historical inability to control cost and schedule
growth
– JWST, CxP, MSL growth cited in several provisions establishing new
reports.
– A 2004 GAO finding that “NASA lacks discipline in cost estimation.”
– Multiple Operating Plans per year to address cost growth or phasing.
• Based on past Performance NASA is required to
report to OMB, Congress and GAO routinely on
Project cost and schedule.
– All appropriations since FY 2008 have included direction for GAO to
“Identify and gauge the progress and potential risks associated with
selected NASA acquisitions.”
1/22/2012 4
5. Typical Results from Quicklook Reports
• Most of NASA’s cost growth for the current
portfolio is attributed to projects baselined
prior to 2009
• NASA projects do not meet GAO-identified
best practice standards for technology
maturity and design stability
• Other challenges that affect NASA project
outcomes include Funding, Launch Vehicle,
Development Partners, Parts Quality,
Contractor Management
1/22/2012 5
6. Why do they ask about TRL’s and drawings?
GAO’S APPROACH
1/22/2012 6
7. GAO’s Report is in Two Parts
• Agency Observations
– Examines adherence to NASA policies such
as 7120.5E and 1000.5A
– Examines potential systemic reasons for
NASA’s cost growth
– Assesses NASA’s efforts to improve cost and
schedule growth, including JCL policy
• Project Assessments
– Provides an update on status of each
project included in the audit
– Outlines challenges specific to each project
that GAO believes may lead to cost growth
GAO Quicklook assesses performance of NASA’s major projects
and the Agency’s management of those projects.
1/22/2012 7
8. How does a project get picked?
• GAO evaluates all spaceflight projects in formulation if a contract is
released that has development content and is greater than $25 M
– Can include projects in Phase A and Pre-Phase A if they have such a
contract
• GAO evaluates all spaceflight projects in development with a life cycle
cost greater than $250 M
• GAO continues to track projects in Operations until NASA negotiates
removal from Quicklook
– Minimal data collection on Operations projects; main focus is cost and
scientific results
• For projects in formulation, GAO expects NASA to provide cost and
schedule ranges
• For projects in development, GAO expects NASA to provide baseline
cost, schedule, design and technical information
1/22/2012 8
9. Projects Included in Audit
2008 2009 2010 2011
Requirement for Inclusion
* Estimated Lifecycle Cost * Estimated Lifecycle Cost * Estimated Lifecycle Cost * Estimated Lifecycle Cost
>$250M >$250M close to $250M (if it is called close to $250M (if it is called
Lifecycle * In development * In development out in the budget as a out in the budget as a
* In formulation if it also has * In formulation if it also has discreet project then it may discreet project then it may
Phase as of
released a development released a development be included) be included)
Project Entrance contract valued at >$50M contract valued at >$25M * In development * In development
Conference * If in operations, it is not * In formulation if it also has * In formulation if it also has
(April 2011) (consistent with other included released a development released a development
external reporting contract valued at >$50M contract valued at >$50M
requirements) * In operations, in the prime * In operations, in the prime
mission phase mission phase
1 Aquarius Development X X X X
2 Ares Formulation X X X
3 Dawn Operations X X X
4 Fermi/GLAST Operations X X X
5 GPM Development X X X X
6 Glory Development X X X X
7 GRAIL Development X X X
8 Herschel Operations X X X X
9 ICESat II Formulation X X
10 JWST Development X X X X
11 Juno Development X X X
12 Kepler Operations X X X X
13 LADEE Development X X
14 LDCM Development X X X X
15 LRO Operations X X X X
16 MAVEN Development X X
17 Mars 2016 (TGO) Formulation X
18 MMS Development X X X
19 MPCV Pre-Formulation X
20 MSL Development X X X X
21 NPP Development X X X X
22 OCO/OCO-2 Development X X X
23 Orion Formulation X X X
24 Phoenix Closed Out X
25 RBSP Development X X X
26 SDO Operations X X X X
27 SLS Pre-Formulation X
28 SMAP Formulation X X
29 SOFIA Development X X X X
30 Solar Probe Plus Formulation X X
31 TDRS Development X X
32 Wise Operations X X X X
1/22/2012 9
10. GAO “Business Case” Model
PDR CDR SIR
Production
Pre- Phase Phase Rarely Applies
NASA phase A A B
Phase C Phase D Phase E
to NASA
KDP-B KDP-C KDP-D
Projects
KDP-A KDP-D
DOD Technology Development
Engineering Manufacturing and
Development
Production and Deployment
Operations and
Support
PDR MS-B CDR MS-C
Concept and Technology Engineering Manufacturing and
GAO Development Development
Production
Knowledge Knowledge Knowledge
Point 1 Point 2 Point 3
• Knowledge Point 1: Resources and Requirements Match
– Indicated by a high level of technology maturity
• Heritage is technology is appropriate in form, fit and function
• New Technologies at least at TRL-6 (for NASA)
• Knowledge Point 2: Product Design is stable “about midway through development”
– Best practice is to achieve design stability by midpoint of development
– GAO cites 90% of engineering drawings releasable as figure of merit
• Knowledge Point 3: Manufacturing Processes are mature
– Product shown to be producible within cost, schedule, and quality targets and have demonstrated
reliability and Design demonstrates needed performance through realistic system-level testing
1/22/2012 10
11. Data Requested Annually
Reporting Year
Data Category Element 2008 2009 2010 2011 Frequency
Design Stability X X X X Annual + Updates
Critcal Technologies X X X X Annual + Updates
Technical Heritage Technologies X X X Annual + Updates
Software Complexity X X Annual
Quality Parts Issues X X Annual
MPAR Baseline X X X X As Occurs
ARRA Funding X X Semi-annual
KDP-B Est. LCC Range X X X X Semi-annual
KDP-C Baseline X X As Occurs
Cost
ICEs X X As Occurs
JCLs X X As Occurs
Project-held UFE X X Monthly (in MSRs)
MD-held UFE by year X Annual
Key Milestones X X X X Semi-annual
Schedule
Agy vs Mgmt Schedule Comparison X Semi-annual
Basic Information X X X X Semi-annual
Contracts Award Fee Strucuture X X X Semi-annual
EVM (CPR Formats) X X Monthly
FAD/PCA X X X X As Occurs
Project Plan X X X X As Occurs
Control Plans X X X X As Occurs
PDR/CDR Packages X X X X As Occurs
Documentation CADRes X X As Occurs
SRB Reports X X As Occurs
KDP-C Decision Memos X As Occurs
KDP-C Datasheets X As Occurs
MSR Presentations X X Monthly
1/22/2012 GAO reviewed this list for 2011, and agreed to reduce frequency of some requests 11
12. So, my project is in Quicklook…
WHAT SHOULD I EXPECT?
1/22/2012 12
13. The Quicklook Annual Cycle (2011)
DCI- Action Item Response Addt’l Actions
contract
Site Visits Agency
2-Pgr Cmts
DCI- Site Visits – Cmts
– MSFC
C&s JPL &GSFC &JSC
DCI- DCI- DCI-
Other Data Requests Addt’l Data
proj C&s C&s
Entrance Exit Final Report
Conference Conference
DCI = “Data Collection Instrument” Three types:
1.Project DCI – for technical data
2.Cost and Schedule (C&S) DCI – provided by OCFO
3.Contracts DCI – provided by OP
1/22/2012 13
14. Process for Working with GAO
Extra explanation may be provided
Quicklook Technical POC (TPOC)
to help contextualize the data or to
discusses request with GAO to
better assist the GAO in answering
understand purpose for
their questions
requested items
GAO transmits Project drafts Response reviewed QLB TPOC
request response by HQ & Ctr Mgmt transmits response
to GAO
• The TPOC and • Provide • Possible • ALR, PM, PE,
relevant Audit document reviewers and reviewers
Liaisons (ALR) and/or include PE, MD, will be copied
should be explanation PCFO,
copied on all Procurement,
• When
requests OCE, OSMA, LSP
document or
• PM please data is not • QLB TPOC and
forward available, MD ALR will
requests to ALR explain why coordinate the
and QLB TPOC appropriate
• Draft response
when necessary provided to PE, reviews.
ALR, and QLB
TPOC
Standard 3 week turn-around when not otherwise stipulated by GAO
1/22/2012 14
15. GAO Data Access Rights
• Legislatively, the GAO has right of access to the data, regardless
of procurement sensitivity or classification
• NASA can negotiate with GAO regarding the provision of data; if
there is a disagreement, GAO’s recourse is legislative action
• GAO is bound by the same rules as NASA for handling the data,
and NASA can specify data restrictions
• NASA actively works with GAO on data formats, types and
context for interpretation
NASA provides the data requested and works with the GAO to add
context and help the GAO analysts to understand NASA’s data.
1/22/2012 15
16. Data Collection Instruments
Example Technical DCI
GAO Data Collection Instrument: Design Stability
• GAO asks PMs to fill in
Date Submitted to NASA Project Office:
NASA Project Office Response Required by:
5/3/2011
5/6/2011
a Data Collection
GAO Contact:
Project:
Joe Analyst
Awesome
Instrument (DCI)
NASA Project Office Instructions Please provide or update the total number of releasable engineering drawings and the total number expected at key dates listed below. The
term "releasable" refers to those drawings (physical hard copy or computer-generated packages) that are sent to be manufactured. Please
annually and following
calculate the number of drawings by using the number sheets within a package, not just the number of packages. Sometimes these
drawings are considered the "build to" drawings. These drawings include details on the parts and work instructions needed to make the PDR and CDR
product and reflect the results of testing. Change any dates or data as needed.
Key Event Dates Defined: The PDR column should reflect information as of the Preliminary Design Review prior to KDP C (NAR). The CDR column should reflect information as of the
• Data collection
Critical Design Review. The "Latest" column should reflect the the number of drawings releasable and expected as of the date this spreadsheet is returned to GAO.
includes
Overall Design Knowledge
KP ESTIMATE PDR CDR September 2010 update Latest – Critical and heritage
Date
No. of Releasable
7/12/2010 Scheduled for July 2011 9/16/2010 5/3/2011
technology TRLs
1537 sheets expected for
947 spacecraft sheets
released to date. 299 – Drawing release
– Parts Quality issues,
spacecraft, 818 drawings 10 released for spacecraft, drawings released for
Drawings 0 expected for instruments. 16 released for instruments. instruments.
No. of Total
2437 sheets of spacecraft including impact ($)
– Software TLOC and
2437 sheets expected for drawings planned, 1023
1879 sheets expected for spacecraft, 1023 drawings drawings expected for
Drawings Expected spacecraft. expected for instruments. instruments.
Source Project Office Project Office Project Office Project Office Blocks
If the total number of drawings is not known or if the total number of drawings has changed, please provide an explanation in the space below. If the project has – Partners
•
completed CDR, please describe any measures and metrics used in addition to engineering drawings to assess design stability. If the project has yet to complete its
CDR, please identify the measures and metrics the project intends to use to assess design stability.
See below for Spacecraft and Instruments comments. The project assesses design stability primarily thru the Systems Engineering organization. A weekly Systems Design Team
GAO requests data as
regularly addresses design, design margins, analyses, interfaces, subsystem and instrument design topics, and proposed changes. Proposed design changes are then processed thru
standard Configuration Control Boards. In addition, weekly status reviews as well as a Monthly Management Review take place that addresses overall status and issues. of PDR, CDR, and
current
Comments & Issues: Enter comments and issues below or on a separate sheet.
• Procurement and
PMs should NOT leave any sections blank; if the data is OCFO fill in similar
DCIs on contracts,
not available or not applicable, say so and explain why. cost, and schedule
1/22/2012 16
17. Site Visits
• GAO will prepare for site visits with the projects by reviewing project monthly
status review (MSR or PSR) packages; discussion questions will be transmitted
in advance of the visit.
• GAO may not visit every project included in the Quicklook; projects not visited
will receive written questions based on the content of their monthly review
packages.
• PMs are invited to bring any charts or personnel that they need to help answer
the questions; however, GAO prefers to have an informal discussion – NOT a
lecture.
• When preparing responses, keep in mind that Quicklook is about cost growth
against the Agency’s MPAR Baseline; responses should keep in mind the full
mission cost – including launch vehicle and UFE
• Refer questions about things you are not responsible for to correct organization
– LV questions should generally be directed to LSP
– MD-held UFE questions should be directed to the MD, although it is ok to state the
value
• When in doubt, it is ok to take an action to provide a response after the visit.
This is the project manager’s opportunity to add context and explain
all the good work their team is doing.
1/22/2012 17
19. Draft 2-Page Project Summaries
What to look for How to Respond
• Overview and Project Essentials 1. Project comments
– Is the information correct? – Will be printed in the “Project Office
• Project Update and Summary Comments” section of the final report
– Limited to 500 characters (appr 100 words)
– Does the GAO accurately portray the
status? 2. General Comments
– Are there any errors in the analysis? – Focus on errors in understanding or
interpretation of information previously
• Project Challenges provided to GAO as well as points of
– GAO’s challenge categories are fixed and disagreement with the GAO’s analysis.
will not change
– These comments will not appear in the
– Assertion of a challenge should be report but may help drive the discussion with
supported by the analysis described in the the GAO about how they can revise
Project Update
3. Technical Comments
• Contract Info, Project Performance, – Comments provided in this section will be
and Project Timeline used to correct any factual errors in the
– no action from PM required; Procurement report.
and OCFO will review to ensure consistency – Corrections must be accompanied by
with other external reporting
evidence
GAO will transmit draft summaries directly to project managers with response requested in
5 business days; Response must go through full agency review process in this time.
1/22/2012 19
20. So we gave GAO all this data…
WHAT DID THE DATA SAY?
1/22/2012 20
21. GAO Quicklook – Mar 2011
• Maintained recommendation for GAO’s “Business Case Approach,” but recognized
improvements due to JCL policy
• Found average cost growth of 14.6% and schedule growth of 8 mo; nearly all growth
attributed to projects baselined before 2009
• Main Observations:
– Technology Challenges:
• NASA begins development with insufficient technology maturity
• Evidence: Majority of projects passed PDR with critical technologies at TRL < 6
– Design Challenges:
• Projects pass CDR with unstable design
• Evidence: Nearly all projects passed CDR with <90% releasable drawings
• Additional Observations
– Funding Challenges: JWST, Ares, Orion funding “did not align with plan”
– Launch Vehicle Challenges: mainly medium class – addressed more fully in another GAO
report
– Development Partner Challenges
– Parts Challenges – addressed more fully in GAO Quality Parts Audit
– Contractor Management Challenges
• Report stated that GAO will conduct a more thorough analysis of NASA’s EVM data in
future audits and that [cost] transparency is insufficient during formulation
1/22/2012 21
22. Mar 2011 QLB - Cost Growth
The majority of
NASA’s cost and
schedule growth is
attributed to
projects baselined
prior to 2009
NASA agrees with GAO’s cost and schedule growth figures
1/22/2012 22
23. Challenges Summary
Quicklooks published in 2009, 2010, 2011
12
2009
2010
10
# Projects with GAO-identified Issue
2011
8
6
4
2
0
Funding Issues Launch Issues Parts Issues Technology Design Issues Contractor Development
Issues Issues Partner Issues
Subjects of other GAO audits
1/22/2012 23
24. Data Provided During 2011
Technology Maturity and Drawing Release
150.0% 150.0%
Percentage Development Cost
130.0% JWST 130.0%
JWST
Percentage cost growth
110.0% 110.0%
Class A GAO hypothesis
90.0% 90.0%
Growth
70.0% MSL Class B 70.0%
50.0% Class C 50.0% NPP
30.0% NPP 30.0% SOFIA
Class D
10.0% 10.0%
Juno
-10.0% -10.0%
0.00% 20.00% 40.00% 60.00% 80.00% 100.00%120.00%
0% 50% 100% GPM
-30.0% -30.0%
% Immature Technologies at PDR % releaseable drawings at CDR
MSL did not provide drawing data; MAVEN is excluded from CDR analysis because CDR was not complete at time of DCI submission
• Self-reported data from DCIs submitted to GAO appear to support the hypothesis that
immature technologies at PDR leads to cost growth for NASA missions
– The distribution does not support a strong correlation
– JWST and MSL drive the correlation, but have cost growth clearly attributable to other factors.
• The data shows low correlation between releasable drawings at CDR and cost growth
1/22/2012 24
25. Data Provided During 2011
Drawing Release
% of Drawings Releasable at CDR
100.00%
90.00%
80.00%
70.00%
60.00%
50.00%
40.00%
30.00%
20.00%
10.00%
0.00%
Aquarius GPM GRAIL Juno JWST LADEE LDCM MMS MSL NPP OCO-2 RBSP SOFIA TDRS
GAO Calculation Assumed NASA calculation
CDR Now • There appears to be some disagreement between how NASA
# Releasable A C and GAO calculate the percent of releasable drawings
• NASA calculates the percentage of releasable drawings at CDR
Total # anticipated B D
based upon the total number of drawings anticipated as of CDR
NASA: % releasable = A/B • Drawing growth after CDR typically occurs as a result of risk
GAO: % releasable = A/D realization; it is impossible to predict which risks will come to
pass and which of those will require drawing changes
NASA’s 90% guidance is based • By NASA’s calculation, several projects did meet the best
on what is known at the time practice guideline of 90% releasable drawings at CDR
1/22/2012 25
26. Data Provided During 2011
Software
• 13 projects provided complete software data
– 4 projects - Aquarius, MSL, NPP, and JWST - reported growth in TLOC greater than 25%
– 5 projects reported a decrease in TLOC since PDR – Juno, LDCM, RBSP, SOFIA, TDRS
• After removing projects whose cost growth is clearly attributed to other
factors, the mathematical correlation between development cost growth in
growth in TLOC is high (.91) but the lack of significant distribution does not
support a relationship between cost growth and software growth
All Quicklook Projects MSL and Aquarius Excluded
200.0% 200.0%
% Development Cost Growth
% Development Cost Growth
150.0% 150.0%
JWST JWST
100.0% 100.0%
MSL
50.0% SOFIA
50.0%
NPP LDCM NPP
SOFIA Aquarius
0.0% GPM 0.0%
-200% 0% 200% 400% 600% 800% -50% 0% 50% 100% 150%
GPM
-50.0% -50.0%
% Growth in TLOC, PDR to Current % Growth in TLOC, PDR to Current
1/22/2012 26
28. Formulation
• Formulation is an iterative
Resources Requirements process that aims to balance
requirements and resources.
• At KDP-C the Decision
Authority agrees that the
design is sufficiently mature
and the requirements match
the resources
• The confirmation decision
results in an agreed-to cost
and schedule baseline; budget
is committed to the project as
part of the PPBE process
• The balance of requirements
and resources is continuously
assessed throughout a
project’s lifecycle
Changes in schedule or projected LCC during formulation should
not be treated as slips or cost growth; rather, this is a normal part of
the formulation process.
1/22/2012 28
29. PDR Road to Confirmation
(For SMD Projects to date)
Project Report Project Report Project Report
JCL JCL JCL
SRB C&S (JCL) SRB C&S (JCL) SRB C&S (JCL)
Assessment Assessment Assessment
SRB Technical SRB Technical SRB Technical
Assessment Assessment Assessment
Project Response Project Response Approval Project Response
Approval
to SRB to SRB to Proceed to SRB to Proceed
ICEs or ICAs1 ICEs or ICAs1
(Cat 2&3)2 ICEs or ICAs1
(Cat 1)
Center Center
Recommendation Recommendation
Program Office SMD
Recommendation Recommendation
CMC Division (Theme) DPMC OCFO/SID Analyst APMC
Recommendation Assessment
The Confirmation decision takes into account many inputs.
There is no “right answer”
1. when available
1/22/2012 29
2. Cat 2 and 3 missions do not require an APMC, but Cat 1 missions do. With 7120.5E, the AA will approve entry to Implementation for all projects
30. JCLs and UFE
• The use of JCL in decision making represents a shift in the paradigm
of how project cost estimates, budgets and schedules are
established.
• This method of estimation moves beyond prior practices of
calculating a “point estimate” and adding an additional level of
reserves.
• Because the UFE is a product of the probabilistic basis of
estimates, any reduction to the UFE will reduce the probability of
achieving the project cost and schedule targets
• Usage of UFE should not be interpreted as a “funding issue;” the
UFE is part of the full amount that NASA budgets for a mission.
• The term “UFE” should be used rather than “reserves” to denote
the more rigorous cost estimating policy NASA now uses
With the release of 7120.5E, NASA no longer uses the term
“reserves.” PMs should update charts to use the term
“Project-held UFE.”
1/22/2012 30
31. Management vs Agency Commitments
Cost Schedule
Agency Baseline Agency Baseline
Commitment Agency Commitment
MD-Held
Schedule • Based upon JCL and
• Based upon JCL and UFE Margin other inputs
other inputs
Project- Project • At 70% JCL or as
• At 70% JCL or as
Schedule determined by the DA
determined by the DA held UFE
Margin • Difference between ABC
• Difference between ABC and MA is additional
and MA is budgeted as schedule margin held by
MD-held UFE the Agency
Management
Management
Agreement
Agreement
• Managed by PM
Managed by PM
• Based upon Project’s
Based upon Schedule
own Cost Estimate
developed by PM
• Should be at 50% JCL or
Should be at 50% JCL or
greater greater
Not to scale
In some cases, the Decision Authority deems it prudent for the Agency to hold
additional schedule margin above the project’s schedule. In these cases, the
Management Schedule should not be viewed as acceleration of the Agency’s
schedule
1/22/2012 31
32. What happens when funds are not
appropriated or performance is poor?
o When requested funds Notional Example
are not
appropriated, or 700
performance is FY10 Budget request
poor, work is pushed 650 FY10 Appropriation + FY11 request
to future years, and FY11 Appropriation + FY12 request
budget requirements
FY12 Appropriation + FY13 request
increase in the out 600
years.
550
o This may lead to cost
growth and operating
plan changes. 500
o If reductions happen in 450
subsequent years, the
profile becomes
unsustainable. 400
FY10 FY11 FY12 FY13 FY14
1/22/2012 32
33. EVM
Total Project (Combined In- and Out- of House)
Out-of-House NASA projects tend to be complex;
Contracts
IIn-House
CP & FPI CP & FPI CP & FPI Partners EVM is not necessarily applied to
FFP
>$50M >$20M <$20M all suppliers but it is addressed at
7 Certified 32
Principles System Guidelines
NA NA NA the total project level.
7 Principles applied across all elements of Total Project
• NASA’s projects are accomplished through combined efforts of suppliers and
NASA in-house efforts
– Total In-House with targeted procurements
– Prime contract with government providing Level-Of-Effort (LOE) oversight
– Many unique mixtures of the above two scenarios
• NASA applies EVM to contractors at the project level and requirements are
flowed down to subcontractors
EVM is one of many project management tools that NASA uses. NASA
recognizes EVM implementation as an area of concern and is committed to
improvement in this area
1/22/2012 33
34. Technical Leading Indicators
• NASA is enhancing the ability to monitor and assess the
stability of programs and projects as well as the maturity of
their products through the development process
• Comprehensive 3-Step Approach
1. The Entrance and Success Criteria for Lifecycle reviews are being
enhanced to further account for product maturity
• These will be documented in the Rev B update to NPR 7123.1
2. A required set of 3 parameters that will address product stability
and maturity will be added as a NID to NPR 7123.1A
• Mass Margins
• Power Margins
• Review item closures
3. A set of recommended Common Leading Indicators is being
developed
• Added to NPR 7120.5E Formulation Agreement and Program/Project
Plan templates
1/22/2012 34
Note: Funding Issues, Launch Issues, and Parts Issues were not addressed in the 2008 Quicklook Report. The 2008 Quicklook Report included issues for several projects that are now in Operations or are completed, and are not listed here (Dawn, Fermi, Herschel, Kepler, LRO, OCO, SDO, WISE).
NASA requires EVM reports to be in compliance with ANSI/EIA-748. CPR is a DOD format.analytical products and risk watch list are provided in MSRs