This document summarizes the findings of a NASA survey of various centers regarding compliance with Office of the Chief Engineer (OCE) policy. It describes the survey objectives, methodology, elements reviewed, and schedule. Some key findings included inconsistent implementation of configuration management, risk management, and technical authority across centers. Strengths identified included lessons learned processes and software engineering at JPL. Opportunities for improvement included updating directives, validating Earned Value Management Systems, and clarifying the roles of technical authority and systems engineering.

1. OCE
Office of the Chief Engineer
Cross-Cutting Look at OCE
Policy Compliance within
OCE
NASA
Office of the Chief Engineer
OCE
Office of the Chief Engineer
(February 2010)
OCE
Office of the Chief Engineer
Presented By: Beth Keer
NASA HEADQUARTERS
OFFICE of the CHIEF ENGINEER
OCE
Office of the Chief Engineer Used with Permission

2. OCE
NASA Engineering &
Survey Team Members
Program/Project
Management
Philosophy:
• Involve personnel that are knowledgeable, experienced and able to
take lessons back to home Center to institute change.
• Take advantage of symmetry with other activities as much as
possible.
• Some consistency with some positions held for POC and unique
skills specific to the Center.
Team Members:
• Beth Keer – OCE Survey Team Manager
• Jim Lawrence – OCE (PSGS/Dell)
• John Kelly – OCE Software Manager
• Dr. David Liskowsky – OCHMO (Part time)
• Participant from Center to be Surveyed
• Participant from Center with interest in chosen Projects
• * Patti Stockman – NASA HQ CIO (Part time)
• * EVMWG
Champion: Mike Ryschkewitsch Chief Engineer
Sponsor: Sandra Smalley 2

3. OCE
NASA Engineering &
Survey Objectives
Program/Project
Management
• Review Center processes and infrastructure for
compliance with OCE requirements, policy,
procedures, processes, statutes, and regulations
• Review at least two Projects/Programs’ documents
and processes for compliance with OCE
requirements, policy, procedures, processes,
statutes, and regulations
• Identify systemic problems or deficiencies
• Recognize areas of excellence/best practices
• Receive Center feedback regarding areas where
Agency policy and requirements should be modified
• Results are used in Agency response to OMB/GAO/
IG issue on oversight of implementation of
requirements
3

4. OCE
NASA Engineering &
Survey Basis
Program/Project
Management
The basis for the survey:
• NPR 7120.5D, NASA Space Flight Program and
Project Management Requirements;
• NPD 2820.1, NASA Software Policy;
• NPR 7150.2, NASA Software Engineering
Requirements;
• NPR 7120.6, Lessons Learned;
• NPD 8070.6, Technical Standards;
• NPR 7120.8, NASA Research and Technology
Program and Project Management Requirements
(Planning Perspective);
• NPR 7123.1, NASA Systems Engineering Processes
and Requirements (beginning with DFRC in 2010).
4

5. OCE
NASA Engineering &
Findings Definition Guidelines
Program/Project
Management
Strength: A finding of OCE Policy implementation
that results in reduced risk to the Program or
Project.
Weakness: A finding of OCE Policy implementation
that results in risk to the Program or Project.
Observation: A finding that is a potential weakness
or potential risk to the Program or Project.
Opportunity: A finding of OCE Policy Implementation
that is potentially a strength.
Non-Conformance: A finding of OCE Policy not
being implemented and with no waiver in place. 5

6. OCE
NASA Engineering &
Survey Core Elements
Program/Project
Management
• Common framework for unified program and
project life cycle
• Program and project review structure
• Technical Authority implementation
• Dissenting Opinions/ Waiver Processes
• Lessons Learned
• Technical Standards
• Software Engineering Management
• Research and Technology (NPR 7120.8)
6

7. OCE
NASA Engineering &
Schedule
Program/Project
Management GSFC – April 2008
KSC – July 2008
MSFC – December 2008**
JSC – February 2009**
LaRC – March 2009**
JPL – August 2009**
ARC – October 2009**
(Current Status) GRC – November 2009
DFRC – *February 2010
(Round 1 complete) SSC - March 2010
HQ MD or PSO – May 2010
GSFC – July 2010
KSC – September 2010
* Planning underway ** Dates consistent with IPS
7

8. OCE
NASA Engineering &
Common Framework
Program/Project
Management
Management Reporting:
• Various degrees of strength to the CMC as far as
Management Reporting of Projects.
– Meet commitments of Center vs Roads and camodes
• Integrated CMCs
– Strong implementation for instances with robotic missions
– Originally had good reports on CxP, (JSC, MSFC, LaRC)
– ARC response not as strong (not clear if external
environment change for CxP or other factors).
Excellent: GSFC, MSFC, LaRC, JPL
8

9. OCE
NASA Engineering &
Common Framework
Program/Project
Management
Configuration Management:
• CM/DM/RM practices are inconsistent across the
Centers.
– Implementation is strong on projects.
• Most Centers have one or two tools that they “bless” for use but do not
require those tools to be used.
– Program driven tool (Windchill) viewed as not user-friendly.
• Some use as repository, not an improvement over working systems.
• Potential for confusion/risk by using separate systems.
– Records Management awareness is improving.
• Records retention schedules are not consistently being completed.
• Records Management planning is not clearly considered when setting
up CM systems.
• Records Management responsibility is consistently being assigned as
the CM Officer or CM lead on the projects. 9

10. OCE
NASA Engineering &
Common Framework
Program/Project
Management
Information Management:
• Directives Management Systems not up to date to
reflect Policy (update processes not perfect).
– HQ requirements are too many and at different levels of
detailed description (what vs how).
Excellent: JPL has contract with definite applicability of policy
AND flowdown of requirements into JPL Rules! and JPL FPP.
Earned Value Management:
• Centers do not have validated EVMS (JPL is
exception).
• Implementation within the Projects not being
institutionalized within the Centers.
• Value of the data is inconsistent to the Managers. 10

11. OCE
NASA Engineering &
Common Framework
Program/Project
Management
Risk Management (RM):
• Inconsistent use of RM as part of the decision making
process across the Centers.
– Some use to allocate reserve for risk mitigation.
– Strongest CMCs include Risk reporting and have strong
discussions at monthly CMCs. (GSFC, MSFC, JPL, LaRC)
• Use of IRMA on CxP is a strength.
– Possible false confidence that risks are being reported.
– Focus is on identifying and tracking, rather than managing
the risks by working levels.
– Perspective is different from working level to Project level
(i.e. rating).
– Separate risk systems maintained to use the risks to
manage. 11

12. OCE
NASA Engineering &
Program and Project Review
Program/Project
Management
Implementation Issues:
• Direction vs Recommendation
• Loss of understanding that SRB is
meeting objectives of TA (OCE/Centers),
MD, and AA (PA&E).
Acceptance of SRB requirements.
12

13. OCE
NASA Engineering &
Technical Authority
Program/Project
Management
Two Implementations on the Projects:
1) Mission/Lead System Engineer is the TA-Eng and
independently funded. (GSFC & MSFC)
• Performs TA as a collaborative effort with the Systems
Engineers on the Project
2) Chief Engineer is the TA-Eng and independently
funded.
• Focus of TA is strongly on the independent funding and
requirements ownership (technical voice not prevalent)
• Performs special studies
• Independent of Systems Engineering functions of the
Project/Program
• Defines specific roles between TA and SE (potential to
have a gap) 13

14. OCE
NASA Engineering &
Technical Authority
Program/Project
Management
• TA-Software responsibilities are not widely
understood.
• TA Implementation Plans require updating
– Most Centers addressed only TA-Eng
– Surveys prompting updates to include SMA and
HMTA
• HMTA Infrastructure is limited to JSC.
– Infrastructure that can be leveraged does exist
– Training/awareness needs to be increased
• Non-Conformance: JPL is submitting a waiver to
HQ for independently funded TA-SMA.
14

15. OCE
NASA Engineering &
Waivers/Dissenting Opinions
Program/Project
Management
• Waiver Process at the Centers is not clearly
understood by Projects and Programs
– Weaknesses exist in content for waivers within Center
documentation (inclusion of rationale)
• Dissenting Opinions Process
– Some Centers have a documented process within specific
areas (i.e. Engineering)
– No Center has a documented process that covers all
possible dissenting opinions (SMA, Eng, procurement,
programmatic, etc.)
15

16. OCE
NASA Engineering &
Technical Standards/Lessons Learned
Program/Project
Management
• Tech standards:
– Overall implementation is good
– Documentation of the working process
• Lessons Learned:
– Overall the requirements of the NPR are
being met.
– Personnel get the lessons they need for
their next Program/Project through informal
means.
16

17. OCE
NASA Engineering &
Software Engineering Management
Program/Project
Management
• Overall implementation strengths exist
at the Centers.
• Policy is being taken seriously.
• NASA workforce is working to assure
requirements are flowed down to
Projects.
17

18. OCE
NASA Engineering &
Software Engineering Management
Program/Project
Management Non-Compliances:
• KSC: Risk Analysis and Waiver approved for Ares 1-X GS
• JSC: SW Engineering Improvement Plan is out of date-working.
Class A s/w being developed by organizations not having CMMI
level 2 rating or higher - (in-house CMMI corrected, contractor
levels are being tracked and reported).
• LaRC: Required 7150.2 Compliance matrix not completed
(corrected)
Firmware development is proceeding without applicable
s/w requirements.
• HQ/JPL: NPR 7150.2 requirements are not flowed to the Contract
Meet the intent and have the ability to apply at the task level.
• ARC: SW Engineering Improvement Plan does not exist.
• GRC: NPR 7150.2 requirements are not flowed to some contracts and
procurement efforts.
Class A software development with no current CMMI rating in the required
18
process areas performed by the Center.

19. OCE
NASA Engineering &
7120.5E Suggestions
Program/Project
Management
• Address all requirements, but allow “Tailoring” of the
approach to implementation of the requirements
consistent with Program/Project characteristics such
as scope, complexity, visibility, cost, safety, and
acceptable risk of a project.
• Address how requirements are determined to be n/a
for work at a Center (elements within the Project).
• Training Requirement on Project Management teams
to have annual training.
• Collecting Lessons learned as go through the life cycle
does not take priority.
• Inclusion of context within policy is viewed as positive.
19

20. OCE
NASA Engineering &
7120.5E Suggestions
Program/Project
Management
• TA-Eng for Software (3.4.1.1 including TA for
Software as described in NPR 7150.2)
• TA implementation focus is on the independent
funding of the technical authority and waiving the
technical requirements at the Project level (good).
– Add language that an additional role of the independently
funded TA is to be the voice of working level engineers that
have not been designated as TA.
• Include table to describe and point to different
classifications (7120.5, 8705.4, 7150.2, etc) and
context for the need of the classifications.
20

21. OCE
NASA Engineering &
7120.5E Suggestions
Program/Project
Management
SRB Implementation Issues:
• Implementation of SRBs is not clear to Projects that all
convening authorities objectives are being met .
• SRBs act as a decisional body rather than as a recommendation
body. (Identified as an implementation issue, not a policy issue)
• Inconsistent implementation of SRB process (personality
dependent).
• Timeframe to complete the report-out process is too long.
– Policy change to allow parallel activities
– Describe process to authorize continuation into the next phase
while the road to the KDP continues.
21

22. OCE
NASA Engineering &
Additional OCE Actions
Program/Project
Management
• Policy-related letters/correspondence to be
attached to the affected NPR/NPD(s) within
NODIS
• Use of SMART Id’s on OCE Policy to assist
Centers with requirements flowdown
• Blanket waiver for Class D missions
22

23. OCE
NASA Engineering &
Topic for Discussion
Program/Project
Management
ISSUE: Class D Payload classification and implementation of
Class B Software requirements in a resource constrained
environment.
1. Comply
2. Ctr level waivers project by project for delegated requirements
(80+% of NPR 7150.2) and implement remaining non-
delegated requirements
3. Ctr level waivers project by project for delegated requirements
(80+% of NPR 7150.2) and submit waiver to HQ for relief on
remaining requirements.
4. Submit blanket waiver to HQ to implement alternate Class B
Software requirements that do not meet or exceed the
requirements of NPR 7150.2.
• Describe the Center process to identify, mitigate,
communicate and accept project risk associated with the
implementation approach proposed by the Center. 23

24. OCE Generic/Blanket Tech Authority Waiver capability
NASA Engineering &
in NPR 7150.2A*, Chapter 6
Program/Project
Management
“6.1.1 For those cases in which a Center or project desires a general
exclusion from requirement(s) in this NPR or desires to generically apply
specific alternate requirements that do not meet or exceed the
requirements of this NPR, the requester shall submit a waiver for those
exclusions or alternate requirements for approval by the NASA
Headquarters Chief Engineer with appropriate justification. [SWE-120]”
“Note: This type of waiver (which is approved by the NASA
Headquarters Chief Engineer) is for generic/blanket relief from a
requirement for a Center, Center organization, or multiple projects over
an extended time. Generic/blanket waivers are not to be confused with
normal waivers that address relief from a requirement on a single project
or in a specific instance (which can be approved at the Center level if so
specified in last column of Appendix D).”
24
* This document has passed NODIS review an is in the Administrator’s suite for signature

25. OCE
NASA Engineering &
Research and Technology
Program/Project
Management
• Definition of roles and responsibilities for
implementation of Technical Authority in R&T Programs
and Projects is needed by HQ. (ARMD TA, Program
TA, Project TA, PI, and Center TAs)
• Roles and responsibilities of the Center TAs need to be
clearly defined in Center documentation.
• R&T Project Management Structure is complex from a
communication perspective.
• Checks and Balances developing within the
organization and Project specific to the R&T
management structure.
25

26. OCE
NASA Engineering &
Topic for Discussion
Program/Project
Management
• Transition from R&T project (NPR 7120.8)
into a flight project (NPR 7120.5).
– Clarity required
– Implementation of NPR 7123.1 requirements
– Implementation of SMA requirements
• ARC and GRC have working processes for
projects that transition from R&T to flight.
• OCE/MDs evaluation of NPR 7120.8 and
NPR 7123.1 for potential clarifications
26

27. OCE
NASA Engineering &
Program/Project
Management
Back Up Charts
27

28. OCE
NASA Engineering &
Survey/Audit Differences
Program/Project
Management
• Higher level of documentation review
• Focus on implementation of OCE
requirements
• Fewer interviews conducted
• Response to findings is different
• Actively looking for Center feedback on areas
where Agency policy and requirements
should be modified
28

29. OCE
NASA Engineering &
Survey Ground Rules
Program/Project
Management
1. The basis for the OCE Survey is NPR 7120.5D - NASA Space
Flight Program and Project Management Requirements, NPD
2820.1 - NASA Software Policy, NPD 8070.6 – Technical
Standards, NPR 7120.6 - Lessons Learned Process and NPR
7150.2, NASA Software Engineering Requirements. NPR
7120.8 – NASA Research and Technology Program and
Project Management Requirements.
2. The Survey will be conducted using the OCE Requirements
Survey Questions checklist developed for the Center.
3. The Center interface point of contact assists the Survey
Team in obtaining documentation and establishing interview
schedules for the areas being surveyed.
29

30. OCE
NASA Engineering &
Survey Ground Rules (cont.)
Program/Project
Management
4. The OCE Survey is being conducted in coordination with the CxP
SA Level 2 Audit. Some Survey areas (such as TA-SMA, Risk
Management, Design documentation, etc.) also have attributes
which are within the scope of the CxP Audit Team. In these areas,
the Audit teams will also review the OCE Survey attributes. The
OCIO Records Management Assessment is also coordinated with
the Survey activities. The integrated effort is taken to ensure
complete coverage of all OCE requirements without duplication of
reviews by multiple teams.
5. Interviews will be conducted. Designated Program/Project
managers, technical authorities, records managers, configuration
managers, institutional directives manager, procurement
acquisition managers and software managers. Additional
interviews may be conducted based on issues found during the
Survey and will be arranged by the Center interface point of
contact.
30

31. OCE
NASA Engineering &
Survey Ground Rules (cont.)
Program/Project
Management
6. The Survey Team (in conjunction with the Audit/Assessment
Teams) will conduct a brief meeting on Tuesday and
Wednesday afternoon. The team members will discuss
their findings from that day, their planned activities for the
following day and any support needed from the Center.
Survey efforts may be refocused or additional areas may be
added at the daily meeting based on findings generated by
the team members. The Center interface point of contact is
requested to attend the daily team meeting to maintain
close coordination with the Survey Team.
7. The Survey Team will conduct a meeting as early in the day
on Thursday to discuss the findings with the Center
interface point of contact and any other Center personnel
invited by the POC.
31

32. OCE
NASA Engineering &
Survey Ground Rules (cont.)
Program/Project
Management
8. OCE Survey Manager will concur with all survey
findings to ensure consistent interpretation of
requirements.
9. An Out-brief will be held on Friday for OCE, SMA,
OCHMO and Center personnel marking the close
of the on-site Survey.
10. The Center POC is provided access to the
detailed dBase (SAARIS) of findings from the
Survey.
11. OCE will work with the Center POC to ensure
understanding of the findings and the responses. 32