The document discusses a Technical Excellence Program (STEP) run by the National Aeronautics and Space Administration's (NASA) Safety and Mission Assurance (SMA) organization. STEP aims to improve project success through technical excellence by developing curriculum for SMA personnel. The agenda outlines a presentation on STEP including its development, rollout, and future plans to expand it to other training needs. It also includes an overview of the System Safety discipline and how it interacts with programs/projects to reduce risk.

1. National Aeronautics and Space Administration
Improving Project Success through
Technical Excellence
February 9 – 10, 2010
John D. Marinaro
Mark M. Kowaleski
http://nsc.nasa.gov/
1
Used with Permission

2. Agenda
• SMA Technical Excellence Program (25 min.)
– NSC Background
– Purpose of STEP
– How the STEP Curriculum was Developed
– STEP Curriculum Elements
– Extension to other Agency training needs and risk reduction
– STEP Level 1 Roll-out and Metrics
– STEP Levels 2 – 4 Planning
• System Safety Discipline Overview (20 min.)
– System Safety & Program/Project Interactions
– System Safety Competencies
– System Safety Lifecycle Products and Services
– Community of Practice
– System Safety State-of-the-Discipline
– System Safety and Program/Project Risk Reduction
• Q&A (10 min.)
• Back-up Materials
– SMA Discipline Curriculums
2

3. SMA Technical
Excellence Program
3

4. NASA SAFETY CENTER
Overview
NSC develops SMA personnel, processes, and tools needed for
the safe achievement of NASA’s strategic goals through:
Establishing a learning Benchmarking from the best
environment and bringing their best
practices into our workplace
Learning from our mishaps Ensuring that we are informed
risk-takers
Managing the routine risks of Preserving our resources for
the workplace effectively the NASA mission
http://nsc.nasa.gov/ 4 2/18/2010

5. NASA Safety Center
FOUR AREAS OF SUPPORT
http://nsc.nasa.gov/ 5 2/18/2010

6. THE PEOPLE OF THE NSC
NASA Safety Center
Alan H. Phillips Rafael Sanabria Dennie Gonia Sandra L. Hardy Charlene L. Anderson
Director Deputy Director Secretary Business Manager Administrative Officer
John Marinaro
Karen Meinert Cheevon (Mi-Mi) Lau Kenneth O’Connor
OCE Technical Chief, GRC NESC
Technical Knowledge Management Audits & Assessments Director Mishap Investigation
Liaison Excellence Director OSMA OCIO Liaison
Discipline Systems Manager Support Manager
Fellows
Kerry Remp Diane Mark Kowaleski Homayoon Carolyn Van Sue Otero Sallie Keith Dr. Jeffrey Dawson Michael Milbert Mark George
COTR Chapman Systems Safety Dezfuli Drei IRIS COTR Information Data Analysis & Analyst MIS Administrative Support
Curriculum Technical Systems Safety ST Enterprise Dissemination Trending Analyst
IRIS
Development Support/ Architecture
EX3
Discipline Manager
EDUCATE Dr. Natesan
Alphaport Fayssal Safie Art Lee Michael Dodson
Jambulingam Reliability & Institutional/ Facility Mishap Investigator
Reliability & Maintainability ST Operational Audit Manager Eastern Region (GRC, GSFC,
Maintainability Brian Follen HQ HQ, KSC, LaRC )
PBMACOTR Robert Cross
NASA Safety Analyst
PBMA
Training Center ARES
Ray Kacmar
Brian Hughitt
Quality Paul Boldon
Quality Kristie French
Engineering Institutional/
Engineering ST Mishap Investigator
Brian Jackson Programmatic Support
Auditor Audit Manager HQ Southern Region
(JSC, MFSC, SSC)
Vacant
Software Assurance Audit support
Martha
from the field
Wetherholt
centers, other HQs Dennis Morehouse
Software Brad Kaustinen
offices/SSCs Mishap Investigator
Assurance ST NCAS COTR
Western Region
Vacant NCAS (ARC, DFRC, JPL)
Operational Aviation Honeywell
Safety
Agency Range
Israel Greenfeld
Safety Efforts
IRIS Requirements Liaison
SARD/KSC Steve Lilley with MIBs/MIWG
MIB Technical Report
Reviews & Corrective
http://nsc.nasa.gov/ 6 Action Tracking 2/18/2010

7. Safety & Mission Assurance
Technical Excellence
• Within NASA, SMA professionals
provide a vital role in support of
NASA’s design and operations focus.
• SMA advises mission specialists on
matters related to risk, safety, and
mission success.
• SMA encompasses all Agency safety,
reliability, maintainability, and
quality engineering and assurance
activities.
7

8. STEP was Designed to be…
1. …a way to assist
the Agency
(specifically SMA)
as it transitions
from an
operational focus
back to a design
focus.
8

9. STEP was Designed to be…
2. …both a source of
knowledge and an
ongoing reference tool in
providing SMA
professionals with
structured, advanced
SMA educational
opportunities in support
of that design focus.
9

10. STEP Curriculum Development
Technical Discipline Teams developed a competency-
driven curriculum structure for each discipline that
included Education, Training & Lifelong Learning.
Job of SMA SMA Competency SMA Products
Professional Disciplines Wheel & Services
EDUCATION TRAINING LIFELONG LEARNING
10

11. STEP Curriculum
• The STEP Curriculum is delivered across four
levels with increasing specialization.
3 Intermediate 4 Advanced
2 Introductory
Discipline Discipline
1 Introduction Discipline
Concepts or Concepts or
to STEP Concepts
Specialty Specialty
[Doer]
[Team Leader] [Expert]
Generalized Specialized
11

12. A Typical STEP Program Commitment
Estimated Hours per STEP Level
STEP Training Elements Level 1 Level 2 Level 3 Level 4
Core Training 3 24 40 N/A
Discipline Training 18 100 137 137
Domain Training (by others) 4 40 TBD TBD
Total Course Hours 25 164 177 (+TBD) 137 (+TBD)
OJT N/A 200 400 400
Years of Experience N/A 2 4 8
Test and/or Peer Board N/A N/A Yes Yes
CEU’s N/A 16 24 32
12

13. STEP Training Elements
13

14. STEP’s Benefit to
NASA’s Programs/Projects
• High confidence in SMA’s competence and capability.
• High confidence in SMA’s products & services.
• Improved technical program/project performance.
• Risk Reduction:
– Increased success likelihood.
– Improved risk characterization.
– Improved risk-based decision-making.
• Serves as showcase for engineering excellence.
• Complements Agency training initiatives:
– Academy of Program/Project and Engineering Leadership (APPEL).
– System Engineering Leadership Development Program (SELDP).
– Senior Executive Service Career Development Program (SES CDP).
14

15. STEP Level 1
Curriculum
• One curriculum across all six disciplines
• Total of 25 contact hours
– Each course is one hour on average
• Topics covered:
– SMA Implementation
– Core
– Domain
– Disciplines
– Case Studies
• On-demand and self-paced
15

16. Level 1 Initiatives
• Agency-wide roll-out
• Online Courses
• Handbook
• Quick-start guide
• Course DVD
• Brochures
16

17. Level 1 Participation
Post Rollout
• Over 1,300 individuals have
been assigned or
registered for the curricula
• 361 graduates as of Nov 1
• 26% of all enrollees have
successfully completed
Level 1 – two months after
rollout!
• Feedback: nearly 70% rate
the program as “high
value”
17

18. STEP Levels 2 - 4
Curriculum Features
• Unlike Level 1’s unified curriculum, Levels 2 - 4 provide a unique curriculum
for each discipline
• Participants will choose a discipline and receive a Course of Study to guide
them through each level.
• Levels 2-4 allow for electives & discipline specialization
• The course of study for each discipline includes a variety of course types:
– Online/e-learning via SATERN
– Instructor-led workshops
– APPEL, NSTC and SARD courses
– Commercially-available courses
– Courses created by and for the NASA Safety Center
Levels 2 – 4 also encourage OJT, Enrichment Experiences, Career
Training & Continuing Education.
18

19. Competency Wheel
• The planning process
started with the
Competency Wheel
for each discipline,
from Level 1.
• The Competency
Wheel lists all the
competencies an SMA
professional needs to
perform.
– The inner ring lists the major
competencies.
– The outer ring provides a
further level of detail to each
of the major competencies.
19

20. Performance Objectives
For each competency, detailed objectives were written to describe exactly what
performance is expected at each level. (Typical performance objectives shown)
Level 2 Level 3 Level 4
• Recognize QE Quality • Perform QE Quality • Evaluate Quality
Management functions. Management functions. Management functions
• Recognize Quality • Apply Quality and Systems.
Management as one of Management principles
four (4) second tier to develop Quality
competencies of the Management Systems
Perform Quality (QMS).
Management & Planning
Competency.
• Recognize the six (6) third
tier competencies in
Quality Management.
20

21. SMA Work Products and Services
SMA Work Products and Services were identified for each
discipline. These are the things SMA Professionals
actually do on the job.
21

22. Course of Study
The Result:
• A complete curriculum
plan – a Course of Study –
was developed for each
discipline, by level
• The course of study
includes a list of
– Courses
– Readings & Resources
– OJT Experiences
– Lifelong Learning Activities
22

23. STEP – A Complete Curriculum
• Overall, STEP will be a
complete curriculum
tailored specifically for
the SMA Professional.
• STEP is specifically
designed to address the
technical needs of
NASA’s programs and
projects.
23

24. System Safety Discipline
OVERVIEW
24

25. Technical Excellence does pay off…
…and you do get what you pay for.
Coach 1st Class
US Airways Flight # 1549 25

26. SS Discipline – Where are we today?
• Most SS professional’s training is ad hoc training and professional
development (not many opportunities for SMA degrees at universities).
• We “learn as we go” and heavy emphasis on unstructured OJT.
• We do a lot of “reviewing” but not a lot of actually “doing” and this is
perceived as a negative thing.
• Don’t address new hires’ training needs.
• Don’t have mentoring for entry or mid-level employees.
• Training needs to apply to “real” work vs. perceived or ideal work that we
think we should be doing.
• Don’t have structured OJT with a clear understanding of what it is that is
expected to be learned.
• Training does not address softer SS discipline areas like resource
estimation, safety in contracts, and SS integration with system and
program lifecycle.
• Training is presently focused on qualitative methods and we would benefit
as a community from more analytical-based training.
26

27. What Do System Safety Engineers Do?
Fundamentally, NASA System Safety Engineers, at some point in
their careers, during any phase of the system life cycle, do any
combination of three basic activities:
1. Assess any situation with safety consequences.
2. Manage safety in acquisitions.
3. Manage, leverage, and interact with safety organizations and NASA
programs.
From this premise, all competencies and products and services are
derived.

28. System Safety’s Interactions
with Programs/Projects
System Safety Products (E.g., Hazard
SS products and
SS Engineer applies Analysis, FTA, PRA, CRM, etc.)
Framework
competencies to
influences
develop SS products
decisions
2
3
SS Engineer Decision Maker
Competencies define
1 how SS Engineers do SS products support milestones
their job at NASA 4 in the Program lifecycle Decisions
influence the
5 outcome of
Programs
System Safety
monitors lifecycle
Safety Performance
6

29. Overall System Safety Framework
Draft 0 of Overview of System
Safety – Not to be Distributed
Outside of SS Working Group

30. System Safety Student Body Estimates
System Safety STEP participants
Civil Servant SMA & SMA's Support Contractor
SMA - Civil SMA -
Center Servant Contractor Total
GRC 14 6 20
JSC 110 233 343
MSFC 32 35 67
LaRC* 10 20 30
DFRC 2 0 2
JPL n/a 10 10
ARC* 5 10 ?
SSC 4 10 14
GSFC 31 73 104
KSC 28 25 53
HQ/NSC* 10 2 12
* Estimated
Total 246 424 670
30

31. SS Competency Wheel
A SMA System Safety
professional needs to possess
these 34 competencies:
Major Competencies
Technical
• System Safety Mathematical Skills
• System Safety Analytical Methods
• System Safety Rationale
Managerial
• System Safety in Acquisition Management
• System Safety in Organizational Management
31

32. Competency Performance Objectives
and Training, OJT, and Reference Material Allocations
(1 of 34 competencies)
Competency: Hazard Analysis
Description: The analytical method and approach used to identify, mitigate, and manage
safety hazards found in systems.
Performed by: System Safety Engineer
STEP Level 2 STEP Level 3 STEP Level 4
• Understand and explain the • Prepare a quantitative HA on • Monitor the performance of
basic engineering principles, a NASA space subsystem or HA management systems and
steps, and elements of a system. processes used by the
Hazard Analysis (HA). • Prepare a Hazard Report (HR) organization.
• Identify and explain NASA with all required content. • Conduct analysis of
policy and procedures • Identify, document, and supporting HA data for trends,
governing the conduct of HA. monitor requirements indicators, and pitfalls.
• Prepare a qualitative hazard associated with a HR. • Interpret, apply, communicate,
analysis on a simple, isolated • Monitor the implementation and monitor requirements
NASA space subsystem. and performance of Hazard related to HA for NASA
• Document and present a HA Controls throughout the programs and projects.
to a senior official. lifecycle. • Influence policy or procedures
• Understand and explain the • Evaluate HR/HA prepared by to improve the conduct of
relationships of HA to fault contractors. HA/HR.
tree analysis and the • Present a HA/HR to decision
relationship of the HA with makers.
other SMA disciplines.
Training and/or Reference and/or32 OJT equals COMPETENCY

33. Core and Advanced
System Safety Products
Core System Safety Products & Services
• Are products & services that all NASA’s System Safety Engineers are
expected to be able to produce at any time.
–E.g., Create a Hazard Analysis.
• Are generally independent of specific job location, project, or job function.
Advanced System Safety Products & Services
• Are products & services that NASA’s System Safety Engineers are expected
to be able to produce if their job assignments demand it or as their careers
progress and their responsibilities escalate.
–E.g., Support a SEB for a major procurement/acquisition activity.
• Are generally dependent on specific job location, project, and job function.
33

34. SS Products & Services
Distributed across Levels
STEP Level 2 STEP Level 3 STEP Level 4
• Conduct Hazard Analysis • Conduct Hazard Analysis • Conduct Hazard Analysis
• Conduct Fault Tree Analysis • Conduct Fault Tree Analysis • Conduct Fault Tree Analysis
• Prepare Safety Rationale • Prepare Safety Rationale • Prepare Safety Rationale
• Prepare Safety Data Package • Prepare Safety Data Package • Prepare Safety Data Package
• Support Anomaly Resolution • Support Anomaly Resolution • Support Anomaly Resolution
• Provide Decision Maker Input • Provide Decision Maker Input • Provide Decision Maker Input
• Communicate Safety • Communicate Safety • Communicate Safety
Requirements Requirements Requirements
• Conduct Surveillance & • Conduct Surveillance & • Conduct Surveillance &
Compliance Verification Compliance Verification Compliance Verification
• Safety Performance Metrics • Safety Performance Metrics • Safety Performance Metrics
Monitoring Monitoring Monitoring
---------------------------------------------- ---------------------------------------------
• Safety Acquisition Management • Safety Acquisition Management
• Probabilistic Risk Assessment • Probabilistic Risk Assessment
• Risk Analysis • Risk Analysis
• Integrated Safety Data Package • Integrated Safety Data Package
• Support Certification of Flight • Support Certification of Flight
Readiness Readiness
----------------------------------------------
Green = Core • Safety Organizational
Management
Red = Advanced • Support Lessons Learned
• Manage Policy/Standards &
Requirements
• Support Special Studies
• Support Mishap/Close-call
investigations
34 • Support Audits & Assessments

35. SS Community of Practice
Q: What is the SS COP?
A: Anything associated with the conduct
of System Safety activities that folks
need to know, share, keep, or access.
Examples:
1. Agency “go-to” SS Website.
2. Mass email notification list.
3. Guest Lecture Series.
4. Webcasts, Podcasts, etc.
5. Blogs/Papers with ratings.
6. Clearing House Concept
Tools, Documentation, Reference,
Resources, Conferences, Speakers,
3
Lessons Learned, Training, Etc. 5
35

36. SS Website
36

37. STEP enables Program/Project
Risk Reduction through…
• Providing a competent and highly capable System Safety
workforce.
• Providing predictable, consistent, and value-added System
Safety products & services to programs/projects.
• Analytical tools proliferation (e.g., SAPHIRE/QRAS).
• Enabling risk-informed decision-making (RIDM).
• Providing safety throughout the entire program acquisition
lifecycle.
• Providing a forum for a community of practice to encourage
lessons learned and communications.
• Proactive human capital management.
37

38. State-of-the-Discipline (SOD)
Asks: “What is the health status
of the SS Discipline?” and
“What are we doing about it?”
Sample SOD Metrics
• STEP Qualification distribution.
• Training metrics/rates/distribution.
• Integrity of SS products over time.
• Analytical tools proliferation.
• Proactive Human Capital Management
• Resources, staffing, work-load.
• Rate of Hazard Controls violations.
• Safety-related Technology Readiness.
38

39. State-of-the-Discipline
Example
Agency CMS Competency Level
Agency CMS Competency Level 1 Level 2 Level 3 Level 4 Total
"SAFENG" 181 321 288 197 987
"SYSTEMENG" 325 742 930 503 2500
506 1063 1218 700 3487
Combined CMS Competencies Levels
Level 1 Level 2 Level 3 Level 4 Total
Combined SAFENG and
17 20 21 26 84
SYSTEMENG
3% 2% 2% 4% 2%
Metric: Proactive Human
Capital Management Desired
Desired SOD
metric
• What is the competency improvement
relationship between
System Safety and System
Engineering at NASA?
• Data supports observation
that SMA and SE could be
better integrated.
• Is an overall 2% cross
competency relationship
acceptable?
• What are we doing about it?
39

40. STEPPING FORWARD
• Providing Education for NASA’s Future…
… one STEP at a time.
40

41. Back-up
41

42. SMA Technical Excellence Program
CURRICULUM BY DISCIPLINE
42

43. STEP Internal Review
SOFTWARE ASSURANCE
KAREN MEINERT
43

44. SA Technical Discipline Team
• NSC TDT Lead – Karen Meinert
• NSC SSC – Jennifer Jones
• HQ – M. Wetherholt, M. Bodeau
• ARC – C. Chow, G. DelCarmen
• DFRC – D. Tran
• GRC – T. Luu • JPL – S. Flanagan, J. Wilf
• GSFC- S. Harris • JSC – R. Hugger
• IV&V – L. Montgomery • KSC – D. Victor
• LaRC – L. Johnson
• MSFC – R. Strickland
• SSC – C. Fallo

45. SA Competency Wheel
• Software Assurance is
comprised of the
following competencies:
– Software Quality
– Software Safety
– Software Reliability
– Software Verification &
Validation
– Software Independent
Verification & Validation
– Software Assurance
Planning & Management
45

46. Competencies by Level
Competency: Software Safety
Description: Knowledge, skills and abilities in identifying, analyzing, tracking, mitigating and controlling
software hazards and hazardous functions to ensure safer operation within a system. Activities performed
ensure that safety is designed into the software.
STEP Level 2 STEP Level 3 STEP Level 4
Performance Expectation Performance Expectation Performance Expectation:
Observe and participate with supervision Under general awareness of supervisor, o Determines criteria for safety critical
in software safety activities throughout perform software safety activities software classification
the lifecycle including: throughout the lifecycle including: o Develop/review safety standards
o Identify safety critical software
o Perform software safety litmus test o Identify safety critical software o Complete hazard reports with Systems
o Assist in reviewing: system PHA o Review system hazard analyses Engineering
requirements, designs, code, analyses, o Perform or review software safety o Develop software hazard reports
change requests, testing documents, analyses o Assure hazards verifications are followed
planning documents, lessons learned, and o Verify compliance to appropriate o Assess adequacy of safety requirements
traceability of safety requirements safety standards and software safety documentation
o Review software safety plans o Plan and manage software safety efforts
o Participate in peer reviews o Assure V&V requirements are met o Perform software safety activities
o Attend/observe milestone reviews o Review/assure software safety throughout the lifecycle, including:
and safety reviews requirements are documented and o Develop/review/assess
o Assist in assurance of CM of safety tracked throughout the life cycle requirements, designs, code, and
requirements o Participate in reviews of designs and test documents;
o Assist in software acquisition code o Assess development and test
activities and evaluation of o Review test documentation environments;
contractor deliverables. o Identify potential risk/safety issues o Determine adequacy of V&V
and verify issues are identified and activities;
tracked to resolution (…continued)
(…continued)
46
This is one of six competencies for this discipline

47. Mapping Training to Competencies
47

48. Current Status
Software Assurance Levels 2-4
Discipline/
Training
Course Type Needs Modification New Development
Available
(30 Courses)
Instructor-Led
93% 7% 17%
Courses
Online Courses 7% .5% 1%
Total 100% 7.5% 18%
Hours 374 28 68

49. Course of Study
• The Course of Study
includes a list of
– 30 Courses
– Readings & Resources
– OJT Experiences
– Lifelong Learning
Activities
49

50. STEP Forward
QUALITY ENGINEERING
RAY KACMAR
50

51. QE Technical Discipline Team
• NSC TDT Lead – Ray Kacmar
• NSC SSC – Jerome Anderson
• HQ – Brian Hughitt
• ARC - Steve Jara, Damon Flansburg
• DFRC - Gary May, Kevin Reilly • JPL - Thomas Smith, Tony Gutierrez
• GRC - Bob Makovec • JSC - Dave Dyer
• GSFC - Louis Thomas, Mike Kelly • KSC - Robert Ellison
• LaRC - Elijah Kent, Charlie Greenhalgh
• MSFC - Ken Crane
• SSC - Tim White
• NCAS - Charles Dionne
51

52. QE Competency Wheel
An SMA Quality Engineering
professional needs to possess
these major competencies:
• Perform Quality Assurance
Management & Planning
• Assure Adequate Design &
Development Controls
• Conduct Procurement QE
• Define and Implement Production
Material & Process Controls
• Define and Perform Inspection
Testing & Acceptance Activities
• Implement Problem Reporting &
Corrective Action
• Conduct Quality Evaluations & Data
Analysis The inner ring lists the major competencies. The outer ring provides
a further level of detail to each of the major competencies.
52

53. Competencies by Level
Competency: Quality Management
Description: Recognize, Perform and Evaluate Quality Management Functions
STEP Level 2 STEP Level 3 STEP Level 4
• Recognize QE Quality Management • Perform QE Quality Management • Evaluate Quality Management
functions. functions. Functions and Systems.
• Recognize Quality Management is one • Apply Quality Management principles • Perform a complete quality
of four (4) second tier competencies of to develop Quality Management management system evaluation to
the Perform Quality Management & Systems (QMS). ensure compliance to all applicable
Planning Competency. • Based on a review of the quality QMS requirements defined in the
• Recognize the six (6) third tier requirements, plan the QMS (to include planned evaluation.
competencies in Quality Management. objectives, scope, schedule, and • Become a Certified Quality Engineer
• Understand the concepts of quality technical expertise). (Elective).
management and quality management • Document the results of the QMS, to • Evaluate human factor issues and
systems. include any findings, observations, or human error prevention techniques.
• Understand the concepts of strategic assessments of risk. (Elective).
planning for quality. • Perform strategic planning (Elective). • Provide instruction to operators and/or
• Understand communication • Develop effective communication inspectors in the methods of
mechanisms. mechanisms (Elective). product/service verification, throughout
• Recognize the factors that influence • Identify human factor issues and the value-stream.
the work environment. human error prevention techniques
• Recognize NASA polices on Human (Elective).
Factors.
• Recognize the importance of NASA’s
training & certification requirements
related to quality and identify related
training.
• Recognize related sources of
information.
This is 1of 48 competencies for this discipline

54. Mapping Training to Competencies
(CIL ems Lis d
nt
te
ning t and
) an
sis
eme
As su Quality
d
De si Adequa
ent
ent
t
Revie puts an
lysis ects
s
Inte g Qua lity
sis
Analy
Critic (FMEA
r
t rols t
stem
Qua ment
awa rd/P ost
Orde
agem
agem
en
es & r e
ana g
en
ring
ns &
Stan nship
Man r ation
ool s
nal y
ent
ent
nts
ilu
Man ce
ent
Eff
l opm
s ed
Man ion
agem
es
n
gn &
al S y
s
ur
n Fa
Tree
,
Proc t
ran
Asse l
agem
agem
r atio
n In
ensio
n of
inee
li ty M
r ime
ings
hase
orm
ss m
M an
Man
ure
nic a
rd A
dard
duc
li ty T
al It
ranc
ram
ws
re
lity
ui sit
kma
rd S
wa
figu
k Ba
w
As su
De ve
Desig
Desig
Desig
Critic
Training
Pla n
Pr e-a
Fault
Draw
Expe
Haza
rev ie
Tech
Perf
Pr og
)
Con
Con
E ng
Data
Purc
Risk
Mod
Tole
Co n
Wor
Qua
Qua
An a
Dim
Acq
Ris
Training Course Hours
Qualit y Management Overview 3 2 2
Qualit y Management Syst ems 8 3
Workforce Development , T raining
& Cert ificat ion 0.5
P rogram Qualit y Int egrat ion
Overview 1 2
P rogram Qualit y Int egrat ion 4 3
Qualit y Assurance P lans 3 2
Lessons Learned 1 2
T echnical Assessment Overview 1.5 2
T echnical Assessment 4 3
Audit s 0.5 2
AS 9100 Lead Audit or Class 40 4
Qualit y Assurance Surveillance P lan 1 2
Qualit y T ools Overview 1.5 3
Qualit y T ools 5 4
Qualit y T ools - NASA Addendum 2 4
Lean Six Sigma 8 4
Qualit y Funct ion Deployment 6 4
Design Input s and Reviews Overview 4 2 2
Design Input s and Reviews 8 3 3
Mist ake-P roofing Execut ive
Overview 4 2
Mist ake P roofing 16 3
Workmanship St andards Overview 1 2
Design of Experiment s Execut ive
Overview 4 3
Design of Experiment s 32 4
Drawings, Dimensions, T olerances
Overview 1 2
Drawings, Dimensions &
T olerancing - S-44 40 3
Failure Modes Effect s Analysis &
Crit ical It ems List (SAT ERN) SMA-
017-01 1 2
Syst em Safet y I 8 3 3
Cont inuous Risk Management -
AP P EL - CRM 24 4
Configurat ion Management 4 2
Funct ional Configurat ion Audit /
P hysical Configurat ion Audit 0.5 2
Dat a Management 2 2
P art s Cont rol Overview 1
P rocurement Qualit y Overview 8 2 2 2 2
NASA Let t er of Delegat ion 2
54

55. Current Status
Quality Engineering Levels 2-4
Discipline/
Training
Course Type Needs Modification New Development
Available
(66 Courses)
Instructor-Led
58% 0% 0%
Courses
Online Courses 18% 14% 10%
Total 76% 14% 10%
Hours 282.5 53.5 38

56. Course of Study
• The Quality Engineering
Course of Study includes
a list of
– 66 Courses
– Readings & Resources
– OJT Experiences
– Lifelong Learning
Activities
56

57. STEP Forward
RELIABILITY & MAINTAINABILITY
DR. NAT JAMBULINGAM
57

58. R & M Technical Discipline Team
• NSC TDT Lead – Nat Jambulingam
• NSC SSC – Maria Havenhill, Dan Deans, Bob Loomis
• HQ – Frank Groen
• HQ Tech Fellow – Fayssal Safie
• ARC – N/A
• DFRC – N/A
• GRC - Edward Zampino
• GSFC - Tony Diventi, Matthew Samuel
• JPL - Chau Brown • JSC - Samadar Roshan-Zamir
• KSC - Chi Yeh, Tim Adams
• LaRC – Pete Parker
• MSFC - Richard Stutts
• SSC - Mike Rewis
58

59. R & M Competency Wheel
A Reliability & Maintainability
professional needs to possess
these major competencies:
• Reliability Management
• R&M in Acquisition
• R&M Methods in Design &
Development
• Reliability Testing and
Demonstration
• Use of Database and Reports for
R&M
• Mathematical Tools
The inner ring lists the major competencies. The outer ring provides
a further level of detail to each of the major competencies.
59

60. Competencies by Level
Competency: FMEA/CIL, FMECA
Description: Systematically identifies system’s potential failure modes, and evaluates
their causes and effects on the system’s ability to perform its functions.
STEP Level 2 STEP Level 3 STEP Level 4
• Define failure modes, integration • Interpret and apply FMEA or FMECA • Perform/evaluate FMEA or FMECA on
characteristics, propagation program requirements. increasingly complex systems.
• Describe the contents of FMEAs, Critical • Perform a FMEA or FMECA on assigned • Accurately & thoroughly document
Items List (CIL) and FMECAs. system/subsystem/component. FMEA and Critical Item List analysis
• Describe how and why Critical Items are • Evaluate a FMEA and Critical Item List performed & describe the implications
created out of the FMEA analyses. analysis & coordinate comments with the conclusions may have.
• Locate and read program requirements. the subsystem engineer. • Explain nuances not readily evident to
• Describe the FMEA, Critical items and • Assess compliance to the reliability less experienced personnel.
FMECA process flow. design requirements verified by the • Assess compliance of submitted
• Describe the relationship of CIL to FMEA or FMECA. FMEA/CIL, FMECA to requirements
hazard reports. • Identify common cause failure modes in document.
• Identify simple generic failure modes addition to system failure modes. • Determine compliance to Reliability
and postulate the effects based on an • Identify failure modes that are specific design requirements verified by the
understanding of component to the design configuration under FMEA or FMECA.
functionality. analysis. • Lead a team in developing a FMEA and
• Identify simple failure mode effects at • Trace the effects of failure modes to a Critical Item List.
the system level the system level. • Coordinate the team activity on a
• Coordinate with design engineering to complex system so that the interfaces
develop retention rationale for the between subsystems have coverage
critical items. and the Critical Item retention rationale
• For those engineers involved in meets all the requirements of the
diagnostics, describe the relationship Program FMEA/CIL or FMECA
between FMEA, FMECA and fault requirements document.
detection and isolation.
This is one of 41 competencies for this discipline 60

61. Mapping Training to Competencies
Competencies
R&M into Govern
contracts, ment-
Potential Source SOWs, Industr Comme
specifications, Defining and y Data rcial
deliverables, controlling Parts NASA Exchan Materia and
R&M R&M Managing and award R&M Reliabil Data and Lesson ge l Contra other
Uncert Sensiti Decisio Boolea Progra Starate Evaluating the structure, and requirements ity Fault Root Probab Parts Reliabil Maintai Availab Mecha Softwa Analysi Materia Reliabil Design Reliabil s NASA Progra Review ctors Givern
ainty vity n n Data Markov R&M m gic elements of a other for Block Tree FMEA/ Cause ilistic Risk Stress ity nability ility nical re Human s and l ity Test of ity Test Produc Reliabil NASA Learne Internal m Board Test ment
Training Probab Statisti Analysi Analysi Analysi Algebr Analysi Analysi Planni Manag Manag contractor's acquisition program/projec Diagra Analysi CIL, Analysi Analysi Analysi Analysi Modeli Modeli Modeli Reliabil Reliabil Reliabil Trendi evaluat Planni Experi Develo t Life ity PRACA d Report (GIDEP Report Report agenci
Training Courses Hours ility cs s s s a s s ng ement ement R&M elements t acquisition m s FMECA s s s s ng ng ng ity ity ity ng ion ng ment pment Testing Growth System System s ) s s es data
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41
1 OSMA+NSC R&M Principles 24 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
2 NSC Design for Reliability 32 2 2 2 2 2 2 2
3 NSC Design for Maintainability 32 2 2 2 2 2 2 2
NSC Reliability, Availability, and Maintainability (RAM) Planning for
4 Programs and Projects 14 2 2 2
5 Commercial Data Collection and Analysis I* 16 3 3 3 3 3 3 3 3
NSC Reliability, Availability, and Maintainability Modeling (including
6 success tree analysis) I 16 3 3 3
GSFC/JPL Parts and Materials Assessment (EEE and Mechanical), Parts
Stress/Derating and also incl. Thermal Analysis and Structural
7 Analysis I 16 3 3
NSC
8 Maintainability and Supportability Analysis and Integration* 16 3
9 SATERN/JPL FMEA/CIL, FMECA 16 3
10 NSC R&M Testing and Demonstration I* 16 3 3 3 3
11 Commercial Reliability Growth 16 3
12 NSC Dependency and Common Cause 16 3 3 3
JPL
13 Single Event Upset/Effects Analysis, Radiation Effects I 16 3 3 3 3
14 Commercial Sneak Circuit Analysis 16 3 3 3
OSMA
15 Root Cause Investigation and Corrective Action/Failure Analysis 8 3
16 NSC GIDEP 8 3 3
17 Commercial Certified Reliability Engineer (CRE) Refresher 16
18 SS Probabilistic Risk Assessment (SS) 24 3 3 3
19 QE Problem Reporting (QE) 4 3 3 3
20 SS Hazard Analysis (SS) 8 3 3 3 3
21 SS Fault Tree Analysis (SS) 24 3
22 NSC Data Collection and Analysis* II 16 4 4 4 4 4 4 4 4
NSC Reliability, Availability, and Maintainability Modeling (including
23 success tree analysis) II 16 4 4 4
NSC Parts and Materials Assessment (EEE and Mechanical), Parts
Stress/Derating and also incl. Thermal Analysis and Structural
24 Analysis II 8 4 4
25 NSC R&M Testing and Demonstration II* 16 4 4 4 4
26 Commercial Reliability Growth 16 4
27 SA Software Reliability (SA) 16 4
28 Commercial Physics of Failure 16 4 4 4 4 4
JPL
29 Single Event Upset/Effects Analysis, Radiation Effects II 16 4 4 4 4
30 JPL Worst Case Analysis 24 4 4
Commercial (NRC)
31 Human Reliability 28 3
NSC/ Commercial
32 Design of Experiments 40 4
Cross Discipline
Course Probability and Statistics Basics : Source Document: NASA SP-
** 2009-659 (SS) 3 3
TOTAL COURSES ADDRESSING COMPETENCY 2 3 1 1 1 1 5 1 3 3 3 3 3 3 3 1 4 2 1 2 2 8 8 9 4 1 1 3 2 3 1 3 3 5 1 2 2 3 2 2 1
61

62. Current Status
R & M Levels 2-4 Discipline
Discipline/
Training
Course Type Needs Modification New Development
Available
(32 Courses)
Instructor-Led
19% 0% 15%
Courses
Online Courses 25% 9% 32%
Total 44% 9% 47%
Hours 166 32 176
Total Discipline Hours = 374 [302 hours are required; 72 hours are elective]

63. Course of Study
• The R & M Course of
Study includes a list of
– 32 Courses
– Readings & Resources
– OJT Experiences
– Lifelong Learning
Activities
63

64. STEP Forward
SYSTEM SAFETY
MARK M. KOWALESKI
64

65. SS Technical Discipline Team
• NSC TDT Lead – Mark Kowaleski
• NSC SSC – Maria Havenhill, Diane
Chapman, Jennifer Baumeister, Kathy Kraft
• HQ – Homayoon Dezfuli
• ARC – N/A
• JPL - Kirk Barrow
• DFRC - Jonathan Brown
• JSC - Gerald Readore, Larry Gregg
• GRC - Bill Schoren, Carrie Green
• KSC - Ronald Long, Robert Schwader
• GSFC - Bo Lewis, Angela Melito
• LaRC - John Greco, Jose Caraballo
• MSFC - Herb Shivers, Keith Layne
• SSC - Robert Gargiulo
65

66. SS Competency Wheel
An SMA System Safety professional
needs to possess these major
competencies:
Technical
• System Safety Rationale
• System Safety Analytical Methods
• System Safety Mathematical Skills
Managerial
• System Safety in Acquisition Management
• System Safety in Organizational
Management
The inner ring lists the major competencies. The outer ring provides
a further level of detail to each of the major competencies.
66

67. Competencies by Level
Competency: Hazard Analysis
Description: The analytical method and approach used to identify, mitigate, manage and communicate safety hazards found
in systems.
STEP Level 2 STEP Level 3 STEP Level 4
• Understand and explain the basic • Prepare a quantitative HA on a • Assess the performance of HA
engineering principles, steps, and NASA space subsystem or system. management systems and
elements of a Hazard Analysis • Prepare a Hazard Report (HR) with processes used by the
(HA). all required content. organization.
• Identify and explain NASA policy • Identify, document, and monitor • Compare actual vs. predicted
and procedures governing the requirements associated with a HR. system performance and formulate
conduct of HA. • Monitor the implementation and and manage corrective actions, as
• Prepare a qualitative hazard performance of Hazard Controls necessary, to ensure efficacy of
analysis on a simple, isolated throughout the lifecycle. hazard control.
NASA space, ground, and/or test • Evaluate HR/HA prepared by • Conduct analysis of supporting HA
hardware/operations subsystem. contractors. data for trends, indicators, and
• Document and present a HA to a • Present a HA/HR to decision pitfalls.
senior official. makers. • Interpret, apply, communicate, and
• Understand and explain the • Utilize anomaly resolution input to monitor requirements related to HA
relationships of HA to fault tree evaluate impacts to system-level for NASA programs and projects.
analysis and other relevant hazard risk through the use of • Influence policy, procedures, and/or
analytical techniques, and the qualitative and/or quantitative tools. requirements documents to
relationship of the HA with other • Utilize Fault Tree Analysis (FTA) or improve, standardize and
SMA disciplines. an equivalent tool to map hazard encourage the conduct of HA/HR
causes and controls to other safety for NASA programs and projects.
and reliability data products (e.g.
FMEA/CIL).
This is 1 of 34 competencies for this discipline. 67

68. Mapping Training to Competencies
les
c qu)
th .
ss .
era b
on
e Me
t
t
u dge
pmen
ts (A
adine
nic ati
t
sme n
etc.
deliv
s is
ale
utc om
rams
m en
and b
Ana ly
evelo
m mu
ht Re
sis
OW ,
ati on
s ses
safety
ons
is
A naly
quire
is
Diag
ism/O
na ly s
ati on
ac ts/S
ming
d Co
ri o D
n aly s
s is
urces
niz ati
ght R
f Flig
bil ity
Source
sis
isk A
s is
Analy
is
duc ts
fety re
aluate
b ra
enc e
A naly
Analy
Estim
ailure
alys is
naly s
nc e A
c han
on an
gram
Reli a
cena
ent o
rror A
o rga
res o
of Fli
contr
s is
ti c R
sis
sis
A lge
ge pro
Ana ly
S equ
rta inty
ge/ev
are F
A naly
e/Me
m Sa
ent S
ion A
rd An
ab ili ty
A naly
r Pro
rs em
Tree
eque
itiv ity
entati
hood
ab ili s
ware
ging
ging
an E
os al/
ents
s tic s
an
Sou rc
Boo le
t
Softw
Dec is
Linea
Ac cid
Sy ste
Mana
Mana
Mana
Mana
Utility
Haz a
Unc e
End o
Cons
Sen s
Likeli
Elem
Ev en
Hard
Fault
Hum
Prob
Prob
Data
Prop
Pres
Sta ti
Ris k
Training Element
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30
Competencies for Core
Products/Services
Class Training
Probability and Statistics Basics I: Source
OSMA
Document: NASA SP-2009-659 2 2 3 3 3 3 3 3 3 3
Risk Management; Source: NASA RIDM
OSMA Handbook (under development) and CRM
Handbook (to be developed) 3 3 3 3 3 3 3 3 3 3 3
Scenario-based System Safety Analysis:
OSMA Source: NASA System Safety Standard (under
development) 3 3 3 3 3 3 3 3 3
Basic Fault Tree Analysis I; Source Document:
OSMA
NASA Fault Tree Handbook 2 2 2 2 2 2 2
System Safety I: Source Document: NASA
OSMA
System Safety Standard (under development) 2 2 2 2 2 2 2 2 2 2 2 2
SMA-SAFE-NSTC-0002, System Safety
NSTC
Fundamentals 3 3 2 2 3 3 2 2 2 2 2 3 2 3
Probabilistic Risk Assessment Methods (PRAM);
OSMA Source Document: NASA PRA Procedures
Guide 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
Probability and Statistics Basics II: Source
OSMA
Document: NASA SP-2009-659 3 3
RIDM Source: NASA RIDM Handbook (under
OSMA
development) 3 3 3 3 3
Basic Fault Tree Analysis II; Source Document:
OSMA
NASA Fault Tree Handbook 3 3 3 3 3 3 3
System Safety II: Source Document: NASA
OSMA
System Safety Standard 3 3 3 3 3 3 3 3 3 3 3 3
APPEL Project Planning Analysis and Control (PPAC) 2 2 2 3 3 3
Management of Space Technology Programs
APPEL
(APPEL-MSTP) 2 2 2
Application of Bayesian Probability Inference to
OSMA
Risk and Reliability Analysis of Space Systems;
Source Document: NASA SP-2009-659 3 3 3 3 3 3 3 3 3 3 3 3
SMA-SAFE-NSTC-0040, System Safety
NSTC
Engineering 3 3 3 3 2 3 3 2 3 3 3 3 3 3
APPEL Assessing Project Performance (APPEL-APP)
Performance Based-Statement of Work (APPEL-
APPEL
PBSOW) 3 2
Application of SAPHIRE to PRA: Source: NASA
OSMA PRA Procedure Guide and SAPHIRE Users
Manual
SMA-SAFE-NSTC-0023, System Safety in
3 3 3 3 3 3 3 3 3 3 3
68
3 3 3
NSTC
Acquisition 2 2 3 3 3
TOTAL COMPETENCIES ADDRESSED 5 5 4 1 0 10 11 11 12 5 5 4 8 4 4 8 12 7 9 5 2 3 5 3 7 5 6 1 1 1

69. Current Status
System Safety Levels 2-4
Discipline/
Training
Course Type Needs Modification New Development
Available
(19 Courses)
Instructor-Led
100% 15% 10%
Courses
Online Courses 0% 0% 0%
Total 100% 15% 10%
Hours 374 94 0

70. Course of Study
• The System Safety
Course of Study includes
a list of
– 19 Courses
– Readings & Resources
– OJT Experiences
– Lifelong Learning
Activities
70

71. STEP Forward
OPERATIONAL SAFETY
VANESSA PELLEGRINO
71

72. OS Technical Discipline Team
• NSC TDT Lead – Vacant
• NSC SSC - Vanessa Pellegrino
• HQ (SARD) – Jonathan Mullin
• HQ - Matt Deptola
• ARC – David King
• DFRC – Leonard Mbuko • JPL – Peter Robles
• GRC – Angela Windau • JSC – Bubba Johnson
• GSFC – Pat Hancock • KSC – Cal Staubus
• IV&V – Kurt Kehl, Dave Sheldon • LaRC – Jose Caraballo
• MSFC – Dennis Davis
• NSTC – Tim Evans
• SSC – David Lorance
• WFF – Kenny Volante
72

73. OS Competency Wheel
• An SMA Operational Safety
professional needs to
possess these major
competencies:
• Recognizing Hazards
• Measuring, Evaluating and Controlling
Hazards
• Safety Management
• Safety Metrics
• Collecting Risk Information
• Assessing Risk
• Managing Risk
• Hazards Specific to NASA The inner ring lists the major competencies. The outer ring provides
a further level of detail to each of the major competencies.
73

74. Competencies by Level
Competency: Personal Protective Equipment (PPE)
Description: Provide oversight and guidance for the Personal Protective Equipment program to ensure that
employees are protected from the hazards in the workplace.
STEP Level 2 STEP Level 3 STEP Level 4
• Understand and describe the • Inspect employees' PPE • Evaluate that PPE is being
Hierarchy of Controls • Perform a PPE Assessment worn properly
• Describe the purpose of PPE • Perform training on • Evaluate and update a
• Understand the usage of the donning/doffing, cleaning, Personal Protective
different types of PPE storage, use and limitations of Assessment Plan
• Understand how to maintain PPE
different types of PPE • Perform selecting and testing
• Know the components and of PPE
requirements of an effective • Purchase appropriate PPE
PPE program as outlined in 29
CFR 1910.32 & 1910.33 and
1910.134-138
• Understand the requirements
in NPR 8715.3 Chapter 3,
Section 3.3
• List in order the common
steps performed during a
hazard assessment
• Understand how each type of
PPE is designed to protect
people
This is one of 38 competencies for this discipline
74

75. Practice
Handling
Engineering
Forklift Safety
Fall Protection
Aerial Platform
Life Safety Code
Safety Refresher
Mobile Crane Safety
Hazardous Locations
Ground Nuclear Safety
Stairw ays and Ladders
Basic Explosives Safety
Fall Protection Refresher
and Operational Personnel
Electrical Safety Refresher
Training Course
Electrical Safety Standards
Fire Protection - Theory and
Laboratory Safety and Health
Crane Operations and Rigging
Overhead Cranes and Material
Safety in High Pressure Systems
Fire Hazards in Oxygen Systems
Explosive Safety Management and
Refresher for Explosives Handlers
Safety in High Pressure Operations
7
3
3
3
3
3
4
7
1
14
14
21
14
21
14
21
14
14
14
12
21
Hours
Training
1
Biolo
g ic al H
aza rd
s
2
2
Chem
ic al H
a zard
s
2
2
3
3
Ele ct
ri
ca l H
az ard
s
4
Na tu
ra l H
az ard
s
2
5
Ra dia
tion H
a za rd
s
2
2
2
2
2
2
2
3
2
2
6
Struct
M ec h ura l and
nanic
a l Ha
Haz a z ards
2
2
3
7
rds R
e la te
d to F
Explo ire &
sions
Haz a
rds R
8
e la te
Hu m
an Fa d to
c tors
Erg on &
omic
M ea su s
9
reme
nt a n
M onit d
oring
Engin
e erin
g C on
trols
10 11
A dm
inistr
ative
Contr
Pers o ols
12
nal P
rotec
Equip tiv e
ment
M ana
g eme
nt Pro
ce ss e
Inspe
ction s
s a nd
Audit
ing
13 14 15
Proje
ct Ma
nag e
me n t
16
Risk M
anag
eme n
t
Sa fe ty
17
M ana
geme
Sys te nt
ms
Tra in
ing a
nd
18
C o mm
unic a
tion
M eth
ods
19
Grou
p Dyn
a mic
s
20
Ba sic
Fina n
cia l Prin
c iple
s
21
Prob
abilit
y and S
ta tis ti
cs
75
Perf o
rman
22
and In ce M
dic ato e tric s
rs
Identi
fy a
23
haz ard nd cha rac
s te rize
De sig
n a nd
mana use d
ata
24
gemn
c olle et s y
ct a n stem
s to
inform d va lidate
ation risk
Colle
ct an
inform d v alida te
ation ris k
c ultu by st
re, m udyin
anag g
25
style eme n
, busi
nes s t
fina n c lima
c ial c
ondit te,
the a ions
va ila
bility &
re sou
rce s of
Re se
arc h
laws , a ppli
polic ca ble
sta nd ies, co
a rds ns en
and sus
prac ti
26
c e s to be st
benc deve
hmark lop
an org s for
aniza a sse
perf o tion's ss ing
rman s afety
ce
Ev alu
ate
injury the ris k o
/prop f
with ert
27
a n org y damag e
aniza
Mapping Training to Competencies
fac ilit tion's
ie s, p
sy ste rodu
ms , c ts ,
e quip proce ss es
ment ,
& em
Audit ploye
ma n a es
sy ste ge me
nt
28
ms to
ag ain c o mp
st e st are
sta nd a blish
a rds ed
Analy
ze tre
and la nds in
gging le a
indic pe rfo ding
ators rma n
29
histo using c
ric al
and st inform
atistic a tion
a l me
thod
s

76. Current Status
Operational Safety Levels 2-4
Discipline/
Training
Course Type Needs Modification New Development
Available
(TBD Courses)
Instructor-Led
TBD TBD TBD
Courses
Online Courses TBD TBD TBD
Total TBD TBD TBD
Hours TBD TBD TBD

77. Course of Study
• The Course of Study
includes a list of
– Courses
– Readings & Resources
– OJT Experiences
– Lifelong Learning
Activities
77

78. STEP Forward
AVIATION SAFETY
DARREN CHILDERS
78

79. AS Technical Discipline Team
• NSC TDT Lead – Vacant
• NSC SSC – Darren Childers
• HQ – Richard Yasky
• DFRC – Jonathan Brown
• GRC – Mike Ernst/Paul Tavernelli • ARC – N/A
• JSC – Jill Brigham • GSFC – N/A
• LaRC – Noel West/Evan Horowitz • IV&V – N/A
• JPL – N/A
• KSC – N/A
• MSFC – N/A
• SSC – N/A

80. AS Competency Wheel
An Aviation Safety professional needs to
possess these major competencies:
• Aviation System Safety Analytical Methods
• Program Reviews
• Unmanned Aircraft Systems (UAS)
• Test Management/Operational Test and
Evaluation
• Basic Performance Flight Testing
• Vehicle Structures, Loads and Flutter Testing
• System Safety
• Modern Flight Control Systems
• Advanced Performance Flight Testing
The inner ring lists the major competencies. The outer ring provides
a further level of detail to each of the major competencies. 80

81. AS Competency Performance Objectives
Competency: Modern Flight Control Systems
Description: In-depth understanding of linear control that includes the impact that
controls and displays have upon flight control evaluations. Extensive knowledge of
frequency domain testing, parameter identification and flight test instrumentation
requirements
STEP Level 2 STEP Level 3 STEP Level 4
• Recognize the pros and cons • Predict results, compare • Defend conclusions based
of different data collection and results to flight test data and upon valid flight test data.
processing techniques. resolve anomalies. • Demonstrate the importance
• Identify and mitigate flight test • Reduce, analyze and explain of credible test and evaluation.
risk. flight test data.
• Identify system problems • Predict the performance and
resulting from flight test and flying qualities of an aircraft
assist in troubleshooting. from first principles
• Understand principles of linear
control theory.
Note: See STEP Forward binder with details of 54 AS Competencies.
81

82. Mapping Training to Competencies
ri tic a s , Effec ts
(FM E An aly sis
Rev ie a dines s
k
Haz a d Report
dels
pla ys
Plan on and
ent
Fli gh gement on
Mana fi guration
cie s
Mana ment/Ris
Boa rd v iew
ning
sis
n
d
tion
gic
a nd
ng
s of
g
e
rfac e
z e,
ws ,
t Safe s s and
ulatio
Effe c Modes an
Elem n trol
Mana onfi gurati
Sy ste l A ir sp ac
lann in
(FM E lys is
Test, Developm
nt
nt
A naly
nitori
c y Lo
ty Mo
fic ien
Link s
reven
Elimin fy , Analy
Plan
ty Re
Airc ra n Rev iew
CA)
d Dis
ning
geme
geme
ents
Airc ra n Rev ie
rd s
ws
ms
ors
A)
s
o
ati
m Inte
t Re
a
ht S im
ion C
li ty
n
ode
ts An
rthine
h Mo
ion P
Evalu
Sy ste
Sen s
ss
ngen
ate a
n
Fideli
m De
Tree
lope
ap P
ol an
ft Co
Data
e
t/Tes
na
A ss e
ire m
ft C
re M
re
Des ig
Miss
Identi
Healt
In-fli g
Nati o
Miss
&
Env e
Airwo
Des ig
Conti
Fail u
Mish
Fault
Sy ste
Contr
Hig h
and C
Fli gh
O ps
Training
Requ
Fail u
Ris k
Hours
# Source Training Course Course Code 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
1 APPEL Introduction to Aeronautics APPEL-I-AERO 32
Project Review Processes and
2 APPEL
Strategies
APPEL-PRPS 40 2 2 2
3 APPEL Assessing Project Performance APPEL-APP 16
Performance Based-Statement of
4 APPEL APPEL-PBSOW 16
Work
Management of Space Technology
5 APPEL APPEL-MSTP 24
Programs
1 OSMA Probability and Statistics Basics I N/A 8
2 OSMA Basic Fault Tree Analysis I N/A 8 2
3 OSMA System Safety I N/A 8
Probabilistic Risk Assessment
4 OSMA N/A 28
Methods (PRAM)
5 OSMA Probability and Statistics Basics II N/A 24
6 OSMA Basic Fault Tree Analysis II N/A 16
Application of Bayesian Probability
7 OSMA Inference to Risk and Reliability N/A 24
fS S SMA-SAFE-
1 NSTC System Safety Engineering 36
NSTC-0040
SMA-SAFE-
2 NSTC System Safety in Acquisition 20
NSTC-0023
Introduction to Jet Engines -- A
1 Practical Aeronautics N/A 16
Practical Perspective
Aeronautics for Scientist and
2 Practical Aeronautics N/A 40
Engineers
National Test Pilots Test Management/Operational Test
1
School and Evaluation
T&E 4002 40 2 2
National Test Pilots
2 Performance Flight Testing I T&E 4101 40
School
National Test Pilots
3 Performance Flight Testing II T&E 4102 40
School
National Test Pilots
4 Flying Qualities Flight Testing I T&E 4103 40
School
National Test Pilots
5 Flying Qualities Flight Testing II T&E 4104 40
School
82

83. Current Status
Aviation Safety Levels 2-4
Discipline/
Training
Course Type Needs Modification New Development
Available
(20 Courses)
Instructor-Led
100% 8% 0%
Courses
Online Courses 0% 0% 0%
Total 100% 8% 0%
Hours 374 31 0

84. Course of Study
• The Aviation Safety
Course of Study includes
a list of
– 16 Courses
– Readings & Resources
– OJT Experiences
– Lifelong Learning
Activities
84

85. Abstract
Presentation Title: Improving Project Success through Technical Excellence
Synopsis: This presentation discusses NASA’s approach to developing Technical Excellence (TE) in Safety
and Mission Assurance (SMA) and addresses how TE enhances safety and mission success for
NASA’s projects.
Abstract: NASA performs high-risk functions in the process of achieving its goals and objectives and
SMA personnel provide valuable risk mitigation products and services to projects that enhance
safety and mission success. The NASA Safety Center (NSC) has been charged with developing an
Agency-wide SMA Technical Excellence Program (STEP). STEP is a structured professional
development program that develops increasing levels of competencies in various SMA disciplines.
STEP also has implications that are of interest to the project management and system engineering
communities and may serve as a model to enhance the technical excellence of those communities
as well. For example, STEP will give projects increased confidence that various SMA products, such
as Hazard Reports, Fault Trees, Risk Assessments, etc., are of the highest quality and caliber
achievable. Further, projects will be assured that SMA products are integrated and treated
holistically as part of the System Engineering and Project Management life-cycles. The tools and
methods to deliver focused and effective professional training may be extended to meet other
Agency training needs and to enhance project communications. In these ways, this presentation
will convey to the audience that STEP is truly a window of opportunity to improve the state-of-SMA
in the Agency and to provide a direct benefit to the Agency’s Projects. 85

86. Biographies
Name: John D. Marinaro
Title: Technical Excellence Director
Company/Organization: NASA Safety Center
Voice: 440-962-3092 Fax: 440-962-3110 E-mail: john.d.marinaro@nasa.gov
Mr. John D. Marinaro is the Director of Technical Excellence at the NASA Safety Center, the former Executive Director of the
NASA Aerospace Safety Advisory Panel, NASA Independent Verification and Validation Facility Chief Engineer, and a retired
Army Helicopter Pilot and Aviation Officer. For more than 25 years, Mr. Marinaro has held management positions devoted
to fields such as: aerospace and electrical engineering; safety management, particularly in NASA operations; software
development and validation; airplane and helicopter flight operations; and other project management responsibilities. His
career has included service with NASA, the Federal Aviation Administration, the Naval Air Systems Command, and a former
active and reserve member of the United States Marine Corps, Navy Reserves, and Army National Guard.
Name: Mark M. Kowaleski
Title: System Safety Discipline Lead
Company/Organization: NASA Safety Center
Voice: 440-962-3044 Fax: 440-962-3110 E-mail: mark.m.kowaleski@nasa.gov
Mr. Mark M. Kowaleski has 21 years of experience in space systems with NASA, including spacecraft, instruments, flight
operations, ground systems, Space Shuttle, and System Safety. Mark has worked as a satellite operations manager, as a
system engineer, project manager, Space Launch Initiative Safety Manager, Space Shuttle Safety Manager, and as the
Aerospace Safety Advisory Panel’s Executive Director. Mark served as the Executive Secretary for the NASA Return-to-Flight
team, served on the Space Shuttle Crew Survivability Study, and was a member of the STS-114 Gap Filler close-call
investigation. Mark is presently working at the NASA Safety Center, being responsible for defining and implementing NASA’s
System Safety Technical Excellence Program. 86