Published on

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide


  1. 1. National Aeronautics and Space Administration Improving Project Success through Technical ExcellenceFebruary 9 – 10, 2010 John D. Marinaro Mark M. Kowaleski 1 Used with Permission
  2. 2. Agenda• SMA Technical Excellence Program (25 min.) – NSC Background – Purpose of STEP – How the STEP Curriculum was Developed – STEP Curriculum Elements – Extension to other Agency training needs and risk reduction – STEP Level 1 Roll-out and Metrics – STEP Levels 2 – 4 Planning• System Safety Discipline Overview (20 min.) – System Safety & Program/Project Interactions – System Safety Competencies – System Safety Lifecycle Products and Services – Community of Practice – System Safety State-of-the-Discipline – System Safety and Program/Project Risk Reduction• Q&A (10 min.)• Back-up Materials – SMA Discipline Curriculums 2
  3. 3. SMA TechnicalExcellence Program 3
  4. 4. NASA SAFETY CENTEROverviewNSC develops SMA personnel, processes, and tools needed forthe safe achievement of NASA’s strategic goals through: Establishing a learning Benchmarking from the best environment and bringing their best practices into our workplace Learning from our mishaps Ensuring that we are informed risk-takers Managing the routine risks of Preserving our resources for the workplace effectively the NASA mission 4 2/18/2010
  5. 5. NASA Safety CenterFOUR AREAS OF SUPPORT 5 2/18/2010
  6. 6. THE PEOPLE OF THE NSC NASA Safety Center Alan H. Phillips Rafael Sanabria Dennie Gonia Sandra L. Hardy Charlene L. Anderson Director Deputy Director Secretary Business Manager Administrative Officer John Marinaro Karen Meinert Cheevon (Mi-Mi) Lau Kenneth O’Connor OCE Technical Chief, GRC NESC Technical Knowledge Management Audits & Assessments Director Mishap Investigation Liaison Excellence Director OSMA OCIO Liaison Discipline Systems Manager Support Manager Fellows Kerry Remp Diane Mark Kowaleski Homayoon Carolyn Van Sue Otero Sallie Keith Dr. Jeffrey Dawson Michael Milbert Mark George COTR Chapman Systems Safety Dezfuli Drei IRIS COTR Information Data Analysis & Analyst MIS Administrative Support Curriculum Technical Systems Safety ST Enterprise Dissemination Trending Analyst IRIS Development Support/ Architecture EX3 Discipline Manager EDUCATE Dr. Natesan Alphaport Fayssal Safie Art Lee Michael Dodson Jambulingam Reliability & Institutional/ Facility Mishap Investigator Reliability & Maintainability ST Operational Audit Manager Eastern Region (GRC, GSFC, Maintainability Brian Follen HQ HQ, KSC, LaRC ) PBMACOTR Robert Cross NASA Safety Analyst PBMA Training Center ARES Ray Kacmar Brian Hughitt Quality Paul Boldon Quality Kristie French Engineering Institutional/ Engineering ST Mishap Investigator Brian Jackson Programmatic Support Auditor Audit Manager HQ Southern Region (JSC, MFSC, SSC) Vacant Software Assurance Audit support Martha from the field Wetherholt centers, other HQs Dennis Morehouse Software Brad Kaustinen offices/SSCs Mishap Investigator Assurance ST NCAS COTR Western Region Vacant NCAS (ARC, DFRC, JPL) Operational Aviation Honeywell Safety Agency Range Israel Greenfeld Safety Efforts IRIS Requirements Liaison SARD/KSC Steve Lilley with MIBs/MIWG MIB Technical Report Reviews & Corrective 6 Action Tracking 2/18/2010
  7. 7. Safety & Mission AssuranceTechnical Excellence• Within NASA, SMA professionals provide a vital role in support of NASA’s design and operations focus.• SMA advises mission specialists on matters related to risk, safety, and mission success.• SMA encompasses all Agency safety, reliability, maintainability, and quality engineering and assurance activities. 7
  8. 8. STEP was Designed to be…1. …a way to assist the Agency (specifically SMA) as it transitions from an operational focus back to a design focus. 8
  9. 9. STEP was Designed to be…2. …both a source of knowledge and an ongoing reference tool in providing SMA professionals with structured, advanced SMA educational opportunities in support of that design focus. 9
  10. 10. STEP Curriculum DevelopmentTechnical Discipline Teams developed a competency-driven curriculum structure for each discipline thatincluded Education, Training & Lifelong Learning. Job of SMA SMA Competency SMA Products Professional Disciplines Wheel & Services EDUCATION TRAINING LIFELONG LEARNING 10
  11. 11. STEP Curriculum• The STEP Curriculum is delivered across four levels with increasing specialization. 3 Intermediate 4 Advanced 2 Introductory Discipline Discipline 1 Introduction Discipline Concepts or Concepts or to STEP Concepts Specialty Specialty [Doer] [Team Leader] [Expert] Generalized Specialized 11
  12. 12. A Typical STEP Program CommitmentEstimated Hours per STEP Level STEP Training Elements Level 1 Level 2 Level 3 Level 4Core Training 3 24 40 N/ADiscipline Training 18 100 137 137Domain Training (by others) 4 40 TBD TBDTotal Course Hours 25 164 177 (+TBD) 137 (+TBD)OJT N/A 200 400 400Years of Experience N/A 2 4 8Test and/or Peer Board N/A N/A Yes YesCEU’s N/A 16 24 32 12
  13. 13. STEP Training Elements 13
  14. 14. STEP’s Benefit toNASA’s Programs/Projects• High confidence in SMA’s competence and capability.• High confidence in SMA’s products & services.• Improved technical program/project performance.• Risk Reduction: – Increased success likelihood. – Improved risk characterization. – Improved risk-based decision-making.• Serves as showcase for engineering excellence.• Complements Agency training initiatives: – Academy of Program/Project and Engineering Leadership (APPEL). – System Engineering Leadership Development Program (SELDP). – Senior Executive Service Career Development Program (SES CDP). 14
  15. 15. STEP Level 1Curriculum• One curriculum across all six disciplines• Total of 25 contact hours – Each course is one hour on average• Topics covered: – SMA Implementation – Core – Domain – Disciplines – Case Studies• On-demand and self-paced 15
  16. 16. Level 1 Initiatives• Agency-wide roll-out• Online Courses• Handbook• Quick-start guide• Course DVD• Brochures 16
  17. 17. Level 1 ParticipationPost Rollout• Over 1,300 individuals have been assigned or registered for the curricula• 361 graduates as of Nov 1• 26% of all enrollees have successfully completed Level 1 – two months after rollout!• Feedback: nearly 70% rate the program as “high value” 17
  18. 18. STEP Levels 2 - 4Curriculum Features• Unlike Level 1’s unified curriculum, Levels 2 - 4 provide a unique curriculum for each discipline• Participants will choose a discipline and receive a Course of Study to guide them through each level.• Levels 2-4 allow for electives & discipline specialization• The course of study for each discipline includes a variety of course types: – Online/e-learning via SATERN – Instructor-led workshops – APPEL, NSTC and SARD courses – Commercially-available courses – Courses created by and for the NASA Safety Center Levels 2 – 4 also encourage OJT, Enrichment Experiences, Career Training & Continuing Education. 18
  19. 19. Competency Wheel• The planning process started with the Competency Wheel for each discipline, from Level 1.• The Competency Wheel lists all the competencies an SMA professional needs to perform. – The inner ring lists the major competencies. – The outer ring provides a further level of detail to each of the major competencies. 19
  20. 20. Performance ObjectivesFor each competency, detailed objectives were written to describe exactly whatperformance is expected at each level. (Typical performance objectives shown) Level 2 Level 3 Level 4 • Recognize QE Quality • Perform QE Quality • Evaluate Quality Management functions. Management functions. Management functions • Recognize Quality • Apply Quality and Systems. Management as one of Management principles four (4) second tier to develop Quality competencies of the Management Systems Perform Quality (QMS). Management & Planning Competency. • Recognize the six (6) third tier competencies in Quality Management. 20
  21. 21. SMA Work Products and ServicesSMA Work Products and Services were identified for eachdiscipline. These are the things SMA Professionalsactually do on the job. 21
  22. 22. Course of StudyThe Result:• A complete curriculum plan – a Course of Study – was developed for each discipline, by level• The course of study includes a list of – Courses – Readings & Resources – OJT Experiences – Lifelong Learning Activities 22
  23. 23. STEP – A Complete Curriculum• Overall, STEP will be a complete curriculum tailored specifically for the SMA Professional.• STEP is specifically designed to address the technical needs of NASA’s programs and projects. 23
  24. 24. System Safety DisciplineOVERVIEW 24
  25. 25. Technical Excellence does pay off……and you do get what you pay for. Coach 1st ClassUS Airways Flight # 1549 25
  26. 26. SS Discipline – Where are we today?• Most SS professional’s training is ad hoc training and professional development (not many opportunities for SMA degrees at universities).• We “learn as we go” and heavy emphasis on unstructured OJT.• We do a lot of “reviewing” but not a lot of actually “doing” and this is perceived as a negative thing.• Don’t address new hires’ training needs.• Don’t have mentoring for entry or mid-level employees.• Training needs to apply to “real” work vs. perceived or ideal work that we think we should be doing.• Don’t have structured OJT with a clear understanding of what it is that is expected to be learned.• Training does not address softer SS discipline areas like resource estimation, safety in contracts, and SS integration with system and program lifecycle.• Training is presently focused on qualitative methods and we would benefit as a community from more analytical-based training.26
  27. 27. What Do System Safety Engineers Do?Fundamentally, NASA System Safety Engineers, at some point intheir careers, during any phase of the system life cycle, do anycombination of three basic activities:1. Assess any situation with safety consequences.2. Manage safety in acquisitions.3. Manage, leverage, and interact with safety organizations and NASA programs.From this premise, all competencies and products and services arederived.
  28. 28. System Safety’s Interactionswith Programs/Projects System Safety Products (E.g., Hazard SS products and SS Engineer applies Analysis, FTA, PRA, CRM, etc.) Framework competencies to influences develop SS products decisions 2 3 SS Engineer Decision Maker Competencies define 1 how SS Engineers do SS products support milestones their job at NASA 4 in the Program lifecycle Decisions influence the 5 outcome of Programs System Safety monitors lifecycle Safety Performance 6
  29. 29. Overall System Safety FrameworkDraft 0 of Overview of SystemSafety – Not to be DistributedOutside of SS Working Group
  30. 30. System Safety Student Body Estimates System Safety STEP participants Civil Servant SMA & SMAs Support Contractor SMA - Civil SMA - Center Servant Contractor Total GRC 14 6 20 JSC 110 233 343 MSFC 32 35 67 LaRC* 10 20 30 DFRC 2 0 2 JPL n/a 10 10 ARC* 5 10 ? SSC 4 10 14 GSFC 31 73 104 KSC 28 25 53 HQ/NSC* 10 2 12 * Estimated Total 246 424 670 30
  31. 31. SS Competency WheelA SMA System Safetyprofessional needs to possessthese 34 competencies:Major CompetenciesTechnical• System Safety Mathematical Skills• System Safety Analytical Methods• System Safety RationaleManagerial• System Safety in Acquisition Management• System Safety in Organizational Management 31
  32. 32. Competency Performance Objectivesand Training, OJT, and Reference Material Allocations(1 of 34 competencies)Competency: Hazard AnalysisDescription: The analytical method and approach used to identify, mitigate, and managesafety hazards found in systems.Performed by: System Safety EngineerSTEP Level 2 STEP Level 3 STEP Level 4• Understand and explain the • Prepare a quantitative HA on • Monitor the performance of basic engineering principles, a NASA space subsystem or HA management systems and steps, and elements of a system. processes used by the Hazard Analysis (HA). • Prepare a Hazard Report (HR) organization.• Identify and explain NASA with all required content. • Conduct analysis of policy and procedures • Identify, document, and supporting HA data for trends, governing the conduct of HA. monitor requirements indicators, and pitfalls.• Prepare a qualitative hazard associated with a HR. • Interpret, apply, communicate, analysis on a simple, isolated • Monitor the implementation and monitor requirements NASA space subsystem. and performance of Hazard related to HA for NASA• Document and present a HA Controls throughout the programs and projects. to a senior official. lifecycle. • Influence policy or procedures• Understand and explain the • Evaluate HR/HA prepared by to improve the conduct of relationships of HA to fault contractors. HA/HR. tree analysis and the • Present a HA/HR to decision relationship of the HA with makers. other SMA disciplines. Training and/or Reference and/or32 OJT equals COMPETENCY
  33. 33. Core and AdvancedSystem Safety ProductsCore System Safety Products & Services• Are products & services that all NASA’s System Safety Engineers are expected to be able to produce at any time. –E.g., Create a Hazard Analysis.• Are generally independent of specific job location, project, or job function.Advanced System Safety Products & Services• Are products & services that NASA’s System Safety Engineers are expected to be able to produce if their job assignments demand it or as their careers progress and their responsibilities escalate. –E.g., Support a SEB for a major procurement/acquisition activity.• Are generally dependent on specific job location, project, and job function. 33
  34. 34. SS Products & ServicesDistributed across LevelsSTEP Level 2 STEP Level 3 STEP Level 4• Conduct Hazard Analysis • Conduct Hazard Analysis • Conduct Hazard Analysis• Conduct Fault Tree Analysis • Conduct Fault Tree Analysis • Conduct Fault Tree Analysis• Prepare Safety Rationale • Prepare Safety Rationale • Prepare Safety Rationale• Prepare Safety Data Package • Prepare Safety Data Package • Prepare Safety Data Package• Support Anomaly Resolution • Support Anomaly Resolution • Support Anomaly Resolution• Provide Decision Maker Input • Provide Decision Maker Input • Provide Decision Maker Input• Communicate Safety • Communicate Safety • Communicate Safety Requirements Requirements Requirements• Conduct Surveillance & • Conduct Surveillance & • Conduct Surveillance & Compliance Verification Compliance Verification Compliance Verification• Safety Performance Metrics • Safety Performance Metrics • Safety Performance Metrics Monitoring Monitoring Monitoring ---------------------------------------------- --------------------------------------------- • Safety Acquisition Management • Safety Acquisition Management • Probabilistic Risk Assessment • Probabilistic Risk Assessment • Risk Analysis • Risk Analysis • Integrated Safety Data Package • Integrated Safety Data Package • Support Certification of Flight • Support Certification of Flight Readiness Readiness ---------------------------------------------- Green = Core • Safety Organizational Management Red = Advanced • Support Lessons Learned • Manage Policy/Standards & Requirements • Support Special Studies • Support Mishap/Close-call investigations 34 • Support Audits & Assessments
  35. 35. SS Community of PracticeQ: What is the SS COP?A: Anything associated with the conduct of System Safety activities that folks need to know, share, keep, or access.Examples:1. Agency “go-to” SS Website.2. Mass email notification list.3. Guest Lecture Series.4. Webcasts, Podcasts, etc.5. Blogs/Papers with ratings.6. Clearing House Concept Tools, Documentation, Reference, Resources, Conferences, Speakers, 3 Lessons Learned, Training, Etc. 5 35
  36. 36. SS Website 36
  37. 37. STEP enables Program/ProjectRisk Reduction through…• Providing a competent and highly capable System Safety workforce.• Providing predictable, consistent, and value-added System Safety products & services to programs/projects.• Analytical tools proliferation (e.g., SAPHIRE/QRAS).• Enabling risk-informed decision-making (RIDM).• Providing safety throughout the entire program acquisition lifecycle.• Providing a forum for a community of practice to encourage lessons learned and communications.• Proactive human capital management. 37
  38. 38. State-of-the-Discipline (SOD)Asks: “What is the health status of the SS Discipline?” and “What are we doing about it?”Sample SOD Metrics• STEP Qualification distribution.• Training metrics/rates/distribution.• Integrity of SS products over time.• Analytical tools proliferation.• Proactive Human Capital Management• Resources, staffing, work-load.• Rate of Hazard Controls violations.• Safety-related Technology Readiness. 38
  39. 39. State-of-the-Discipline Example Agency CMS Competency Level Agency CMS Competency Level 1 Level 2 Level 3 Level 4 Total "SAFENG" 181 321 288 197 987 "SYSTEMENG" 325 742 930 503 2500 506 1063 1218 700 3487 Combined CMS Competencies Levels Level 1 Level 2 Level 3 Level 4 Total Combined SAFENG and 17 20 21 26 84 SYSTEMENG 3% 2% 2% 4% 2%Metric: Proactive Human Capital Management Desired Desired SOD metric• What is the competency improvement relationship between System Safety and System Engineering at NASA?• Data supports observation that SMA and SE could be better integrated.• Is an overall 2% cross competency relationship acceptable?• What are we doing about it? 39
  40. 40. STEPPING FORWARD• Providing Education for NASA’s Future… … one STEP at a time. 40
  41. 41. Back-up 41
  42. 42. SMA Technical Excellence ProgramCURRICULUM BY DISCIPLINE 42
  44. 44. SA Technical Discipline Team • NSC TDT Lead – Karen Meinert • NSC SSC – Jennifer Jones • HQ – M. Wetherholt, M. Bodeau • ARC – C. Chow, G. DelCarmen • DFRC – D. Tran • GRC – T. Luu • JPL – S. Flanagan, J. Wilf • GSFC- S. Harris • JSC – R. Hugger • IV&V – L. Montgomery • KSC – D. Victor • LaRC – L. Johnson • MSFC – R. Strickland • SSC – C. Fallo
  45. 45. SA Competency Wheel• Software Assurance is comprised of the following competencies: – Software Quality – Software Safety – Software Reliability – Software Verification & Validation – Software Independent Verification & Validation – Software Assurance Planning & Management 45
  46. 46. Competencies by Level Competency: Software Safety Description: Knowledge, skills and abilities in identifying, analyzing, tracking, mitigating and controlling software hazards and hazardous functions to ensure safer operation within a system. Activities performed ensure that safety is designed into the software. STEP Level 2 STEP Level 3 STEP Level 4 Performance Expectation Performance Expectation Performance Expectation: Observe and participate with supervision Under general awareness of supervisor, o Determines criteria for safety critical in software safety activities throughout perform software safety activities software classification the lifecycle including: throughout the lifecycle including: o Develop/review safety standards o Identify safety critical software o Perform software safety litmus test o Identify safety critical software o Complete hazard reports with Systems o Assist in reviewing: system PHA o Review system hazard analyses Engineering requirements, designs, code, analyses, o Perform or review software safety o Develop software hazard reports change requests, testing documents, analyses o Assure hazards verifications are followed planning documents, lessons learned, and o Verify compliance to appropriate o Assess adequacy of safety requirements traceability of safety requirements safety standards and software safety documentation o Review software safety plans o Plan and manage software safety efforts o Participate in peer reviews o Assure V&V requirements are met o Perform software safety activities o Attend/observe milestone reviews o Review/assure software safety throughout the lifecycle, including: and safety reviews requirements are documented and o Develop/review/assess o Assist in assurance of CM of safety tracked throughout the life cycle requirements, designs, code, and requirements o Participate in reviews of designs and test documents; o Assist in software acquisition code o Assess development and test activities and evaluation of o Review test documentation environments; contractor deliverables. o Identify potential risk/safety issues o Determine adequacy of V&V and verify issues are identified and activities; tracked to resolution (…continued) (…continued) 46 This is one of six competencies for this discipline
  47. 47. Mapping Training to Competencies 47
  48. 48. Current Status Software Assurance Levels 2-4 Discipline/ Training Course Type Needs Modification New Development Available (30 Courses) Instructor-Led 93% 7% 17% Courses Online Courses 7% .5% 1% Total 100% 7.5% 18% Hours 374 28 68
  49. 49. Course of Study• The Course of Study includes a list of – 30 Courses – Readings & Resources – OJT Experiences – Lifelong Learning Activities 49
  51. 51. QE Technical Discipline Team • NSC TDT Lead – Ray Kacmar • NSC SSC – Jerome Anderson • HQ – Brian Hughitt • ARC - Steve Jara, Damon Flansburg • DFRC - Gary May, Kevin Reilly • JPL - Thomas Smith, Tony Gutierrez • GRC - Bob Makovec • JSC - Dave Dyer • GSFC - Louis Thomas, Mike Kelly • KSC - Robert Ellison • LaRC - Elijah Kent, Charlie Greenhalgh • MSFC - Ken Crane • SSC - Tim White • NCAS - Charles Dionne 51
  52. 52. QE Competency WheelAn SMA Quality Engineeringprofessional needs to possessthese major competencies: • Perform Quality Assurance Management & Planning • Assure Adequate Design & Development Controls • Conduct Procurement QE • Define and Implement Production Material & Process Controls • Define and Perform Inspection Testing & Acceptance Activities • Implement Problem Reporting & Corrective Action • Conduct Quality Evaluations & Data Analysis The inner ring lists the major competencies. The outer ring provides a further level of detail to each of the major competencies. 52
  53. 53. Competencies by Level Competency: Quality Management Description: Recognize, Perform and Evaluate Quality Management Functions STEP Level 2 STEP Level 3 STEP Level 4 • Recognize QE Quality Management • Perform QE Quality Management • Evaluate Quality Management functions. functions. Functions and Systems. • Recognize Quality Management is one • Apply Quality Management principles • Perform a complete quality of four (4) second tier competencies of to develop Quality Management management system evaluation to the Perform Quality Management & Systems (QMS). ensure compliance to all applicable Planning Competency. • Based on a review of the quality QMS requirements defined in the • Recognize the six (6) third tier requirements, plan the QMS (to include planned evaluation. competencies in Quality Management. objectives, scope, schedule, and • Become a Certified Quality Engineer • Understand the concepts of quality technical expertise). (Elective). management and quality management • Document the results of the QMS, to • Evaluate human factor issues and systems. include any findings, observations, or human error prevention techniques. • Understand the concepts of strategic assessments of risk. (Elective). planning for quality. • Perform strategic planning (Elective). • Provide instruction to operators and/or • Understand communication • Develop effective communication inspectors in the methods of mechanisms. mechanisms (Elective). product/service verification, throughout • Recognize the factors that influence • Identify human factor issues and the value-stream. the work environment. human error prevention techniques • Recognize NASA polices on Human (Elective). Factors. • Recognize the importance of NASA’s training & certification requirements related to quality and identify related training. • Recognize related sources of information. This is 1of 48 competencies for this discipline
  54. 54. Mapping Training to Competencies (CIL ems Lis d nt te ning t and ) an sis eme As su Quality d De si Adequa ent ent t Revie puts an lysis ects s Inte g Qua lity sis Analy Critic (FMEA r t rols t stem Qua ment awa rd/P ost Orde agem agem en es & r e ana g en ring ns & Stan nship Man r ation ool s nal y ent ent nts ilu Man ce ent Eff l opm s ed Man ion agem es n gn & al S y s ur n Fa Tree , Proc t ran Asse l agem agem r atio n In ensio n of inee li ty M r ime ings hase orm ss m M an Man ure nic a rd A dard duc li ty T al It ranc ram ws re lity ui sit kma rd S wa figu k Ba w As su De ve Desig Desig Desig Critic Training Pla n Pr e-a Fault Draw Expe Haza rev ie Tech Perf Pr og ) Con Con E ng Data Purc Risk Mod Tole Co n Wor Qua Qua An a Dim Acq Ris Training Course HoursQualit y Management Overview 3 2 2Qualit y Management Syst ems 8 3Workforce Development , T raining& Cert ificat ion 0.5P rogram Qualit y Int egrat ionOverview 1 2P rogram Qualit y Int egrat ion 4 3Qualit y Assurance P lans 3 2Lessons Learned 1 2T echnical Assessment Overview 1.5 2T echnical Assessment 4 3Audit s 0.5 2AS 9100 Lead Audit or Class 40 4Qualit y Assurance Surveillance P lan 1 2Qualit y T ools Overview 1.5 3Qualit y T ools 5 4Qualit y T ools - NASA Addendum 2 4Lean Six Sigma 8 4Qualit y Funct ion Deployment 6 4Design Input s and Reviews Overview 4 2 2Design Input s and Reviews 8 3 3Mist ake-P roofing Execut iveOverview 4 2Mist ake P roofing 16 3Workmanship St andards Overview 1 2Design of Experiment s Execut iveOverview 4 3Design of Experiment s 32 4Drawings, Dimensions, T olerancesOverview 1 2Drawings, Dimensions &T olerancing - S-44 40 3Failure Modes Effect s Analysis &Crit ical It ems List (SAT ERN) SMA-017-01 1 2Syst em Safet y I 8 3 3Cont inuous Risk Management -AP P EL - CRM 24 4Configurat ion Management 4 2Funct ional Configurat ion Audit /P hysical Configurat ion Audit 0.5 2Dat a Management 2 2P art s Cont rol Overview 1P rocurement Qualit y Overview 8 2 2 2 2NASA Let t er of Delegat ion 2 54
  55. 55. Current Status Quality Engineering Levels 2-4 Discipline/ Training Course Type Needs Modification New Development Available (66 Courses) Instructor-Led 58% 0% 0% Courses Online Courses 18% 14% 10% Total 76% 14% 10% Hours 282.5 53.5 38
  56. 56. Course of Study• The Quality Engineering Course of Study includes a list of – 66 Courses – Readings & Resources – OJT Experiences – Lifelong Learning Activities 56
  58. 58. R & M Technical Discipline Team • NSC TDT Lead – Nat Jambulingam • NSC SSC – Maria Havenhill, Dan Deans, Bob Loomis • HQ – Frank Groen • HQ Tech Fellow – Fayssal Safie • ARC – N/A • DFRC – N/A • GRC - Edward Zampino • GSFC - Tony Diventi, Matthew Samuel • JPL - Chau Brown • JSC - Samadar Roshan-Zamir • KSC - Chi Yeh, Tim Adams • LaRC – Pete Parker • MSFC - Richard Stutts • SSC - Mike Rewis 58
  59. 59. R & M Competency WheelA Reliability & Maintainabilityprofessional needs to possessthese major competencies: • Reliability Management • R&M in Acquisition • R&M Methods in Design & Development • Reliability Testing and Demonstration • Use of Database and Reports for R&M • Mathematical Tools The inner ring lists the major competencies. The outer ring provides a further level of detail to each of the major competencies. 59
  60. 60. Competencies by LevelCompetency: FMEA/CIL, FMECADescription: Systematically identifies system’s potential failure modes, and evaluatestheir causes and effects on the system’s ability to perform its functions.STEP Level 2 STEP Level 3 STEP Level 4• Define failure modes, integration • Interpret and apply FMEA or FMECA • Perform/evaluate FMEA or FMECA on characteristics, propagation program requirements. increasingly complex systems.• Describe the contents of FMEAs, Critical • Perform a FMEA or FMECA on assigned • Accurately & thoroughly document Items List (CIL) and FMECAs. system/subsystem/component. FMEA and Critical Item List analysis• Describe how and why Critical Items are • Evaluate a FMEA and Critical Item List performed & describe the implications created out of the FMEA analyses. analysis & coordinate comments with the conclusions may have.• Locate and read program requirements. the subsystem engineer. • Explain nuances not readily evident to• Describe the FMEA, Critical items and • Assess compliance to the reliability less experienced personnel. FMECA process flow. design requirements verified by the • Assess compliance of submitted• Describe the relationship of CIL to FMEA or FMECA. FMEA/CIL, FMECA to requirements hazard reports. • Identify common cause failure modes in document.• Identify simple generic failure modes addition to system failure modes. • Determine compliance to Reliability and postulate the effects based on an • Identify failure modes that are specific design requirements verified by the understanding of component to the design configuration under FMEA or FMECA. functionality. analysis. • Lead a team in developing a FMEA and• Identify simple failure mode effects at • Trace the effects of failure modes to a Critical Item List. the system level the system level. • Coordinate the team activity on a • Coordinate with design engineering to complex system so that the interfaces develop retention rationale for the between subsystems have coverage critical items. and the Critical Item retention rationale • For those engineers involved in meets all the requirements of the diagnostics, describe the relationship Program FMEA/CIL or FMECA between FMEA, FMECA and fault requirements document. detection and isolation. This is one of 41 competencies for this discipline 60
  61. 61. Mapping Training to Competencies Competencies R&M into Govern contracts, ment- Potential Source SOWs, Industr Comme specifications, Defining and y Data rcial deliverables, controlling Parts NASA Exchan Materia and R&M R&M Managing and award R&M Reliabil Data and Lesson ge l Contra other Uncert Sensiti Decisio Boolea Progra Starate Evaluating the structure, and requirements ity Fault Root Probab Parts Reliabil Maintai Availab Mecha Softwa Analysi Materia Reliabil Design Reliabil s NASA Progra Review ctors Givern ainty vity n n Data Markov R&M m gic elements of a other for Block Tree FMEA/ Cause ilistic Risk Stress ity nability ility nical re Human s and l ity Test of ity Test Produc Reliabil NASA Learne Internal m Board Test ment Training Probab Statisti Analysi Analysi Analysi Algebr Analysi Analysi Planni Manag Manag contractors acquisition program/projec Diagra Analysi CIL, Analysi Analysi Analysi Analysi Modeli Modeli Modeli Reliabil Reliabil Reliabil Trendi evaluat Planni Experi Develo t Life ity PRACA d Report (GIDEP Report Report agenci Training Courses Hours ility cs s s s a s s ng ement ement R&M elements t acquisition m s FMECA s s s s ng ng ng ity ity ity ng ion ng ment pment Testing Growth System System s ) s s es data 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 411 OSMA+NSC R&M Principles 24 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 22 NSC Design for Reliability 32 2 2 2 2 2 2 23 NSC Design for Maintainability 32 2 2 2 2 2 2 2 NSC Reliability, Availability, and Maintainability (RAM) Planning for4 Programs and Projects 14 2 2 25 Commercial Data Collection and Analysis I* 16 3 3 3 3 3 3 3 3 NSC Reliability, Availability, and Maintainability Modeling (including6 success tree analysis) I 16 3 3 3 GSFC/JPL Parts and Materials Assessment (EEE and Mechanical), Parts Stress/Derating and also incl. Thermal Analysis and Structural7 Analysis I 16 3 3 NSC8 Maintainability and Supportability Analysis and Integration* 16 39 SATERN/JPL FMEA/CIL, FMECA 16 310 NSC R&M Testing and Demonstration I* 16 3 3 3 311 Commercial Reliability Growth 16 312 NSC Dependency and Common Cause 16 3 3 3 JPL13 Single Event Upset/Effects Analysis, Radiation Effects I 16 3 3 3 314 Commercial Sneak Circuit Analysis 16 3 3 3 OSMA15 Root Cause Investigation and Corrective Action/Failure Analysis 8 316 NSC GIDEP 8 3 317 Commercial Certified Reliability Engineer (CRE) Refresher 1618 SS Probabilistic Risk Assessment (SS) 24 3 3 319 QE Problem Reporting (QE) 4 3 3 320 SS Hazard Analysis (SS) 8 3 3 3 321 SS Fault Tree Analysis (SS) 24 322 NSC Data Collection and Analysis* II 16 4 4 4 4 4 4 4 4 NSC Reliability, Availability, and Maintainability Modeling (including23 success tree analysis) II 16 4 4 4 NSC Parts and Materials Assessment (EEE and Mechanical), Parts Stress/Derating and also incl. Thermal Analysis and Structural24 Analysis II 8 4 425 NSC R&M Testing and Demonstration II* 16 4 4 4 426 Commercial Reliability Growth 16 427 SA Software Reliability (SA) 16 428 Commercial Physics of Failure 16 4 4 4 4 4 JPL29 Single Event Upset/Effects Analysis, Radiation Effects II 16 4 4 4 430 JPL Worst Case Analysis 24 4 4 Commercial (NRC)31 Human Reliability 28 3 NSC/ Commercial32 Design of Experiments 40 4 Cross Discipline Course Probability and Statistics Basics : Source Document: NASA SP-** 2009-659 (SS) 3 3 TOTAL COURSES ADDRESSING COMPETENCY 2 3 1 1 1 1 5 1 3 3 3 3 3 3 3 1 4 2 1 2 2 8 8 9 4 1 1 3 2 3 1 3 3 5 1 2 2 3 2 2 1 61
  62. 62. Current Status R & M Levels 2-4 Discipline Discipline/ Training Course Type Needs Modification New Development Available (32 Courses) Instructor-Led 19% 0% 15% Courses Online Courses 25% 9% 32% Total 44% 9% 47% Hours 166 32 176 Total Discipline Hours = 374 [302 hours are required; 72 hours are elective]
  63. 63. Course of Study• The R & M Course of Study includes a list of – 32 Courses – Readings & Resources – OJT Experiences – Lifelong Learning Activities 63
  65. 65. SS Technical Discipline Team • NSC TDT Lead – Mark Kowaleski • NSC SSC – Maria Havenhill, Diane Chapman, Jennifer Baumeister, Kathy Kraft • HQ – Homayoon Dezfuli • ARC – N/A • JPL - Kirk Barrow • DFRC - Jonathan Brown • JSC - Gerald Readore, Larry Gregg • GRC - Bill Schoren, Carrie Green • KSC - Ronald Long, Robert Schwader • GSFC - Bo Lewis, Angela Melito • LaRC - John Greco, Jose Caraballo • MSFC - Herb Shivers, Keith Layne • SSC - Robert Gargiulo 65
  66. 66. SS Competency WheelAn SMA System Safety professionalneeds to possess these majorcompetencies:Technical• System Safety Rationale• System Safety Analytical Methods• System Safety Mathematical SkillsManagerial• System Safety in Acquisition Management• System Safety in Organizational Management The inner ring lists the major competencies. The outer ring provides a further level of detail to each of the major competencies. 66
  67. 67. Competencies by LevelCompetency: Hazard AnalysisDescription: The analytical method and approach used to identify, mitigate, manage and communicate safety hazards foundin systems. STEP Level 2 STEP Level 3 STEP Level 4• Understand and explain the basic • Prepare a quantitative HA on a • Assess the performance of HA engineering principles, steps, and NASA space subsystem or system. management systems and elements of a Hazard Analysis • Prepare a Hazard Report (HR) with processes used by the (HA). all required content. organization.• Identify and explain NASA policy • Identify, document, and monitor • Compare actual vs. predicted and procedures governing the requirements associated with a HR. system performance and formulate conduct of HA. • Monitor the implementation and and manage corrective actions, as• Prepare a qualitative hazard performance of Hazard Controls necessary, to ensure efficacy of analysis on a simple, isolated throughout the lifecycle. hazard control. NASA space, ground, and/or test • Evaluate HR/HA prepared by • Conduct analysis of supporting HA hardware/operations subsystem. contractors. data for trends, indicators, and• Document and present a HA to a • Present a HA/HR to decision pitfalls. senior official. makers. • Interpret, apply, communicate, and• Understand and explain the • Utilize anomaly resolution input to monitor requirements related to HA relationships of HA to fault tree evaluate impacts to system-level for NASA programs and projects. analysis and other relevant hazard risk through the use of • Influence policy, procedures, and/or analytical techniques, and the qualitative and/or quantitative tools. requirements documents to relationship of the HA with other • Utilize Fault Tree Analysis (FTA) or improve, standardize and SMA disciplines. an equivalent tool to map hazard encourage the conduct of HA/HR causes and controls to other safety for NASA programs and projects. and reliability data products (e.g. FMEA/CIL). This is 1 of 34 competencies for this discipline. 67