Published on

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide


  1. 1. Risk Informed Design and Test OnNASA’s Constellation Program John V. Turner, PhD Constellation Program Risk Manager Used with permission
  2. 2. Program Goals • NASA identified goals for the CxP related to ISS Support and Lunar Exploration – Intent is to lay groundwork for Mars exploration as well • Exploration Systems Architecture Study conducted to develop exploration systems architecture to support these missions • Constellation program chartered to develop and field this architecturePage 2 NASA CxP John V. Turner, PMC 2009
  3. 3. The Challenge • Develop an architecture that optimally meets goals and objectives, within cost and schedule, and with acceptable safety and mission success risk  Risk Informed Design: aims to support design activities in identifying acceptable and optimal safety  Risk Informed Test: aims to support test activities in identifying ways to best reduce uncertainty and risk (uncover defects in design, manufacturing, and processing prior to IOC)Page 3 NASA CxP John V. Turner, PMC 2009
  4. 4. Risk Timeline – ISS Mission • Risk changes in character, intensity, source over time • Risk prevention and mitigation must be considered in every system and activity across all mission phases • Understanding the integrated implications of system risks is critical to success 8-10 Minutes 180-210 Days! Entry / Ignition Staging MECO Landing / Ground First Stage Ops Second Stage Orbit Ops Docked at ISS Rescue Mission Elapsed Time Crew Ingress A Leading Risk! Timeframes and Intensity are illustrative – not to actual scale.Page 4 NASA CxP John V. Turner, PMC 2009
  5. 5. Sources of Failure • Where do Defects Enter into the flight equipment and operations that result in failure? • Defects can arise through – Actual system design flaws – Inadequate testing to uncover defects – Manufacturing errors – Integration or processing errors – Bad decisions during real time • Note: History indicates that manufacturing, integration and processing are very significant defect sources • Goal: Put in place processes that identify and eliminate defects leading to failure Defect Manufacturing Integration / Source Design Test Operations. Processing Mitigation RI Design RI Test Robust Quality Assurance Mission OperationsPage 5 NASA CxP John V. Turner, PMC 2009
  6. 6. Risk Informed Design (RID) • Probabilistic Requirements to drive risk performance in the design • Loss of Crew (LOC) and Loss of Mission (LOM) risk factored into significant design and planning trades • Risk assessment embedded in Integrated Design Analysis Cycles to inform all key analysis tasks • “Zero Based Design” • Risk Informed Test Plans • Focus additional analysis and test resources on High risk / High Uncertainty areasPage 6 NASA CxP John V. Turner, PMC 2009
  7. 7. RID Approach • Premise: Risk is a design commodity like mass or power • Qualitative and Quantitative risk analyses expose dominant risk contributors and support design and planning trades to assign critical design commodities (mass, volume, power, cost, etc.) • Iterative systems engineering design cycles incorporate risk in trade space and identify design solutions that are risk informed • Risk analysis considers all significant failure types, including: functional, phenomenological, software, human reliability, common cause, and external or environmental events, • Complexity and fidelity of analysis consistent with the available data and information during each design cyclePage 7 NASA CxP John V. Turner, PMC 2009
  8. 8. “Zero Based Design” 1. Early design concepts are defined with minimally required functionality to perform the mission and no redundancy – Focus on implementing “Key Driving Requirements” vs establishing a fully functional, acceptably safe, or highly reliable design. – Risk analyses are performed during this phase to understand the risk vulnerabilities of this “zero based design” (ZBD).Page 8 NASA CxP John V. Turner, PMC 2009
  9. 9. “Zero Based Design” 2. Prioritize design enhancements with a focus on enhanced functionality and LOC risk. – Focus: “Make the design work”, “Make the design safe” – Identify optimal use of design commodities, cost, and schedule to reduce risk – with priority on diversity vs simple redundancy. – Major Premise: Simple redundancy is one option to improve safety and reliability. It is not the only option. It is not always the safest or most cost effective option. – Compare different investment portfolios using FOMs derived from key risk commodities, including LOC risk – Goal: Spend scarce risk mitigation resources (mass, power, volume, cost) most effectively to maximally address riskPage 9 NASA CxP John V. Turner, PMC 2009
  10. 10. “Zero Based Design” 3. Finally, additional enhancements are considered which more fully address functional requirements and focus on reliability and loss of mission (LOM) risk. – A portfolio approach to comparing investments is again used – Ensures that the final design iteration produces a vehicle that better meets functional requirements, safely, reliably, and within budget.Page 10 NASA CxP John V. Turner, PMC 2009
  11. 11. Zero Based Design Summary • “Build-Up” approach from the zero based design to a risk balanced system design, its complexity, and the existence of each system element. – Rationale exists to justify resource allocations such as: mass, power, and assures that affirmative rationale is used for the cost. – Build up approach lessens the likelihood of having to make dramatic design changes later in the design cycle to resolve critical commodity shortfalls and get back “in the box.” • This approach is described in detail in two NESC reports: – “Crew Exploration Vehicle Smart Buyer Design Team Final Report” – “DDT&E Considerations for Safe and Reliable Human Rated Spacecraft Systems”Page 11 NASA CxP John V. Turner, PMC 2009
  12. 12. Results • Program – Original ESAS Loss of Crew (LOC) and Loss of Mission (LOM) requirements were derived using initial architecture trade study that included conceptual design concepts and underestimated certain significant risk drivers – Requirements have been adjusted based on current design and environments, improved analysis and a better understanding of what is challenging yet achievable – CxP architecture currently meeting mission level LOC and LOM requirements • Orion Project – Orion early design conducted prior to inauguration of RID activities – Began RID design cycles in late 2007 – Significant design changes (4X improvement in LOC, 3X in LOM)) – Implemented Apollo 13 Low Power Emergency Return Capability – improvements in safety and mission success while resolving mass challengesPage 12 NASA CxP John V. Turner, PMC 2009
  13. 13. Results • Ares Project − Ares conducted RID design trades early in the DDTE process and incorporated design changes in multiple subsystems − Ares I risk analysis currently projects significant improvement in reliability of previous manned launch systems • Altair Project – Conducted ZBD approach from project initiation, completed LOM and LOC risk buyback design iterations from Zero Based design configuration – Significantly Improved safety and mission success and developed stronger design concept to enter next stage of designPage 13 NASA CxP John V. Turner, PMC 2009
  14. 14. Some Lessons Learned • RID brings designers and analysts together early to evaluate sources of risk, the integrated implications of risks, and the efficacy of different design implementations in maximizing safety and mission success • RID drives designers toward dissimilar or functional redundancy vis traditional redundant system approach – Reduced weight penalty incurred by traditional method • Requirements are met more effectively wrt use of design commodities – design features can be prioritized to determine where reductions are best applied in the event of mass issues • Risk Informed Campaign Analysis provides insight into “program” vs “mission” success as a function of system design issues • Evaluating DRM LOM requires strong understanding of operational flexibility – forces early operations criteria development and operations driven design • Current methods for evaluating Maturity Growth require improvement – Assumed maturity for design analysis – Need better way to address maturity growth and determine early mission riskPage 14 NASA CxP John V. Turner, PMC 2009
  15. 15. Some Lessons Learned • The tools used to model LOC and LOM should evolve from early concept development to verification phase – Simple, historical data driven models early – Conventional Linked Fault Tree / Event Tree models later – Models increase in complexity and fidelity with the design • Application of Qualitative Top Down Functional Modeling to identify significant hazards that should drive both the Integrated Hazard Analysis and PRA Master Logic Diagrams • Consistency and Visibility are Critical! – Models, – Data – Methods – Tools • Three types of risk analysis to support RID – Mission Risk Models – Hazard Quantification – Focused assessments and trades – Different methods potentially used for eachPage 15 NASA CxP John V. Turner, PMC 2009
  16. 16. Risk Informed Design Continuum CxP Early SRR SDR PDR CDR TBD Concept Exploration Define initial Define Preliminary Detailed Verification Early Design mission Requirements Design Design architecture Design FidelityAnalysisDesign Cycles Risk Analysis Fidelity Simple models…………………………………………………………………………………………....Complex Models Heritage and surrogate data………………………………………………………………………Test ./ Demonstrated Data Architecture trades……………….………………Design Improvement…………………………………..VerificationPage 16 NASA CxP John V. Turner, PMC 2009
  17. 17. Architecture Trade Studies Mars Mission Architecture Risk Assessment Architecture 6 Architecture 10 Systems Reliability Architecture 5 Entry / Landing Risk FOM Architecture 8 Architecture 3 Mars Orbit Insertion Architecture 1 Launch / Integration Architecture 7 Trans Mars Injection Architecture 4 Mars Ascent Architecture 9 Architecture 2 Trans Earth Injection 0.00 1.00 2.00 3.00 Other Hazards Reference Missions Example Only – Not Real DataPage 17 NASA CxP John V. Turner, PMC 2009
  18. 18. Architecture and System Level Assessments Example Only – Not Real DataPage 18 NASA CxP John V. Turner, PMC 2009
  19. 19. LOC Uncertainty Results Example Only – Not Real DataPage 19 NASA CxP John V. Turner, PMC 2009
  20. 20. Prioritizing Design Mitigation Example Only – Not Real DataPage 20 NASA CxP John V. Turner, PMC 2009
  21. 21. Mission Success Depends Upon a Combination of Many Variables Launch Strategy: Launch: • Two launch Vehicle Reliability: • Time increment • LOM/LOC • Single Launch between launches • Launch Availability Target Characteristics: • Launch Probability • Redundant Landing Sites • Order of Launches • Multiple opportunities to access a select landing site LEO Loiter: • Lighting constraints at • LEO Loiter Duration target Vehicle Performance: • Ascent Rendezvous Opportunities • Orbital Mechanics Variation Tolerance • TLI Windows • Additional Propulsive Capability • Vehicle Life • Launch Mass ConstraintsPage 21 NASA CxP John V. Turner, PMC 2009
  22. 22. RITOS Overview • RITOS Objective: Elicit expert opinion and historical data related to top program flight risk drivers in order to: 1. Better understand the risks and associated uncertainties 2. Identify potential mitigations and/or controls and effective test and verification strategies 3. Qualitatively assess the adequacy of the currently planned mitigations/controls and test and verification activities • RITOS Approach: – Identify top program risk drivers based on SR&QA products, history, and judgment – Elicit expert opinion and historical data related to the risk driver – Assess currently planned mitigation/control and test and verification strategies based on elicitation results, historical data, and judgment – Provide recommendations to T&V for enhancing currently planned approach to risk driver mitigation/control and test and verification NASA CxP John V. Turner, PMC 2009Page 22
  23. 23. SR&QA Scope ♦ Are planned analysis and test (ground/flight) SR&QA adequate to characterize and burn down risk? Focus • Type, scope, and fidelity of tests • Frequency of tests ♦ Is the plan executable? • Budget − Enough $ SE&I • Schedule Focus − Fabrication / Integration / Need dates − Test, Fix, Fly • Analysis and Reaction Time • Facilities − Do we have the right facilities − Availability • Test articles − Availability, fidelity, re-use issues, timingPage 23 NASA CxP John V. Turner, PMC 2009
  24. 24. Risk Topic Selection • Risk topics are chosen based on their priority in the various SR&QA risk product results, historical data, and SR&QA judgment • Initial topic list: – MMOD Impact to Orion for ISS DRM – First Stage/Upper Stage Separation – Orion descent and landing – Upper Stage Engine – Launch Abort System – Upper Stage/Orion Separation – Thermal Protection System • List can be further expanded as new risk topics are identifiedPage 24 NASA CxP John V. Turner, PMC 2009
  25. 25. Expert Elicitation • Each risk topic is researched in order to understand the mechanisms and/or phenomena that drive the risk • Attempt to identify an expert from each discipline area related to the risk – External candidates – Historical failure experts – CxP Internal subject matter experts – NESC panelists from applicable studies • Elicitation is a structured one-on-one discussion with the candidate in which various topics related to the risk are discussed, but in context to: – Risk calculation, characterization, and uncertainty – Test and Verification – Mitigations and Controls • Following elicitation, results are combined into themes and organized such that they are useful to the assessmentPage 25 NASA CxP John V. Turner, PMC 2009
  26. 26. Results Format • RITOS objective is to provide results that are beneficial to CxP IT&V and SR&QA • RITOS approach can be modified for each risk topic to accommodate IT&V needs • Results are qualitative, but can provide “sanity check” of T&V plans • Results could be useful in prioritizing test objectives • Results will be presented in two formats: 1. Bulleted form as elicitation result conclusions 2. Swimlane chart depiction of currently planned T&V activities with RITOS recommendations mapped into process flowPage 26 NASA CxP John V. Turner, PMC 2009
  27. 27. RITOS Progress to Date Initial Candidate ID / Schedule and Conduct Assess Existing Status Research Question Dev Elicitations; Compile Results Test Plan MMOD FS/US Sep US Engine ED&L TPS LAS US/Orion Sep On-hold Reduced progress Normal progressPage 27 NASA CxP John V. Turner, PMC 2009
  28. 28. RITOS Lessons Learned • Obtaining access to CxP Internal Subject Matter Experts has proven to be challenging. Working through Level II representatives has helped but not fully solved problem. • Coordination with Projects is challenging and time consuming. In cases where delay to study progress occurred we moved ahead to future topics to continue progressing. • Obtaining test plans is challenging and in some cases test plans do not exist. In cases where test plans are not available we package results in way that can be used during test plan development. Test plan will be assessed once it becomes available. • Typical RITOS elicitation results are qualitative, but can provide sanity check of test plans and insight into test prioritization when reductions are being considered. Conclusions obtained from elicitation themes are provided to Cx IT&V.Page 28 NASA CxP John V. Turner, PMC 2009
  29. 29. Conclusions • Risk Informed Design Provides a methodology to incorporate risk information early in the design process and obtain a more optimal balance of design commodities and risk than traditional rule of thumb safety design criteria • Risk Informed Test utilizes risk information to identify areas where test can be used more effectively to reduce uncertainty and risk prior to transition to operations. • Experience to date in the Constellation program indicates the value of the RID and RIT, but additional work is need to develop more consistent methods and tools to accomplish RID and RIT • In order to eliminate defects and thus reduce actual failures, programs and projects need to proactively address defect sources in Design, Test, Mission Assurance, and operations – This presentation only addresses two of these four “buckets”Page 29 NASA CxP John V. Turner, PMC 2009
  30. 30. BackupPage 30 NASA CxP John V. Turner, PMC 2009
  31. 31. COMPONENTS AND FLOW OF A TYPICAL PRA MODEL Cut Sets CCF A,B,C 1E-3 End States For Shuttle: Gas Explosion 2E-4 List of consequence LOCV (Loss of Crew & Vehicle) A fails, B fails, C fails 1.5 E-4 of interest Etc. Phase I Results MLD Event Trees FMEAs/CILs Development Hazard Reports Functional Analyses SAPHIRE List of Previous Risk Assessments Initiating Events Risk Levels for selected end states Fault Trees  Flight Rules  Training Manuals  System Architecture  Engineering Expertise • Assumptions Data Analyses  MADS  PRACA Relative risk drivers  Industry databases  Other assessments (e.g. off-line simulation Something that this graphic does not display is models) the necessary engineering analysis that must be done Reviewed by to support success criteria and capacity Program Organizations A large number of pages of detailed documentation are requiredPage 31 NASA CxP John V. Turner, PMC 2009
  32. 32. RITOS ProcessPage 32 NASA CxP John V. Turner, PMC 2009