1. Integrated Testing - Adding Value to Systems Engineering Process NASA PM Challenge 2011 Keith Britton/NASA KSC Dawn Schaible/NASA LaRC

2. Testing has long been recognized as important yet there are many examples of how more testing could have prevented failures Testing alone does not make a program successful but does increase the likelihood of success 2 Problem Statement

3. 3 Problem Statement It is easier to find and fix problems on the ground… …than it is on-orbit

4. Research Methodology This presentation is based on previous research for the MIT Systems Design and Management Program Thesis. Based in interviews with NASA and Industry Experts Role of testing/requirements definition Decision making factors Testing budgets and cost considerations Organizational factors/knowledge transfer Focused on testing portion of verification process Emphasized integration phase of test programs Human-rated spacecraft were investigated 4

5. Common Themes from Research Barriers: Subjectivity of test requirement development Paradoxical nature of testing Vulnerability to changes and cutbacks Inadequate attention to testing Optimism varies with organizational position Cost tracking methods for testing are insufficient 5

6. Barriers Subjectivity of test requirement development Test requirement decision process is usually subjective, rather than a documented process Risk factors do not perfectly relate to test requirements (due to complexity of interactions) 6

7. Paradoxical nature of testing Testing raises confidence in the system, but can often be seen as a drain on program resources A test should be considered successful if many problems are found, or if the system works as designed Difficult to measure cost avoidance 7 Barriers

8. Vulnerability to changes and cutbacks Most testing activities occur at end of development phase, when budget and schedule pressure is the greatest Incomplete or changing test requirements are more vulnerable to cutbacks than stringent requirements 8 Barriers

9. Inadequate attention to testing Testing is often overlooked during the early planning phases General lack of formal training in testing Sometimes test engineers do not hold the same status as other engineers in the project 9 Barriers

10. Barriers Optimism varies with organizational position Due to risk aversion level, systems perspective, fidelity of available information Detailed technical information may not be consistent across the organization 10

11. Cost tracking methods for testing are insufficient Difficult to determine pure testing cost from current budget structures Fidelity of life-cycle cost estimating not useful for test requirement decision 11 Barriers

12. Common Themes from Research Enablers: While testing practices vary, decision factors do not Upfront planning is a key to success, but be prepared for change Testing is more of an art than a science 12

14. Risk

15. Confidence building

16. New design/technology

17. Complexity

18. Reuse

19. Team experience

21. Process

22. Individual decision-making behavior

24. Enablers Testing is more of an art than a science Experience and mentoring can be more important than formal training Test engineers need to know the system design as well as the designers – and be creative in finding system faults Good sub-system engineering is not a substitute for proper system engineering Complexity of current systems requires strong integration and communication 15

25. Recommendations Understand the barriers and enablers to good test programs and account for them in planning Include testing in the earliest stages of the spacecraft’s development process Establish improved training and mentoring programs for test engineers Consider test engineering a valid profession with a viable career path for test engineers Recognize and understand personal risk-tolerance levels and how individual decision-making styles affect decisions Continue research into dealing with the complexity of today’s systems and how testing can be used to discover unintended interactions 16