Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Itea 1:2013 beyond_Test and Evaluation


Published on

System Engineering large scale Test and Evaluation. Emphasizes determining Fitness for Purpose and/or Readiness for Operation of deployed systems.Focuses on producing actionable knowledge for the warfighter.

Published in: Technology, Education
  • Be the first to comment

  • Be the first to like this

Itea 1:2013 beyond_Test and Evaluation

  1. 1. System Engineering Next Generation T&E The Fit For Purpose Paradigm ITEA SoS Workshop: The NIE Experience El Paso, TX, July 17, 2013 Attributed reuse permitted Jack Ring Educe LLC
  2. 2. Intended Takeaways The tenets of Test & Evaluation currently taught and practiced are not sufficient for net-centric SoS, especially at brigade scale. Key gaps are: a) In-field modeling of the anticipated engagement and formulating a ConOps for the warfighter’s intervention system. b) System Design and Engineering of a responsive T&E system. c) Inadequate SoSSE and Acquisition policies and practices. Next Step: Develop a cadre of T&E Systemists. 11/07/13, attributed reuse permitted 2
  3. 3. You don’t know Jack ?  1957 – Present. GE 20, Honeywell 10, Edelbrock 3, Ascent Logic 2, IBM OTP 1, Entrepreneur 20. Kennen Technologies LLC, OntoPilot LLC, Educe LLC.  System Test & Evaluation (Atlas ICBM Radio Guidance System)  System Engineering (State-determined  Stochastic  Non-deterministic Systems).  Inaugural chair, 1970, GE-wide workshop on Software Engineering.  Led SAFE system concept definition for multi-agency federation of intelligence collection, analysis and production.  Product Manager, Distributed Transaction Processing products suite.  Led Computer Integrated Manufacturing shop floor control network demo involving 20 vendors using GM technologies.  SME: Autonomous System T&E MAP protocol.  Tutorials & Papers; ITEA, INCOSE, INCOSE IL, ISSS, IEEE-SMC, IEEE SysCon, ICSEng, NIST.  Patent co-author, General Purpose Set Theoretic Processor Architecture and Method. 11/07/13 3
  4. 4. The Net-Centric Paradigm Lead Time? Predictable + Adaptive + Autonomous assets SoS Warfighter’s SoS SoSSE Anticipated Engagement Realized & Net-centric Praxis(i) Design SoS Engagement Deployed Model Warfighter Action  Actionable Conduct T&E Sessions Knowledge Is this a system? Will it be Fit For Purpose? Is it still Fit For Purpose? Realize SysEng T&E T&E System System  Red Force, Blue Force, Green Force, Grey Force 11/07/13, attributed reuse permitted Adaptatio n Cycle Time 4
  5. 5. Warfighters Deserve To Know Is This System FIT FOR PURPOSE, F4P? POSIWID: the purpose of a system is what it does, regardless of designer or operator intent. Is This System Still FIT FOR PURPOSE, F4P? The Mean Time to Configuration Change. MTTCC, of a brigade-scale system may be < 15 minutes. The F4P test must be run at MTTCC/2 11/07/13, attributed reuse permitted 5
  6. 6. Current T&E Situation Gap: Inadequate warfighter-trusted knowledge regarding the dynamic and integrity limits of multi-node networks of heterogeneous, autonomous systems. Status: Current T&E, e.g., NIE, is a) 10X too expensive and time consuming and b) Does not determine Fit For Purpose. Talent: T&E community competencies have dwindled toward instrumentation and data technicians. Does not leverage NCO knowledge and learning. Remediation: The current resurgence of system engineering and system of systems engineering does not provide the necessary facts. 11/07/13, attributed reuse permitted 6
  7. 7. Why Beyond T&E? Current T&E < <Factor > > System of Interest Focus Does system meet Requirements? Data TENA Errors, Operational Capabilities Data Question Acquisition 11/07/13 Key Capability Findings Work Product Serves Beyond Effect of SOI on larger system Will SoS be Fit for Purpose(s)? System modeling Design of Experiment Loci of dynamic and integrity limits Actionable Knowledge Warfighter, attributed reuse permitted 7
  8. 8. Warfighter ConOps 2 Operators e.g., Warfighters Effects Capabilities Problematic 1 Situation MOE’s Anticipated 3 Engagement N Operational Y F4P? System 7 Engineering Application Knowledge Base 4 5 Acquisition System 6 Inventory of Realization Components, attributed reuse permitted & Resilience 11/07/13 8
  9. 9. SoS Interoperability Principles • A system exists only when deployed, activated and responding to stimuli. • Entails orchestration of “N” self-adapting systems that exhibit the desired effects. • The dynamic and integrity limits of any system are determined by a) Progress properties: starting from some state reach a desired state in a finite number of steps. b) Safety/Integrity properties: maintain certain states that always insure correct progress. 11/07/13, attributed reuse permitted 9
  10. 10. Key Gap Closing Actions 1. Engagement Modeling 2. SE of T&E of SoS F4P 3. Develop a cadre of T&E Systemists 11/07/13, attributed reuse permitted 10
  11. 11. 1. Engagement Modeling • Warfighters, envision their intervention system by deciding a) the intended Effects, Measures of Effectiveness and Capabilities of their intended SoS and b) the likely evolution as interaction progresses and c) update their SoS model at the pace of engagement or resource change. • Warfighters need an modeling method and tool to clarify the explicit and implicit effects of the problem system and intervention system on each other, notably [Input/Output, Performance, Technology, Tradeoff Gradients, Testability, and Cost]* • Warfighters need an asset knowledge base provided by the Acquisition community. * Model-based System Engineering by A. Wayne Wymore, CRC Press, 1995 11/07/13, attributed reuse permitted 11
  12. 12. 1.1 The Warfighter SoS Model The truth, the whole truth, and nothing but the truth. Informatics Thermodynamics Biomatics Teleonomics Social Dynamics Economics Ecologics Intended: Emergence & Prevention of Emergence Model must be directly executable. based on a formal ontology. reflexive. 11/07/13 Minimal Implicate Order, attributed reuse permitted 12
  13. 13. Intended Outcomes: A new T&E paradigm. Enthusiasm to Transition. Justifiable budgets. Joint Operations Brigade-scale Warfighters PEOs PMs Test Range Executives FAA, FCC, DHS Allies, Congress Actionable Knowledge WHY With Warfighters By Purple-suiters And Civilian SME’S WHO Development Field Ops Training Exercises WHERE in situ (Battlefield) Simple  Autonomous Engagement-specific Physical & Cyber 11/07/13 2. 2. SE of T&E SE of T&E Of F4P Of F4P WHAT Knowledge Discovery/Usage New User Paradigms More with less, attributed reuse permitted “Day 2”of SoS Projects Until End of Life WHEN At speed of SoS evolution Whole system span, e.g., DOTMLPF Third Generation SE HOW Non-deterministic SoS’s Adequate, Accurate, Timely Trusted, Cyber-Assured Independent & Objective T&E enterprise Many Kinds of T&E Family of T&E systems 13
  14. 14. 2.1 Is Each Stage of the Warfighter SoS viable? 1) Ensure Acquisition SoSSE a) describes progress properties and safety/integrity properties and b) Includes sufficient self-test in components. 2) Leverage new technology that automates System Readiness Assessment: a) Software problems: Reduced to ≈ 1% of current. b) System Integration cost/time: Reduced to ≈ 20% of current experience. c) Test abort delays: Reduced by ≈ 40% of current T&E experience. d) Cybersecurity: Reduced vulnerability. 11/07/13, attributed reuse permitted 14
  15. 15. 2.2 Is the Warfighter SoS Fit For Purpose? • Confirm that anticipated and demonstrated dynamic and integrity limits of warfighter’s SoS are necessary, sufficient and efficient. • Apply from Day 2 of a new engagement project through Year N of the SoS usage/evolution cycle --- as often as MTTCC/2. 11/07/13, attributed reuse permitted 15
  16. 16. 2.3.a Is the SoS Personnel Subsystem Systemic? Z = model of the local system of interest = fidelity of Z to S Beneficiaries Node 1 Supply = U * P * C where; U = degree of understanding P = level of role proficiency C = level of collaboration ability Beneficiaries Supply Operators Operators DB Admin DB Admin Config Admin In-service Eng. Z(S1) 11/07/13 Node (i) Dev.Eng. Systems Eng. Config Admin In-service Eng. Dev. Eng. Systems Eng. J. Ring & A. Madni, “Key Challenges in SoS Engineering” 2005 IEEE International Conference on Systems, Man and Cybernetics, Waikoloa, Hawaii October 10-12, 2005 Z(S2) 16
  17. 17. 2.3.b Are SoS-wide Changes Coherent? Adjust: Gradients on relationships Arrange: Pattern of relationships Co-align: Content of system with context and resources. Consistent with Conservation Laws? X, d(X)/dt, d2(X)/dt2 Thermodynamics: mass, momentum and energy Informatics: data, information and knowledge Teleonomics: skills, rate of learning, and rate of invention Human social dynamics: trust, enthusiasm, co-evolution Economic: Investment, ROI, Liquidity Ecology: Waste, Fads, Unintended Consequences 11/07/13 17, attributed reuse permitted
  18. 18. 3. Develop a cadre of T&E Systemists a) Conduct 12-15 person expeditions that implement ConOps. 3X100 day cycles. 20 teams in five years. b) Adept at all degrees of Extent, Variety, Ambiguity. c) Executable models enable system composition. d) Ensures requisite information from acquisition programs. e) Accelerates co-learning of all involved. 11/07/13, attributed reuse permitted 18
  19. 19. 3.1 Use Work Program of Complexity Discovery: Description is done in a group process. focus on problematic situation and underlying problem system. Diagnosis is done by an experienced individual professional, who iterates with the group until the description is fully understood and accepted. Resolution: Design is done in a group process. involves both formal logic and behavioral pathologies. Implementation is carried out by whatever means the design specifies. UNDERSTANDING COMPLEXITY: THOUGHT & BEHAVIOR, 2002, A HANDBOOK OF INTERACTIVE MANAGEMENT, 1990/1994, 11/07/13, attributed reuse permitted 19
  20. 20. 3.2 Create Self-synergizing Systemists Relationship Meaning Mediators Co-evolve Morphing toward Win-Win-Win Joy-enabled Level of Consciousness Co-facilitate Co-learn Collaborate Co-celebrate Value Out/Value In ≈ eN Stewardship by N participants Meaningful reflection Shared knowledge claims Help one another Desire to serve En-joying one another Time & Space, F2F Cooperate Commit Converge Compatible Actions Willing to wait Principled relationship Courage to plan Common compelling purpose Shared self-respect Communicate Connect Share interests and values Common language Two discover one another Accessible attributes 11/07/13, attributed reuse permitted 20
  21. 21. 3.3 Encourage Reflective Practitioners Four ascending levels of behavior: •Know how. •Reflection -- on how 'know how' was applied. •Knowing-in-action (devising while doing) •Reflection-in-action. A Practitioner must have two kinds of knowing: • Objectivist - descriptive • Constructivist – prescriptive - world making Designing cannot be taught -- but can be coached • Joint experimentation. • Follow Me! Educating the Reflective Practitioner, • Hall of Mirrors. Donald Schon, Jossey Bass, 1987 11/07/13, attributed reuse permitted 21
  22. 22. Recap: F4P Paradigm Benefits 1) 2) 3) 4) 5) 6) Free. Return >> Investment. Consistent with DoD 5000.2 (effects, capabilities). Consistent with Swarms of Autonomous Systems. Benefits Acquisition community as well as warfighters. Dynamic and integrity limits inform Design of Experiments. Effective in at least defense, aviation, homeland security, industry supply chains, knowledge discovery/vetting networks, and human activity systems. 7) Re-orients and prepares system engineering personnel to leverage next generation technologies. 11/07/13, attributed reuse permitted 22
  23. 23. Clarifications? Questions? Comments? 11/07/13 The castle, Jack, besiege the CASTLE!, attributed reuse permitted Thank you! 23