• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Itea 1:2013 beyond_Test and Evaluation
 

Itea 1:2013 beyond_Test and Evaluation

on

  • 114 views

System Engineering large scale Test and Evaluation. Emphasizes determining Fitness for Purpose and/or Readiness for Operation of deployed systems.Focuses on producing actionable knowledge for the ...

System Engineering large scale Test and Evaluation. Emphasizes determining Fitness for Purpose and/or Readiness for Operation of deployed systems.Focuses on producing actionable knowledge for the warfighter.

Statistics

Views

Total Views
114
Views on SlideShare
114
Embed Views
0

Actions

Likes
0
Downloads
0
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

CC Attribution-ShareAlike LicenseCC Attribution-ShareAlike License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Good afternoon, ladies and gentlemen. My purpose is two fold. First, to make you aware that Test and Evaluation as it is described, taught and practiced is not sufficient for the era of large scale, networked, heterogenous systems. Second, to briefly outline a new paradigm that will serve system users/operators. <br />
  • Here is your chance to detect my biases and blind spots. <br /> Hopefully you can overcome them. <br /> Lots of actual experience mixed with formal learning and action research. <br /> One of the experiences I greatly enjoyed was resolving a disagreement between Bob Joehnck and Max Balchowsky about camshaft lobe duration. <br />
  • To delve deeper into the Challenges, two systems depicted as cartoons, the red one on the left and the blue one on the right. Each, of course, has subsystems of various kinds embedded in respective infrastructures. <br /> The simplistic approach to creating a system of systems is to presume interconnecting the two. <br /> But there is much more to it than that. Consider that the left system has associated Beneficiaries, Logistics supplier, Operators and Data Base Administrators. Each has a Level of Proficiency, U*P*C, indicated by the solid connector. Likewise, the system on the left has similar associated components, each with their own U, P, and C. <br /> The rest of the story is that the system on the left depends on other Key personnel, Each has a U , P , C coefficient but is further hampered by Knowing the system only through a model, Z(S1), of the system. Likewise with Z (S2). <br /> Accordingly , the interconnection is conditioned not only by the technical link but also by the human links. <br /> Consider the number of failure modes. <br />
  • From the systemics view we see that autonomy involves one or more of Adjust, Arrange and Co-align. Too often overlooked are the mandatory constraints on Adjust, Arrange and Co-align. Most of us are familiar with the Conservation of mass, momentum and energy as a physical constraint on the dynamics of change. UAS’s require that we also acknowledge and honor contraints associated with Informatics, Teleonomics,, Human Social Dynamics, Economics and Ecologics.. <br />
  • Although humans exhibit a variety of behaviors one trajectory of relationships is often seen. <br /> This chart suggest labels (leftmost column) for the kinds of relationships that can evolve among purposeful communities. Also, the chart indicates the meaning of the label (center column) and identifies a key mediator or catalyst (rightmost column). <br /> Relationships start with Connect (when attributes descriptors of each party are accessible to both, directly or through a broker) and progress through Communicate to Co-evolve. <br /> Current (2003) literature is largely concerned with collaboration. We note that achieving a collaborative community is not the ultimate. In fact, there are three more levels of human interaction beyond collaboration. And perhaps more. <br /> Not shown on this chart of relationships is the pursuit of Personal Best. Often the Desire to Serve is manifest by stimulating another toward higher levels of their personal best. . <br />
  • Implementing these propositions will be easier for reflective practitioners than for those who follow some rote recipe for systems engineering. <br /> Prof. Donald Schon, MIT, has advised of four levels of behavior and two kinds of knowing. Further, he says that Design cannot be taught but design can be learned. <br /> You can be a SE craftsman or an SE practitioner depending on how well you learn reflection. <br />
  • The dominant failure mode will reflect the Law of Unintended Consequences. Rather than hard failures that can be detected and fixed the SoS will quietly fail in pernicious ways while the human components (with low U, P, C) think they are accomplishing their mission, <br /> SoSE must guard against LUC. Two ways are listed here. <br />

Itea 1:2013 beyond_Test and Evaluation Itea 1:2013 beyond_Test and Evaluation Presentation Transcript

  • System Engineering Next Generation T&E The Fit For Purpose Paradigm ITEA SoS Workshop: The NIE Experience El Paso, TX, July 17, 2013 Attributed reuse permitted Jack Ring Educe LLC jring7@gmail.com
  • Intended Takeaways The tenets of Test & Evaluation currently taught and practiced are not sufficient for net-centric SoS, especially at brigade scale. Key gaps are: a) In-field modeling of the anticipated engagement and formulating a ConOps for the warfighter’s intervention system. b) System Design and Engineering of a responsive T&E system. c) Inadequate SoSSE and Acquisition policies and practices. Next Step: Develop a cadre of T&E Systemists. 11/07/13 jring7@gmail.com, attributed reuse permitted 2
  • You don’t know Jack ?  1957 – Present. GE 20, Honeywell 10, Edelbrock 3, Ascent Logic 2, IBM OTP 1, Entrepreneur 20. Kennen Technologies LLC, OntoPilot LLC, Educe LLC.  System Test & Evaluation (Atlas ICBM Radio Guidance System)  System Engineering (State-determined  Stochastic  Non-deterministic Systems).  Inaugural chair, 1970, GE-wide workshop on Software Engineering.  Led SAFE system concept definition for multi-agency federation of intelligence collection, analysis and production.  Product Manager, Distributed Transaction Processing products suite.  Led Computer Integrated Manufacturing shop floor control network demo involving 20 vendors using GM technologies.  SME: Autonomous System T&E MAP protocol.  Tutorials & Papers; ITEA, INCOSE, INCOSE IL, ISSS, IEEE-SMC, IEEE SysCon, ICSEng, NIST.  Patent co-author, General Purpose Set Theoretic Processor Architecture and Method. 11/07/13 3
  • The Net-Centric Paradigm Lead Time? Predictable + Adaptive + Autonomous assets SoS Warfighter’s SoS SoSSE Anticipated Engagement Realized & Net-centric Praxis(i) Design SoS Engagement Deployed Model Warfighter Action  Actionable Conduct T&E Sessions Knowledge Is this a system? Will it be Fit For Purpose? Is it still Fit For Purpose? Realize SysEng T&E T&E System System  Red Force, Blue Force, Green Force, Grey Force 11/07/13 jring7@gmail.com, attributed reuse permitted Adaptatio n Cycle Time 4
  • Warfighters Deserve To Know Is This System FIT FOR PURPOSE, F4P? POSIWID: the purpose of a system is what it does, regardless of designer or operator intent. Is This System Still FIT FOR PURPOSE, F4P? The Mean Time to Configuration Change. MTTCC, of a brigade-scale system may be < 15 minutes. The F4P test must be run at MTTCC/2 11/07/13 jring7@gmail.com, attributed reuse permitted 5
  • Current T&E Situation Gap: Inadequate warfighter-trusted knowledge regarding the dynamic and integrity limits of multi-node networks of heterogeneous, autonomous systems. Status: Current T&E, e.g., NIE, is a) 10X too expensive and time consuming and b) Does not determine Fit For Purpose. Talent: T&E community competencies have dwindled toward instrumentation and data technicians. Does not leverage NCO knowledge and learning. Remediation: The current resurgence of system engineering and system of systems engineering does not provide the necessary facts. 11/07/13 jring7@gmail.com, attributed reuse permitted 6
  • Why Beyond T&E? Current T&E < <Factor > > System of Interest Focus Does system meet Requirements? Data TENA Errors, Operational Capabilities Data Question Acquisition 11/07/13 Key Capability Findings Work Product Serves Beyond Effect of SOI on larger system Will SoS be Fit for Purpose(s)? System modeling Design of Experiment Loci of dynamic and integrity limits Actionable Knowledge Warfighter jring7@gmail.com, attributed reuse permitted 7
  • Warfighter ConOps 2 Operators e.g., Warfighters Effects Capabilities Problematic 1 Situation MOE’s Anticipated 3 Engagement N Operational Y F4P? System 7 Engineering Application Knowledge Base 4 5 Acquisition System 6 Inventory of Realization Components jring7@gmail.com, attributed reuse permitted & Resilience 11/07/13 8
  • SoS Interoperability Principles • A system exists only when deployed, activated and responding to stimuli. • Entails orchestration of “N” self-adapting systems that exhibit the desired effects. • The dynamic and integrity limits of any system are determined by a) Progress properties: starting from some state reach a desired state in a finite number of steps. b) Safety/Integrity properties: maintain certain states that always insure correct progress. 11/07/13 jring7@gmail.com, attributed reuse permitted 9
  • Key Gap Closing Actions 1. Engagement Modeling 2. SE of T&E of SoS F4P 3. Develop a cadre of T&E Systemists 11/07/13 jring7@gmail.com, attributed reuse permitted 10
  • 1. Engagement Modeling • Warfighters, envision their intervention system by deciding a) the intended Effects, Measures of Effectiveness and Capabilities of their intended SoS and b) the likely evolution as interaction progresses and c) update their SoS model at the pace of engagement or resource change. • Warfighters need an modeling method and tool to clarify the explicit and implicit effects of the problem system and intervention system on each other, notably [Input/Output, Performance, Technology, Tradeoff Gradients, Testability, and Cost]* • Warfighters need an asset knowledge base provided by the Acquisition community. * Model-based System Engineering by A. Wayne Wymore, CRC Press, 1995 11/07/13 jring7@gmail.com, attributed reuse permitted 11
  • 1.1 The Warfighter SoS Model The truth, the whole truth, and nothing but the truth. Informatics Thermodynamics Biomatics Teleonomics Social Dynamics Economics Ecologics Intended: Emergence & Prevention of Emergence Model must be directly executable. based on a formal ontology. reflexive. 11/07/13 Minimal Implicate Order jring7@gmail.com, attributed reuse permitted 12
  • Intended Outcomes: A new T&E paradigm. Enthusiasm to Transition. Justifiable budgets. Joint Operations Brigade-scale Warfighters PEOs PMs Test Range Executives FAA, FCC, DHS Allies, Congress Actionable Knowledge WHY With Warfighters By Purple-suiters And Civilian SME’S WHO Development Field Ops Training Exercises WHERE in situ (Battlefield) Simple  Autonomous Engagement-specific Physical & Cyber 11/07/13 2. 2. SE of T&E SE of T&E Of F4P Of F4P WHAT Knowledge Discovery/Usage New User Paradigms More with less jring7@gmail.com, attributed reuse permitted “Day 2”of SoS Projects Until End of Life WHEN At speed of SoS evolution Whole system span, e.g., DOTMLPF Third Generation SE HOW Non-deterministic SoS’s Adequate, Accurate, Timely Trusted, Cyber-Assured Independent & Objective T&E enterprise Many Kinds of T&E Family of T&E systems 13
  • 2.1 Is Each Stage of the Warfighter SoS viable? 1) Ensure Acquisition SoSSE a) describes progress properties and safety/integrity properties and b) Includes sufficient self-test in components. 2) Leverage new technology that automates System Readiness Assessment: a) Software problems: Reduced to ≈ 1% of current. b) System Integration cost/time: Reduced to ≈ 20% of current experience. c) Test abort delays: Reduced by ≈ 40% of current T&E experience. d) Cybersecurity: Reduced vulnerability. 11/07/13 jring7@gmail.com, attributed reuse permitted 14
  • 2.2 Is the Warfighter SoS Fit For Purpose? • Confirm that anticipated and demonstrated dynamic and integrity limits of warfighter’s SoS are necessary, sufficient and efficient. • Apply from Day 2 of a new engagement project through Year N of the SoS usage/evolution cycle --- as often as MTTCC/2. 11/07/13 jring7@gmail.com, attributed reuse permitted 15
  • 2.3.a Is the SoS Personnel Subsystem Systemic? Z = model of the local system of interest = fidelity of Z to S Beneficiaries Node 1 Supply = U * P * C where; U = degree of understanding P = level of role proficiency C = level of collaboration ability Beneficiaries Supply Operators Operators DB Admin DB Admin Config Admin In-service Eng. Z(S1) 11/07/13 Node (i) Dev.Eng. Systems Eng. Config Admin In-service Eng. Dev. Eng. Systems Eng. J. Ring & A. Madni, “Key Challenges in SoS Engineering” 2005 IEEE International Conference on Systems, Man and Cybernetics, Waikoloa, Hawaii October 10-12, 2005 Z(S2) 16
  • 2.3.b Are SoS-wide Changes Coherent? Adjust: Gradients on relationships Arrange: Pattern of relationships Co-align: Content of system with context and resources. Consistent with Conservation Laws? X, d(X)/dt, d2(X)/dt2 Thermodynamics: mass, momentum and energy Informatics: data, information and knowledge Teleonomics: skills, rate of learning, and rate of invention Human social dynamics: trust, enthusiasm, co-evolution Economic: Investment, ROI, Liquidity Ecology: Waste, Fads, Unintended Consequences 11/07/13 17 jring7@gmail.com, attributed reuse permitted
  • 3. Develop a cadre of T&E Systemists a) Conduct 12-15 person expeditions that implement ConOps. 3X100 day cycles. 20 teams in five years. b) Adept at all degrees of Extent, Variety, Ambiguity. c) Executable models enable system composition. d) Ensures requisite information from acquisition programs. e) Accelerates co-learning of all involved. 11/07/13 jring7@gmail.com, attributed reuse permitted 18
  • 3.1 Use Work Program of Complexity Discovery: Description is done in a group process. focus on problematic situation and underlying problem system. Diagnosis is done by an experienced individual professional, who iterates with the group until the description is fully understood and accepted. Resolution: Design is done in a group process. involves both formal logic and behavioral pathologies. Implementation is carried out by whatever means the design specifies. UNDERSTANDING COMPLEXITY: THOUGHT & BEHAVIOR, 2002, www.jnwarfield.com A HANDBOOK OF INTERACTIVE MANAGEMENT, 1990/1994, www.jnwarfield.com 11/07/13 jring7@gmail.com, attributed reuse permitted 19
  • 3.2 Create Self-synergizing Systemists Relationship Meaning Mediators Co-evolve Morphing toward Win-Win-Win Joy-enabled Level of Consciousness Co-facilitate Co-learn Collaborate Co-celebrate Value Out/Value In ≈ eN Stewardship by N participants Meaningful reflection Shared knowledge claims Help one another Desire to serve En-joying one another Time & Space, F2F Cooperate Commit Converge Compatible Actions Willing to wait Principled relationship Courage to plan Common compelling purpose Shared self-respect Communicate Connect Share interests and values Common language Two discover one another Accessible attributes 11/07/13 jring7@gmail.com, attributed reuse permitted 20
  • 3.3 Encourage Reflective Practitioners Four ascending levels of behavior: •Know how. •Reflection -- on how 'know how' was applied. •Knowing-in-action (devising while doing) •Reflection-in-action. A Practitioner must have two kinds of knowing: • Objectivist - descriptive • Constructivist – prescriptive - world making Designing cannot be taught -- but can be coached • Joint experimentation. • Follow Me! Educating the Reflective Practitioner, • Hall of Mirrors. Donald Schon, Jossey Bass, 1987 11/07/13 jring7@gmail.com, attributed reuse permitted 21
  • Recap: F4P Paradigm Benefits 1) 2) 3) 4) 5) 6) Free. Return >> Investment. Consistent with DoD 5000.2 (effects, capabilities). Consistent with Swarms of Autonomous Systems. Benefits Acquisition community as well as warfighters. Dynamic and integrity limits inform Design of Experiments. Effective in at least defense, aviation, homeland security, industry supply chains, knowledge discovery/vetting networks, and human activity systems. 7) Re-orients and prepares system engineering personnel to leverage next generation technologies. 11/07/13 jring7@gmail.com, attributed reuse permitted 22
  • Clarifications? Questions? Comments? 11/07/13 The castle, Jack, besiege the CASTLE! jring7@gmail.com, attributed reuse permitted Thank you! 23