Standards Conformance Testing as M
Upcoming SlideShare
Loading in...5
×
 

Standards Conformance Testing as M

on

  • 816 views

 

Statistics

Views

Total Views
816
Views on SlideShare
816
Embed Views
0

Actions

Likes
0
Downloads
1
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Standards Conformance Testing as M Standards Conformance Testing as M Presentation Transcript

  • Standards Conformance Testing and M&S Web Services Bernard P. Zeigler, Ph.D., Arizona Center for Integrative Modeling and Simulation and Joint Interoperability Test Command Fort Huachuca, AZ 85613-7051
  • Outline
    • Role of Standards Conformance Testing in Net-Centric T&E
      • JITC responsibility for Link-16 standards conformance testing
      • Link-16: Challenges to implementation and testing
      • M&S–based Automated Test Case Generation
      • Application to Link-16 in the IABM SIAP Context
    • Service Oriented Architecture: M&S-based Testing Services
      • Link-16 standards conformance testing
      • certification for GIG deployable services
      • exposing M&S-based T&E assets as services
  • Net-Centric Test and Evaluation: Some Assertions
    • Net-Centric T&E
    • places an increased emphasis on standards conformance testing (SCT)
      • SCT will be the first gate before more extensive T&E
      • necessary, but not sufficient, for interoperability and mission effectiveness
    • performed on distributed net-centric test-bed infrastructure
      • Global Information Grid (GIG)
      • provide on-site or downloadable SCT packages
    • emphasizes services rather than systems
      • testing of Service Oriented Architecture (SOA)
      • GIG enterprise services
    View slide
  • JITC is the Responsible Test Organization for Tactical Data Link Standards
    • Joint Interoperability Test Command (JITC) is responsible for ensuring systems that implement Tactical Data Link * (TDL)
      • are interoperable and
      • in compliance with the applicable joint standards
    • This is accomplished by conducting the following types of tests:
      • Joint / NATO /Combined Interoperability
      • Performance Assessment in Operational Environments
      • Standards Validation
      • Standards Conformance
    • JITC employs a variety of tools that provide its analysts the ability to evaluate TDL system performance in both the lab and live environments.
    source: http:// jitc.fhu.disa.mil * Link 11/11B/16 and Variable Message Format (VMF) View slide
  • Link-16: Challenges to Implementation and Testing
    • Joint Single Link Implementation Requirements Specification
    • JSLIRS is an evolving standard (MIL-STD-6016c) for tactical data link information exchange and networked command/control of radar systems
    • Presents significant challenges to automated conformance testing:
      • The specification document states requirements in natural language
      • open to ambiguous interpretations
      • The document is voluminous
      • many interdependent chapters and appendixes
      • labor intensive and prone to error
      • potentially incomplete and inconsistent.
    • Problem: how to ensure that a certification test procedure
      • is traceable back to specification
      • completely covers the requirements
      • can be consistently replicated across the numerous contexts
      • military service, inter-national, and commercial companies
  • Benefits of Formalization and Automation
    • Provides traceability to original specification
    • Reduces ambiguity from textual specification
    • Facilitates integrating Modeling & Simulation into the testing process
    • Enables testing of complex:
      • Standards
      • Systems
      • Functions
      • Families of systems
  • SIAP/IABM — Successor to Link-16
    • SIAP (Single Integrated Air Picture) Goal: Improve the adequacy and fidelity of information to form a shared understanding of tactical situation
    • Integrated Architecture Behavior Model (IABM) requires that all sensors utilize a standard reference frame for conveying information about the location of targets.
    • Developed by the Joint SIAP System Engineering Organization (JSSEO), Arlington, Va., a sub-office of the Assistant Secretary of the Army for Acquisition, Logistics and Technology.
    source: http://www.navyleague.org/sea_power/mar_04_18.php
  • Automated Test Case Generator (ATC-Gen)
    • IABM is an extension of Link-16 developed in HLA environment and requires HLA simulation-based testing
    • JITC has taken the initiative to integrate modeling and simulation into the automation of the testing process
    • Funded the development of Automated Test Case Generator (ATC-Gen) led by ACIMS using DEVS (Discrete Event System Specification) technology.
    • In R&D of two years, proved the feasibility and the general direction
    • First major test of IABM – Config05 – ATC-Gen is the basis for testing link-16 and extended IABM requirements
  • Discrete Event Nature of Link-16 Specification Constraints (Exception) Rules Stop Modify C2 Record for TN 1 2 3 Rule Processing Stop, Do Nothing, Alerts, Or jump to other Transaction Track Display Operator decisions Validity checking Transmit Msg Other ConsequentProcessing Jumps (stimuli) to other Transactions of specification Transaction Level - example P.1.2 = Drop Track Transmit Preparation Processing Time outs Periodic Msg Input to system DEVS Output from system t 1 t 2 t 3 t 4
  • ATC Gen Goals & Approach
    • Goals :
    • To increase the productivity and effectiveness of standards conformance testing (SCT) at Joint Interoperability Test Command (JITC)
    • To apply systems theory, modeling and simulation concepts, and current software technology to (semi-)automate portions of conformance testing
    Objective: Automate Testing Capture Specification as If-Then Rules in XML Analyze Rules to Extract I/O Behavior Synthesize DEVS Test Models Test Driver Executes Models to Induce Testable Behavior in System Under Test (SUT) Interact With SUT Over Middleware Network DEVS Simulator Test Driver HLA SUT HLA
  • ATC Gen Tool
    • Standard to XML Translation
      • Analyst interprets the requirements text to extract state variables and rules, where rules are written in the form:
        • If P is true now Condition
        • Then do action A later Consequence
        • Unless Q occurs in the interim Exception
    • Dependency Analysis & Test Generation
      • Dependency Analyzer (DA) determines the relationship between rules by identifying shared state variables
      • Test Model Generator converts Analyst defined test sequences to executable simulation models
    • Test Driver
      • Test Driver interacts with and connects to SUT via HLA or Simple J interfaces to perform conformance testing
      • Validated against legacy test tools
  • Capturing the Link-16 Standard in XML Repository Appendix P Appendix U Section 4.11.13 Appendix D XML FILES Appendix P Appendix U Section 4.11.13 Appendix D MIL-STD-6061C
  • Rule Analysis Extract I/O Behaviors RESULTS RULE ANALYSIS Appendix P Appendix U Section 4.11.13 Appendix D MIL-STD-6061C
  • Test Sequence Manually Derive & Create Paths ResponderNnot specified> D.1.3.1.9ND.1.3.1> D.1.3.3.10.1a, D.1.3.3.13.1ND.1.3.1> D.1.3.12.1ND.1.3.1> Automatic Generation of Path File (Text)
  • Test Sequence Validation & Generation DEPENDENCY ANALYZER Validates Test Sequence Appendix P Appendix U Section 4.11.13 Appendix D XML FILES GENERATED TEST SEQUENCE (XML) Completed Path File Repository
  • Test Case Generation Test Sequence with Assigned Values TEST CASE GENERATOR GENERATED TEST CASE (XML)
  • Test Model Validation & Generation TEST MODEL GENERATOR #include "hierSequence.h" #include "PPLI.h" #include "RemoteTNdrop.h"   const port_t hierSeqDigraph::start=0; const port_t hierSeqDigraph::inJmsg=1; const port_t hierSeqDigraph::pass=2; const port_t hierSeqDigraph::outJmsg=3;   hierSeqDigraph::hierSeqDigraph():staticDigraph() { PPLI *pp = new PPLI(); add(pp); couple( this , this ->start, pp, pp->start); couple(pp, pp->outJmsg, this , this ->outJmsg); RemoteTNdrop *p1 = new RemoteTNdrop(); add(p1); couple( this , this ->start, p1, p1->start); couple( this , this ->inJmsg, p1, p1->inJmsg); couple(p1, p1->outJmsg, this , this ->outJmsg); } Test Case Test Model GENERATED TEST CASE MIRROR (XML)
  • Test Model Execution TEST DRIVER SYSTEM UNDER TEST Test Model (C++) #include "hierSequence.h" #include "PPLI.h" #include "RemoteTNdrop.h"   const port_t hierSeqDigraph::start=0; const port_t hierSeqDigraph::inJmsg=1; const port_t hierSeqDigraph::pass=2; const port_t hierSeqDigraph::outJmsg=3;   hierSeqDigraph::hierSeqDigraph():staticDigraph() { PPLI *pp = new PPLI(); add(pp); couple( this , this ->start, pp, pp->start); couple(pp, pp->outJmsg, this , this ->outJmsg); RemoteTNdrop *p1 = new RemoteTNdrop(); add(p1); couple( this , this ->start, p1, p1->start); couple( this , this ->inJmsg, p1, p1->inJmsg); couple(p1, p1->outJmsg, this , this ->outJmsg); }
  • Extending M&S-Based T&E to the GIG
    • Migrate ATC-Gen capability to the GIG-SOA
      • accessible to all developers requiring Link-16 conformance testing and certification
      • enable self-tests for Link-16 standards conformance
      • expose Link-16 XML-rule repository as authorized reference model
      • register ATC-Gen test capabilities as web services
    • Extend M&S-Based automated testing
      • provide mission thread testing for GIG deployable services
      • certify transitions through the stages of DISA’s Federated Distributed Certification Environment
    • Expose M&S-based T&E assets as services
      • experimental frame components: generators, acceptors, transducers
      • exploit DEVS reusability support
      • characterize services for discovery and reuse
  • Link-16 XML Repository Web-Service Asset
    • Organized according to MIL-STD-6016C macro-structure hierarchy
    • Folders store aggregations/abstractions of lower level rules
    • MIL-STD-6016C Reference Model
      • Removes ambiguity
      • Annotates problems areas, improving the ability to find and fix issues
      • Provides organization for indexing states, rules, and variables
      • Supports test generation and executable rule construction
  • Transferring DEVS-based Testing Methodology to SOA
    • DEVS-based Testing Methodology readily migrates to SOA environment as shown below
    • In particular, the ATC-Gen Test Driver can be offered as a service for Link-16 conformance self-test
    • Further, the methodology can be applied to other test and evaluation contexts
    DEVS Simulator DEVS Model Packaging:FOM Messaging:Interactions,Updates Communication: RTI HLA Service Discovery: UDDI DEVS Simulator DEVS Model Sevice Description: WSDL Packaging:XML Messaging:SOAP Communication: HTTP SOA
  • Automating Mission Thread/Vignettes for GES Testing
    • Objective – automate to the maximum extent the definition of mission threads/vignettes and the generation of test services based on them
    • Enable COI and community wide service users and developers to easily write threads/scenarios in (restricted) Natural Language form
    • Interacting with such users/developers, implement these scenarios with DEVS test federation models that can execute over Core ES
    • Mission thread/Vignette scenario testing applies objectives-driven testing to all service levels and throughout service offering life cycle
    • Repository can support re-use and re-application for regression testing to preserve service behavior under continual change in SOA environment
  • Mission Thread Testing Applies to all Service Levels and Throughout Service Offering Life Cycle Mission Threads enable objectives driven testing throughout service offering life-cycle CORE Enterprise Services COI Enterprise Services Community-wide Enterprise Services Mission Threads enable objectives driven testing at every development level
  • NLP Mission Thread Scenario Generation JTAC working with FourthInfDiv … . … USNAircraft gets deconfliction from USMCAircraft Parser outputs tagged equivalent of input Natural Language Parser Transform to System Entity Structure (actor = JTAC, action = work, actee = FourthInfDiv) … . (actor = JTAC, action = requests, message = ImmediatCAS, actee = DirectAirSuppAirborne) … . Restricted Natural Language form of Mission Thread Scenario Specification Scenario in XML Transform to DEVS DEVS Test Driver for NCES Mission Threads
  • Example: Joint Close Air Support Scenario
    • Aircraft get situational brief from AWACS aircraft while in route, then switch to SOF JTAC for Terminal Attack Control and deconfliction from orbiting UAV.  A 9-Line brief will be given to each section/single aircraft.  JTAC will continue to execute CAS missions until all weapons are expended.   
    C.      Joint Special Operations Task Force (JSOFT) approves the request and CAOC assigns a section of USMC F/A-18Ds, F-15Es, and a single B-1B.  Ordnance consists of 20mm, Joint Direct Attack Munitions (JDAMs), and Laser Guided Bombs (LGBs). B.      SOF JTAC contacts AWACS with request.  AWACS passes the request to Special Operations Liaison Element (SOLE) in the Combine Air Operations Center (CAOC).    A.     Special Operations Force (SOF) (AFSOC and NSW) JTAC working with Operational Detachment-Alpha (ODA) is tasked to request Immediate CAS on a stationary mechanized target in mountainous terrain.  A Predator unmanned aerial vehicle (UAV) is on station for support. JCAS JMT Operational Scenario #1
  • NLP Test Generation Example: Joint Close Air Support Scenario JTAC works with ODA! JTAC is supported by a Predator! JTAC requests ImmediateCAS to AWACS ! AWACS passes requestImmediateCAS to CAOC! CAOC assigns USMCAircraft to JTAC! CAOC sends readyOrder to USMCAircraft ! USMCAircraft sends sitBriefRequest to AWACS ! AWACS sends sitBrief to USMCAircraft ! USMCAircraft sends requestForTAC to JTAC ! JTAC sends TACCommand to USMCAircraft ! USMCAircraft sends deconflictRequest to UAV! USMCAircraft gets targetLocation from UAV! ! NCES
  • DEVS-based Web-Services Testing NCES/GIG DEVS Test Federation Live Test Player Service Under Test DEVS Simulator Node SOAP-XML DEVS Test Player
  • Demo: http://acims9.acims.arizona.edu Web Service: Find aircraft available for CAS given coordin-ates and time frame. D. Aircraft get situational brief from AWACS aircraft while in route, then switch to SOF JTAC for Terminal Attack Control and deconfliction from orbiting UAV.  A 9-Line brief will be given to each section/single aircraft.  JTAC will continue to execute CAS missions until all weapons are expended.    C.      Joint Special Operations Task Force (JSOFT) approves the request and CAOC assigns a section of USMC F/A-18Ds, F-15Es, and a single B-1B.  Ordnance consists of 20mm, Joint Direct Attack Munitions (JDAMs), and Laser Guided Bombs (LGBs). B.      SOF JTAC contacts AWACS with request.  AWACS passes the request to Special Operations Liaison Element (SOLE) in the Combine Air Operations Center (CAOC). A.     Special Operations Force (SOF) (AFSOC and NSW) JTAC working with Operational Detachment-Alpha (ODA) is tasked to request Immediate CAS on a stationary mechanized target in mountainous terrain.  A Predator unmanned aerial vehicle (UAV) is on station for support. JCAS JMT Operational Scenario #1
  • M&S-Based T&E Services on the SOA: Experimental Frame Repository
    • Compose EF from generator, acceptor, and transducer components in repository
    • Characterize functionalities using M&S Theory and WSDL
    • Support composability based on migration of DEVS technology to SOA
    Generator Repository Request Acceptor Repository Transducer Repository SUT Generator Acceptor Transducer Experimental Frame
  • Example: Synthetic Environment Repository Experimental Frames needed for Testing Objectives Environmental and Standards Reference Models available for use in Testing Experimental Frame Partial Order Experimental Frame- Model Applicability Relations
  • Summary and Conclusions
    • There is an acute need for a new Net-centric testing paradigm at the enterprise level where joint and coalition operations are conducted (Editorial, ITEA Jnl, Sept, 2005)
    • Net-Centric T&E places an increased emphasis on standards conformance self-testing over the GIG-SOA and emphasizes services rather than systems
    • Due to its formal and abstract basis, the DEVS functionality migrates quite directly to execute over the web-services middleware based on XML and SOAP
    • This extension enables semi-automated construction of DEVS test federations for verification testing of a wide variety of proposed web services:
      • ATC-Gen Link-16 and other TDL standards conformance self testing
      • Mission thread or vignette scenario specification using natural language input can replace Link-16 style standards formalization as the source input for test case generation
    • M&S-based T&E services can exploit experimental frame methodology
  • Bernard P. Zeigler [email_address] ACIMS www.acims.arizona.edu JITC jitc.fhu.disa.mil Contact: More information:
  • Joint Interoperability, Assured Security, Best Value, Customer Success Global Net-Centric Solutions -- The Warfighter's Edge