2010.11.04 - The GM-VV

482 views
416 views

Published on

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
482
On SlideShare
0
From Embeds
0
Number of Embeds
13
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

2010.11.04 - The GM-VV

  1. 1. Generic Methodology for Verification, Validation (GM-VV) to support the Acceptance of Models, Simulations and Data Constantinos Giannoulis PhD student (SU/FOI) Seminarium Kompetensnätverk VV&A 2010-11-04 FMV/SMART-lab
  2. 2. Agenda <ul><li>The GM-VV </li></ul><ul><ul><li>Overview </li></ul></ul><ul><ul><li>Basic concepts </li></ul></ul><ul><ul><li>An international standard </li></ul></ul><ul><li>Why the GM-VV? </li></ul><ul><li>NMSG 073 - Case Study </li></ul><ul><li>Reusability </li></ul><ul><li>The GM-VV currently </li></ul>
  3. 3. The GM-VV; An overview
  4. 4. VV&A of M&S; what? <ul><li>Verification </li></ul><ul><ul><li>The process which is used to construct, under a set of time, cost, skills, and organizational constraints a justified belief about model correctness </li></ul></ul><ul><ul><li>“ Did I build the thing right?” </li></ul></ul><ul><li>Validation </li></ul><ul><ul><li>The process which is used to construct, under a set of time, cost, skills, and organizational constraints a justified belief about model validity </li></ul></ul><ul><ul><li>“ Did I built the right thing?” </li></ul></ul><ul><li>Acceptance / Accreditation </li></ul><ul><ul><li>The process whereby the customer accepts that the M&S system is fit for its intended purpose </li></ul></ul><ul><ul><li>“ Should it be used?” </li></ul></ul>
  5. 5. Current status <ul><li>Existing methods and approaches are: </li></ul><ul><ul><li>Not generic, thus development and technology dependent </li></ul></ul><ul><ul><li>Not adequately dealing with knowledge sharing and re-usability (formalized sets of products and processes) </li></ul></ul><ul><ul><li>Lack of tool support </li></ul></ul><ul><ul><li>Project based (VV&A on an Enterprise level is not supported) </li></ul></ul><ul><ul><ul><li>Confidence, objectivity, quality assessment, risks and costs </li></ul></ul></ul><ul><ul><ul><li>VV levels and maturity (residual uncertainty, rigor, etc.) </li></ul></ul></ul>
  6. 6. Current Status
  7. 7. Design rationale & assumptions <ul><li>Technical </li></ul><ul><ul><li>Based on ISO/IEEE Stds, INCOSE, etc. (eg.15288) </li></ul></ul><ul><ul><li>Problem Frame Concept and Goal-Based Req. Eng. </li></ul></ul><ul><ul><li>ISO/IEC 15026 Std on Safety Cases </li></ul></ul><ul><ul><li>Systems Approach to M&S, Wymore, Zeigler & Traore </li></ul></ul><ul><ul><li>AIF, Goal Structure Notation, and Claim, Argument, Evidence Networks </li></ul></ul><ul><ul><li>Statistical, Bayesian, Constraint Satisfaction Programming, Kleijnen, Kelton, Sanchez, Whitt </li></ul></ul><ul><ul><li>Data-base, Knowledge-Base, Data-Mining Concepts </li></ul></ul><ul><li>Contextual </li></ul><ul><ul><li>M&S Development and Technology Neutral </li></ul></ul><ul><ul><li>Costumer and Goal Oriented </li></ul></ul><ul><ul><li>M&S Acceptance from Employment Perspective </li></ul></ul><ul><ul><li>Information and Product Centred </li></ul></ul><ul><ul><li>Compliant and Complementary to System Eng. Standards </li></ul></ul><ul><ul><li>Semi-formal Definitions to Support Tool Development </li></ul></ul><ul><ul><li>Argumentation Based Acceptance </li></ul></ul><ul><ul><li>Tailorable to Various Application Domains </li></ul></ul>
  8. 8. The GM-VV; Basic concepts
  9. 9. Worldview
  10. 10. The 3 pillars
  11. 11. Acceptance
  12. 12. Property: Acceptance <ul><li>Utility (U) </li></ul><ul><ul><li>The property of the relative satisfaction gained by the use of an M&S system, expressed in terms of value and cost </li></ul></ul><ul><li>Validity (V) </li></ul><ul><ul><li>The property of an M&S system to comply with the behaviour of the referent </li></ul></ul><ul><li>Correctness (C) </li></ul><ul><ul><li>The property of an M&S system to comply with formal rules and bodies of reference information for its representation and transformation of its representation into another one </li></ul></ul>
  13. 13. Evidence Collection
  14. 14. Argumentation framework Claim Argument Sub-Claim Sub-Claim Argument Evidence Evidence Argument Evidence Goal Strategy Sub-Goal Sub-Goal Strategy Strategy Acceptance Criteria Acceptance Criteria
  15. 15. Product
  16. 16. The GM-VV matrix
  17. 17. The GM-VV; An international standard
  18. 18. The GM-VV document architecture <ul><li>Volume I: Overview </li></ul><ul><ul><li>Provide a high level overview of the methodology (a walkthrough to the organization, process and products) </li></ul></ul><ul><li>Volume II: Implementation Guide </li></ul><ul><ul><li>Provide guidance on how to apply the GM-VV considering the organization, the process and the product and how to tailor them </li></ul></ul><ul><li>Volume III: Reference Manual </li></ul><ul><ul><li>Provide the specification of the methodology </li></ul></ul><ul><ul><li>Facilitate the development of tools supporting a VV&A effort </li></ul></ul><ul><ul><li>Specify the concepts, components and interrelationships of the methodology </li></ul></ul>
  19. 19. Concept mapping (e.g. VV&A Requirement Definition Process) HB (obligations) RPG (activities) <ul><li>Problem Owner’s </li></ul><ul><li>Provide the VV&A preconditions specification </li></ul><ul><li>Provide the VV&A requirements specification </li></ul><ul><li>Contribute to the VV&A system of interest </li></ul><ul><li>Contribute to the VV&A experimental frame </li></ul><ul><li>Contribute to the VV&A results </li></ul><ul><li>Acceptance Leader’s </li></ul><ul><li>Provide the target of acceptance </li></ul><ul><li>Contribute to the VV&A requirements specification </li></ul><ul><li>Contribute to the VV&A system of interest </li></ul><ul><li>Contribute to the VV&A experimental frame </li></ul><ul><li>Contribute to the VV&A results </li></ul><ul><li>Provide VV&A Preconditions </li></ul><ul><ul><li>Identify the M&S Intended Use, the M&S Use Risks, M&S Requirements, M&S Constraints, the M&S System & VV&A results </li></ul></ul><ul><li>Provide VV&A Requirements </li></ul><ul><li>Specify the VV&A Intended Use, VV&A Requirements & VV&A Constraints </li></ul><ul><li>Provide a VVA-SoI </li></ul><ul><li>Identify any observable </li></ul><ul><li>Provide a VVA-EF </li></ul><ul><li>Specify a VV&A EF (according to the VVA-SoI) </li></ul><ul><li>Provide an Acceptance Goal (initiates the ToA). </li></ul><ul><li>Identify the VV&A Intended Use meeting the Acceptance Goal & confidence required on meeting the Acceptance Goal </li></ul>VV&A Preconditions VV&A Requirements VV&A SoI VV&A EF ToA (initiated) VV&A Results
  20. 20. Why the GM-VV?
  21. 21. Current status <ul><li>Existing methods and approaches are: </li></ul><ul><ul><li>Not generic, thus development and technology dependent </li></ul></ul><ul><ul><li>Not adequately dealing with knowledge sharing and re-usability (formalized sets of products and processes) </li></ul></ul><ul><ul><li>Lack of tool support </li></ul></ul><ul><ul><li>Project based (VV&A on an Enterprise level is not supported) </li></ul></ul><ul><ul><ul><li>Confidence, objectivity, quality assessment, risks and costs </li></ul></ul></ul><ul><ul><ul><li>VV levels and maturity (residual uncertainty, rigor, etc.) </li></ul></ul></ul>
  22. 22. Why the GM-VV ? <ul><li>Is generic & tailorable, thus development and technology independent </li></ul><ul><li>Deals with knowledge sharing, re-usability (formalized sets of products and processes) and traceability </li></ul><ul><li>Supports tool development </li></ul><ul><li>Is goal based, rooted to GORE (i*, KAOS, GSN, etc.) </li></ul><ul><li>Supports beyond traditional project based VV&A (service) </li></ul><ul><li>Improves the documentation of V&V efforts and encouraging uniformity </li></ul><ul><li>Provides support for policy making </li></ul><ul><li>Elaborates VV&A requirements for V&V activities </li></ul><ul><li>Provides support as an agreement/contracting mechanism for VV&A projects and services </li></ul><ul><li>Supports confidence on the use of M&S means </li></ul>
  23. 23. The GM-VV; NMSG 073 Case Study Spring - Summer 2010
  24. 24. Case Study: Intended Use & research question <ul><li>FileProof </li></ul><ul><li>Rijkswaterstaat (2006-2008) aimed at: </li></ul><ul><ul><li>Immediate reduction of traffic jams by means of smart, innovative and practical approaches </li></ul></ul><ul><li>What are the effects of showing one or multiple flashing lights on: </li></ul><ul><ul><li>The safety of rescue workers on site, and </li></ul></ul><ul><ul><li>Traffic flow </li></ul></ul><ul><li>Effect of light color (blue or amber / yellow) </li></ul><ul><li>Effect of time of day (daylight / evening) </li></ul><ul><li>Effect of position accident (own lane or other direction) </li></ul><ul><li>Introduction </li></ul>
  25. 25. Case Study; Objectives <ul><li>Deliverables </li></ul><ul><ul><li>White Paper </li></ul></ul><ul><ul><ul><li>Authoritative & Educational Illustration </li></ul></ul></ul><ul><ul><ul><li>Benefits, Limitations & Improvements </li></ul></ul></ul><ul><ul><li>Introductory Tutorial Presentation </li></ul></ul><ul><ul><ul><li>For the M&S Community </li></ul></ul></ul><ul><ul><ul><li>Illustration & Guidance Application </li></ul></ul></ul><ul><li>Approach </li></ul><ul><ul><li>Focus on Technical Processes & Products </li></ul></ul><ul><ul><li>Spiral Application (iterative & incremental) </li></ul></ul><ul><li>Introduction </li></ul>
  26. 26. Scope & Execution <ul><li>Scope & Execution </li></ul><ul><li>The Case Study </li></ul>
  27. 27. The VV&A Goal Network
  28. 28. The Acceptance Goal
  29. 31. <ul><li>Estimated resources </li></ul><ul><ul><li>3 members </li></ul></ul><ul><ul><ul><li>Manfred Roza (NL) </li></ul></ul></ul><ul><ul><ul><li>Constantinos Giannoulis (SE) </li></ul></ul></ul><ul><ul><ul><li>Martin Adelantado (FR) </li></ul></ul></ul><ul><ul><li>13 workshop days per member </li></ul></ul><ul><ul><li>7 ”homework” days per member </li></ul></ul><ul><ul><li>4 nations (NL, SE, FR, DEU) </li></ul></ul><ul><li>Resources consumed </li></ul><ul><ul><li>5 members + guests </li></ul></ul><ul><ul><ul><li>Manfred Roza (NL) </li></ul></ul></ul><ul><ul><ul><li>Constantinos Giannoulis (SE) </li></ul></ul></ul><ul><ul><ul><li>Martin Adelantado (FR) </li></ul></ul></ul><ul><ul><ul><li>Karl-Gustav Pettersson (SE) </li></ul></ul></ul><ul><ul><ul><li>LtC Jens Zimmermann (DEU) </li></ul></ul></ul><ul><ul><li>13 workshop days per member </li></ul></ul><ul><ul><li>30+ ”homework” days (NL, SE) </li></ul></ul><ul><ul><li>4 nations (NL, SE, FR, DEU) </li></ul></ul><ul><li>Introduction </li></ul>Case Study; post application info
  30. 32. The GM-VV; Reusing VV&A of M&S
  31. 33. Example: Missile proximity sensor <ul><li>The proximity sensor: </li></ul><ul><ul><li>does not send detonation event until arm delay </li></ul></ul><ul><ul><li>can record a hit at a fixed point in its path </li></ul></ul><ul><ul><li>can hit the target aproaching at high speed (e.g. the target has an off set in y-direction of 10m (blast radius is 15m) </li></ul></ul><ul><ul><li>can correctly detect a target with constant accelaration ( parallel to the y-axis at 50 m/s² and has an off set in altitude close to the blast radius ) </li></ul></ul>
  32. 34. Example: Missile proximity sensor
  33. 35. Reusing VV&A of M&S <ul><li>Concrete parts of the goal network could be reused (relevant evidence and claims) </li></ul><ul><ul><li>Within similar context (SME) </li></ul></ul><ul><ul><li>Supported by the goal/claim network (tracebility) </li></ul></ul><ul><li>Generic branches of goals could be defined and reused (application specific requirements, value ranges, etc.) </li></ul><ul><li>Rigorous structuring and handling of information (goal & claim network) </li></ul><ul><li>Become a service provider for multiple projects (enrich maturity levels, improve cost) </li></ul>
  34. 36. The GM-VV currently
  35. 37. The GM-VV: Today <ul><li>NATO </li></ul><ul><ul><li>NMSG 073: standardization effort for the GM-VV </li></ul></ul><ul><li>France </li></ul><ul><ul><li>DGA: part of ITCS architecture (GM-VV Repository & case tools) </li></ul></ul><ul><ul><li>Case study: CBRN </li></ul></ul><ul><li>Canada </li></ul><ul><ul><li>SECO: implementation case study experiments (MALO, OC1) </li></ul></ul><ul><ul><li>Case study: Helmet (in collaboration with TNO) </li></ul></ul><ul><li>The Netherlands </li></ul><ul><ul><li>An independent M&S VV&A certification centre (feasibility study phase) </li></ul></ul><ul><ul><li>Case studies: Joint Fire effects trainer (planning phase), CBRN (planning phase) </li></ul></ul><ul><li>USA & EU </li></ul><ul><ul><li>Growing interests expressed by NASA & Airbus </li></ul></ul><ul><li>Sweden </li></ul><ul><ul><li>Build a network of expertise; involve more </li></ul></ul><ul><ul><li>Run Case Studies; apply, evaluate and improve the GM-VV </li></ul></ul>
  36. 38. Steps forward <ul><li>Existence of a standard under SISO; end of 2011 </li></ul><ul><li>Development of VV&A support tools , patterns, guidance and training courses and materials </li></ul><ul><li>Provide recommendations for specifics areas </li></ul><ul><li>Network of expertise; FMV, FOI, academia (KTH, SU, Skövde Högskola, etc) & Industry </li></ul><ul><ul><li>VV&A as a service </li></ul></ul><ul><ul><li>Introduce VV&A in M&S contracts </li></ul></ul><ul><ul><li>How to implement VV&A </li></ul></ul><ul><li>Maturity improvement (Case Studies) </li></ul><ul><li>Process formal specification (e.g. BPMN) </li></ul><ul><li>Support of service-oriented view </li></ul><ul><li>Decision making: </li></ul><ul><ul><li>Risks, uncertainties,… </li></ul></ul><ul><ul><li>Resources, costs, time </li></ul></ul><ul><ul><li>levels </li></ul></ul>
  37. 39. The GM-VV i Sverige <ul><li>Swedish Defence Material Administration (FMV) </li></ul><ul><ul><li>Fredrik Jonsson, [email_address] , [email_address] </li></ul></ul><ul><ul><li>Håkan Lagerström, [email_address] </li></ul></ul><ul><ul><li>Karl Gustav Pettersson , [email_address] </li></ul></ul><ul><li>Swedish Defence Research Agency (FOI) </li></ul><ul><ul><li>Sten-Åke Nilsson, [email_address] </li></ul></ul><ul><ul><li>Constantinos Giannoulis (Guest Doktorand - SU), [email_address] </li></ul></ul>
  38. 40. Vad kan vi göra tillsammans

×