The GM-VV

6,418 views

Published on

Invited seminar on the GM-VV at SAAB Test & Evaluation Group's biannual meeting, Stockholm, January 2010.

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
6,418
On SlideShare
0
From Embeds
0
Number of Embeds
80
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

The GM-VV

  1. 1. Generic Methodology for Verification, Validation (GM-VV)<br /> to support the Acceptance of Models, Simulations and Data<br />Constantinos GiannoulisPhD student (FOI)<br />Stockholm, January 20th 2010<br />
  2. 2. Agenda<br />VV&A of M&S; What, why and where are we?<br />The GM-VV<br />Overview<br />Basic concepts<br />An international standard<br />Why the GM-VV?<br />Moving forward<br />
  3. 3. VV&A of M&S; What, why and where are we?<br />
  4. 4. VV&A<br /><ul><li>Verification
  5. 5. The process which is used to construct, under a set of time, cost, skills, and organizational constraints a justified belief about model correctness
  6. 6. “Did I build the thing right?”
  7. 7. Validation
  8. 8. The process which is used to construct, under a set of time, cost, skills, and organizational constraints a justified belief about model validity
  9. 9. “Did I built the right thing?”
  10. 10. Acceptance / Accreditation
  11. 11. The process whereby the customer accepts that the M&S system is fit for its intended purpose
  12. 12. “Should it be used?”</li></ul>VV&A of M&S; What, why and where are we?<br />
  13. 13. Why VV&A?<br />“Engineering analysis (involving the Crater simulation) conducted during the flight concluded for NASA managers that although the foam might have caused some structural damage to the wing area, it would not have been sufficient to cause a catastrophic event.” R. Dittemore, Columbia mission manager, February 3, 2003<br />"We have found the smoking gun. The test we conducted ... demonstrates that this (foam debris) is in fact the most probable cause creating the breach that led to the accident of the Columbia and the loss of the crew and vehicle." S. Hubbard, Columbia Accident Investigation Board (CAIB) member, July 7, 2003<br />Crater Simulation Analysis<br />“... the debris assessment team used the Crater software developed by Boeing Co. engineers. Crater is normally intended for prelaunch predictions about how small debris, usually ice, could damage the shuttle's external tank. The software is also used postlaunch to analyze divots in the shuttle's exterior tiles.” R. Edwards, FCW.com, September 8, 2003<br /> (Boeing) “gave their findings to NASA on Jan. 23. Within the report, however, were uncertainties raised by the program: the foam could potentially cut a gouge deeper than the thickness of tile, though the report assured NASA that Crater was ‘conservative,’ that is, it tended to overestimate damage. The report emphasized the view that the tile would survive, and the engineers suggested that a more dense layer at the base of each tile would further blunt the effect of the foam. The mission management team quickly accepted the analysis the next day and moved on.” J. Schwartz, New York Times, August 25, 2003<br />“The use of Crater in this new and very different situation compromised NASA's ability to accurately predict debris damage in ways that Debris Assessment Team engineers did not full comprehend.” CAIB Report, August 2003<br />F6.3-11 Crater initially predicted tile damage deeper than the actual tile depth, but engineers used their judgment to conclude that damage would not penetrate the densified layer of tile. Similarly, RCC damage conclusions were based primarily on judgment and experience rather than analysis. <br />Relevant CAIB Findings<br />F6.3-18 After Program managers learned about the foam strike, their belief that it would not be a problem was confirmed (early, and without analysis) by a trusted expert who was readily accessible and spoke from "experience." No one in management questioned this conclusion. <br />In the beginning, we thought that NASA had made a Type II error using an unvalidated simulation but found that they had made a Type I error instead.<br />
  14. 14. Why VV&A ?<br />Threat analysis<br />Cognitive<br />Human physiology<br />VV&A of M&S; What, why and where are we?<br />Transfer of Training<br />Motion cueing<br />Engineering<br />Terrain<br />Physics<br />
  15. 15. Current status<br />National Initiatives<br /><ul><li>USA
  16. 16. MSCO
  17. 17. VPMM
  18. 18. DVDT
  19. 19. RPG
  20. 20. Templates
  21. 21. DoE
  22. 22. NASA
  23. 23. STD-7009
  24. 24. FR
  25. 25. DGA
  26. 26. ONERA
  27. 27. CA
  28. 28. SECO
  29. 29. NL
  30. 30. MoD
  31. 31. TNO
  32. 32. SE
  33. 33. FMV
  34. 34. FOI
  35. 35. UK
  36. 36. DSTL</li></ul>NATO<br /><ul><li>NMSG 054
  37. 37. NMSG 073</li></ul>SISO<br /><ul><li>SCM
  38. 38. VPMM
  39. 39. VV&A Overlay</li></ul>Europa MoU<br /><ul><li>REVVA 2</li></ul>ISO<br /><ul><li>15026</li></ul>VV&A of M&S; What, why and where are we?<br />IEEE<br /><ul><li>15288-2008
  40. 40. 1516.4-2007
  41. 41. 1278.4-1997
  42. 42. 1220-2005</li></li></ul><li>Current status <br />Existing methods and approaches are:<br /><ul><li>Not generic, thus development and technology dependent
  43. 43. Not adequately dealing with knowledge sharing and re-usability (formalized sets of products and processes)
  44. 44. Lack of tool support
  45. 45. Project based (VV&A on an Enterprise level is not supported)</li></ul>Confidence, objectivity, quality assessment, risks and costs<br />VV levels and maturity (residual uncertainty, rigor, etc.)<br />VV&A of M&S; What, why and where are we?<br />
  46. 46. Current Status<br />VV&A of M&S; What, why and where are we?<br />
  47. 47. Current Status: Sweden<br />Swedish Defence Material Administration (FMV)<br />Fredrik Jonsson, fredrik.jonsson@fmv.se<br />Håkan Lagerström, hakan.lagerstrom@fmv.se<br />Swedish Defence Research Agency (FOI)<br />Sten-Åke Nilsson, nil@foi.se<br />Constantinos Giannoulis (PhD Student), constantinos.giannoulis@foi.se<br />VV&A of M&S; What, why and where are we?<br />
  48. 48. The GM-VV; An overview<br />
  49. 49. The GM-VV matrix<br />
  50. 50. The GM-VV; Basic concepts<br />
  51. 51. The GM-VV concepts<br /><ul><li>Structure</li></ul>World view<br />The 3 pillars<br />VV&A life cycle (development, employment, etc.)<br /><ul><li>Property</li></ul>Acceptance<br />U,V,C<br /><ul><li>Behavior</li></ul>System of interest (SoI) observables<br />Experimental frames (EF)<br />Oracles, evidence, etc.<br /><ul><li>Information (Knowledge)</li></ul>Argumentation framework<br />Evidence collection (Goal & Claim networks)<br />Project & Corporate memory<br /><ul><li>Integration</li></ul>Implementation & Conformance<br />Project & service view (SOvvA)<br />The GM-VV; Basic concepts<br />
  52. 52. Worldview<br />The GM-VV; Basic concepts<br />Structure<br />
  53. 53. The 3 pillars<br />The GM-VV; Basic concepts<br />Structure<br />
  54. 54. Organization<br />The GM-VV; Basic concepts<br />Structure<br />
  55. 55. Organization<br />The GM-VV; Basic concepts<br />Structure<br />
  56. 56. Product<br />The GM-VV; Basic concepts<br />Structure<br />
  57. 57. Process<br />The GM-VV; Basic concepts<br />Structure<br />
  58. 58. Acceptance<br />The GM-VV; Basic concepts<br />Property<br />
  59. 59. Property: Acceptance<br /><ul><li>Utility (U)
  60. 60. The property of the relative satisfaction gained by the use of an M&S system, expressed in terms of value and cost
  61. 61. Validity (V)
  62. 62. The property of an M&S system to comply with the behaviour of the referent
  63. 63. Correctness (C)
  64. 64. The property of an M&S system to comply with formal rules and bodies of reference information for its representation and transformation of its representation into another one</li></ul>The GM-VV; Basic concepts<br />Property<br />
  65. 65. Concepts from SE to VV&A<br />From framing systems through a modular and hierchical view to the M&S context<br />Behavior<br />
  66. 66. Argumentation framework<br />Claim<br />Argument<br />Sub-Claim<br />Sub-Claim<br />Argument<br />Evidence<br />Evidence<br />Argument<br />Evidence<br />Goal<br />Strategy<br />Sub-Goal<br />Sub-Goal<br />Strategy<br />Strategy<br />Acceptance<br />Criteria<br />Acceptance<br />Criteria<br />The GM-VV; Basic concepts<br />Information<br />
  67. 67. Semi formal specification<br />An argumentation framework based on the argumentation interchange format ontology (AIF)<br />The GM-VV; Basic concepts<br />Information<br />
  68. 68. Evidence Collection<br />The GM-VV; Basic concepts<br />Information<br />
  69. 69. Project and Corporate Memory<br /><ul><li>Repository model:
  70. 70. Tools
  71. 71. Documentation
  72. 72. Re-use
  73. 73. Standardization
  74. 74. Exchange</li></ul>The GM-VV; Basic concepts<br />Information<br />
  75. 75. The GM-VV Matrix design view<br />Integration<br />
  76. 76. The GM-VV; An international standard<br />
  77. 77. Design rationale & assumptions<br />Technical<br />Based on ISO/IEEE Stds, INCOSE, etc. (eg.15288)<br />Problem Frame Concept and Goal-Based Req. Eng.<br />ISO/IEC 15026 Std on Safety Cases<br />Systems Approach to M&S, Wymore, Zeigler & Traore<br />AIF, Goal Structure Notation, and Claim, Argument, Evidence Networks<br />Statistical, Bayesian, Constraint Satisfaction Programming,Kleijnen, Kelton, Sanchez, Whitt<br />Data-base, Knowledge-Base, Data-Mining Concepts<br />Contextual<br />M&S Development and Technology Neutral<br />Costumer and Goal Oriented<br />M&S Acceptance from Employment Perspective<br />Information and Product Centred<br />Compliant and Complementary to System Eng. Standards<br />Semi-formal Definitions to Support Tool Development<br />Argumentation Based Acceptance<br />Tailorable to Various Application Domains<br />The GM-VV; An international standard<br />
  78. 78. REVVA2: The GM-VV<br /><ul><li>Objective of REVVA 2 (currently NMSG-073)
  79. 79. Provide a set of documents which will be proposed as a standard for a Verification and Validation methodology of data, models and simulations submitted to an appropriate international standardisation body
  80. 80. Outcome
  81. 81. The Generic Methodology for Verification, Validation (GM-VV) to support the Acceptance of Models, Simulations and Data</li></ul>The GM-VV; An international standard<br />
  82. 82. The GM-VV document architecture<br />Handbook<br />Help to understand the organisational component of a VV&A endeavour<br />Provide a high level overview of the methodology (a walkthrough to the organization, process and products)<br />Recommended Practice Guide<br />Provide guidance on how to apply the GM-VV<br />Reference Manual<br />Provide the specification of the methodology<br />Facilitate the development of tools supporting a VV&A effort<br />Specify the concepts, components and interrelationships of the methodology<br />The GM-VV; An international standard<br />
  83. 83. Document mapping(e.g. VV&A Requirement Definition Process)<br />HB (obligations)<br />RPG (activities)<br />Problem Owner’s<br />Provide the VV&A preconditions specification<br />Provide the VV&A requirements specification<br />Contribute to the VV&A system of interest<br />Contribute to the VV&A experimental frame<br />Contribute to the VV&A results<br />Provide VV&A Preconditions<br />Identify the M&S Intended Use, the M&S Use Risks, M&S Requirements, M&S Constraints, the M&S System & VV&A results<br />Provide VV&A Requirements<br /> Specify the VV&A Intended Use, VV&A Requirements & VV&A Constraints<br />Provide a VVA-SoI<br /> Identify any observable<br />Provide a VVA-EF<br /> Specify a VV&A EF (according to the VVA-SoI)<br />Provide an Acceptance Goal (initiates the ToA).<br /> Identify the VV&A Intended Use meeting the Acceptance Goal & confidence required on meeting the Acceptance Goal<br />The GM-VV; An international standard<br />Acceptance Leader’s<br />Provide the target of acceptance<br />Contribute to the VV&A requirements specification<br />Contribute to the VV&A system of interest<br />Contribute to the VV&A experimental frame<br />Contribute to the VV&A results<br />ToA<br />(initiated)<br />VV&A<br />Preconditions<br />VV&A<br />Requirements<br />VV&A<br />EF<br />VV&A<br />Results<br />VV&A<br />SoI<br />
  84. 84. Why the GM-VV?<br />
  85. 85. Why the GM-VV ?<br /><ul><li>Supports confidence on the use of M&S means
  86. 86. Improves the documentationof V&V efforts and encouraging uniformity
  87. 87. Provides support for policy making
  88. 88. Provides support as a contracting mechanism for VV&A projects and services
  89. 89. Elaborates VV&A requirements for V&V activities</li></ul>because<br /><ul><li>Is generic and tailorable, thus development and technology independent
  90. 90. Deals with knowledge sharing, re-usability (formalized sets of products and processes) and traceability
  91. 91. Supports tool development
  92. 92. Is not project based and supports service view
  93. 93. Is goal based</li></ul>Existing methods and approaches are:<br /><ul><li>Not generic, thus development and technology dependent
  94. 94. Not adequately dealing with knowledge sharing and re-usability (formalized sets of products and processes)
  95. 95. Lack of tools support
  96. 96. Project based (VV&A on an Enterprise level is not supported)</li></ul>Confidence, objectivity, quality assessment, risks and costs<br />VV levels and maturity (residual uncertainty, rigor, etc.)<br />Why the GM-VV?<br />
  97. 97. Moving forward<br />
  98. 98. The GM-VV: Today<br />NATO<br />NMSG 073: standardization effort for the GM-VV<br />France<br />DGA: part of ITCS architecture (GM-VV Repository & case tools)<br />Case study: CBRN<br />Canada<br />SECO: implementation case study experiments (MALO, OC1)<br />Case study: Helmet (in collaboration with TNO)<br />The Netherlands<br />An independent M&S VV&A certification centre (feasibility study phase)<br />Case studies: Joint Fire effects trainer (planning phase), CBRN (planning phase)<br />USA & EU<br />Growing interests expressed by NASA & Airbus<br /><ul><li>Sweden
  99. 99. Build a network of expertise; involve more
  100. 100. Run Case Studies; apply, evaluate and improve the GM-VV </li></ul>Moving forward<br />
  101. 101. Steps forward<br />Existence of a standard under SISO; end of 2010<br />Development of VV&A support tools, patterns, guidance and training courses and materials<br />Provide recommendations for specifics areas<br />Network of expertise; FMV, FOI, academia (KTH, SU, Skövde Högskola, etc) & Industry<br />VV&A as a service<br />Introduce VV&A in M&S contracts<br />How to implement VV&A in the MoD<br />Process formal specification (e.g. BPMN)<br />Support of service-oriented view<br />Maturity improvement<br />Decision making:<br />Risks, uncertainties,…<br />Resources, costs, time<br />levels<br />

×