Invited seminar on the GM-VV at SAAB Test & Evaluation Group's biannual meeting, Stockholm, January 2010.

1. Generic Methodology for Verification, Validation (GM-VV) to support the Acceptance of Models, Simulations and Data Constantinos GiannoulisPhD student (FOI) Stockholm, January 20th 2010

2. Agenda VV&A of M&S; What, why and where are we? The GM-VV Overview Basic concepts An international standard Why the GM-VV? Moving forward

3. VV&A of M&S; What, why and where are we?

5. The process which is used to construct, under a set of time, cost, skills, and organizational constraints a justified belief about model correctness

6. “Did I build the thing right?”

7. Validation

8. The process which is used to construct, under a set of time, cost, skills, and organizational constraints a justified belief about model validity

9. “Did I built the right thing?”

10. Acceptance / Accreditation

11. The process whereby the customer accepts that the M&S system is fit for its intended purpose

12. “Should it be used?”VV&A of M&S; What, why and where are we?

13. Why VV&A? “Engineering analysis (involving the Crater simulation) conducted during the flight concluded for NASA managers that although the foam might have caused some structural damage to the wing area, it would not have been sufficient to cause a catastrophic event.” R. Dittemore, Columbia mission manager, February 3, 2003 "We have found the smoking gun. The test we conducted ... demonstrates that this (foam debris) is in fact the most probable cause creating the breach that led to the accident of the Columbia and the loss of the crew and vehicle." S. Hubbard, Columbia Accident Investigation Board (CAIB) member, July 7, 2003 Crater Simulation Analysis “... the debris assessment team used the Crater software developed by Boeing Co. engineers. Crater is normally intended for prelaunch predictions about how small debris, usually ice, could damage the shuttle's external tank. The software is also used postlaunch to analyze divots in the shuttle's exterior tiles.” R. Edwards, FCW.com, September 8, 2003 (Boeing) “gave their findings to NASA on Jan. 23. Within the report, however, were uncertainties raised by the program: the foam could potentially cut a gouge deeper than the thickness of tile, though the report assured NASA that Crater was ‘conservative,’ that is, it tended to overestimate damage. The report emphasized the view that the tile would survive, and the engineers suggested that a more dense layer at the base of each tile would further blunt the effect of the foam. The mission management team quickly accepted the analysis the next day and moved on.” J. Schwartz, New York Times, August 25, 2003 “The use of Crater in this new and very different situation compromised NASA's ability to accurately predict debris damage in ways that Debris Assessment Team engineers did not full comprehend.” CAIB Report, August 2003 F6.3-11 Crater initially predicted tile damage deeper than the actual tile depth, but engineers used their judgment to conclude that damage would not penetrate the densified layer of tile. Similarly, RCC damage conclusions were based primarily on judgment and experience rather than analysis. Relevant CAIB Findings F6.3-18 After Program managers learned about the foam strike, their belief that it would not be a problem was confirmed (early, and without analysis) by a trusted expert who was readily accessible and spoke from "experience." No one in management questioned this conclusion. In the beginning, we thought that NASA had made a Type II error using an unvalidated simulation but found that they had made a Type I error instead.

14. Why VV&A ? Threat analysis Cognitive Human physiology VV&A of M&S; What, why and where are we? Transfer of Training Motion cueing Engineering Terrain Physics

16. MSCO

17. VPMM

18. DVDT

19. RPG

20. Templates

21. DoE

22. NASA

23. STD-7009

24. FR

25. DGA

26. ONERA

27. CA

28. SECO

29. NL

30. MoD

31. TNO

32. SE

33. FMV

34. FOI

35. UK

38. VPMM

40. 1516.4-2007

41. 1278.4-1997

43. Not adequately dealing with knowledge sharing and re-usability (formalized sets of products and processes)

44. Lack of tool support

45. Project based (VV&A on an Enterprise level is not supported)Confidence, objectivity, quality assessment, risks and costs VV levels and maturity (residual uncertainty, rigor, etc.) VV&A of M&S; What, why and where are we?

46. Current Status VV&A of M&S; What, why and where are we?

47. Current Status: Sweden Swedish Defence Material Administration (FMV) Fredrik Jonsson, fredrik.jonsson@fmv.se Håkan Lagerström, hakan.lagerstrom@fmv.se Swedish Defence Research Agency (FOI) Sten-Åke Nilsson, nil@foi.se Constantinos Giannoulis (PhD Student), constantinos.giannoulis@foi.se VV&A of M&S; What, why and where are we?

48. The GM-VV; An overview

49. The GM-VV matrix

50. The GM-VV; Basic concepts

52. Worldview The GM-VV; Basic concepts Structure

53. The 3 pillars The GM-VV; Basic concepts Structure

54. Organization The GM-VV; Basic concepts Structure

55. Organization The GM-VV; Basic concepts Structure

56. Product The GM-VV; Basic concepts Structure

57. Process The GM-VV; Basic concepts Structure

58. Acceptance The GM-VV; Basic concepts Property

60. The property of the relative satisfaction gained by the use of an M&S system, expressed in terms of value and cost

61. Validity (V)

62. The property of an M&S system to comply with the behaviour of the referent

63. Correctness (C)

64. The property of an M&S system to comply with formal rules and bodies of reference information for its representation and transformation of its representation into another oneThe GM-VV; Basic concepts Property

65. Concepts from SE to VV&A From framing systems through a modular and hierchical view to the M&S context Behavior

66. Argumentation framework Claim Argument Sub-Claim Sub-Claim Argument Evidence Evidence Argument Evidence Goal Strategy Sub-Goal Sub-Goal Strategy Strategy Acceptance Criteria Acceptance Criteria The GM-VV; Basic concepts Information

67. Semi formal specification An argumentation framework based on the argumentation interchange format ontology (AIF) The GM-VV; Basic concepts Information

68. Evidence Collection The GM-VV; Basic concepts Information

70. Tools

71. Documentation

72. Re-use

73. Standardization

74. ExchangeThe GM-VV; Basic concepts Information

75. The GM-VV Matrix design view Integration

76. The GM-VV; An international standard

77. Design rationale & assumptions Technical Based on ISO/IEEE Stds, INCOSE, etc. (eg.15288) Problem Frame Concept and Goal-Based Req. Eng. ISO/IEC 15026 Std on Safety Cases Systems Approach to M&S, Wymore, Zeigler & Traore AIF, Goal Structure Notation, and Claim, Argument, Evidence Networks Statistical, Bayesian, Constraint Satisfaction Programming,Kleijnen, Kelton, Sanchez, Whitt Data-base, Knowledge-Base, Data-Mining Concepts Contextual M&S Development and Technology Neutral Costumer and Goal Oriented M&S Acceptance from Employment Perspective Information and Product Centred Compliant and Complementary to System Eng. Standards Semi-formal Definitions to Support Tool Development Argumentation Based Acceptance Tailorable to Various Application Domains The GM-VV; An international standard

79. Provide a set of documents which will be proposed as a standard for a Verification and Validation methodology of data, models and simulations submitted to an appropriate international standardisation body

80. Outcome

81. The Generic Methodology for Verification, Validation (GM-VV) to support the Acceptance of Models, Simulations and DataThe GM-VV; An international standard

82. The GM-VV document architecture Handbook Help to understand the organisational component of a VV&A endeavour Provide a high level overview of the methodology (a walkthrough to the organization, process and products) Recommended Practice Guide Provide guidance on how to apply the GM-VV Reference Manual Provide the specification of the methodology Facilitate the development of tools supporting a VV&A effort Specify the concepts, components and interrelationships of the methodology The GM-VV; An international standard

83. Document mapping(e.g. VV&A Requirement Definition Process) HB (obligations) RPG (activities) Problem Owner’s Provide the VV&A preconditions specification Provide the VV&A requirements specification Contribute to the VV&A system of interest Contribute to the VV&A experimental frame Contribute to the VV&A results Provide VV&A Preconditions Identify the M&S Intended Use, the M&S Use Risks, M&S Requirements, M&S Constraints, the M&S System & VV&A results Provide VV&A Requirements Specify the VV&A Intended Use, VV&A Requirements & VV&A Constraints Provide a VVA-SoI Identify any observable Provide a VVA-EF Specify a VV&A EF (according to the VVA-SoI) Provide an Acceptance Goal (initiates the ToA). Identify the VV&A Intended Use meeting the Acceptance Goal & confidence required on meeting the Acceptance Goal The GM-VV; An international standard Acceptance Leader’s Provide the target of acceptance Contribute to the VV&A requirements specification Contribute to the VV&A system of interest Contribute to the VV&A experimental frame Contribute to the VV&A results ToA (initiated) VV&A Preconditions VV&A Requirements VV&A EF VV&A Results VV&A SoI

84. Why the GM-VV?

86. Improves the documentationof V&V efforts and encouraging uniformity

87. Provides support for policy making

88. Provides support as a contracting mechanism for VV&A projects and services

90. Deals with knowledge sharing, re-usability (formalized sets of products and processes) and traceability

91. Supports tool development

92. Is not project based and supports service view

94. Not adequately dealing with knowledge sharing and re-usability (formalized sets of products and processes)

95. Lack of tools support

96. Project based (VV&A on an Enterprise level is not supported)Confidence, objectivity, quality assessment, risks and costs VV levels and maturity (residual uncertainty, rigor, etc.) Why the GM-VV?

97. Moving forward

99. Build a network of expertise; involve more

100. Run Case Studies; apply, evaluate and improve the GM-VV Moving forward

101. Steps forward Existence of a standard under SISO; end of 2010 Development of VV&A support tools, patterns, guidance and training courses and materials Provide recommendations for specifics areas Network of expertise; FMV, FOI, academia (KTH, SU, Skövde Högskola, etc) & Industry VV&A as a service Introduce VV&A in M&S contracts How to implement VV&A in the MoD Process formal specification (e.g. BPMN) Support of service-oriented view Maturity improvement Decision making: Risks, uncertainties,… Resources, costs, time levels