This document provides a status update on GM-VV, a generic methodology for the verification and validation of modeling and simulation. It discusses the structure, objectives, and key concepts of GM-VV. GM-VV is designed to improve the quality and efficiency of VV&A processes. It aims to provide a common basis for understanding VV&A and a generally applicable yet tailorable methodology. The document outlines GM-VV's three-volume structure and discusses its customer-oriented and generic nature. The overarching goal of GM-VV is to demonstrate with sufficient confidence that a model or simulation fits its intended use.

1. Status update on GM-VV
Constantinos Giannoulis, DSV/FOI
Swedish VV&A Competence Network
2011-11-01
FOI

2. Agenda
 VV&A of M&S
 Status update on GM-VV
 Document Structure
 Objectives
 Rationale
 Concepts

3. Modeling & Simulation
 Abstracting and Approximating Reality

4. Ref:
VV&A W.F. Waite AEGIS TG
2004 JSMARTS vision 2020
Representation
Transformation
Abstraction

5. What‟s is the Issue?
 Which One is Acceptable for My Intended
Use and Which One is Not?
 Capabilities
 Uncertainty
 Use risks
 Need for Convincing Evidence
 M&S system capabilities and limitations
 Credible M&S results
 M&S system improvements

6. Acceptance
 Decision Making
 Confidence in success of M&S solution
 Justification decision and mitigation of risks
 M&S Solution Success
 Relative to intended use
 M&S system development dependent
 Decision Errors
 Falsely rejecting M&S system
 Falsely accepting M&S system
 Risks
 Reduced by evidence

7. Verification & Validation (V&V)
The objective of V&V is to collect, generate,
maintain and reason with a body of evidence to
support and justify a decision to use M&S

8. Verification & Validation (V&V)
 Verification
 Are you building or using the M&S right?
V&V is enhancing M&S return on investment!
 Validation
 Are you building or procuring the right M&S?
V&V is mitigating M&S use risks!

9. Crater Simulation Analysis
“... the debris assessment team used the Crater software
developed by Boeing Co. engineers. Crater is normally
intended for prelaunch predictions about how small
debris, usually ice, could damage the shuttle's external tank.
The software is also used postlaunch to analyze divots in the
shuttle's exterior tiles.” R. Edwards, FCW.com, September
8, 2003
(Boeing) “gave their findings to NASA on Jan. 23. Within the
report, however, were uncertainties raised by the program:
the foam could potentially cut a gouge deeper than the
thickness of tile, though the report assured NASA that Crater
“Engineering analysis (involving the Crater simulation) conducted was „conservative,‟ that is, it tended to overestimate damage.
during the flight concluded for NASA managers that although the The report emphasized the view that the tile would
survive, and the engineers suggested that a more dense
foam might have caused some structural damage to the wing
layer at the base of each tile would further blunt the effect of
area, it would not have been sufficient to cause a catastrophic the foam. The mission management team quickly accepted
event.” R. Dittemore, Columbia mission manager, February 3, 2003 the analysis the next day and moved on.” J. Schwartz, New
York Times, August 25, 2003
“The use of Crater in this new and very different situation
compromised NASA's ability to accurately predict debris
damage in ways that Debris Assessment Team engineers
did not full comprehend.” CAIB Report, August 2003
"We have found the smoking
gun. The test we conducted ... Relevant CAIB Findings
demonstrates that this (foam F6.3-11 Crater initially predicted tile damage deeper than the actual tile depth, but
debris) is in fact the most engineers used their judgment to conclude that damage would not penetrate the
probable cause creating the densified layer of tile. Similarly, RCC damage conclusions were based primarily on
breach that led to the accident judgment and experience rather than analysis.
of the Columbia and the loss of
F6.3-18 After Program managers learned about the foam strike, their belief that it
the crew and vehicle." S.
would not be a problem was confirmed (early, and without analysis) by a trusted expert
Hubbard, Columbia Accident
who was readily accessible and spoke from "experience." No one in management
Investigation Board (CAIB)
questioned this conclusion.
member, July 7, 2003

10. Current V&V efforts
NATO National SISO
 NMSG 054  USA  SCM
 NMSG 073 MSCO  VPMM
•VPMM  VV&A Overlay
•DVDT
Europa MoU •RPG
•Templates
 REVVA 2 DoE
NASA ISO
•STD-7009  15026
 FR
DGA
ONERA IEEE
 CA
SECO  15288-2008
 NL  1516.4-2007
MoD  1278.4-1997
TNO  1220-2005
 SE
FMV
FOI
 UK
DSTL

11. GM-VV
The Generic Methodology for Verification and
Validation of M&S: GM-VV

12. GM-VV

13. GM-VV: document structure
 The GM-VV Volume 1: Introduction & Overview
 The GM-VV Volume 2: Implementation Guide
 The GM-VV Volume 3: Reference Manual
The Generic Methodology for Verification & Validation
Is introduced and defined Provides guidance on
in implementation and use of
Learns the basics Provides technical
of GM-VV from GM-VV Vol 1. and referential
the Introduction & background
Overview information for
M&S/VVA GM-VV Vol 2.
Organizations Implementation
Apply GM-VV by using the
and Personnel guide
GM-VV Vol 3.
Reference
Obtain VV&A and related background information on and for use within GM-VV
Manual
from the

14. GM-VV
 Customer Oriented
 Generic
 Complete Methodology
 V&V Efficiency & Quality

15. GM-VV: objectives
 Improvement of V&V quality and cost-
efficiency (ROI)
 Common basis for communicating and
understanding V&V
 Generally applicable and tailorable V&V
methodology
 Basis for future V&V tools and techniques
development

16. Ref:
VV&A W.F. Waite AEGIS TG
2004 JSMARTS vision 2020
Representation
Transformation
Abstraction

17. GM-VV: rationale
 M&S based problem solving
 M&S considered as a system of systems
 M&S objective: to mimic or replicate another system
 M&S is part of larger context: frame system /
environment

18. GM-VV: rationale
 VV&A User/Sponsor
 Stakeholder inside M&S problem solving worlds
 Responsibility: acceptance decision on M&S assets
 What is “good enough” for intended use and the “risk”?

19. GM-VV: rationale
 VV&A World Scope
 Acceptance decision-support to VV&A User/Sponsor
 Delivers evidenced-based recommendation

20. GM-VV: concepts
 The Acceptance Goal for M&S
 Demonstrate with sufficient confidence that the M&S
system fits the intended use
 Four key activities to accomplish this goal
 Define acceptability criteria
 Demonstrate satisfaction by evidence
 Assess the evidential quality
 Build arguments for acceptance recommendation
claims

21. GM-VV: concepts
V&V Application or
GM-VV Related Application
Problem Specific or Problem Domain
V&V Conceptual
• Needs • Standards
• Budget Framework Guides
• Policies
• Time schedule concepts • Patterns
• Resources • Tools
• Risks • Techniques
• Human Skills V&V Implementation • Practices
• ... • Knowledge
Framework Basic input for
• ...
products, processes and roles
Accounts for V&V Tailoring Integrates & uses
Framework Produces
V&V Method and Application
Instances

22. Tack!
Frågor?
For more background information,
presentations, videos, and contact info,
please visit: http://vva.smart-lab.se/