Download to read offline
![V&VinDoD
• DoD focuses on “operational effectiveness,
operational suitability and survivability
(including cybersecurity), or lethality.”
[Defense Acquisition Guidebook (DAG),
Chapter 8]
• Organizationally, DoD divides into
Developmental Test and Evaluation (DT&E)
and Operational Test and Evaluation (OT&E)
• Development of a Test and Evaluation Master
Plan (TEMP) is considered critical
• DoD supports major test ranges and facilities,
worldwide
• An entire command focuses on
interoperability (Joint Interoperability Test
Command – JITC)
Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam](https://image.slidesharecdn.com/howtombsept-220602011055-06077d4a/75/How-to-Verify-and-Validate-a-System-or-Process-5-2048.jpg)
Uploaded byElizabeth Steiner
How to Verify and Validate a System or Process
The document summarizes a webinar by Dr. Steven Dam on verifying and validating systems, covering customer perspectives, requirements development, and the importance of planning for test and evaluation. It discusses the differences in verification and validation approaches used by organizations like DOD and NASA while emphasizing the need for traceability and comprehensive test planning. Additionally, it introduces tools such as Innosoft and Sopatra for enhancing the V&V process and managing test results effectively.
More Related Content
Similar to How to Verify and Validate a System or Process
More from Elizabeth Steiner
How to Verify and Validate a System or Process
- 1. HowtoVerifyandValidateaSystemorProcess Presented by: StevenH. Dam, Ph.D., ESEP – Part Four of the “How to MBSE” webinar series.
- 2. AskUsYourQuestions • Ask usyour questions using the panel on the right • This presentation is being recorded and will be made available to you. • Contact us after the webinar through • support@Innoslate.com • Call 571.485.7800 • LinkedIn Innoslate User Group • Twitter Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam
- 3. MeetYourHost • President andFounder of SPEC Innovations • Participated in the development of C4ISR and the DoDAF • Expert Systems Engineering Professionals Certificate • steven.dam@specinnovations.com • @stevenhdam Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam
- 4. WhatYouWillLearnToday • How DifferentCustomers View V&V • How to Develop Your V&V Requirements, V&V Plans, and Test Suites from the Beginning • How to Capture and Report Test Results in Innoslate® • How to Evaluate Processes using Sopatra® Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam
- 5. V&VinDoD • DoD focuseson “operational effectiveness, operational suitability and survivability (including cybersecurity), or lethality.” [Defense Acquisition Guidebook (DAG), Chapter 8] • Organizationally, DoD divides into Developmental Test and Evaluation (DT&E) and Operational Test and Evaluation (OT&E) • Development of a Test and Evaluation Master Plan (TEMP) is considered critical • DoD supports major test ranges and facilities, worldwide • An entire command focuses on interoperability (Joint Interoperability Test Command – JITC) Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam
- 6. V&VinNASA • According toNASA: • “Product verification proves that an end product (whether built, coded, bought, or reused) for any element within the system structure conforms to its requirements or specifications.” • “While verification proves whether ‘the product was done right,’ validation proves whether ‘the right product was done.’” • These definitions then map back to the DoD approach: • DT&E == Verification • OT&E == Validation • So, frequently different customers will frame V&V in different terms and have different processes and procedures, but the end goals are the same Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam From NASA Systems Engineering Handbook
- 7. EachOrganizationWillDefineTheirOwnProcessesforCM • Our processhas many of the same elements as the DoD and NASA processes • We include the use of our “SE Knowledgebase” as a key part of our process • We currently use Innoslate® as our primary SE Knowledgebase and tool • It can ingest or capture all the work products and may even be the source of many of those products • We can also use it to conduct analyses and create any necessary reports Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam SE Knowledgebase Review Test Results with ERB Develop Deployment Plan and Conduct Beta Testing Assess Risk and Plan Mitigation Updated SE Knowledgebase Return Item to “Integrate Solution” (if necessary) Beta Test Report Deficiencies? YES NO Workaround Identified Verify/Validate to Requirements Verification/ Validation Requirements From Design & Analysis Phase Perform Regression Testing Acceptable Regression Tests? YES NO Deployment Plan Submit Changes to CM Process Regression Test Report Inspection and Test Report Updated Test Plan
- 8. “HowtoMBSE”WebinarSeries: WhereDoesV&VFitintotheSystemLifecycle? Proprietary of SPECInnovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam Documents View Requirements View Modeling Simulation Risk Analysis Timeline WBS Design Engineering Integrations Test Center Sopatra EVERYWHERE!
- 9. HowtoDevelopYourV&VRequirements,V&VPlans, andTestSuitesfromtheBeginning Proprietary of SPECInnovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam
- 10. CreateVerification/ValidationRequirementsandDocuments • Start withValidation Requirements • Customer needs get translated into implementation independent requirements • Determine then how you will validate that these requirements have been met • May require a separate set of requirements • Once system requirements have been decomposed to specification level, create the verification requirements as well Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam
- 11. CreateTestPlansandTestSuites • Test Planscan be developed using built-in templates or create your won • Essential to be done early in the lifecycle to identify long lead items, such as test facilities and automated test equipment (ATE) • Preliminary Test Suites can also be created as verification requirements are developed • Just about everything can be planned well ahead of the actual V&V execution activities • This upfront planning ensures better requirements and much quicker V&V Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam
- 12. EnsureRequirementsTraceabilityEarly • The TraceabilityMatrix provides a way to trace requirements to Test Cases • The traceability assist provides a check on the probability that a requirement and test case should be connected Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam
- 13. IncludeVerificationMethodsforRequirements • Verification Methodsare identified as labels in Innoslate® • Analysis • Demonstration • Inspection • Modeling & Simulation • Test • Additional labels can be added, if needed by ensuring that the “Verification Method” label is used as a parent • These labels are provided at the Organization level Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam
- 14. HowtoCaptureandReportTestResultsinInnoslate® Proprietary of SPECInnovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam
- 15. UseTestCenterandTestSuitestoCaptureResults • Test Centerprovides a means to define Test Suites for each major V&V activity, using all methods (analysis, demonstration, inspection, simulation, test) • Innoslate® can support the simulation portion of V&V as well • Data recorded in files can be stored as Artifacts or a “File” attribute can be added to the Test Case class to associate files with each Test Case Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam
- 16. UseAlltheTestCaseAttributesandDiagrams • Test Casesare a subclass of Actions • This feature enables the Test case to be use in Action Diagrams, Timelines, and other project planning and execution activities • Test Cases have extra attributes to capture V&V related information • Expected Result • Actual Result • Setup • Add others as needed, e.g. • Result Link • File Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam
- 17. CreateTestReportsandVerificationMatrices • Test Reportsare another document type in Innoslate® • No templates are currently available • Add your own custom templates • Verification Matrices are reports in the Document View or can be developed in Database View Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam
- 18. HowtoEvaluateProcessesusingSopatra® Proprietary of SPECInnovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam
- 19. Sopatra®WasDesignedtoEvaluateProcessesandProcedures • Sopatra® translatesa text-based Standard Operating Procedure into an Action Diagram • It uses other information (input devices, output displays, acronyms, and time distribution databases) to better develop the process diagram • Best practice actions (Hear, See, Decide) are automatically added • Special metrics were defined to identify the Available Operational Time Window (AOTW) and Time of Procedure (ToP) to calculate the Procedure Buffer Time (PBT) Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam
- 20. Sopatra®’sMonteCarloSimulatorProvidestheanalysisResults • A welldeveloped procedure will show “all green” • Any failed runs will show up in “red” • Some procedures may be able to have an acceptable failure rate and the check for that will highlight the fail rate in red when exceeded Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam
- 21. LiveDemo Let’s dive onin. Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam
- 22. PredicttheFuture! A well calibratedsimulation can do this • Remember what we do and how we do it matters • High-quality analyses save projects, products, systems, and lives • Develop high-quality simulations by using Innoslate’s discrete event and Monte Carlo Simulators • Reduce operational errors by using Sopatra Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam
- 23. Questions and Answers Use thepanel on the right to ask your questions Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam
- 24. Next Webinar in “Howto MBSE” “Webinar: How to Verify and Validate” May 26, 2022, 11:00 am ET Go To Webinar Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam
- 25. MoreResources SPEC Innovations offerstraining, books, videos, documentation, trials, and more Training: specinnovations.com/training Books: “Real MBSE” textbook and lab manual available on Amazon Videos: Visit the SPEC Innovations YouTube channel LinkedIn: Innoslate and Systems Engineers User Group Documentation: help.Innoslate.com Trial: cloud.Innoslate.com Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam
- 26. SPEC Innovations offerstraining, books, videos, documentation, trials, and more Thank you! Visit cloud.innoslate.com for a trial. SPEC Innovations @Innoslate Innoslate User Group Innoslate.com/blog 571.485.7800 innoslate.com Proprietary of SPEC Innovations® 05/26/2022 at 11:00 am ET, Dr. Steven Dam
Editor's Notes
- #23 Exercise: Quality