Your SlideShare is downloading. ×
IV&V Cox Overview
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

IV&V Cox Overview

589

Published on

Published in: Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
589
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Independent Verification & Validation Overview P3 PARTNERS IV&V METHODOLOGY – NASA PRESENTATION www.p3express.com 3600 Dallas Hwy, N.W., Suite 230 #126 Marietta, GA 30075 Toll-Free (877) 729-0990 Fax (404) 745-0990 Copyright – P3 Partners L.L.C., – 2005 – All rights reserved, Confidential Information, do not copy, reproduce, distribute or disclose
  • 2. What is IV&V? Independent Verification & Validation Overview
  • 3. IV&V Function
    • Software Independent Verification & Validation (IV&V) is a systems engineering process employing rigorous methodologies for evaluating the correctness and quality of the software product throughout the software life cycle. Software IV&V
    • is adapted to the characteristics
    • of the project.
  • 4. IV&V Characteristics
    • Includes Risk Identification and Mitigation Techniques
    • Provides Independent Evaluation / Assessment of:
      • Are we building the product right? = Verification
      • Are we building the right product? = Validation
    • Requires Technical, Managerial and Financial Independence
    • Makes a value added contribution, everyone shares the same mission success objective
      • For NASA Management - Provides Mission Assurance
      • For Project Management - Provides Unbiased Source of Help
    • Helps deliver
      • Risk Identification and Mitigation
      • Increased Quality and Safety
      • Improved Timeliness and Reliability
      • Reduced Rework Cost
    • NPD 8730.4: Requires NASA programs and projects that contain mission or safety critical software to document decisions concerning the use of IV&V.
  • 5. IV&V Defined
    • Independent
    • Technical: IV&V prioritizes its own efforts
    • Managerial: Independent reporting route to NASA Headquarters
    • Financial: Budget is allocated by NASA to the IV&V Facility such that IV&V effectiveness is not compromised
    • Verification (Are we building the product right?)
    • The process of determining whether or not the products of a given phase of the software development lifecycle fulfill the requirements established during the previous phase
    • The product is internally complete, consistent and correct will support the next phase
    • Validation (Are we building the right product?)
    • The process of evaluating software throughout its development process to ensure compliance with software requirements. This process ensures:
      • Expected behavior when subjected to anticipated events
      • No unexpected behavior when subjected to unanticipated events
      • System performs to the customer's expectations under all operational conditions
  • 6. IV&V and the SDLC
    • IV&V Process provides tools and analysis procedures appropriate to each phase of the Software Development Life Cycle (SDLC):
    • Concept Phase:
      • Verify candidate reuse software will satisfy the domain of the new system
      • Assess the proposed architectural schema for feasibility
      • Analyze the system requirements
    • Requirements Phase:
      • Verify that system and software requirements are correct, complete, traceable and testable
      • Verify test plans and acceptance criteria are sufficient to validate system requirements and operational needs
      • Ensure testing methods are sufficient to verify and validate software requirements
      • Verify the correct software development, management, and support processes are in place
    • Design Phase:
      • Verify the design will satisfy the requirements levied against it
      • Ensure test plans and test environments are sufficient to verify and validate software and operational requirements
      • Verify that the design does not have any characteristics that will cause it to fail under operational scenarios
  • 7. IV&V and the SDLC (cont.)
    • Coding Phase:
      • Verify the code reflects the design
      • Verify the code is correct
      • Verify that test cases trace to and cover software requirements and operational needs
      • Verify that software test cases, expected results, and evaluation criteria fully meet testing objectives
      • Analyze selected code unit test plans and results to verify full coverage of logic paths, range of input conditions, error handling, etc.
    • Test Phase:
      • Verify correct disposition of software test anomalies
      • Validate software test results versus acceptance criteria
      • Verify tracing and successful completion of all software test objectives
    • Operational Phase:
      • Verify that regression tests are sufficient to identify adverse impacts of changes
  • 8. IV&V Testing Philosophy
    • Developer is motivated to show software works (verification).
    • IV&V attempts to stress the software (verification & validation).
      • IV&V works to improve the project's test planning and does not typically focus on independent testing.
      • In some cases, IV&V may independently test highly critical software.
  • 9. IV&V Lifecycle Activities
    • Life-cycle IV&V is designed to mesh with the Project schedule and provide timely inputs to mitigate risk
    • Dialog between the IV&V Facility and the Project must begin before SRR
    • For most Projects, IV&V ends (and the Final Report is delivered) on or about MRR. Some Projects have extended S/W development post-launch or major upgrades/maintenance (e.g. Shuttle, MER)
    System Requirements Review Preliminary Design Review Critical Design Review System Test S/W FQT Initial IVVP Signed Mission Readiness Review Concept Phase 2.0 Requirements Phase 3.0 Design Phase 4.0 Implementation Phase 5.0 Test Phase 6.0 Operations & Maintenance Phase 7.0 Baseline IVVP Signed
    • - IV&V provides support and reports for Project milestones
    • Technical Analysis Reports document major phases
    • IVVP is updated to match changes in Project
    IV&V Provides CoFR IV&V Final Report IV&V Phase Independent Support 1.0 System Retirement Launch Note: numbers correspond to IV&V WBS
  • 10. Why IV&V? Independent Verification & Validation Overview
  • 11. The Problem …
    • In the 90's, the Commanding General of the Army's Operational Test and Evaluation Agency noted that 90 percent of systems that were not ready for scheduled operational tests had been delayed by immature software.
    Data provided by The Army’s Software Metrics Newsletter “Insight”, Winter 1997 http://www.armysoftwaremetrics.org/documents/INSIGHT/winter97.pdf
  • 12. Software Project Resolution
    • Successful Projects
      • Completed and operational, and:
        • On Schedule
        • On Cost
        • With all originally specified features and functions
    • Challenged Projects
      • Completed and operational, but:
        • Behind Schedule
        • Over Cost
        • With fewer features and functions than originally specified
    • Failed Projects:
      • Cancelled before completion or never implemented
    Project Resolution is commonly categorized into three resolution types:
  • 13. Software CHAOS
    • The Standish Group has examined 30,000 Software Projects in the US since 1994. This "CHAOS" research has revealed a decided improvement in IT project management with the implementation of standards and practices such as IV&V. This improvement correlates with the rise in project success depicted in the chart below:
    Project Resolution History (1994-2000) The Standish Group International, Inc.: Extreme CHAOS (2001) - The 2001 update to the CHAOS report. http://www.standishgroup.com/sample_research/PDFpages/extreme_chaos.pdf
  • 14. Error Introduction
    • The Carnegie Mellon Software Engineering Institute 1 reports that at least 42-50 percent of software defects originate in the requirements phase.
    • The Defense Acquisition University Program Manager Magazine 2 reports that a Department of Defense study that over 50 percent of all software errors originate in the requirements phase.
    1 – Carnegie Mellon Software Engineering Institute, The Business Case for Requirements Engineering, RE’ 2003, 12 September 2003 2 - Defense Acquisition University Program Manager Magazine, Nov-Dec 1999, Curing the Software Requirements and Cost Estimating Blues
  • 15. Error Detection/Correction
    • Early error detection and correction are vital. The cost to correct software errors multiplies during the software development lifecycle. Early error detection and correction reduce costs and save time.
    Direct Return on Investment of Software Independent Verification and Validation: Methodology and Initial Case Studies, James B. Dabney and Gary Barber, Assurance Technology Symposium, 5 June 2003.
  • 16. IV&V Benefits
    • Reduced frequency of operational change
    • Reduced maintenance cost
    • Second source technical alternative
    • Better decision criteria
    • Better visibility into development
    • Criteria for program acceptance
    • Compliance between specs & code
    • Higher confidence in software reliability
    • Better software/system performance
  • 17. NASA IV&V Policy
    • NASA will:
    • Establish and apply criterion, tools, and methodology to evaluate and assess software risk to identify appropriate level of IV&V
    • Task the NASA IV&V Facility in Fairmont, WV, to manage the performance of all IV&V for software in Provide Aerospace Products and Capabilities (PAPAC) programs and projects identified per the above criterion and any other safety critical software (as defined in NASA-STD-8719.13B )
    • Require NASA programs and projects that contain mission or safety critical software to document decisions concerning the use of IV&V
    • Responsibilities delineated for Chief Safety and Mission Assurance Officer, Chief Engineer, Chief Information Officers, Mission Office Associate Administrators (AAs), Governing Program Management Councils (GPMCs), and IV&V.
    • GPMCs will review results of the software IV&V process to assure that it meets project needs
  • 18. NASA IV&V Facility
    • NPD 2820.1C for Software IV&V Policy states: " Task the IV&V Facility in Fairmont, West Virginia to manage the performance of all IV&V for software identified per the established criteria, and for any other safety critical software (as defined in NASA-STD-8719.13) "
  • 19. How is IV&V Started? Independent Verification & Validation Overview
  • 20. IV&V Process Overview
    • NASA Headquarters determines the list of NASA missions slated to receive IV&V for each fiscal year and the priority for each mission.
    • The IV&V Facility prepares a rough order of magnitude IV&V cost estimate for the list of missions in support of budget formulation.
    • The IV&V Facility initiates a planning and scoping effort for each new mission to develop the following:
        • Lifecycle IV&V approach based on software criticality and risk
        • Startup Assessment Report (including a refined full lifecycle IV&V cost)
    • The IV&V Facility develops an IV&V Plan in accordance with the IV&V Work Breakdown Structure (WBS) and initiates IV&V activities.
    • The IV&V documented in the project plan is subject to IV&V Facility review.
  • 21. IV&V Process Overview (cont.)
    • The project's GPMC is responsible for overseeing the project's IV&V approach.
    • Differences between the project's approach and the IV&V Facility recommendation are resolved by the project's GPMC.
    • The IV&V Facility determines the distribution of resources between the project's development sites and the IV&V Facility.
    • IV&V covers all phases of the software development lifecycle.
    • IV&V status is reported to Project Offices, Office of Safety and Mission Assurance, and GPMCs.
  • 22. How is IV&V Performed? Independent Verification & Validation Overview
  • 23. Appropriate Level Of IV&V
    • Requirements, Design, Code, and Test Analysis for mission critical software functions:
      • Based on the Criticality Analysis
      • Provides appropriate IV&V breadth and depth across relative to the development phase for the mission’s most critical functions
      • Includes all software aspects of project (flight, ground, etc.)
    • System-level analysis of the acceptability of integrated system test plans and results
    To confidently support a decision to deploy the system, the following tasks are required to be performed: 
  • 24. Criticality Analysis
    • Criticality Analysis (CA) forms the foundation for IV&V
      • The results of CA are used to determine the set of tasks to be performed on each software component and to focus the emphasis and intensity of the IV&V effort on the most important areas.
    • The CA process assesses each software component against a set of criteria to determine its relative importance.
      • The criteria includes size, complexity, risk, and consequences of failure in accordance with NPD 2820.1C
      • The output of the CA process is called the Critical Functions List (CFL) and it documents the software integrity level of each component
    • More information on general CA processes and software integrity levels can be found in IEEE Std. 1012-1998
  • 25. Criticality Analysis and Planning Process Criticality Analysis Ranked Mission Critical Functions List IV&V Plan with critical software components and IV&V WBS elements to be performed
  • 26. IV&V Work Breakdown Structure Independent Verification & Validation Overview
  • 27.
    • The Work Breakdown Structure presented on the following slides represents the nominal set of tasks that will be performed on some part of the software in each mission type.
    Work Breakdown Structure Key Other systems that support the NASA mission such as Integrated Financial Management Data Analysis Where NASA is responsible for only an instrument, not the complete mission Instrument Missions that are not human rated Robotic Missions where human life is at risk (human-rated) Human
  • 28. Work Breakdown Structure X X X X Final Report Generation 1.3 X X X X Management and Planning of IV&V 1.1 X Traceability Analysis 2.6 X Software/User Requirements Allocation Analysis 2.5 X Concept Document Evaluation 2.4 X X System Requirements Review 2.3 X X X System Architecture Assessment 2.2 X X X X Reuse Analysis 2.1         Concept Phase 2.0 X X X X Identify Process Improvement Opportunities in the Conduct of IV&V 1.7 X X X X Criticality Analysis 1.6 X X X X Management and Technical Review Support 1.5 X X X X IV&V Tool Support 1.4 X X X X Issue and Risk Tracking 1.2         Phase Independent Support 1.0 Data Analysis Instrument Robotic Human IV&V Services
  • 29. Work Breakdown Structure (cont.) Data Analysis Instrument Robotic Human IV&V Services X Data Flow Analysis 4.8 X X Acceptance Test Plan Analysis 3.5 X X Component Test Plan Analysis 4.7 X X X X Database Analysis 4.6 X X X X Software Integration Test Plan Analysis 4.5 X X X X Software FQT Plan Analysis 4.4 X X X X Interface Analysis – Design 4.3 X X X X Software Design Evaluation 4.2 X X X X Traceability Analysis – Design 4.1         Design Phase 4.0 X Timing and Sizing Analysis 3.6 X X X X System Test Plan Analysis 3.4 X X X X Interface Analysis – Requirements 3.3 X X X X Software Requirements Evaluation 3.2 X X X X Traceability Analysis – Requirements 3.1         Requirements Phase 3.0
  • 30. Work Breakdown Structure (cont.) Data Analysis Instrument Robotic Human IV&V Services X X System Test Results Analysis 6.4 X Simulation Analysis 6.3 X Regression Test Analysis 6.2 X X X X Traceability Analysis – Test 6.1         Test Phase 6.0 X* Software FQT Procedure Analysis 5.12 X* System Test Procedure Analysis 5.11 X X Component Test Case Analysis 5.10 X X Software Integration Test Results Analysis 5.9 X X Software Integration Test Procedure Analysis 5.8 X X Acceptance Test Case Analysis 5.7 X X X X Software Integration Test Case Analysis 5.6 X X X X Software FQT Case Analysis 5.5 X X X X System Test Case Analysis 5.4 X X X X Interface Analysis – Code 5.3 X X X X Source Code and Documentation Evaluation 5.2 X X X X Traceability Analysis – Code 5.1         Implementation Phase 5.0
  • 31. Work Breakdown Structure (cont.) Test Phase 6.0 X X Software FQT Results Analysis 6-5 Data Analysis Instrument Robotic Human IV&V Services Component Test Design Analysis 8.6 Component Test Procedure Analysis 8.7 Component Test Results Analysis 8.8 Configuration Management Assessment 8.9 Disaster Recovery Plan Assessment 8.10 X Migration Assessment 7.3 Audit Support 8.5 Algorithm Analysis 8.4 Acceptance Test Results Analysis 8.3 Acceptance Test Procedure Analysis 8.2 Acceptance Test Design Analysis 8.1         Optional Tasks 8.0 X Retirement Assessment 7.4 X Anomaly Evaluation 7.2 X Operating Procedure Evaluation 7.1         Operations and Maintenance Phase 7.0
  • 32. Work Breakdown Structure (cont.) Feasibility Study Evaluation 8.11 Independent Testing 8.12 Data Analysis Instrument Robotic Human IV&V Services Software FTQ Design Analysis 8.17 Software Integration Test Design Analysis 8.18 System Test Design Analysis 8.19 Training Documentation Evaluation 8.20 Project Management Oversight Support 8.15 Security Assessment 8.16 Operational Evaluation 8.13 Performance Monitoring 8.14 User Documentation Evaluation 8.21         Optional Tasks 8.0
  • 33. Independent Verification & Validation Overview Summary
    • Software Independent Verification & Validation (IV&V) is a systems engineering process employing rigorous methods for evaluating the correctness and quality of the software product throughout the software life cycle
      • Software IV&V is executed across the full project life cycle
      • Software IV&V is an adaptive process based on the characteristics of a project
    • Software IV&V is a value-added approach to ensuring that software is fit for operations and meets its requirements for safety, availability and function with the shared goal of mission success with the project
    • Points of Contact
      • Redacted for online use.

×