Using VectorCAST to Satisfy
                  FDA Software Testing Requirements

The VectorCAST family of products automat...
Using VectorCAST to Automate FDA Software Testing                                                 2

The VectorCAST prod...
Using VectorCAST to Automate FDA Software Testing                                                            3

•   Autom...
Using VectorCAST to Automate FDA Software Testing                                                    4

•   Reports pass/...
Using VectorCAST to Automate FDA Software Testing                                                                   5

Using VectorCAST to Automate FDA Software Testing                                                     6

Method 1 typicall...
Using VectorCAST to Automate FDA Software Testing                                                    7

    Tool Qualifica...
Upcoming SlideShare
Loading in …5

Using VectorCAST to Satisfy FDA Software Testing Requirements


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Using VectorCAST to Satisfy FDA Software Testing Requirements

  1. 1. Using VectorCAST to Satisfy FDA Software Testing Requirements The VectorCAST family of products automates testing activities across the software development lifecycle. The purpose of this paper is to provide a high-level view of what the VectorCAST products do, how they work together, and how you might best use them to fully meet the software testing requirements as specified in “General Principles of Software Validation; Final Guidance for Industry and FDA Staff”, Section 5.2.5 “Testing by the Software Developer”. The FDA guideline specifically addresses: • Unit Testing • Integration Testing • System Testing The following quotes are taken directly from Section 5.2.5: • “Testing entails running software products under known conditions with defined inputs and documented outcomes” • “Essential element of test is the expected result” • “Testing starts with unit test and concludes with system testing” • “The amount of structural coverage is commensurate with the level of risk posed” • “Regression analysis and testing are employed to provide assurance that a change has not created a problem” • “A software products testing can be organized into unit, integration, and system testing” VectorCAST addresses all of the types of testing mentioned above and provides the automation necessary to streamline the generation of all test artifacts. VectorCAST is an invaluable asset to help your organization meet the stringent requirements of the FDA guideline.
  2. 2. Using VectorCAST to Automate FDA Software Testing 2 The VectorCAST product family consists of four complementary tools: VectorCAST/C++ This tool automates the process of testing source modules written in C or C++. It automates both unit testing and integration testing. VectorCAST/Cover This tool performs code-coverage analysis during functional or system test. It can share coverage data with VectorCAST/C++. VectorCAST/RSP This tool serves as an extension to VectorCAST/C++. It allows (Runtime Support you to execute test cases for real-time applications in an Package) embedded-target or simulator environment. Figure 1 shows a high-level view of how the VectorCAST tools address the full range of testing. Unit / Integration Testing System Testing Shared Coverage VectorCAST/C++ Data VectorCAST/Cover Support for Unit / Integration Testing of Embedded Apps VectorCAST/RSP Figure 1. How the VectorCAST tools apply to testing. VectorCAST/C++ VectorCAST/C++ automates the key activities associated with software testing. Normally these activities are time-consuming, expensive, and for these reasons often short-changed. For software-development projects of any size or complexity, VectorCAST/C++:
  3. 3. Using VectorCAST to Automate FDA Software Testing 3 • Automatically generates an executable test harness on the basis of one or more files of source code. As illustrated in Figure 2, the executable harness generated by VectorCAST/C++ consists of a test driver, the source file(s) under test, any user-designated stubs for dependent functions, and all unstubbed dependents. The test harness is data-driven, meaning test data is read by the harness during execution. This approach precludes any need of having to compile and link a new executable harness each time the same tests are performed with different test cases. • Builds test cases for stimulating the source code under test. You can have VectorCAST/C++ generate complete test cases automatically, or you can specify the input data and expected results for these. The test cases used (input data and expected results) are automatically preserved for subsequent regression testing. VCAST builds all driver and stub code. Units (.c, .cpp) Individual Functions within a Unit Test Driver Real Dependent Unit(s) Under Stubs Test Executable Test Harness Application Select a single unit or Select units to be stubbed, Source Code multiple units for test or use real units Figure 2. Automated generation of test harness. • Executes Tests You can use VectorCAST/C++ to execute tests in any of three ways: (1) on a host machine running natively on a PC or UNIX box; and in conjunction with VectorCAST/RSP, (2) to execute with a simulator or emulator; or (3) directly on an embedded target. You can execute tests from either a GUI or a command-line-interface (CLI).
  4. 4. Using VectorCAST to Automate FDA Software Testing 4 • Reports pass/fail results and code coverage VectorCAST/C++ generates pass/fail reports on the tests performed, as well as coverage reports on the associated code. VectorCAST/C++ can provide coverage information at the statement, branch, and MC/DC levels, as well as at levels A, B, and C for DO-178B. This report data serves to document the test project. As illustrated in Figure 1, VectorCAST/C++ can share coverage data with VectorCAST/Cover, and vice versa, to ensure 100% coverage • Facilitates regression testing VectorCAST/C++ maintains all the information it needs for automated regression testing over the course of a developing project. There is no need to manually change or redo test code, as was necessary previous to test automation. VectorCAST/RSP VectorCAST/RSP allows you to use VectorCAST/C++ to conduct unit and integration tests on real-time applications either on the target itself or in a simulator. VectorCAST/RSP integrates with the target CLI to invoke the cross compiler and to establish an I/O conduit between the target environment and VectorCAST/C++. Test execution is transparent to the user. GUI Host/RSP Simulator VectorCAST Compiler Interface Test Harness VectorCAST VectorCAST I/O I/O Monitor Download Tools RTOS Figure 3. VectorCAST/RSP architecture. In reference to Figure 3, you would use VectorCAST/C++ on a host platform (with RSP enabled) to build a test harness for testing one or more units of application code. RSP would ensure proper compilation, and would automatically download the executable harness into the target environment. You would then use VectorCAST/C++ on the host platform to run test cases against the application modules residing on the target. You would also use VectorCAST/C++ to report the coverage of your tests.
  5. 5. Using VectorCAST to Automate FDA Software Testing 5 VectorCAST/Cover VectorCAST/Cover allows you to gauge the effectiveness of system testing by determining which areas of an application have been exercised (or covered). You can perform coverage analysis on a portion of an application or on the entire application. The coverage information generated by VectorCAST/Cover differs from the information generated by VectorCAST/C++. The coverage information generated by VectorCAST/Cover derives from system or functional testing. These tests are usually application specific and constructed to test high-level requirements. The coverage information generated by VectorCAST/C++ derives from unit or integration testing. These tests are typically constructed to determine whether algorithms are behaving as expected, whether anomalous conditions are being tested, and whether all paths are being tested. Usually it is not possible to test an entire application during system testing. For this reason, a combination of unit, integration, and system test is necessary to achieve 100% coverage. Ensuring 100% Coverage Figure 4 illustrates two methods of using VectorCAST/Cover and VectorCAST/C++ to ensure 100% coverage. Method 1 – Traditional Approach Unit Test Integration Test Functional/System Test Coverage Coverage = Driver Units Stubs Driver Units Stubs Full Application 100 % Coverage Method 2 – Functional-Requirements Approach Functional/System Test Unit Test + = Driver Units Stubs 100 % Structural Coverage Coverage Coverage 60% - 70% 30% - 40% Achieved Achieved Figure 4. Two methods of ensuring 100% coverage.
  6. 6. Using VectorCAST to Automate FDA Software Testing 6 Method 1 typically applies to the traditional approach to testing, in which the testing progression is from individual components to aggregates, to the full system. Method 2 typically applies to requirements-based testing, in which high-level system requirements are typically tested first, followed by low-level unit or integration tests. Achieving 100% coverage is the goal of both methods. Under the first method, you would use VectorCAST/C++ to conduct unit and integration testing on components and aggregates. In conjunction with this testing, you would use the built-in coverage utility to determine and report on source coverage. At functional/system test, you would use the VectorCAST/Cover tool (along with coverage data from VectorCAST/C++) to ensure 100% coverage. Under the second method, in conjunction with performing system testing, you would use VectorCAST/Cover to identify any ‘holes’ in source coverage. You would then use VectorCAST/C++ to augment the coverage by running test cases on the files of source code containing ‘holes’. In addition, you would use VectorCAST/C++’s coverage-analysis utility (along with coverage data from VectorCAST/Cover) to ensure 100% coverage. (Note: VectorCAST/Cover supports the McCabe Cyclomatic Complexity Metric.) VectorCAST Tool Qualification Requirement The “qualification” of a software testing tool is necessary when test processes are eliminated, reduced, or automated. VectorCAST automates several levels of testing during the software verification process. Examples of VectorCAST test automation include: • The generation of test harnesses for unit and integration testing • Automatically comparing expected and actual results during regression testing • Providing code coverage metrics during system, integration, and unit test • Building test cases for boundary level conditions and for the basis path cases VectorCAST Tool Qualification Process To qualify VectorCAST for FDA projects, the following deliverables are produced by Vector Software for each project: Tool Operational Requirements document, which includes: • VectorCAST functionality in verifiable requirements • Project operational environment • Configuration management process • Method of attaining verification that VectorCAST has been satisfactorily tested against specified requirements
  7. 7. Using VectorCAST to Automate FDA Software Testing 7 Tool Qualification document, which includes: • Tool qualification test data and results for each requirement specified The tool qualification process normally includes interaction with the qualifying user as well as the associated certification authority if applicable. Samples of these documents are available upon request. Additional Resources This white paper is intended to serve as an introduction to the product literature available under the Download option on VectorCAST/C++ Unit and Integration Testing for C/C++ VectorCAST/Ada Unit and Integration Testing for Ada83 and Ada95 VectorCAST/Cover The Multi-Language Code Coverage Tool VectorCAST/RSP Real-Time Embedded testing.