Using VectorCAST to Satisfy FDA Software Testing Requirements
Using VectorCAST to Satisfy
FDA Software Testing Requirements
The VectorCAST family of products automates testing activities across the software development
lifecycle. The purpose of this paper is to provide a high-level view of what the VectorCAST
products do, how they work together, and how you might best use them to fully meet the software
testing requirements as specified in “General Principles of Software Validation; Final Guidance
for Industry and FDA Staff”, Section 5.2.5 “Testing by the Software Developer”.
The FDA guideline specifically addresses:
• Unit Testing
• Integration Testing
• System Testing
The following quotes are taken directly from Section 5.2.5:
• “Testing entails running software products under known conditions with defined inputs
and documented outcomes”
• “Essential element of test is the expected result”
• “Testing starts with unit test and concludes with system testing”
• “The amount of structural coverage is commensurate with the level of risk posed”
• “Regression analysis and testing are employed to provide assurance that a change has not
created a problem”
• “A software products testing can be organized into unit, integration, and system testing”
VectorCAST addresses all of the types of testing mentioned above and provides the automation
necessary to streamline the generation of all test artifacts. VectorCAST is an invaluable asset to
help your organization meet the stringent requirements of the FDA guideline.
Using VectorCAST to Automate FDA Software Testing 2
The VectorCAST product family consists of four complementary tools:
VectorCAST/C++ This tool automates the process of testing source modules written
in C or C++. It automates both unit testing and integration testing.
VectorCAST/Cover This tool performs code-coverage analysis during functional or
system test. It can share coverage data with VectorCAST/C++.
VectorCAST/RSP This tool serves as an extension to VectorCAST/C++. It allows
(Runtime Support you to execute test cases for real-time applications in an
Package) embedded-target or simulator environment.
Figure 1 shows a high-level view of how the VectorCAST tools address the full range of testing.
Unit / Integration
Testing System Testing
VectorCAST/C++ Data VectorCAST/Cover
Support for Unit /
Integration Testing of
Figure 1. How the VectorCAST tools apply to testing.
VectorCAST/C++ automates the key activities associated with software testing. Normally these
activities are time-consuming, expensive, and for these reasons often short-changed.
For software-development projects of any size or complexity, VectorCAST/C++:
Using VectorCAST to Automate FDA Software Testing 3
• Automatically generates an executable test harness on the basis of one or more
files of source code.
As illustrated in Figure 2, the executable harness generated by VectorCAST/C++
consists of a test driver, the source file(s) under test, any user-designated stubs for
dependent functions, and all unstubbed dependents. The test harness is data-driven,
meaning test data is read by the harness during execution. This approach precludes
any need of having to compile and link a new executable harness each time the same
tests are performed with different test cases.
• Builds test cases for stimulating the source code under test.
You can have VectorCAST/C++ generate complete test cases automatically, or you
can specify the input data and expected results for these. The test cases used (input
data and expected results) are automatically preserved for subsequent regression
VCAST builds all driver and stub code.
Individual Functions within a Unit
Unit(s) Under Stubs
Application Select a single unit or Select units to be stubbed,
Source Code multiple units for test or use real units
Figure 2. Automated generation of test harness.
• Executes Tests
You can use VectorCAST/C++ to execute tests in any of three ways: (1) on a host
machine running natively on a PC or UNIX box; and in conjunction with
VectorCAST/RSP, (2) to execute with a simulator or emulator; or (3) directly on an
You can execute tests from either a GUI or a command-line-interface (CLI).
Using VectorCAST to Automate FDA Software Testing 4
• Reports pass/fail results and code coverage
VectorCAST/C++ generates pass/fail reports on the tests performed, as well as
coverage reports on the associated code. VectorCAST/C++ can provide coverage
information at the statement, branch, and MC/DC levels, as well as at levels A, B, and
C for DO-178B. This report data serves to document the test project.
As illustrated in Figure 1, VectorCAST/C++ can share coverage data with
VectorCAST/Cover, and vice versa, to ensure 100% coverage
• Facilitates regression testing
VectorCAST/C++ maintains all the information it needs for automated regression
testing over the course of a developing project. There is no need to manually change
or redo test code, as was necessary previous to test automation.
VectorCAST/RSP allows you to use VectorCAST/C++ to conduct unit and integration tests on
real-time applications either on the target itself or in a simulator. VectorCAST/RSP integrates
with the target CLI to invoke the cross compiler and to establish an I/O conduit between the
target environment and VectorCAST/C++. Test execution is transparent to the user.
GUI Host/RSP Simulator
Download Tools RTOS
Figure 3. VectorCAST/RSP architecture.
In reference to Figure 3, you would use VectorCAST/C++ on a host platform (with RSP enabled)
to build a test harness for testing one or more units of application code. RSP would ensure proper
compilation, and would automatically download the executable harness into the target
environment. You would then use VectorCAST/C++ on the host platform to run test cases against
the application modules residing on the target. You would also use VectorCAST/C++ to report the
coverage of your tests.
Using VectorCAST to Automate FDA Software Testing 5
VectorCAST/Cover allows you to gauge the effectiveness of system testing by determining which
areas of an application have been exercised (or covered). You can perform coverage analysis on a
portion of an application or on the entire application.
The coverage information generated by VectorCAST/Cover differs from the information
generated by VectorCAST/C++. The coverage information generated by VectorCAST/Cover
derives from system or functional testing. These tests are usually application specific and
constructed to test high-level requirements.
The coverage information generated by VectorCAST/C++ derives from unit or integration testing.
These tests are typically constructed to determine whether algorithms are behaving as expected,
whether anomalous conditions are being tested, and whether all paths are being tested.
Usually it is not possible to test an entire application during system testing. For this reason, a
combination of unit, integration, and system test is necessary to achieve 100% coverage.
Ensuring 100% Coverage
Figure 4 illustrates two methods of using VectorCAST/Cover and VectorCAST/C++ to ensure
Method 1 – Traditional Approach
Unit Test Integration Test Functional/System Test
Driver Units Stubs Driver Units Stubs Full Application 100 % Coverage
Method 2 – Functional-Requirements Approach
Functional/System Test Unit Test
Driver Units Stubs
100 % Structural
60% - 70% 30% - 40%
Figure 4. Two methods of ensuring 100% coverage.
Using VectorCAST to Automate FDA Software Testing 6
Method 1 typically applies to the traditional approach to testing, in which the testing progression
is from individual components to aggregates, to the full system. Method 2 typically applies to
requirements-based testing, in which high-level system requirements are typically tested first,
followed by low-level unit or integration tests. Achieving 100% coverage is the goal of both
Under the first method, you would use VectorCAST/C++ to conduct unit and integration testing
on components and aggregates. In conjunction with this testing, you would use the built-in
coverage utility to determine and report on source coverage. At functional/system test, you would
use the VectorCAST/Cover tool (along with coverage data from VectorCAST/C++) to ensure
Under the second method, in conjunction with performing system testing, you would use
VectorCAST/Cover to identify any ‘holes’ in source coverage. You would then use
VectorCAST/C++ to augment the coverage by running test cases on the files of source code
containing ‘holes’. In addition, you would use VectorCAST/C++’s coverage-analysis utility
(along with coverage data from VectorCAST/Cover) to ensure 100% coverage.
(Note: VectorCAST/Cover supports the McCabe Cyclomatic Complexity Metric.)
VectorCAST Tool Qualification Requirement
The “qualification” of a software testing tool is necessary when test processes are eliminated,
reduced, or automated. VectorCAST automates several levels of testing during the software
Examples of VectorCAST test automation include:
• The generation of test harnesses for unit and integration testing
• Automatically comparing expected and actual results during regression testing
• Providing code coverage metrics during system, integration, and unit test
• Building test cases for boundary level conditions and for the basis path cases
VectorCAST Tool Qualification Process
To qualify VectorCAST for FDA projects, the following deliverables are produced by Vector
Software for each project:
Tool Operational Requirements document, which includes:
• VectorCAST functionality in verifiable requirements
• Project operational environment
• Configuration management process
• Method of attaining verification that VectorCAST has been satisfactorily
tested against specified requirements
Using VectorCAST to Automate FDA Software Testing 7
Tool Qualification document, which includes:
• Tool qualification test data and results for each requirement specified
The tool qualification process normally includes interaction with the qualifying user
as well as the associated certification authority if applicable.
Samples of these documents are available upon request.
This white paper is intended to serve as an introduction to the product literature available under
the Download option on www.vectorcast.com:
VectorCAST/C++ Unit and Integration Testing for C/C++
VectorCAST/Ada Unit and Integration Testing for Ada83 and Ada95
VectorCAST/Cover The Multi-Language Code Coverage Tool
VectorCAST/RSP Real-Time Embedded testing.