Verification and validation are processes to ensure a software system meets user needs. Verification checks that the product is being built correctly, while validation checks it is the right product. Both are life-cycle processes applying at each development stage. The goal is to discover defects and assess usability. Testing can be static like code analysis or dynamic by executing the product. Different testing types include unit, integration, system, and acceptance testing. An effective testing process involves planning test cases, executing them, and evaluating results.

1. 10- Verification and Validation

2. • Assuring that a software system
meets a user’s needs
• Verification:
• “Are we building the product right”
– The software should conform to its specification.
• Validation:
• “Are we building the right product”
– The software should do what the user really requires.
Verification vs Validation

3. • Is a whole life-cycle process - V & V must be
applied at each stage in the software process.
• Has two principal objectives:
– The discovery of defects in a system.
– The assessment of whether or not the system is
usable in an operational situation.
The V & V Process

4. • Dynamic V & V Concerned with exercising
and observing product behavior (testing).
• Static verification Concerned with analysis
of the static system representation to
discover problems.
Dynamic and Static Verification

5. Static and Dynamic V&V
Formal
specification
High-level
design
Requirements
specification
Detailed
design
Program
Prototype
Dynamic
validation
Static
verification

7. Example

8. Testing – Terminology
• Test case
– Individual test
• Test suite
– Collection of test cases
• Test harness
– Program that executes a series of test cases
• Test framework
– Software that facilitates writing & running tests
– Example – JUnit

9. Testing – Terminology
• Test driver
– Program to create environment for running tests
– Declares variables, creates objects, assigns values
– Executes code and displays results of tests
• Tester (Quality Assurance)
– Person devising and / or performing tests
– More effective if 2nd person writes tests
• Walkthrough
– Programmer explains code to 2nd person

10. Types of Testing
• Program testing can be used to show the
presence of bugs, but never their absence.”
• Static: Done without actually executing program
– Code inspections
– Walkthroughs
• Dynamic: Done by executing program or its parts
– Module or unit testing
– Integration testing
– System testing

11. Who Tests the Software?
developer independent tester
Understands the system
but, will test "gently"
and, is driven by
"delivery"
Must learn about the system,
but, will attempt to break it
and, is driven by quality

12. Testing Principles
• All tests should be traceable to customer
requirements.
• Tests should be planned before testing begins.
• 80% of all errors are in 20% of the code.
• Testing should begin in the small and progress to
the large.
• Exhaustive testing is not possible.
• Testing should be conducted by an independent
third party if possible.

13. Software Defect Causes
• Specification may be wrong.
• Specification may be a physical impossibility.
• Faulty program design.
• Program may be incorrect.

14. Types of Errors
• Algorithmic error.
• Computation & precision error.
• Documentation error.
• Capacity error or boundary error.
• Timing and coordination error.
• Performance error.
• Recovery error.
• Hardware & system software error.
• Standards & procedure errors.

15. Test Strategies
• Black-box or behavioral testing
– knowing the specified function a product is to
perform and demonstrating correct operation based
solely on its specification without regard for its
internal logic
• White-box or glass-box testing
– knowing the internal workings of a product, tests
are performed to check the workings of all possible
logic paths

16. Black-box Testing
• Focus: I/O behavior. If for any given input, we can predict the
output, then the module passes the test.
– Almost always impossible to generate all possible inputs
("test cases")
• Goal: Reduce number of test cases by equivalence
partitioning:
– Divide input conditions into equivalence classes
– Choose test cases for each equivalence class. (Example: If
an object is supposed to accept a negative number, testing
one negative number is enough)

17. Black-box Testing (Continued)
• Selection of equivalence classes (No rules, only guidelines):
– Input is valid across range of values. Select test cases from
3 equivalence classes:
• Below the range
• Within the range
• Above the range
– Input is valid if it is from a discrete set. Select test cases
from 2 equivalence classes:
• Valid discrete value
• Invalid discrete value
• Another solution to select only a limited amount of test cases:
– Get knowledge about the inner workings of the unit being
tested => white-box testing

18. White-box Testing
• Focus: Thoroughness (Coverage). Every statement in
the component is executed at least once.
• Four types of white-box testing
– Statement Testing
– Loop Testing
– Path Testing
– Branch Testing

19. White-box Testing (Continued)
• Statement Testing (Algebraic Testing): Test single statements
(Choice of operators in polynomials, etc)
• Loop Testing:
– Cause execution of the loop to be skipped completely.
(Exception: Repeat loops)
– Loop to be executed exactly once
– Loop to be executed more than once
• Path testing:
– Make sure all paths in the program are executed
• Branch Testing (Conditional Testing): Make sure that each
possible outcome from a condition is tested at least once

20. • Unit testing
– testing of individual components
• Module testing
– testing of collections of dependent components
• Sub-system testing
– testing collections of modules integrated into sub-systems
• System testing
– testing the complete system prior to delivery
• Acceptance testing
– testing by users to check that the system satisfies
requirements. Sometimes called alpha testing
Testing Stages

21. The Testing Process
Sub-system
testing
Module
testing
Unit
testing
System
testing
Acceptance
testing
Component
testing
Integration testing User
testing

22. Unit Testing
• Program reviews.
• Formal verification.
• Testing the program itself.
– black box and white box testing.

23. Types of Testing
• Unit Testing:
– Individual subsystem
– Carried out by developers
– Goal: Confirm that subsystems is correctly coded
and carries out the intended functionality
• Integration Testing:
– Groups of subsystems (collection of classes) and
eventually the entire system
– Carried out by developers
– Goal: Test the interface among the subsystem

24. System Testing
• System Testing:
– The entire system
– Carried out by developers
– Goal: Determine if the system meets the requirements
(functional and global)
• Acceptance Testing:
– Evaluates the system delivered by developers
– Carried out by the client. May involve executing typical
transactions on site on a trial basis
– Goal: Demonstrate that the system meets customer
requirements and is ready to use
• Implementation (Coding) and testing go hand in hand

25. Eaxample:Test Cases
• Test case 1 : ? (To execute loop exactly
once)
• Test case 2 : ? (To skip loop body)
• Test case 3: ?,? (to execute loop more
than once)
These 3 test cases cover all control flow
paths

26. Generating Test Data
• Ideally want to test every permutation of valid
and invalid inputs
• Equivalence partitioning it often required to
reduce to infinite test case sets
– Every possible input belongs to one of the
equivalence classes.
– No input belongs to more than one class.
– Each point is representative of class.

27. Regression Testing
• Check for defects propagated to other modules by
changes made to existing program
– Representative sample of existing test cases is used to
exercise all software functions.
– Additional test cases focusing software functions likely to
be affected by the change.
– Tests cases that focus on the changed software components.

28. Integration Testing
• Bottom - up testing (test harness).
• Top - down testing (stubs).
• Modified top - down testing - test levels
independently.
• Big Bang.
• Sandwich testing.

29. Top-Down Integration Testing
• Main program used as a test driver and stubs are
substitutes for components directly subordinate to it.
• Subordinate stubs are replaced one at a time with real
components (following the depth-first or breadth-first
approach).
• Tests are conducted as each component is integrated.
• On completion of each set of tests and other stub is
replaced with a real component.
• Regression testing may be used to ensure that new
errors not introduced.

30. Bottom-Up Integration Testing
• Low level components are combined in clusters that
perform a specific software function.
• A driver (control program) is written to coordinate
test case input and output.
• The cluster is tested.
• Drivers are removed and clusters are combined
moving upward in the program structure.

31. Acceptance Testing
• Making sure the software works correctly for
intended user in his or her normal work
environment.
• Alpha test
– version of the complete software is tested by customer
under the supervision of the developer at the
developer’s site
• Beta test
– version of the complete software is tested by customer
at his or her own site without the developer being
present

32. System Testing
• Recovery testing
– checks system’s ability to recover from failures
• Security testing
– verifies that system protection mechanism prevents
improper penetration or data alteration
• Stress testing
– program is checked to see how well it deals with
abnormal resource demands
• Performance testing
– tests the run-time performance of software

33. Performance Testing
• Stress test.
• Volume test.
• Configuration test
(hardware &
software).
• Compatibility.
• Regression tests.
• Security tests.
• Timing tests.
• Environmental tests.
• Quality tests.
• Recovery tests.
• Maintenance tests.
• Documentation tests.
• Human factors tests.

34. Testing Life Cycle
• Establish test objectives.
• Design criteria (review criteria).
– Correct.
– Feasible.
– Coverage.
– Demonstrate functionality .
• Writing test cases.
• Testing test cases.
• Execute test cases.
• Evaluate test results.

35. Testing Tools
• Simulators.
• Monitors.
• Analyzers.
• Test data generators.

36. Test Team Members
• Professional testers.
• Analysts.
• System designers.
• Configuration management specialists.
• Users.

37. Test Documentation Needed
• Requirement being tested.
• Design verification methodology.
• Code verification methodology.

38. Document Each Test Case
• Requirement tested.
• Facet / feature / path tested.
• Person & date.
• Tools & code needed.
• Test data & instructions.
• Expected results.
• Actual test results & analysis
• Correction, schedule, and signoff.

39. Test Case Design
"Bugs lurk in corners
and congregate at
boundaries ..."
Boris Beizer
OBJECTIVE
CRITERIA
CONSTRAINT
to uncover errors
in a complete manner
with a minimum of effort and time

40. Debugging
• Debugging (removal of a defect) occurs as a
consequence of successful testing.
• Some people better at debugging than others.
• Is the cause of the bug reproduced in another
part of the program?
• What “next bug” might be introduced by the fix
that is being proposed?
• What could have been done to prevent this bug
in the first place?

41. The Debugging Process
Locate
error
Design
error repair
Repair
error
Re-test
program

42. Debugging Approaches
• Brute force
– memory dumps and run-time traces are examined for
clues to error causes
• Backtracking
– source code is examined by looking backwards from
symptom to potential causes of errors
• Cause elimination
– uses binary partitioning to reduce the number of
locations potential where errors can exist

43. End of lecture