The document discusses software test management and planning. It notes that errors found early in the development process are less costly to fix. A graph shows that errors discovered during maintenance are 368 times more expensive to fix than requirements errors. The document recommends optimizing the software process to find errors early. It also provides guidance on test planning, including designing for testability, defining metrics, covering all requirements with tests, and integrating the test plan into the project plan.

1. Software Architecture
Prof.Dr.ir. F. Gielen
Software Test Management
Vakgroep Informatietechnologie – IBCN

2. The relative cost of faults.
This graphs indicates the relative cost of correcting a fault or error:
An error introduced during the requirements analysis and discovered during implementation costs on the average
four times as much to correct as to when the error would have been discovered during a requirements review.
Errors discovered during maintenance (i.e. after the product has been put to operation) cost 368 times as much !
400 368
350
300
250
200
150
100 52
50 1 3 4 10
0 Design
Integration
Specification
Implementation
Maintenance
Requirements
Optimizing the software process to reduce costs
should focus on discovering possible errors early in the process.
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN p. 2

3. Test Planning Basics.
 Do this NOW !!!
 Test Design parallels Application Design
 Test requirements impact architecture: design for testability.
 Monitors
 Logging
 …
 The test plan should be an integrated part of the overall
project plan.
 The test plan must define how success will be measured
 Metrics are needed
 All requirements must have a test.
 Many different kinds of tests, some trivial, some not
 Degree of testing depends on risk considerations
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

4. Test Planning Process (ISO 29119)
Understand Scope
Context
Organise
Test Plan
Development
Identify & Analysed Risks
Analyze Risks
Treatment
Identify Risk
Approaches
Treatment
Approaches
Design Test
Strategy
Schedule,
Staffing Profile Determine
Staffing and Test
Draft Scheduling Strategy
Test Plan Document
Test Plan
Approved
Test Plan
Gain
Consensus on
Test Test Plan
Plan Publish
Test Plan

5. Creating a Test Plan
 Step 1: Requirements Analysis
 For each requirement choose test types, and methodologies ->
document it in the TEST PLAN
 Each requirement may need multiple test types and methodologies
 If you are having trouble choosing a test, the requirement may need to
be more clearly specified
 Functional Requirements have use-cases / scenarios
 Quality Requirements (e.g performance) may need additional
performance
specifications for test purposes
 A test is defined as a combination of the test types and the
methodologies chosen for testing the requirement.
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

6. Creating a Test Plan (Cont’d)
 Step 2: Prioritize
 For each test type, determine order (account for dependencies) and
priority (critical, important, moderate, low) based on risk
considerations.
 Step 3: Elaborate
 For each test, specify the details in a Test Description
 Step 4: Planning
 Account for test time/effort and other resource costs in the project plan.
 Don’t neglect the priories and risks.
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

7. Why prioritize test cases ?
Acceptance,
Risk Defect-density
Exposure testing
from (all defects equal)
Defects
Value/Risk-
Driven Testing
(80-20 value distribution)
Time to Ship (amount of testing)
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN p. 7

8. Test Types
There are many “types” of tests that can be
performed :
 Unit and Validation
 white-box
 black-box
 Integration
 Interface
 System
 Stress
 Regression
 Acceptance
 …
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

9. Unit and Validation Testing
• A unit test is any that is specific to some component of your
system (i.e. not the whole system).
• Validation tests determine how well a component corresponds
to its specification in the design.
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

10. Unit testing: White-box
 A white-box test uses knowledge of component’s
implementation to test it.
 A tester needs to be very familiar with the language and
technology.
 The tester reads the code, looking for mistakes or ways to
break it (boundary conditions, errors, limits).
 White-box testing is time-consuming, and requires much
skill. Thus it is often only done thoroughly on a per-
component basis (ie, as a unit test), and only for critical
components.
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

11. Unit testing: Black box
 A black-box test is not concerned with the
component’s implementation, but gauges whether it
accepts the correct inputs and produces correct
outputs.
 We can use equivalence partitioning to guide black-
box/validation testing:
 Since we rarely have time to try all the possible inputs, group them, and test
representatives from each group.
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

12. Black box testing (cont’d)
 Use Equivalence Partitioning to test the system
 group valid inputs together
 group invalid inputs together
 group questionable inputs together
 group outputs and identify erroneous groupings
 Focus on boundary values, and inputs which may be
strange or unexpected
 For a search routine, search for the first and last items &
 Search for the middle item
 Search for a non-existent item
 Search an empty database
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

13. Integration and Interface tests
 Interface tests focus on the points of contact
between two components. Interfaces include:
 Method invocations.
 Network or other stream-based communications.
 Shared memory, files and other system resources (and therefore,
resource locking).
 Integration tests see whether components that
work separately work together.
 More then just getting all of it to compile!
 Use Black Box methods, and a limited number of white box
methods to confirm that the sub-systems are interacting correctly
 They must be designed to exercise all of the use cases and error
conditions
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

14. System and Stress testing
 System tests test the entire assembled system.
 The system should be tested in a realistic environment, on it’s
actual hardware, by real users as well as the developers.
 System testing is often done in two phases, “Alpha” and “Beta”.
Beta release often go to many people outside the development
team.
 A stress test is a type of system test, where the
system’s performance limits are stretched (and
broken, usually).
 Stress tests are interested in gathering performance statistics, and
determining whether the system handles failure gracefully.
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

15. Regression Testing
Regression testing tests a new version of software
against a rigorously defined set of inputs and outputs, to
ensure the new version has introduced no
inconsistencies with old versions.
 Regression testing is one reason why we keep old tests
around (and document them): the old tests, passed once,
should be applied again with every significant change to the
system.
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN

16. Acceptance tests
 An acceptance test is any where some person or people (usually the
customer) decide whether an aspect of the software is acceptable, or not.
 Use acceptance tests when you have something concrete to test:
 At the ends of development cycles.
 With prototypes.
 With “beta” versions of your system, or components.
 Delivery to the customer.
 Acceptance tests return a boolean value: PASS/FAIL
 Watch out for “that’s nice, but….” responses from your customer (i.e.,
feature creep).
Vakgroep Informatietechnologie – Onderzoeksgroep IBCN