11 software testing_strategy

Software Testing Strategy

What kind of testing approach we should take to apply
the already developed test cases ?

13 January 2012 Made By Utpal Ray 1


 Strategic Approach
It is about applying designed test cases strategically to uncover
more number of errors in a shorter span of time.
Deciding a suitable test strategy calls for a suitable test plan and a
test procedure document.
Testing should begin at the component level ( unit level ) and move
‘outward’ towards integration of the entire computer based system.
Different testing techniques are required at different point of
testing span.
Testing should be performed by an Independent Test Group, who
did not contribute towards the design and development of the
software.
Any testing strategy must incorporate test planning, test case
design, test execution, test result collection, test result evaluation
etc.



 A typical Software Test Strategy
Considering the fact that testing is a big task, the entire
testing cycle can be divided into four different phase where
different test strategy is employed in different phase; also,
different test techniques are used in different phase.
The test target and the test focus also changes along this
four different phases of testing cycles.
The most interesting part is how error finding rate changes
as testing progresses along this four different phases and so
as the cost associated with fixing all those errors.
It is to be noted that test strategy for a newly developed
software ( which was developed from scratch ) will be
different than the test strategy for a moderately modified
software.



 A typical Software Test Strategy for a newly developed software

Test Phase Early Phase Integration Validation Final Phase
( Phase – I ) Phase Phase ( Phase – IV )
Scenario ( Phase – II ) ( Phase – III )

Test Unit Integration Validation System
Strategy Testing Testing Testing Testing

Test Mainly White White Box Black Box Black Box
Technique Box And Black Testing Testing
Testing Box Testing
Test Individual A group of The Whole The entire
Target Module Closely Software s/w in the
related operational
Module environment



 A typical Software Test Strategy for a newly developed software
( Contd. )
Scenario ( Phase – I ) Phase Phase ( Phase–IV )
( Phase – II ) ( Phase-III )
Testing Coding Design and Functional The s/w along
Focus the s/w Requirement with the other
architecture and other system elements
requirement

Error Finding
Rate



 A typical Software Test Strategy for a newly developed software (
Contd. )
Scenario ( Phase – I ) Phase Phase ( Phase–IV )
( Phase – II ) ( Phase-III )

Difficulty and
cost of fixing
those errors

The
responsible
persons for Individual Project Team The whole The whole
the errors Department Organization



 Unit Testing
It is done on the individual module level.
Component level design is used as a guide.
It is important to fix the scope of the unit testing.
The module interface needs to be examined and tested.
The boundary conditions need to be examined ( some sort of
black box testing ).
The basis path testing ( white box testing ) need to be
performed.
The error conditions, the error handlings and the error
messages – all these criteria need to be tested.



 Unit Testing ( contd. )
If it is necessary driver and suitable stubs need to be
developed and integrated with the modules ( illustrated
below ).
DRIVER

Test Cases Test Results

Module To be
Tested

STUB STUB



 Integration Testing
This is the testing phase after unit testing. This phase
deals with the combination of different modules.
Unless each of the module is unit tested
satisfactorily, one cannot enter this testing phase.
The best approach for integration testing is the
incremental integration; where modules are added one
after another. Integration testing is performed every
time after each of the module is added. This process of
testing is good enough to find out which module is
causing the error.
There are two distinct methods of incremental
integration – Top-Down integration and Bottom-Up
integration.



 Top-Down Integration
Modules are added by moving downward through the
control hierarchy, beginning with the main control
module ( top most executive module ) first.
The subordinate modules ( to the main module ) can
be added either in a breadth-first manner or in a
depth-first manner.
For the Top-Down integration, you may need stub to
effectively test the effect of module addition.



 Top-Down Integration ( contd. )
Breadth-First

M1

M2 M3 M4

M5 M6 M7 Sequence of Module Addition :-

M1, M2, M3, M4, M5, M6, M7, M8
M8



 Top-Down Integration ( contd. )
Depth-First

M1

M2 M3 M4

M5 M6 M7
Sequence of Module Addition :-
M1, M2, M5, M8, M6, M3, M7, M4
M8



 Bottom-Up Integration
Modules are added from the bottom; which means
modules from the lower most level of the ‘call and
return’ architecture are added first. Then the addition
process moves upwards.
The low level components are combined into multiple
clusters and each of this cluster is tested first using a
driver. While testing each of the cluster; modules are
added from the lower most level for each of this
cluster.
Once the clusters have been successfully tested; the
clusters are attached to the main branch ( after
removing the drivers ) and addition of modules
process continues. This way the integration testing is
performed on the whole ‘call and return’ arch.



 Bottom-Up Integration ( contd. )
An example of a ‘Call and Return’ Architecture

MA

MB MC MD

M11 M12 M21 M22 M31

M13 M23 M24 M32

M14 M25 M26

CLUSTER 1 CLUSTER 2 CLUSTER 3



Testing Cluster 1

DRIVER1

M14, M13, M12, M11
M11 M12

M13

M14



Testing Cluster 2

DRIVER2

M26, M25, M24, M22, M23, M21
M21 M22

M23 M24

M25 M26



Testing Cluster 3

DRIVER3

M32, M31
M31

M32



 Validation Testing
This phase is entered after the successful completion
of the integration testing.
The software validation is achieved through a series of
black box tests that demonstrate conformity with the
requirements.
One needs a test plan to identify a series of tests which
need to be used as a validation test. One needs a valid
test procedure also, regarding executing all those
individual tests.



 Validation Testing ( contd. )
The validation test ensures that the following criteria are
satisfied successfully :-
- All Functional Requirements
- All Behavioral characteristics
- All Performance Requirements
- All Documentation Correctness
- All Other Requirements ( Transportability,
Compatibility, Error Recoverability,
Maintainability etc. )
If some of the criteria mentioned above do not meet the
requirements, a deficiency list need to be prepared
identifying which one is deviating from the normal
reference.


 System Testing
It is the final phase of the testing cycle.
It is about integrating the software with hardware,
people and information.
The primary purpose of the system testing is to fully
exercise the computer based system.
The system test is really effective if it can be done in the
customer’s premises. Otherwise the customer’s
environment is re-created in-house.



 System Testing ( contd. )
There are quite a few varieties of system testing as
described below :-
- Recovery Testing [ How system recovers
from Faults? ]
- Security Testing [ How secure the system is? ]
- Stress Testing [ How much load the system can
take? ]
- Performance Testing [ How system performs in
normal condition and stresses condition ]



 Verification & Validation ( V&V )
One has to Validate the software to ensure that all the
functional requirements are met by the software.
One has to verify that one particular functionality has
been correctly implemented in the software.
Verification : Are we building the product right ?
Validation : Are we building the right product ?
Validation comes nearly at the end of the testing cycle
Verification may come throughout the testing cycle.



 Entry and Exit Criteria for testing
When to exit testing from the current test phase and
enter the next testing phase ?
Since the quote ‘You are never done with testing, the
burden simply shifts from the engineer to the
customer’, is always true; the decision to end testing
mainly depends upon the resources available and also
depends upon the confidence of the testing personnel.
Usually, the rate of finding errors drops as testing
progresses. When one cannot find any more errors;
that could be the time to exit the current phase and
enter the next phase.



 Sample Test Plan Template
1. Document Control
- Distribution
- Approvers/Reviewers
- Change History
2. Overview
- Project Summary
- Overall Test Goals and Objectives
3. Test Environment
- Hardware and Software Configuration
4. Test Tools To Be Used
5. Administration
- Test Assumptions and Dependencies
- Entry and Exit Criteria
- Status Tracking Approach
- Problem Reporting and Tracking Approach
- Maintenance Strategy
- Deliverables
6. Schedules
7. Test Matrices and Scenario



 Test plan template, IEEE 829 format

1.0 References
2.0 Introduction
3.0 Test Items
4.0 Features to be Tested
5.0 Features not to be Tested
6.0 Approach
7.0 Item Pass/Fail Criteria
8.0 Suspension Criteria and Resumption Requirements
9.0 Test Deliverables
10.0 Remaining Test Tasks
11.0 Environmental Needs
12.0 Staffing and Training Needs
13.0 Responsibilities
14.0 Schedule
15.0 Planning Risks and Contingencies
16.0 Approvals
17.0 Glossary



 Other Types of Testing Scenario
1. Alpha Testing
2. Beta Testing
3. Gamma Testing
4. Smoke Testing
5. Regression Testing
6. Automated Testing
7. Fuzz Testing
8. Static Testing



 Other Types of Testing Scenario ( contd. )
Alpha Testing
Different Software Organization has a different view
regarding definition of this test. This may be also
called ‘Engineering’ Test. Usually this test is attempted
after the integration testing is over. This test is
conducted in-house. This is a some kind of validation
test in which different in-house department may take
participation.



Beta Testing
This testing is attempted after the alpha testing is over.
This test is always conducted in the customer
premises; that’s why it is a case of ‘live’ testing. Usually
developers are not present during this kind of testing.
The customer records all problems that were
encountered during beta testing and reports this to
the developers at regular intervals.



Gamma Testing
Gamma testing is a little-known informal phrase that
refers derisively to the release of "buggy" (defect-
ridden) software. It is not a term of art among
testers, but rather an example of referential humor.
Cynics have referred to all software releases as "gamma
testing" since defects are always found in the customer
premises.



Smoke Testing
It is a ritual of daily testing.
It need not be an exhaustive testing rather it is an end
to end testing.
The test cases are designed keeping the frequently
performing functions in mind.
This is a one type of integration testing, where the
entire software is smoke tested daily.



Regression Testing
It is about running a subset of old test suite, to
make sure that the newly added changes did not
break the earlier functionality.
This is very much necessary when software moves
from one version to another version upwardly.
One has to run a regression test suite to make sure
the functionality of the earlier version did not
change even if the new functions has been added.



Automated Testing
Considering the fact that testing is a repetitive
process, most of the testing execution process can be
automated.
In it’s simplest form, an automated test suite does not wait
for user’s input; rather it takes the input from a already
prepared file. At the same time it does not leave the test
results for user’s interpretation but compares it with an
already prepared sample output file.
It is comparatively easier to write automated test suite
when input/output involves with keyboard and monitor. In
case of GUI testing, a different method of automated
testing should be employed.



Fuzz Testing
The basic idea is to attach the inputs of a program to a
source of random data ("fuzz"). If the program fails
(for example, by crashing, or by failing built-in code
assertions), then there are defects to correct.



Static Testing
Static Testing (also known as "Dry Run Testing") is a
form of software testing where the software isn't
actually used. Syntax checking and manually reading
the code to find errors are methods of static testing.
This type of testing is mostly used by the developer
himself (who designed or code the module). Static
testing is usually the first type of testing done on any
system.



Still More Varieties of Testing

Proposal Testing; Requirement Testing; Design Testing;
Big Bang Testing; Sandwich Testing; Complexity Testing;
Compatibility Testing; Security Testing; Performance Testing;
Volume Testing; Stress Testing; Recovery Testing; Installation Testing;
Error Handling Testing; Manual Support Testing; Intersystem Testing;
Control Testing; Sanity Testing; Adhoc Testing (Monkey Testing,
Exploratory Testing, Random Testing); Execution Testing;
Operations Testing; Compliance Testing; Usability Testing;
Decision Table Testing (Axiom Testing); Documentation Testing;
Training Testing; Rapid Testing; ‘COTS’ Testing; Client-Server Testing;
Web Application Testing; Mobile Application Testing;
eBusiness/eCommerce Testing; Agile Development Testing;
Data Warehouse Testing;

[ Courtesy; Software Testing – M G Limaye; McGraw-Hill



 Home Task
1. Why is a highly coupled module difficult to unit test
?
2. Develop an integration testing strategy for the
problems which you have already got the
architectural design.
3. Who should perform the validation test – the
software developer, an independent tester or the
software user ? Justify your answer.


11 software testing_strategy

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (17)

Similar to 11 software testing_strategy

Similar to 11 software testing_strategy (20)

More from University of Computer Science and Technology

More from University of Computer Science and Technology (20)

Recently uploaded

Recently uploaded (20)

11 software testing_strategy