BH0-003
ISEB Foundation Certificate in Software Testing Certification
Exam: BH0-003
Edition: 1.0
C
C CERT MAGIC
1 http://www.certmagic.com
BH0-003
Table of Contents
StudyGuide 1
SOFTWARE TESTING........................................................................ 10
THE DEFINITION: 10
GOALS OF TESTING: 11
THE EIGHT BASIC PRINCIPLES OF TESTING: 11
Regression Testing 13
Using Bugs ....................................................................................... 14
More about using bug reports ............................................................. 14
Using Product Structure ..................................................................... 16
A good tester builds a mental model of product internals. ....................... 16
Using the User .................................................................................. 19
A good tester understands how users spend their days. .......................... 19
Getting there .................................................................................... 20
Reprise: Testing Problems .................................................................. 21
Failure to handle code details.............................................................. 23
Mistakes applying programming clichés ................................................ 24
Misunderstanding the environment ...................................................... 25
Domain Testing: .............................................................................. 26
Software Testing: 26
Test Case: 27
Bug/Error/Fault: 27
1.1.1 White -Box Testing Approach .................................................... 28
Black-Box Testing Approach................................................................ 28
Complete Testing/Exhaustive Testing ................................................... 28
Domain Testing................................................................................. 29
Similarities among Different Interpretations of Domain Testing ................ 30
Differences between Different Interpretations of Domain Testing ............. 30
Domain Testing: A Literature Review ................................................... 32
Partitioning of Input Domain ............................................................... 33
White or Black: ................................................................................. 33
Driving Factor:.................................................................................. 33
Type of Domain: ............................................................................... 33
Overlapping or Disjoint Subdomains: ................................................... 34
Size of Subdomains: .......................................................................... 34
White or Black .................................................................................. 34
White-Box Testing Approach ............................................................... 34
Black-Box Testing Approach/ Specification-Based Approach..................... 37
Combination of Black-Box and White-Box Testing Approaches ................. 40
Driving Factor ................................................................................... 40
Confidence-Based Approach................................................................ 41
Risk-Based Approach ......................................................................... 41
Linear or Non-Linear .......................................................................... 43
2 http://www.certmagic.com
BH0-003
Overlapping or Disjoint Subdomains..................................................... 43
Size of Subdomains – Equally Sized or Unequally Sized? ......................... 44
Selecting Representatives from Subdomains ......................................... 44
Which Test Case Selection Method Should We Use?................................ 51
Testing Multiple Variables in Combination 52
Cause-Effect Graphs .......................................................................... 52
Pairwise / Orthogonal Arrays Testing.................................................... 53
Combinatorial Testing Using Input-Output Analysis ................................ 54
All Pairs Combination ......................................................................... 54
Weak Robust Equivalence -Class Testing .............................................. 55
When to Use What Combination Technique? .......................................... 55
Testing and Process Recommendations for Software Engineering... 56
Software Engineering Problems 56
Requirements Documentation 58
Source Control of Tools, Deliverables, and Documentation 59
Bug Tracking 61
Written Test Scripts and Test Automation 63
Testing Strategy .............................................................................. 66
Verification & Validation 66
Testing – Techniques 69
Success with Test Automation ......................................................... 77
Taking Test Automation Seriously 78
Who Should Automate Tests? 79
Choosing What To Automate 80
Building Maintainable Testsuites 83
Building Test Interpreters 84
Keeping Your Testsuite Reliable 86
Using Error Recovery Systems 87
Living with Test Automation 88
Structurally Guided Black Box Testing ............................................. 90
Control FlowAdequacy Criteria 93
Data Flow Adequacy Criteria 93
The Guided Approach 94
Software Development/Test Phases.............................................. 102
Software Test Methodology 104
Requirements phase Tools 105
Software Testing Fundamentals 110
White Box Testing 110
1.1.2 The Nature of Software Defects ................................................110
Basis Path Testing ............................................................................110
Loop Testing....................................................................................112
Other White Box Techniques ..............................................................113
Black Box Testing 113
Equivalence Partitioning ....................................................................114
Boundary Value Analysis ...................................................................114
Cause-Effect Graphing Techniques ......................................................115
GUI Testing Checklist .................................................................... 115
Section 2 - Screen Validation Checklist 118
3 http://www.certmagic.com
BH0-003
General Conditions: ..........................................................................121
Validation Testing - Standard Actions 124
Overview of System X 128
Objectives of System Test 129
Software Quality Assurance involvement 130
Scope of Test Approach - System Functions 131
EXCLUSIONS ...................................................................................131
Testing Process 132
Testing Scope 133
Functional Testing ............................................................................133
Integration Testing ...........................................................................133
Business (User) Acceptance Test ........................................................134
Performance Testing .........................................................................134
Regression Testing ...........................................................................134
Bash & Multi-User Testing..................................................................134
Technical Testing .............................................................................134
Operations Acceptance Testing (OAT)..................................................134
System Test Entrance/Exit Criteria 135
Entrance Criteria ..............................................................................135
Exit Criteria .....................................................................................135
TEST PHASES AND CYCLES ............................................................ 136
System Testing Cycles 136
Software Delivery 137
Formal Reviewing138
Test Schedule ................................................................................ 139
System Test Schedule 139
RESOURCES ................................................................................... 140
Human 140
Resource Type .................................................................................140
Resource Title..................................................................................140
No. 140
Date Req'd ......................................................................................140
Who 140
Status 140
Hardware 141
Hardware components required ..........................................................142
Software 143
Test IMS environments .....................................................................143
Test Environment Software................................................................143
Error Measurement System ...............................................................143
ROLES AND RESPONSIBILITIES .................................................... 143
Management Team 143
Testing Team 143
Business Team 144
Testing Support Team 144
External Support Team 144
Error Management & Configuration Management .......................... 145
STATUS REPORTING ...................................................................... 146
4 http://www.certmagic.com
BH0-003
Status Reporting 146
Issues, Risks and Assumptions 146
Assumptions 147
Formal Signoff 147
Purpose of Error Review Team. 148
Error Review Team Meeting Agenda. 148
Classification of Bugs 149
Procedure for maintenance of Error Management system. 149
Overnight Processing - Checking Accounting & Audit & CIS 150
SOFTWARE QUALITY ASSURANCE MEASURES................................ 151
(i) DATES. 151
(ii) EFFORT. 151
(iii) VOLUME. 151
(iv) QUALITY. 151
(v) TURNAROUND. 151
Test Project Setup Checklist .......................................................... 151
Item 151
Status 151
Who 151
2. Test Preparation ...........................................................................152
3. Build System Test Environment ......................................................152
4. Prepare System Test .....................................................................153
5. Execute System Test.....................................................................153
6. Signoff ........................................................................................153
Software Quality Assurance & Usability Testing ............................ 154
3 Level Error Classification Method................................................ 158
Explanation of Classifications .............................................................159
5 Level Error Classification Method................................................ 159
The Importance of Scalability & Load Testing 163
Preparing for a Load Test 164
Measuring Current Load Levels...........................................................164
Concurrent Users .............................................................................165
Estimating Test Duration ...................................................................165
Ramp-up Rate .................................................................................166
Script Execution ...............................................................................167
System Performance Monitoring .........................................................168
Suggested Execution Strategy: ..........................................................169
Results Analysis ...............................................................................169
The Rise and Rise of Outsourcing .................................................. 170
Why choose to Outsource Testing? 170
The Benefits of Outsourcing 171
The First Steps to Successful Outsourcing 172
Choosing an Outsourced Testing Partner 173
Risk Planning for Outsourced Testing 174
Managing Outsourced Testing Projects 174
Unit Test Tools .............................................................................. 179
Rational Test RealTime Unit Testing ....................................................182
C++Test .........................................................................................183
5 http://www.certmagic.com
BH0-003
JsUnit (Schaible) ..............................................................................208
Perl Test::MockObject.......................................................................209
HarnessIt ........................................................................................209
COMPUTE........................................................................................209
TBrun 211
Cantata++ ......................................................................................211
AdaTEST 95.....................................................................................212
.TEST 213
SimpleTest ......................................................................................213
DUnit 214
GJTester .........................................................................................214
CUnit 214
Tessy 215
TestGen4J .......................................................................................215
Quality Assurance.......................................................................... 216
Definition ........................................................................................223
Goals of SQA ...................................................................................223
Responsibilities of SQA......................................................................224
Establishing the SQA Program 224
Creating SQA Plan ............................................................................224
SQA Activities ..................................................................................226
Tailoring SQA to Project ....................................................................226
Other Considerations ........................................................................226
Measuring Software Quality 227
Quality Definition .............................................................................227
Measuring Software Quality ...............................................................228
Selecting Quality Measures ................................................................234
FUNCTIONAL SYSTEM TESTING TECHNIQUES................................ 262
Functional Testing techniques are designed to ensure the system
requirements and specifications are achieved. 262
1) Requirements Testing Technique : ....................................... 262
Usage 263
2) Regression Testing Technique : ............................................ 263
Background .....................................................................................263
Usage 264
3) Error – Handling Testing Technique : ................................... 265
Back ground ....................................................................................265
Usage 265
4) Manual Support Testing Technique :..................................... 266
Usage 266
5) Intersystem Testing Technique: ........................................... 267
Background .....................................................................................267
Usage 267
6) Control Testing Technique :.................................................. 268
Background .....................................................................................268
Usage 268
7) Parallel Testing Technique : ................................................ 269
Usage 269
7 http://www.certmagic.com
BH0-003
IEEE Software Testing Standards .................................................. 269
Other External Standards 275
Testing working days = (Development working days) / 3. ........ 403
Testing engineers = (Development engineers) / 2. ................... 403
Options not present … These are six Structural Testing ................ 409
Chapter 1 Quiz............................................................................... 411
Chapter 6 Self-Check Quiz ............................................................. 429
Chapter 7 Self-Check Quiz ............................................................. 432
Chapter 8 Self-Check Quiz ............................................................. 436
Chapter 9 Self-Check Quiz ............................................................. 440
Chapter 10 Self-Check Quiz ........................................................... 443
Chapter 11 Self-Check Quiz ........................................................... 445
Chapter 12 Self-Check Quiz ........................................................... 449
Chapter 13 Self-Check Quiz ........................................................... 453
Chapter 14 Self-Check Quiz ........................................................... 457
Chapter 15 Self-Check Quiz ........................................................... 460
Chapter 16 Self-Check Quiz ........................................................... 464
Chapter 17 Self-Check Quiz ........................................................... 465
Chapter 18 Self-Check Quiz ........................................................... 469
Chapter 19 Self-Check Quiz ........................................................... 473
Chapter 20: Object-Oriented Concepts and Principles 477
Chapter 20 Self-Check Quiz ........................................................... 477
Chapter 21 Self-Check Quiz ........................................................... 479
Chapter 22 Self-Check Quiz ........................................................... 483
Chapter 23 Self-Check Quiz ........................................................... 486
Chapter 24: Technical Metrics for Object-Oriented Systems....................488
Chapter 24 Self-Check Quiz ........................................................... 489
Chapter 25 Self-Check Quiz ........................................................... 491
Chapter 27: Component-Based Software Engineering 497
General.......................................................................................... 513
GUI Interface 514
Chapter 1 Self-Check Quiz ............................................................. 537
1. Go the web site http://www.mhhe.com/pressman and click on
"Student Center". Choose the chapter that you wish to answer and click
on "Student Quiz" which opens up in a new window. After you answer
the questions and click "Submit" button, the answers are displayed and
any explanation for incorrect answer(s) (those attempted by you) is also
given. If you can get hold of the book, it's good - else, go through the
Student Outlines available on or CSTEHelpdesk yahoo groups, which
should help you in answering most of the questions. The testing method
that uses statistical techniques to find out how faults in the program
affect it's failure rate is: ....................................................................542
2. faults that cause an input to be associated with the wrong path
domain are called.............................................................................542
3. If you know that a developer tends to have extra errors in date-
processing code, and decide to test dates harder than usual as a result,
you are doing: .................................................................................542
4. Which of these is NOT an example of stress testing? ......................543
8 http://www.certmagic.com
BH0-003
5. Fault seeding is instrumentation designed to .................................543
6. _______ attempts to decide what constitutes a sufficient set of
paths to test....................................................................................544
7. Testing to determine whether the system can meet the specific
performance criteria is referred to as: .................................................544
8. Which type of testing gives you the best coverage?........................544
9. Tests to determine the ability of the application to properly process
incorrect transactions are:.................................................................545
10. Crashing a server to ensure that backup data and processes are
adequate is an example of: ...............................................................545
11. Syntax testing evaluates the program's ability to handle:.............545
12. Which of these is an objective of Operations testing?...................546
13. Cost benefits analysis is particularly important during _________
testing, otherwise large amounts of effort can be expended with minimal
payback. .........................................................................................546
14. Code reviews and inspections to make sure programming
standards are followed is known as:....................................................546
15. Which of these are all types of structural testing? .......................547
16. Which of these is a main objective of Security Testing?................547
17. When testing a grade calculation system, a tester determines that
all scores from 90 to 100 will yield a grade of A, but scores below 90 will
not. This analysis is known as: ...........................................................547
18. Which of these are examples of Requirements testing? ................548
19. A concise method of representing equivalence partitioning is:.......548
20. Local extent, finite breadth and global extent, infinite breadth are
two types of: ...................................................................................548
21. Complexity measures, data flow analysis, and symbolic execution
are all static testing methods known as: ..............................................549
22. Input Domain testing uses.......................................................549
23. A technique that produces a finite class of faults, but will crash the
whole program, is known as: .............................................................549
24. Selecting test data on the basis of features of the function to be
computed is called............................................................................550
25. Coming up with tasks for the end user to do, and then watching
them do it for the purposes of making sure their procedures are correct
is: 550
26. When is it appropriate to use intersystem testing? ......................550
27. Output domain coverage is ......................................................551
28. Which of the following can be used as test oracles? .....................551
29. Control testing is a broader term that includes ...........................551
30. Functional analysis is a ______ testing technique, whereas
functional testing is a _________ testing technique. .............................552
31. Test stubs and harnesses are most often used during:.................552
32. Comparing the old version of a program against a new one under
test, in order to determine that the new system produces the correct
results, is:.......................................................................................552
33. All of the following are specification techniques EXCEPT ...............553
34. All of the following might be done during unit testing EXCEPT:......553
9 http://www.certmagic.com
BH0-003
Test Answers ................................................................................. 553
SOFTWARE TESTING
THE DEFINITION:
Testing is a process designed to
• Prove that the program is error free
• Establish that the software performs its functions correctly
• Establish with confidence that the software does its job fully
If yours matches, you are in for a surprise because testing is none of
these. Rather it is properly defined as follows (see Concepts):
Concepts - Testing is the task of locating errors.
Any definition of testing is more than a passive description. It has a
profound impact on the way testing is carried out. Since people are
highly goal-oriented, the setting of proper goals can mean the
difference between success and failure. If the goal of testing were to
prove [that a program or process functions properly], the tester would
subconsciously work towards this goal, choosing test data that would
prove that point.
The reverse would be true if the goal were to locate and correct errors.
Test data would be selected with an eye toward providing the program
with cases that would likely cause the program to fail. This would be a
more desirable result. Why? We begin with the assumption that the
system, like most systems, contains errors. The job of testing is to
discover them before the user does. In that case, a good tester is one
who is successful in crashing the system, or in causing it to perform in
some way that is counter to the specification.
The mentality of the tester, then, is a destructive one -quite different
from the constructive attitude of the programmer, the "creator". This
is useful information for the analyst. Who is acting as a project leader,
and is responsible for staffing. Staff should be selected with the
appropriate personality traits in mind.
Another effect of having a proper working definition of testing regards
the way the project leader assesses the performance of the test team.
Without a proper definition of testing, the leader might describe a
successful test run as one which proves the program is error free and
describe an unsuccessful test as one which found errors. As is the case
with the testers themselves, this mind-set is actually counter-
productive to the testing process.
10 http://www.certmagic.com
BH0-003
GOALS OF TESTING:
To satisfy its definition, testing must accomplish the following goals:
1. Find cases where the program does not do what it is supposed to
do.
2. Find cases where the program does things it is not supposed to do.
The first goal refers to specifications which were not satisfied by the
program while the second goal refers to unwanted side-effects.
THE EIGHT BASIC PRINCIPLES OF TESTING:
Following are eight basic principles of testing:
1. Define the expected output or result.
2. Don't test your own programs.
3. Inspect the results of each test completely.
4. Include test cases for invalid or unexpected conditions.
5. Test the program to see if it does what it is not supposed to do as
well as what it is supposed to do.
6. Avoid disposable test cases unless the program itself is disposable.
7. Do not plan tests assuming that no errors will be found.
8. The probability of locating more errors in any one module is directly
proportional to the number of errors already found in that module.
Let's look at each of these pints.
1) DEFINE THE EXPECTED OUTPUT OR RESULT
More often that not, the tester approaches a test case without a set of
predefined and expected results. The danger in this lies in the
tendency of the eye to see what it wants to see. Without knowing the
expected result, erroneous output can easily be overlooked. This
problem can be avoided by carefully pre-defining all expected results
for each of the test cases. Sounds obvious? You’d be surprised how
many people miss this pint while doing the self-assessment test.
2) DON'T TEST YOUR OWN PROGRAMS
Programming is a constructive activity. To suddenly reverse
constructive thinking and begin the destructive process of testing is a
difficult task. The publishing business has been applying this idea for
years. Writers do not edit their own material for the simple reason that
the work is "their baby" and editing out pieces of their work can be a
very depressing job.
The attitudinal l problem is not the only consideration for this principle.
System errors can be caused by an incomplete or faulty understanding
11 http://www.certmagic.com
BH0-003
of the original design specifications; it is likely that the programmer
would carry these misunderstandings into the test phase.
3) INSPECT THE RESULTS OF EACH TEST COMPLETELY
As obvious as it sounds, this simple principle is often overlooked. In
many test cases, an after-the-fact review of earlier test results shows
that errors were present but overlooked because no one took the time
to study the results.
4) INCLUDE TEST CASES FOR INVALID OR UNEXPECTED CONDITIONS
Programs already in production often cause errors when used in some
new or novel fashion. This stems from the natural tendency to
concentrate on valid and expected input conditions during a testing
cycle. When we use invalid or unexpected input conditions, the
likelihood of boosting the error detection rate is significantly increased.
5) TEST THE PROGRAM TO SEE IF IT DOES WHAT IT IS NOT
SUPPOSED TO DO AS WELL AS WHAT IT IS SUPPOSED TO DO
It's not enough to check if the test produced the expected output. New
systems, and especially new modifications, often produce unintended
side effects such as unwanted disk files or destroyed records. A
thorough examination of data structures, reports, and other output
can often show that a program is doing what is not supposed to do
and therefore still contains errors.
6) AVOID DISPOSABLE TEST CASES UNLESS THE PROGRAM ITSELF IS
DISPOSABLE
Test cases should be documented so they can be reproduced. With a
non-structured approach to testing, test cases are often created on-
the-fly. The tester sits at a terminal, generates test input, and submits
them to the program. The test data simply disappears when the test is
complete.
Reproducible test cases become important later when a program is
revised, due to the discovery of bugs or because the user requests
new options. In such cases, the revised program can be put through
the same extensive tests that were used for the original version.
Without saved test cases, the temptation is strong to test only the
logic handled by the modifications. This is unsatisfactory because
changes which fix one problem often create a host of other apparently
12 http://www.certmagic.com
BH0-003
unrelated problems elsewhere in the system. As considerable time and
effort are spent in creating meaningful tests, tests which are not
documented or cannot be duplicated should be avoided.
7) DO NOT PLAN TESTS ASSUMING THAT NO ERRORS WILL BE FOUND
Testing should be viewed as a process that locates errors and not one
that proves the program works correctly. The reasons for this were
discussed earlier.
8) THE PROBABILITY OF LOCATING MORE ERRORS IN ANY ONE
MODULE IS DIRECTLY PROPORTIONAL TO THE NUMBER OF ERRORS
ALREADY FOUND IN THAT MODULE
At first glance, this may seem surprising. However, it has been shown
that if certain modules or sections of code contain a high number of
errors, subsequent testing will discover more errors in that particular
section that in other sections.
Consider a program that consists of two modules, A and B. If testing
reveals five errors in module A and only one error in module B, module
A will likely display more errors that module B in any subsequent tests.
Why is this so? There is no definitive explanation, but it is probably
due to the fact that the error-prone module is inherently complex or
was badly programmed. By identifying the most "bug-prone" modules,
the tester can concentrate efforts there and achieve a higher rate of
error detection that if all portions of the system were given equal
attention.
Regression Testing
Extensive testing of the system after modifications have been made is
referred to as regression testing.
There are many different kinds of testers, working in different
situations. Most of my testers have been in situations like the
following. Testers in similar situations will be best served by this
paper, though I believe it has broader applicability.
1. They were “black box testers”. That is, they executed their tests
through the same interface that the end user uses.
2. They were typically testing a particular feature in a product,
such as a mail program’s Address Book feature. (Such a feature
allows you to associate an abbreviation with an email address,
13 http://www.certmagic.com
BH0-003
so that you can send mail to “dawn” instead of having to
remember d132@ddress.com.)
3. They were usually brought into the project late, after the feature
was mostly coded.
4. "Upstream" documentation like requirements documents,
specifications, and design documents were often missing. When
they existed, they were out of date. User documentation (user
manuals, help pages, and the like) probably existed, but were
often incomplete.
Everyone would probably agree that (3) and (4) cause problems. I
think (1) and (2) do as well, for reasons I’ll explain later.
The Tester’s Triad compensates for these problems.
Using Bugs
Many bug tracking systems have an email notification feature.
You can ask to be mailed copies of any bug reports filed against
a particular area. In a medium-sized project, I would expect a
good tester to get – and read – email for all the bugs filed
against any part of the system. In a larger project, there are
too many bugs to do that. But I’d still expect a good tester to
read bug reports in areas other than her own.
More about using bug reports
There’s more to using bug reports than that. Let me explain in
the context of James Bach’s model of project risk. Here’s a
picture of his model:
What does it mean to say that a product has a bug? First, the
product must be presented with some Threat – something in
14 http://www.certmagic.com
BH0-003
the input or environment that it cannot handle correctly. In this
case, the threat is two cute but identical twins. (This
corresponds to the common case of software being unable to
handle duplicate inputs. That’s not the threat we talked about in
the previous section, but I couldn’t find a good illustration for
that one.) Next, the Product must perform its processing on
that input or in the context of that environment. If it cannot
handle the threat, it produces a Problem in the form of a bad
result. But a problem means little or nothing unless there is
some Victim who is damaged by it.
By reading bug reports, you learn about “hot spots” in the code.
For example, you might discover that the product as a whole
has a lot of problems handling network timeouts and various
other network glitches. That might inspire you to think of how
network glitches could affect your feature.
Problem
In desktop applications, a window’s title bar often
contains useful information (program name, file being
worked on, etc.) When there’s more text than fits in the
title bar, a product should do something sensible (not
crash, truncate meaningfully, etc.). Many testers before
you have discovered that their product isn’t sensible.
Perhaps your product has similar Problems. Perhaps you
can think of ways to make your title bars overflow (such
as asking for translations into a language that, on
average, uses more characters per word than yours
does).
Victim
Bug reports don’t just go into the bug database and sit
there. Someone reads them and decides what to do about
them. Should they be fixed? Deferred? Marked as “not a
bug”? In some companies, that someone is called the
“bug triage team” or “bug council”.
Ideally, the bug triage team would be composed of purely
rational entities that weigh each bug on its merits. It is in
fact composed of human beings, who are both busy and
filled with idiosyncrasies.
It is your job to write bug reports that are effective, that
convince the bug triage team to make the right decisions.
It is not enough for you to be right; the triage team must
be too. By tracking the progress of other people’s bug
reports, you learn how to deal with the triage team’s
idiosyncrasies, learn about how to make persuasive bug
15 http://www.certmagic.com
BH0-003
reports. (For more on this, see [Kaner93], [Black99], and
[TestingCraft].)
One way to make a convincing bug report is to tell a
story. You must put into the imaginations of the bug
triage team a plausible tale about the damage a bug
would do to an important victim. Again, to do this well,
you must understand what the bug triage team thinks is
important.
Let us summarize how a tester uses a bug report with this
table:
Threat Product Problem Victim
Bug Threat Hot spots Problem Convincing bug
types types reports
The two other elements of the Tester’s Triad will add more rows.
Using Product Structure
One place you’ll find a good tester is sitting in a room while a
developer draws circles and arrows on the whiteboard. Testers
talk to developers to learn about product structure: what are
the pieces? How do they fit together?
A good tester builds a mental model of product internals.
A good way to begin such a conversation is by asking for an
explanation of what went wrong to cause the failure in bug
3434.
Why are testers talking to developers? Why aren’t they reading
design documentation?
I’m not entirely sure, but I suppose that the reason is that
design documents are static. They describe relationships
between unmoving entities. People seem to learn better – and
to better retain knowledge – when those static relationships are
put to use in a dynamic description, when the explanation is of
the form “First this happens, then this module passes this data
to that module, and that module stashes it here for when…
Asking for an explanation of a bug report makes for a good
example to trace through the system.
16 http://www.certmagic.com
BH0-003
Why is an internal model useful? Here’s the picture you as a
black box tester see of the product:
The fuzzy blobs are features to be tested. (I’ll explain why
they’re fuzzy in a moment.)
But there’s a great deal happening beneath that surface you
see. The whole product might look like this:
The surface features communicate with subsystems that
perform parts of the task. Those subsystems communicate in
turn with other subsystems. The subsystems are quite often
shared among features.
This sharing explains why it’s often difficult to tell whether a
particular something the product does is part of your feature or
some other feature (and thus whether testing it is your job or
someone else’s). The action is in fact done by something shared
between the two features. (It’s this fuzzy division of
responsibility at the feature or external interface level that I
meant to imply with the fuzzy blobs in the first picture.)
If you know about the internal structure, you can make better
testing decisions:
1. Without knowledge of internals, testers often write
redundant tests. If your feature and Betty’s feature both
17 http://www.certmagic.com
BH0-003
use a math library, it makes no sense for both of you to
test whether the sqrt function actually works correctly
(that is, that the square root of 4 is 2). That testing
should be done by only one of you (or by some “white
box” tests specifically targeted at the subsystem). Both of
you will have to test that your feature uses the sqrt
function appropriately (for example, that neither of you
try to take the square root of a negative number), but
that’s usually a much smaller set of tests.
2. Suppose Betty finds a bug in the math library, but it could
affect a user of her feature in only a minor way. If you
know something of how your feature uses the math
library, you might be able to find a corresponding, but
much more serious, manifestation of the underlying
problem. (Note that you need to read Betty’s bug report
to know to try.)
3. It’s often the case that subsystems store data. That data
is, in effect, shared among different features. A tester
who knows what data is shared can often exploit that
knowledge.
Here’s a fanciful (and implausible) example. (For a real
but more technical example, see [Marick99b].) Suppose
that you know that putting someone’s email address in
the To field of a mail header stores that address in the
“most recent person” variable. And suppose that you
know that, for some efficiency reason, putting an address
in an address book does the same thing.
That’s a bug. A persuasive bug report might explain it
with this story of how a Victim experiences a Problem:
We can add internal knowledge to our table of why good testers
do what they do:
Threat Product Problem Victim
Bug Threat types Hot spots Problem Convincing bug
types reports
Product Support code Connections
problems between
subsystems
Now, one objection to this idea is that testers don’t have time to
learn about the guts of the product. However, it’s been my
18 http://www.certmagic.com
BH0-003
observation that the product model doesn’t have to be very
detailed. You can get a lot of benefit out of a rough, high-level
model, one that a non-programmer can readily develop.
In fact, I’ve seen testers work effectively with models that I
knew to be flat-out wrong. But even those wrong models
provided value in the form of useful testing ideas.
As in many things testing, perfection isn’t the goal – the goal is
to be good enough. I am actively researching what “good
enough” means, and what techniques one uses to get good
enough knowledge [VisibleWorkings].
Using the User
A good tester’s conversation is often in terms of the users: what
their goals are, what tasks they perform, and how they go about
those tasks. When first looking at a feature, a good tester will as
a matter of course think about the user’s reaction to that
feature, how it will affect what the user does, and what
mismatches it might have with the user’s habits.
A good tester understands how users spend their days.
Why is this useful?
One reason has to do with a problem with the approach in the
previous section. It’s all very well to talk about exercising
interactions between subsystems or between features. But
which interactions between which subsystems? Even though
you’ll unaware of many interactions (remember, you have
imperfect knowledge of the interior), there will still be too many
possibilities to try them all.
If you can’t exercise all interactions, it’s best to exercise those
that are most likely to cause problems to users (victims). To do
that, you must understand the users. You must, for example,
realize that a user may be performing several tasks at once.
When the computer is crunching away at one of the tasks, she’s
likely to switch to another. Maybe she’ll start up another copy of
the same program. If that new copy changes some files being
used by the first copy… blam!
These are exactly the sort of interactions that programmers
overlook. Feature testing, which concentrates on one feature at
a time, also overlooks them. But the total testing effort should
not.
19 http://www.certmagic.com
BH0-003
There should be task-based testing that’s guided by an
understanding of the user. In addition to covering the
interactions you’ve decided are most important, it will also
unintentionally exercise important interactions you didn’t know
about.
Another reason for understanding users is that it enables better
bug reports. The product’s usability is important. But we as
testers have been trained not to report usability problems. And
yet we are probably the closest approximation to the expert
user available to the project team. (Usability testing typically
concentrates on novice users, not on people like us – people
who use the product many hours a day for many days). As such,
we can discover expert tasks that cannot be done, tasks that
are needlessly difficult, tasks that are needlessly error-prone,
and features that promote misunderstandings.
When we report these bugs, the likely reaction is “works as
designed”. Our counter is, essentially, “So redesign it!” We have
to express that message politely, of course, and we have to
present it persuasively: by being a credible and knowledgeable
representative of the user.
Getting there
The question is how one learns about the user. I don’t have a
satisfying answer for that.
I have observed that good testers really care about the users.
They have a protective attitude that is often not shared – at
least not so strongly – by the rest of the product team. I think
this leads to testers thinking of users as people, rather than
abstractions. That engages empathy and imagination, and leads
to better test cases.
Let me compare testing to a related field, software marketing
and product design. An important book in that field is Geoffrey
Moore’s Crossing the Chasm [Moore91]. In it, he says:
Neither the names nor the descriptions of markets evoke any
memorable images – they do not elicit the cooperation of one’s
intuitive faculties. We need to work with something that gives
more clues about how to proceed. We need something that feels
a lot more like real people. However, since we do not have real
live customers as yet, we are just going to have to make them
up. Then, once we have their images in mind, we can let them
guide us to developing a truly responsive approach to their
needs. (pp. 94-95)
20 http://www.certmagic.com
BH0-003
He describes a process of target-customer characterization that
includes naming the person, sketching their history and goals,
and so forth. That seems a little silly, a little far afield from our
technical task of finding bugs. But finding bugs is a creative
task, in part, and creativity comes on its own terms.
Testing that is entirely analytical and painstaking and planned
will miss bugs. It is difficult to keep the user in mind while
you’re doing something quite different from what a user does
(writing an automated test script; stepping through test case
checklists written months ago). For that reason, many good
testers practice some form of exploratory testing, which has
more of an emphasis on an interweaving of test design and
creation – just the kind of thing users do, except they’re “doing
their job” instead of “creating tests”. For some writings on
exploratory testing see [Exploratory].
I believe the act of exploratory testing will increase your
understanding of the user.
This final table summarizes how keeping the user in mind
further addresses Bach’s four risk areas:
Threat Product Problem Victim
Bug Threat types Hot spots Problem Convincing bug
types reports
Product Support Connections
code between
problems subsystems
User User actions Picking Reactions to
subsystem program actions;
traversals impact of bugs
Reprise: Testing Problems
There were four problems listed in the earlier section. Two were
unexceptional: being brought in late and not having upstream or
high-level documents. Two I expect to be controversial: being a
black box tester, and feature testing.
I list these as problems because it seems to me that I observe
good testers pretending to be the first, and to perform the
second, while actually doing something more. They make rough
models of the internals despite supposedly working from the
interface a user sees.While supposedly doing feature testing,
21 http://www.certmagic.com
BH0-003
they add “irrelevant” steps to tests in order to exercise
interactions. They lean toward exercising their features in the
context of user tasks.
That’s good. It grates against the rather tired, rather rote way
we look at testing. Our terminology and approach haven’t
changed much in the last 20-some years, and I think that’s a
problem [Marick99]. If the best testers are doing something
beyond the received wisdom, we need to fix the received
wisdom.
I don’t know if this is a true story, but it’s truly a story I’ve
heard. A new jet fighter was being tested. The test pilot
strapped in, turned the key in the ignition (or the equivalent),
and flipped the switch to raise the landing gear. The plane
wasn’t moving, but the avionics software dutifully raised the
landing gear. The plane fell down and broke.
I haven’t seen the actual code, but it’s reasonable to assume
that it looked like the pseudocode in Figure 1. The bug is that
some code is missing—code that in Figure 2 is shown underlined
and in a different color.
Figure 2:
Code that
corrects
the bug
(but
other
bugs may
Figure1:
remain,
The
since
buggy
there
code
may be
other
reasons
not to
raise the
landing
gear)
This type of bug—one that is corrected by adding code that
someone left out—is called a fault of omission. Numerous
studies have shown it to be a common fault in production code.
22 http://www.certmagic.com
BH0-003
Failure to handle code details
Sometimes the programmer’s mistake is caused by details of
specific code. For example, programmers often read textual
data into fixed-size buffers without anticipating that the data
might be too big to fit. When the data overflows the buffer, bad
things happen. Such faults are a major cause of security
problems on the InternetHere’s an example of a corrected buffer
overrun bug:
fscanf(f, "%99s", command_buffer);
A version without the “99” would have no checks on the length
of the input. The corrected version cuts off the input at 99
characters. That may not be ideal if the input was legitimately
long, but it’s less likely to be catastrophic.
One fault of omission in my USENET survey was due to memory
allocation. Some ways of allocating memory initialize it to zero;
because of that, it’s easy to fall into the trap of assuming that
all of them do. The faulty code allocated a gdbm_file_info
structure in a way that left it full of garbage. The garbage was
then processed as if it were meaningful data. The code was
corrected by zeroing the structure just after it was allocated.
The new code looks like the following, where dbf is the location
of the new memory:
bzero((char )dbf, sizeof(gdbm_file_info));
Bugs like these are not prevented by any abstract theory of
good programming or good design. They’re prevented when
programmers are trained to think specific thoughts like “Always
double-check that allocated memory is initialized” and “Never,
never, never read into a buffer without guarding against
overruns.” When the code is important, you should supplement
prevention with reviews by other programmers trained to think
the same thoughts.
Thinking is easier when checklists are written down. It would be
helpful if documentation for library functions were explicit about
potential faults of omission. (“Watch out for buffer overruns!” in
flashing red letters would be nice.) But they often aren’t.
Omissions are easy to make because the documentation is
obscure about special cases, especially error cases.
But why force people to avoid errors that are begging to be
made? Instead of relying on programmers to zero memory,
23 http://www.certmagic.com
BH0-003
have every memory allocation routine do it for them. Trade an
occasional performance penalty for greater reliability.
When prevention fails, you test. But conventional “black box”
testing that doesn’t look at the code is weak at finding these
code-specific omissions. For many of these bugs (such as
problems with memory allocation), there’s no clue in the
external interface or documentation that would prompt you to
try a test that would fail. For others, there is. You could search
for buffer overruns by looking for all places where you might
enter variable-length data. But code inspections are a better use
of your money.
Code-aware “clear box” testing isn’t useful for finding faults of
omission. When you see the call to fscanf and wonder about
buffer overruns, you don’t run a test—you simply check if the
code to handle long inputs exists. (If the code does exist, you
might run a test to check whether it works, but that’s searching
for a different fault.)
Mistakes applying programming clichés
Now let’s move away from the specifics of code to a slightly more
abstract realm. When programmers write code that searches a
collection of objects, they sometimes overlook special cases like
these:
• The object being sought doesn’t appear.
• The object being sought appears twice.
• The collection doesn’t contain any objects to search through.
Notice that these mistakes are independent of code. It doesn’t
matter whether the search is done by library routines like (in C)
bsearch and strrchr, or by hand-crafted searching code. That’s why
I say we’re in a more abstract realm—we’re dealing with abstract
ideas like “searching” and “collection.”
This has two practical consequences. First, it’s an additional source
of checklists. As a programmer writing code, or a reviewer reading
code, I can ask, “Can I interpret this code as doing a search?” If so,
I refer to a checklist of special cases that the code should handle. I
used the word “interpret” deliberately. Sometimes you can say, “If
I look at the program from this perverse point of view and consider
these two fairly unrelated operations, I could argue that there’s a
sort of a search going on. What, then, would happen if the object
being ‘sought’ appears twice?” In my experience, perverse
interpretations find important bugs. That makes sense:
24 http://www.certmagic.com
BH0-003
programmers are more likely to overlook the special case if they
don’t realize that the general situation applies.
The second practical consequence is that black box testers do
better at finding this kind of omission than the previous kind, which
was so tightly tied to specific code constructs. Someone familiar
with the internals is likely to unconsciously reject the notion that
searching applies—after all, there’s no searching code. Black box
testers are not hampered by the same misleading knowledge.
Misunderstanding the environment
A third type of fault of omission has nothing to do with either
specific code or abstract programming concepts. It’s what I call
the “who would want to do that?” fault. The fighter fault was of
this type. Programmers and designers build products to help
people. Designers know something of what those people do
before they have the product, but never enough. And designers’
predictions of how the product will change people’s behavior are
always wrong. So, armed with this incomplete knowledge,
programmers create a product. Having done that, they find it
very difficult to break free from what are now ingrained
assumptions. They don’t anticipate other uses. When the users
(or other actors in the environment, like other products on the
same machine) behave in unexpected ways, that uncovers
faults of omission.
Better up-front knowledge—better requirements analyses, more
thorough and detailed user scenarios—will surely help. The state
of the practice is shockingly bad. But even after it’s improved, I
believe the black box tester has a crucial role to play. Why? We
humans are a species that manipulates—we learn by picking
things up, shaking them, and seeing what happens. Using
programs is eye-opening: so much that was hidden becomes
obvious. Some of the things that become obvious are ways to
use the program that the designers never intended—“Hey, that
operation takes a long time, so why don’t I browse over here
while I’m waiting?…” <crash Black box testers—if they have
some domain expertise and if they pay attention to how people
behave—are a concentrated and efficient way of churning
through odd but inevitable uses.
The fault of omission is not your friend, but it will be your
constant companion. Learn to keep it from dragging you down.
Here are the three (admittedly rough) categories of faults of
omission, together with comments about preventing or
detecting them.
25 http://www.certmagic.com
BH0-003
Understanding
Applying
Handling the Product’s
Programming
Specific Code Environment
Abstractions
(buffer (“Who would
(collections,
overruns, etc.) want to do
searching, etc.)
that?”)
Yes, especially if
based on Inspections of the
checklists. code are rarely
useful.
Yes, especially if
Inspections of Inspections of
Inspections based on
design and other requirements
checklists
“upstream” documents or use
documents can cases are
prevent definitely useful.
omissions.
Yes. Inspections
Code-aware Inspections are
are probably Rarely useful.
testing better
better.
Yes, especially if
testers have
Problems will for
Yes. The domain
the most part be
“Black box” independent point knowledge or are
found by dumb
testing of view will find good at putting
luck. There are
bugs. themselves in the
better ways.
shoes of the
users.
Do a better job of
Don’t use
requirements
predefined data
analysis. Zero in
Miscellaneous structures and
on undocumented
functions that
and unexamined
invite error
assumptions.
Domain Testing:
Software Testing:
26 http://www.certmagic.com
BH0-003
“Testing is the process of executing a program with the intent of
finding errors”
“The purpose of testing is to determine whether a program contains
any errors”
“Testing--A verification method that applies a controlled set of
conditions and stimuli for the purpose of finding errors”
“Software testing in all its guises is a study of the software input
space, the domain over which a program under test is supposed to
operate. The whole question of ‘how should testing be done?’ is a
matter of input selection”
Test Case:
(1) A set of test inputs, execution conditions, and expected results
developed for a particular objective, such as to exercise a particular
program path or to verify compliance with a specific requirement.
(2) Documentation specifying inputs, predicted results, and a set of
execution conditions for a test item.
Bug/Error/Fault:
We test to find bugs, also called errors. defined a bug, error or fault
as:
(1) The difference between a computed, observed, or measured value
or condition and the true, specified, or theoretically correct value or
condition. For example, a difference of 30 meters between a computed
result and the correct result.
(2) An incorrect step, process, or data definition. For example, an
incorrect instruction in a computer program.
(3) An incorrect result. For example, a computed result of 12 when the
correct result is 10.
(4) A human action that produces an incorrect result. For example, an
incorrect action on the part of a programmer or operator.
There are two general approaches to doing testing- ‘white-box’ testing
and ‘black-box’ testing.
27 http://www.certmagic.com
BH0-003
1.1.1 White -Box Testing Approach
In this testing approach, the program under test is treated as a
white box or a glass box, something you can see through
(Kaner, Falk & Nguyen, 1999; Myers, 1979). According to Myers
(1979), this approach can also be called a logic-driven testing
approach in which the tester does testing based on the internal
structure of the program, usually ignoring the specifications.
This approach is also known as structural testing or glass-box
testing. IEEE Std. 610.12 (1990) defined structural, glass-box or
white-box testing as “Testing that takes into account the
internal mechanism of a system or component”.
Black-Box Testing Approach
In this testing approach, the program under test is treated as a
black box, something you cannot see through (Kaner et al.,
1999; Myers, 1979). A black-box tester is unconcerned with the
internals of the program (Howden, 1980a; Myers, 1979;
Podgurski & Yang, 1993) and tests the program using the
specifications of the program (Myers, 1979). Black-box testing
has been defined by IEEE Std. 610.12 (1990) as “(1) Testing
that ignores the internal mechanism of a system or component
and focuses solely on the outputs generated in response to
selected inputs and execution conditions” (p. 35). Specification
can be defined as “A document that specifies, in a complete,
precise, verifiable manner, the characteristics of a system or
component, and, often, the procedures for determining whether
these provisions have been satisfied” (IEEE Std. 610.12, 1990,
p. 69).
According to Myers (1979), the black-box testing approach can
also be called a data-driven or input/output-driven testing
approach since black-box testers feed the program a set of input
values and observe the output to see if it is in accordance with
what is expected in the specification document.
Complete Testing/Exhaustive Testing
One of the problems with software testing is that it is practically
impossible to achieve complete or exhaustive testing (Burnstein,
2003; Collard, personal communication, July 22, 2003;
Goodenough & Gerhart, 1975; Huang, 1975; Myers, 1979;
Kaner, 2002a; Kaner et al., 1999). According to Kaner (2002a),
exhaustive or complete testing is not possible because:
1. The domain of possible inputs is too large.
28 http://www.certmagic.com
BH0-003
2. There are too many combinations of data to test.
3. There are too many possible paths through the program to
test.
4. There are user interface errors, configuration and
compatibility failures, and dozens of other dimensions of
analysis.
Huang (1975) gave a simple yet very powerful example of why
complete testing is just impossible. He described a program that
has just two input variables, x and y, and one output variable z.
If we were to assume that the variables are integers and the
maximum value for x or y is 232, then it means that each has
232 possibilities of input values. This in turn means that the
total possible combinations of input values for x and y is 232 x
232. Huang (1975) said, “Now suppose this program is
relatively small, and on the average it takes one millisecond to
execute the program once. Then it will take more than 50 billion
years for us to complete the test!” (p. 289). This is called
combinatorial explosion.
Domain Testing
Domain testing, one of the widely used software testing
methodologies, was designed to alleviate the impossibility of
complete testing and thereby enable effective testing with
reduced effort. Kaner and Bach (2003) described the
fundamental goal or question in domain testing, saying, “This
confronts the problem that there are too many test cases for
anyone to run. This is a stratified sampling strategy that
provides a rationale for selecting a few test cases from a huge
population”
Kaner, Bach and Pettichord (2002) define domain of a variable
as “a (mathematical) set that includes all possible values of a
variable of a function” (p.36). According to Hamlet and Taylor
(1990), “Input partitioning is the natural solution to the two
fundamental testing problems of systematic method and test
volume” (p. 1402).
Generally speaking, tasks in domain testing consist of
partitioning the input domain, the set of all possible input
values, into a finite number of subsets or equivalence classes
and then choosing a few representatives from each class as the
candidates for testing (Goodenough & Gerhart, 1975;
Jorgensen, 2002; Kaner et al.,1999; Myers, 1979). All members
of an equivalence class are equivalent to each other in the sense
that they are all sensitive to the same type of error
29 http://www.certmagic.com
BH0-003
(Goodenough & Gerhart, 1975; Ho wden, 1976; Jeng &
Weyuker, 1989; Kaner & Bach, 2003;
Kaner et al., 2002; Kaner et al., 1999; Myers, 1979; Richardson
& Clarke, 1981;
Weyuker & Ostrand, 1980).
However, one member of a class might be more likely to expose
such an error or may also be able to expose a different error. In
either of these cases, we would treat that member as a better
(more powerful) member of the equivalence class. When one is
available, we select the best representative (most powerful
member) of the equivalence class (Kaner & Bach, 2003, part 2).
Although most of the literature describes partitioning of input
domain, similar analysis can be applied to the output domain as
well (Kaner et al., 2002; Kaner et al., 1999; Myers, 1979;
Whittaker & Jorgensen, 2002). For programs having multiple
variables, the next task would be to combine the test cases in
order to perform combination testing.
In sum, domain testing is a divide-and-conquer method of
testing in which researchers systematically reduce an
enormously large test data set to a few subsets and further
select a few representatives from each.
Similarities among Different Interpretations of Domain Testing
There are differing interpretations of the domain testing
methodology, but a few things are common to all of them. They
all agree that input domain of any average program is
enormously large and it is impossible to test for all data points.
They advocate partitioning the input domain into a finite number
of subsets based on some criterion. They also agree that all the
members of any such subset are equivalent to each other with
respect to the criterion that was used to classify them in their
respective subset. All interpretations concur that one or more
representatives must be chosen from each such subset.
Differences between Different Interpretations of Domain Testing
One of the differences lies in the criterion that is used for
partitioning the input domain. Some researchers have used path
analysis as the criterion for partitioning
Some others have described mutation testing as a form of
domain testing. Path analysis and mutation testing approaches
are basically white-box approaches.
30 http://www.certmagic.com
BH0-003
Other researchers have used the features/functions and
input/output properties described in the specification as the
criterion for partitioning This is a black-box testing
approach.
Some have described a specific form of black-box domain
testing approach called functional testing. This has been defined
under section 2.01.01.02.01. In the literature, Howden’s works
on functional testing have been described to have laid the
foundation of black-box testing. There are some others who
have suggested incorporating blackbox and white-box domain
testing approaches into a combined approach that takes
advantage of both the specification and the internal structure of
a program
Another difference is that some researchers have proposed that
partitioning of input domain should result in non-overlapping or
disjoint subdomains or partitions (Howden, 1976; Jorgensen,
2002; Myers, 1979; White & Cohen, 1980).
However, others allowed overlapping subdomains in their
analyses as they noted that in practice it may be impossible to
get perfectly disjoint subdomains
Yet another difference comes from the way best representatives
are selected from an equivalence class. Some have asserted
that since all members of an equivalence class are equivalent to
each other, any arbitrary member can be chosen at random to
represent its respective equivalence class. But others have
described strategies such as boundary value analysis for
selecting best representatives from each equivalence class
(Binder, 1999; Hamlet, 2000; Howden, 1980b; Hutcheson,
2003; Jorgensen, 2002; Kaner & Bach, 2003; Kaner et al.,
2002; Kaner et al., 1999; Myers, 1979; White & Cohen, 1980).
In boundary value analysis strategy, test data at the boundaries
and just beyond the boundaries is selected. In addition, some
researchers have described methods such as special value and
worst case testing to supplement boundary value analysis
(Jorgensen, 2002). Some have also described proportional
partition testing, a partition testing strategy in which the test
data selection is random but the number of test cases selected
from a subdomain depends on the probability of failure of inputs
in the subdomain (Chan, Mak, Chen & Shen, 1997; Chan, Mak,
Chen & Shen, 1998; Chen, Wong & Yu, 1999; Chen & Yu, 1994;
Chen & Yu, 1996b; Chen & Yu, 1996c; Leung & Chen, 2000;
Ntafos, 1998; Ntafos, 2001). In other words, if we were to
31 http://www.certmagic.com
BH0-003
assume that each input in a subdomain is equally likely to occur,
then the number of test cases selected would depend on the
size of the subdomain.
There is also a difference in how linear and non-linear domains
and continuous and discrete domain spaces are analyzed. Some
researchers have described domain testing only for linear
continuous space domains (Beizer, 1995; Clarke, Hassell &
Richardson, 1982; White & Cohen, 1980). Others have extended
their description of domain testing methodology to non-linear
and discrete domain spaces as well (Jeng & Weyuker, 1994;
Kaner et al., 1999; Zeil et al., 1992b).
While most people in the literature have not described forming
equally sized subdomains, there are some who have suggested
partitioning input domain into equally sized subdomains (Chan
et al., 1998; Weyuker & Jeng, 1991).
Finally, there is the difference in the driving force behind
testing. Some researchers have described domain testing
strategy as a method of gaining confidence in the program or
for proving the correctness of the program (Goodenough &
Gerhart, 1975; Howden, 1976; Howden, 1979; Howden, 1981;
Howden, 1986; White & Cohen, 1980). Others have described it
as a risk-based or failure-based testing approach (Beizer, 1995;
Collard, personal communication,
July 22, 2003; Frankl, Hamlet, Littlewood & Strigini, 1998;
Gerrard & Thompson, 2002; Hamlet & Taylor, 1990; Hamlet,
2000; Kaner, 2002b; Kaner & Bach, 2003; Podgurski & Yang,
1993; Whittaker & Jorgensen, 2000). In the latter approach,
partitions or subdomains are formed anticipating failures,
whereas in the former approach the goal is to prove that the
program is working well. This means that if the program works
properly for a representative of a partition, then it is assumed to
give confidence of correctness for all the remaining members in
the partition. Testing approaches that concentrate on achieving
code coverage, such as the path analysis approach mentioned
earlier, fall under this category.
Domain Testing: A Literature Review
since exhaustive testing is impossible we have to be able to do
selective testing, but in a way that the entire population is
represented. Partitioning the input domain and selecting best
representatives from each partition is one way to achieve this
goal. In this testing strategy, called domain testing, the three
main tasks are:
32 http://www.certmagic.com
BH0-003
1. Dividing or partitioning the set of all possible test cases into
partitions
based on some criterion.
2. Selecting candidates that best represent each partition.
3. Combining variables in case of programs having multiple
variables.
Partitioning of Input Domain
This is the process of dividing the input domain or the set of all
possible test cases into partitions such that all test cases in one
partition are equivalent to each other with respect to some
criterion. Although most of the literature describes partitioning
of input domain, similar analysis can be applied to the output
domain as well (Kaner et al., 2002; Kaner et al., 1999; Myers,
1979).
According to Kaner et al. (2002), test cases can be grouped into
an equivalence class if they satisfy the following conditions: “(a)
they all test the same thing; (b) if one of them catches a bug,
the others probably will too; and (c) if one of them doesn’t catch
a bug, the others probably won’t either” (p. 36).
Partitioning of input domain differs in the literature along the
following dimensions:
White or Black:
White-Box Testing Approach
Path Analysis Approach
Mutation Testing Approach
Black-Box Testing Approach/Specification-Based Approach
Functional Testing Approach
Combination of White-Box and Black-Box Testing
Approaches
Driving Factor:
Confidence-based approach
Risk-based approach
Type of Domain:
Linear
Non-linear
33 http://www.certmagic.com
BH0-003
Overlapping or Disjoint Subdomains:
Partitioning into overlapping subdomains
Partitioning into disjoint subdomains
Size of Subdomains:
Equally sized subdomains
No particular sized subdomains
A detailed description of the elements in the above classification
ofpartitioning of the input domain follows.
White or Black
There are some testers that rely solely on the implementation of
a program, which demonstrates the actual behavior of a
program. We call them the 'white-box testers'. There are others
that rely on specification, which describes or is at least
supposed to describe the intended behavior of a program. We
call this group the 'black-box testers'.
Black-box testers may or may not rely on a specification, and
white-box testers often rely on specifications. White-box testers
have knowledge of the code, while black-box testers do not.
Some others have realized that both implementation and
specification are important sources of information for testing. In
this context, different approaches to partitioning the input
domain have been described in the literature. The following is a
classification of the approaches.
White-Box Testing Approach
As mentioned before, there are two visible variations of partition
testing under this category:
Path Analysis Approach
Mutation Testing Approach
Path Analysis Approach
Some of those who have described the path analysis
approach to doing partition testing are Boyer et al. (1975),
Clarke and Richardson (1982), DeMillo et al. (1978), Duran
34 http://www.certmagic.com
BH0-003
and Ntafos (1981), Ferguson and Korel (1996), Goodenough
and Gerhart (1975), Hajnal and Forgács (1998), Hamlet
(2000), Howden (1976), Howden (1986), Huang (1975),
Jeng and Weyuker (1989), Jeng and Weyuker (1994), Koh
and Liu (1994), Podgurski and Yang (1993), Weyuker and
Jeng (1991), White and Cohen (1980), White and Sahay
(1985), Zeil et al. (1992a), and Zeil and White (1981).
These are definitions of some basic terms concerned with
path-analysis: Path: IEEE Std. 610.12 (1990) defined a path
by stating, “(1) In software engineering, a sequence of
instructions that may be performed in the execution of a
computer program” (pp. 54-55).
Howden (1976) defined a path this way: “A path through a
program corresponds to some possible flow of control” (p.
209).
Kaner et al. (1999) defined a path as “…a sequence of
operations that runs from the start of the program to an exit
point. This is also called an end-to-end path. A subpath is a
sequence of statements from one place in the program to
another. Subpaths are also called paths” (p. 43).
Path Condition: IEEE Std. 610.12 (1990) defined the path
condition as “A set of conditions that must be met in order
for a particular program path to be executed”
Path Analysis: This has been defined by IEEE Std. 610.12
(1990) as “Analysis of a computer program to identify all
possible paths through the program, to detect incomplete
paths, or to discover portions of the program that are not on
any path”
In the path analysis approach to doing domain testing,
partitioning of the input domain is done based on paths. To
understand what is meant by path in this context, consider
an example of a very simple program:
If x < 10 then
Event A occurs
Else
Event B occurs.
Depending on what the value of the variable x is, the
program would either go down the path that leads to the
execution of event A or would go down the path that leads to
the execution of event B.
35 http://www.certmagic.com
BH0-003
In the path analysis approach to doing partition testing, the
input domain corresponding to a program would be the set of
all paths that the program can take.
For instance, in the above example, there are two possible
paths. One path is that event A occurs and the other path
corresponds to the possibility that event B occurs. Of course,
there is another possible path for all the cases when the
value of x is non-numeric. In the above example it is difficult
to tell, but normally if there were an error-handling capability
incorporated in the program, the path taken in the event of
such an error would be whatever is mentioned in the
corresponding errorhandling construct of the program.
In domain testing, the first main task is to partition the input
domain into partitions or equivalence classes based on some
criterion. As mentioned before, the criterion in the path
analysis approach is the path. All members of one partition
or subset of the input domain are expected to result in the
execution of the same path of the program (Howden, 1976;
Weyuker & Jeng, 1991; White & Cohen, 1980). In the
example above, the set of all values of variable x less than
10 forms one partition and the set of all values of x greater
than or equal to 10 forms another.
Mutation Testing Approach
If a program passed all the tests from a subdomain, then can
one really be sure that the program is actually correct?
Either the program really is correct over that subdomain or
the tester is using a set of ineffective test cases to test the
program (DeMillo et al., 1978). This is where mutation
testing comes into play.
Mutation testing involves creating a set of wrong versions of
a program, called mutants, for every subdomain over whose
members the original program seems to operate correctly.
That is, it is known to have passed all the tests in that
subdomain (DeMillo et al., 1978). A mutant of a program is
usually formed by modifying a single statement of the
original program (Weyuker & Jeng, 1991). The statement
that is modified will depend on which portion of the program
the
associated subdomain corresponds to. Adrion, Branstad and
Cherniavsky (1982), DeMillo et al. (1978), Howden (1981),
Jeng and Weyuker (1989), Podgurski and Yang (1993), and
36 http://www.certmagic.com
BH0-003
Weyuker and Jeng (1991) are among the researchers that
have described how mutation testing can be incorporated in
the process of achieving partitioning of input domain.
So, how are the mutants really used? If testing the mutant
gives different results compared to the original program,
then it is confirmed that the original program was indeed
correct. But if the mutant yields results identical to that of
the original program, then there is definitely something
amiss (DeMillo et al., 1978).
Hence, mutation testing is taking domain testing or partition
testing a step further. It ensures that when a program
passes a set of tests, it really means that the program is
correct. However, Howden (1981) has pointed out one
outstanding disadvantage of the mutation testing approach--
the large number of mutants that one might end up
generating for a program. Howden said, “It is estimated that
there are on the order of n2 mutants for an n-line program.
This implies that if a proposed test set T contains t elements
it is necessary to carry out on the order of n2 to n2t program
executions to determine its completeness” (1981, p. 67).
Black-Box Testing Approach/ Specification-Based Approach
Those who have described the process of domain testing
using specifications include Beizer (1995), Chen et al.
(2003), Hamlet (1996), Howden (1980), Hutcheson
(2003), Jorgensen (2002), Kaner (2002a), Kaner and
Bach (2003), Kaner et al. (2002), Kaner et al. (1999),
Mayrhauser et al. (1994), Myers (1979), Ostrand and
Balcer (1988), Patrick and Bogdan (2000), Podgurski and
Yang (1993), Richardson et al. (1989), Reid (1997), and
Weiss and Weyuker (1988).
Kaner et al. (2002) have suggested doing domain testing
by first identifying both input and output variables for
every function either by looking at the specification for
the program at hand or by looking at the program or
prototype from the outside by considering the program or
function as a black box. Next, they suggested finding the
domain for each variable and partitioning this domain into
equivalence classes. After that, they suggested choosing
a few candidates from each class that best represent that
class.
Myers (1979) suggested associating an equivalence class
with every input or output condition mentioned in the
37 http://www.certmagic.com
BH0-003
specification of the product under test. Myers cited two
kinds of equivalence classes associated with every input
or output condition:
1. A valid equivalence class consists of all values that
serve as valid input to the product or program under test,
such that the corresponding input or output condition is
satisfied.
2. An invalid equivalence class consists of all erroneous
values or invalid inputs. Ostrand and Balcer (1988)
described the classic category partition method of doing
functional testing using specifications. The first three
tasks in their method are:
1. Analyze the specification. The tester identifies
individual functional units that can be tested separately.
For each unit, the tester identifies:
⇒ parameters of the functional unit
⇒ characteristics of each parameter
⇒ objects in the environment whose state could
affect the functional unit’s operation
⇒ characteristics of each environment object
The tester then classifies these items into categories that
have an effect on the behavior of the functional unit.
2. Partition the categories into choices. The tester
determines the different significant cases that can occur
within each parameter and environment category.
3. Determine constraints among the choices. The
tester decides how the choices interact, how the
occurrence of one choice can affect the existence of
another, and what special restrictions might affect any
choice. (p. 679) Kaner et al. (1999) described an example
of a program whose specification includes 64K to 256K
RAM memory requirements. In this case, one could
identify
three equivalence classes. The first one contains all cases
that require operating the program over RAM capacities
below 64K. The second class consists of all cases that
require operating the program over RAM capacities
between 64K and 256K.
Finally, the third equivalence class consists of all cases
that require operating the program over RAM capacities
greater than 256K.
38 http://www.certmagic.com
BH0-003
Functional testing has been described in the literature as
a specific form of specification-based or black-box domain
testing, which is described next
Functional Testing Approach
IEEE Std. 610.12 defined a function as:
(1) A defined objective or characteristic action of a system or
component. For example, a system may have inventory control
as its primary function.
(2) A software module that performs a specific action, is invoked
by the appearance of its name in an expression, may receive
input values, and returns a single value. (p. 35)
Hamlet et al. (2001), Howden (1981), Howden (1986), Howden
(1989),Podgurski and Yang (1993), Vagoun (1996) and Zeil et
al. (1992b) are among the researchers that have described the
functional testing approach to doing domain testing.
Howden's work on functional testing is often cited in the
literature as having laid the foundation of black-box testing
because the approach he described focuses only on the inputs
and outputs of a program’s functions and not on how the
program executes the functions. According to Podgurski and
Yang (1993), “Perhaps the most widely used form of partition
testing is functional testing. This approach requires selecting
test data to exercise each aspect of functionality identified in a
program’s requirement specification. The inputs that invoke a
particular feature comprise a subdomain” (p. 170).
In the functional approach to doing domain testing, partitioning
of the input domain is done based on functions. Functional
structures described in the specification are studied and test
data is developed around these structures (Howden, 1986). The
program at hand is analyzed to identify the functions involved,
the input variables involved and their constraints. According to
Howden
(1981), identifying functions and selecting reliable test cases are
two important tasks in functional testing. Depending on what
kind of function it is, the approach to deriving a reliable test
data set for the corresponding function will be different, since
each will have different kinds of errors associated with it
(Howden, 1981). One of the five types of functions identified by
Howden (1981) based on functional structure is the arithmetic
39 http://www.certmagic.com
BH0-003
relation function. “Arithmetic relation functions are computed by
expressions of the form E1 r E2, where E1 and E2 are arithmetic
expressions and r is one of the relational operators <, , =, =, =
or ?” (p.70).
According to Howden (1981), of the many errors that can be
associated with this type of function, a simple one is use of
illegal or incorrect relation, which in turn means use of an
incorrect relational operator. To test for such an error, a tester
would consider two partitions. One corresponds to the set of test
cases that have the expressions related by the correct relational
operator, and the other partition consists of all test cases that
have the expressions related by one of the remaining sets of
relational operators.
Combination of Black-Box and White-Box Testing Approaches
As previously mentioned, there are also a few researchers who
have talked about leveraging the advantages of both white-box
and black-box testing strategies and described a combined
approach to doing domain testing. Among the proponents of
such a combined approach have been Binder (1999), Chen et al.
(2000), Goodenough and Gerhart (1975), Hamlet and Taylor
(1990), Howden (1980a), Howden (1980b), Howden (1982),
Richardson and Clarke (1981), Weyuker and Ostrand (1980)
and White (1984). In Weyuker and Ostrand’s (1980) approach
to doing domain testing, both program-dependent sources (such
as the underlying code) and programindependent sources (such
as the program specifications) are used. Programdependent
sources lead to path domains and program-independent sources
lead to problem domains. These two domains are intersected to
get the final subdomains. Since a program’s specification
describes the intended behavior of the program and the
implementation represents the actual behavior, Weyuker and
Ostrand (1980) suggested that the differences between the path
and problem domains are good places to look for errors because
that is where the inconsistency
between the specification and the implementation lies.
Driving Factor
Are we trying to gain confidence that the program works well, or
are we trying to find failures? To achieve confidence in a
program, the kind of test cases selected will be different from
the ones chosen to break the program. This in turn will affect
how a tester forms partitions of the input domain. In this
40 http://www.certmagic.com
BH0-003
context, there are two general approaches to doing domain
testing, which are explained in the following sections.
Confidence-Based Approach
Some researchers have described partition testing as a method
of gaining confidence in the correctness of the program, rather
than testing the program to specifically make it fail in as many
ways as possible (Goodenough & Gerhart, 1975; Howden, 1976;
Howden, 1979; Howden, 1981; Howden, 1986; White & Cohen,
1980).
Most testing strategies that strive for code coverage fall under
this category since the main goal there is to obtain coverage
and in turn gain confidence in the correctness of the program.
Code coverage-based testing methods like the path analysis
approach seem to give confidence in the program since they
test all or most portions of the code. This approach will probably
miss all or some of the bugs and issues that would have been
revealed if the criterion for testing was to expose all risky areas
of the program and make the program fail in many interesting
and challenging ways.
Risk-Based Approach
There are other researchers who have talked about strategizing
domain testing
effort based on risks . In other words, they described a fault-
based or suspicionbased
approach to forming partitions (Beizer, 1995; Collard, personal
communication, July 22, 2003; Frankl et al., 1998; Gerrard &
Thompson, 2002; Hamlet, 2000; Hamlet & Taylor, 1990; Kaner,
2002b; Kaner & Bach, 2003; Myers, 1979; Podgurski & Yang,
1993; Whittaker & Jorgensen, 2002).
Gerrard and Thompson (2002) defined risk as: “A risk threatens
one or more of a project’s cardinal objectives and has an
uncertain probability” (p. 14). Kaner (2002b) characterized risk
as: “Possibility of suffering loss or harm (probability of an
accident caused by a given hazard)” (slide 8). In other words, a
statement describing a risk is an assertion about how a program
or system could fail.
41 http://www.certmagic.com
BH0-003
Collard said, “I often say in classes that a list of assumptions is
a list of risk factors with plans for how we are going to manage
them. I also emphasize the criticality of assumptions (how much
does each one matter?), which is another way of saying we
need to prioritize based on risk” (personal communication, July
22, 2003).
Most of those who describe risk-based approaches to doing
domain testing do not specifically describe forming equivalence
classes or partitions based on risks , but their approach
describes how risks should be identified and how test data
should be selected based on identified risks. Some researchers
have described how data points that represent the most risky
areas in an equivalence class should be selected (Beizer, 1995;
Collard, personal communication, July 22, 2003; Hamlet &
Taylor, 1990; Kaner, 2002b; Kaner & Bach, 2004; Myers,
1979). An example is
selection of test data lying on the boundaries of equivalence
classes. However, Hamlet and Taylor (1990) and Kaner and
Bach (2004) have not specifically described forming partitions or
equivalence classes based on risks or anticipated failures. Kaner
and Bach (2004) identified specific tasks involved in the risk-
based domain testing approach:
The risk-based approach looks like this:
– Start by identifying a risk (a problem the program might
have).
– Progress by discovering a class (an equivalence class) of tests
that could expose the problem.
– Question every test candidate
• What kind of problem do you have in mind?
• How will this test find that problem? (Is this in the right
class?)
• What power does this test have against that kind of problem?
• Is there a more powerful test? A more powerful suite of tests?
(Is this the best representative?)
– Use the best representatives of the test classes to expose
bugs. (part 5, slide 3) Hamlet and Taylor (1990) suggested
development of fault-revealing sub domains. They described the
partition testing method to doing domain testing as a failure-
based approach. They also asserted that a good partition testing
method will help create subdomains, each of which is associated
with a particular failure, and that testing samples from a
subdomain should enable detection of the
associated failure.
42 http://www.certmagic.com
BH0-003
According to Kaner (2002b), risk-based domain testing leads to
development of powerful tests and optimal prioritization,
assuming that correct risks are first identified and then
prioritized. The hazard with using the risk-based approach is
that testers might miss certain risks because they might not
think they are likely or just not be aware of them (Kaner,
2002b).
Whittaker and Jorgensen (2002) described an attack-based
approach to doing domain testing. They argued that testing any
software along four (input, output, storage and computational)
dimensions with the right set of attacks will, to a large extent,
ensure finding the major bugs in software.
Linear or Non-Linear
Not all domains of programs are linear in nature. Kaner and
Bach (2003) described linearizable and non-linearizable
domains. Linearizable variables are ones whose values can be
mapped onto a number line, such as a variable representing a
range of numbers. On the other hand, non-linearizable variables
are those whose values cannot be mapped onto a number line,
such as printers (Kaner et al., 1999; Kaner & Bach, 2003).
Kaner and Bach (2003) also characterized linearizable variables
as variables whose values represent ordered sets and non-
linearizable variables as ones whose values represent non-
ordered sets. Most of the literature on domain testing refrains
from wandering into the territory of non-linear domains, but
there are some who do discuss it.
Jeng and Weyuker (1994) described a simplified domain testing
strategy that is applicable to non-linear domains as much as it is
to linear domains. Zeil et al. (1992b) depicted detection of linear
errors in non-linear domains. According to both Jeng and
Weyuker (1994) and Zeil et al. (1992b), it does not matter
whether the domain is continuous or discrete.
Overlapping or Disjoint Subdomains
Some researchers have suggested that partitioning of input
domain should result in non-overlapping or disjoint partitions
since their analysis is based on a pure mathematical model of
doing partitioning (Howden, 1976; Jorgensen, 2002; Myers,
1979; White & Cohen, 1980). Some people talk about having
disjoint or nonoverlapping partitions because according to them,
43 http://www.certmagic.com
BH0-003
if two partitions (for example, A 24 and B) overlap and someone
were to select test data from the area that forms the
intersection of the two partitions, how would one decide whether
to consider the test data as be longing to partition A or to
partition B, or both?Ntafos (1998) proposed that one way to
resolve the problem of overlapping subdomains is to further
partition the overlapping portion until one gets disjoint
subdomains. Other researchers have relaxed their requirements
and considered the possibility of overlapping subdomains in
their analyses, observing that in practice, overlapping
subdomains are very probable (Jeng & Weyuker, 1989; Kaner et
al.,1999; Weyuker & Jeng, 1991).
Size of Subdomains – Equally Sized or Unequally Sized?
Most partition testing discussions in the literature do not
necessitate forming equally sized domains because the size of
the subdomains is an irrelevant criterion for them. However,
Chan et al. (1998) and Weyuker and Jeng (1991) have asserted
that forming equally sized partitions actually helps improve the
effectiveness of detecting failures by partition testing over
random testing. There is a detailed discussion about this under
section 2.02.02.
Selecting Representatives from Subdomains
Partitioning the input domain is only half the battle. The next
task is to select candidates from each partition to represent
their respective partitions. The literature outlines some visible
distinctions in how the selection of representatives can be done:
Random Selection
Proportional Partition Testing
Risk-Based Selection
o Boundary Value Analysis
o Special Value Testing
o Robustness Testing
o Worst Case Testing
The following section contains detailed descriptions of each of
the items listed above.
Random Selection
Some researchers have taken the word “equivalence” in the
equivalence class quite literally. They have considered all the
44 http://www.certmagic.com
BH0-003
elements of an equivalence class to be equivalent in all
respects, so much so that they would not know how to prefer
a member over any other member of an equivalence class.
Hence, they have suggested random selection of one or
more members from each equivalence class.
Their theory is that the program under test will either pass
(behave as expected) over all members of an equivalence
class or fail (behave differently from what is expected) over
all. According to them, it really does not matter which one is
chosen from a partition (Howden, 1976; Ntafos, 1998;
Podgurski & Yang, 1993; Weyuker & Jeng, 1991; Weyuker &
Ostrand, 1980).
In the literature, those who describe such a random process
of choosing best representatives are usually the ones that do
not specifically describe a risk-based approach to doing
domain testing.
Proportional Partition Testing
This method is a slight variation of the above-defined
random selection method. Some researchers have portrayed
proportional partition testing, a partition testing strategy in
which the test data selection is random but the number of
test cases selected from a subdomain depends on the
probability of failure of inputs in the subdomain. In other
words, if we were to assume that each input in a subdomain
is equally likely to occur, then the number of test cases
selected would depend on the size of the subdomain (Chan
et al., 1997; Chan et al., 1998; Chen et al., 1999; Chen &
Yu, 1994; Chen & Yu, 1996; Leung & Chen, 2000; Ntafos,
1998; Ntafos, 2001).
There have been several discussions in the literature about
the relative merits of partition testing and pure random
testing. Pure random testing is not random selection of test
cases from subdomains, but is just random selection of test
cases from the entire input domain without forming any
partitions. Random and partition testing have been found to
be better than each other under certain conditions, and the
two are almost equivalent under certain other conditions.
Proportional partition testing has mostly been discussed as a
method that evolved in order to improve partition testing and
to make its effectiveness at finding failures greater than that
of random testing. Weyuker and Jeng (1991) noted that
when the original partition testing method is refined so that
45 http://www.certmagic.com
BH0-003
we form all subdomains of equal size and then select an
equal number of test cases from each subdomain, pure
random testing can never beat partition testing in terms of
effectively finding failures. This is assuming that either the
number of subdomains formed is very large or the number of
test cases chosen from each subdomain is very large
compared to the size of the subdomain itself.
Chan et al. (1998) cited a modified version of the
proportional partition testing method that somewhat blends
the refined version of partition testing described by Weyuker
and Jeng (1991) with the traditional proportional partition
testing method.
Chan et al. (1998) called their method the Optimally Refined
Proportional Sampling (ORPS) strategy. This method involves
partitioning the input domain into equally sized partitions as
described by Weyuker and Jeng (1991), and then selecting
one test case at random from each equally sized subdomain.
Chan et al. (1998) noted that the results of trying out their
ORPS strategy on several programs seemed positive enough
to recommend this method as a
subdomain testing strategy.
Ntafos (1998) has illustrated an experiment in which he
compared random testing with proportional partition testing.
When doing proportional partition testing, he considered 50
subdomains. He applied the two strategies to 50 test cases
to start with, and then increased to 2,000 total test cases in
increments of 50 cases.
He referred to the number of test cases as n, the probability
of detecting at least one
failure in subdomain i as Pi and the number of subdomains
as k. Ntafos (1998) found the following:
The experiment was repeated 1,000 times using random
values for the probabilities and the failure rates (but keeping
the overall failure rate about the same). Test case allocation
for proportional partition testing was done by starting with
an initial allocation of ni = max(l, floor(n Pi )) and allocating
each of the remaining test cases so as to minimize the
percentage difference between the current and the true
proportional allocation. With 50 test cases we have that
proportional partition testing was outperformed by random
testing in 154 out of 1,000 cases but the probability of
detecting at least one error is 22.5% higher for proportional
partition testing than
random testing (averaged over the 1,000 cases). With 2,000
test cases, proportional partition testing is outperformed in
46 http://www.certmagic.com
BH0-003
only 35 cases but the two strategies perform equally well in
581 cases.
The probability of detecting at least one error is now only
0.06% higher for proportional partition testing. The variation
of the various values with n is mostly the expected one; note
that the number of cases in which random testing
outperforms proportional partition testing tends to decrease
but the decrease is not monotonic. It is also somewhat
surprising that even with
2,000 test cases random testing still outperforms
proportional partition testing in a significant number of
cases.
Ntafos (1998) repeated his experiment with 100
subdomains. This time, he started with 100 test cases and
increased to a total of 4,000 test cases in increments of 100
cases. “The results are similar; even with 4,000 test cases,
there is still one case when random outperforms proportional
partition testing while the difference in performance is only
0.009%”
Ntafos (1998) concluded that if it requires at least one test
case selected from each subdomain and thousands of test
cases overall to prove that proportional partition testing
beats random testing by an insignificant amount, then the
effectiveness of proportional partition testing when compared
with random testing is questionable not only in terms of cost
(overhead of generation of thousands of test cases) but also
in terms of how effective it is in detecting failures.
Ntafos (1998) also pointed out that partition testing
strategies that rely on random selection of representatives
from subdomains that are considered homogenous are
ineffective in finding certain errors when compared with
testing strategies that involve knowledge-based selection of
test cases. These would include boundary values in domain
testing, selection of test cases based on anticipated failures
or risk-prone areas in case of risk-based approach and
selection of test cases based on important features or
functions of a software. He further argued that forming
perfectly homogeneous subdomains has been practically
impossible, so random selection of test cases from so-called
homogeneous subdomains might not be a great idea after
all.
Risk-Based Selection
47 http://www.certmagic.com
BH0-003
Here we select test cases based on risk. Section 2.01.02.02
discussed how riskbased testing strategies determine risks
and select test cases based on these risks. One of the most
familiar risk-based test case selection strategies is called
boundary values analysis. Jorgensen (2002) also identified
certain other risk-based test selection strategies, such as
special value testing and worst case testing.
Boundary Value Analysis
“Experience shows that test cases that explore boundary
conditions have a higher payoff than test cases that do not.
Boundary conditions are those situations directly on, above,
and beneath the edges of input equivalence classes and
output equivalence classes” .
“A boundary describes a change-point for a program. The
program is supposed to work one way for anything on one
side of the boundary. It does something different for
anything on the other side” (Kaner et al., 1999, p. 399).
Hutcheson (2003) described boundary value analysis as one
of the most important testing techniques. “Boundary value
analysis is a test data selection
technique in which values are chosen to lie along data
extremes. Boundary values include maximum, minimum,
just inside and outside boundaries, typical values, and error
values” (p. 316). Those who practice boundary value analysis
believe that areas on the boundary and around it are risky
areas. This, in fact, is a risk-based strategy.
According to Kaner et al. (2002), in boundary testing the
values of an equivalence class are mapped onto a number
line and the boundary values, which are the extreme
endpoints of the mapping and the values just beyond the
boundaries, are chosen as the best representatives of that
equivalence class. “A best representative of an equivalence
class is a value that is at least as likely as any other value in
the class to expose an error in the software” Kaner et al.
(1999) depicted different kinds of boundaries, some of which
are described below.
⇒ Numeric boundaries: lower and upper boundaries
defined by a range of values or a single boundary
defined by equality.
48 http://www.certmagic.com
BH0-003
⇒ Boundaries on numerosity: boundaries defined by
the length (or ranges of length) of the elements or the
number of constituent characters in the elements.
⇒ Boundaries in loops: minimum and maximum
number of times a loop can execute will determine the
lower and upper boundaries, respectively, for the loop
iterations.
⇒ Boundaries within data structures: boundaries
defined by the lower and upper bounds of structures
that store data.
⇒ Boundaries in space: boundaries defined by the
bounds of objects in two-dimensional or three-
dimensional space.
⇒ Boundaries in time: boundaries defined by time-
determined tasks.
⇒ Hardware-related boundaries: boundaries defined by
the upper and lower bounds of hardware needs and
requirements.
According to Hutcheson (2003), boundary value analysis is
based on the belief that if a system works correctly for the
boundary values, it will also work correctly for all the values
within the range. This makes boundary values the most
important test cases. Hutcheson (2003) explained that
applying boundary value analysis to a month variable will
yield the following six test cases: {0, 1, 2, 11, 12, 13}. The
test cases or data points 1 and 12 are minimum and
maximum values, respectively, that a month variable can
take. Test cases 0 and 2 are just outside and just inside the
boundary defined by test case 1, respectively. Similarly, test
cases 11 and 13 are just inside and just outside the
boundary defined by test case 12, respectively. Hutcheson
(2003) further noted that test cases 2 and 11 seem
redundant, as both of these serve as data points from the
region within the two endpoints. The researcher therefore
suggested replacing the two test cases 2 and 11 with mid-
point data value 6, which makes {0, 1, 6, 12, 13} the final
set of test cases due to boundary value analysis. Some other
researchers have not recommended using any values other
than those on the boundaries and ones that are just beyond
the boundaries, since the
others would simply be redundant. According to them, in the
aforementioned example the test case 6 would also be
redundant (Kaner et al., 2002; Kaner et al., 1999; Kaner &
Bach, 2003; Myers, 1979).
Jorgensen (2002) pointed out one outstanding disadvantage
of boundary value analysis. This analysis works well only
49 http://www.certmagic.com
BH0-003
when there are independent variables that belong to
domains with well-defined boundaries.
Jorgensen (2002) gave an example of a function that
evaluates the next date of the current date. Just applying
boundary value analysis to the date variable of this function
will miss errors corresponding to that of a leap year. Also,
boundary value analysis cannot be applied to non-
linearizable discrete variables.
Special Value Testing
Jorgensen (2002) described this strategy as a widely
practiced form of functional testing. According to him, in this
strategy a tester tests the program or software based on
their knowledge of the problem domain. Testers who have
tested similar software before or are familiar with its domain
of functionality have their own lists of encountered issues
and risky areas for such software and the corresponding
problem domain. Such testers know where the most risk-
prone areas in the software are. Based on their knowledge
and experience, they would then select such “special” test
cases that address those risks and reveal the vulnerability of
the corresponding risk-prone areas.
Jorgensen (2002) noted that for the date example described
in the previous section, special value testing would generate
test cases that test for leap year risks that would be missed
by the normal boundary value analysis technique.
Robustness Testing
Jorgensen (2002) described robustness testing as a simple
extension of boundary value analysis. He explained that the
main focus here is on outputs. This kind of testing is done to
determine how well the program handles situations in which
the value of the expected output exceeds the maximum
tolerated or perhaps falls below the minimum required.
Hence, this testing would involve test case generation that
yields those values for input variables that push the expected
values of the output variables to the extreme and beyond.
Worst Case Testing
50 http://www.certmagic.com
BH0-003
Jorgensen (2002) illustrated this as an extreme version of
boundary value analysis. In this case, one would test using
various combinations of test cases that were generated for
individual variables using boundary value analysis. The
intention here is to see what happens to the software when
variables with such extreme values interact together.
Jorgensen (2002) also asserted that this kind of testing is
useful when variables are known to heavily interact with
each other. However, this has similar shortcomings as with
boundary value analysis when there are dependent variables
involved.
Beizer (1995) contended that extreme points combination
testing might not be an effective test case generation
technique. This is because the number of combinations
increases exponentially with the number of input variables
and when there are dependent variables involved, a large
number of the tests that are generated might be
meaningless. For example, certain combinations might be
impossible due to certain dependency relationships among
the input variables.
Which Test Case Selection Method Should We Use?
Risk-based strategy is the best strategy, especially where cost
and effective failure detection are concerned. Random testing
might be used as a supplement and might be valuable in high-
volume automated test case execution. High-volume automated
random testing would be inexpensive since it is automated, and
having hundreds or thousands of random test cases might help
in finding any bugs that were not revealed by the risk-based
strategy.
In conclusion, risk-based selection of test cases is something
that definitely needs to be done, but high-volume random test
selection might be done in addition if time and resources permit,
especially for regression testing. Kaner et al. (2002) have
described regression testing as follows:
Regression testing involves reuse of the same tests, so you can
retest (with these) after change. There are three kinds of
regression testing. You do bug fix regression after reporting a
bug and hearing later on that it’s fixed. The goal is to prove that
the fix is no good. The goal of old bugs regression is to prove
that a change to the software has caused an old bug fix to
become
51 http://www.certmagic.com
BH0-003
unfixed. Side-effect regression, also called stability regression,
involves retesting of substantial parts of the product. The goal is
to prove that the change has caused something that used to
work to now be broken.
Testing Multiple Variables in Combination
While testing multiple variables together, a test case represents a
combination of input values of these multiple variables. As noted in
the introductory chapter, one would ideally test for all possible
combinations of inputs. But this is practically impossible because
even with a normal commercial program with few variables, all
possible combinations can lead to combinatorial explosion. The
following are some of the techniques described in the literature that
involve combination testing of multiple variables with a reduced
test case set:
• Cause-effect graphs
• Combinatorial testing using input-output analysis
• Pairwise or orthogonal arrays testing
• All pairs combination testing
• Weak robust equivalence class testing
The following sections include brief discussions of each of the
above-listed
items.
Cause-Effect Graphs
Bender (2001), Elmendorf (1973), Elmendorf (1974), Elmendorf
(1975), Myers (1979), and Nursimulu and Probert (1995)
described cause-effect graphing as a combination testing
technique. To start with, using the specification of the program
under test, one identifies causes, effects and constraints due to
the external environment. Next, Boolean graphs are formed with
causes and effects as the nodes and the links joining causes and
the respective effects that represent the relationship between
the causes and effects. The graph is then traced to build a
decision table, which is used to produce test cases.
According to Ostrand and Balcer (1988), the cause-effect
graphing technique can get very complicated and very difficult
to implement, especially when the number of causes is too
large.
52 http://www.certmagic.com
BH0-003
Pairwise / Orthogonal Arrays Testing
One of the solutions to combinatorial explosion is pairwise
testing. “Pairwise testing (or 2-way testing) is a specification
based testing criterion, which requires that for each pair of input
parameters of a system, every combination of valid values of
these two parameters be covered by at least one test case” (Lei
& Tai, 1988, p. 1).
Bolton (2004) defined an orthogonal array (OA) as: “An
orthogonal array has specific properties. First, an OA is a
rectangular array or table of values, presented in rows and
columns, like a database or spreadsheet. In this spreadsheet,
each column represents a variable or parameter”
The value of each variable is chosen from a set known as an
alphabet. This alphabet doesn't have to be composed of
letters—it's more abstract than that; consider the alphabet to be
"available choices" or "possible values". A specific value,
represented by a symbol within an alphabet is formally called a
level. That said, we often use letters to represent those levels;
we can use numbers, words, or any other symbol. As an
example, think of levels in terms of a variable that has Low,
Medium, and High settings. Represent those settings in our
table using the letters A, B, and C. This gives us a three-letter,
or three-level alphabet.
At an intersection of each row and column, we have a cell. Each
cell contains a variable set to a certain level. Thus in our table,
each row represents a possible combination of variables and
values…
Hence, a row in an orthogonal array would represent one
possible combination of values of the existing variables and all
the rows together would comprise the set of all possible
combinations of values for the variables. The combinations
generated due to orthogonal array strategy can quickly become
overwhelming and difficult to manage, especially with today’s
normal commercial
programs that have hundreds of variables. Pairwise testing
tends to alleviate this problem somewhat. It involves testing
variables in pairs and without combining them with other
variables.
The pairwise strategy is helpful if the tester is trying to test for
risks associated with relationships that exist among the pairs,
but not otherwise (Kaner, 2002c). As pairwise testing has been
described in the literature, it seems that the test cases used are
53 http://www.certmagic.com
BH0-003
actually a subset of the set of all test cases that are generated
by orthogonal array testing.
Combinatorial Testing Using Input-Output Analysis
Schroeder and Korel (2000) described the input-output analysis
approach to doing combinatorial testing, asserting that the
credibility of pairwise testing or orthogonal arrays testing in
terms of fault-detecting capability is quite doubtful since it has
not been proven that the reduced data set resulting from these
methods actually has a good failure-detection rate. They
suggested reducing the set of all possible combinations without
losing the ability to detect failures in the given program.
Schroeder and Korel (2000) observed that since not every input
to a program affects every possible output of the program, if
one can identify the influencing subset of input values for every
possible output of a program, the number of combinations of
input-output for a particular output will be significantly lower
compared to the set of all possible input-output combinations
for that output.
They drew input-output relationship diagrams that have inputs
at the top and outputs at the bottom and arrows going down
from inputs to the corresponding outputs they affect. After
doing this, for every output they listed its influencing inputs as
columns of a table. Next, they filled in the rows with values of
the inputs that lead to the corresponding output. After they had
done this for each output, they used their brute-force algorithm
to form a minimal set of combinations of inputs that best
represent each of the outputs. The two researchers considered
their method better than pairwise or orthogonal testing.
All Pairs Combination
Yet another solution to combinatorial explosion is all pairs
combination. This is an extension of pairwise testing, but it
yields more combinations. According to Kaner et al. (2002),
“Every value of every variable is paired with every value of
every other variable in at least one test case” (p. 54). If there
are n number of variables in a program, then combinations
generated due to the all pairs combination technique will contain
n(n-1)/2 pairs in each combination (Kaner et al., 1999).
Cohen, Dalal, Parelius and Patton (1996) considered the all pairs
approach to be far better than the orthogonal arrays approach.
They noted that in the pairwise version of orthogonal testing,
54 http://www.certmagic.com
BH0-003
every pair of values must occur an equal number of times,
which makes this combinatorial approach very difficult to
implement in practice. They gave an example to prove their
point. “For example, for 100 parameters with two values each,
the orthogonal array requires at least 101 tests, while 10 test
cases are sufficient to cover all pairs” (p. 87). However, this
technique is only applicable to independent variables (Kaner &
Bach, 2004, part 19).
Weak Robust Equivalence -Class Testing
Jorgensen (2002) cited four forms of equivalence class testing
involving multiple variables:
Weak Normal Equivalence Class Testing: It is called normal
because only hose test cases that contain valid values for each
variable in the input combination will be considered. It is called
weak because not all possible combinations of valid equivalence
classes of variables involved will be
considered.
Strong Normal Equivalence Class Testing: This is called
normal for the reasons stated above and it is called strong
because there will be at least one test case for each combination
of valid equivalence classes of input variables involved.
Weak Robust Equivalence Class Testing: This is called robust
because there is at least one variable in an input combination
whose value is a representative of an invalid equivalence class
of that variable. This method is weak because in a given
combination of input variables, only one variable has its value
coming from an invalid equivalence class.
Strong Robust Equivalence Class Testing: As mentioned
before, the robust part comes from consideration of invalid
values. The strong part refers to the fact that a single test case
has multiple variables with values coming from their invalid
equivalence classes.
When to Use What Combination Technique?
According to Kaner and Bach (2004), when independent
variables are involved, all pairs is perhaps the most effective
combination technique to use because it generates a minimal
number of combinations when compared with other combination
techniques, such as orthogonal and pairwise. It also considers
55 http://www.certmagic.com
BH0-003
several pairs of multiple variables simultaneously. When
dependent variables are present in a program, then the cause-
effect graphing technique can be used, although this is quite a
complex combination technique (part 19). Kaner and Bach
(2004) also presented a method of combining dependent
variables by constructing relationship tables for variables in a
program and analyzing the dependency relationships existing
between them to generate only meaningful combinations.
Hence, if there are both independent and dependent variables in
a program, one might choose to apply the all pairs method to
the independent variables and then separately test the
dependent variables. This is what I have done in my domain
testing training. The variables are first categorized as
independent or dependent. The independent variables are
combined using the all pairs combination technique and then the
dependent variables, if there are any, are tested separately
using dependency relationship tables.
Testing and Process Recommendations for Software Engineering
Software Engineering Problems
Many small software engineering organizations have problems
cooperating and communicating on projects. A primary reason for this
is that these small organizations often grow out of the individual
efforts of the one or two initial programmers. Projects undertaken by
an individual programmer are small enough for that one person to
understand the requirements, write the code, find the bugs, produce
the deliverable, and provide the ongoing support, often without really
being aware that these are separate tasks. As a result, comments in
the code and user documentation are usually sparse. Other
engineering processes and artifacts - such as repeatable (documented
and/or automated) tests, requirement and specification documents,
and records of bugs that were found and fixed - are often extremely
informal or completely absent.
Reasons why this happens include:
• Many software developers have not had appropriate education
in software processes. Many students have had courses in the
details of specific languages and general programming
constructs (linked lists, hash tables, and so on), but few have
had specific instruction in how to write a requirements
document, how to think about developing unit tests, etc.
56 http://www.certmagic.com
BH0-003
• Small projects can and do succeed without these extra
documents and processes, so the need for those processes are
not perceived. An individual engineer who is working on a
project small enough to keep in his or her head, may reach a
state that he or she can call "deliverable" much more quickly
without producing requirements documents, unit tests, etc. He
or she may not perceive post-delivery support as part of the
engineering cost and may not keep careful track of how much
time is spent on that part of the project. (This is especially true
if the end users come to expect and accept low-quality work and
don’t bother the engineer with reports of annoying but non-
critical bugs.) The engineer may therefore not believe in the
value of these additional processes and may in fact be openly
disdainful.
As a software engineering organization achieves successes, it will tend
to tackle more ambitious projects. Soon there are two features of the
projects that were extremely minimal or absent earlier on:
• The organization is working with customers that
require some useful estimate of when the project will be
completed and how much it will cost. Too many inaccurate
estimates will ruin the organization’s reputation and could
bankrupt it.
• Several engineers (and sometimes a newly-promoted
or newly-hired engineering team leader or manager) will have to
cooperate closely on facets of the project, and must therefore
be able to communicate openly, freely, and in high detail.
These two factors drive the need for improved engineering processes
in a small project group.
The question then is: what processes will provide the payoffs (better
cost and time estimates and efficient cooperation among engineers)
without requiring too much overhead (thereby spending more money
and time than will be saved)?
It is recommended that a small organization should implement
lightweight, informal versions of the following processes:
• Record user requirements, the engineering organization’s later
questions about those requirements, and subsequent resolutions
of those requirements.
• Provide source control of tools, deliverables, and
documentation.
• Record and track bugs.
• Record the tests that are to be run against the deliverable.
• Record the details of each test run.
57 http://www.certmagic.com
BH0-003
• Establish standardized coding and commenting guidelines.
• Conduct peer code review.
This is enough process to show quick payoff without too much time
expenditure. Attempting to implement more process than this too
early in the maturation of an engineering organization is likely to fail.
Individual engineers may be confused and annoyed at the imposition
of too much "overhead", and engineering leadership may not have the
time to educate and enforce additional process. Attempting to
implement more processes than can be taught and enforced is a sure
road to failure.
Let’s look at each of those recommendations in a little more detail.
Each recommendation will be kept as separate as possible. This way,
an organization that quickly comes to agreement about how to
implement any of these suggestions can do so without having to
implement them all at the same time. This is extremely important! The
goal is to help the organization be more productive through slow and
steady change, and not to spend all your time implementing and
arguing about engineering process improvements.
Requirements Documentation
One problem, which is not unique to small organizations, but which
tends to affect them much more than individual programmers, is
incomplete communication and understanding of customer
requirements. One person talks to the customer and finds out what
they want, and other people end up writing the actual code. This
results not only in cases where the first person wasn’t clearly
communicating with the customer, but cases where the engineers
within the organization weren’t clearly communicating with each other.
The first step towards solving this problem is to maintain a written
users requirements document. This means more than just the initial
document that the first customer contact person writes while
discussing the project with the customer! All subsequent questions and
amendments should be recorded directly in the same document. There
are two reasons to do this.
First, at some point in your organization’s growth and maturation, you
will want to be sure that the customer cannot continually withhold
payment with claims of "That wasn’t what I wanted." Maintaining a
customer requirements document means that, at some point, you can
easily derive a contract from the users requirements document, by
adding some language like "...and anything that is not specified above
can be implemented in any way we like."
58 http://www.certmagic.com
BH0-003
Second, by recording the questions and amendments to customers’
requirements document, you build a body of data that can be reviewed
before future customer contacts, so that your people know what issues
to clarify in subsequent projects.
Specific detail about any particular format for these customers’
requirements document will not be discussed. It is much more
important that the document be written down in full, and that
subsequent questions and amendments be written into the same
document, than that the document follow any particular format.
It is strongly recommend that you combine this detailed tracking of
customer requirements with source control, as discussed in the next
section.
Source Control of Tools, Deliverables, and Documentation
One problem that crops up when multiple people cooperate on a
project is that they are simultaneously changing the files that make up
the project. This may result in cases where engineers lose each other’s
changes, nobody knows which of multiple versions of the project are
the "current" one, etc. This problem affects not only the actual
deliverable product, but also internal documents and tools used to
build, test, and describe that product.
It is recommend that a small organization use a simple source code
control tool - say SourceSafe for Windows or RCS for Unix - to allow
engineers to more easily find (and discuss) the versions of these files.
The organization should keep a central version of the project on a
server. This central version should include all source files, internal
tools, documentation, and the fully built version of the product that
corresponds exactly to the versions of those files.
Each engineer should keep a local copy of the entire project on his or
her own machine. When the engineer performs some identifiable
chunk of work, the engineer should build the project on his or her own
machine and check for problems.
Then he or she should warn the organization that he or she is going to
be changing the central repository. It is very important that only one
engineer have the right to update the central repository at one time!
There should be a note on a central whiteboard, or a toy that gets
passed around indicating the right to change the central repository, or
some such physical representation of this right. Once the engineer
gains the right to change the repository, he or she should then update
their local copy of the project with any changes that happened to the
59 http://www.certmagic.com
BH0-003
central repository while he was doing this work, and build again and
check for problems. Then he or she should submit the changes to the
central repository, build again there, and check for problems. Once the
central repository builds with no problems, the engineer should tell the
organization that he or she is done with this batch of work and
relinquish the right to change the repository.
Note that in this section there has been no discussion on what exactly
the engineer should do to check for problems. That is a separate issue.
It is strongly recommend that a small engineering organization be sure
to separate the discussion of source code control from the discussion
of how to check for problems!
The project should be arranged so those tools used to build the project
and the documentation about the project are parallel to the source
files for the project. For example, a simple project that has source
files, database definition file(s) for a standard commercial database,
and some documentation, might be arranged with the following
directories:
MyProject/src: a directory containing all the source files (this directory
may of course have additional structure inside it)
MyProject/bin: a directory that starts out empty, into which the
executables are placed when the project is built. (Opinions vary on
whether intermediate files - object files etc. - should go into this
directory or into the source directory as the project is being built.)
MyProject/doc: a directory that contains documentation relating to the
project. This directory might have further structure to it - for example,
if you have some documentation intended for internal use only, and
other documentation intended for end users, you might create a
MyProject/doc/internal directory and a MyProject/doc/external
directory. If you are recording user requirements with the rigor
recommended in the previous section, there should be a
MyProject/doc/requirements directory as well.
MyProject/database: a directory containing the database-related
file(s).
MyProject/tools: a directory containing tools relating to the project
(test helper programs, documentation readers, etc.)
MyProject/tests: a directory containing the tests.
Someone in the organization should understand the source code
control system well enough to "own" problems with it, provide
information and tools that speed up everyone’s ability to get fresh
60 http://www.certmagic.com
BH0-003
versions of the project and build them, etc. One tool that this person
should provide is a single command that will retrieve a clean copy of
any version of the product and copy it into a clean local area on the
developer’s machine. This command might first just completely blow
away anything in that local area on the developer’s machine, or to
save time it might simply blow away any built pieces (object files,
executables, etc.) and retrieve source files only. The source code
control system owner will be in the best position to decide how to do
this.
Bug Tracking
Bug identification and tracking can be a controversial subject in a
small organization. On the one hand, engineers may not like it when
others point out errors in their code. (Or, to put it another way, they
may worry that records of their mistakes may be used against them in
subsequent performance reviews.) On the other hand, recording bugs
helps make sure that a bug, once discovered, is not forgotten. It’s a
delicate balancing act.
It is recommended that a small organization’s first efforts to track
bugs should be extremely informal. One specific way to do this might
be to implement internal newsgroups for the discussion of bugs related
to the project(s).
So, for example, if OurCompany.com is working on a project called
Project1, you might want to make two internal newsgroups in which
you can discuss issues relating to that project:
oc.project1 -- for discussion relating to Project1
oc.project1.bugs -- for discussion of bugs found in Project1
A bug report might then be a message to the bugs newsgroup that
looks something like this:
Subject: BUG: [jsmith] Tab order in customer-data screen is broken
When you go to the Customer Data screen, click in any of the text
fields, and hit the tab key repeatedly, the order in which the fields get
the keyboard focus jumps all around the screen, rather than
progressing through the screen in an orderly fashion.
The subject line indicates that this is a new bug report, that the
reporter thinks that jsmith is the person who will need to fix it, and
includes a short description of the bug.
61 http://www.certmagic.com
BH0-003
Discussion of the bug ticket should be done with the ‘reply’
mechanism, so that the discussion of each bug is segregated into its
own thread.
Then, when jsmith fixes the bug, he should reply to a message in this
bug thread and change the subject line:
Subject: FIX: [jsmith] Tab order in customer-data screen is broken
I found and fixed the problem. See MyProject/src/customer/foo.asp.
The fix is in the central repository as of 2pm today.
By using the subject field this way, the discussion of a bug will look
like this:
Mary Jones BUG: [jsmith] Tab order in customer-data screen is
broken
John Smith  :Re: BUG: [jsmith] Tab order in customer-data
screen is broken
Mary Jones Re: Re: BUG: [jsmith] Tab order in customer-data
screen is broken
John Smith FIX: [jsmith] Tab order in customer-data screen is
broken
Anyone reading these discussions can easily see which bugs have been
fixed and which ones haven’t.
It is predicted that, some time after the organization is trained to use
internal newsgroups this way, people will want to answer additional
questions about bugs:
• How many bugs are open against the current project?
• What bug should I be working on first?
• Did John Smith’s fix to this bug really fix the problem?
• How many bugs have we been finding against this project, both
now and at various times in the past?
Be aware that a commercial bug-tracking system will make it easier
to answer those questions - but at some cost. Everyone in the
organization has to be consistent in how they report and track
bugs. It often takes a little longer to report, comment on, and
move bugs around in the system than with this newsgroup method.
So it is recommended that you wait until you are sure you are
spending too much time manually sifting through newsgroup
discussions before you go looking for a commercial bug-tracking
solution. By the time you know you are ready for a commercial bug
tracking system, you will know how to identify one that suits your
needs (or at least how to start looking).
62 http://www.certmagic.com
BH0-003
Written Test Scripts and Test Automation
Finding bugs often begins as a time-consuming and haphazard
process. The easiest (and, in some ways, best) way to find bugs is to
have someone use the program exactly as your end users will. Each
engineer should run the product in his or her own area after making
any significant changes before committing the results to the central
repository. Engineering organizations soon learn that testing is a costly
but unavoidable part of the process. You must do a lot of testing
because usually your organization does not fully appreciate how
"interconnected" everything in the product is. Changes made in one
directory or module may not seem to have effects elsewhere, when
they actually do. Unexpected inter-relationships among modules may
result in bugs popping up in unexpected places.
An organization’s first response should be to have a periodic review of
the version of the product in the central repository, where engineers
(and maybe even other people in the organization) play around with
the product in order to find and report bugs.
After you go through this stage a few times, you will probably notice
that "coverage" is haphazard. Bugs can lurk around for quite a while
before being discovered. The same testing tasks may be repeated
many times before other tasks are attempted the first time.
One way to make this "manual" testing effort more efficient and
effective is to write down "scripts" that are to be periodically executed
against the software. These test scripts should be placed under source
code control, just like any other file associated with the project. The
test script should be written in plain English such that someone who is
computer-aware but not necessarily a developer can understand and
carry it out.
For example, consider a simple web page where there are two fields
and an "OK" button. The intention is that the users will type a number
into each of the text fields and then hit the "OK" button. This should
cause another page, which says "the sum of your numbers is
(whatever)", to load.
The test script might start out like this:
1. For each of the following combinations below, enter the first string
in the first field, and then the second string in the second field, and
then click OK. Make sure that the next page shows the correct sum.
a. 3 1
b. 1.2 3.
c. .3 0.2
d. -4 -.3
63 http://www.certmagic.com
BH0-003
2. For each of the following combinations below, enter the first string
in the first field, and then the second string in the second field, and
then hit OK. Make sure that an error dialog comes up saying "Illegal
value" and that you can dismiss that error dialog by hitting its OK
button.
a. a 3.0
b. 1.2 #
c. 0 ‘
d. --4 2
e. 3 3..
f. (blank) 4.0
g. 2.1 (blank)
h. (blank) (blank)
It might then go on to include test cases for more complicated cases,
tab ordering, hitting forward and back on the browser, etc.
There are many reasons to produce and use test scripts like this:
• Written test scripts can be reviewed for ideas that can apply to
other test scripts.
• Multiple people can more easily discuss and cooperate in testing
when a significant portion of the test work is written down.
• Members of your organization who are not programmers can run
test scripts and learn how to write very good test scripts.
Note that even when you have test scripts, you should encourage people
to do some "free play" with the product during test cycles. Encourage
people to keep some kind of record of what they’re doing. Then bugs that
are discovered can be more readily reproduced and isolated, and the
techniques used to uncover those bugs can make their way into
subsequent versions of test scripts. "Free play" testing is very valuable.
After the organization has produced good test scripts and developed the
habit of performing periodic tests, it will become evident that some (or
many) of the tests had great value in their first few iterations, but are
much less likely to actually discover bugs later. You can decrease the rate
at which you run those tests, so as not to spend more time (and money)
running those tests than they’re worth. Another way to attack this
problem is by automating those tests.
Test automation is a fairly sophisticated technique. It is strongly
suggested that your organization go through a few projects with
increasingly rigorous manual test scripts before you embark on any kind
of significant test automation plan. You need to understand many things
about your engineering process before you can make a good decision
about test automation.
64 http://www.certmagic.com
BH0-003
It is suggested at this point that, when the time comes to consider test
automation, you think along the lines of purchasing a commercial
capture-and-playback test automation tool (like Segue’s
QAPartner/SilkTest package) to automate the rigorous manual test scripts
that you have developed. By the time it is a good business idea to do this
test automation, you will recognize the need and will have had plenty of
practice with engineering process improvement. Do your test automation
slowly and carefully!
Written Test Results
Each time someone carries out a test script, or performs some free form
testing, he or she should write down what script was run, what version of
the product it was run against, and what the results were. If you have
implemented the simple newsgroup model discussed above, then a test
results report might look something like this:
Newsgroup: oc.project1
Subject: Test Results - User Data Script 5/13/00 vs. Project1 v1.2.63
I ran the User Data script dated 5/13/00 on Project1 version 1.2.63 (the
version in the central repository as of today at 3pm).
All tests passed except 2c. (the single quote test). I checked the bugs
newsgroup and saw that there is an open bug on this already so I didn’t
file a new one.
Note the distinction between the test script and the test results report.
The script should be submitted to the central repository and versioned the
same as source code. The test results reports can be sent to the project
newsgroup.
These test results reports help engineers reproduce and isolate bugs by
eliminating any possible confusion over when a particular bug was first
discovered.
Coding and Commenting Guidelines
Standardized guidelines ensure that code and documentation is uniform in
appearance, concept, and presentation, regardless of who actually does
the programming. In addition, guidelines truly add professionalism to the
coding process.
A set of coding guidelines should be defined for all of the computer
languages and technologies that are used inhouse. It is recommended
that the guidelines be arrived at by mutual agreement with all the
programmers. Equally important, is a standardized format for
commenting code. Indeed, uncommented code could prove to be a
65 http://www.certmagic.com
BH0-003
potential financial liability in the future, while properly commented code
will make it easier for developers to become acquainted with legacy code
or code created by other programmers.
For development environments/languages that offer it, automated
formatting tools can be purchased to help in the application of the
standards.
Code Reviews
It is recommended that code reviews be performed immediately after
each significant program module or function is created. Ideally, two
reviewers should peruse the code independently. At a minimum, the
reviewers should examine the code with respect to:
• The code must adhere to the coding and comments guidelines.
• The code must be clear and understandable.
• The code must be efficient and logical.
Next, they should meet with the developer to discuss the code and
recommend changes. Such a meeting should stress both the positive and
problematic aspects of the code. All of the agreed upon changes, if any,
should be documented. The developer should then implement the
changes and this should then be verified by either one of the reviewers or
a third person. Finally, the review case is marked closed.
As you can gather, the code review process is essentialy treated in the
same way as bug reports. Issues should be properly documented, they
should be marked complete once they have been addressed, and the
product should not ship until all matters have been addressed.
Testing Strategy
To consider Verification and Validation at the strategic level – what do
the terms mean, what is the difference between static and dynamic
V&V, what do we mean by testing, how does testing relate to the
development process?
Verification & Validation
Any engineered product can be checked in one of two ways:
• check vs. the specified functions - Validation “doing the right thing”
66 http://www.certmagic.com
BH0-003
• check vs. the internal workings – Verification “doing the thing right”
Static and dynamic V&V
• Static V&V - software inspections - concerned with analysis of the
static system representation, in order to discover problems.
– May be supplemented by tool-based document and code analysis.
• Dynamic V&V - software testing - concerned with exercising and
observing product behaviour.
– The system is executed with test data and its operational behaviour is
observed.
What is Testing?
• Testing is the process of executing a program with the intent of finding
errors
• A good test case has a high probability of finding an as-yet
undiscovered error
• A successful test is one that uncovers an as-yet undiscovered error.
Types of testing
• Defect testing
– Tests designed to discover system faults.
– A successful test is one which reveals the presence of faults in a
system.
• Statistical testing
– Tests designed to reflect the frequency of user inputs (i.e., the
operational profile). Is used for reliability estimation.
Testing Shows…
“Testing cannot show the absence of faults, it can only show
that software faults are present.”
However, testing also shows function and performance.
Testing is also used to indicate the quality of the software.
Independent Testing
The DEVELOPERS understand the system - because they built it!
Unsurprisingly, they test to show it works; they test gently,and are
driven by ‘delivery’.
INDEPENDENT testers have to learn about the system, but will attempt
to break it. They are driven by quality.
67 http://www.certmagic.com
BH0-003
Exhaustive Testing
Even in small programs the number of possible logical paths can be
enormous. Take a program about 100 lines long, with a couple of
nested loops executing 20 times each. There are approximately 1014
possible paths that may be executed. At one test per millisecond, that
would be 3170 years alone!
Exhaustive testing is a ‘non-starter’.
Selective Testing
Even if exhaustive testing is a non-starter, white box testing should
not be dismissed. Important logical paths and loops should be tested.
Selective testing validates the interfaces and gives confidence in the
internal workings of the software.
Where Errors Occur
• Requirements Analysis
• System design & specfn
• Program design
• Program implementation
• Unit and integration test
• System test
• Maintenance
Why Errors Occur
1. Incorrect or unclear requirements
2. Incorrect or unclear translation
3. Incorrect or unclear design spec.
4. Misinterpretation of system design
5. Misinterpretation of prog. Design
6. Incorrect documentation
7. Incorrect syntax or semantics
8. Incomplete test procedures
9. Adding new errors when fixing old
10.User documentation, human factors, changing requirements,.....
The Testing ‘V’ Lifecycle
1. Test Plans
2. Test process
3. Requirements traceability
4. Items under test
5. Test schedules
6. Test recording procedures
7. Hardware and software requirements
8. Constraints
68 http://www.certmagic.com
BH0-003
Test Case Design
• OBJECTIVE to uncovers errors
• CRITERIA in a complete manner
• CONSTRAINT with a minimum of effort &time
Testing – Techniques
Testing Shows - “ Testing cannot show the absence of defects, it can
only show that software errors are present.”
Verification & Validation Any engineered product can be tested in one
of two ways:
• test vs. the specified functions - Validation “doing the right thing”
• test vs. the internal workings – Verification “doing the thing right”
Unit Testing
• individual units are tested separately
• units or modules may be single functions, procedures or programs
• done incrementally, usually by the programmer who coded it
• uses stubs and drivers
• white box testing most appropriate at this stage
• tests local data structures, boundary conditions, independent paths,
error handling paths
• a formal test plan should be specified and written
INTERNAL WORKINGS
- testing the working of each function BLACK BOX TEST
- testing that ‘all the gears work WHITE BOX TEST
White Box Testing “to ensure that all statements and conditions have
been executed at least once....”
- Basis path testing...... flow graphs, cyclomatic complexity, deriving
tests
-Control structure testing..... condition tests, data flow tests, loop
tests
Why Cover?
69 http://www.certmagic.com
BH0-003
• Logic errors and incorrect assumptions are inversely proportional to a
path’s execution probability.
• Although we often believe a path is not likely to be executed, reality is
counter-intuitive.
• Typographical errors are random; it’s likely that untested paths will
contain some.
Basis Path Testing (i) First, compute the cyclomatic complexity:
Number of simple decisions (predicate nodes) +1
or
Number of areas
or
Edges - nodes + 2
Basis Path Testing (ii) Next we derive the independent paths.
Since cyclomatic complexity = 4 there are four independent paths:
path 1: 1,2,3,6,7,8
path 2: 1,2,3,5,7,8
path 3: 1,2,4,7,8
path 4: 1,2,4,7,1,2,4,...7,8
Basis Path Testing (iii)
• You don’t need a flow chart, but a picture will help when you trace
program paths.
• Count each simple logical test, compound tests count as two or more.
• Basis path tests should be applied to critical modules
Simple Loop Tests
Minimum conditions for simple loops
• skip the loop entirely
• only one pass through the loop
• two passes through the loop
• m passes through the loop, m<n
• (n-1), n , (n+1) passes through the loop
where n is the maximum number of allowable passes
Nested Loops
• Start at the innermost loop, setting all outer loops to their minimum
values.
• Test the loop, using the steps for Simple Loops (add tests for out-of-
range or excluded values).
70 http://www.certmagic.com
BH0-003
• Move to the next loop out, set up and test as in previous step, holding
all inner loops at typical values and outer loops at their minimum
values.
• Continue until the outermost loop has been tested.
Concatenated Loops
• If the loops are independent of one another then treat each as a
simple loop.
• If the loops are not independent, loop counter for the first loop is the
initial value for the second, treat them as Nested Loops.
Testing Stages
– Subsystem & System Testing
– Acceptance Testing
Other Types of Testing
– Regression Testing
– Debugging
Software Reviews
Look at software reviews from the point of view of software
quality assurance…..
SQA Pays Off
STAGE COST TO FIND & FIX
REQTS ~1
DESIGN ~ 1.5
CODE ~2
TEST ~4
SYS. TEST ~ 15
OPERATION ~ 80-100
(DATA NORMALISED TO COST AT REQTS PHASE)
Quality reviews
• The principal method of validating the quality of a process or of
a product
• Team examines part or all of a process or system and its
documentation to find potential problems
• There are different types of review with different objectives
– Inspections for defect removal (product)
– Reviews for progress assessment (product and process)
– Quality reviews (product and standards)
71 http://www.certmagic.com
BH0-003
Types of Review and Review examples
LIFE CYCLE STAGE REVIEW
REQTS ANALYSIS SPECn WALKTHROUGHS
SOFTWARE DESIGN DESIGN WALKTHROUGHS
CODING CODE WALKTHROUGHS
TESTING TEST PLAN REVIEW
Effectiveness of Reviews
LESS FORMAL CONVERSATION
v PEER GROUP REVIEW
v INFORMAL RESENTATION
v FORMAL PRESENTATION
v WALKTHROUGH
MORE FORMAL INSPECTION
Arguments have been made by various authorities, e.g. see Pressman,
that effectiveness increases with formality
Review Functions
• Quality function - they are part of the general quality management
process
• Project management function – they provide information for project
managers
• Training and communication function - product knowledge is passed
between development team members
Objectives of a Technical Review
• Uncover errors in function, logic, implementn. of the software
• Verify the reviewed item meets requirements
• Ensure the item has been represented using pre-defined standards
• Ensure uniform development
• Make projects more manageable
The Review Team
Review teams should be relatively small and reviews should be fairly
short (90 mins.)
– REVIEW LEADER
– RECORDER
– PRODUCER
– REVIEWER(S)
The Review Process
• Select (put together) review team
• Arrange date, time and location
• Distribute document(s) to be reviewed
72 http://www.certmagic.com
BH0-003
• Confirm (chase-up) review team
• Conduct the review
– Identify actions and those assigned
– Complete review forms
• Monitor review action points
Reviewer Preparation
• Be sure you understand the CONTEXT
• Skim all the product material to understand the location and format of
the information
• Read the product material and annotate
• Pose comments as questions
• Avoid issues of style
• Inform the review leader if you can’t prepare
Conducting the Review
• Evaluate the product before the review
• Review the product, not the producer
• Keep the tone mild, ask questions instead of making accusations
• Stick to the review agenda
• Raise issues - don’t resolve them
• Avoid discussion of style - stick to tech. correctness
• Schedule reviews as project tasks
• Report and record all review results – keep records
Review Results I
• Comments made during the review should be classified.
– No action ~ no change to the software or documentation is required.
(see next PowerPoint)
– Refer back for repair ~ designer or programmer should correct an
identified fault.
– Reconsider overall design ~ the problem identified in the review
impacts other parts of the design - some overall judgement must be
made about the most cost-effective way of solving the problem.
• Requirements and specification errors may have to be referred to the
client.
Review Results II
Occasionally, plans, design, code, or other document(s) are 'signed off'
at a review!
~ Progress to the next review point, or to the next development stage
has been OK’d.
Metrics Derived from Reviews
• inspection time per page of documentation
73 http://www.certmagic.com
BH0-003
• inspection time per KLOC or FP
• inspection effort per KLOC or FP
• errors uncovered per reviewer hour
• errors uncovered per preparation hour
• errors uncovered per SE task (e.g., design)
• number of minor errors (e.g., typos)
• number of major errors (e.g. nonconformance to reqs.)
Tutorial
Software Quality Assurance & Software Quality Control
The Nature of Quality
• Management Commitment
• Words and Money
• Planning (at all levels)
• Communication
• Involvement (“walk the halls”)
• Control
• Measurement - key
Development of Quality
• Deming, Juran, Ishikawa, Shingo, Crosby
• TQM
– kaizen
– atarimae hinshitsu
– kansei
– miryokuteki hinshitsu
What is Quality?
• Basic concept: ‘Fitness for purpose’
• Quality Definition – “The totality of features and characteristics of a
product or service that bear on its ability to satisfy stated or implied
needs.”- ISO 8402
Quality Concepts
• general objective: “reduce the variation between samples” ... what
does mean for software?
• quality control: inspections, reviews, tests
• quality assurance: analysis, auditing and reporting activities
• cost of quality
– appraisal costs, prevention costs
• cost of poor quality
74 http://www.certmagic.com
BH0-003
– failure costs, external (field) failure costs
Software Quality Assurance
Statistical SQA
• Measurement (- of what?)
– Product and Process
• Collect data
• Find causes in the process
• Find a fix, in each case, in the process
Deming
• “ Someone once said: “You cannot inspect quality into a product.”
- he meant that you must build it in!
• Quality Participation – employee participation in decisions
• 14 point quality plan
Quality Assurance
– Juran defined quality assurance, in his Quality Control Handbook, as:
“the activity of providing to all concerned the evidence needed
to establish confidence that the quality function is being performed
adequately”
– Also defined as:
“Systematic activities providing evidence of fitness for purpose of the
total software product”
QA Activities
• prepare a SQA plan
• participate in the definition of a project’s software development plan
(and process model)
• review (software engineering) activities to verify compliance with
defined software process
QA Activities – contd.
• review selected (software) work products to verify compliance with
specifications
• ensure deviations from defined activities and products are documented
and handled according to defined procedures
• record any non-compliances
• regularly report to senior management
Software Quality Planning
“If we fail to plan, we plan to fail….”anon.
Objective:
75 http://www.certmagic.com
BH0-003
Provide a framework for understanding the scope of the problem and
for making estimates of resources, cost and schedule.
Launching a SQA Programme
• initiate the programme
• identify the issues
• write the plan
• establish standards
• establish the function
• train and promote
• implement the plan
• evaluate the programme
Watts Humphrey says:
“The people responsible for the software projects are the only ones
who can be responsible for quality. The role of SQA is to monitor the
way these groups perform their responsibilities.”
Pitfalls
• it is a mistake to think that SQA people (alone) can do anything about
quality
• the existence of a SQA function does not ensure that the standard
procedures are followed
• unless management periodically demonstrates its support for SQA, by
following their recommendations, SQA will be ineffective
• unless line management requires that SQA tries to resolves their
issues with project management before escalation, SQA and
development will not work together effectively.
QA vs. QC
• Precise definitions vary, but broadly we may distinguish between these
as follows:
– software quality assurance concerns the design and monitoring of
an appropriate regime of standards and procedures to achieve high
quality outcomes from system development activities;
– software quality control consists in the conformance to this regime
by all members of a system development team.
Software Quality Factors
• McCall et al, Factors in Software Quality, (1977), sets out a checklist of
factors divided into three categories:
– system operations: factors which focus on the day-to-day use of the
system;
– system revision: factors which address the ease with which changes
can be made to the system;
– system transition: factors which consider the system in relation to
other systems.
76 http://www.certmagic.com
BH0-003
System Operations Factors
• correctness: does the system operate according to its specification?
• reliability: is the system consistently able to produce accurate
results? (Narrow definition?)
• efficiency: does the system avoid unwarranted resource demands?
• integrity: is the system secure from intrusion?
• usability: is it easy for users to learn how to use the system, and
then convenient to use it?
System Revision Factors
• maintainability: how easy is it to fix bugs in the system?
(this is a much narrower definition of ‘maintenance’ than is
usually applied.)
• flexibility: how readily can the system be modified to meet
new/changed requirements?
• testability: has the system been designed to facilitate systematic and
thorough testing?
System Transition Factors
• portability: how easy is it to adapt the system to enable it to operate
in a different hardware and/or software environment?
• reusability: would it be possible and cost-effective to reuse all or
some parts of the system in future development projects?
• interoperability: how readily can the system communicate and
interact with other systems?
Software QA
From the point of view of software quality assurance…..
Quality Management System Standards
To review software quality management system standards;
Quality Management System Elements
Quality Management System Standards
The Auditor
ISO Standards
Review
Success with Test Automation
Over the past several years, tools that help programmers quickly create
applications with graphical user interfaces have dramatically improved
programmer productivity. This has increased the pressure on testers, who
are often perceived as bottlenecks to the delivery of software products.
Testers are being asked to test more and more code in less and less time.
77 http://www.certmagic.com
BH0-003
They need to dramatically improve their own productivity. Test automation is
one way to do this.
This paper presents advice on how to staff, plan and design a test
automation system for GUI applications. I will present some ideas that have
helped me make testsuites that are reliable and easy to maintain. These
concepts and suggestions will be demonstrated by reference to the system
built to test BMC's MetaSuite family of products. These client/server
applications provide an easy-to-use interface for administering open systems
databases.
Taking Test Automation Seriously
Software testers, under pressure to do more testing in less time, often
find themselves rushed and eager for anything that will give them a
hand. Their fear and desperation lead them to seek a "silver bullet"
that will help them regain a rational pace to their work. Test
automation is often this silver bullet. The fantasy is that it will make
their job simpler and easier and help them contend with unrealistic
schedules.
Automating tests for a graphical user interface presents significant
difficulties not found in character based interfaces, much less
command line interfaces or programming interfaces (APIs). Graphical
user interfaces tend to be made of complex components and tend to
be constantly redesigned during the development process. Significant
successes have been made in delivering test tools that are able to
identify and manipulate graphical user interfaces. QA Partner by Segue
Software and Xrunner and WinRunner by Mercury Interactive are
examples.
Most experienced software testers have excellent insights into the
kinds of practices that are critical for software development. They see
the consequences of their developers coding before designing,
neglecting code reviews or ignoring feedback: more bugs and slipped
schedules. However, testers' clear insight into the development
process often fades when they undertake test automation. They fail to
realize that test automation is itself a software development activity
and that it therefore needs to adhere to critical development practices.
Like developers who are stressed for time, testers are prone to
skipping steps, taking the big leap, and blindly hoping to come out
with a success at the other side. But frustration and disappointment
are more likely consequences.
Some testers even try to develop test automation in their spare time. I
have rarely seen this deliver testsuites that can bear much weight.
78 http://www.certmagic.com
BH0-003
Major challenges for GUI test automation are maintainability and
reliability. These challenges demand that a software engineering
approach be taken to address them. Different teams define the
software development process differently. This is fine. The important
thing to remember is that this process should be used with test
automation as well.
Let's look at some of the ways testers and developers are tempted to
underestimate test automation. First off, the name "test tools" makes
them sound simple and easy to use. But they are really development
environments specialized for creating testing programs.
Many testers do not have strong programming skills. This combined
with the repetitive nature of much testing, leads people to use record
and playback techniques. There indeed are many tools that allow
scripts to be recorded and then played back, using screen captures for
verification. The problem that always crops up is that the layouts are
changed, invalidating the screen captures and then the interface
controls change making playback fail. Now the scripts must be re-
recorded from scratch. Record and playback tools provide an easy way
to create throwaway testsuites.
It is particularly hard for people who have had so much success using
these techniques with character-based interfaces to understand the
difficulty of using them with graphical interfaces.
Recording tests and reviewing the created code is an excellent way to
learn how to use a test tool. But most successful test automators
move on to more generalized testing methods.
Someone using a test tool, whether she has the title of tester or
developer, needs to understand how to design, develop and maintain
software. Test automation systems will need to be tested themselves
and should be subjected to frequent review and improvement to make
sure that they are actually addressing the testing needs of the
organization.
Who Should Automate Tests?
I have been a GUI test automation specialist for a couple different
testing groups in the past four years and currently head up a small
team of automation specialists. I think it makes a lot of sense to have
someone focus on these aspects of the testing. These are some of my
thoughts on how to select someone to do this task.
A test automator needs to have good testing and development skills.
She needs to understand testing requirements and the situations
testers face. Automating tests should not be an opportunity to impose
79 http://www.certmagic.com
BH0-003
a particular testing methodology on testers. They will find fault with it
and refuse to use it. Rather it needs to build from existing testing
methodologies.
If the test automator has background as a tester, you will need to ask
if she will show the necessary discipline. Sometimes testers who really
want to be programmers seize on test automation as a way for them
to develop these skills. It is important that they have good judgment
and not get carried away with the programming. Be wary if they are
hoping to automate all of their testing. They need to be focusing on
the big wins. They may focus on improving the automation when it is
actually good enough for the job.
A test automator needs to know how to develop software. She needs
to be particularly aware of issues such as maintenance and reliability.
Making the system easy to update with changes to the product under
test should be the priority.
If her background is as a developer, you will need to ask if she has
understanding and respect for the testing process.
Sometimes you can find independent contractors who have well-
matched backgrounds. With them, you will have to ask who will be
maintaining the testing system after they have left. Maintenance will
be a critical challenge.
If you have access to good training in test automation, take advantage
of it. Developments in test automation are being made very quickly.
It's often cheaper to pay to people learn from someone else's mistakes
than to have to make them make the mistake again themselves.
Don't assign automation to rejects from programming or testing.
Unless test automation is done by someone who is motivated and
working closely with the rest of the development group, it will not
succeed.
Choosing What To Automate
A colleague once asked me if I thought it was theoretically possible to
automate 100% of testing. This question threw me. Theoretically,
testing should never be necessary in the first place. The programs
should be coded correctly first off! But we're not really talking about
the theoretical. Testing is the art of the pragmatic. Good software
testing requires good judgment.
Look at where you are spending a lot of time testing manually. This is
what you should consider automating. If you are a conscientious
tester, you are often aware of things you wished you only had time to
80 http://www.certmagic.com
BH0-003
test. Don't focus your automation efforts on these tasks that may
otherwise go untested. It's usually a mistake. For one thing, you only
want to code automation after you have a determined testing
procedure. If you've run through your manual tests a couple times,
you probably have a solid sense of how to proceed with the testing.
You don't want to automate tests you haven't run much manually and
then realize that there is a more-effective procedure. This may mean
re-working your automation or just giving up on it. Another problem
with automating tests you haven't found the time to run manually is
that you're not likely to find the time to maintain them later. Test
automation always breaks down at some point. Make sure the tests
are important enough that you will be devoting the time to maintain
them when the opportunity arises. First, get your testing procedures
and practices standardized and effective. Then, look at how you can
use automation to improve your productivity.
Testing can be boring. Sometimes people want to automate even
casual tests that will only be executed a couple times. The thought is
that automation may allow them to avoid the tedium. But there are
snags. Complications arise. Test automation will itself have to be
debugged. This may often take as much time as it would to just
execute the tests manually in the first place. I use a rule of thumb that
says test automation will take at least ten times the time it would take
to execute the tests manually. Don't fall into the temptation to
automate simply to make your job more exciting.
Many testers really want to be programmers. Test automation may
provide these people with an opportunity to develop their
programming skills. If you find yourself in this circumstance, try to
stay clear on your goals and how to sensibly use your programming
skills to accelerate the testing. Don't let your attention get caught into
the programming for its own sake. Don't try to automate all of your
testing. Don't be a perfectionist. If your program does the testing,
great. It can have a couple bugs. You're not creating a commercial
product; if there are fatal bugs, you'll be around to fix them. Later I'll
discuss some advice regarding the parts of test automation that must
be reliable. If you are intent to become a programmer, hone your
testing skills while you seek a programming position. They will be
extremely valuable when you get programming work.
Performance is an area where wasted effort can be applied to test
automation. Performance improvements generally depend on
assumptions about the product under test. But since maintainability is
usually fostered by making as few assumptions about how the product
works as is practical, improving performance often reduces
maintainability. Don't do this. Make maintainability a priority. I have
seen performance improvements to test automation that had to be
ripped out when the product changed. They made it harder to maintain
81 http://www.certmagic.com
BH0-003
the testsuite and didn't last long anyway. The best way to allow more
tests to be run in the day is to design your testing system to allow for
unattended testing. I have more say about this later.
Test automation efforts have failed by trying to do too much. You are
better off trying to get first results quickly. This has several
advantages. It will allow you to quickly identify any testability issues.
These may require cooperation from developers and may take some
time to resolve. The sooner they are identified, the better off you are.
You may also wish to just automate the most laborious part of the
testing, leaving such items as setup and verification to be done
manually.
Starting small and building from there will allow you to validate your
testing strategy. Getting early feedback from testers, programmers
and build engineers will allow you to grow your testsuite into
something that will benefit many people. Demonstrate to your
programmers the assumptions you are depending on.
If you've been asked to specialize on the test automation, you may
find a tendency to try to get a big chunk all worked out before handing
it off. Fight this tendency. The sooner you hand off bits to others that
they can use in their daily testing, the better off you all will be. Test
automation may require testers to rethink how they are doing their
job. The sooner they start this, the better. Late in a testing cycle, they
may find themselves putting all their energy into keeping up with the
product changes and the bug list. They may not put much energy into
learning how to use the automation and you may find yourself
frustrated when it goes under used and unappreciated.
First, try to get one test to run. Then build up your testsuite. Realize
that the people using test automation don't care much code you've
written to support the testing. All they will notice is how many tests
are automated and how reliable the testsuite is. After you have a small
suite, you can work on generalizing code and putting together a more
general testing system.
Build acceptance tests are the tests that are run before software is
moved into testing. These should be able to be run quickly, often in
less than an hour. These tests are excellent candidates for automation.
These tests are run frequently. These tests should try to cover as
many different functions of the product as possible. The aim is
breadth, not depth.
It's worth the investment to make your acceptance tests easy to setup
and run. Once the acceptance testsuite is put in place, smart
programmers will want to run it on their code before checking it in.
Finding their own bugs will help them avoid embarrassment and lot of
82 http://www.certmagic.com
BH0-003
rushing around. As a tester, you will want to do all that you can to
make this happen.
In my experience, making good decisions about what to automate can
be critical to successful test automation. There are often many simple
things that can give big paybacks. Find them.
Building Maintainable Testsuites
One of the biggest challenges to using automated testsuites is keeping
them functional as the product interface changes. The BMC Meta
testsuite uses several techniques to allow it to be easily maintained as
our product interface changes.
We use QA Partner for our test automation. This includes tools for
creating "window declarations" which map window controls to logical
names. If the name of a control changes, we can update the window
declaration. All of our test scripts will now work with the revised
product. Another nice feature of this tool is that it can often locate
moved controls. In these cases, we don't need to make any changes to
our testsuite. Window declarations are just one technique we use to
keep our testsuites easy to maintain.
When we have different dialogs with largely the same controls, we use
QA Partner's class hierarchy to set up a superclass that contains the
common controls. Only the unique items are defined for the individual
dialogs. This also simplifies maintenance.
It is very important for us to be able to anticipate user interface
changes. We keep in close communication with our developers on this.
They understand that late and unannounced changes to the user
interface may delay testing. By knowing what parts of the interface
remain subject to design changes, we are able to use common
routines that can be easily updated later.
We also use common code for testing equivalent features in the
different products in the Meta product family. Different products
administer different databases, such as Oracle, Sybase, DB2 and the
like. Generalizing the common aspects has made it easy for us to port
our testing apparatus to new products. It has also reduced the total
lines of code, thus reducing the amount of code to maintain and
debug.
Probably the most significant thing we do to reduce the maintenance
burden is we write our testcases in an abstract testing language.
Testcases only indicate the objects to be manipulated in the testcase.
We build an interpreter and test driver to actually execute the
83 http://www.certmagic.com
BH0-003
testcases. This has many advantages, only one of which is easing
maintenance. I'd like to explain how we do this in more detail.
Building Test Interpreters
A test interpreter and driver allow testcases to be easily specified by a
domain expert. Many testers do not want to have to deal with the
various details of a testing tool. A test interpreter allows them to focus
on testing requirements rather than automation implementation. The
testcase indicates the details of what to test. The test interpreter and
driver actually do the testing. They know how to do the testing. This
arrangement is particularly effective when different people are
responsible for the testcases and the test automation.
Here is an example of one of our testcases.
TEST CASE ID: dtbed101
EDIT TABLE: SA3.TB03
ADD COLUMN(S)
Position NAME TYPE NULLS DEFAULT FOR BIT
DATA LOGGED COMPACT
11 NEW_CHAR_COL_LEN18 CHAR(100) Y N
Note: Column name is of maximum length and is of type char.
EDIT TABLE: SA2.TB03
ADD COLUMN(S)
Position NAME TYPE NULLS DEFAULT FOR BIT
DATA LOGGED COMPACT
11 NEW_INTEGER_COL INTEGER Y N
Note: Column is of type integer.
EDIT TABLE: SA3.TB02
ADD COLUMN(S)
Position NAME TYPE NULLS DEFAULT FOR BIT
DATA LOGGED COMPACT
35 NEW_LOB_COL_AT_END CLOB(5K) Y N - Y
Note: Column is of type clob. Logged is the default.
The format for this testcase was originally based on documentation
that was meant strictly for use by other people. We formalized it and
made it be the actual input language for our test driver. Let me review
some of the advantages to using this kind of format for specifying
testcases.
Testcases are independent of implementation details. Many of our
testcases are specified long before our testers know what the user
84 http://www.certmagic.com
BH0-003
interface will look like. Also, when interface changes are made later,
the testcases don't have to be updated. The testcases only need to be
changed when the product requirements change.
Testers don't have to know test tool details. We leave this for our
automation specialists and those testers who have an interest in the
testing tools.
Testing can focus on requirements. We are able to leverage the
knowledge of our domain experts. We document the testcase format
and this is what they need to know.
Tests are self-documenting. Since the format was originally based on
documentation, the tests are easy to review and understand.
Let me give some more information about how we develop our
interpreters and drivers.
The testcase format which I gave an example of above is actually the
second generation. An example of the first generation format is given
below.
|A|dtbed101|E|TB||SA3|TB03||C|011|colname||NEW_CHAR_COL_LEN
18
| |dtbed101|E|TB||SA3|TB03||C|011|datatype||CHAR(100)
| |dtbed101|E|TB||SA3|TB03||C|011|null||Y
| |dtbed101|E|TB||SA3|TB03||C|011|default||N
| |dtbed101|E|TB||SA2|TB03||C|011|colname||NEW_INTEGER_COL
| |dtbed101|E|TB||SA2|TB03||C|011|datatype||INTEGER
| |dtbed101|E|TB||SA2|TB03||C|011|null||Y
| |dtbed101|E|TB||SA2|TB03||C|011|default||N
|
|dtbed101|E|TB||SA3|TB02||C|035|colname||NEW_LOB_COL_AT_END
| |dtbed101|E|TB||SA3|TB02||C|035|datatype||CLOB(5K)
| |dtbed101|E|TB||SA3|TB02||C|035|null||Y
| |dtbed101|E|TB||SA3|TB02||C|035|default||N
| |dtbed101|E|TB||SA3|TB02||C|035|logged||Y
This format is easier for our automation to parse and execute but is
more difficult to write. It was difficult for our testers to create these
files and they would often make mistakes like putting items in the
wrong column. This would lead to a long process of debugging
testcases. This was often frustrating for testers, who would rather be
finding bugs in the product than in their own test data.
85 http://www.certmagic.com
BH0-003
This testcase was actually created from the first by means of a
translator that converts the information from the first format to the
second. We wrote our translator in Perl. The column format is then the
input to the test driver written in QA Partner.
Let me review several components of our testing system that allows us
to support easy-to-read testcase formats.
Translator. This converts the testcase into a format that is easier for a
program to read. Our translator is written in Perl. QA Partner does not
support the kind of string-matching commands (regular expressions)
that this required.
Testcase Reader. This QA Partner function reads and parses the
intermediate format. Errors in the test data are reported.
Test Driver. This QA Partner script starts the testcase reader and
executes the lines of the testcase. It embodies the testing
methodology and conventions. The driver finishes by triggering the
product to generate a work script. Our test drivers also test things like
memory leaks.
Window Declarations. This QA Partner file identifies the controls. Any
special handling for non-standard controls can be specified here.
Verification. A separate Unix shell script compares the generated work
script against a pre-defined baseline. The script automatically ignores
insignificant differences such as time stamps.
Keeping Your Testsuite Reliable
You will want to be able to depend on your automated testsuite. You
will want it to be able to run it new builds need to be tested. You will
want to trust its accuracy.
The absolute worse thing that can happen to an automated testsuite is
that it reports that tests have passed when there are in fact problems
with the functions they should be testing. This situation is called a
false positive. If your testsuites get a reputation for false positives, no
one will want to use them. They'd rather do the testing manually.
Your test automation will have bugs in it. You will be able to live with
this if automation bugs either result in aborts (the test didn't run) or
false negative (reported failure but no product bug). Generally, you
will want to be manually reproducing reported problems anyway.
86 http://www.certmagic.com
BH0-003
The goal of test automation should be to reduce the number of tests
that need to be run manually, not to eliminate manual testing entirely.
As long as no more than a small portion result in false negatives,
automation will have saved you significant amounts of time. Now you
know the likely places to find bugs: the testcases that failed.
When you are coding your testsuite, you will want to take some
measures to ensure that errors are not hidden or ignored. That is
generally the cause of false positives. The easiest way of inserting this
kind of problem is to make a mistake with exception handling. Double-
check any exception handling code you write or better yet have
someone else review it. The rule of thumb is "When in doubt, call it
out." What this means is that unless your code is sure of the cause of
an error, it should not suppress the reporting of the error. I have also
seen false positives result from switch statements that did not include
a default clause.
Being very careful to avoid false positives will allow you experiment
more with other parts of your testing system. It does not have to be
perfect to be useful.
I have also found that usability is important for perceived reliability. If
it is easy to set up the tests incorrectly without getting good error
messages, testers won't think well of the testing system. They will be
frustrated if they review the results of a test run only to realize they
forgot to set a parameter, meaning that they tested the wrong thing.
If this happens repeatedly, they will realize that test automation is not
saving them time. Be sensitive to these issues. Think about how to
reduce the likelihood for these kinds of problems.
The biggest way to keep your testsuites reliable is to design them so
that they can be run unattended. This will allow you test at overnight
or while you are at meetings. It will also allow you to be testing on
multiple machines at once.
Using Error Recovery Systems
A common problem that prevents productive unattended testing is
cascading failures. This is what happens. One of the tests fails. The
product is left in an unexpected state (perhaps an error dialog is
displayed). Subsequent tests attempt to run but can't because the
error dialog is still displayed. To run the testsuite, the product must be
reset and the testsuite must be restarted after the failed test.
Successive failures will require the testsuite to be rerun again and
again.
An error recovery system is the solution to this problem. It
automatically records the error and then restores the application. This
87 http://www.certmagic.com
BH0-003
allows successive tests to run reliably. Cascading failures are avoided,
allowing unattended testing to occur.
A recovery system needs to know what the "base state" of the product
is. After each testcase, it will check to see if it is in this state. If not, it
will reset the product.
Testcases must be set up properly in order to take advantage of a
recovery system. Each testcase must start and end at the predefined
base state. The base state for our products is just the main window
that appears after starting it. This is a somewhat different approach
from manual testing. Typically, manual testers don't reset the product
before each test, but rather run several tests in succession, only
resetting the product if a problem arises.
One consequence of this is that tests cannot depend on earlier tests.
This principle is called "testcase independence." If a test is meant to
pick up where another finished, it will have to be redesigned. One
option is to include the repetition of the earlier test as part of the
second test. Adhering to testcase independence will allow your tests to
work with a recovery system and run unattended. They will also be
able to be run singly or in a group.
Testcases that are not independent can cause difficulties unrelated to
unattended testing. One may fail when run as part of a battery, but
pass when run alone. You may decide that the bug is irreproducible,
when the problem is really with the testcase.
We've built our error recovery system from code included with QA
Partner as well as code we've written ourselves. Our recovery system
has been customized to recognize our products' error dialogs and to be
able to close down various product dialogs.
We began building ours very early. It helped us debug our test drivers
and test data. Errors that our recovery system logs and handles
include scripting errors, product error dialogs, unexpected product
behavior, and product crash.
We also included code to handle time-out situations, but we are
planning to remove this. The recovery system has not been able to
cleanly shut down the product during a time-out. We've decided it's
better just to wait.
Don't try to engineer your recovery system to recover from all
"possible" errors. Instead, make it handle the actual errors you are
encountering.
Living with Test Automation
88 http://www.certmagic.com
BH0-003
Here are some recent results on our use of test automation during a
busy testing cycle. In one week in March of 1996, we ran the following
number of testcases of the type described above.
Product Unique testcases run Total runs
Product A 399 1005
Product B 43 155
Product C 54 82
You can see that on average tests were run about three times during
that week.
By dedicating people on test automation, we have multiplied the other
testers' productivity and have focused on the testing areas where the
big wins are.
We have accepted "Good Enough" test automation. We've realized that
just like with products we sell, software quality is a combination of
reliability, features and timeliness. Understanding the importance of
avoiding false positives has allowed us to make reasonable quality
trade-offs.
We have tightened our testing cycle. Our testing is more consistent
and repeatable. We are able to test on more configurations. We are
constantly improving our test battery.
Some quotes out of the book from Glenford Myers, ``The Art of
Software Testing'':
``Testing is the process of executing a program with the intent of
finding errors.''
``A good test case is one that has a high probability of detecting an
as-yet undiscovered error.''
``A successful test case is one that finds an error. An unsuccessful
test case is one that causes a program to produce the correct result.''
``One should start with the assumption that the program contains
errors and then test the program to find as many of the errors as
possible.''
Initial Conclusions:
89 http://www.certmagic.com
BH0-003
Testing is a destructive process.
The money invested in the development of tests pays off if enough
errors are found which would have been more costly if they would
have been detected much later (e.g. at the customer's installation).
Tests allow to proof the presence of faults but not the correctness of a
program.
Structurally Guided Black Box Testing
Black-box testing [1, 13, 11] can be easily automated and involves less
processing than white box testing because it does not use information about
the program structure. However it is very hard to achieve high coverage with
black-box testing. Some branches can be very hard to reach.
These branches influence the testability of the code they encapsulate and the
module in turn. A technique which can help black-box testing to cover these
hard to test branches easily will significantly enhance test effectiveness. In
this paper, we propose a simple guided approach which makes black-box
testing more effective using some easily available structural information. we
propose a new dynamic measure termed potential of a branch.
During testing, we extract useful structural information and combine it with
changing coverage information to evaluate the current potential of a branch.
We use thismeasure to guide the black-box testing techniques so that the
new tests generated aremore likely to exercise branches which are so far not
covered. We also present an instrumentation approach called magnifying
branches which may enhance the effectiveness of the new approach. These
magnifying branches help the guided approach to increase the focus on these
harder branches. The coverage results of a black-box testing technique
(random testing) with and without guiding are then compared. Further we
compare these coverage results with those obtained after instrumenting the
programs with the magnifying branches. The results show that this simple
guided technique significantly improves coverage, especially for programs
with complex structural properties.
Software testing is an extensive and difficult process for any realistic
software system. By automating the test generation process, the overall cost
can be significantly reduced. There are several ways of classifying software
testing techniques. One way is to classify them by the the amount of
program information they use. Black-box or functional testing [1, 13, 11]
strategy uses the specifications or the required behaviour of the software as
a starting point to design test cases. White-box testing[1, 13], on the other
hand, uses internal structure of the program to derive test cases. Black-box
testing is conceptually simpler and can be easily automated. It is a commonly
followed strategy for testing a wide variety of software systems. This
90 http://www.certmagic.com
BH0-003
provides the motivation for us to focus on improving the efficacy of black-box
testing techniques.
Black-box testing has a limitation in that we do not know how much of the
code has been covered.
To overcome this limitation, test data adequacy criteria can be used to
measure what has been tested. There are several adequacy criteria [26] that
can be classified by the source of information used to specify testing
requirements(specification or program-based) or by their underlying testing
approach(structural, fault or error-based). We select a structural coverage
criterion : branch coverage. The branch coverage criterion is formally
described in [26] : “ A set P of execution paths satisfies the
branch coverage criterion if and only if for all edges e in the flow graph, there
is at least one path p in P such that P contains the edge e”. This has been
alternatively described as Decision Coverage Criterion [13] and can be stated
as “each branch direction (true,false) must be traversed at least once”. In
actual practice, for large programs, it may not be feasible to completely
satisfy the branch coverage criterion. If the branch coverage achieved is
some minimum acceptable level (85% - 90%), then testing may be regarded
as reasonably thorough. However increasing the coverage achieved helps to
improve the reliability of the software.
Depending on the decision logic in software programs certain branches are
harder to cover, or enter, than others. They are not infeasible but they need
more testing effort to cover. We term these branches as hard to test
branches. Simple black-box testing techniques are not effective for these
branches. This is because they do not use any structural information. This
combined black-box/white-box approach is found to increase coverage
especially as we approach high coverage regions[5] i.e 90% to 100%
coverage. We identify certain structural features that make branches hard to
test. In this paper we propose a simple approach which makes use of this
readily extracted structural information to improve the branch coverage
capabilities of black- box testing techniques. We call this a guided approach
since structural information is used to automatically guide black-box test
generation. We have evaluated this approach by using 4 publicly available
benchmark programs. The results show that a high coverage is obtained with
considerably less effort i.e. number of test vectors required has been at least
halved.
6.1 Testability and Hard to Detect Branches
In this section, we explain how covering the hard to test branches improves
the testability of software programs.
Voas and Miller [20] defines software testability as “the probability that a
piece of software will fail on it’s next execution during testing (with a
particular assumed input distribution) if the software includes a fault”. While
testability is a complex issue, it is usually considered as an attribute of a
91 http://www.certmagic.com
BH0-003
module or a software as a whole. Voas et al [18] give a method to calculate
the testability of a module based on the PIE model. Further, Voas et al [19]
define a metric, fault revealing ability of a test case as a measure of the
likelihood that the test case will produce a failure if a fault were to exist
anywhere in the program. This metric is also calculated based on the PIE
model. This metric is similar in concept to the idea of detectability of a
testing method as defined by Howden and Huang[7]. The PIE model is a
technique that is based on the three part model of software failure. The three
necessary and sufficient conditions for a fault to actually cause a failure
which is detectable is
1. Execution : the location where the fault exists or has an impact on must
be executed.
2. Infection : the program data state is changed by the erroneous
computation.
3. Propagation : the erroneous data state is propagated to the program’s
output causing an error in the output.
Probabilities for each stage have to be calculated for the PIE model. It uses
all three probabilities to calculate either the testability of the code or used to
distinguish test cases by calculating the fault revealing ability metric for each
test case. However the PIE model restates this : If the code is never
executed in the first place then faults can never be detected. This is then
primarily a control flow problem, which means that we have to look at
branches that influence which locations are executed, i.e. they affect the
probability of execution of those locations. Thus testability of a software can
be improved with respect to different test cases used.
If we can execute locations more often we can improve the testability. This is
where branch coverage and particular test data generation schemes becomes
important: to find out which branches are hard to cover and come up with
ways to cover them. This would lead to execution of straight line code that
lies within branches. To further support this notion we could introduce a
modified definition of the branch coverage criterion as in ‘a branch is said to
be covered when it is executed or entered n times’; where n is related to the
testability of the straight line code enclosed by the branch under
examination.
Related Work
There is a lot of literature on various testing techniques. We attempt here to
briefly discuss some of the more relevant approaches. Since our technique
aims to guide testing using structural information, structural testing is of
interest to us. This can be subdivided into program-based and specification-
based[26]. In program-based testing a classification can be based on which
criteria are used. The criteria are explained below.
92 http://www.certmagic.com
BH0-003
Control FlowAdequacy Criteria
Branch coverage is one criterion that falls within this category.
Statement coverage is the most basic criterion that one can use. Path
Coverage Criterion[26, 23] is another which is used as a basis for
pathwise test data generation methods[17].
Two of these methods are symbolic execution[2, 6] and execution-
oriented [8] test data generation. Of particular interest is Korel’
dynamic test data generation method[9]. This differs from the
execution oriented test data generation method in that there is no
longer a focus on covering paths. The goal now is to execute a
selected location. The program is executed with random input and
based on the program execution flow an event sequence is generated.
An event sequence is similar to a path but less formal in that it is
focused towards achieving the goal. Based on this, the search
procedure decides which branch should be taken to achieve the goal. If
a certain branch in the event sequence is not taken, than a real valued
function is associated and function minimization search algorithms are
used to find input that will cover that branch. This basic idea has been
extended to use data dependence analysis[4] and to handle
procedures in programs. Our approach is much simpler. It focuses on
covering branches, at least currently. Moreover it does not need to
extract a function for each branch. So this makes our method
computationally cheaper. This would make our approach more viable
for any realistic software system.
Data Flow Adequacy Criteria
Data flow analysis of test adequacy criteria is based on covering the
flow of data within the program. Based on a variety of data flow
artifacts c-use, p-use, du-paths there are several criteria [21, 3, 10,
15]. These use far more information from the program and though
complex can be used to detect errors much better than control flow
based testing methods. Our objective being to extract as little
information from the program, and to simplify the computational
aspects, we do not consider methods based on these criteria to be
comparable.
The above two categories fall into the white-box category because of
their dependence on information extracted from the program and the
amount and complexity of information needed. Another class of
methods are domain testing[22, 16]. Domain analysis and domain
testing try to partition the input-output behaviour space into sub-
93 http://www.certmagic.com
BH0-003
domains and to come up with test cases that can test these
subdomains.
This is a black-box testing technique since it uses only the functional
information about the program. There are also methods which
generate sub-domains based on some information extracted from the
program. The path condition for a particular path in the program is
derived based on the predicates of the branches in that path. This
serves to define a sub-domain. Then by testing the sub-domain, the
branches that constitute the path will get covered. Our approach is
conceptually similar in that we use some structural information to
guide black-box testing. However, we do not derive sub-domain
conditions and so do not focus on paths nor on deriving path
conditions based on the branch predicates.
The Guided Approach
Our proposed approach makes it possible to generate tests so as to
maximize the branch coverage criterion as quickly as possible. The
actual test generation mechanism is not fixed. What we have is a
framework where we can guide a test generation mechanism to satisfy
the branch coverage criterion.
This is primarily a framework for black box testing techniques.
However the guiding algorithm calculatesmetrics based on the
structure of the branches within the code, thus this could be labeled a
gray-box framework. Any test generation technique or even a suitable
search procedure could be placed into the framework. The framework
only guides the test generation process based on the metric it
calculates.
Factors that make a branch hard to test
What makes a branch hard to test is in itself an interesting problem.
Based on random testing several programs, we have come up with the
follow structural criteria that make a branch hard to test.
1. Nesting factor : Branches that are deeply nested are more difficult
to cover. In the potential measure that we have come up with, we
quantify this difficulty associated with the nesting factor.
2. Branch Predicates : Branches with certain operators in their
predicates are usually harder to cover. We have identified three such
operators and they are : equal to (=), not equal to (!=) and logical
And (&&). We instrument the programs with magnifying branches
around such branches in order to help the guiding algorithm cover
these branches faster.
The Metric : Potential of a branch
94 http://www.certmagic.com
BH0-003
We introduce a term potential of a branch. It is a dynamic metric that
is calculated for each branch after the execution of each test case.
Briefly it is the number of branches that are nested within a branch
which are yet to be covered. It is a combination of both the nesting
level and the number of branches which have not yet been covered.
More significantly it indicates what the potential is, i.e.how many new
branches is it possible to cover if we can somehow cover the current
branch (for which we are calculating this metric). Formally we define it
as follows.
Potential of branch i with respect to a test vector is 0 if it has not been
covered. If branch i has been covered/entered with respect to the test
vector then it’s potential is the sum of the number of uncovered
branches nested immediately inside this branch i and the potentials of
the covered branchesnested immediately inside branch i. If all
branches within branch i have been covered the potential is 0.
The pseudo-code for calculating the potential of a branch is given
below.
potential(branch i ) # with respect to a test vector begin procedure
if branch i is covered
potential = 0
for x in branches j to k # where j to k are branches nested
# immediately inside branch i
if ( branch x is covered by this test vector )
potential = potential + potential ( branch x )
if ( branch x has never been covered )
potential = potential + 1
return potential
else
return 0
end procedure
An example for the calculation of the potential for a slice of code.
if (a <= b) # branch i
{
if (a c) # branch a
{
:
}
if (a < d) # branch b
{
if (d b) # branch x
{
:
}
else # branch y
{
:
}
95 http://www.certmagic.com
BH0-003
7
else # branch c
{
:
}
}
After a test case (or set of test cases),
_ If branch i has not been covered then potential (i) = 0.
_ If branch i has been covered and if only branch c was covered then
potential(i) = (a,b) + potential(
c) = 2 + 0 = 2.
_ If only branch b was covered then potential(i) = (a,c) + potential(b)
= 2 + 2 = 4.
Components of the Framework
The framework itself consists of the metric calculation after each test
vector. Based on the changes in the potential for each branch, the
branch on which the test generation is to be focus on is obtained.
Based on the changes of the potential for that branch, range reduction
or range expansion are carried out to provide constraints for the test
generation mechanism. This is a simple search method to find the data
point that would cover a certain branch or a set of uncovered
branches. During this process, the branch being focused upon might
change when for a test vector some other branch might seem more
promising ( this is decided based on the potential metric).
The components which are needed to use this framework are
1. Test Generation Mechanism
A test generation scheme is needed to generate test vectors. These
test generation schemes need to be able to adjust the test generation
based on constraints which will be specified based on the changes in
the potentials of the branches in the program under test. The test
generation schemes can be Random, Anti-random or any other black-
box test generation techniques. But we currently have used the
random generation technique. In the future we plan to use the anti-
random technique [11, 25] with this framework.
2. Range Adjustment Mechanisms
This would involve specifying a constraint to the test generator to
focus on a certain portion of the input space with a certain test vector
as the center of the focus i.e. to localize the input space.
The way this is done would be dependent on the test generation
technique being used.
_ Reduction. The constraint would reduce the input space under
consideration for the test generator. For the random test generation
96 http://www.certmagic.com
BH0-003
method, this would imply reducing the range for the random
generator.
_ Expansion. The constraint would increase the input space under
consideration for the test generator. For the random test generation
method, this would imply increasing the range for the random
generator.
However the reduction and expansion would be different for the
different methods. In the
case of the anti-random test generation method, reduction could be
taken as reducing the maximum hamming distance that is used to
obtain the next test vector. In this way, the reduction and expansion
operations would be very much dependent on the test generation
method being used.
Description of the algorithm
Initially the test generation technique that has been selected is used
without the guiding mechanism, to cover those brancheswhich are
easy to cover. During the next stage, the potentialmetric is used to
guide further testing. After exercising the program with each test
vector the potentials of all the branches
in the program under test are calculated and the branch with the
highest potential is selected. If the potential of this branch has
increased, then the last test vector (after which the metric was
calculated) is made the center of the input space and range reduction
is carried out so as to localize the input space for further test
generation. If there is no change in the potential after several test
vectors ( number of retries ), then range expansion is carried out so
that other promising regions could be identified in the input space.
When another promising region is found, range reduction is done so as
to focus the test generation in this newly found promising region. The
number of retries is the consecutive number of test cases generated in
a localized region for which there is no change in the potential of the
branch with the maximum potential and the potential of all the other
branches stays lower than this potential. If the number of retries is
exceeded, range expansion takes place. A flowchart of the guiding
algorithm
is shown in Fig 1.
The flow chart includes these steps:
1. Generate a test vector with the selected test generation mechanism
2. If any new branches get covered by this input, then the number of
retries is reset to zero. Otherwise
the number of retries is incremented.
3. If the number of retries exceeds the maximum number of retries
permitted, then range expansion
is carried out.
4. The potential for each of the branches with respect to this test
vector is calculated.
97 http://www.certmagic.com
BH0-003
5. The branch i with the maximum potential is identified.
6. The current maximum potential is compared with the previous
chosen potential. If the current
potential is greater, then the number of retries is initialized to zero and
range reduction is carried
out with a view to focus on branch i.
7. Start from step (1) again till the specified coverage is achieved.
2.5 Magnifying Branches
In order to increase the effectiveness of the above framework, we can
instrument the code. What we propose to instrument with are
magnifying branches. These magnifying branches are used to surround
branches that have predicates whichmay make the branch difficult to
cover. These magnifying branches help the guiding mechanism cover
this branch faster. The magnifying branches’ predicates are derived
from the predicate of the branch under scrutiny. They serve two
purposes
1. to make more effective use of the the range reduction mechanism
2. to increase focus on that branch since the magnifying branch is
easer to cover and has a higher potential.
The magnifying branch is meant to be easier to cover because it’s
predicate is derived from the
original branch’ predicate by relaxing the original predicate. So more
test vectors satisfy this new predicate than that of the original branch.
So whenever this magnifying branch is entered, its potential becomes
higher because of the uncovered branch nested within. When this
potential becomes higher, the guiding algorithmmakes the test
generation mechanism focus on the branch encapsulated by this
magnifying branch. In this way, the magnifying branches enhance the
effectiveness of the guiding mechanism.
A few examples to illustrate the instrumentation process are shown
below.
example 1 if ( a == b )
after instrumentation becomes
if ( ( a = ( b - LOCALITY_RANGE ) ) && ( a <= ( b +
LOCALITY_RANGE) ) )
if ( a== b )
LOCALITY RANGE is a measure of how much a difficult predicate has
been relaxed by. So the
magnifying branch is easier to cover since the range of values that
satisfy it has been increased.
example 2 if ( a && b )
after instrumentation becomes
if ( a ) magnifying branch
if ( b ) magnifying branch
if ( a && b )
98 http://www.certmagic.com
BH0-003
In this way, other difficult predicates too can be magnified. This can be
done recursively if each
expression is a compound expression. This instrumentation is done at
the beginning before the test process is started.
Experiments and Results
Programs used
An experiment was carried out to assess the effectiveness of these
techniques. We selected 4 programs of varying sizes from literature or
which have accepted standard implementations. The programs are
listed in Table 1. They are a mix of program sizes, number of
branches, predicate complexity and nesting level. Some of the
programs have been slightly modified to allow it to interface properly
with ourtool.
Triangle: This program is a standard example for most testing related
literature [24]. It accepts the lengths of three sides of a triangle and
classifies it as scalene, isosceles, equilateral or not a triangle at all.
This is a small example but has a complex decision logic.
Calendar: Calendar is the UNIX calendar program. This is the GNU
implementation of cal. It is a
relatively large program. It accepts thmonth, year and certain options.
The output is the calendar
for the specified range of dates.
Roots: This is an implementation of algorithm 326 from the Collected
Algorithms of the ACM [14].
It accepts the coefficients of a bi-quadratic equation and calculates it’s
roots.
Max: This is a simple program used to test the tool we built initially. It
can be found in any basic
programming text. It accepts 3 numbers and outputs the greatest of
the 3 numbers.
Program Number of lines Number of branches
triangle 90 20
calendar 419 42
roots 245 41
max 37 19
Table 1: Some details of the programs used in the experiment.
Description of the Experiment
We use a tool written by us which automates the technique that we
have described in previous sections. Each program under test is
instrumented by the tool to track which branches are covered. The tool
also keeps track of which test covered which branches. It provides an
interface to the test generator component to calculate the potential of
any branch after each test vector is applied. Currently the magnifying
branches are introduced manually but they could be automated easily.
There are a lot of tools that can calculate the coverages so those tools
could be used to handle this feature of our tool. However to keep our
99 http://www.certmagic.com
BH0-003
tool integrated with advanced features of our technique it includes
even these simple features too.
The test generator generates random test vectors. It interfaces with
the tool in order to use the
potential information. The test generator uses this information along
with the algorithm outlined in the previous section to localize or
expand the ranges for the random test generator. The test generator
has two phases
1. Pure random test vectors are generated to cover all the easy to
cover branches.
2. Random vectors are generated within ranges guided by our scheme
to cover the harder branches.
For each of the programs, we test using three different methods.
Simple Random testing: We carry out pure random testing because
the test generator component usedwith the guided scheme is a
random test generator. So this serves as a benchmark to determine
how the guided random test generator performs better.
Guided testing: The tool is used to help guide the random test
generation mechanism.
Guided testing with Magnifying branches: The program is instrumented
with magnifying branches.
Then guided testing is performed on the instrumented program.
For each of the above methods,we stop the testing after reaching a
certain percentage of coverage.
cover any new (i.e. previously uncovered) branch within a specific
number of retries. If this is
not successful the phase ends indicating failure to satisfy the required
coverage criterion goals. For the current experiment testing was
carried out till 100% branch coverage was achieved or till a plateau w
Results
The results of these tests are shown in graphs Fig (2) - Fig (5). Each
graph plots the % of coverage achieved for the three methods for each
of the programs under study. Table 2 summarizes the complete
results, showing themaximumcoverage achieved for each program and
the number of tests required by each method. In the graphs, the
scales of the x and y axes have been chosen so as to show the
differences between the methods clearly.
Program % Coverage Number of Test Vectors
Achieved Random Testing Guided Testing Using Magnifying Branches
triangle 100 5527 166 316
100 http://www.certmagic.com
BH0-003
calendar 97.62 408 127 243
roots 92.68 3426 1598 429
max 100 8 8 8
Table 2: Coverage achieved (%) Vs Number of tests needed by each
Method
For all programs, the basic technique of potential guided random
testing performs at least as well
as random testing by itself. In case of the triangle, calendar and
roots programs it performs much better, giving a high coverage with
far fewer test cases. In case of the max program, it performs the
same as random testing. This is to illustrate the case when the
program is easily testable.
The magnifying branches method is still a preliminary idea. We are still
studying the benefits of
this method. Currently, it does not always perform better than the
guided random testing method.
The reason for this behaviour is that we magnify branches in the
program under test irrespective of whether they are easy or hard to
test. So even for the relatively easy branches, the magnifying
branches cause quite a few unnecessary tests. The number of
unnecessary tests is exacerbated by the retry factor. So the number of
retries can be adjusted to improve the performance of the magnifying
branches method. The number of retries also affects the simple guided
testing method but to a much lesser ( negligible ) extent.
For the triangle program (figure 2), both the guided random testing
methods perform much better
than the pure random testing method. The graph shows that in order
to obtain about 95% branch
coverage, the pure random testing, guided testing and the magnifying
branches methods require 442, 90 and 303 test vectors respectively.
This in particular shows the effectiveness of the guided testing scheme
for particularly complex decision logic.
For the cal program (figure 3) also, both the guided random testing
methods perform better than
the pure random testing method. The graph shows that in order to
obtain about 90% branch coverage, the pure random testing, guided
testing and the magnifying branches methods require about 400, 120
and 170 test vectors respectively. However the simple guided testing
performs better than that with magnifying branches. From the graph,
we can see the point at which these two diverge. From the study of
the sequence of tests for this program it shows that from this point on,
our previous reasoning for this anomaly holds. This could be rectified
by combining the simple guided and the guidedwith magnifying
101 http://www.certmagic.com
BH0-003
branches methods into amore effectivemethod. This can be done by
having amethod in which after the benefit of the simple guided method
is realized, the magnifying branches can be made use of. For this
program, the maximum coverage achieved is 97.62% i.e. one branch
was not covered. This is because of the weakness of the random test
generator.
The results for the roots program (figure 4) mirror that of theCalendar
program. The pure random
testing scheme takes as much as 10 times ( as seen in the table ) the
number of test cases as does the best method, guided testing with
magnifying branches. The graph shows that in order to obtain about
92% branch coverage, the pure random testing, guided testing and
the magnifying branches methods require about 717, 246 and 400 test
vectors respectively. Also for this program, the guided testing method
with magnifying branches performs much better for the uncovered (
the harder branches ) branches after reaching about 90% coverage
than the simple guided method. This is because it turns out that only
the branch that was hardest to test qualified to have amagnifying
branch. Themaximum coverage achieved for this program is 92.68%
i.e. 3 branches were not covered. In order to understand this, one
must know the structure of this program. There are three functions in
this program. They are quadratic, cubic and biquadratic root
calculation functions. Our driver tests the biquadratic function only
because this in turn makes use of the other two functions. There is
some redundant code in these functions since each of them was coded
to be used independently. Since only the biquadratic function is being
used directly, the redundant code in the other functions is never
exercised and thus the 3 uncovered branches.
The max program (figure 5) is used to point out the fact that
programs with in which the branches
are easy to test do not benefit from the guiding technique. All three
methods achieve 100% branch coverage after 8 test vectors. This is
because for any of the methods, initially the pure random method is
run till saturation after which the guiding is used.
Software Development/Test Phases
A competitive and mature software development organization targets a high
reliability
objective from the very beginning of software development. Generally, the
software life cycle is divided into the following phases. As we will see later,
different testing related tools may be required for different phases.
102 http://www.certmagic.com
BH0-003
A. Requirements and definition: In this phase the developing organization
interacts with the customer organization to specify the software system to be
built. Ideally, the requirements should define the system completely and
unambiguously. In actual practice, there is often a need to do corrective
revisions during software development. A review or inspection during this
phase is generally done by the design team to identify conflicting or missing
requirements. A significant number of errors can be detected by this process.
A change in the requirements in the later phases can cause increased defect
density.
B. Design: In this phase, the system is specified as an interconnection of
units, such that each unit is well defined and can be developed and tested
independently. The design is reviewed to recognize errors.
C. Coding: In this phase, the actual program for each unit is written,
generally in a higher level language such as C or C++. Occasionally,
assembly level implementation may be required for high performance or for
implementing input/output operations. The code is inspected by analyzing
the code (or specification) in a team meeting to identify errors.
D. Testing: This phase is a critical part of the quest for high reliability and
can take 30 to 60% of the entire development time. It is generally divided
into these separate phases.
Unit test: In this phase, each unit is separately tested, and changes are done
to remove the defects found. Since each unit is relatively small and can be
tested independently, they can be exercised much more thoroughly than a
large program.
Integration testing: During integration, the units are gradually assembled
and partially assembled subsystems are tested. Testing subsystems allows
the interface among modules to be tested. By incrementally adding units to a
subsystem, the unit responsible for a failure can be identified more easily.
System testing: The system as a whole is exercised during system testing.
Debugging is continued until some exit criterion is satisfied. The objective of
this phase is to find defects as fast as possible. In general the input mix may
not represent what would be encountered during actual operation.
Acceptance testing: The purpose of this test phase is to assess the system
reliability and
performance in the operational environment. This requires collecting (or
estimating) information about how the actual users would use the system.
This is also called alpha-testing. This is often followed by beta-testing, which
involves actual use by the users.
Regression testing: When significant additions or modifications are made to
an existing version, regression testing is done on the new or "build" version
to ensure that it still works and has not "regressed" to lower reliability.
103 http://www.certmagic.com
BH0-003
E. Operational use: Once the software developer has determined that an
appropriate reliability criterion is satisfied, the software is released. Any bugs
reported by the users are recorded but are not fixed until the next release.
Software Test Methodology
To test a program, a number of inputs are applied and the program
response is observed. If the response is different from expected, the
program has at least one defect. Testing can have one of two separate
objectives. During debugging, the aim is to increase the reliability as
fast as possible, by finding faults as quickly as possible. On the other
hand during certification, the objective is to assess the reliability, thus
the fault finding rate should be representative of actual operation. The
test generation approaches can be divided into the classes.
A. Black-box (or functional) testing: When test generation is done by
only considering the
input/output description of the software, nothing about the
implementation of the software is
assumed to be known. This is the most common form of testing.
B. White-box (or structural) testing: In this approach the actual
implementation is used to generate the tests. While test generation
using the white-box approach is not common, evaluation of test
effectiveness often requires use of structural information.
Coverage Measures
The extent to which a program has been exercised can be evaluated
by measuring software test coverage [mal95]. Test coverage in
software is measured in terms of structural or data-flow units that
have been exercised. These units can be statements (or blocks),
branches, etc. as defined below: Some popular coverage measures are
often referred to by using a compact notation, these are given the
parenthesis.
Statement coverage (C0): the fraction of the total number of
statements that have been executed by the test data.
Branch coverage (C1): the fraction of the total number of branches
that have been executed by the test data.
C-use coverage: the fraction of the total number of computation uses
(c-uses) that have been covered during testing. A c-use pair includes
two points in the program, a point where the value of a variable is
defined or modified followed by a point where it is used for
computation (without the variable being modified along the path).
P-use coverage: the fraction of the total number of predicate uses (p-
uses) that have been covered during testing. A p-use pair includes two
points in the program, a point where the value of a variable is defined
or modified followed by a point which is a destination of a branching
104 http://www.certmagic.com
BH0-003
statement where it is used as a predicate (without modifications to the
variable along the path).
It has been shown that if all paths in the program have been
exercised, then all p-uses must have been covered. This means that
all-paths coverage requirement is stronger than all-p-uses. Similarly
all-p-use coverage implies all-branches coverage and all-branches
coverage implies allinstructions coverage. This is termed the
subsumption hierarchy.
Module coverage (S0): the fraction of the total number of modules
that have been called during testing. A module is a separately
invocable element of a software system, sometimes also called
procedure, function, or program.
Call-pair coverage (S1): the fraction of the total number of Call-pairs
that have been used during testing that have been used during
testing. A call-pair is connection between two functions in which one
function "calls" (references) the other function.
Path coverage: the fraction of the total number of paths that have
been used during testing. A path is any sequence of branches taken
from the beginning to the end of a program. To achieve 100% path
coverage, all permutations of paths must be executed. Tools for
different phases are examined below. Some tools are applicable to
multiple phases. Some of the types of tools are now widely used,
others have just started to emerge.
Requirements phase Tools
Requirement Recorder/Verifier:
Requirements can be recorded informally in a natural language like
English or formally
using Z, LOTOS, etc. Use of formal methods results in a more
thorough recording of requirements. The requirement information
needs to be unambiguous, consistent and complete. A term or an
expression is unambiguous if it has one and only one definition. A
requirements specification is consistent if each term is used in the
same manner for each occurrence. Completeness implies presence of
all needed statements, and all required components for each
statement. The requirement verifiers can automatically check for
ambiguity, inconsistency and completeness of statements. However,
they cannot determine that the set of requirement statements is
complete. This would require review by human testers. A requirements
recorder may also assist in specification-based test case generation.
Test Case Generation:
105 http://www.certmagic.com
BH0-003
Automatic test case generation can be an extremely important part of
achieving high
reliability software. Manual test care generation is a slow and labor
intensive process and may be insufficient if not done carefully.
Arbitrarily generated tests can find defects with high testability
relatively easily; however, these tests can become ineffective as
testing progresses. Specificationbased test generation can ensure that
the different test cases cover at least some different functionality by
partitioning the functionality and probing the portions. Either the
input-space or the state space may be partitioned. Poston [ pos98]
classifies the strategies used as active driven (to test for missing
actions), data driven, logic driven, event driven and state driven.
Both Validator (Aonix) and T-VEC (T-VEC) include specification
verification and test case generation.
Orthogonal to the test generation strategy is question of test vector
distribution. The
distribution may be chosen to confirm with the operational profile, so
that the tests replicate the normal operation. On the other hand, the
strategy, at reach step, may choose to probe the functionality that has
been relatively untouched by testing so far. The second approach may
be implemented in the form of antirandom testing [yin97]. A
combinatorial design based test generation can significantly reduce the
number of combinations to be considered.
It is also possible to generate tests using the software implementation
formation. Some
tools can use this approach termed “white-box” testing. Such test
generation can require enormous amounts of computation, and thus
should be considered only for branches, p-uses, etc. which are very
hard to test otherwise. Beizer has called such testing “kiddie testing at
its worst.” Such tests cannot detect missing functions
Programming Phase Tools
These are often called “static” tools, because they do not involve
actual execution of the
software.
Metrics Evaluators:
Many metrics have been used to evaluate aspects of complexity of
programs. They include lines of code, number of modules, operands,
operators or data/control flow measures. The belief is that if a module
is more complex, it is more fault-prone and thus deserves special
attention. It has been shown that many metrics are strongly correlated
to the number of lines of code, and may not provide any information
[ros97]. Still when the resources and time are limited, it may be a
106 http://www.certmagic.com
BH0-003
good strategy to identify the fault prone modules. Poston regards such
tools to be “nice” but not
essential.
Code Checkers:
These are also static tools like metric evaluators. These tools look for
violations of good
programming practices to generate warnings. They can identify
misplaced pointers, uninitialized variables and non-conformance to
coding standards Testing Phase Tools. STW/Advisor (Software
Research) includes both code checking and metric evaluation.
Inspection Based Error Estimation:
A design document or code can be inspected. Many defects can often
be detected simply by inspection. If separate teams or individuals do
inspection independently, it amounts to sampling the defects present.
Statistical methods are available that can be used to obtain a
preliminary estimate of the remaining number of bugs remaining
Testing Phase Tools
These tools were the earliest to appear and are now widely used. They
are often termed
“dynamic” because they involve actual execution of the software using
the test cases selected and evaluation of test thoroughness.
Capture-Playback Tool:
This is somewhat like a VCR or perhaps more closely like recording and
running
spreadsheet macros. Older capture-playback tools are worked at the
bit-map level. Modern tools can capture and replay information at bit-
map, widget, object or control levels. Information captured can be
edited to replace hard coded values and path name to make them
more general by passing setting environment variables and passing
parameter values. One can build a library of small test scripts which
can be combined to obtain different test sequences. A test sequence
can be implemented by using a state table as a driver.
An alternative is to have data driven scripts that input data as well as
parameters and environment variables. Using appropriate data values,
the same test scripts can be made to cover different functionalities of
the program. The data files can also contain the expected results for
specific test cases, e.g. success or failure. Most capture-replay tools
today incorporate a
comparator, which compares the expected and actual outputs. QA
Partner (Seague) and WinRunner/Xrunner (Mercury Interactive) are
examples of this class of tools.
Memory Leak Detectors:
107 http://www.certmagic.com
BH0-003
Modern programming practices use dynamic memory allocation. If a
program fails to
deallocate memory that is no longer being used, it keeps on reserving
more and more of the memory to itself, until eventually it runs out of
memory. Such memory leaks can be detected by tools like
BoundsChecker (Relational Software) or Purify (Purify).
Test Harness:
A software under test needs to interface with a capture-replay tool as
well as a data-base system and perhaps with other systems also.
These interfaces also need to be tested. Such a test execution
environment is termed a test harness. They may include “stubs” to
stimulate missing parts. In the past, test harnesses have been custom
built. Some test harness generators like Cantata (Information
Processing) have recently become available
Coverage Analyzers:
It is impossible to test a program exhaustively. The testers must
decide if a program has
been exercised sufficiently thoroughly. One way is to use a coverage
analyzer which will keep track of the fraction of all structural or data-
flow units that have been exercised. It has been shown that coverage
measures are approximately linearly correlated with the defect
coverage
Most analyzers are intrusive. They “instrument” the code by inserting
test probes in the
software, before it is compiled. Instrumenting affects the performance
slightly. Non-intrusive analyzers are a much more expensive
alternative; they collect information using a separate hardware
processor.
Statement coverage is not a rigorous measure even with 100%
coverage, the residual defect density ca still be high. Branch coverage
is a popular measure. Sometimes a threshold value, say 85% branch
coverage is used. Pure coverage is stricter than branch coverage and
is suitable for high reliability programs. Module coverage and call-pair
coverage are common system level coverage measures. At the present
time use path coverage is feasible only if its definition is revised to
reduce the total path count. GCT (Testing Foundations) and ATAC
(Bellcore) are coverage analyzers.
Load/performance tester:
These tools allow stress testing of client/server applications, which are
often expected to
work correctly under high loads. SQA LoadTest (SQA) allows stress
testing of Windows
108 http://www.certmagic.com
BH0-003
client/server applications, Final Exam Internet Load Test (Platinum) is
specifically for web applications.
Bug-tracker:
A bug-tracker records the status of each bug found. Depending on the
strategy used, a bug may or may not be fixed immediately after it is
found. A bug-taker is basically a data-base tool.
Reliability Growth Modeling tools:
As defects are found and removed, the reliability of the program
increases. This is
manifested by a gradual decline in the defect finding rate. A wealth of
methods is available that use software reliability growth models
(SRGMs). Several SRGM tools are available that have these features:
1. Pre-processing or smoothing of data
2. Estimating parameters of a selected SRGM
3. Answering queries like how much larger the software needs to be
tested
SMERFS (NSWRC) is a popular SRGM tool. ROBUST (CSU) allows
parameters of SRGMs to be estimated even before testing begins,
which can be useful for preliminary resource planning.
Coverage based Reliability Tools:
Recently, a model describing defect coverage and test coverage has
been proposed and validated. The model tends to fit the data quite
closely and can yield very stable estimates of the number of residual
defects . ROBUST (CSU) allows coverage to be used as the stopping
rule criterion. It also allows stable estimation of the number of residual
defects.
Identifying Tools Needed
The software testing tools can be expensive. The cost to license a tool
can be just a fraction of the overall cost. The testers need to
understand the tools and become familiar with them. The use of the
tools needs to be incorporated in the process.
The “Big 3” tools.
1. Requirement recorder/test case generator
2. Test execution tool
3. Test evaluation tool
Not all projects need sophisticated tools. Many can significantly benefit
by using some of the simpler tools. One good approach to identify the
tools needed is to consider the quality required in the final project. A
measure of quality called TestWorks Quality Index has been defined by
Software Research .
109 http://www.certmagic.com
BH0-003
Software Testing Fundamentals
Testing objectives include
1. Testing is a process of executing a program with the intent of
finding an error.
2. A good test case is one that has a high probability of finding an
as yet undiscovered error.
3. A successful test is one that uncovers an as yet undiscovered
error.
Testing should systematically uncover different classes of errors in a
minimum amount of time and with a minimum amount of effort. A
secondary benefit of testing is that it demonstrates that the software
appears to be working as stated in the specifications. The data
collected through testing can also provide an indication of the
software's reliability and quality. But, testing cannot show the absence
of defect -- it can only show that software defects are present.
White Box Testing
White box testing is a test case design method that uses the control
structure of the procedural design to derive test cases. Test cases can
be derived that
1. guarantee that all independent paths within a module have been
exercised at least once,
2. exercise all logical decisions on their true and false sides,
3. execute all loops at their boundaries and within their operational
bounds, and
4. exercise internal data structures to ensure their validity.
1.1.2 The Nature of Software Defects
Logic errors and incorrect assumptions are inversely
proportional to the probability that a program path will be
executed. General processing tends to be well understood while
special case processing tends to be prone to errors.
We often believe that a logical path is not likely to be executed
when it may be executed on a regular basis. Our unconscious
assumptions about control flow and data lead to design errors
that can only be detected by path testing.
Typographical errors are random.
Basis Path Testing
110 http://www.certmagic.com
BH0-003
This method enables the designer to derive a logical complexity
measure of a procedural design and use it as a guide for
defining a basis set of execution paths. Test cases that exercise
the basis set are guaranteed to execute every statement in the
program at least once during testing.
Flow Graphs
Flow graphs can be used to represent control flow in a program
and can help in the derivation of the basis set. Each flow graph
node represents one or more procedural statements. The edges
between nodes represent flow of control. An edge must
terminate at a node, even if the node does not represent any
useful procedural statements. A region in a flow graph is an
area bounded by edges and nodes. Each node that contains a
condition is called a predicate node. Cyclomatic complexity is a
metric that provides a quantitative measure of the logical
complexity of a program. It defines the number of independent
paths in the basis set and thus provides an upper bound for the
number of tests that must be performed.
The Basis Set
An independent path is any path through a program that introduces at
least one new set of processing statements (must move along at least
one new edge in the path). The basis set is not unique. Any number of
different basis sets can be derived for a given procedural design.
Cyclomatic complexity, V(G), for a flow graph G is equal to
1. The number of regions in the flow graph.
2. V(G) = E - N + 2 where E is the number of edges and N is the
number of nodes.
3. V(G) = P + 1 where P is the number of predicate nodes.
Deriving Test Cases
1. From the design or source code, derive a flow graph.
2. Determine the cyclomatic complexity of this flow graph.
o Even without a flow graph, V(G) can be determined by counting
the number of conditional statements in the code.
3. Determine a basis set of linearly independent paths.
o Predicate nodes are useful for determining the necessary paths.
4. Prepare test cases that will force execution of each path in the basis
set.
o Each test case is executed and compared to the expected
results.
Automating Basis Set Derivation
111 http://www.certmagic.com
BH0-003
The derivation of the flow graph and the set of basis paths is amenable to
automation. A software tool to do this can be developed using a data
structure called a graph matrix. A graph matrix is a square matrix whose size
is equivalent to the number of nodes in the flow graph. Each row and column
correspond to a particular node and the matrix corresponds to the
connections (edges) between nodes. By adding a link weight to each matrix
entry, more information about the control flow can be captured. In its
simplest form, the link weight is 1 if an edge exists and 0 if it does not. But
other types of link weights can be represented:
• the probability that an edge will be executed,
• the processing time expended during link traversal,
• the memory required during link traversal, or
• the resources required during link traversal.
Graph theory algorithms can be applied to these graph matrices to help in
the analysis necessary to produce the basis set.
Loop Testing
This white box technique focuses exclusively on the validity of loop
constructs. Four different classes of loops can be defined:
1. simple loops,
2. nested loops,
3. concatenated loops, and
4. unstructured loops.
Simple Loops
The following tests should be applied to simple loops where n is the
maximum number of allowable passes through the loop:
1. skip the loop entirely,
2. only pass once through the loop,
3. m passes through the loop where m < n,
4. n - 1, n, n + 1 passes through the loop.
Nested Loops
The testing of nested loops cannot simply extend the technique of simple
loops since this would result in a geometrically increasing number of test
cases. One approach for nested loops:
1. Start at the innermost loop. Set all other loops to minimum values.
112 http://www.certmagic.com
BH0-003
2. Conduct simple loop tests for the innermost loop while holding the
outer loops at their minimums. Add tests for out-of-range or excluded
values.
3. Work outward, conducting tests for the next loop while keeping all
other outer loops at minimums and other nested loops to typical
values.
4. Continue until all loops have been tested.
Concatenated Loops
Concatenated loops can be tested as simple loops if each loop is independent
of the others. If they are not independent (e.g. the loop counter for one is
the loop counter for the other), then the nested approach can be used.
Unstructured Loops
This type of loop should be redesigned not tested!!!
Other White Box Techniques
Other white box testing techniques include:
1. Condition testing
o exercises the logical conditions in a program.
2. Data flow testing
o selects test paths according to the locations of definitions and
uses of variables in the program.
Black Box Testing
Black box testing attempts to derive sets of inputs that will fully exercise
all the functional requirements of a system. It is not an alternative to
white box testing. This type of testing attempts to find errors in the
following categories:
1. incorrect or missing functions,
2. interface errors,
3. errors in data structures or external database access,
4. performance errors, and
5. initialization and termination errors.
Tests are designed to answer the following questions:
1. How is the function's validity tested?
2. What classes of input will make good test cases?
3. Is the system particularly sensitive to certain input values?
4. How are the boundaries of a data class isolated?
113 http://www.certmagic.com
BH0-003
5. What data rates and data volume can the system tolerate?
6. What effect will specific combinations of data have on system
operation?
White box testing should be performed early in the testing process, while
black box testing tends to be applied during later stages. Test cases
should be derived which
1. reduce the number of additional test cases that must be designed
to achieve reasonable testing, and
2. tell us something about the presence or absence of classes of
errors, rather than an error associated only with the specific test at
hand.
Equivalence Partitioning
This method divides the input domain of a program into classes of data from
which test cases can be derived. Equivalence partitioning strives to define a
test case that uncovers classes of errors and thereby reduces the number of
test cases needed. It is based on an evaluation of equivalence classes for an
input condition. An equivalence class represents a set of valid or invalid
states for input conditions.
Equivalence classes may be defined according to the following guidelines:
1. If an input condition specifies a range, one valid and two invalid
equivalence classes are defined.
2. If an input condition requires a specific value, then one valid and two
invalid equivalence classes are defined.
3. If an input condition specifies a member of a set, then one valid and
one invalid equivalence class are defined.
4. If an input condition is boolean, then one valid and one invalid
equivalence class are defined.
Boundary Value Analysis
This method leads to a selection of test cases that exercise boundary values.
It complements equivalence partitioning since it selects test cases at the
edges of a class. Rather than focusing on input conditions solely, BVA derives
test cases from the output domain also. BVA guidelines include:
1. For input ranges bounded by a and b, test cases should include values
a and b and just above and just below a and b respectively.
2. If an input condition specifies a number of values, test cases should be
developed to exercise the minimum and maximum numbers and
values just above and below these limits.
3. Apply guidelines 1 and 2 to the output.
114 http://www.certmagic.com
BH0-003
4. If internal data structures have prescribed boundaries, a test case
should be designed to exercise the data structure at its boundary.
Cause-Effect Graphing Techniques
Cause-effect graphing is a technique that provides a concise representation
of logical conditions and corresponding actions. There are four steps:
1. Causes (input conditions) and effects (actions) are listed for a module
and an identifier is assigned to each.
2. A cause-effect graph is developed.
3. The graph is converted to a decision table.
4. Decision table rules are converted to test cases.
GUI Testing Checklist
For Each Window in the Application
⇒ If Window has a Minimise Button, click it. Window Should return to an
icon on the bottom of the screen
⇒ Double Click the Icon to return the Window to its original size.
⇒ The window caption for every application should have the name of the
application and the window name - especially the error messages.
These should be checked for spelling, English and clarity , especially on
the top of the screen. Check does the title of the window makes
sense.
⇒ If the screen has an Control menu, then use all ungreyed options.
⇒ Check all text on window for Spelling/Tense and Grammar
⇒ Use TAB to move focus around the Window. Use SHIFT+TAB to move
focus backwards.
⇒ Tab order should be left to right, and Up to Down within a group box
on the screen. All controls should get focus - indicated by dotted box,
or cursor. Tabbing to an entry field with text in it should highlight the
entire text in the field.
⇒ The text in the Micro Help line should change - Check for spelling,
clarity and non-updateable etc.
⇒ If a field is disabled (greyed) then it should not get focus. It should not
be possible to select them with either the mouse or by using TAB. Try
this for every greyed control.
⇒ Never updateable fields should be displayed with black text on a grey
background with a black label.
⇒ All text should be left-justified, followed by a colon tight to it.
⇒ In a field that may or may not be updateable, the label text and
contents changes from black to grey depending on the current status.
⇒ List boxes are always white background with black text whether they
are disabled or not. All others are grey.
115 http://www.certmagic.com
BH0-003
⇒ In general, do not use goto screens, use gosub, i.e. if a button causes
another screen to be displayed, the screen should not hide the first
screen, with the exception of tab in 2.0
⇒ When returning return to the first screen cleanly i.e. no other
screens/applications should appear.
⇒ In general, double-clicking is not essential. In general, everything can
be done using both the mouse and the keyboard.
⇒ All tab buttons should have a distinct letter.
Text Boxes
⇒ Move the Mouse Cursor over all Enterable Text Boxes. Cursor should
change from arrow to Insert Bar. If it doesn't then the text in the box
should be grey or non-updateable. Refer to previous page.
⇒ Enter text into Box
⇒ Try to overflow the text by typing to many characters - should be
stopped Check the field width with capitals W.
⇒ Enter invalid characters - Letters in amount fields, try strange
characters like + , - etc. in All fields.
⇒ SHIFT and Arrow should Select Characters. Selection should also be
possible with mouse. Double Click should select all text in box.
Option (Radio Buttons)
⇒ Left and Right arrows should move 'ON' Selection. So should Up and
Down.. Select with mouse by clicking.
Check Boxes
⇒ Clicking with the mouse on the box, or on the text should SET/UNSET
the box. SPACE should do the same.
Command Buttons
⇒ If Command Button leads to another Screen, and if the user can enter
or change details on the other screen then the Text on the button
should be followed by three dots.
⇒ All Buttons except for OK and Cancel should have a letter Access to
them. This is indicated by a letter underlined in the button text. The
116 http://www.certmagic.com
BH0-003
button should be activated by pressing ALT+Letter. Make sure there is
no duplication.
⇒ Click each button once with the mouse - This should activate
Tab to each button - Press SPACE - This should activate
Tab to each button - Press RETURN - This should activate
The above are VERY IMPORTANT, and should be done for EVERY
command Button.
⇒ Tab to another type of control (not a command button). One button on
the screen should be default (indicated by a thick black border).
Pressing Return in ANY no command button control should activate it.
⇒ If there is a Cancel Button on the screen , then pressing <Esc should
activate it.
⇒ If pressing the Command button results in uncorrectable data e.g.
closing an action step, there should be a message phrased positively
with Yes/No answers where Yes results in the completion of the action.
Drop Down List Boxes
⇒ Pressing the Arrow should give list of options. This List may be
scrollable. You should not be able to type text in the box.
⇒ Pressing a letter should bring you to the first item in the list with that
start with that letter. Pressing ‘Ctrl - F4’ should open/drop down the
list box.
⇒ Spacing should be compatible with the existing windows spacing (word
etc.). Items should be in alphabetical order with the exception of
blank/none which is at the top or the bottom of the list box.
⇒ Drop down with the item selected should be display the list with the
selected item on the top. Make sure only one space appears, shouldn't
have a blank line at the bottom.
Combo Boxes
⇒ Should allow text to be entered. Clicking Arrow should allow user to
choose from list
List Boxes
⇒ Should allow a single selection to be chosen, by clicking with the
mouse, or using the Up and Down Arrow keys.
⇒ Pressing a letter should take you to the first item in the list starting
with that letter.
117 http://www.certmagic.com
BH0-003
⇒ If there is a 'View' or 'Open' button beside the list box then double
clicking on a line in the List Box, should act in the same way as
selecting and item in the list box, then clicking the command button.
⇒ Force the scroll bar to appear, make sure all the data can be seen in
the box.
Section 2 - Screen Validation Checklist
Aesthetic Conditions:
1. Is the general screen background the correct colour?
2. Are the field prompts the correct colour?
3. Are the field backgrounds the correct colour?
4. In read-only mode, are the field prompts the correct colour?
5. In read-only mode, are the field backgrounds the correct colour?
6. Are all the screen prompts specified in the correct screen font?
7. Is the text in all fields specified in the correct screen font?
8. Are all the field prompts aligned perfectly on the screen?
9. Are all the field edit boxes aligned perfectly on the screen?
10.Are all groupboxes aligned correctly on the screen?
11.Should the screen be resizable?
12.Should the screen be minimisable?
13.Are all the field prompts spelt correctly?
14.Are all character or alpha-numeric fields left justified? This is the
default unless otherwise specified.
15.Are all numeric fields right justified? This is the default unless
otherwise specified.
16.Is all the microhelp text spelt correctly on this screen?
17.Is all the error message text spelt correctly on this screen?
18.Is all user input captured in UPPER case or lower case
consistently?
19.Where the database requires a value (other than null) then this
should be defaulted into fields. The user must either enter an
alternative valid value or leave the default value intact.
20.Assure that all windows have a consistent look and feel.
21.Assure that all dialog boxes have a consistent look and feel.
Validation Conditions:
1. Does a failure of validation on every field cause a sensible user
error message?
2. Is the user required to fix entries which have failed validation tests?
118 http://www.certmagic.com
BH0-003
3. Have any fields got multiple validation rules and if so are all rules
being applied?
4. If the user enters an invalid value and clicks on the OK button (i.e.
does not TAB off the field) is the invalid entry identified and
highlighted correctly with an error message.?
5. Is validation consistently applied at screen level unless specifically
required at field level?
6. For all numeric fields check whether negative numbers can and
should be able to be entered.
7. For all numeric fields check the minimum and maximum values and
also some mid-range values allowable?
8. For all character/alphanumeric fields check the field to ensure that
there is a character limit specified and that this limit is exactly
correct for the specified database size?
9. Do all mandatory fields require user input?
10.If any of the database columns don't allow null values then the
corresponding screen fields must be mandatory. (If any field which
initially was mandatory has become optional then check whether
null values are allowed in this field.)
Navigation Conditions:
1. Can the screen be accessed correctly from the menu?
2. Can the screen be accessed correctly from the toolbar?
3. Can the screen be accessed correctly by double clicking on a list
control on the previous screen?
4. Can all screens accessible via buttons on this screen be accessed
correctly?
5. Can all screens accessible by double clicking on a list control be
accessed correctly?
6. Is the screen modal. i.e. Is the user prevented from accessing other
functions when this screen is active and is this correct?
7. Can a number of instances of this screen be opened at the same time
and is this correct?
Usability Conditions:
1. Are all the dropdowns on this screen sorted correctly? Alphabetic
sorting is the default unless otherwise specified.
2. Is all date entry required in the correct format?
3. Have all pushbuttons on the screen been given appropriate Shortcut
keys?
4. Do the Shortcut keys work correctly?
119 http://www.certmagic.com
BH0-003
5. Have the menu options which apply to your screen got fast keys
associated and should they have?
6. Does the Tab Order specified on the screen go in sequence from Top
Left to bottom right? This is the default unless otherwise specified.
7. Are all read-only fields avoided in the TAB sequence?
8. Are all disabled fields avoided in the TAB sequence?
9. Can the cursor be placed in the microhelp text box by clicking on the
text box with the mouse?
10.Can the cursor be placed in read-only fields by clicking in the field with
the mouse?
11.Is the cursor positioned in the first input field or control when the
screen is opened?
12.Is there a default button specified on the screen?
13.Does the default button work correctly?
14.When an error message occurs does the focus return to the field in
error when the user cancels it?
15.When the user Alt+Tab's to another application does this have any
impact on the screen upon return to The application?
16.Do all the fields edit boxes indicate the number of characters they will
hold by there length? e.g. a 30 character field should be a lot longer
Data Integrity Conditions:
1. Is the data saved when the window is closed by double clicking on the
close box?
2. Check the maximum field lengths to ensure that there are no
truncated characters?
3. Where the database requires a value (other than null) then this should
be defaulted into fields. The user must either enter an alternative valid
value or leave the default value intact.
4. Check maximum and minimum field values for numeric fields?
5. If numeric fields accept negative values can these be stored correctly
on the database and does it make sense for the field to accept
negative numbers?
6. If a set of radio buttons represent a fixed set of values such as A, B
and C then what happens if a blank value is retrieved from the
database? (In some situations rows can be created on the database by
other functions which are not screen based and thus the required
initial values can be incorrect.)
7. If a particular set of data is saved to the database check that each
value gets saved fully to the database. i.e. Beware of truncation (of
strings) and rounding of numeric values.
Modes (Editable Read-only) Conditions:
120 http://www.certmagic.com
BH0-003
1. Are the screen and field colours adjusted correctly for read-only mode?
2. Should a read-only mode be provided for this screen?
3. Are all fields and controls disabled in read-only mode?
4. Can the screen be accessed from the previous screen/menu/toolbar in
read-only mode?
5. Can all screens available from this screen be accessed in read-only
mode?
6. Check that no validation is performed in read-only mode.
General Conditions:
1. Assure the existence of the "Help" menu.
2. Assure that the proper commands and options are in each menu.
3. Assure that all buttons on all tool bars have a corresponding key
commands.
4. Assure that each menu command has an alternative(hot-key) key
sequence which will invoke it where appropriate.
5. In drop down list boxes, ensure that the names are not abbreviations /
cut short
6. In drop down list boxes, assure that the list and each entry in the list
can be accessed via appropriate key / hot key combinations.
7. Ensure that duplicate hot keys do not exist on each screen
8. Ensure the proper usage of the escape key (which is to undo any
changes that have been made) and generates a caution message
"Changes will be lost - Continue yes/no"
9. Assure that the cancel button functions the same as the escape key.
10.Assure that the Cancel button operates as a Close button when
changes have be made that cannot be undone.
11.Assure that only command buttons which are used by a particular
window, or in a particular dialog box, are present. - i.e make sure they
don't work on the screen behind the current screen.
12.When a command button is used sometimes and not at other times,
assure that it is grayed out when it should not be used.
13.Assure that OK and Cancel buttons are grouped separately from other
command buttons.
14.Assure that command button names are not abbreviations.
15.Assure that all field labels/names are not technical labels, but rather
are names meaningful to system users.
16.Assure that command buttons are all of similar size and shape, and
same font & font size.
17.Assure that each command button can be accessed via a hot key
combination.
18.Assure that command buttons in the same window/dialog box do not
have duplicate hot keys.
19.Assure that each window/dialog box has a clearly marked default value
(command button, or other object) which is invoked when the Enter
key is pressed - and NOT the Cancel or Close button
121 http://www.certmagic.com
BH0-003
20.Assure that focus is set to an object/button which makes sense
according to the function of the window/dialog box.
21.Assure that all option buttons (and radio buttons) names are not
abbreviations.
22.Assure that option button names are not technical labels, but rather
are names meaningful to system users.
23.If hot keys are used to access option buttons, assure that duplicate
hot keys do not exist in the same window/dialog box.
24.Assure that option box names are not abbreviations.
25.Assure that option boxes, option buttons, and command buttons are
logically grouped together in clearly demarcated areas "Group Box"
26.Assure that the Tab key sequence which traverses the screens does so
in a logical way.
27.Assure consistency of mouse actions across windows.
28.Assure that the color red is not used to highlight active objects (many
individuals are red-green color blind).
29.Assure that the user will have control of the desktop with respect to
general color and highlighting (the application should not dictate the
desktop background characteristics).
30.Assure that the screen/window does not have a cluttered appearance
31.Ctrl + F6 opens next tab within tabbed window
32.Shift + Ctrl + F6 opens previous tab within tabbed window
33.Tabbing will open next tab within tabbed window if on last field of
current tab
34.Tabbing will go onto the 'Continue' button if on last field of last tab
within tabbed window
35.Tabbing will go onto the next editable field in the window
36.Banner style & size & display exact same as existing windows
37.If 8 or less options in a list box, display all options on open of list box -
should be no need to scroll
38.Errors on continue will cause user to be returned to the tab and the
focus should be on the field causing the error. (i.e the tab is opened,
highlighting the field with the error on it)
39.Pressing continue while on the first tab of a tabbed window (assuming
all fields filled correctly) will not open all the tabs.
40.On open of tab focus will be on first editable field
41.All fonts to be the same
42.Alt+F4 will close the tabbed window and return you to main screen or
previous screen (as appropriate), generating "changes will be lost"
message if necessary.
43.Microhelp text for every enabled field & button
44.Ensure all fields are disabled in read-only mode
45.Progress messages on load of tabbed screens
46.Return operates continue
47.If retrieve on load of tabbed window fails window should not open
122 http://www.certmagic.com
BH0-003
Specific Field Tests
Date Field Checks
• Assure that leap years are validated correctly & do not cause
errors/miscalculations
• Assure that month code 00 and 13 are validated correctly & do not
cause errors/miscalculations
• Assure that 00 and 13 are reported as errors
• Assure that day values 00 and 32 are validated correctly & do not
cause errors/miscalculations
• Assure that Feb. 28, 29, 30 are validated correctly & do not cause
errors/ miscalculations
• Assure that Feb. 30 is reported as an error
• Assure that century change is validated correctly & does not cause
errors/ miscalculations
• Assure that out of cycle dates are validated correctly & do not cause
errors/miscalculations
Numeric Fields
• Assure that lowest and highest values are handled correctly
• Assure that invalid values are logged and reported
• Assure that valid values are handles by the correct procedure
• Assure that numeric fields with a blank in position 1 are processed or
reported as an error
• Assure that fields with a blank in the last position are processed or
reported as an error an error
• Assure that both + and - values are correctly processed
• Assure that division by zero does not occur
• Include value zero in all calculations
• Include at least one in-range value
• Include maximum and minimum range values
• Include out of range values above the maximum and below the
minimum
• Assure that upper and lower values in ranges are handled correctly
Alpha Field Checks
• Use blank and non-blank data
• Include lowest and highest values
• Include invalid characters & symbols
123 http://www.certmagic.com
BH0-003
• Include valid characters
• Include data items with first position blank
• Include data items with last position blank
Validation Testing - Standard Actions
Examples of Standard Actions - Substitute your specific
commands
Add
View
Change
Delete
Continue - (i.e. continue saving changes or additions)
Add
View
Change
Delete
Cancel - (i.e. abandon changes or additions)
Fill each field - Valid data
Fill each field - Invalid data
Different Check Box / Radio Box combinations
Scroll Lists / Drop Down List Boxes
Help
Fill Lists and Scroll
Tab
Tab Sequence
Shift Tab
Shortcut keys / Hot Keys
Note: The following keys are used in some windows applications, and
are included as a guide.
Key No Modifier Shift CTRL ALT
F1 Help Enter Help na na
Mode
124 http://www.certmagic.com
BH0-003
F2 na na na na
F3 na na na na
F4 na na Close Close
Document / Application.
Child window.
F5 na na na na
F6 na na na na
F7 na na na na
F8 Toggle extend Toggle Add na na
mode, if mode, if
supported. supported.
F9 na na na na
F10 Toggle menu na na na
bar activation.
F11, F12 na na na na
Tab Move to next Move to Move to next Switch to
active/editable previous open previously
field. active/editable Document or used
field. Child window. application.
(Adding SHIFT (Holding down
reverses the the ALT key
order of displays all
movement). open
applications).
Alt Puts focus on na na na
first menu
command
(e.g. 'File').
Control Shortcut Keys
Key Function
CTRL + Z Undo
125 http://www.certmagic.com
BH0-003
CTRL + X Cut
CTRL + C Copy
CTRL + V Paste
CTRL + N New
CTRL + O Open
CTRL + P Print
CTRL + S Save
CTRL + B Bold
CTRL + I Italic
CTRL + U Underline
These shortcuts are suggested for text formatting applications, in the
context for
which they make sense. Applications may use other modifiers for these
operations.
Important Considerations for Test Automation
Often when a test automation tool is introduced to a project, the
expectations for the return on investment are very high. Project
members anticipate that the tool will immediately narrow down the
testing scope, meaning reducing cost and schedule. However, I have
seen several test automation projects fail - miserably.
The following very simple factors largely influence the effectiveness of
automated testing, and if not taken into account, the results is usually
a lot of lost effort, and very expensive ‘shelfware’.
1. Scope - It is not practical to try to automate everything, nor is
there the time available generally. Pick very carefully the
functions/areas of the application that are to be automated.
2. Preparation Timeframe - The preparation time for automated
test scripts has to be taken into account. In general, the
preparation time for automated scripts can be up to 2/3 times
longer than for manual testing. In reality, chances are that
initially the tool will actually increase the testing scope. It is
therefore very important to manage expectations. An automated
126 http://www.certmagic.com
BH0-003
testing tool does not replace manual testing, nor does it replace
the test engineer. Initially, the test effort will increase, but when
automation is done correctly it will decrease on subsequent
releases.
3. Return on Investment - Because the preparation time for test
automation is so long, I have heard it stated that the benefit of
the test automation only begins to occur after approximately the
third time the tests have been run.
4. When is the benefit to be gained? Choose your objectives
wisely, and seriously think about when & where the benefit is
to be gained. If your application is significantly changing
regularly, forget about test automation - you will spend so much
time updating your scripts that you will not reap many benefits.
[However, if only disparate sections of the application are
changing, or the changes are minor - or if there is a specific
section that is not changing, you may still be able to
successfully utilise automated tests]. Bear in mind that you may
only ever be able to do a complete automated test run when
your application is almost ready for release – i.e. nearly fully
tested!! If your application is very buggy, then the likelihood is
that you will not be able to run a complete suite of automated
tests – due to the failing functions encountered.
5. The Degree of Change – The best use of test automation is for
regression testing, whereby you use automated tests to ensure
that pre-existing functions (e.g. functions from version 1.0 - i.e.
not new functions in this release) are unaffected by any changes
introduced in version 1.1. And, since proper test automation
planning requires that the test scripts are designed so that they
are not totally invalidated by a simple gui change (such as
renaming or moving a particular control), you need to take into
account the time and effort required to update the scripts. For
example, if your application is significantly changing, the scripts
from version 1.0. may need to be completely re-written for
version 1.1., and the effort involved may be at most prohibitive,
at least not taken into account! However, if only disparate
sections of the application are changing, or the changes are
minor, you should be able to successfully utilise automated tests
to regress these areas.
6. Test Integrity - how do you know (measure) whether a test
passed or failed ? Just because the tool returns a ‘pass’ does
not necessarily mean that the test itself passed. For example,
just because no error message appears does not mean that the
next step in the script successfully completed. This needs to be
taken into account when specifying test script fail/pass criteria.
127 http://www.certmagic.com
BH0-003
7. Test Independence - Test independence must be built in so
that a failure in the first test case won't cause a domino effect
and either prevent, or cause to fail, the rest of the test scripts in
that test suite. However, in practice this is very difficult to
achieve.
8. Debugging or "testing" of the actual test scripts
themselves - time must be allowed for this, and to prove the
integrity of the tests themselves.
9. Record & Playback - DO NOT RELY on record & playback as
the SOLE means to generates a script. The idea is great. You
execute the test manually while the test tool sits in the
background and remembers what you do. It then generates a
script that you can run to re-execute the test. It's a great idea -
that rarely works (and proves very little).
10.Maintenance of Scripts - Finally, there is a high maintenance
overhead for automated test scripts - they have to be
continuously kept up to date, otherwise you will end up
abandoning hundreds of hours work because there has been too
many changes to an application to make modifying the test
script worthwhile. As a result, it is important that the
documentation of the test scripts is kept up to date also.
Overview of System X
To aim of this phase of the project is to implement a new X System
platform that will enable:
• Removal of legacy office systems
• Introduction of ABC
• Processing of Special Transactions
• No constraint on location of capture
• Enable capture of transactions for other processing systems
• New Reconciliation Process
• Positioning for European ECU Currency and future initiatives
This programme will result in significant changes to the current
departmental and inter-office processes. The functionality will be
delivered on a phased basis.
Phase 1 will incorporate the following facilities :
128 http://www.certmagic.com
BH0-003
• Replacement of the legacy System A
• New Reconciliation System
• Outsourcing system for departments in different european
countries.
• New/Revised Audit Trail & Query Facilities
[Detailed inclusions are listed later in this document]
Preparation for this test consists of three major stages:-
• The Test Approach sets the scope of system testing, the overall
strategy to be adopted, the activities to be completed, the general
resources required and the methods and processes to be used to
test the release. It also details the activities, dependencies and
effort required to conduct the System Test.
• Test Planning details the activities, dependencies and effort
required to conduct the System Test.
• Test Conditions/Cases documents the tests to be applied, the
data to be processed, the automated testing coverage and the
expected results.
Formal Reviewing
There will be several formal review points before and during system
test. This is a vital element in achieving a quality product.
Formal Review Points
1. Design Documentation
2. Testing Approach
3. Unit Test Plans
4. Unit Test Conditions & Results
5. System Test Conditions
6. System Test Progress
7. Post System Test Review
Objectives of System Test
At a high level, this System Test intends to prove that :-
129 http://www.certmagic.com
BH0-003
• The functionality, delivered by the development team, is as
specified by the business in the Business Design Specification
Document and the Requirements Documentation.
• The software is of high quality; the software will replace/support
the intended business functions and achieves the standards
required by the company for the development of new systems.
• The software delivered interfaces correctly with existing
systems, including Windows 98.
[Detailed objectives are listed later in this document.]
Software Quality Assurance involvement
The above V Model shows the optimum testing process, where test
preparation commences as soon as the Requirements Catalogue is
produced. System Test planning commenced at an early stage, and for
this reason, the System test will benefit from Quality initiatives
throughout the project lifecycle.
The responsibility for testing between the Project & Software Qualtiy
Assurance (S.Q.A.) is as follows:
• Unit Test is the responsibility of the Development Team
• System Testing is the responsibility of SQA
• User Acceptance Testing is the Responsibility of the User
Representatives Team
130 http://www.certmagic.com
BH0-003
• Technology Compliance Testing is the responsibility of the
Systems Installation & Support Group.
Scope of Test Approach - System Functions
The contents of this release are as follows :-
Phase 1 Deliverables
o New & revised Transaction Processing with automated support
o New Customer Query Processes and systems
o Revised Inter-Office Audit process
o Relocate Exceptions to Head Office
o New centralised Agency Management system
o Revised Query Management process
o Revised Retrievals process
o New International Reconciliation process
o New Account Reconciliation process
EXCLUSIONS
When the scope of each Phase has been agreed and signed off, no
further inclusions will be considered for inclusion in this release,
except:
(1) Where there is the express permission and agreement of the
Business Analyst and the System Test Controller;
(2) Where the changes/inclusions will not require significant effort on
behalf of the test team (i.e. requiring extra preparation - new test
conditions etc.) and will not adversely affect the test schedule.
SPECIFIC EXCLUSIONS
• Cash management is not included in this phase
• Sign On/Sign Off functions are excluded - this will be addressed by
existing processes
• The existing Special Order facility will not be replaced
• Foreign Currency Transactions
• International Data Exchanges
• Accounting or reporting of Euro transactions
Reference & Source Documentation:
131 http://www.certmagic.com
BH0-003
1. Business Processes Design Document - Document Ref: BPD-1011
2. Transaction Requirements for Phase 1 - Document Ref: TR_PHASE1-
4032
3. Project Issues & Risks Database - T:DataProjectPROJECT.MDB
4. The System Development Standards - Document Ref: DEVSTD-1098-2
5. System Development Lifecycle - Document Ref: SDLC-301
Testing Process
The diagram above outlines the Test Process approach that will be followed.
a. Organise Project involves creating a System Test Plan, Schedule & Test Approach, and
requesting/assigning resources.
b. Design/Build System Test involves identifying Test Cycles, Test Cases, Entrance &
Exit Criteria, Expected Results, etc. In general, test conditions/expected results will be
identified by the Test Team in conjunction with the Project Business Analyst or Business
Expert. The Test Team will then identify Test Cases and the Data required. The Test
conditions are derived from the Business Design and the Transaction Requirements
Documents
c. Design/Build Test Procedures includes setting up procedures such as Error
Management systems and Status reporting, and setting up the data tables for the
Automated Testing Tool.
d. Build Test Environment includes requesting/building hardware, software and data set-
ups.
e. Execute Project Integration Test - See Section 3 - Test Phases & Cycles
f. Execute Operations Acceptance Test - See Section 3 - Test Phases & Cycles
g. Signoff - Signoff happens when all pre-defined exit criteria have been achieved. See
Section 2.4.
132 http://www.certmagic.com
BH0-003
Exclusions
SQA will not deal directly with the business design regarding any
design / functional issues / queries.
The development team is the supplier to SQA - if design / functional
issues arise they should be resolved by the development team and its
suppliers.
Testing Scope
Outlined below are the main test types that will be performed for this
release. All system test plans and conditions will be developed from
the functional specification and the requirements catalogue.
Functional Testing
The objective of this test is to ensure that each element of the
application meets the functional requirements of the business as
outlined in the :
• Requirements Catalogue
• Business Design Specification
• Year 2000 Development Standards
• Other functional documents produced during the course of the
project i.e. resolution to issues/change requests/feedback.
This stage will also include Validation Testing - which is intensive
testing of the new Front end fields and screens. Windows GUI
Standards; valid, invalid and limit data input; screen & field look and
appearance, and overall consistency with the rest of the application.
The third stage includes Specific Functional testing - these are low-
level tests which aim to test the individual processes and data flows.
Integration Testing
This test proves that all areas of the system interface with each other
correctly and that there are no gaps in the data flow. Final Integration
133 http://www.certmagic.com
BH0-003
Test proves that system works as integrated unit when all the fixes are
complete.
Business (User) Acceptance Test
This test, which is planned and executed by the Business
Representative(s), ensures that the system operates in the manner
expected, and any supporting material such as procedures, forms etc.
are accurate and suitable for the purpose intended. It is high level
testing, ensuring that there are no gaps in functionality.
Performance Testing
These tests ensure that the system provides acceptable response
times (which should not exceed 4 seconds).
Regression Testing
A Regression test will be performed after the release of each Phase to
ensure that -
• There is no impact on previously released software, and
• to ensure that there is an increase in the functionality and
stability of the software.
The regression testing will be automated using the automated testing
tool.
Bash & Multi-User Testing
Multi-user testing will attempt to prove that it is possible for an
acceptable number of users to work with the system at the same time.
The object of Bash testing is an ad-hoc attempt to break the system.
Technical Testing
Technical Testing will be the responsibility of the Development Team.
Operations Acceptance Testing (OAT)
This phase of testing is to be performed by the Systems Installation
and Support group, prior to implementing the system in a live site.
The SIS team will define their own testing criteria, and carry out the
tests.
134 http://www.certmagic.com
BH0-003
System Test Entrance/Exit Criteria
Entrance Criteria
The Entrance Criteria specified by the system test controller, should be
fulfilled before System Test can commence. In the event, that any
criterion has not been achieved, the System Test may commence if
Business Team and Test Controller are in full agreement that the risk
is manageable.
• All developed code must be unit tested. Unit and Link Testing
must be completed and signed off by development team.
• System Test plans must be signed off by Business Analyst and
Test Controller.
• All human resources must be assigned and in place.
• All test hardware and environments must be in place, and free
for System test use.
• The Acceptance Tests must be completed, with a pass rate of
not less than 80%.
Acceptance Tests:
25 test cases will be performed for the acceptance tests. To achieve
the acceptance criteria 20 of the 25 cases should be completed
successfully - i.e. a pass rate of 80% must be achieved before the
software will be accepted for System Test proper to start. This means
that any errors found during acceptance testing should not prevent the
completion of 80% of the acceptance test applications.
Note: These tests are not intended to perform in depth testing of the
software.
Resumption Criteria
In the event that system testing is suspended resumption criteria will
be specified and testing will not re-commence until the software
reaches these criteria.
Exit Criteria
The Exit Criteria detailed below must be achieved before the Phase 1
software can be recommended for promotion to Operations Acceptance
status. Furthermore, I recommend that there be a minimum 2 days
effort Final Integration testing AFTER the final fix/change has been
retested.
• All High Priority errors from System Test must be fixed and
tested
135 http://www.certmagic.com
BH0-003
• If any medium or low-priority errors are outstanding - the
implementation risk must be signed off as acceptable by
Business Analyst and Business Expert
• Project Integration Test must be signed off by Test Controller
and Business Analyst.
• Business Acceptance Test must be signed off by Business
Expert.
TEST PHASES AND CYCLES
There will be two main stages of testing for the new application during
System Test :-
• System Testing
• Operations Acceptance Testing
System Testing Cycles
The main thrust of the approach is to intensively test the front end in
the first two releases, thus raising approximately 80% of errors in this
period. With the majority of these errors fixed, standard and/or
frequently used actions will be tested to prove individual elements and
total system processing in Release v0.3. Regression testing of
outstanding errors will be performed on an ongoing basis.
When all errors (which potentially impact overall processing) are fixed,
an additional set of test cases are processed in Release v0.4 to ensure
the system works in an integrated manner. It is intended that Release
v0.4 be the final proving of the system as a single application. There
should be no A or B class errors outstanding prior to the start of
Release v0.4 testing.
Test Cases by Release version:
Testing by Phase
Acceptance 1
Release v0.1 Functional 1
User Acceptance
Acceptance 2
Release v0.2 Functional 2
Regression 1
Acceptance 3
Functional 3
Release v0.3 Performance 1
136 http://www.certmagic.com
BH0-003
Bash & Multi-User Testing
Regression 1
Regression 2
Integration 1
Technical 1
Release v0.4 Regression 1
Regression 2
Regression 3
Installation Test
Contingency Per Bug Fix Test Only
Automated Testing
Automated testing tools will be used in the test environment for
functional and regression testing. The main focus of the automated
testing will be the regression testing of the previously delivered
functionality - i.e. when development version 0.2 of the software is
delivered the majority of the regression testing of the functionality
delivered in development version 0.1 will be automated. It is
estimated that the full benefit of the automated testing will only occur
when the tests have been executed three or more times.
Software Delivery
During System Test the release of new versions of the software will be
co-ordinated between the Development Team leader and the System
Test Controller. However, unless it concerns a fix to a very serious
error, new versions should only be released when a agreed targets
have been reached (i.e. the next version contains fixes to X or more
numbers of bugs).
Release Schedule:
Functionality
to v0.1 v0.2 v0.3 v0.4 v1.0
be Delivered 1st May 17th May 31st May 18th June 29th June
1. Function A
2. Process B No New Bug Fix
3. Euro Reqs' Functionality contingency
4. Y2K Reqs. to be release
5. Inter Office delivered only.
137 http://www.certmagic.com
BH0-003
Trans
6. International
Trans. in this
7. Other. release.
(per functional spec, by priority)
Notes:
It is intended that 80% of the functionality will have been tested in full
prior to the Phase 3 Release.
All the functionality must be present in the Phase 3 Release.
No previously undelivered functionality will be accepted for testing
after Phase 3.
Formal Reviewing
There will be several formal review points before and during system test,
including the review of this document. This is a vital element in achieving a
quality product.
Formal Review Points
1. Design Documentation - Requirements Specification & Functional
Specification
2. System Test Plan
3. Unit Test Plans & Test Conditions
4. Unit Test Results
138 http://www.certmagic.com
BH0-003
5. System Test Conditions
6. System Test Progress/Results
7. Post System Test Review
8. Integration Test Results
9. Pilot Implementation Review
10. Project Review
The diagram above outlines the Test Approach. Boxes 1 - 6 show the major
review stages prior to Test execution. Boxes 7 - 10 show the phases planned
for & after test execution.
While the above diagram concentrates on the testing aspect of SQA's role,
there is an ongoing role also, in ensuring the quality of the major
deliverables throughout the lifecycle of the project. SQA's role will be to
ensure that all Quality Inspections occur for all the agreed deliverables and
that follow up actions and initiatives are pursued.
Progress/Results Monitoring
• Acceptance Test 1 Results
• Test Results - Release v0.1
• Test Results - Release v0.2
• Test Results - Release v0.3
• Performance Test 1 Results
• Regression 1 & 2 Results
• Test Results - Release v0.4
• Technical Test Results
Test Schedule
System Test Schedule
These are screenshots of several high level views of the project schedule.
These schedules are intended as examples only and probably will not
correspond exactly with the rest of the test plan.
Click on the small image to view the main schedule.
139 http://www.certmagic.com
BH0-003
RESOURCES
Human
Resource Type Resource Title No. Date Who Status
Req'd
Project Business Analyst 1 - A.N. Assigned
Mgmt/Functional Other
Testing Test Controller 1 - A. Assigned
Smith
Testers 4 1st May To Be
Assigned
Test Support Team Support 4 15th May To be
Programmers Assigned
140 http://www.certmagic.com
BH0-003
Technical Support 1 1st May To be
Assigned
WAN Support 1 25th May To be
Assigned
Technical - External CIS Support 1 25th May To be
Assigned
Bookkeeping 1 15th May To be
Support Assigned
External Liaison 1 25th May C. Assigned
Support Jones
Business Business Expert/ 1 1st May To be
Business Assigned
Representative
Hardware
One seperate, controlled system will be required for the initial phase of
testing, setup as per one standard, complete office environment. In order to
maintain the integrity of the test environment his network will not be
accessible to anybody outside this project. The printers are also exclusively
for use by the test network.
141 http://www.certmagic.com
BH0-003
Hardware components required
• 1 Network Controller
• 6 Networked PC's (See below)
• 1 DAP Workstation
• 1 Motorola 6520
• 1 Alpha AXP Server
• 1 Batch Waste Printer
• 1 HP LaserJet 4v Printer
PC Specifications
The 6 PC's required for the test environment will include the
following:
1 x P100, 1Gb HD, 16Mb RAM [Current Minimum Specification]
3 x P166, 1.5Gb HD, 32Mb RAM [Current Standard Specification]
1 x P333, 2.5Gb HD, 64Mb RAM [Current Maximum Specification]
These specifications are the various specifications currently in use in different
branches.
1 x Pentium running Windows NT is also required as the Test center for
controlling and executing the automated testing.
142 http://www.certmagic.com
BH0-003
Software
Test IMS environments
Test IMS region X will be required for System Testing. Additional or amended
data will be populated where required.
Test Environment Software
System Test will be run on the following Software Versions :-
Custom Destop Vers.97.0.1
Windows 95 Operating System
Visual Basic 5 Runtime Files
MS Office 97
Novell Netware
Error Measurement System
This system test will use a bespoke MS Access database Error Management
system.
A new database will be implemented for the sole use of this project.
ROLES AND RESPONSIBILITIES
Management Team
Project Leader - AAAA
Ensure Phase 1 is delivered to schedule, budget & quality
Ensure Exit Criteria are achieved prior to System Test Signoff
Regularly review Testing progress with Test Controller.
Liaise with external Groups e.g. New Systems
Raise and manage issues/risks relating to project or outside Test
Teams control.
Review & sign off Test approach, plans and schedule.
SQA Project Leader - BBBB
Ensure Phase 1 is delivered to schedule, budget & quality
Regularly review Testing progress
Manage issues/risks relating to System Test Team
Provide resources necessary for completing system test.
Testing Team
Test Planner / Controller - DDDD
Ensure Phase 1 is delivered to schedule, budget & quality
143 http://www.certmagic.com
BH0-003
Produce High Level and Detailed Test Conditions
Produce Expected Results
Report progress at regular status reporting meetings
Co-ordinate review & signoff of Test Conditions
Manage individual test cycles & resolve tester queries/problems.
Ensure test systems outages/problems are reported immediately and
followed up.
Ensure Entrance criteria are achieved prior to System Test start.
Ensure Exit criteria are achieved prior to System Test signoff.
Testers
Identify Test Data
Execute Test Conditions and Markoff results
Raise Software Error Reports
Administer Error Measurement System
Business Team
Business Analyst - EEEE
Review high level / detailed test plans for System Test
Define Procedures
Resolve design issues
Resolve Business issues
Take part in daily test Error Review Team meetings
Business Representative - FFFF
Execute User Acceptance Testing
Define Test Conditions/Expected Results for Business Acceptance Test
Resolve user issues
Resolve Design issues
Testing Support Team
Support Programmers
Take part in daily Error Review Team meetings
Co-ordinate/provide support for system test.
Resolve errors
Re-release test software after amendments
Support Systems Testers
External Support Team
CIS Support
Provide CIS support, if required.
Resolve CIS queries, if required.
IMS Support
Provide System Test Support
144 http://www.certmagic.com
BH0-003
Support IMS Regions
Resolve Spooling Issues (if necessary)
Bookkeeping Integration & Compliance (if necessary)
Resolve queries arising from remote backup
Bookkeeping Support
Provide Bookkeeping Technical support, if required.
Resolve queries, if required.
Technical Support
Provide support for hardware environment
Provide support for Test software
Promote Software to system test environment
Access Support
Provide and support Test Databases
Error Management & Configuration Management
During System Test, errors will be recorded as they are detected on Error
Report forms. These forms will be input on the Error Management System
each evening with status "Error Raised" or "Query Raised". The Error
Review Team will meet each morning (10am, Conference Room) to review
and prioritise DN's raised the previous day, and assign them or drop them
as appropriate. This team will consist of the following representatives:-
• A. Boring - Development Team Leader
• B. Curie - Business Analyst
• C. Durine - Test Controller
• D. Ewards - Business Representative
Errors, which are agreed as valid, will be categorised as follows by the
Error Review Team :-
• Category A - Serious errors that prevent System test of a
particular function continuing or serious data type error
• Category B - Serious or missing data related errors that will not
prevent implementation.
• Category C - Minor errors that do not prevent or hinder
functionality.
Category A errors should be turned around by Bug Fix Team in 48 hours
(this is turn around from time raised at Error Review Team meeting to
time fix is released to System Test environment). In the event of an A
error that prevents System Test continuing, the turnaround should be
within 4 hours.
145 http://www.certmagic.com
BH0-003
Category B errors should be turned around in 1 day; while
Category C errors should be turned around in 3 days.
However, the release of newer versions of the software will be co-
ordinated with the Test Controller - new versions should only be released
when agreed, and where there is a definite benefit (i.e. contains fixes to X
or more numbers of bugs).
STATUS REPORTING
Status Reporting
• Test preparation and Testing progress will be formally reported
during a weekly Status Meeting.
A status report will be prepared by the Test Controller to facilitate this
meeting. This report will contain the following information :-
1. Current Status v. Plan (Ahead/Behind/On Schedule)
2. Progress of tasks planned for previous week
3. Tasks planned for next week including tasks carried from previous
week
4. Error Statistics from Error Measurement system
5. Issues/Risks
6. AOB.
Issues, Risks and Assumptions
1. No further changes or inclusions will be considered for inclusion in
this release except (1) where there is the express permission and
agreement of the Business Analyst and the System Test Controller; (2)
Where the changes/inclusions will not require significant effort on
behalf of the test team and will not adversely affect the test schedule.
This is a potentially serious issue, as any major changes to design will
entail additional time to re-plan testing and to create or amend test
conditions
.
2. The design of the software must be final, and design documentation
must be complete, informative and signed off by all parties prior to
System Test proper commences.
3. A weakness in the 'phased delivery' approach is that the the high
degree of interdependency in the code means that the smallest
changes can have serious effects to areas of the application which
146 http://www.certmagic.com
BH0-003
apparently have not been changed. The assumption of the test team is
that previously delivered and tested functionality will only require
regression testing to verify that it 'still' works. I.e. testing will not be
intended to discover new errors. Because of this I recommend that
there be a minimum 2 days regression testing AFTER the final
fix/change has been retested. This however, imposes a fixed time
constraint on the completion of system testing which requires the
agreement of the Project Leader.
4. Automated Testing
The majority of the Regression testing will be performed using the
automated test tool. However, due to the workload required to
implement (and debug) the test tool fully it is likely that the return will
only be maximised after the 3rd time running the regression test suite
for each release. The other major uses of the test tool are for (1) Load
Testing, (2) Multi-User Testing, and (3) Repetitive data entry.
Resp : Test Controller
Assumptions
• Software will be delivered on time.
• Software is of the required quality.
• The software will not be impacted by impending Y2K compliance
changes to the external software infrastructure - i.e. any external
software changes will have to be compatible with this application.
• All "Show-Stopper" bugs receive immediate attention from the
development team.
• All bugs found in a version of the software will be fixed and unit
tested by the development team before the next version is
released.
• Functionality is delivered to schedule.
• Required resources available.
• All service agreements will be met.
• The automated test tool will function & interface correctly with the
software.
• All documentation will be up to date and delivered to the system
test team.
• Functional and technical specifications will be signed off by the
business.
• All service agreements will be met.
• The Intranet will be fully functional prior to project
commencement.
Formal Signoff
147 http://www.certmagic.com
BH0-003
This document must be formally approved before System Test can
commence. The following people will be required to sign off :-
Group
Signatures:
Project Manager
SQA
Testing Team
Development Team
Purpose of Error Review Team.
Ensure maximum efficiency of the development and system testing
teams for the release of the new office software through close co-
operation of all involved parties.
This will be achieved through daily meetings whose function will be to
• Agree status of each raised Error
• Prioritisation of valid Error's
• Ensure that enough documentation is available with Error's.
• Agree content and timescale for software releases into System
test.
• Ensure one agreed source of Error reporting information.
• Identify any issues which may affect the performance of system
testing.
Error Review Team Meeting Agenda.
• Review any actions from last meeting.
Classify and prioritise each Error.
• Review Error's raised for Duplicates etc.
• Agree priority of each Error
• Determine adequacy of documentation associated with raised
Error's.
• Agree release content and timescale.
• Review of assigned actions from meeting.
• AOB
148 http://www.certmagic.com
BH0-003
Classification of Bugs
1. An "A" bug is a either a showstopper or of such importance as to
radically affect the functionality of the system i.e. :
- Examples of showstoppers
- If, because of a consistent crash during processing of a
particular type of application, a user could not complete that type of
application.
- Incorrect data is passed to legacy system resulting in
corruption or system crashes
- Example of severally affected functionality
- Calculation of repayment term/amount are incorrect
- Incorrect credit agreements produced
2. Bugs would be classified as "B" where :
- a less important element of functionality is affected
- Example : a value is not defaulting correctly and it is
necessary to input the correct value
- data is affected which does not have a major impact
- Example : where, for instance, some element of client
capture was not propagated to the database
- there is an alternative method of completing a particular process
- Example : a problem might occur reading all the details of a
credit - this change can be manually input.
3. "C" type bugs are mainly cosmetic bugs i.e. :
- incorrect / no helptext on screens
- drop down lists repeat an option
Procedure for maintenance of Error Management system.
1. The Test Controller will refer any major error/anomaly to either
Devopment Team Leader or designated representative on the
development team before raising a formal error record. This has several
advantages :-
- it prevents the testers trying to proceed beyond 'showstoppers'
- it puts the developer on immediate notice of the problem
- it allows the developer to put on any traces that might be necessary to
track down the error.
2. All bugs raised will be on the correct Error form, and contain all relevant
data.
149 http://www.certmagic.com
BH0-003
3. These errors will be logged on the day they occur with a status of
'RAISED'
4. There will be a daily 'System Test Support Group' meeting to discuss,
prioritise and agree all logged errors.
During this meeting some errors may be dropped, identified as
duplicates, passed to programmer, etc.
5. The Error Log will be updated with the status of all errors after this
meeting. e.g. with pgmr, dropped, duplicate.
6. Once errors have been fixed and 'rebundelled' for a release the paper
forms must be passed to the Test Controller and he will change their
status to 'Fixed to be retested'
7. Once the error has been retested and proven to be corrected the status
will be changed to 'Closed'
8. Regular status reports will be produced from the Error system, for use in
the Error Review Team meetings.
Overnight Processing - Checking Accounting & Audit & CIS
Test Requirement Check Items Level of
Testing
Accounting
When spooling complete the Summary report 1. Legacy Txs on 1. Checking
should be checked against : Report at
1. Similar Legacy Transactions V field level
2. Test Input forms Office
Transactions on 2. Checking
Report at
field level
2. Summary
report
V
Applic input
forms
Accounting : after open/amend the 1. Amendment 1. Satisfy
amendment report should be checked: report as to
1. For rejected open/amend instructions reasons for
2. Detail should correspond to input Applic 2. Amendment rejection
Forms report
V 2. Checking
Test input forms at
field level
Print off Account and Customer records and Bookkeeping - Checking
check field detail against applic input Input tx's at
forms/branch summary report V field level
150 http://www.certmagic.com
BH0-003
Test Input
forms/Amend rpt
SOFTWARE QUALITY ASSURANCE MEASURES
(i) DATES.
- Start date of SQA involvement.
(ii) EFFORT.
- No. of SQA Man Days Test Planning
- No. of SQA Man Days Reviewing Test Plans
- No. of SQA Man Days Executing Tests
(iii) VOLUME.
- No. of Tests Identified
(iv) QUALITY.
- No. of Tests Passed First Time
- Percentage of Tests Passed First Time
- No. of Error's Raised During Regression Testing
- No. of Error's Generated as a Result of Incorrect Fixes
- No. of Error's Raised by Category (A/B/C)
- No. of Error's Raised by Reason Code
- No. of Error's Raised by High Level Business Function
(v) TURNAROUND.
- Average Error Turnaround Time
Test Project Setup Checklist
Item Status Who
1. Project Initiation
1.1. Prepare System Test Estimates
1.2. Define System Test Approach
1.3. Define Testing Scope
1.4. Prepare DRAFT System Test Plan
1.5. Review System Test Plan
151 http://www.certmagic.com
BH0-003
1.6. Prepare Test Schedule
1.7. Request Test Resources
1.8. Request Business Expert
1.9. Request Management Support
1.10. Request Environment/Technical Support
1.11. Request Test Hardware (pc’s & servers)
1.12. Request Facilities (desks, chairs etc.)
1.13. Setup Test Project Folder
1.14. Revise Test Estimates
1.15. Define Entrance/Acceptance Criteria
1.16. Agree Communication Channels
1.17. Agree Reporting Procedures, Method &
Frequency
1.18. Define Exit Criteria
1.19. Design Release Notes Template
2. Test Preparation
2.1. Agree Builds/Drops Schedule & Contents
2.2. Agree Release Notes Contents & Format
2.3. Agree Error Management Procedures
2.4. Define & Agree Error Management Roles
2.5. Define System Test Roles & Responsibilities
2.6. Assign Test Roles & Responsibilities
2.7. Assign Test Case Preparation Primary
Responsibilities
2.8. Assign Test Case Preparation Secondary
Responsibilities
2.9. Prepare High Level Test Cases
2.9. Prepare Detailed Low Level Test Cases
2.9. Define Test Environment Setup
(Network/Server)
2.10. Define Test PC Setups & Configurations
(clients)
2.11. Review Test Plan
2.12. Review Test Schedule
2.13. Setup Test Execution Progress Tracking
Database
3. Build System Test Environment
3.1. Setup Test Environment (server)
3.2. Setup Test PC’s
3.3. Setup Bug Database
3.4. Verify Test Environment (shakedown)
3.5. Verify Bug Database External Access
3.6. Setup Test Data
152 http://www.certmagic.com
BH0-003
3.7. Setup & Install Test Peripherals (card readers,
receipt printers)
4. Prepare System Test
4.1. Review System Test Cases
4.2. Revise System Test Cases
4.3. System Test Readiness Review
4.4. Verify Entrance Criteria Reached
4.5. Receive B36 Build 1
4.6. Install B36 Build 1
4.7. Execute Acceptance Tests
4.8. Review Acceptance Test Results (accept –
yes/no)
5. Execute System Test
5.1. Execute System Test
5.2. Execute Cycle 1 - GUI Tests
5.3. Execute Cycle 2 – Functional Tests
5.4. Execute Cycle 3 – Scenario Tests
5.5. Log & Track Defects
5.6. Maintain & Administer Error Management
System
5.7. Progress Measurement & Reporting
5.8. Measure Progress – Actual vs Planned
5.9. Manage & Track new builds
5.10. Perform Build Regression Tests
5.11. Regression Test Fixed Bugs
5.12. Close Regressed Bugs / Re-open “Not Fixed”
bugs
5.13. Measure Error statistics & Metrics
5.14. Report Error Statistics & Metrics
5.15. Track & Record Error Turnaround Time
5.16. Escalate Issues as appropriate
5.17. Perform Final Regression Test
6. Signoff
6.1. Signoff System Test
6.2. Produce Post-Testing Report
6.3. Washup & Lessons Learnt Meeting
6.4. Review Exit Report
6.5. Cleanup Test Environment
6.6. Return Peripherals
6.7. Post Execution Test Case Review
6.8. Handover Test Documentation
153 http://www.certmagic.com
BH0-003
Software Quality Assurance & Usability Testing
The role of User Testing in Software Quality Assurance.
What is 'Usability Testing'
'Usability Testing' is defined as: "In System Testing, testing which attempts
to find any human-factor problems". A better description is "testing the
software from a users point of view". Essentially it means testing software to
prove/ensure that it is 'user-friendly', as distinct from testing the
functionality of the software. In practical terms it includes ergonomic
considerations, screen design, standardisation etc.
Why Usability Testing should be included as an element of the testing
cycle.
I believe that QA have a certain responsibility for usability testing. There are
several factors involved, but the main reason is the 'perspective differences'
or different viewpoints of the various teams involved in the development of
the software.
To demonstrate, assume a new application is developed, that is exactly,
100%, in accordance with the design specifications - yet, unfortunately, it is
not fit for use - because it may be so difficult/awkward to use, or it ends up
so complicated that the users don't want it or won't use it. Yet, it is what the
design specified. This has happened, and will happen again.
I remember a diagram that vividly showed this - it showed the design of a
swing, with sections on "what the customer ordered", "What the
development team built", "What the engineers installed" etc., with the effect
of illustrating the different perspectives of the various people involved.
This is especially true where the business processes that drive the design of
the new application are very complex (for example bespoke financial
applications).
Secondly, when a totally new or custom application is being developed, how
many of the coders themselves (1) have actual first hand experience of the
business processes/rules that form the basis of the application being
developed; and/or (2) how many of the coders will actually end up using the
finished product ? Answer: Usually none. (3) How many of the test team do
have first hand experience or the expert knowledge of the underlying
business logic/processes ? Answer: Usually minimal.
154 http://www.certmagic.com
BH0-003
Even if the testers are indeed experts in their area, they may miss the big
picture, so I think that usability testing is a sub-specialty that often is not
best left to the average tester. Only some specific personnel should be
responsible for doing Usability Testing.
Thirdly, apart from the usual commercial considerations, the success of some
new software will depend on how well it is received by the public - whether
they like the application . Obviously if the s/w is bug ridden then the
popularity of the s/w will suffer; aside from that, if it is a high quality
development the popularity of the s/w will still depend on the usability (albeit
to a lesser degree). It would be a pity (but it wouldn't be the first time) that
an application was not a success because it wasn't readily accepted -
because it was not user friendly, or because it was too complex or difficult to
use.
How to approach Usability Testing
How to Implement Usability Testing
The best way to implement usability testing is two fold - firstly from a design
& development perspective, then from a testing perspective.
From a design viewpoint, usability can be tackled by (1) Including actual
Users as early as possible in the design stage. If possible, a prototype should
be developed - failing that, screen layouts and designs should be reviewed
on-screen and any problems highlighted.. The earlier that potential usability
issues are discovered the easier it is to fix them.
(2) Following on from the screen reviews, standards should be documented
i.e. Screen Layout, Labelling/Naming conventions etc. These should then be
applied throughout the application.
Where an existing system or systems are being replaced or redesigned,
usability issues can be avoided by using similar screen layouts - if they are
already familiar with the layout the implementation of the new system will
present less of a challenge, as it will be more easily accepted (provided of
course, that that is not why the system is being replaced).
3). Including provisions for usability within the design specification will assist
later usability testing. Usually for new application developments, and nearly
always for custom application developments, the design team should either
have an excellent understanding of the business processes/rules/logic behind
the system being developed; and include users with first hand knowledge of
same. However, although they design the system, they rarely specifically
include usability provisions in the specifications.
155 http://www.certmagic.com
BH0-003
An example of a usability consideration within the functional specification
may be as simple as specifying a minimum size for the 'Continue' button.
4). At the unit testing stage, there should be an official review of the system
- where most of those issues can more easily be dealt with. At this stage,
with screen layout & design already reviewed, the focus should be on how a
user navigates through the system. This should identify any potential issues
such as having to open an additional window where one would suffice. More
commonly though, the issues that are usually identified at this stage relate to
the default or most common actions. For example, where a system is
designed to cope with multiple eventualities and thus there are 15 fields on
the main input screen - yet 7 or 8 of these fields are only required in rare
instances. These fields could then be set as hidden unless triggered, or
moved to another screen altogether.
5). All the previous actions could be performed at an early stage if
Prototyping is used. This is probably the best way to identify any potential
usability/operability problems. You can never lessen the importance of user-
centered design, but you can solve usability problems before they get to the
QA stage (thereby cutting the cost of rebuilding the product to correct the
problem) by using prototypes (even paper prototypes) and other "discount
usability" testing methods.
6). From a testing viewpoint, usability testing should be added to the
testing cycle by including a formal "User Acceptance Test". This is done by
getting several actual users to sit down with the software and attempt to
perform "normal" working tasks, when the software is near release quality. I
say "normal" working tasks because testers will have been testing the
system from/using test cases - i.e. not from a users viewpoint. User testers
must always take the customer's point of view in their testing.
User Acceptance Testing (UAT) is an excellent exercise, because not only will
it give you there initial impression of the system and tell you how readily the
users will take to it, but this way it will tell you whether the end product is a
closer match to their expectations and there are fewer surprises. (Even
though usability testing at the later stages of development may not impact
software changes, it is useful to point out areas where training is needed to
overcome deficiencies in the software.
(7) Another option to consider is to include actual users as testers within the
test team. One financial organization I was involved with reassigned actual
users as "Business Experts" as members of the test team. I found their input
as actual "tester users" was invaluable.
8). The final option that may be to include user testers who are eventually
going to be (a) using it themselves; and/or (b) responsible for training and
156 http://www.certmagic.com
BH0-003
effectively "selling" it to the users.
The Benefits of Usability Testing
The benefits of having had usability considerations included in the
development of computer software are immense, but often unappreciated.
The benefits are too numerous to list - I'd say it's similar to putting the coat
of paint on a new car - the car itself will work without the paint, but it doesn't
look good. To summarise the benefits I would just say that it makes the
software more "user friendly". The end result will be:
• Better quality software.
• Software is easier to use.
• Software is more readily accepted by users.
• Shortens the learning curve for new users.
The Role and benefits of "Usability Testers"
Apart from discovering and preventing possible usability issues, the addition
of 'Usability Testers' to the test team can have a very positive effect on the
team itself. Several times I have seen that testers become too familiar with
the "quirks" of the software - and not report a possible error or usability
issue. Often this is due to the tester thinking either "It's always been like
that" or "isn't that the way it's supposed to be ?". These types of problem
can be allieviated by including user testers in the test team.
They can also help to:
• Refocus the testers and increase their awareness to usability issues,
by providing a fresh viewpoint
• Provide and share their expert knowledge - training the testers to the
background and purpose of the system
• Provide a "realistic" element to the testing, so that test scenarios are
not just "possible permutations".
Summary:
1. Usability evaluation should be incorporated earlier in the software development cycle to
minimize resistance to changes in a hardened user interface;
2. Organizations should have an independent usability evaluation of software products to avoi
the temptation to overlook problems to release the product;
157 http://www.certmagic.com
BH0-003
3. Multiple categories of dependent measures should be employed in usability testing because
subjective measurement is not always consonant with user performance; and
4. Even though usability testing at the later stages of development may not impact software
changes, it is useful to point out areas where training is needed to overcome deficiencies in
software.
In my experience, the greater the involvement of key users, the more
pleased they will be with the end product. Getting management to commit
their key people to this effort can be difficult, but it makes for a better
product in the long run.
Classification of Errors by Severity
Often the severity of a software defect can vary even though the
software never changes. The reason being is that a software defect’s
severity depends on the system in which it runs.
For example, the severity of the Pentium’s floating-point defect
changes from system to system. On some systems, the severity is
small; whereas on other systems, the severity is high.
Another problem (which occurs regularly) is that the definitions of
the severity levels (or categories) themselves change depending on
the type of system. For example, a catastrophic defect in a nuclear
system means that the fault can result in death or environmental
harm; a catastrophic defect in a database system means that the
fault can (or did) cause the loss of valuable data.
Therefore, the system itself determines the severity of a defect based
on the context for which the defect applies. The context makes all
the difference in how to classify a defect’s severity.
I have attached two sample classification methods – a 3 level
classification method, and a more comprehensive 5 level
classification method, which I hope you may find useful.
3 Level Error Classification Method
Errors, which are agreed as valid, will be categorised as follows :-
• Category A - Serious errors that prevent System test of a particular
function continuing or serious data type error
• Category B - Serious or missing data related errors that will not
prevent implementation.
158 http://www.certmagic.com
BH0-003
• Category C - Minor errors that do not prevent or hinder
functionality.
Explanation of Classifications
1. An "A" bug is a either a showstopper or of such importance as to radically
affect the functionality of the system i.e. :
If, because of a consistent crash during processing of a
new application, a user could not complete that application.
Incorrect data is passed to system resulting in corruption
or system crashes
Example of severally affected functionality:
Calculation of repayment term/amount are incorrect
Incorrect credit agreements produced
2. Bugs would be classified as "B" where a less important element of
functionality is affected, e.g.:
a value is not defaulting correctly and it is necessary to
input the correct value
data is affected which does not have a major impact, for
example - where an element of a customer application was not
propagated to the database
there is an alternative method of completing a particular
process - e.g. a problem might occur which has a work-around.
Serious cosmetic error on front-end.
3. "C" type bugs are mainly cosmetic bugs i.e.:
Incorrect / misspelt text on screens
drop down lists missing or repeating an option
5 Level Error Classification Method
1. Catastrophic: Defects that could (or did) cause ditrous consequences for
the system in question.
E.g.) critical loss of data, critical loss of system availability,
critical loss of security, critical loss of safety, etc.
159 http://www.certmagic.com
BH0-003
2. Severe: Defects that could (or did) cause very serious consequences for the
system in question.
E.g.) A function is severely broken, cannot be used and there is no
workaround.
3. Major: Defects that could (or did) cause significant consequences for the
system in question - A defect that needs to be fixed but there is a
workaround.
E.g. 1.) losing data from a serial device during heavy loads.
E.g. 2.) Function badly broken but workaround exists
4. Minor: Defects that could (or did) cause small or negligible consequences
for the system in question. Easy to recover or workaround.
E.g.1) Error messages misleading.
E.g.2) Displaying output in a font or format other than what the
customer desired.
5. No Effect: Trivial defects that can cause no negative consequences for the
system in question. Such defects normally produce no erroneous
outputs.
E.g.1) simple typos in documentation.
E.g.2) bad layout or mis-spelling on screen.
1. What is Software Testing?
Software testing is more than just error detection;
Testing software is operating the software under controlled
conditions, to (1) verify that it behaves “as specified”; (2) to detect
errors, and (3) to validate that what has been specified is what the
user actually wanted.
1. Verification is the checking or testing of items, including software, for
conformance and consistency by evaluating the results against pre-
specified requirements. [Verification: Are we building the system
right?]
160 http://www.certmagic.com
BH0-003
2. Error Detection: Testing should intentionally attempt to make things
go wrong to determine if things happen when they shouldn’t or things
don’t happen when they should.
3. Validation looks at the system correctness – i.e. is the process of
checking that what has been specified is what the user actually
wanted. [Validation: Are we building the right system?]
In other words, validation checks to see if we are building what the customer
wants/needs, and verification checks to see if we are building that system
correctly. Both verification and validation are necessary, but different
components of any testing activity.
The definition of testing according to the ANSI/IEEE 1059 standard is that
testing is the process of analysing a software item to detect the differences
between existing and required conditions (that is defects/errors/bugs) and to
evaluate the features of the software item.
Remember: The purpose of testing is verification, validation and error
detection in order to find problems – and the purpose of finding those
problems is to get them fixed.
2. Why Testing CANNOT Ensure Quality
Testing in itself cannot ensure the quality of software. All testing can do is
give you a certain level of assurance (confidence) in the software. On its
own, the only thing that testing proves is that under specific controlled
conditions, the software functioned as expected by the test cases executed.
3. What is Software “Quality”?
Quality software is reasonably bug-free, delivered on time and within budget,
meets requirements and/or expectations, and is maintainable.
However, quality is a subjective term. It will depend on who the ‘customer’ is
and their overall influence in the scheme of things. A wide-angle view of the
‘customers’ of a software development project might include end-users,
customer acceptance testers, customer contract officers, customer
management, the development organisation’s
management/accountants/testers/salespeople, future software maintenance
engineers, stockholders, magazine reviewers, etc. Each type of ‘customer’
will have their own view on ‘quality’ - the accounting department might
161 http://www.certmagic.com
BH0-003
define quality in terms of profits while an end-user might define quality as
user-friendly and bug-free.
4. What is “Quality Assurance”?
“Quality Assurance” measures the quality of processes used to create a
quality product.
Software Quality Assurance (‘SQA’ or ‘QA’) is the process of monitoring and
improving all activities associated with software development, from
requirements gathering, design and reviews to coding, testing and
implementation.
It involves the entire software development process - monitoring and
improving the process, making sure that any agreed-upon standards and
procedures are followed, and ensuring that problems are found and dealt
with, at the earliest possible stage. Unlike testing, which is mainly a
‘detection’ process, QA is ‘preventative’ in that it aims to ensure quality in
the methods & processes – and therefore reduce the prevalence of errors in
the software.
Organisations vary considerably in how they assign responsibility for QA and
testing. Sometimes they’re the combined responsibility of one group or
individual. Also common are project teams that include a mix of testers and
developers who work closely together, with overall QA processes monitored
by project managers or quality managers.
5. Quality Assurance and Software Development
Quality Assurance and development of a product are parallel activities.
Complete QA includes reviews of the development methods and standards,
reviews of all the documentation (not just for standardisation but for
verification and clarity of the contents also). Overall Quality Assurance
processes also include code validation.
A note about quality assurance: The role of quality assurance is a superset of
testing. Its mission is to help minimise the risk of project failure. QA people
aim to understand the causes of project failure (which includes software
errors as an aspect) and help the team prevent, detect, and correct the
problems. Often test teams are referred to as QA Teams, perhaps
acknowledging that testers should consider broader QA issues as well as
testing.
162 http://www.certmagic.com
BH0-003
6. What’s the difference between QA and testing?
Simply put:
TESTING means “Quality Control”; and
QUALITY CONTROL measures the quality of a product; while
QUALITY ASSURANCE measures the quality of processes used to
create a quality product.
7. The Mission of Testing
In well-run projects, the mission of the test team is not merely to perform
testing, but to help minimise the risk of product failure. Testers look for
manifest problems in the product, potential problems, and the absence of
problems. They explore, assess, track, and report product quality, so that
others in the project can make informed decisions about product
development. It's important to recognise that testers are not out to "break
the code." We are not out to embarrass or complain, just to inform. We are
human meters of product quality.
The Importance of Scalability & Load Testing
Some very high profile websites have suffered from serious outages
and/or performance issues due to the number of people hitting their
website. E-commerce sites that spent heavily on advertising but not
nearly enough on ensuring the quality or reliability of their service
have ended up with poor web-site performance, system downtime
and/or serious errors, with the predictable result that customers are
being lost.
In the case of toysrus.com, its web site couldn't handle the
approximately 1000 percent increase in traffic that their advertising
campaign generated. Similarly, Encyclopaedia Britannica was unable to
keep up with the amount of users during the immediate weeks
following their promotion of free access to its online database. The
truth is, these problems could probably have been prevented, had
adequate load testing taken place.
When creating an eCommerce portal, companies will want to know
whether their infrastructure can handle the predicted levels of traffic,
to measure performance and verify stability.
163 http://www.certmagic.com
BH0-003
These types of services include Scalability / Load / Stress testing, as
well as Live Performance Monitoring.
Load testing tools can be used to test the system behaviour and
performance under stressful conditions by emulating thousands of
virtual users. These virtual users stress the application even harder
than real users would, while monitoring the behaviour and response
times of the different components. This enables companies to minimise
test cycles and optimise performance, hence accelerating deployment,
while providing a level of confidence in the system.
Once launched, the site can be regularly checked using Live
Performance Monitoring tools to monitor site performance in
real time, in order to detect and report any performance
problems - before users can experience them.
Preparing for a Load Test
The first step in designing a Web site load test is to measure as
accurately as possible the current load levels.
Measuring Current Load Levels
The best way to capture the nature of Web site load is to identify and
track, [e.g. using a log analyzer] a set of key user session variables
that are applicable and relevant to your Web site traffic.
Some of the variables that could be tracked include:
the length of the session (measured in pages)
the duration of the session (measured in minutes and seconds)
the type of pages that were visited during the session (e.g.,
home page, product information page, credit card information
page etc.)
the typical/most popular ‘flow’ or path through the website
the % of ‘browse’ vs. ‘purchase’ sessions
the % type of users (new user vs. returning registered user)
Measure how many people visit the site per week/month or
day. Then break down these current traffic patterns into one-
hour time slices, and identify the peak-hours (i.e. if you get lots
of traffic during lunch time etc.), and the numbers of users
during those peak hours. This information can then be used to
estimate the number of concurrent users on your site.
164 http://www.certmagic.com
BH0-003
Concurrent Users
Although your site may be handling x number of users per day,
only a small percentage of these users would be hitting your
site at the same time. For example, if you have 3000 unique
users hitting your site on one day, all 3000 are not going to be
using the site between 11.01 and 11.05 am.
So, once you have identified your peak hour, divide this hour
into 5 or 10 minute slices [you should use your own judgement
here, based on the length of the average user session] to get
the number of concurrent users for that time slice.
Estimating Target Load Levels
Once you have identified the current load levels, the next step is to
understand as accurately and as objectively as possible the nature of
the load that must be generated during the testing.
Using the current usage figures, estimate how many people will visit
the site per week/month or day. Then divide that number to attain
realistic peak-hour scenarios.
It is important to understand the volume patterns, and to determine
what load levels your web site might be subjected to (and must
therefore be tested for).
There are four key variables that must be understood in order to
estimate target load levels:
how the overall amount of traffic to your Web site is expected to
grow
the peak load level which might occur within the overall traffic
how quickly the number of users might ramp up to that peak
load level
how long that peak load level is expected to last
Once you have an estimate of overall traffic growth, you’ll need
to estimate the peak level you might expect within that overall
volume.
Estimating Test Duration
165 http://www.certmagic.com
BH0-003
The duration of the peak is also very important-a Web site that may
deal very well with a peak level for five or ten minutes may crumble if
that same load level is sustained longer than that. You should use the
length of the average user session as a base for determining the load
test duration.
Ramp-up Rate
As mentioned earlier, Although your site may be handling x number of
users per day, only a small percentage of these users would be hitting
your site at the same time.
Therefore, when preparing your load test scenario, you should take
into account the fact that users will hit the website at different times,
and that during your peak hour the number of concurrent users will
likely gradually build up to reach the peak number of users, before
tailing off as the peak hour comes to a close.
The rate at which the number of users build up, the "Ramp-up Rate"
should be factored into the load test scenarios (i.e. you should not just
jump to the maximum value, but increase in a series of steps).
Scenario Identification
The information gathered during the analysis of the current traffic is
used to create the scenarios that are to be used to load test the web
site.
The identified scenarios aim to accurately emulate the behavior of real
users navigating through the Web site.
for example, a seven-page session that results in a purchase is going
to create more load on the Web site than a seven-page session that
involves only browsing. A browsing session might only involve the
serving of static pages, while a purchase session will involve a number
of elements, including the inventory database, the customer database,
a credit card transaction with verification going through a third-party
system, and a notification email. A single purchase session might put
as much load on some of the system’s resources as twenty browsing
sessions.
Similar reasoning may apply to purchases from new vs. returning
users. A new user purchase might involve a significant amount of
account setup and verification —something existing users may not
require. The database load created by a single new user purchase may
166 http://www.certmagic.com
BH0-003
equal that of five purchases by existing users, so you should
differentiate the two types of purchases.
Script Preparation
Next, program your load test tool to run each scenario with the
number of types of users concurrently playing back to give you a the
load scenario.
The key elements of a load test design are:
test objective
pass/fail criteria
script description
scenario description
Load Test The objective of this load test is to determine if the Web
Objective site, as currently configured, will be able to handle the X
number of sessions/hr peak load level anticipated. If the
system fails to scale as anticipated, the results will be
analyzed to identify the bottlenecks.
Pass/Fail Criteria The load test will be considered a success if the Web site
will handle the target load of X number of sessions/hr
while maintaining the pre-defined average page response
times (if applicable). The page response time will be
measured and will represent the elapsed time between a
page request and the time the last byte is received.
Since in most cases the user sessions follow just a few navigation
patterns, you will not need hundreds of individual scripts to achieve
realism—if you choose carefully, a dozen scripts will take care of most
Web sites.
Script Execution
Scripts should be combined to describe a load testing scenario. A basic
scenario includes the scripts that will be executed, the percentages in
which those scripts will be executed, and a description of how the load
will be ramped up.
167 http://www.certmagic.com
BH0-003
By emulating multiple business processes, the load testing can
generate a load equivalent to X numbers of virtual users on a Web
application. During these load tests, real-time performance monitors
are used to measure the response times for each transaction and
check that the correct content is being delivered to users. In this way,
they can determine how well the site is handling the load and identify
any bottlenecks.
The execution of the scripts opens X number of HTTP sessions (each
simulating a user) with the target Web site and replays the scripts
over and over again. Every few minutes it adds X more simulated
users and continues to do so until the web site fails to meet a specific
performance goal.
System Performance Monitoring
It is vital during the execution phase to monitor all aspects of the
website. This includes measuring and monitoring the CPU usage and
performance aspects of the various components of the website – i.e.
not just the webserver, but the database and other parts aswell (such
as firewalls, load balancing tools etc.)
For example, one etailer, whose site fell over (apparently due to a high
load), when analysing the performance bottlenecks on their site
discovered that the webserver had in fact only been operating at 50%
of capacity. Further investigation revealed that the credit card
authorisation engine was the cause of failure – it was not responding
quick enough for the website, which then fellover when it was waiting
for too many responses from the authorisation engine. They resolved
this issue by changing the authorisation engine, and amending the
website coding so that if there were any issues with authorisation
responses in future, the site would not crash.
Similarly, another ecommerce site found that the performance issues
that they were experiencing were due to database performance issues
– while the webserver CPU usage was only at 25%, the backend db
server CPU usage was 86%. Their solution was to upgrade the db
server.
Therefore, it is necessary to use (install if necessary) performance
monitoring tools to check each aspect of the website architecture
during the execution phase.
168 http://www.certmagic.com
BH0-003
Suggested Execution Strategy:
Start with a test at 50% of the expected virtual user capacity
for 15 minutes and a medium ramp rate. The different
members of the team [testers will also need to be monitoring
the CPU usage during the testing] should be able to check
whether your website is handling the load efficiently or some
resources are already showing high utilization.
After making any system adjustments, run the test again or
proceed to 75% of expected load. Continue with the testing
and proceed to 100%; then up to 150% of the expected load,
while monitoring and making the necessary adjustments to
your system as you go along.
Results Analysis
Often the first indication that something is wrong is the end user
response times start to climb. Knowing which pages are failing will
help you narrow down where the problem is.
Whichever load test tool you use, it will need to produce reports that
will highlight the following:
Page response time by load level
Completed and abandoned session by load level
Page views and page hits by load level
HTTP and network errors by load level
Concurrent user by minute
Missing links report, if applicable
Full detailed report which includes response time
by page and by transaction, lost sales opportunities, analysis
and recommendations
Important Considerations
When testing websites, it is critically important to test from outside the
firewall. In addition, web-based load testing services, based outside
the firewall, can identify bottlenecks that are only found by testing in
this manner.
Web-based stress testing of web sites are therefore more accurate
when it comes to measuring a site's capacity constraints.
169 http://www.certmagic.com
BH0-003
Web traffic is rarely uniformly distributed, and most Web sites exhibit
very noticeable peaks in their volume patterns. Typically, there are a
few points in time (one or two days out of the week, or a couple of
hours each day) when the traffic to the Web site is highest.
The Rise and Rise of Outsourcing
According to a recent Gartner report, increased spending on IT
outsourcing will be instrumental in driving an average annual growth
rate of 19 per cent for the professional services industry over the next
three years.
Furthermore, according to Forrester Research, some 60 per cent of
companies will employ three or more service providers to manage
their core business processes by 2002.1[1]
More recently it has also been claimed that the use of outsourcers to
supply non-core IT and business processes will increase in direct
proportion to the downturn in the wider economy.
In line with these predictions, more and more organisations are now
looking to outsource part of their software testing function,
traditionally the department that gets axed first when cutbacks are
required.
Why choose to Outsource Testing?
For the majority of organisations, the testing effort is often cyclical,
with high and low peaks of activity. However, because of this it is not
efficient or practical to maintain a fully resourced test team all year
round, leaving excess capacity during the low peaks.
One solution is to use a specialist outsourcing testing company to
provide the help you need, either in the form of attaining in-house
assistance (consultants onsite), or, as discussed in this article,
outsourcing this function completely.
170 http://www.certmagic.com
BH0-003
Other reasons for outsourcing their testing includes: to cut costs, to
speed testing, to improve their testing, to save office space, acquire
test environment facilities, and release in-house staff to take on
development activities.
The Benefits of Outsourcing
The use of outsourced test teams has increased dramatically in recent
years with the recognition that companies can benefit from a wealth of
testing expertise that can add to, and complement, in-house
knowledge.
The main advantage is that instead of maintaining a fully resourced
test team all year round, when test activity peaks core permanent
employees can either be supplemented by outsourced testing
personnel, or the excess work is outsourced completely.
Companies can now contact service suppliers to avail of end-to-end
software quality services, for use at any point during development, to
shorten time-to-market cycles without taxing internal resources.
Additionally, they can benefit from contracting in testers who will be
specialists in the field.
Utilising an outside test support services firm can provide the ability to
utilise resources in a "just-in-time" fashion. Ideally, the outsource
vendor can provide all of the services that may be required for the
system testing. The energy expended in building the relationship can
help the client achieve a level of confidence in the quality of the
product, which would historically require a larger internal team.
The ability to then call on those resources when they are needed, and
then have the ability to give them back when they are no longer
needed, provides a significant cost saving. In addition, it allows
everybody focus on what they do best, thereby maximising everyone's
resources.
There are a number of additional benefits that can be achieved,
including -
These specialists will seamlessly increase the value of
your company and allows for more efficiency because both
partners in this outsource relationship will be focused on what
they do best - a software developer needs to focus on making
great software.
171 http://www.certmagic.com
BH0-003
The use of independent test teams can validate that the
existing testing and procedures are up to the mark.
As a complementary service, it can improve web-
enabled software quality, accelerate development cycles, and
reduce cost of building software.
Experienced & skilled testers can be acquired for short
periods of time.
The alliance enables the organisation to perform a
greater range of testing services than is currently possible on
it’s own.
This solution dramatically improves software reliability
while accelerating time-to-market and reducing the cost of
development.
Compatibility lab testing can provide for testing your
application against various combinations of Hardware, Operating
Systems & Internet Browsers – particularly important for
eBusinesses, but one that might not be achievable by the
company alone. [For example, they may have a server farm
that you cannot afford or justify for the testing, no matter how
important].
* Return on Investment - learn how to find costly and
embarrassing problems before your customers find them.
The First Steps to Successful Outsourcing
I would recommend that you consider the following points when
deciding whether to outsource a project or application for testing.
1. The nature of the project
Firstly, consider the nature of the project or application that you are
considering outsourcing.
172 http://www.certmagic.com
BH0-003
If the project is an enhancement version of a stable product, with
existing test scripts that could be used by the outsourcing partner, it
would be ideal to outsource.
However, if it is the first release of a totally new application, it may not
be ideal to outsource the testing. For example, if there are major
problems with the application, testing is not going to solve them, and
if your testing partner encounters a large number of serious errors the
test coverage will probably be lessened.
Lastly, carefully consider what you expect to achieve by outsourcing.
In all likelihood, outsourcing will never (and probably should never)
completely do away with the necessity to have in-house expertise, so
you should consider the loss of knowledge and hands-on experience
that your test team will lose.
2. What testing do you want them to do?
It is also important to consider what types of testing that you actually
require to be performed. For example, if stress testing is important, an
outsourced testing firm may be more experienced and suited towards
performing this. Or, you may wish to perform a certain aspect of the
testing that is very complex and that you believe you would not
sufficiently benefit from by outsourcing.
3. The Practicalities
Who is going to be fixing the bugs? If your company is going to, the
process and procedures by which the error reports are going to be
exchanged is critical. If your error management system cannot be
remotely updated [i.e. by the outsourced partner reporting the bugs
directly] then I would suggest either not outsourcing at all, or else
implementing an error management tool which can cater for remote
logging, and can be under your direct control. It is important that you
retain control (or at least possession) of the bug database for future
reference.
Secondly, how are you going to hand over new builds of the s/w? If
you have to burn CD’s, unless there is a quick way of transferring the
new builds over to the outsourced testing partner, valuable testing
time may be lost.
Choosing an Outsourced Testing Partner
173 http://www.certmagic.com
BH0-003
When choosing to utilise outsourced testing services, it is important to
check the following:
• Their past experience – specifically in testing the same type of
project
• Their current (not historical) client list
• Bios of their key personnel that are going to be working on your
project (and not just Bio’s of their best people).
• Detailed evidence of the existence of testing processes and
procedures, particularly for preparing test cases/scripts;
executing test cases/scripts; and (most importantly) their entire
review process.
Risk Planning for Outsourced Testing
What’s the big deal - after all, they will actually do the work, won’t
they? I mean, what could go wrong?
As the extent of the outsourcing grows, however, so too will the
challenge of managing relationships with them. But if companies fail to
do this effectively, projects could be jeopardised. The danger is that
weak outsourcing partners can stall entire projects. So, how do you
provide for this situation?
Apart from the contractual elements to formalise the relationship, you
must naturally consider how you can actually verify their progress.
Additionally, you need also to be sure that the work is being performed
to a high level of quality.
In my opinion, the only way to ensure this is to hold regular reviews
and assessments with the partner. (See next section on reviewing).
If possible, send a representative (one of your testers) to be part of
their team. In addition to providing you a direct insight into the actual
progress and quality, the representative can assist the outsourcer
either by having previous experience of that application, or at the very
least, an informal channel of communication.
Managing Outsourced Testing Projects
174 http://www.certmagic.com
BH0-003
One of the key ingredients for success when outsourcing is the
relationship. There must be trust between both parties. This trust is
necessary in order to communicate the value points and the methods
of communication that are critical for supporting the flexibility that
using a test support services firm requires.
You should take a collaborative approach, and work together –
after all, you are both working towards the same goal –
releasing a quality product.
I would recommend the following:
Firstly, define the channels of communication. Appoint a person
from your company, to act as the first level of contact between
your company and the outsourced partner, who will deal
directly with an agreed specified person on their side (ideally
the test manager for the project).
These two people are the first-level team for resolving
problems as they arise, and additionally your representative
also provides and archives all information passed to the
outsourcing partner.
Secondly, agree escalation procedures, and specify to whom
the problem is to be escalated to - on each side.
Set out milestones – but beware when setting milestones, that
YOU are the ones who state whether the outsourced testing
company has reached the milestone or not – for example, if one
milestone is the production of the test plan, if you are not
happy with the test plan then you must be in a position to state
that they have NOT reached the milestone. Otherwise you could
end up in a position whereby they gave you a test plan that you
considered inadequate, yet they are looking for payment for
reaching the milestone.
To cater for this, and other concerns, it is vitally important to
specify formal review points – stages at which progress can be
measured, and where issues can be highlighted.
Formal Review Points:
Test Plan Review Insist that you must sign off on the test
plan before test preparation can
commence. Hold a round table review (by
phone or videoconferencing as available)
and ensure the test plan clearly sets out
175 http://www.certmagic.com
BH0-003
the approach to be taken, identifying
what levels and types of testing are to be
applied and the methods, techniques,
tools and resources to be used, as well as
the review points, schedule, test coverage
required, and signoff criteria. [See here
for more details about test plan
contents].
If you are not satisfied with the test plan,
do not signoff. Ensure that one of the
criteria for proceeding to the next stage is
your signoff.
Test Case Review Ensure that all test cases are reviewed
and signed off by your own business
analysts/product specialists, to ensure
that they are of sufficient quality, depth,
accuracy and coverage.
Insist on modifications to the test cases
until you are satisfied with their quality.
Remember – the testing will probably
only be as good as the test cases – you
should assume that anything that is not
defined as a test scenario will not be
tested, no matter how obvious it may be
to you.
Execution Readiness An execution readiness review is
Review performed to ensure that the outsourcing
company are ready to move into the Test
Execution phase. This readiness review
checks to ensure that the test entrance
criteria, as specified in the Test Plan,
have been met.
Execution Progress Ensures that regular detailed progress
Reporting information, including full bug details and test
case pass/fail statistics are passed to you
throughout the execution phase.
During this phase it is important that you can
analyse the bugs that are being logged and
match the bugs to specific test cases. This can
then serve as an independent check of the test
execution progress.
176 http://www.certmagic.com
BH0-003
In parallel you can perform some testing
yourself, to double-check the test cases that
are passing.
Build Acceptance If you hand over software that is not of
Tests ‘testable’ quality, the outsourced partner is not
going to be able make much progress and
neither of you are going to benefit at all.
Therefore, you should insist that ‘Build
Acceptance Tests’ are performed on each
build/drop of software - either perform these
yourself, or get the outsourcing partner to
perform them. These tests are simple checks
to ensure that the software is of ‘testable’
quality in that they are going to able to test
the various functions and not be stuck by a
very obvious showstopper bug. If they cannot
proceed with the testing there is no point
blaming them.
Bug Turnaround As I am sure you are aware, waiting for bugs
Times to be fixed can be a serious bottleneck for all
testing projects, and possibly even more so for
an outsourced project. Therefore, you should
actively monitor bug turnaround times for
bottlenecks.
It is probably also more appropriate that
somebody from your company be the one
escalating/hassling the developers to fix the
bugs, rather than an external person trying to.
Exit Report & Final Ensure they prepare an Exit Report, which will
Signoff. summarise the testing and includes the
following details:
• The full bug list
• List of any outstanding bugs
• List of any uncompleted test cases
• A copy of all Test spreadsheets
• A copy of all test results & statistics
Do not sign off on this document until you are
satisfied as to the accuracy of the contents,
particularly the test coverage metrics. Ask to
see log files or any other evidence of test
execution having taken place.
177 http://www.certmagic.com
BH0-003
Final Deliverables
A final consideration to take into account is that you may have to test
this project yourself in the future. However, this could be more
difficult because you do not have the benefit of the knowledge and
hands-on experience of actually testing the project in the first place.
In order to minimise this risk it is important to specify as a condition
of the project that upon completion of the testing, you should be
handed back all test documentation.
For example, you should include the following as specific
deliverables:
• The final test plan document
• All updated and revised test cases and expected results
• All test data requirements
• Testing pre-conditions, including test sequence, test
environment setup and initialisation
• The Bug database
• All scripts etc.
In this way if you have to test it yourself at least you will not be
starting from scratch.
Conclusion
Once you have picked a good partner, and gone through the first
project together, outsourced testing can be an excellent option. But
it must be noted that to make a success of it, it requires continuous
monitoring and assessment, and careful choice of projects to
outsource.
There must be trust between both parties. This trust is necessary in
order to communicate the value points and the methods of
communication that are critical for supporting the flexibility that
using a test support services firm requires.
You should also take into account that the old adage ‘garbage in,
garbage out’ is particularly true in these situations. If you handover
a product or information for testing that is not of ‘testable’ quality, it
is not going to help either of you.
178 http://www.certmagic.com
BH0-003
Upon completion of each project you should perform a final
acceptance test on all returned software – but be open about this
process and give constructive feedback. In addition, you should also
assess each other’s performance, particularly with regard to
bottlenecks, and be open to accepting criticism of your own
company’s performance.
Keep in mind that if you do decide to outsource some or all of your
testing, a lot of effort will still be required by yourselves in order to
support and enable the outsource partner to maximise performance
(for example, reviewing test cases etc.)
Previously testing often focussed on functional issues and risks as
being the main concerns. However, with the advent of eBusiness
development, this focus has changed. Non-functional issues such as
cross browser compatibility / configuration, usability, performance
and security have become more and more important considerations
when planning test projects. It is also becoming more prevalent that
companies are increasingly making use of outsourced testing to get
the expertise quickly without expending all of their capital.
Specialisation breeds a continued need for companies to focus on
core competencies, and by using strategic outsourcing to fill gaps
helps fast-growing companies in the high-tech industry keep focused
on their initial goals and keeps resources allocated to the areas that
need them most.
Unit Test Tools
These tools, frameworks, and libraries support unit testing, which is
usually performed by the developer, usually using interfaces below the
public interfaces of the software under test.
AdaTEST 95
AQtest
Aunit
C++Test
Cantata++
Check
COMPUTE
179 http://www.certmagic.com
BH0-003
Tessy
.TEST
Test Mentor - Java Edition
TestGen4J
unit++
vbUnit3 Basic
VectorCAST
XMLUnit
XSLTunit
These tools fit better into a different category, but are worth mentioning
here as well. Their main listing is in the other category.
jfcUnit
Jtest
Rational Test RealTime Unit Testing
Kind of Tool
Rational Test RealTime's Unit Testing feature automates C, C++, Ada
83 and 95 software component testing.
Organization
IBM Rational Software
http://www.rational.com/
Software Description
Rational Test RealTime Unit Testing performs black-box/functional
testing, i.e. verifies that all units behave according to their
specifications without regard to how that functionality is implemented.
The Unit Testing feature has the flexibility to naturally fit any
development process by matching and automating developers' and
testers' work patterns, allowing them to focus on value-added tasks.
Rational Test RealTime is integrated with native development
environments (Unix and Windows) as well as with a large variety of
cross-development environments. Find out more about Rational Test
RealTime at: http://www.rational.com/products/testrt/index.jsp
Platforms
182 http://www.certmagic.com
BH0-003
Rational Test RealTime is available for most development and target
systems including Windows, Unix (Solaris, HP-UX, Linux)
C++Test
Kind of Tool
A C/C++ unit testing tool that automatically tests any C/C++ class,
function, or component.
Organization
ParaSoft Corporation
http://www.parasoft.com/
Entry added April 4, 2001.
JUnit
Kind of Tool
A regression testing framework used by developers who implement
unit tests in Java. (freeware)
Organization
JUnit.org
E-mail: junit@objectmentor.com
http://www.junit.org/
Software Description
JUnit is a regression testing framework written by Erich Gamma and
Kent Beck. It is used by the developer who implements unit tests in
Java.
Platforms
Platforms running Java.
Entry added April 25, 2001.
AQtest
Kind of Tool
Automated support for functional, unit, and regression testing
Organization
AutomatedQA Corp.
http://www.automatedqa.com/
Software Description
183 http://www.certmagic.com
BH0-003
AQtest automates and manages functional tests, unit tests and
regression tests, for applications written with VC++, VB, Delphi,
C++Builder, Java or VS.NET. It also supports white-box testing, down
to private properties or methods. External tests can be recorded or
written in three scripting languages (VBScript, JScript, DelphiScript).
Using AQtest as an OLE server, unit-test drivers can also run it directly
from application code. AQtest automatically integrates AQtime when it
is on the machine. Entirely COM-based, AQtest is easily extended
through plug-ins using the complete IDL libraries supplied. Plug-ins
currently support Win32 API calls, direct ADO access, direct BDE
access, etc.
Platforms
Windows 95, 98, NT, or 2000
Entry updated June 13, 2001.
JsUnit (Hieatt)
Kind of Tool
JavaScript unit testing framework (freeware)
Organization
Edward Hieatt
E-mail: edward@jsunit.net
http://www.jsunit.net/
Software Description
JsUnit is a Unit Testing framework for client-side (in-browser)
JavaScript. It is essentially a port of JUnit to JavaScript.
Licensed under the GNU Public License 2.0, GNU Lesser Public License
2.1, and the Mozilla Public License 1.1
Platforms
Platforms supporting JavaScript 1.4 or higher.
MinUnit
Kind of Tool
Minimal Unit Testing Framework for C (freeware)
Organization
Jera Design
E-mail: jbrewer@jera.com
http://www.jera.com/techinfo/jtns/jtn002.html
Software Description
184 http://www.certmagic.com
BH0-003
A minimal unit testing framework for C. It doesn't use malloc, so it
may be more suitable for certain kinds of embedded systems.
Platforms
Any platform with an ANSI C compiler.
TBGEN
Kind of Tool
Ada test harness generator, unit/integration testing environment
Organization
Testwell Oy
http://www.testwell.fi/tbgdesc.html
Software Description
TBGEN (Test Bed Generator for Ada'83) generates test driver (and
stubs, if needed), which are compiled with the units under test
resulting in a test bed program. Using Ada-like command language the
test bed facilitates specification-based (black-box) unit and integration
testing. Both interactive and script-based tests are supported. The
work becomes automated, effective, intuitive, visible, documented,
standardized, measurable. Can be used with GNAT Ada compiler.
Platforms
Windows-2000/NT/9x, VAX/VMS, Solaris. Ask also for source code
licensing (written in 'vanilla' Ada'83). On a given platform independent
of the used Ada compiler.
Entry updated April 23, 2001.
CTB
Kind of Tool
C test harness generator, unit/integration testing environment
Organization
Testwell Oy
http://www.testwell.fi/
Software Description
CTB (C Test Bed System) generates test driver (and stubs, if needed),
which are compiled with the units under test resulting in a test bed
program. The test bed building can be incremental and arranged on
"as needed" basis using makefiles. Using C-like command language
the test bed facilitates specification-based (black-box) unit and
185 http://www.certmagic.com
BH0-003
integration testing. Both interactive and script-based tests are
supported. The work becomes automated, effective, intuitive, visible,
documented, standardized, measurable. Read more from
http://www.testwell.fi/ctbdesc.html
Platforms
Windows 2000/NT/9x, HPUX, Solaris, Linux.
Entry updated April 23, 2001.
OTF - An Object Testing Framework
Kind of Tool
Testing Framework for Smalltalk Objects
Organization
MCG Software, Inc.
320 NW Uptown Terrace, #1A
Portland, OR 97210
Phone: (503) 827-3933
FAX: (503) 827-3934
E-mail: info@mcgsoft.com
http://www.mcgsoft.com/
Software Description
OTF is an easy--to-use framework for the developing; editing, keeping,
sharing and running suites of tests for Smalltalk objects. Regression
testing is automatic with full logging of results. Tests may be looped
and conditional code executed for stress testing. While OTF focuses on
testing modeling objects, there is also a simple mechanism for testing
user interfaces. Extensions are easily added to OTF to expand OTF
functionality and tailor OTF to site requirements. OTF is available on all
the major Smalltalks.
Platforms
Windows, OS/2, & Unix via Visual Smalltalk; Visual Smalltalk
Enterprise; Visual Works; Visual Age
VectorCAST
Kind of Tool
Software Module Test System
Company
Vector Software, Inc.
1130 Ten Rod Road, E-307
186 http://www.certmagic.com
BH0-003
North Kingstown, RI 02852
Phone: 401-295-5855
Fax: 401-295-5856
E-mail: info@vectors.com
http://www.vectors.com/
Software Description
VectorCAST is a software module test system that automates the
testing of individual software components prior to system test.
Automation includes: building of complete test harnesses, generation
and execution of test cases, code coverage, and pass/fail reports.
VectorCAST is customized to the target CPU, cross compiler and run-
time environments for leading embedded development tools from:
Wind River Systems, Green Hills, TI, Rational, Cosmic, Tasking, Mentor
Graphics, Analog Devices, and MetaWare.
Platforms
IBM RS/6000(AIX), HP (HP/UX), Linux (Red Hat, Hard Hat & SuSE),
Sun (Solaris and SunOS), and Windows (95/NT/2000)
Entry updated January 13, 2003.
CTA++
Kind of tool
C++ test harnessing tool, unit/integration testing
Organization
Testwell Oy
Hermiankatu 8
FIN-33720 Tampere
Finland
Phone: +358-3-316-5464
FAX: +358-3-359-9660
E-mail: info@testwell.fi
http://www.testwell.fi
Software Description
CTA++ (C++ Test Aider) is a tool for unit testing C++ classes,
libraries and subsystems. CTA++ facilitates effective testing
characterized as: easy-to-use and powerful arrangement to model the
test suite into test cases, various forms of assertions for automating
the test result checking, clear PASS/FAIL reporting on test cases and
the whole test session, making the test runs visible, compact HTML
browsable reporting of test results, regression testing, reading actual
and expected values from command line or from compact textual data
files, support for stub functions, reusing test cases of base class when
testing inherited classes, testing multi-threaded code, testing all the
advanced features of C++ (inheritance, overloading, exceptions,
187 http://www.certmagic.com
BH0-003
private parts, etc.), and more. Read more from
http://www.testwell.fi/ctadesc.html
Platforms
Windows-2000/NT/9x, Solaris, HPUX, Linux
Entry updated April 23, 2001.
Test Mentor - Java Edition
Kind of Tool
Java component, unit and function test automation
Organization
SilverMark, Inc.
www.javatesting.com/
Software Description
A functional test and test modeling tool for Java developers & QA
Engineers to use as they develop their Java classes, clusters,
subsystems, frameworks, and other components, either deployed on
the client or the server during unit and i ntegration testing.
Platforms
Client (user-interface) runs on Windows platforms only, test execution
on al l Java platforms
Entry added January 8, 2001.
Aunit
Kind of Tool
Ada unit testing framework (freeware)
Organization
ACT Europe
8 rue de Milan
75009 Paris
France
Phone: +33 1 49 70 67 16
Fax: +33 1 49 70 05 52
E-mail: sales@act-europe.fr
http://libre.act-europe.fr/aunit/
Software Description
AUnit is a set of Ada packages based on the xUnit family of unit test
frameworks. It's intended as a developer's tool to facilitate confident
writing and evolution of Ada software. It is purposely lightweight, as
188 http://www.certmagic.com
BH0-003
one of its main goals is to make it easy to develop and run unit tests,
rather than to generate artifacts for process management. The
framework supports easy composition of sets of unit tests to provide
flexibility in determining what tests to run for a given purpose.
Platforms
Unix, Windows
Entry updated April 9, 2003.
Check
Kind of Tool
A unit test framework for C (freeware)
Organization
SourceForge
http://check.sourceforge.net/
Software Description
Check features a simple interface for defining unit tests, putting little
in the way of the developer. Tests are run in a separate address space,
so Check can catch both assertion failures and code errors that cause
segmentation faults or other signals. The output from unit tests can be
used within source code editors and IDEs.
Check was inspired by similar frameworks that currently exist for most
programming languages; the most famous example being JUnit for
Java.
Licensed under the GNU LGPL.
Platforms
POSIX-compliant systems.
Entry updated April 9, 2003.
CppUnit
Kind of Tool
C++ unit test tool (freeware)
Organization
SourceForge
http://cppunit.sourceforge.net/
189 http://www.certmagic.com
BH0-003
Software Description
CppUnit is the C++ port of the famous JUnit framework for unit
testing. Test output is in XML or text format for automatic testing and
GUI based for supervised tests.
Licensed under the GNU LGPL.
Platforms
BeOS, MacOS, Windows, Linux, possibly others.
Entry updated April 9, 2003.
csUnit
Kind of Tool
"Complete Solution Unit Testing" for Microsoft .NET (freeware)
Organization
csUnit.org
E-mail: info@csunit.org
http://www.csunit.org/
Software Description
csUnit is a unit testing framework for the Microsoft .NET Framework. It
targets test driven development using .NET languages such as C#,
Visual Basic .NET, Visual J# and managed C++.
Licensed under the GNU GPL.
Platforms
Microsoft Windows
Entry updated July 22, 2003.
cUnit
Kind of Tool
C unit testing framework (freeware)
Organization
Christian Holmboe
E-mail: spotty@codefactory.se
http://people.codefactory.se/~spotty/cunit/
Software Description
190 http://www.certmagic.com
BH0-003
cUnit is my first attempt to make a unit testing framework for the C
programming language. It helps a developer to organize and run
automated tests of his/her C source code. I wrote this so that I could
try the CodeUnitTestFirst approach while developing C programs. cUnit
is very simple and only has a few features:
• Test suites for organizing tests and test suites
• Unit test context management (setup/teardown)
• Convenient running and listing of tests and test suites
Platforms
Gnu/Linux
Entry updated April 9, 2003.
CUT
Kind of Tool
C Unit Test System (freeware)
Organization
SourceForge
http://cut.sourceforge.net/cgi-bin/piki
Software Description
CUT is a tool to help programmers develop ProgrammerTests for their
C-based software. CUT can be used to help test C, C++, Objective-C,
and depending on the run-time environment, even assembly language
software. CUT can be used with most operating systems, and is
entirely processor independent. CUT is ideal for embedded software
testing frameworks and full-blown desktop application testing alike.
Licensed under the zlib/libpng License.
Platforms
Linux, Windows
Entry updated April 9, 2003.
dotunit
Kind of Tool
Unit test framework for .net (freeware)
191 http://www.certmagic.com
BH0-003
Organization
Christian Sepulveda
E-mail: csepulv@atdesigntime.com
http://dotunit.sourceforge.net/
Software Description
dotunit is a port of JUnit to the Microsoft .net platform. This testing
framework allows for automated unit and functional tests which are
vital for refactoring and regression testing.
Distributed under the BSD license.
Platforms
Windows
Entry updated April 10, 2003.
EasyMock
Kind of Tool
A class library that provides an easy way to use mock objects for given
interfaces. (freeware)
Organization
OFFIS
http://www.easymock.org/
Software Description
Unit testing is the testing of software units in isolation. However, most
units do not work alone, but they collaborate with other units. To test
a unit in isolation, we have to simulate the collaborators in the test.
Mock Objects are test-oriented replacements for concrete
implementations. They are configured to simulate the object they
replace in a simple way. In contrast to stubs, they also verify whether
they were used correctly.
Made available under the terms of the MIT license.
Platforms
Platforms supporing Java and JUnit.
Entry updated April 10, 2003.
192 http://www.certmagic.com
BH0-003
GrandTestAuto
Kind of Tool
Unit test tool for Java (freeware)
Organization
Tim Lavers
http://grandtestauto.org/
Software Description
GrandTestAuto is a tool for unit testing applications written in Java.
GrandTestAuto automatically runs all unit tests for an application and
at the same time checks that the unit tests provide sufficient
coverage.
GrandTestAuto is free for use and modification under the terms of the
Wide Open License.
Platforms
Platforms supported by JDK1.4.
Entry updated April 11, 2003.
HtmlUnit
Kind of Tool
Java unit testing framework for web applications (freeware)
Organization
Gargoyle Software
Phone: 416-822-0973
Fax: 416-822-0975
http://htmlunit.sourceforge.net/
Software Description
HtmlUnit is a java unit testing framework for testing web based
applications. It is similar in concept to httpunit but is very different in
implementation. Which one is better for you depends on how you like
to write your tests. HttpUnit models the http protocol so you deal with
request and response objects. HtmlUnit on the other hand, models the
returned document so that you deal with pages and form and tables.
Released under an apache style license
Platforms
Platforms that run Java.
193 http://www.certmagic.com
BH0-003
Entry updated April 11, 2003.
HttpUnit
Kind of Tool
Java API for accessing web sites without a browser (freeware)
Organization
Russell Gold
E-mail: russgold@httpunit.org
http://httpunit.sourceforge.net/
Software Description
The normal way to access web sites is via a browser; however, there
are times when it is desirable to bypass the browser and access a site
from a program, including:
• automated web site testing
• using a web-site as part of a distributed application
HttpUnit makes this easy. Written in Java, HttpUnit emulates the
relevant portions of browser behavior, including form submission,
JavaScript, basic http authentication, cookies and automatic page
redirection, and allows Java test code to examine returned pages
either as text, an XML DOM, or containers of forms, tables, and links.
Distributed under the MIT license.
Platforms
Platforms that support Java.
Entry updated April 11, 2003.
JavaScript Assertion Unit (jserUnit)
Kind of Tool
JavaScript unit testing framework (freeware)
Organization
Daniel Fournier
E-mail: zenon48@users.sourceforge.net
http://jsertunit.sourceforge.net/
Software Description
JavaScript unit testing framework.
194 http://www.certmagic.com
BH0-003
Licensed under the GNU GPL.
Platforms
Platforms supporting Javascript.
Entry updated April 11, 2003.
JTestCase
Kind of Tool
Data-driven testing add-on for JUnit (freeware)
Organization
Yuqing Wang
http://jtestcase.sourceforge.net/
Software Description
JUnit test framework provides an excellent way to formalize your test
code. But due to its "none-input-param, none-return" design, generally
you need to hard-code all test data for each testing method. And for
each test cases of one unit test, you need to change test code,
recompile it, and run it.
JTestCase is an open-sourced project that helps you in separating test
data from test code. You can organize all your test cases of multiple
unit tests into one data file - an XML file, and bulk-load them into
memory via sets of easy-to-use APIs that JTestCase provides.
In a word, JTestCase provides a way for java unit tests to be test-
case-oriented and full-test-automatable.
Licensed under the Common Public License Version 0.5.
Platforms
Platforms supporting Java.
Entry updated April 11, 2003.
JUnitX
Kind of Tool
195 http://www.certmagic.com
BH0-003
JUnit extension module (freeware)
Organization
Extreme Java
E-mail: andreas.heilwagen@acm.org
http://www.extreme-java.de/junitx/
Software Description
Module for testing private and protected Java classes, methods and
variables.
Platforms
Platforms supported by Java.
Entry updated April 22, 2003.
LingoUnit
Kind of Tool
Unit testing framework for Macromedia Director. (freeware)
Organization
Koseki Kengo
E-mail: kengo@tt.rim.or.jp
http://sourceforge.net/projects/lingounit/
Software Description
LingoUnit is a Unit Testing framework for Macromedia Director. Most of
this framework is based on JUnit.
Platforms
Requires Macromedia Director 8.5.
Entry updated April 22, 2003.
MockMaker
Kind of Tool
A program for creating source code for mock object classes (freeware)
Organization
Matthew Cooke
E-mail: mpcooke3@hotmail.com
http://www.mockmaker.org/
Software Description
Given an interface, MockMaker writes the source code for a mock
object class that implements the interface and allows instances of that
class to have expectations set about how many times a method is
called, what parameters each method is called with, and to pre-set
196 http://www.certmagic.com
BH0-003
return values for methods. In many cases (possibly most cases), the
classes produced by MockMaker are exactly what you want a mock
class to do.
MockMaker is released under the MIT License.
Platforms
MockMaker has been tried on Windows 98 using Sun's JDK1.3, and
should work with other versions of Java 1.1.
Entry updated April 22, 2003.
Mock Objects
Kind of Tool
Framework for developing unit tests in the mock object style
(freeware)
Organization
Mock Objects
http://www.mockobjects.com/
Software Description
Unit testing is a fundamental practice in Extreme Programming, but
most nontrivial code is difficult to test in isolation. You need to make
sure that you test one feature at a time, and you want to be notified
as soon as any problem occurs. Normal unit testing is hard because
you are trying to test the code from outside.
We propose a technique called Mock Objects in which we replace
domain code with dummy implementations that both emulate real
functionality and enforce assertions about the behaviour of our code.
These Mock Objects are passed to the target domain code which they
test from inside, hence the term endo-testing. Unlike conventional
testing techniques, however, in endo-testing assertions are not left in
production code but gathered together in unit tests.
Distributed under The Apache Software License.
Platforms
Platforms supported by Java.
Entry updated April 22, 2003.
197 http://www.certmagic.com
BH0-003
Mockry
Kind of Tool
Mock Object generator for Java (freeware)
Organization
Montreal Extreme Programming Users Group
http://mockry.sourceforge.net/
Software Description
Mockry is a tool to create mockobjects. MockObjects are used to help
developers to automate their unit tests. This project has been initiated
by the Montreal's XP users group.
Distributed under the BSD License.
Platforms
Platforms supported by Java.
Entry updated April 22, 2003.
Mock Creator
Kind of Tool
Generates mock objects from Java Interfaces. (freeware)
Organization
abstrakt gmbh
Henriettenstrasse 73
20259 Hamburg
Germany
Phone: +49-40-398 04 630
E-mail: info@abstrakt.de
http://www.abstrakt.de/mockcreator.html
Software Description
Mock Creator was developed by Christian Junghans at abstrakt. It is
used to generate mock objects from Java Interfaces.
Distributed under the GNU GPL and the GNU LGPL.
Platforms
MockCreator is available for: Visual Age for Java 4.0, Eclipse 2.0,
Commandline (and other IDEs)
Entry updated April 22, 2003.
198 http://www.certmagic.com
BH0-003
NUnit
Kind of Tool
A unit-testing framework for all .Net languages. (freeware)
Organization
Nunit.org
http://www.nunit.org/
Software Description
An xUnit based unit testing tool for Microsoft .NET. It is written entirely
in C# and has been completely redesigned to take advantage of many
.NET language features, for example custom attributes and other
reflection related capabilities. NUnit brings xUnit to all .NET languages.
Distributed under the zlib/libpng License.
Platforms
Microsoft .NET
Entry updated April 22, 2003.
ObjcUnit
Kind of Tool
Unit testing framework for Objective-C on Mac OS X (freeware)
Organization
Oops AB
Gullbringa 130
SE-442 95 H lta
Sweden
Phone: +46 (0)303 24 79 00
Fax: +46 (0)303 24 79 09
E-mail: info@oops.se
http://oops.se/objcunit/
Software Description
Formerly called O2Unit, ObjcUnit is a unit testing framework for
Objective-C on Mac OS X. It's design was copied from JUnit, written by
Erich Gamma and Kent Beck, and then adapted somewhat for
Objective-C.
Platforms
Mac OS X
Entry updated April 22, 2003.
199 http://www.certmagic.com
BH0-003
OCUnit
Kind of Tool
Testing framework for Objective C (freeware)
Organization
Sen:te
Petit-Ch ne 18 ter
CH-1003 Lausanne
Switzerland
Phone: + 41 21 693 83 83
E-mail: feedback@sente.ch
http://www.sente.ch/software/ocunit/
Software Description
With OCUnit, testing becomes integrated with development. You can
test frameworks, bundles, or applications.
This Objective C testing framework is a based on SUnit, Kent Beck's
Smalltalk unit testing framework, also available for Java under the
name JUnit, and is distributed as open source.
Platforms
Mac OS X, Mac OS X Server, YellowBox/Cocoa and WebObjects
environments
Entry updated April 22, 2003.
PalmUnit
Kind of Tool
Unit testing framework for the Palm device (freeware)
Organization
Yutaka Kato
E-mail: ycatoo@eirene.dricas.com
http://www.geocities.co.jp/SiliconValley-
SanJose/7344/PalmUnit/README-en.html
Software Description
A port of CppUnit to the Palm platform.
Licensed under the GNU GPL.
Platforms
Palm
200 http://www.certmagic.com
BH0-003
Entry updated April 22, 2003.
PBUnit
Kind of Tool
Unit test framework for PowerBuilder (freeware)
Organization
John Urberg
E-mail: pbunit@yahoo.com
http://www.geocities.com/pbunit/
Software Description
PBUnit is PowerBuilder 8.0 port of JUnit 2.1 which was originally
implemented by Kent Beck and Erich Gamma.
Distributed under the Common Public License.
Platforms
PowerBuilder 8.0
Entry updated April 22, 2003.
PerlUnit
Kind of Tool
Unit test framework for Perl (freeware)
Organization
Adam Spiers
http://perlunit.sourceforge.net/
Software Description
This project has been set up to unify perl unit testing frameworks for
use in Extreme Programming.
Platforms
Platforms supported by Perl
Entry updated April 22, 2003.
phpAsserUnit
201 http://www.certmagic.com
BH0-003
Kind of Tool
Unit testing framework for PHP (freeware)
Organization
Daniel Fournier
http://jsertunit.sourceforge.net/docs/phpassertunit.html
Software Description
A unit testing framework based on jserUnit. It's a kind of wrapper of
the assertion methods found in jserUnit. So it provides the same
functionalities, but in a PHP development environment.
Platforms
Platforms supported by PHP.
Entry updated April 22, 2003.
PhpUnit
Kind of Tool
Testing framework for PHP (freeware)
Organization
Fred Yankowski
E-mail: fred@ontosys.com
http://phpunit.sourceforge.net/
Software Description
Provides a testing framework for PHP, similar to JUnit for Java.
Platforms
Platforms supported by PHP.
Entry updated April 22, 2003.
PyUnit
Kind of Tool
Unit testing framework for Python (freeware)
Organization
Steve Purcell
http://pyunit.sourceforge.net/
Software Description
This unit testing framework, dubbed 'PyUnit' by convention, is a
Python language version of JUnit. JUnit was written by smart cookies
Kent Beck and Erich Gamma, and is, in turn, a Java version of Kent's
Smalltalk testing framework.
202 http://www.certmagic.com
BH0-003
Provides a standard, proven, simple and elegant way to write unit
tests for Python software. GUI also provided.
Platforms
Platforms supported by Python.
Entry updated April 22, 2003.
QtUnit
Kind of Tool
Unit testing framework for C++ with Qt (freeware)
Organization
UWYN
Lambermontlaan 148
B-1030 Brussels
Belgium
Phone: +32 (0)2 245 41 06
http://www.uwyn.com/projects/qtunit/
Software Description
QtUnit is a unit testing framework for C++, originally based on
CppUnit 1.5 written by Michael Feathers. All code has been refactored
and ported to exclusively use Qt 3.x as it base library. This makes it
highly portable to all the platforms supported by Qt, without
compromising on the advanced features that are currently expected
from modern software (Qt is a trademark of Trolltech AS).
Licensed under the GNU GPL.
Platforms
Platforms supported by the Qt library.
Entry updated April 22, 2003.
Ruby Test::Unit
Kind of Tool
Unit testing framework for the Ruby programming language (freeware)
Organization
203 http://www.certmagic.com
BH0-003
Nathaniel Talbott
E-mail: testunit@talbott.ws
http://testunit.talbott.ws/
Software Description
A framework for unit testing in Ruby, helping you to design, debug and
evaluate your code by making it easy to write and have tests for it.
Platforms
Platforms supported by Ruby.
Entry updated April 22, 2003.
Ruby/Mock
Kind of Tool
Mock objects for Ruby (freeware)
Organization
Nat Pryce
E-mail: nat.pryce@b13media.com
http://www.b13media.com/dev/ruby/mock.html
Software Description
Ruby/Mock is a package that makes it easy to implement Mock Objects
in RubyUnit test cases. It provides a generic Mock class that can be
used to create mock objects for any interface. A test uses closures to
define the expectations of a mock object. Tests can also encapsulate
protocols into reusable mock-object classes.
Licensed under the GNU GPL.
Platforms
Platforms supported by Ruby.
Entry updated April 23, 2003.
SUnit
Kind of Tool
Unit testing framework for Smalltalk (freeware)
Organization
Camp Smalltalk
http://sunit.sourceforge.net/
204 http://www.certmagic.com
BH0-003
Software Description
SUnit is the mother of all xUnit testing frameworks, and serves as one
of the cornerstones of test-driven development methodologies such as
eXtreme Programming.
Platforms
Platforms supported by Smalltalk
Entry updated April 23, 2003.
TagUnit
Kind of Tool
Tag library for testing custom tags within JSP pages. (freeware)
Organization
Simon Brown
http://www.tagunit.org/
Software Description
In the same way that JUnit allows us to write unit tests for Java
classes, TagUnit allows us to unit test JSP custom tags, inside the
container.
TagUnit provides a way to perform assertions on the content that
custom tags generate and the side-effects that they have on the
environment such as the introduction of scoped
(request/page/session/application) attributes, cookies and so on. In
addition to this, assertions can be made on the constraints specified
within the tag library descriptor file that give us a way to verify the
contract that a tag provides.
TagUnit is distributed under a BSD style license.
Platforms
Java 2 Standard Edition, SDK 1.2 or above, JavaServer Pages (JSP)
1.2 compatible web or application server such as Jakarta Tomcat 4.0
and above
Entry updated April 23, 2003.
JUnitEE
Kind of Tool
205 http://www.certmagic.com
BH0-003
JUnit extension for running tests inside an application server
(freeware)
Organization
junitee.org
http://www.junitee.org/
Software Description
JUnitEE provides a TestRunner which outputs HTML and a servlet
which can be used as an entry point to your test cases. Building your
test harness as a standard J2EE web application means:
• Your tests are packaged conveniently into a .war file which can
easily be moved between servers; you can leave the .war file in
the main .ear file and simply avoid enabling the test web
application on the production server.
• Your test classes will be dynamically reloaded by the app server
(assuming your server supports this).
• Your test cases look just like your production code, and can use
the same beans (or whatever) you use as a facade for your
EJBs.
Platforms
Platforms supported by Java.
Entry updated April 23, 2003.
unit++
Kind of Tool
C++ unit testing framework (freeware)
Organization
Claus Dr by
http://unitpp.sourceforge.net/
Software Description
Unit++ is a C++ unit testing framework similar to junit, yet intended
to be more C++ like than CppUnit (the C++ port of junit).
Licensed under the GNU GPL.
Platforms
Platforms supported by C++. Seems to be developed primarily on
Linux.
Entry updated April 23, 2003.
206 http://www.certmagic.com
BH0-003
vbUnit3 Basic
Kind of Tool
Unit test tool for Visual Basic and COM objects. (freeware)
Organization
ShareIt!
Element 5 AG
Vogelsanger Str. 78
50823 Cologne
Germany
Phone: +49-221-31088-20
Fax: +49-221-31088-29
E-mail: support@shareit.com
http://www.vbunit.com/
Software Description
The free version of vbUnit3. Open source and free of charge. This
package includes:
• vbUnit3 Framework with complete source code and unit tests
• vbUnit2 TestRunner with source code
• full documentation including the vbUnit3 tutorial
• project and class templates
Platforms
vbUnit3 requires an existing installation of Visual Basic 6.0 with Visual
Studio Service Pack 5 to work correctly. vbUnit3 has been tested to
function correctly on the following operating systems: Windows 98
Second Edition, Windows NT4 with Service Pack 6, Windows 2000 with
Service Pack 2.
Entry updated April 23, 2003.
XMLUnit
Kind of Tool
JUnit-style testing for XML (freeware)
Organization
Tim Bacon
E-mail: timbacon@users.sourceforge.net
http://xmlunit.sourceforge.net/
Software Description
XMLUnit enables JUnit-style assertions to be made about the content
and structure of XML. It is an open source project that grew out of a
207 http://www.certmagic.com
BH0-003
need to test a system that generated and received custom XML
messages.
Platforms
Platforms supported by Java.
Entry updated April 23, 2003.
XSLTunit
Kind of Tool
Unit testing framework for XSLT transformations (freeware)
Organization
Eric van der Vlist
E-mail: vdv@dyomedea.com
http://xsltunit.org/
Software Description
The purpose of XSLTunit is to provide a unit testing framework for
XSLT transformations similar to the "unit" environments available for
other languages (i.e. Junit for Java).
Although not a general purpose programming language, XSLT is turing
complete and allows to develop powerful (and complex)
transformations that deserve unit testing.
Platforms
EXSLT compliant XSLT processor (such as Saxon, libxslt, 4xslt, Xalan-
J, ...)
Entry updated April 23, 2003.
JsUnit (Schaible)
Kind of Tool
JavaScript unit test framework (freeware)
Company
J rg Schaible
E-mail: joehni@mail.berlios.de
http://jsunit.berlios.de/
Software Description
JsUnit is a simple framework to write repeatable tests in JavaScript. It
is an instance of the xUnit architecture for unit testing frameworks.
208 http://www.certmagic.com
BH0-003
JsUnit is a port of JUnit 3.8.1 originally written by Erich Gamma and
Kent Beck. It covers the core system and the examples.
Licensed under the GNU GPL.
Platforms
ECMA implementations that support exceptions.
Perl Test::MockObject
Kind of Tool
Perl extension for emulating troublesome interfaces (freeware)
Organization
chromatic
E-mail: chromatic@wgz.org
http://search.cpan.org/~chromatic/Test-MockObject/
Software Description
Testing is a lot easier when you can control the entire environment.
With Test::MockObject, you can get a lot closer.
Test::MockObject allows you to create objects that conform to
particular interfaces with very little code. You don't have to
reimplement the behavior, just the input and the output.
Platforms
Platforms supported by Perl.
HarnessIt
Kind of Tool
Unit testing software for all aspects of .NET development
Organization
United Binary, LLC.
http://www.unittesting.com/
Software Description
HarnessIt is a unit testing software for the Microsoft.NET languages.
Designed from the ground up for the .NET Framework, HarnessIt takes
full advantage of language innovations provided by .NET to create a
simpler, more flexible unit testing framework.
Platforms
Windows (All), .NET Framework 1.0 and up
COMPUTE
209 http://www.certmagic.com
BH0-003
Kind of Tool
Tests Microsoft Component Technologies with writing a single line of
code
Company
iUnitTest
Unit 47
24-28 St Leonards Road
Windsor
Berkshire
SL4 3BB
United Kingdom
Phone: +44 1753 669 340
Fax: +44 1753 669 341
E-mail: sales@iunittest.co.uk
http://www.iunittest.co.uk/
Software Description
COMPUTE is a software tool for unit testing Microsoft COM, MTS,
COM+ and .NET components. Simply load your component into
COMPUTE, enter your test data and call your methods/transactions.
Test data can be compiled into test scripts and re-tested as often as
you like. Just think, no more test harnesses, constantly entering data
and recording the results. Its features include:
• No Coding Required!
• No More Test Harnesses!
• Simulate real life load, performance and stress testing with its
virtual client support.
• Consistent, easy to use User Interface for all your components.
• Supports Regression Testing and even migrates Test Data from
different versions of components.
• Test Results are automatically recorded and retrieved instantly.
• All Test Scripts, Test Data and Test Results is kept as XML.
• Automatically creates Documentation.
• Uses XSL to customise Documentation to your standards.
• Documentation conforms to to IEEE 1008-1987.
• Unit Test Process follows existing standards.
• Powerful Object and Parameter Referencing features.
• Drag and Drop .NET Component support.
• One Dimensional and Two Dimensional Array Support.
• Supports interactive debugging of your components while
COMPUTE executes your Test Script. Works with VB, VC++ and
VS.NET debuggers.
• Supports COM and .NET Collections (including ArrayList,
HashTable, Queue, SortedList, Stack and StringCollection).
Platforms
Windows NT, 2K and XP
210 http://www.certmagic.com
BH0-003
TBrun
Kind of Tool
Automated Unit Test tool
Organization
LDRA
http://www.ldra.co.uk/tbrun_moreinfo.asp
Software Description
TBrun utilises the powerful static and dynamic analysis facilities of the
LDRA tool suite to provide a sophisticated, fully automated, unit test
solution. TBrun automatically generates test harnesses for the unit
under test and, in so doing, saves time, frees up highly qualified staff,
increases test efficiency and improves motivation to test through a
repeatable, less error-prone process.
Features of TBrun include:
• Automatically generates test drivers/harnesses (wrapper code)
• Runs tests on code units
• Automatic test vector generation
• Detects changes in source code and documents changes
required in tests
• Performs regression tests
• Maintains test data and results
• Easy to use graphical interface
• Command line mode for automation
• Interactive testing with immediate result checking and coverage
metrics
• Detailed knowledge of code not needed
• Tests stored for easy regression
• Essential for C/C++, C#, Ada83, Ada95, Java & VB.net unit
testing
Platforms
Windows 9x/NT/2000/XP and Unix. Also available for any host / target
environment.
Cantata++
Kind of Tool
Test Driver, Code Coverage Analyzer and Static Metrics for C, C++,
and EC++
Company
IPL
http://www.ipl.com/tools
Software Description
211 http://www.certmagic.com
BH0-003
Cantata++ has been designed around the requirements of the C/C++
languages to produce a tool which allows developers to efficiently
perform unit and integration testing. The product offers high
productivity and a unique set of testing, coverage analysis and static
analysis features. Cantata++ is the market leader in its class, and has
been used in the development of high-integrity software worldwide.
Cantata++ V5 is due for release in September 2006. This version is
built on Eclipse and features many new productivity enhancers.
Major features:
• Unit and Integration testing: on both host and target platforms
• Integrated Coverage Analysis: statement, decision, MC/DC,
entry point and call-return metrics
• Full support for: ANSI C, ISO C++ and EC++
• GUI: Graphical results analysis and wizard-driven test
preparation
• Object Oriented: OO-aware testing and coverage analysis
• Wrapping and Stubbing: to simulate and control external
interfaces
• Static Analysis: code complexity and size metrics
• Automated test script generation with option to stub/wrap all
external calls for simulation
• Automated global data update/corruption checks
• Memory leak and buffer overflow detection (MemProbes)
Platforms
Windows, Unix, Linux and most target environments
AdaTEST 95
Kind of Tool
Test Driver, Code Coverage Analyzer and Static Metrics for Ada
Company
IPL
http://www.ipl.com/tools
Software Description
AdaTEST 95 has been designed around the requirements of the Ada
language to produce a tool which allows developers to efficiently
perform unit and integration testing. The product offers high
productivity and a unique set of testing, coverage analysis and static
analysis features. AdaTEST 95 is the market leader in its class, and
has been used in the development of high-integrity and safety-critical
software worldwide.
Major features:
212 http://www.certmagic.com
BH0-003
• Unit and Integration testing: on both host and target platforms
• Integrated Coverage Analysis: statement, decision, MC/DC,
entry
• point and call-return metrics
• Full support for: Ada 95, Ada 83 and several Ada subsets
• GUI: Graphical wizard-driven test preparation
• Object Oriented: OO-aware testing
• Stubbing: to simulate and control external interfaces
• Static Analysis: code complexity and size metrics
Platforms
Windows, Unix, Linux and most target environments
.TEST
Kind of Tool
Automatic Static Analysis & Unit Testing for .NET
Organization
ParaSoft Corporation
http://www.parasoft.com/
Software Description
Parasoft .TEST is an automated unit testing and static analysis tool for
the Microsoft(r) .NET Framework that analyzes assembly code of
tested assemblies and, based on gathered information, performs
different types of analysis to help prevent errors. It works with any
programming language that targets the .NET Framework, including
Microsoft Visual C# .NET, Visual Basic .NET, and managed Visual C++
.NET.
Platforms
TEST runs on the Windows NT, Windows 2000 or Windows XP
operating systems and works with Microsoft Visual Studio .NET.
SimpleTest
Kind of Tool
Unit tester and web tester for PHP (freeware)
Organization
Last Craft
http://www.lastcraft.com/simple_test.php
Software Description
Unit tester, mock objects and web tester for PHP. Similar to JUnit,
JWebUnit and JMock.
Platforms
Platforms suported by PHP
213 http://www.certmagic.com
BH0-003
DUnit
Kind of Tool
xUnit Testing Tool (freeware)
Organization
DUnit group http://dunit.sourceforge.net/
Software Description
DUnit is a port of the popular JUnit tool to Borland's Delphi (Object
Pascal).
Platforms
Windows, Linux
Entry updated August 14, 2003.
GJTester
Kind of Tool
General Java testing tool
Organization
TreborSoft
http://www.gjtester.com/
Software Description
GJTester provides a powerful GUI to aid developers in building test
cases and test scripts. It allows the testers to accomplish unit and
regression test without programming effort. The tool is useful for
testing CORBA, RMI and other server technologies as well.
Platforms
Platforms supported by Java.
Entry updated February 11, 2004.
CUnit
Kind of Tool
Unit testing framework for C (freeware)
Organization
Anil Kumar
E-mail: anilsaharan@users.sourceforge.net
http://cunit.sourceforget.net/
Software Description
Unit testing framework for C. It is based on the initial architecture of
CppUnit and supports various test runner interfaces like console,
curses and XML based.
Platforms
214 http://www.certmagic.com
BH0-003
Platform independent
Entry updated May 26, 2004.
Tessy
Kind of Tool
Automated Unit Testing
Organization
Hitex Development Tools GmbH
http://www.hitex.com/
Software Description
Tessy automates the unit testing of embedded software written in C.
Tessy provides code coverage and creates test documentation in
various formats. Tessy supports many cross compilers and debuggers
for embedded systems and thus is able to execute the tests on the real
hardware. Test case specification according to the Classification Tree
Method is supported by the Classification Tree Editor, which is included
in Tessy. Tessy is ideal for regression tests.
Platforms
Windows NT, 2000 and XP
Entry updated January 5, 2006.
TestGen4J
Kind of Tool
An automatic unit test generator for java (freeware)
Organization
SpikeSource, Inc
http://www.spikesource.com/projects/testgen4j/
Software Description
TestGen4J is a collection of open-source tools that automatically
generates unit test cases. The first released component of TestGen is
TestGen4J. TestGen4J automatically generates test cases from your
own Java class files, or source files. Its primary focus is to exercise
boundary value testing of the arguments passed to the method. It
uses a rules engine, with a user-configurable XML file, that defines
boundary conditions for the data types. The test code is separated
from test data with the help of JTestCase.
Platforms
All Java platforms
215 http://www.certmagic.com
BH0-003
Quality Assurance
An important part of defining the end-product of the project is the
specification of its quality related features - which the project must then aim
to deliver.
Quality has been an issue at the forefront of organizational concerns for the
past decade. The development of quality conscious industrial and business
practices has been identified as being of the utmost importance in gaining
and retaining a competitive edge.
In the context of a project that aims to deliver a complex end-product, the
quality aspects of that end-product will need to be planned, designed and
worked for.
Quality assurance is a global term used to incorporate the quality policy,
quality management and quality control functions which combine to assure
the client that the product will be consistently manufactured to the required
condition. Its aim is to achieve and assure quality through the adoption of a
cost effective quality control system and through external inspections and
audits.
Quality planning is an integral part of the planning activity. It manifests itself
in the product descriptions and in the scheduling of quality related activities
in the PERT charts. The results of the quality planning activities are reflected
in the resource and technical plans, at each level of the project.
Quality control is concerned with ensuring that the required qualities are built
into all of the products throughout their development life cycles. Quality
control utilizes measurable quality criteria and is exercised via quality
reviews, project reviews and by the testing of products.
Quality assurance requires agreement on the level of quality controls to be
adopted, both specifically relating to the project and to the overall
organizational policy. It is important that all three interests represented by
the project owner are taken into account when deciding the mechanisms to
be adopted.
Quality planning manifests itself in the product descriptions and in the
scheduling of quality related activities in the PERT charts. The results of the
quality planning activities are reflected in the resource and technical plans, at
each level of the project.
Quality control is concerned with ensuring that the required qualities are built
into all of the products throughout their development life cycles. It utilizes
216 http://www.certmagic.com
BH0-003
measurable quality criteria and is exercised via quality reviews, project
reviews and by the testing of products.
Quality Assurance – Quality Planning
The planning process should specifically address the quality issues raised by
a proposed project. Quality planning should have a significant effect on the
overall size and scope of the projects plans.
Quality planning is integral with general planning and it manifests itself in the
product descriptions, specifically in their quality criteria. In addition, it should
influence the scheduling of activities in the PERT chart. Quality related
activities should also be explicitly integrated into the resource and technical
plans.
Quality Assurance – Product Descriptions
The product description should describe the purpose, form and components
of a product. It should also list, or refer to, the quality criteria applicable to
that product.
Product descriptions should be created as part of the planning process, to
shadow the identification of the products that are required by the project.
Each product description may either apply to a specific item, or to all the
products of a given type.
The component products of a complex product may be described in separate
descriptions, giving rise to a hierarchy of product descriptions for that
product.
Quality Assurance – Quality Criteria
Quality criteria should be used to define the characteristics of a product in
terms that are quantifiable, and therefore allow it to be measured - at
various points in its development life cycle, if required.
The criteria effectively define 'quality' in the context of the product and are
used as a benchmark against which to measure the finished product. Quality
criteria should be detailed, or referred to, in the related product description.
If the criteria are common to an already defined product, or even a class of
products, it is sufficient to include a reference to the appropriate product
description(s).
Quality criteria should be established by considering what the important
characteristics of a product are in satisfying the need that it addresses, and
they should always be stated objectively. Subjective or descriptive criteria
such as 'quick response' or 'maintainable' are unsatisfactory - as they do not
permit meaningful measurement.
Quality Assurance – Incorporating Quality
217 http://www.certmagic.com
BH0-003
Quality planning should ensure that all quality related activities are planned
and incorporated into the projects PERT charts. The PERT chart is useful in
project planning, as it assists in estimating and scheduling the work required.
It contains detailed information on the duration of each activity and the
sequence in which they should be performed.
The tasks required to ensure the quality of the delivered products are often
overlooked, with the result that the PERT chart fails to represent quality
related work. This can have serious consequences for either the quality levels
achieved, the overall budget, or both.
The chart also clearly identifies the critical path, which is the
sequence of related activities which will take the longest time.
It is important that every task required to accomplish the project is included,
or accurate project planning will be jeopardized.
Quality planning should have a significant effect on the overall scope and size
of the projects plans. It should ensure that all quality related activities are
identified and incorporated into the PERT chart.
Product descriptions should be created as part of the planning process, to
shadow the identification of the products. Quality criteria should be used to
define the characteristics of a product objectively, so that it can undergo
meaningful inspection.
Quality Assurance – Quality Control
Quality control is concerned with ensuring that the required qualities are built
into all of the products throughout their development life cycles.
It defines the method of inspection, in-process inspection and final inspection
to determine if the product has met its quality specification.
Quality control utilizes measurable quality criteria and is exercised via change
control, quality reviews, project reviews and by the testing of products.
Quality control is exercised via change control mechanisms, quality reviews,
project reviews and by the testing of products.
The quality review is a formal procedure for inspecting a product, or group of
related products, against an agreed set of quality criteria. Quality reviews are
dealt with in detail in the next section.
The change control mechanism, is the formal system for raising any project
concern, this includes the highlighting of any product development problems
. Change control is also covered in detail later in this course.
Project reviews should be staged to monitor the actual progress against the
baselined plans and to review any current or potential problems. These
218 http://www.certmagic.com
BH0-003
meetings are usually held at regular intervals, for example weekly or
fortnightly and their aim is to disseminate information among project team
members.
In addition to the formal quality reviews associated with each product, a
number of products may require independent or integration testing. This is
particularly true of products of a technical nature - that may require testing
to ensure that they perform their required task to an acceptable standard.
There are four main mechanisms for exercising project quality control -
change control, quality reviews, project reviews and the testing of products.
These should be tailored to suit the needs of each individual project.
Change control is a formal mechanism that enables any member of the
project team to raise any concern, including the highlighting of any product
development problems.
Quality Assurance – Quality Reviews
Quality reviews are planned and documented inspections of a review item.
The review item may be a product, a group of related products or part of a
product. The scheduling of quality reviews is likely to represent a significant
effort in terms of required resources.
The earlier an error is identified the less costly are the implications and the
penalty for failing to conduct adequate reviews is illustrated by the graph
shown. The early detection of errors also minimizes the degree to which
faults in one product can be compounded - either in its later development or
in the development of related products.
The quality review is a formal procedure for reviewing a product, or group of
related Products, against an agreed set of quality criteria. The review
meeting should be run by a designated chairman and brings together the
reviewers and a designated presenter.
The chairman should be responsible for organizing the review, including the
initial selection of reviewers. They should set the agenda for and ensure the
smooth running of the review itself, striving to minimize deviation and
protracted discussion of potential solutions.
This role is normally undertaken by the lead developer of the product to be
reviewed. The presenter should ensure that all reviewers have the
information they require, ahead of the review meeting. The Presenter is
normally responsible for correcting errors found at the review and therefore
should record all the agreed points.
The main duty of the reviewers is to check the item under review against the
relevant criteria as defined in the product descriptions and other review
checklists. During the follow-up phase one or more reviewers should confirm
219 http://www.certmagic.com
BH0-003
that the corrective work was carried out and that no undesirable side effects
occurred.
Quality Assurance – The 3 Phases of Quality Reviews
Quality reviews should be seen as a three phase process - the first phase
being preparation, which precedes the actual review meeting. It is the
responsibility of the chairman and presenter to organize the quality review
and notify all those invited. Invitations and copies of the products being
reviewed should be issued, allowing sufficient time for each reviewer to
compile an error list - which they should bring to the review meeting.
In the preparation phase copies of the review item and relevant product
descriptions should be distributed to the reviewers, along with their
invitations to the review meeting.
Compiled independently by each reviewer during the preparation phase, error
lists detail any significant defects observed in the review item. Each reviewer
should inspect the review item from a particular viewpoint - for example, a
user or operator.
In this phase they should check the product for defects or omissions, using
the product descriptions and review checklists. Where minor errors or
typographical inaccuracies are identified the reviewers can note these directly
onto their copy of the product - thereby creating an annotated product.
Quality Assurance – The Quality Review Meeting
The central phase of the quality review process is the review meeting itself.
During the review meeting the emphasis should be on error detection, in line
with the criteria, and only limited discussion of corrective action should
occur. It is important that 'personalities' and 'politics' are kept out of the
review - in particular it is essential that the review remains focused on the
review item and not on the presenter.
For reviews that are 'complete' the action list tasks are allocated along with
designated reviewers, to verify that the work is done. If the review ends
abnormally it should be deemed 'incomplete'.
Error lists and copies of annotated products should be brought to the meeting
by each reviewer, or sent to the chairman before the meeting if the reviewer
is unable to attend.
If the review is deemed 'complete', then the chairman should forward the
follow-up action list and all copies of the annotated product to the presenter
together these represent all of the errors identified.
220 http://www.certmagic.com
BH0-003
The chairman should also distribute copies of the result notification to the
presenter and the reviewers. A project exception report should be raised for
any errors detected in non-review items.
Quality Assurance –Quality Review Meeting Follow-up
Following the quality review meeting there should be a follow-up period
during which the errors identified at the review that were committed to the
follow-up action list are rectified and signed off. If there is an urgent need for
the product, and the rework cannot be carried out in the allotted time, then a
project exception report should be generated, requiring the project managers
approval.
Following the quality review meeting the follow-up action list as agreed by
the reviewers and completed by the chairman should be forwarded along
with all copies of the annotated product to the presenter.
At the end of the follow-up period, which is typically restricted to one week,
the follow-up action list should be signed-off by the chairman.
A reviewer may be delegated to sign-off areas of remedial work, usually
those that they identified. Once it has been signed-off the re-worked product
should be resubmitted to the configuration management system.
Quality Assurance – Options Following the Quality Review Meeting
At the end of a review the chairman should designate it as 'complete' or
'incomplete'. The objective of the re-review is to ensure that the review item
receives adequate quality review, given that the initial review was designated
'incomplete'. The supporting documentation should be passed, along with the
chairmans recommendation, to the appropriate project manager who should
decide, as soon as possible, on subsequent action.
Re-work, this option is useful where the product was deemed inadequate, or
requires a second review to verify that all errors have been satisfactorily
resolved.
Reconvene, if key reviewers were unable to attend or the review ran out of
time this option enables the later continuation of the meeting. This may
delay the project if the reconvening moves the product completion date back.
Reorganize, this option is useful in situations where the project manager
wishes to change the composition of the review team, perhaps to include
people with more specialist skills.
Declare Complete, on reviewing the information the project manager may
decide that most, or all, of the errors have in fact been identified and that a
further review is not justified. This is effectively bypassing the normal rigors
of the quality control system and should only be considered in exceptional
221 http://www.certmagic.com
BH0-003
circumstances - for example, resource shortages, and within the context of a
non-critical product.
Abandon, in exceptional circumstances where the product under review is
urgently required the decision may be taken to abandon further review, using
the product 'as is'. Errors identified in the initial review will not be corrected
but noted on a project exception report recommending that a document
update request is used to highlight these deviations in the delivered product.
Quality assurance practices are a component of good management and are
essential to the achievement and demonstration of high quality in products
and operation.
Quality assurance practices cover:
• A detailed analysis of the objectives to be achieved;
• An analysis of the tasks to be performed;
• The identification of skills required;
• The selection and training of personnel;
• The use of appropriate equipment and procedures;
• The use of document control and record systems;
• The creation of a satisfactory working environment; and
• A recognition of individual responsibilities.
There are different means to achieve these practices.
Means to achieve quality assurance practices
Organizational arrangements for sound quality assurance practices are
requisite for all parties concerned, to provide a clear definition of the
responsibilities of component groups and channels of communication and co-
ordination between them. The extent and type of quality verification need to
reflect the safety significance and nature of the individual tasks. Such
verification methods include audits, checks, documents verification and
examinations to ensure that each task has been satisfactorily performed or
that any necessary corrective actions have been taken.
The basic responsibility for achieving quality remains with the performer of
the task, not the verifier.
Quality assurance documentary verification
An essential component of quality assurance is the documentary verification.
It verifies that tasks have been performed as required, deviations have been
identified and corrected, and action has been taken to prevent the recurrence
of errors.
The necessary facilities are provided for this, including a hierarchy of
documentation, quality control procedures which provide sampling of work
products, opportunity for observation of actual practices and witnessing of
tests and inspections, and sufficient staff and other resources.
222 http://www.certmagic.com
BH0-003
Software quality Assurance (SQA) is a planned and systematic approach to
ensure that software process and products conforms to the established
standards, processes, and procedures. The goals of SQA are to improve
software quality by appropriately monitoring both software and the
development process to ensure full compliance with the established
standards and procedures. The first step to establish an SQA programis to
get the top management's agreement on its goal. It then needs to identify
SQA issues, to write SQA plan, to establish standards and SQA functions, to
implement the SQA plan and to evaluate SQA program. For SQA to be
effective, they must have good people and full management support. High
quality software product must be able to run correctly and consistently, have
few defects (if there are), handle abnormal situation nicely, and need few
installation effort. The defects should not affect the normal use of the
softwre, will not do any destructive things to system, and rarely be evident
to the users. Before deciding what measures to use, it is essential to consider
the objectives of the measurement program. If the measures will be used to
manage software development, they should be objective, timely available
and controllable. On the other hand, if the measures are to support decisions
on product acceptance, they must reasonably represent user needs.
Definition
SQA is the planned and systematic set of activities that ensure that software
process and products conform to requirements, standards, and procedures.
"Processes" include all activities involved in designing, coding, testing and
maintaining; "products" include software, associated data, documentation,
and all supporting and reporting paperwork.[6]
The role of SQA is to give management the assurance that the officially
established process is actually being implemented. It ensures that:
1. An appropriate development methodology is in place.
2. The projects use standards and procedures in their work.
3. Reviews and audits are conducted.
4. Documentation is produced to support maintenance and enhancement.
5. Software Configuration Management is set up to control change.
6. Testing is performed and passed.
Any deficiencies and deviations are identified and brought to management's
attention.
Goals of SQA
The software development is a complex process full risks. There are technical
risks such as software will not perform as intended or be too hard to operate,
modify, and/or maintain; there are programmatic risks such as the project
will overrun cost or schedule. The goals of SQA is to reduce these risks by:
223 http://www.certmagic.com
BH0-003
• Appropriately monitoring the software and the development process.
• Ensuring full compliance with standards and procedures for software
and process.
• Ensuring that inadequacies in product, process, or standards are
brought to management's attention so that they can be fixed.
SQA is not responsible for producing quality products or for making quality
plans. They are responsible for auditing the quality actions and for alerting
management to any deviations.
Responsibilities of SQA
To achieve its goals, SQA is responsible for:
1. Review all development and quality plans for completeness.
2. Participate as inspection moderators in design and code inspections.
3. Review all test plans for adherence to standards.
4. Review samples of all test results to determine adherence to plans.
5. Periodically audit SCM performance to determine adherence to
standards.
6. Participate in all project phase reviews and write down any
nonconformance.
Establishing the SQA Program
To make SQA program successful, the most important step is to get senior
management commit to it. The senior managers must first agree on SQA
goals, and agree to resolve major SQA issues between SQA and line
management. Without management support, SQA can not be effective.
Creating SQA Plan
An effective SQA program requires forward planning and following through it.
The SQA plan specifies its goals, tasks to be performed, and the standards
and procedures against which the development work is to be measured.
The IEEE standard for SQA plan preparation contains the following:
1. Purpose
2. Reference Documents
3. Management
4. Documentation
5. Standards, Practices, and Conventions
6. Reviews and Audits
7. Software Configuration Management
224 http://www.certmagic.com
BH0-003
8. Problem Reporting and Corrective Action
9. Tools, Techniques, and Methodologies
10.Code Control
11.Media Control
12.Supplier Control
13.Records Collection, Maintenance, and Retention
Many of these topics are relatively clear from their headings, the
documentation, standards sections are given more explaination in the
following part. The SQA plan section on reviews and audits should describe
both the technical and the managerial reviews and audits to be conducted.
Documentation
The documentation section should describe the documentation to be
produced and how it is to be previewed. These include:
1. Software requirement specification, which specifies each software
function, performance parameter, interface, or other attribute with
sufficient precision to permit its verification.
2. Software Design Description, which describes the major components,
databases, and internal interfaces.
3. Software Verification and Validation Plan, which describes the methods
used to verify that the requirements are implemented in the design,
that the design is implemented in the code, and that the code meets
the requirements.
4. Software Verification and Validation Report, which is used to report on
the SQA verification and validation activities.
5. User Documentation, which is required for installation, operation, and
maintenance of the software.
6. Other, includes software development plan, the software configuration
management plan, the standards and procedures manual, together
with the planned review methods.
Standards and Procedures
The standards are the criteria to which software products are compared.
Procedures are the criteria to which development and control processes are
compared. They are closely related. The standards define what should be
done; while procedures define how the work is actually to be done, by whom,
when and what is done with the results.[6]
The minimum requirement for standards include:
1. Documentation Standards specify form and content for planning,
control, and product documentation and provide consistency
throughout a project.
225 http://www.certmagic.com
BH0-003
2. Design Standards specify the form and content of the design product.
They provide rules and methods for translating the software
requirements into the software design and for representing it in the
design documentation.
3. Code Standards specify the language in which the code is to be written
and define any restrictions on use of language features. They define
legal language structure, style conventions, rules for data structures,
and internal code documentation.
SQA Activities
SQA activities include product evaluation and process monitoring, which
ensure the product and the process used in development are correctly carried
out and standards are followed. SQA audit, another SQA activity, is a key
technique used to perform product evaluation and process monitoring.
Production evaluation is to ensure that standards are followed. It assures
that clear and achievable standards exist and evaluate compliance of
software product with the standards.
Process monitoring is to ensure that appropriate steps to carry out process
are being followed. SQA monitors processes by comparing actual steps
performed with established procedures.
Audit is a fundamental SQA technique. It looks at a process or product in
depth, comparing them with established standards and procedures.
Tailoring SQA to Project
Each project has its specific attributes and SQA program should be tailored to
accommodate to the project needs. The characteristics that should be
considered are: mission critical of project, schedule and budget, size and
complexity of project, size and complexity of project staff organization.
The relationship between SQA program and mission critical level is very
straight forward. The more critical the project, the more important and
formal the SQA should be. The relationship between SQA and budget and
schedule is not so intuitive; the tighter the budget and schedule, the more
critical it is to have a well planned and effective SQA program. This does not
mean that SQA program for project with more resources can be lax.
The project organization structure also influence the SQA program. For large
or dispersed staff, more formal SQA program is required. A small, centralized
development staff can easily inform each other the nonconformance and
helping each other in meeting standards, less formal SQA effort is required.
Other Considerations
226 http://www.certmagic.com
BH0-003
The hardest problems the software managers face is to get good people into
SQA. Good engineers want to stay in the development group. Rotation
scheme may be effective, but generally software development is transferring
their poorer performers to SQA and not taking them back. To hire new ones
outside into SQA only when there are enough experienced people there
already.
Without experienced people staffed in SQA, it can not be effective even with
senior managers support.
Measuring Software Quality
Software quality is a key measure of the software process. It provide a
clear record of development progress, a basis for setting objectives,
and a framework for current action.
Quality Definition
ANSI(American National Standard Institution) defines quality as the
totality of features and characteristics of a product that bears on its
ability to satisfy given needs.
Then what is software quality? Suppose you receive a software product
that is delivered on time, within budget, and which correctly and
efficiently performs all its specified functions. Does it mean a high
quality software? For several reasons, the answer may be "no". Here
are some problems you may find:
• The software may be hard to understand and difficult to modify.
This leads to excessive costs in software maintenance. Software
maintenance cost may be very expensive, for example, General
Motors spends 75% of their software effort in software
maintenance.
• The software may be difficult to use, or easy to misuse.
Software quality is defined as the degree to which software conforms
to quality criteria. Quality criteria include but are not limited to:
Economy Correctness Resilience
Integrity Reliability Usability
Documentation Modifiability Clarity
Understandability Validity Maintainability
Flexibility Generality Portability
Interoperability Testability Efficiency
227 http://www.certmagic.com
BH0-003
Modularity Resuability
Because there are so many criteria, we can not use them all for
measuring software quality. But some of desired characteristics can be
used in software product standard, guiding actual development work
to be done toward the right direction.
To be a high quality software, the product must be able to run
correctly and consistently, have few defects (if there are), handle
abnormal situation nicely, and need few installation effort. The defects
should not affect the normal use of the softwre, will not do any
destructive things to system, and rarely be evident to the users.
Measuring Software Quality
Software product quality is a key measure of software process. It
provides a clear record of development progress, a basis for setting
objectives, and a framework for current action. Software quality can
be measured in many ways. There are many models to measure
quality, but numerical quality measures are preferred.
Classes of Quality Measures
When we measure software quality, the measures generally fall into
following categories:
• Development: Defects found, change activities
• Product: Defects found, software structure, documentation
structure, controlled test
• Acceptance: Problems, effort to install, effort to use
• Usage:Problems, availability, effort to install, effort to use, user
opinions
• Repair: Defects, Repair effort
As shown in table 1, these measures can be characterized as
following:
• Objective: Can the measure be got objectively, i.e., it can be
repeatedly produced by different people?
• Timely:Is it available in time to be used in development or
maintenance process?
• Available: How hard to get the measure?
• Representative: To what extent does it represents customer's
view of quality software?
228 http://www.certmagic.com
BH0-003
• Controllable: To what extent can its valuable be changed by
development or maintenance group?
Objective Timely Available Representative Controllable
Development
Defects yes yes yes moderate yes
Change
yes yes yes poor no
activity
Product
Error seeding moderate yes difficult doubtful moderate
Software
depends yes moderate doubtful yes
structure
Controlled
moderate yes difficult good yes
tests
Acceptance
Problems no late yes good moderate
Install effort moderate late difficult good yes
Usage
Problems no late yes good moderate
Operating
moderate late difficult good yes
effort
Surveys no late difficult very good no
Availability yes late moderate very good yes
Repair
Defects yes late yes moderate yes
Repair effort moderate late moderate moderate yes
Table 1. Classes of Quality Measures
It is preferable that the measures are objective, readily available, easy
to get, represent customers' needs very well, and can be directly
controlled by development or maintenance group. But unfortunately no
single measure satisfy all these criteria.
Before deciding what measures to use, it is essential to consider the
objectives of the measurement program. If the measures will be used
to manage software development, they should be objective, timely
available and controllable. On the other hand, if the measures are to
support decisions on product acceptance, they must reasonably
represent user needs.
229 http://www.certmagic.com
BH0-003
FURPS+ Model
FURPS+ model is used to identify most important attributes of a
product and define them in measurable terms. It can help developers
establish their priorities to achieve better customer satisfaction. The
FURPS+ stands for functionality, usability, reliability, performance, and
supportability.[3] Components of FURPS+ are show as below:
Feature Set
Functionality Capabilities
Generality
Security
Human Factors
Usability Aesthetics
Consistency
Documentation
Frequency/Severity of Failure
Recoverability
Reliability Predictability
Accuracy
Mean Time to Failure
Speed
Efficiency
Performance Resoure Consumption
Throughput
Response Time
Teastability
Extensibility
Adaptability
Maintainability
Supportability Computability
Configurability
230 http://www.certmagic.com
BH0-003
Serviceability
Installability
Localization
Through customer surveys and interviews, the important software product
attributes that customer needs are identified and given a weight range from
1 to 5. For example, if customer thinks that the system should be of high
security, then the "security" in functionality is weighted 5; and the rating for
"security" in the implemented system can be from 0 to 5.
231 http://www.certmagic.com
BH0-003
232 http://www.certmagic.com
BH0-003
Figure 1. Example of FURPS+ Model
Defect Measures
While its generally true that programs with large number of defects
have poor quality and customer satisfaction, once a base level quality
is reached, defect measures not longer predict customer satisfaction.
To a certain pointer, users care more about usability, performance,
functionality.
Change Activity
Change activity can be a useful measure of development quality. When
change activity remains high late in the development process, it
usually indicates that there are some quality problems, such as
unstable requirements, or high defect rates. But development group
usually has no direct control over such factors.
Error Seeding
The idea of error seeding is to inject a known number of "dummy"
defects into program and check how many of them are found by
various inspections and testing. If, for example, 60% of them are
found, the presumption is that 60% of other defects have been found
as well.
Problem Measures
The number of problems reported by customers after software product
installation. This measure have moderate correlation with customer
satisfaction because these problems waste customers' time and give
them inconvenience. But the shortcomings of this measure is that data
is only available after development is over and some of problems may
be caused by lack of comprehension, simple human mistake.
Reliability and Availability
Availability means the ability to perform intended function whenever
needed. This measure is particularly important for system and
communication programs that are fundamental to overall system
operation.
It can not be measured directly but must be calculated from such
probabilistic measures as mean time between failure(MTBF) and mean
time required to repair and restore the system full operation(MTTR).
Availability = (1 - MTTR / (MTTR+MTBF)) 100%
233 http://www.certmagic.com
BH0-003
This is a useful measure for some system, but is very difficult to get
needed before operational testing.
Defect Removal Efficiency
Defect removal efficiency is defined as the reduction of defects that
are present at the beginning of a removal operation by certain
percentage. Cumulative defect removal efficiency is the percent of
defect that have been removed by a series of defect removal
operations based on previously injected defects.
Figure 2. Example of Defect Removal Efficiency
The higher the cumulative removal efficiency, the less defects will be
left on delivered products. To determine the actual removal
efficiencies, each defect must be evaluated to determine the phase
where it was injected.
Selecting Quality Measures
Quality measure should be chose based on its intended needs; while
another important consideration is the kind of data available. Because
defect data is all that can be obtained before shipment, it should be
used. Other steps, such as customer survey and interview, involving
users in early design and testing, can also be conducted to address
those attributes concerning software quality.
With numerical quality measures, management can set up quality
goals for the company to improve product quality. The goals are set to
motivate actions toward the goals. The goal can not be set too low, it
should be aggressive enough that demands extraordinary efforts to
achieve the goal. Each time the goal is achieved, higher goals are set
so the quality will continuously get improved. This is what quality
program try to achieve.
Software assurance is the planned and systematic set of activities that
ensures that software processes and products conform to
234 http://www.certmagic.com
BH0-003
requirements, standards, and procedures. "Processes" include all of
the activities involved in designing, developing, enhancing, and
maintaining software; "products" include the software, associated
data, its documentation, and all supporting and reporting paperwork.
The three mutually supportive activities involved in the software life
cycle are management, engineering, and assurance. Software
management is the set of activities involved in planning, controlling,
and directing the software project. Software engineering is the set of
activities that analyzes requirements, develops designs, writes code,
and structures databases. Software assurance makes sure the
management and engineering efforts put forth result in a product that
meets all of its requirements.
B. Goals of Software Assurance
Software development, like any complex development activity, is a
process full of risks. The risks are both technical and programmatic;
that is, risks that the software will not perform as intended or will be
too difficult to operate, modify, or maintain are technical risks, while
risks that
the project will overrun cost or schedule are programmatic risks. The
goal of software assurance is to reduce these risks. For example,
coding standards are set to specify a minimum quality of code. If no
standards are set, there exists some risk that the code will not come
up to a minimum
usable standard, and that the code will require rework. If standards
are set but there is no explicit process for assuring that all code meets
the standards, then there is some risk that some coders will produce
code that does not meet the standards. The assurance process
involved is
quality assurance, and to have no quality assurance activity is to
increase the risk that unacceptable code will be produced.
Similarly, the lack of a nonconformance reporting and corrective action
system increases the risk that problems in the software will be
forgotten and not corrected, or that important problems will not get
priority attention. Other risk-related examples can be provided to
support all of the
activities in this guidebook. The point is that software assurance
activities can help to reduce risks.
Every software development, enhancement, or maintenance project
includes some assurance activities. The types, amount, and formality
of such activities are decisions of the project manager, based on an
assessment of the project, its risks, and its development and
operational environments.
235 http://www.certmagic.com
BH0-003
Even a simple, one person development job has assurance activities
embedded in it, even if the programmer denies that "quality
assurance" plays any part in what is to be done. Each programmer
has some idea of how code should be written, and this idea functions
as a coding standard for that programmer. Likewise, each of us has
some idea of how documentation should be written, and this is a
personal documentation standard. Each programmer reviews his/her
products to make sure they meet their internal standards, and this is
an assurance review or audit. Each programmer tests and inspects
his/her own work, and these are verification and validation processes.
The list could go on, but the idea should be clear. A project software
assurance program involves the processes that each programmer goes
through, but requires the planning and formal establishment of
project, rather than personal, standards and processes.
Tailoring Software Assurance to the Project
Specific project characteristics and risks influence assurance needs,
and assurance planning should be tailored to reflect this fact.
Characteristics that should be considered include safety and mission
criticality of the software, schedule and budget, size and complexity of
the
product to be produced, and size and organizational complexity of the
development staff.
The relationship of criticality to assurance is as one would expect: the
more critical the software, the more important and formal the software
assurance effort must be. The relationship of schedule and budget is
not intuitive, however; the tighter the budget and schedule, the more
critical it is to have a well planned and effective assurance effort. This
does not mean that projects with more resources can afford to be lax,
it just means that tight resources increase risks that should be offset
by a strong assurance program.
The projected size of the software to be produced influences the level
of assurance required. A large project requires explicit and detailed
standards for all of the products in order to get at least a minimum
standard of quality from the varied ideas and experience of many
different programmers.
In addition, a large project requires significant efforts in testing and
other verification activities, which have to be planned and the plans
followed. In short, just due to the size of the activity, a significant and
formal assurance program must be established or risks of poor quality
products must be accepted. On the other hand, a small project may
require little formal assurance, and on a very small one, the assurance
efforts may be left to the programmer involved if adequate, informal
planning is done.
236 http://www.certmagic.com
BH0-003
Another factor that influences assurance planning is the project's
organizational structure. A small, centralized development staff can
easily participate in reviews and inspections, keep each other informed
on the status of nonconformances, and help each other in meeting
coding and
documentation standards. A large or dispersed staff will have many
different ideas of the best ways of doing things and many more
difficulties in communicating them. In the latter case, a more formal
assurance program and a larger assurance effort will be needed.
A last but very important characteristic is the difference between the
requirements of a software providing organization and a software
acquiring organization. A software provider actually develops the
products by developing designs and writing code, etc., and therefore
needs a full assurance program. An acquirer does not develop
software and thus can limit its assurance activities to those that
ensure that the provider is adhering to agreed- to methods and
standards and producing the agreed-to products.
Creating the Software Assurance Plan
An effective assurance program requires planning and follow through;
it cannot simply evolve along with the project. Adequate assurance
planning ensures that the assurance activities are focused on the
quality requirements and risks associated with the specific project.
The purpose of creating a software assurance plan is to
document/specify the conduct of the activities that will comprise
software assurance for a specific project. Armed with information
about the project and the available software assurance resources, the
project manager is ready to develop the plan. A useful guide for
documenting assurance plans is provided in the assurance sections of
the SMAP Management Plan Documentation Standard. In addition, the
following should be considered:
Plan software assurance in conjunction with management and
engineering planning, i.e., during the project concept and initiation
phase.
Phase assurance activities properly. For example, design standards
must be produced well before design is to be done.
Complete tool development or procurement before the tools are
needed. Especially important is the development of test tools and
test data sources.
Project Structure Considerations
237 http://www.certmagic.com
BH0-003
In planning and establishing a software assurance program, one
consideration is the software project organization and the location in
that organization of the assurance activities. Experience has indicated,
both in hardware and software, that some assurance functions are
best done by organizational entities that are separate from the ones
doing engineering activities. Software Quality Assurance (SQA) is one
activity that should be organizationally
separated from the producing organizations. Administratively, the SQA
organization should report no lower than the project manager; indeed,
many large successful software producing organizations have the SQA
organization report administratively to top corporate
management and interface with the project manager. The reason for
this separation of function is that the SQA organization is
management's arm that assures that standards are met and that
procedures are followed. If SQA is not independent of the
development activity, clear and impartial assessment will be difficult.
In addition, many organizations have had success using an
independent test team, or at least an independent test development
team. The team is responsible for developing test plans, procedures,
and test cases for formal acceptance tests. Independence is required
because these tests should be requirements driven and not influenced
by the design structure and coding details.
Completion Criteria
Because of the nature of software, it is difficult to ascertain the status
of a development or maintenance activity. It is important, therefore,
to define criteria for the completion of specific development stages.
For example, during the implementation phase, one has to do the
lowest level detailed design of small program elements, code the
elements, and unit test them. When a significant number of program
elements are involved, it is difficult for anyone to ascertain the status
of the units without specific completion criteria. For example, if there
is a criterion that detailed design is complete only after the rework
that finishes a design inspection, then the design can be said to be
either complete or incomplete depending on the status of the rework.
The setting of completion criteria is a management activity, but the
audit of records is an SQA activity. The accuracy of the reported
status can then be determined. This is important to both providers
and acquirers of software, and this "status auditing" is an important
SQA function.
Implementation of the Software Assurance Plan
238 http://www.certmagic.com
BH0-003
Once the project needs have been determined and the software
assurance planning is complete, the plan must be implemented.
Qualified, trained staff must be obtained, and special training must be
made available where needed. If standards and procedures are not
available for reuse on this project, they must be written. Staff must
be trained in the standards and procedures, since merely writing them
down does not guarantee compliance. All of the above are
management activities, but the assurance staff is a resource to help
complete them.
Staff devoted purely to assurance activities is usually small compared
to the project staff. On the other hand, it is important to have people
with specific assurance responsibilities, even if they must be shared
organizationally with other duties. Too often the truism that "quality is
everybody's business" becomes "quality is nobody's business" if
specific responsibilities are not
assigned.
SOFTWARE QUALITY ASSURANCE
A. Concepts and Definitions
Software Quality Assurance (SQA) is defined as a planned and
systematic approach to the evaluation of the quality of and adherence
to software product standards, processes, and procedures. SQA
includes the process of assuring that standards and procedures are
established and are followed throughout the software acquisition life
cycle. Compliance with agreed-upon standards and procedures is
evaluated through process monitoring, product evaluation, and audits.
Software development and control processes should include
quality assurance approval points, where an SQA evaluation of the
product may be done in relation to the applicable standards.
B. Standards and Procedures
Establishing standards and procedures for software development is
critical, since these provide the framework from which the software
evolves. Standards are the established criteria to which the software
products are compared. Procedures are the established criteria to
which
the development and control processes are compared. Standards and
procedures establish the prescribed methods for developing software;
the SQA role is to ensure their existence and adequacy. Proper
documentation of standards and procedures is necessary since the
SQA activities of process monitoring, product evaluation, and auditing
rely upon unequivocal definitions to measure project compliance.
239 http://www.certmagic.com
BH0-003
Software Quality Assurance Activities
Product evaluation and process monitoring are the SQA activities that
assure the software development and control processes described in
the project's Management Plan are correctly carried out and that the
project's procedures and standards are followed. Products are
monitored for conformance to standards and processes are monitored
for conformance to procedures. Audits are a key technique used to
perform product evaluation and process monitoring. Review of the
Management Plan should ensure that appropriate SQA approval points
are built into these processes.
Product evaluation is an SQA activity that assures standards are being
followed. Ideally, the first products monitored by SQA should be the
project's standards and procedures. SQA assures that clear and
achievable standards exist and then evaluates compliance of the
software product to the established standards. Product evaluation
assures that the software product reflects the requirements of the
applicable standard(s) as identified in the Management Plan.
Process monitoring is an SQA activity that ensures that appropriate
steps to carry out the process are being followed. SQA monitors
processes by comparing the actual steps carried out with those in the
documented procedures. The Assurance section of the Management
Plan specifies the methods to be used by the SQA process monitoring
activity.
A fundamental SQA technique is the audit, which looks at a process
and/or a product in depth, comparing them to established procedures
and standards. Audits are used to review management, technical, and
assurance processes to provide an indication of the quality and status
of the software product.
The purpose of an SQA audit is to assure that proper control
procedures are being followed, that required documentation is
maintained, and that the developer's status reports accurately reflect
the status of the activity. The SQA product is an audit report to
management consisting of
findings and recommendations to bring the development into
conformance with standards and/or procedures.
SQA Relationships to Other Assurance Activities
Some of the more important relationships of SQA to other
management and assurance activities are described below.
1. Configuration Management Monitoring
240 http://www.certmagic.com
BH0-003
SQA assures that software Configuration Management (CM) activities
are performed in accordance with the CM plans, standards, and
procedures. SQA reviews the CM plans for
compliance with software CM policies and requirements and provides
follow-up for nonconformances. SQA audits the CM functions for
adherence to standards and procedures and prepares reports of its
findings.
The CM activities monitored and audited by SQA include baseline
control, configuration identification, configuration control, configuration
status accounting, and configuration authentication. SQA also
monitors and audits the software library. SQA assures that:
Baselines are established and consistently maintained for use in
subsequent baseline development and control.
Software configuration identification is consistent and accurate with
respect to the numbering or naming of computer programs, software
modules, software units, and associated software documents.
Configuration control is maintained such that the software
configuration used in critical phases of
testing, acceptance, and delivery is compatible with the associated
documentation.
Configuration status accounting is performed accurately including the
recording and reporting of data reflecting the software's configuration
identification, proposed changes to the configuration identification,
and the implementation status of approved changes.
Software configuration authentication is established by a series of
configuration reviews and audits that exhibit the performance
required by the software requirements specification and the
configuration of the software is accurately reflected in the software
design documents.
Software development libraries provide for proper handling of software
code, documentation, media, and related data in their various forms
and versions from the time of their initial approval or acceptance until
they have been incorporated into the final media.
Approved changes to baselined software are made properly and
consistently in all products, and no unauthorized changes are made.
2. Verification and Validation Monitoring
241 http://www.certmagic.com
BH0-003
SQA assures Verification and Validation (V&V) activities by monitoring
technical reviews, inspections, and walkthroughs. The SQA role in
formal testing is described in the next
section. The SQA role in reviews, inspections, and walkthroughs is to
observe, participate as needed, and verify that they were properly
conducted and documented. SQA also ensures that any actions
required are assigned, documented, scheduled, and updated.
Formal software reviews should be conducted at the end of each phase
of the life cycle to identify problems and determine whether the
interim product meets all applicable requirements.
Examples of formal reviews are the Preliminary Design Review (PDR),
Critical Design Review
(CDR), and Test Readiness Review (TRR). A review looks at the
overall picture of the product being developed to see if it satisfies its
requirements. Reviews are part of the
development process, designed to provide a ready/not-ready decision
to begin the next phase. In formal reviews, actual work done is
compared with established standards. SQA's main objective in reviews
is to assure that the Management and Development Plans have been
followed, and that the product is ready to proceed with the next phase
of development. Although the decision to proceed is a management
decision, SQA is responsible for advising management and
participating in the decision.
An inspection or walkthrough is a detailed examination of a product on
a step-by-step or line-of-code by line-of-code basis to find errors. For
inspections and walkthroughs, SQA assures, at a minimum, that the
process is properly completed and that needed follow-up is done. The
inspection process may be used to measure compliance to standards.
Formal Test Monitoring
SQA assures that formal software testing, such as acceptance testing,
is done in accordance with plans and procedures. SQA reviews testing
documentation for completeness and adherence to standards. The
documentation review includes test plans, test specifications, test
procedures, and test reports. SQA monitors testing and provides
follow-up on nonconformances. By test monitoring, SQA assures
software completeness and readiness for delivery.
The objectives of SQA in monitoring formal software testing are to
assure that:
The test procedures are testing the software requirements in
accordance with test plans.
The test procedures are verifiable.
242 http://www.certmagic.com
BH0-003
The correct or "advertised" version of the software is being tested (by
SQA monitoring of the CM activity).
The test procedures are followed.
Nonconformances occurring during testing (that is, any incident not
expected in the test procedures) are noted and recorded.
Test reports are accurate and complete.
Regression testing is conducted to assure nonconformances have been
corrected.
Resolution of all nonconformances takes place prior to delivery.
Software testing verifies that the software meets its requirements.
The quality of testing is assured by verifying that project requirements
are satisfied and that the testing process is in accordance with the test
plans and procedures.
Software Quality Assurance During the Software
Acquisition Life Cycle
In addition to the general activities described in subsections C and D,
there are phase-specific SQA activities that should be conducted
during the Software Acquisition Life Cycle. At the conclusion of each
phase, SQA concurrence is a key element in the management decision
to initiate the following life cycle phase. Suggested activities for each
phase are described below.
1. Software Concept and Initiation Phase
SQA should be involved in both writing and reviewing the Management
Plan in order to assure that the processes, procedures, and standards
identified in the plan are appropriate, clear, specific, and auditable.
During this phase, SQA also provides the QA section of the
Management Plan.
2. Software Requirements Phase
During the software requirements phase, SQA assures that software
requirements are complete, testable, and properly expressed as
functional, performance, and interface requirements.
3. Software Architectural (Preliminary) Design Phase
243 http://www.certmagic.com
BH0-003
SQA activities during the architectural (preliminary) design phase
include:
Assuring adherence to approved design standards as designated in the
Management Plan.
Assuring all software requirements are allocated to software
components.
Assuring that a testing verification matrix exists and is kept up to date.
Assuring the Interface Control Documents are in agreement with the
standard in form and content.
Reviewing PDR documentation and assuring that all action items are
resolved.
Assuring the approved design is placed under configuration
management.
4. Software Detailed Design Phase
SQA activities during the detailed design phase include:
Assuring that approved design standards are followed.
Assuring that allocated modules are included in the detailed design.
Assuring that results of design inspections are included in the design.
Reviewing CDR documentation and assuring that all action items are
resolved.
5. Software Implementation Phase
SQA activities during the implementation phase include the audit of:
Results of coding and design activities including the schedule contained
in the Software Development Plan.
Status of all deliverable items.
Configuration management activities and the software development
library.
Nonconformance reporting and corrective action system.
6. Software Integration and Test Phase
244 http://www.certmagic.com
BH0-003
SQA activities during the integration and test phase include:
Assuring readiness for testing of all deliverable items.
Assuring that all tests are run according to test plans and procedures
and that any nonconformances are reported and resolved.
Assuring that test reports are complete and correct.
Certifying that testing is complete and software and documentation
are ready for delivery.
Participating in the Test Readiness Review and assuring all action
items are completed.
7. Software Acceptance and Delivery Phase
As a minimum, SQA activities during the software acceptance and
delivery phase include assuring the performance of a final
configuration audit to demonstrate that all deliverable items are ready
for delivery.
8. Software Sustaining Engineering and Operations Phase
During this phase, there will be mini-development cycles to enhance or
correct the software. During these development cycles, SQA conducts
the appropriate phase-specific activities described above.
F. Techniques and Tools
SQA should evaluate its needs for assurance tools versus those
available off-the-shelf for applicability to the specific project, and must
develop the others it requires. Useful tools might include audit and
inspection checklists and automatic code standards analyzers.
IV. SOFTWARE QUALITY ENGINEERING
A. Concepts
Software Quality Engineering (SQE) is a process that evaluates,
assesses, and improves the quality of software. Software quality is
often defined as the degree to which software meets requirements for
reliability, maintainability, transportability, etc., as contrasted with
functional, performance, and interface requirements that are satisfied
as a result of software engineering.
245 http://www.certmagic.com
BH0-003
Quality must be built into a software product during its development to
satisfy quality requirements established for it. SQE ensures that the
process of incorporating quality in the software is done properly, and
that the resulting software product meets the quality requirements.
The degree of conformance to quality requirements usually must be
determined by analysis, while functional requirements are
demonstrated by testing. SQE performs a function complementary to
software development engineering. Their common goal is to ensure
that a safe, reliable, and quality engineered software product is
developed.
B. Software Qualities
Qualities for which an SQE evaluation is to be done must first be
selected and requirements set for them. Some commonly used
qualities are reliability, maintainability,transportability, interoperability,
testability, useability, reusability, traceability, sustainability, and
efficiency. Some of the key ones are discussed below.
1. Reliability
Hardware reliability is often defined in terms of the Mean-Time-To-
Failure, or MTTF, of a given set of equipment. An analogous notion is
useful for software, although the failure mechanisms are different and
the mathematical predictions used for hardware have not yet been
usefully applied to software. Software reliability is often defined as the
extent to which a program can be expected to perform intended
functions with required precision over a given
period of time. Software reliability engineering is concerned with the
detection and correction of errors in the software; even more, it is
concerned with techniques to compensate for unknown software errors
and for problems in the hardware and data environments in which the
software
must operate.
2. Maintainability
Software maintainability is defined as the ease of finding and
correcting errors in the software. It is analogous to the hardware
quality of Mean-Time-To-Repair, or MTTR. While there is as yet no
way to directly measure or predict software maintainability, there is a
significant body of knowledge about software attributes that make
software easier to maintain. These include modularity, self (internal)
documentation, code readability, and structured coding techniques.
These same attributes also improve sustainability, the ability to make
improvements to the
software.
246 http://www.certmagic.com
BH0-003
3. Transportability
Transportability is defined as the ease of transporting a given set of
software to a new hardware and/or operating system environment.
4. Interoperability
Software interoperability is the ability of two or more software systems
to exchange information and to mutually use the exchanged
information.
5. Efficiency
Efficiency is the extent to which software uses minimum hardware
resources to perform its functions.
There are many other software qualities. Some of them will not be
important to a specific software system, thus no activities will be
performed to assess or improve them.
Maximizing some qualities may cause others to be decreased. For
example, increasing the efficiency of a piece of software may require
writing parts of it in assembly language. This will decrease the
transportability and maintainability of the software.
C. Metrics
Metrics are quantitative values, usually computed from the design or
code, that measure the quality in question, or some attribute of the
software related to the quality. Many metrics have been invented, and
a number have been successfully used in specific environments, but
none has gained widespread acceptance.
D. A Software Quality Engineering Program
The two software qualities which command the most attention are
reliability and maintainability. Some practical programs and
techniques have been developed to improve the reliability and
maintainability of software, even if they are not measurable or
predictable. The types of activities
that might be included in an SQE program are described here in terms
of these two qualities. These activities could be used as a model for
the SQE activities for additional qualities.
1. Qualities and Attributes
An initial step in laying out an SQE program is to select the qualities
that are important in the context of the useof the software that is
247 http://www.certmagic.com
BH0-003
being developed. For example, the highest priority qualities for flight
software are usually reliability and efficiency. If revised flight software
can be up-linked during flight, maintainability may be of interest, but
considerations like transportability will not drive the design or
implementation. On the other hand, the use of science analysis
software might require ease of change and maintainability, with
reliability a concern and
efficiency not a driver at all.
After the software qualities are selected and ranked, specific attributes
of the software that help to increase those qualities should be
identified. For example, modularity is an attribute that tends to
increase both reliability and maintainability. Modular software is
designed to result in code that is apportioned into small, self-
contained, functionally unique components or units. Modular code is
easier to maintain, because the interactions between units of code are
easily understood, and low level functions are contained in few units of
code. Modular code is also more reliable, because it is easier to
completely test a small, self contained unit.
Not all software qualities are so simply related to measurable design
and code attributes, and no quality is so simple that it can be easily
measured. The idea is to select or devise measurable, analyzable, or
testable design and code attributes that will increase the desired
qualities. Attributes like information hiding, strength, cohesion, and
coupling should be considered.
2. Quality Evaluations
Once some decisions have been made about the quality objectives and
software attributes, quality evaluations canbe done. The intent in an
evaluation is to measure the effectiveness of a standard or procedure
in promoting the desired attributes of the software product. For
example, the design and coding standards should undergo a quality
evaluation. If modularity is desired, the standards should clearly say
so and should set standards for the size of units or components. Since
internal documentation is linked to maintainability, the documentation
standards should be clear and require good internal documentation.
Quality of designs and code should also be evaluated. This can be
done as a part of the walkthrough or inspection process, or a quality
audit can be done. In either case, the implementation is evaluated
against the standard and against the evaluator's knowledge of good
software engineering practices, and examples of poor quality in the
product are identified for possible correction.
3. Nonconformance Analysis
248 http://www.certmagic.com
BH0-003
One very useful SQE activity is an analysis of a project's
nonconformance records. The nonconformances should be analyzed
for unexpectedly high numbers of events in specific sections or
modules of code. If areas of code are found that have had an
unusually high error count (assuming it is not because the code in
question has been tested more thoroughly), then the code should be
examined. The high error count may be due to poor quality code, an
inappropriate design, or requirements that are not well understood or
defined. In any case, the analysis may indicate changes and rework
that can improve the reliability of the completed software. In addition
to code problems, the analysis may also reveal software development
or maintenance processes that allow or cause a high proportion of
errors to be introduced into the software. If so, an evaluation of the
procedures may lead to changes, or an audit may discover that the
procedures are not being followed.
4. Fault Tolerance Engineering
For software that must be of high reliability, a fault tolerance activity
should be established. It should identify software which provides and
accomplishes critical functions and requirements. For this software,
the engineering activity should determine and develop techniques
which will ensure that the needed reliability or fault tolerance will be
attained. Some of the techniques that have been developed for high
reliability environments include:
Input data checking and error tolerance. For example, if out-of-range
or missing input data can affect reliability, then sophisticated error
checking and data interpolation/extrapolation schemes may
significantly improve reliability.
Proof of correctness. For limited amounts of code, formal "proof of
correctness" methods may be able to demonstrate that no errors exist.
N-Item voting. This is a design and implementation scheme where a
number of independent sets of software and hardware operate on the
same input. Some comparison (voting) scheme is used to determine
which output to use. This is especially effective where subtle timing or
hardware errors may be present.
Independent development. In this scheme, one or more of the N-
items are independently developed units of software. This helps
prevent the simultaneous failure of all items due to a common coding
error.
E. Techniques and Tools
249 http://www.certmagic.com
BH0-003
Some of the useful fault-tolerance techniques are described under
subsection D, above. Standard statistical techniques can be used to
manipulate nonconformance data. In addition, there is considerable
experimentation with the Failure Modes and Effects Analysis (FMEA)
technique adapted from hardware reliability engineering. In particular,
the FMEA can be
used to identify failure modes or other assumable (hardware) system
states which can then lead the quality engineer to an analysis of the
software that controls the system as it assumes those states.
There are also tools that are useful for quality engineering. They
include system and software simulators, which allow the modeling of
system behavior; dynamic analyzers, which detect the portions of the
code that are used most intensively; software tools that are used to
compute metrics from code or designs; and a host of special purpose
tools that can, for example, detect all system calls to help decide on
portability limits.
V. VERIFICATION AND VALIDATION
A. Concepts and Definitions
Software Verification and Validation (V&V) is the process of ensuring
that software being developed or changed will satisfy functional and
other requirements (validation) and each step in the process of
building the software yields the right products (verification). The
differences between verification and validation are unimportant except
to the theorist; practitioners use the term V&V to refer to all of the
activities that are aimed at making sure the software will function as
required.
V&V is intended to be a systematic and technical evaluation of
software and associated products of the development and maintenance
processes. Reviews and tests are done at the end of each phase of
the development process to ensure software requirements are
complete and testable and that design, code, documentation, and data
satisfy those requirements.
B. Activities
The two major V&V activities are reviews, including inspections and
walkthroughs, and testing.
1. Reviews, Inspections, and Walkthroughs
Reviews are conducted during and at the end of each phase of the life
cycle to determine whether established requirements, design concepts,
and specifications have beenmet. Reviews consist of the presentation
250 http://www.certmagic.com
BH0-003
of material to a review board or panel. Reviews are most effective
when conducted by personnel who have not been directly involved in
the development of the software being reviewed.
Informal reviews are conducted on an as-needed basis. The developer
chooses a review panel and provides and/or presents the material to
be reviewed. The material may be as informal as a computer listing or
hand-written documentation.
Formal reviews are conducted at the end of each life cycle phase. The
acquirer of the software appoints the formal review panel or board,
who may make or affect a go/no-go decision to proceed to the next
step of the life cycle. Formal reviews include the Software
Requirements Review, the Software Preliminary Design Review, the
Software Critical Design Review, and the Software Test Readiness
Review.
An inspection or walkthrough is a detailed examination of a product on
a step-by-step or line-of-code by line-of-code basis. The purpose of
conducting inspections and walkthroughs is to find errors. The group
that does an inspection or walkthrough is composed of peers from
development, test, and quality assurance.
2. Testing
Testing is the operation of the software with real or simulated inputs to
demonstrate that a product satisfies its requirements and, if it does
not, to identify the specific differences between expected and actual
results. There are varied levels of software tests, ranging from unit or
element testing through integration testing and performance testing,
up to software system and acceptance tests.
a. Informal Testing
Informal tests are done by the developer to measure the development
progress. "Informal" in this case does not mean that the tests are
done in a casual manner, just that the acquirer of the software is not
formally involved, that witnessing of the testing is not required, and
that the prime purpose of the tests is to find errors. Unit, component,
and subsystem integration tests are usually informal tests.
Informal testing may be requirements-driven or design-driven.
Requirements-driven or black box testing is done by selecting the
input data and other parameters based on the software requirements
and observing the outputs and reactions of the software. Black box
testing can be done at any level of integration. In addition to testing
for satisfaction of requirements, some of the objectives of
requirements-driven testing are to ascertain:
251 http://www.certmagic.com
BH0-003
Computational correctness.
Proper handling of boundary conditions, including extreme inputs and
conditions that cause extreme outputs.
State transitioning as expected.
Proper behavior under stress or high load.
Adequate error detection, handling, and recovery.
Design-driven or white box testing is the process where the tester
examines the internal workings of code. Design-driven testing is done
by selecting the input data and other parameters based on the internal
logic paths that are to be checked. The goals of design-driven testing
include ascertaining correctness of:
All paths through the code. For most software products, this can be
feasibly done only at the unit test level.
Bit-by-bit functioning of interfaces. Size and timing of critical elements
of code.
b. Formal Tests
Formal testing demonstrates that the software is ready for its intended
use. A formal test should include an acquirer-approved test plan and
procedures, quality assurance witnesses, a record of all discrepancies,
and a test report. Formal testing is always requirements-driven, and
its purpose is to demonstrate that the software meets its
requirements.
Each software development project should have at least one formal
test, the acceptance test that concludes the development activities and
demonstrates that the software is ready for operations.
In addition to the final acceptance test, other formal testing may be
done on a project. For example, if the software is to be developed
and delivered in increments or builds, there may be incremental
acceptance tests. As a practical matter, any contractually required
test is usuallyconsidered a formal test; others are "informal."
After acceptance of a software product, all changes to the product
should be accepted as a result of a formal test. Post acceptance testing
should include regression testing. Regression testing involves
rerunning previously used acceptance tests to ensure that the change
did not disturbfunctions that have previously been accepted.
252 http://www.certmagic.com
BH0-003
C. Verification and Validation During the Software Acquisition Life
Cycle
The V&V Plan should cover all V&V activities to be performed during all
phases of the life cycle. The V&V Plan Data Item Description (DID)
may be rolled out of the Product AssurancePlan DID contained in the
SMAP Management Plan Documentation Standard and DID.
1. Software Concept and Initiation Phase
The major V&V activity during this phase is to develop a concept of
how the system is to be reviewed and tested. Simple projects may
compress the life cycle steps; if so, the reviews may have to be
compressed. Test concepts may involve simple generation of test
cases by a user representative or may require the development of
elaborate simulators and test data generators. Without an adequate
V&V concept and plan, the cost, schedule, and complexity of
the project may be poorly estimated due to the lack of adequate test
capabilities and data.
2. Software Requirements Phase
V&V activities during this phase should include:
Analyzing software requirements to determine if they are consistent
with, and within the scope of, system requirements.
Assuring that the requirements are testable and capable of being
satisfied.
Creating a preliminary version of the Acceptance Test Plan, including a
verification matrix, which relates requirements to the tests used to
demonstrate that requirements are satisfied.
Beginning development, if needed, of test beds and test data
generators.
The phase-ending Software Requirements Review (SRR).
3. Software Architectural (Preliminary) Design Phase
V&V activities during this phase should include:
Updating the preliminary version of the Acceptance Test Plan and the
verification matrix.
253 http://www.certmagic.com
BH0-003
Conducting informal reviews and walkthroughs or inspections of the
preliminary software and data base designs.
The phase-ending Preliminary Design Review (PDR) at which the
allocation of requirements to the software architecture is reviewed and
approved.
4. Software Detailed Design Phase
V&V activities during this phase should include:
Completing the Acceptance Test Plan and the verification matrix,
including test specifications andunit test plans.
Conducting informal reviews and walkthroughs or inspections of the
detailed software and data base designs.
The Critical Design Review (CDR) which completes the software
detailed design phase.
5. Software Implementation Phase V&V activities during this phase
should include:
Code inspections and/or walkthroughs.
Unit testing software and data structures.
Locating, correcting, and retesting errors.
Development of detailed test procedures for the next two phases.
6. Software Integration and Test Phase
This phase is a major V&V effort, where the tested units from the
previous phase are integrated into subsystems and then the final
system. Activities during this phase should include:
Conducting tests per test procedures.
Documenting test performance, test completion, and conformance of
test results versus expected results.
Providing a test report that includes a summary of nonconformances
found during testing.
Locating, recording, correcting, and retesting nonconformances.
254 http://www.certmagic.com
BH0-003
The Test Readiness Review (TRR), confirming the product's readiness
for acceptance testing.
7. Software Acceptance and Delivery Phase
V&V activities during this phase should include:
By test, analysis, and inspection, demonstrating that the developed
system meets its functional, performance, and interface requirements.
Locating, correcting, and retesting nonconformances.
The phase-ending Acceptance Review (AR).
8. Software Sustaining Engineering and Operations Phase
Any V&V activities conducted during the prior seven phases are
conducted during this phase as they pertain to the revision or update
of the software.
D. Independent Verification and Validation
Independent Verification and Validation (IV&V) is a process whereby
the products of the software development life cycle phases are
independently reviewed, verified, and validated by an organization that
is neither the developer nor the acquirer of the software. The IV&V
agent should have no stake in the success or failure of the software.
The IV&V agent's only interest should be to make sure that the
software is thoroughly tested against its complete set of
requirements.
The IV&V activities duplicate the V&V activities step-by-step during the
life cycle, with the exception that the IV&V agent does no informal
testing. If there is an IV&V agent, the formal acceptance testing may
be done only once, by the IV&V agent. In this case, the developer will
do a formal demonstration that the software is ready for formal
acceptance.
E. Techniques and Tools
Perhaps more tools have been developed to aid the V&V of software
(especially testing) than any other software activity. The tools
available include code tracers, special purpose memory dumpers and
formatters, data generators, simulations, and emulations. Some tools
are essential for testing any significant set of software, and, if they
have to be developed, may turn out to be a significant cost and
schedule driver.
255 http://www.certmagic.com
BH0-003
An especially useful technique for finding errors is the formal
inspection. Formal inspections were developed by Michael Fagan of
IBM. Like walkthroughs, inspections involve the line-by-line evaluation
of the product being reviewed. Inspections, however, are significantly
different from walkthroughs and are significantly more effective.
Inspections are done by a team, each member of which has a specific
role. The team is led by a moderator, who is formally trained in the
inspection process. The team includes a reader, who leads the team
through the item; one or more reviewers, who look for faults in the
item; a recorder, who notes the faults; and the author, who helps
explain the item being inspected.
This formal, highly structured inspection process has been extremely
effective in finding and eliminating errors. It can be applied to any
product of the software development process, including documents,
design, and code. One of its important side benefits has been the
direct feedback to the developer/author, and the significant
improvement in quality that results.
VI. NONCONFORMANCE REPORTING AND CORRECTIVE ACTION
1. Nonconformance Detection and Reporting
By definition, a nonconformance is a deviation of any product from its
requirements or standards. Nonconformance reports may be filed
against any product in any phase of the software life cycle by anyone
associated with the project. Normally, the NRCA system is used after a
product is firstapproved or baselined by its developer and released for
wider use. For example, while a developer is unit testing his/her code,
errors found may be tracked only locally. After the code is declared
correct and released for integration, the NRCA system is used to
inform users of the code about nonconformances and to assure that
the nonconformances are corrected and not overlooked.
Usually, a special form is used to make the nonconformance report.
Examples of the information the form might contain are:
Date and time of detection of the nonconformance.
Error identification (report number and title).
Reporting individual and organization.
Individual responsible for corrective action.
Criticality of the nonconformance.
256 http://www.certmagic.com
BH0-003
Statement of the nonconformance.
Proposed fix for the nonconformance.
Identifier of the unit of code, data, or documentation in which
corrective action must be taken.
Life cycle phase in which the nonconformance was introduced.
Life cycle phase in which the nonconformance was detected.
Final closure resolution.
Date and/or version of the configuration item in which the correction will
be included.
Date on which the nonconformance is closed.
2. Tracking and Management Reports
After the report is prepared by the individual who found the
nonconformance, the data are entered into some form of controlling
system. Data base management systems are often used to help
automate the otherwise laborious clerical effort of tracking the
nonconformances and providing management reports.
A nonconformance tracking and reporting system should be able to
provide management reports containing such information as error and
correction status, the number of errors found per product, and the
criticality of open problems. The data enable the impact of
nonconformances to be evaluated so that the use of resources may be
prioritized.
3. Impact Assessment and Corrective Action
Nonconformance reports should be evaluated for criticality and level of
importance. Factors to be considered include:
The impact of not correcting the nonconformance.
The resources required for correcting the nonconformance.
The impact on other baselined items if the nonconformance is
corrected.
If the decision is made to correct a nonconformance, there should be
procedures to control the corrective action process. Such procedures
should include followup to ensure the nonconformance has been
257 http://www.certmagic.com
BH0-003
documented and corrected in the appropriate version of software, and
to assure that adequate testing, including regression testing, is done.
C. Interrelationships
If the nonconformance is in a code or data product, those responsible
for V&V activities must develop tests to ensure that the problem is
indeed satisfactorily corrected. In addition, regression testing is
needed to make sure that no new problems have been introduced by
the fix.
SQA must assure that proper procedures are followed in processing
nonconformances, and that shortcuts are not taken since they would
threaten product integrity. SQA must also ensure that the modified
product still meets its standards. SQA may use the numbers of
nonconformances detected in specific areas of the system or that
occurred in specific life cycle phases to identify process or product
areas that might benefit from an audit.
D. Techniques and Tools
Each project should consider an automated tracking system for
nonconformance reports and an automated updating capability to
identify and record the product changes that occur as a result of the
resolution of the nonconformances.
VII. SOFTWARE AND SYSTEM SAFETY
A. Concepts and Definitions
System safety is concerned with the possibility of catastrophic failure
of systems in such a way as to compromise the safety of people or
property, or result in mission failure. Software safety is definable only
in the system context. Software has no inherent dangers; however,
systems controlled or monitored by software do fail, and some failures
of some systems will have safety impacts. To the extent that system
failures can be caused or fail to be prevented by software, there is a
need for an activity called "software safety."
If we are to be concerned with the safety of software only in a system
context, we must then be concerned with nonconformances in the
software and with the software requirements as well. Indeed, the
most serious problems with software-based systems are those that
develop when the
software requirements are incorrect, inappropriate, or incomplete for
the system situation.
258 http://www.certmagic.com
BH0-003
B. Software Problems
System failures that are caused by software are due to one of two
types of software problems: nonconformances (or failures to satisfy
requirements) or an error or omission in the software requirements. A
nonconformance may be simple (the most common is a coding error or
"bug"), or more complex (i.e., a subtle timing error that delays a
shuttle launch). The important point about nonconformances is that
verification and validation techniques are designed to
detect them and assurance techniques are designed to prevent them;
improvements in these methods and a safety program based on
specialized application of them are improving the safety and reliability
of software controlled systems.
An error or omission in requirements is less tractable. The software
may perform exactly as required, but the requirements do not
correctly deal with some system state. When the system enters the
undefined state, unexpected and undesirable behavior may result.
This type of problem cannot be handled within the software discipline;
it results from a failure of the system and software engineering
processes which developed and allocated the system requirements to
the software.
C. Methods for Improving Software Safety
Improving the software development process and building better
software are ways to increase system reliability, i.e., by producing
software with fewer faults. Intuitively, more reliable software is
probably safer software, but from a safety standpoint more
concentration on safety-related
software functions is needed. A first order approach is to identify the
critical software that controls system safety-related functions and give
it special attention through the development and testing process. This
is just a special case of the "build it better" method, but it focuses
scarce
resources on critical areas.
D. Software Safety Program (Example)
System hazard analysis may indicate that some software requires a
more formal safety program because it is included in a safety critical
system component. The software safety program begins with a
preliminary software safety analysis. The purpose of the preliminary
software safety analysis is to identify software controlled functions that
affect the safety critical component and the software components that
execute the functions. These software components are safety
critical. When a safety critical software component is identified, then
software safety activities are initiated on that component and
259 http://www.certmagic.com
BH0-003
continued through the requirements, design, and code analyses and
testing phases in the software development process.
1. Requirements Analysis
Software safety requirements analysis forms the basis for subsequent
software safety activities. The process of requirements analysis
evaluates both software and interface requirements. The analysis is
intended to identify errors and deficiencies in the software
requirements that could
result in the identified hazardous system states. Techniques employed
in performing requirements analysis include criticality analysis;
specification analysis; and timing, sizing, and throughput analysis.
Criticality analysis evaluates each requirement in terms of the safety
objectives derived for a given software component. This evaluation is
to determine whether the requirement has safety implications. If so,
the requirement is deemed critical and must be tracked throughout the
software development cycle; that is, through design, coding, and
testing. It must be traceable from the highest level specification all
the way to the code and documentation.
Specification analysis evaluates the completeness, correctness,
consistency, and testability of identified software safety critical
requirements. Specification analysis considers each requirement
singly and all requirements as a set.
Timing, sizing, and throughput analysis evaluates software
requirements that relate to execution time, memory allocation, and
channel usage. Timing, sizing, and throughput analysis focuses on
noting and defining program constraints based on maximum required
and allowable execution times, maximum memory usage and
availability, and throughput considerations based on I/O channel
usage.
2. Design Analysis
Design analysis verifies that the program design correctly implements
safety critical requirements.
Design logic analysis evaluates the equations, algorithms, and control
logic of the software design.
Design data analysis evaluates the description and intended usage of
each data item used in design of the critical component. Interrupts
and their effect on data must receive special attention in safety critical
areas to verify that interrupts and interrupt handling routines do not
alter critical data items used by other routines.
260 http://www.certmagic.com
BH0-003
Design interface analysis verifies the proper design of a software
component's interfaces with other components of the system,
including hardware, software, and operators.
Design constraint analysis evaluates the design solutions against
restrictions imposed by requirements and real-world limitations. The
design must be responsive to all known or anticipated restrictions on
the software component. These restrictions may include timing,
sizing, and throughput constraints, equation and algorithm limitations,
input and output data limitations, and design solution limitations.
3. Code Analysis
Code analysis verifies that the coded program correctly implements
the verified design and does not violate safety requirements. The
techniques used in the performance of code analysis mirror those used
in design analysis.
4. Safety Testing
Software safety testing verifies analysis results, investigates program
behavior, and confirms that the program complies with safety
requirements. Special safety testing, conducted in accordance with
the safety test plan and procedures, establishes the compliance of the
software with the safety requirements. Safety testing focuses on
locating program weaknesses and identifying extreme or unexpected
situations that could cause the software to fail in ways that would
cause a violation of safety requirements. The safety testing effort is
limited to those software requirements classified as safety critical
items.
E. Techniques and Tools
In the last few years, there has been much effort to adapt methods
used in hardware safety and reliability to software. Tools like fault tree
analysis and sneak circuit analysis have been applied to software with
some success. Modeling of software using Petri nets has been tried,
and other
modeling techniques have been advocated, but with only limited
success to date. While some techniques may have some limited
usefulness, their success depends heavily on the ability of the analyst
that applies them.
261 http://www.certmagic.com
BH0-003
FUNCTIONAL SYSTEM TESTING TECHNIQUES
Functional Testing techniques are designed to ensure the system
requirements and specifications are achieved.
Objective is to –
♦ The process normally involves creation of test conditions and
then using them to evaluate that system developed is correct.
The Different Functional System Testing Techniques are –
1. Requirements Testing
2. Regression Testing
3. Error – Handling Testing
4. Manual – support Testing
5. Intersystem Testing
6. Control Testing
7. Parallel Testing
1) Requirements Testing Technique :
262 http://www.certmagic.com
BH0-003
Usage To ensure that system performs correctly
To ensure that correctness can be sustained for a
considerable period of time.
System can be tested for correctness through all
phases of SDLC but incase of reliability the
programs should be in place to make system
operational.
Objectives Successfully implementation of user requirements
Correctness maintained over considerable period of
time
Processing of the application complies with the
organization’s policies and procedures.
Secondary users needs are fulfilled –
− Security officer
− DBA
− Internal auditors
− Record retention
− Comptroller
How to Use Test conditions created
− These test conditions are generalized ones,
which becomes test cases as the SDLC
progresses until system is fully operational.
− Test conditions are more effective when created
from user’s requirements.
− Test conditions if created from documents then
if there are any error in the documents those
will get incorporated in Test conditions and
testing would not be able to find those errors.
− Test conditions if created from other sources
(other than documents) error trapping is
effective.
Functional Checklist created.
When to use Every application should be Requirement tested
Should start at Requirements phase and should
progress till operations and maintenance phase.
The method used to carry requirement testing and
the extent of it is important.
Example Creating test matrix to prove that system
requirements as documented are the requirements
desired by the user.
Creating checklist to verify that application
complies to the organizational policies and
procedures.
2) Regression Testing Technique :
Background One segment of system is developed and
thoroughly tested.
263 http://www.certmagic.com
BH0-003
Another segment is changed which has ditrous
effect on the tested segment.
The implemented change, new data or
parameters have created change in the already
tested segment.
Usage All aspects of system remain functional after
testing.
Change in one segment does not change the
functionality of other segment.
Objectives Determine –
− system documents remain current
− System test data and test conditions remain
current.
− Previously tested system functions properly
without getting effected though changes are
made in some other segment of application
system.
How to Use Test cases, which were used previously for the
already tested segment is, re-run to ensure that
the results of the segment tested currently and
the results of same segment tested earlier are
same.
Test automation is needed to carry out the test
transactions (test condition execution) else the
process is very time consuming and tedious.
In this case of testing cost/benefit should be
carefully evaluated else the efforts spend on
testing would be more and payback would be
minimum.
When to use When there is high risk that the new changes
may effect the unchanged areas of application
system.
In development process: Regression testing
should be carried out after the pre-determined
changes are incorporated in the application
system.
In Maintenance phase : regression testing
should be carried out if there is a high risk that
loss may occur when the changes are made to
the system
Example Re-running of previously conducted tests to
ensure that the unchanged portion of system
functions properly.
Reviewing previously prepared system
documents (manuals) to ensure that they do not
get effected after changes are made to the
application system.
Obtaining printout of data dictionary to ensure
264 http://www.certmagic.com
BH0-003
that the data elements reflect the changes being
incorporated.
Disadvantage Time consuming and tedious if test automation
not done.
3) Error – Handling Testing Technique :
Back ground Pre determination of Error handling features is the
basic difference between Automated and manual
systems.
Manual System: can deal with problems as they
occur.
Automated Systems: Must pre program error
handling.
Usage It determines the ability of applications system to
process the incorrect transactions properly
Errors encompass all unexpected conditions.
In some system approx. 50% of programming
effort will be devoted to handling error condition.
Objectives Determine:
Application system recognizes all expected error
conditions.
Accountability of processing errors has been
assigned and procedures provide a high probability
that errors will be properly corrected.
During correction process reasonable control is
maintained over errors.
How to Use A group of knowledgeable people is required to
anticipate what can go wrong in the application
system.
It is needed that all the application knowledgeable
people assemble to integrate their knowledge of
user area, auditing and error tracking.
Then logical test error conditions should be created
based on this assimilated information.
The error handling testing technique should test –
− Error
− Processing of error
− Control condition
− Reentry of condition is proper or not.
The iterative process should be used where first
the errors in the system trapped should be
corrected and then the corrected system should be
re-tested to check the authenticity of application
operation and to complete the error handling
265 http://www.certmagic.com
BH0-003
testing cycle.
Tester should think negatively to trap errors.
Testers should determine how the system should
fail so that they can test to determine if the
software can properly process the erroneous data.
When to use Throughout SDLC
Impact from errors should be identified and should
be corrected to reduce the errors to acceptable
level.
Used to assist in error management process of
system development and maintenance.
Example Create a set of erroneous transactions and enter
them into the application system then find out
whether the system is able to identify the
problems.
Using iterative testing enters transactions and trap
errors. Correct them. Then enter transactions with
errors, which were not present in the system
earlier.
4) Manual Support Testing Technique :
Usage It involves testing of all the functions performed by
the people while preparing the data and using
these data from automated system.
Objectives Verify – manual support documents and
procedures are correct.
Determine –
− Manual support responsibility is correct
− Manual support people are adequately
trained.
− Manual support and automated segment
are properly interfaced.
How to Use It involves:
− Evaluation of adequacy of process
− Execution of process
Process evaluated in all segments of SDLC.
Execution of the can be done in conjunction with
normal system testing.
Instead of preparing, execution and entering actual
test transactions the clerical and supervisory
personnel can use the results of processing from
application system.
It involves several iterations of process.
To test people it requires testing the interface
between the people and application system.
266 http://www.certmagic.com
BH0-003
When to use Verification that manual systems function properly
should be conducted throughout the SDLC.
Should not be done at later stages of SDLC.
Best done at installation stage so that the clerical
people do not get used to the actual system just
before system goes to production.
Example Provide input personnel with the type of
information they would normally receive from their
customers and then have them transcribe that
information and enter it in the computer.
Users can be provided a series of test conditions
and then asked to respond to those conditions.
Conducted in this manner, manual support testing
is like an examination in which the users are asked
to obtain the answer from the procedures and
manuals available to them.
5) Intersystem Testing Technique:
Background Application systems are frequently interconnected
to other application system.
The interconnection may be data coming from
another application system, leaving for another
application system or both.
Frequently multiple systems (applications)
sometimes called cycles or functions are involved.
Usage To ensure interconnection between application
functions correctly.
Objectives Determining –
− Proper parameters and data are correctly
passed between the applications
− Documentation for involved system is correct
and accurate.
Ensure Proper timing and coordination of
functions exists between the application system.
How to Use Operations of multiple systems are tested.
Multiple systems are run from one another to
check that they are acceptable and processed
properly.
When to use When there is change in parameters in
application system
The parameters, which are erroneous then risk
associated to such parameters, would decide the
extent of testing and type of testing.
Intersystem parameters would be checked /
267 http://www.certmagic.com
BH0-003
verified after the change or new application is
placed in the production.
Examples Develop test transaction set in one application
and passing to another system to verify the
processing.
Entering test transactions in live production
environment and then using integrated test
facility to check the processing from one system
to another.
Verifying new changes of the parameters in the
system, which are being tested, are corrected in
the document.
Disadvantage Time consuming
Cost may be expensive if system is run several
times iteratively.
6) Control Testing Technique :
Background One half of total system development effort is
directly attributable to controls.
Controls include:
− Data validation
− File integrity
− Audit trail
− Back up and recovery
− Documentation.
− Other aspects of system related to
integrity
Control is system within a system.
Control looks at the totality of the system.
Usage Control is a management tool to ensure that
processing is performed in accordance to what
management desire or intents of management.
Objectives Accurate and complete data
Authorized transactions
Maintenance of adequate audit trail of information.
Efficient, effective and economical process.
Process meeting the needs of the user.
How to Use To test controls risks must be identified.
Develop risk matrix, which identifies the risks,
controls; segment within application system in
which control resides.
Testers should have negative approach i.e. should
determine or anticipate what can go wrong in the
application system.
When to use Should be tested with other system tests.
268 http://www.certmagic.com
BH0-003
Example file reconciliation procedures work
Manual controls in place.
7) Parallel Testing Technique :
Usage To ensure that the processing of new application
(new version) is consistent with respect to the
processing of previous application version.
Objectives Conducting redundant processing to ensure that
the new version or application performs correctly.
Demonstrating consistency and inconsistency
between 2 versions of the application.
How to Use Same input data should be run through 2 versions
of same application system.
Parallel testing can be done with whole system or
part of system (segment).
When to use When there is uncertainty regarding correctness of
processing of new application where the new and
old version are similar.
In financial applications like banking where there
are many similar applications the processing can
be verified for old and new version through
parallel testing.
Example Operating new and old version of a payroll system
to determine that the paychecks from both
systems are reconcilable.
Running old version of application to ensure that
the functions of old system are working fine with
respect to the problems encountered in the new
system.
IEEE Software Testing Standards
IEEE: Institute of Electrical and Electronics Engineers
A transnational organization founded in 1884, IEEE consists of dozens of
specialized societies within geographical regions throughout the world.
Software testing standards are developed within the technical committees of
the IEEE Societies and the Standards Coordinating Committees of the IEEE
Standards Board.
These standards are created through a process of obtaining the
consensus of practicing professionals. This consensus process, which includes
269 http://www.certmagic.com
BH0-003
careful discussion and debate among members of the various committees
who serve voluntarily, is one of the fundamental themes of the standards
process. Another key theme is to provide standards in a timely manner, from
project approval to standard approval is approximately 3 years.
To obtain the full version of the IEEE Standards:
IEEE
445 Hoes Lane
PO Box 1331
Piscataway, NJ 08855-1331
The following standards are those a tester should be aware of, and include a
short abstract about each one:
610.12-1990: IEEE Standard Glossary of Software Engineering
Terminology
Topics covered include addressing; assembling, compiling, linking,
loading, computer performance evaluation, configuration management, data
types; errors, faults, and failures; evaluation techniques; instruction types;
language types; libraries; microprogramming; operating systems; quality
attributes; software documentation; software and system testing; software
architectures; software development process; software development
techniques; and software tools. This standard promotes clarity and
consistency in the vocabulary of software engineering and associated fields.
730-1998: IEEE Standard for Software Quality Assurance Plans
The purpose of this standard is to provide uniform, minimum
acceptable requirements for preparation and content of SQAPs (Software
Quality Assurance Plans).
The plan includes the following sections:
Purpose
Defines purpose and scope
Reference Documents
Complete list of documents referenced within SQAP
Management
Describes organization, tasks, responsibilities
Documentation
Identifies the documentation governing development,
verification and validation, use, and maintenance of the software.
Documents include: Software Requirements Specification (SRS),
Software Design Description (SDD), Software Verification and Validation Plan
(SVVP), Software Verification and Validation Report (SVVR), User
documentation, Software Configuration Management Plan (SCMP), Software
Development Plan, Standards and procedures manual, software project
management plan, software maintenance manual.
Standards, practices, conventions, and metrics
270 http://www.certmagic.com
BH0-003
Identifies the standards, practices, conventions and metrics to
be applied, and states how compliance with these items is to be monitored
and assured.
Reviews and audits
Defines the technical and managerial reviews and audits to be
conducted, states how the reviews and audits are to be accomplished, and
states what further actions are required and how they are to be implemented
and verified.
Test
Identifies all the tests not included in the SVVP for the software
covered by the SQAP and states what methods are to be used.
Problem reporting and corrective action
Describes the practices and procedures to be followed for
reporting, tracking, and resolving problems identified in both software items
and the software development and maintenance process
Tools, techniques, and methodologies
Identifies the special software tools, techniques, and
methodologies that support SQA, state their purposes, and describe their
use.
Code control
Defines the methods and facilities used to maintain, store,
secure, and document controlled versions of the identified software during all
phases of the software life cycle.
Media control
Identifies the media for each computer product and the
documentation required to store the media, including the copy and restore
process, and protects the computer program physical media from
unauthorized access or inadvertent damage or degradation during all phases
of the software life cycle.
Supplier control
States the provisions for assuring that software provided by
suppliers meets established requirements. Also states the methods that will
be used to assure that the software supplier receives adequate and complete
requirements. This section also states the methods to be employed to assure
that the developers comply with the requirements of this standard.
Records collection, maintenance, and retention
Identifies the SQA documentation to be retained; states the
methods and facilities to be used to assemble, safeguard, and maintain this
documentation, and designates the retention period.
Training
Identifies the training activities necessary to meet the needs of
the SQAP
Risk management
Specifies the methods and procedures employed to identify,
assess, monitor, and control areas of risk arising during the portion of the
software life cycle covered by the SQAP.
828-1998: IEEE Standard for Software Configuration Management Plans
271 http://www.certmagic.com
BH0-003
This standard establishes the minimum required contents of a software
configuration (SCM) plan. It is supplemented by 1042-1987 (which provides
approaches to good software configuration management planning). This
standard applies to the entire life cycle of critical software. The plan
documents what SCM activities are to be done, how they are to be done, who
is responsible for doing specific activities, when they are to happen, and what
resources are required.
829-1998: IEEE Standard for Software Test Documentation
This standard describes a set of basic test documents that are
associated with the dynamic aspects of software testing. The standard
defines the purpose, outline, and content of each basic document.
Documents included:
Test Plan
Test Design Specification
Test Case Specification
Test Procedure Specification
Test Item Transmittal Report (a document identifying test
items)
Test Log
Test Incident Report
Test Summary Report
830-1998: IEEE Recommended Practice for Software Requirements
Specifications
This standard describes the recommended approaches for the
specification of software requirements. It describes the content and qualities
of a good software requirements specification (SRS), and includes several
sample SRS outlines.
1008-1987 (R1993): IEEE Standard for Software Unit Testing (ANSI)
This standard defines an integrated approach to systematic and
documented unit testing. The approach uses unit design and unit
implementation information, in addition to unit requirements, to determine
the completeness of the testing. The standard describes a testing process
composed of a hierarchy of phases, activities, and tasks. Further, it defines a
minimum set of tasks for each activity, although additional tasks may be
added to any activity.
1012-1998: IEEE Standard for Software Verification and Validation
The purpose of this standard is to:
1. Establish a common framework for V&V processes,
activities, and tasks in support of all software life cycle
processes, including acquisition, supply, development,
operation, and maintenance processes.
2. Define the V&V tasks, required inputs, and required
outputs
272 http://www.certmagic.com
BH0-003
3. Identify the minimum V&V tasks corresponding to
software integrity levels using a four-level scheme
4. Define the content of a software V&V Plan (SVVP)
1012a-1998: IEEE Standard for Software Verification and Validation
(Supplement to 1012-1998 – Content Map to IEEE 12207.1)
The 2 requirements (1012 and 12207.1) place requirements on plans
for verification of software and validation of software. The purpose of this
annex is to explain the relationship between the two sets of requirements, so
that users producing documents intended to comply with both standards may
do so.
1016-1998: IEEE Recommended Practice for Software Design
Descriptions
This is a recommended practice for describing software designs. It
specifies the necessary information content, and recommended organization
for a Software Design Description (SDD). An SDD is a representation of a
software system that is used as a medium for communicating software
design information.
1028-1997: IEEE Standard for Software Reviews
The purpose of this standard is to define systematic reviews applicable
to software acquisition, supply, development, operation, and maintenance.
This standard describes how to carry out a review. Other standards or local
management define the context within which a review is performed, and the
use made of the results of the review. Software reviews can be used in
support of the objectives of project management, system engineering,
verification and validation, configuration management, and QA. Different
types of reviews reflect differences in the goals of each review type.
Systematic reviews are described by their defined procedures, scope, and
objectives.
1044-1993: IEEE Standard Classification for Software Anomalies
(ANSI)
(Anomaly: any condition that departs from the expected. The
expectation can come from documentation or someone’s perceptions or
experiences). The methodology of this standard is based on a process
(sequence of steps) that pursues a logical progression from the initial
recognition of an anomaly to its final disposition. Each step interrelates with
and supports the other steps.
1045-1992: IEEE Standard for Software Productivity Metrics (ANSI)
This standard describes the data collection process and calculations for
measuring software productivity.
1058-1998: IEEE Standard for Software Project Management Plans
This standard prescribes the format and content of Software Project
Management Plans (SPMPs). An SPMP is the controlling document for
273 http://www.certmagic.com
BH0-003
managing a software project; it defines the technical and managerial
processes necessary to develop software work products that satisfy the
product requirements.
1058.1-1987(R1993): IEEE Standard for Software Project Management
Plans (ANSI)
Explains the relationship between 1058 standard and 12207.1
standard, so that users producing documents intended to comply with both
standards may do so.
1061-1998: IEEE Standard for Software Quality Metrics Methodology
Scope: Provides a methodology for establishing quality requirements
and identifying, implementing, analyzing, and validating process and product
software quality metrics. This methodology applies to all software at all
phases of any software life cycle.
Framework: Software Quality is the degree to which Software
possesses a desired combination of quality attributes. The purpose of
Software Metrics is to make assessments throughout the Software Life cycle
as to whether the Software Quality requirements are being met. The use of
software metrics reduces subjectivity in the assessment and control of
software quality by providing a quantitative basis for making decisions about
software quality. However the use of Software Metrics does not eliminate the
need for human judgment in Software assessments. The use of software
metrics within an organization or project is expected to have a beneficial
effect by making software quality more visible.
274 http://www.certmagic.com
BH0-003
Other External Standards
The use of standards simplifies communication, promotes consistency and
uniformity, and eliminates the need to invent yet another (often different and
even incompatible) solution to the same problem. Standards, whether
‘official’ or merely agreed upon, are especially important when we’re talking
to customers and suppliers, but it’s easy to underestimate their importance
when dealing with different departments and disciplines within an
organization. They also provide vital continuity so that we are within our own
organization. They also provide vital continuity so that we are not forever
reinventing the wheel. They are a way of preserving proven practices above
and beyond the inevitable staff changes within organizations.
Some standards are particularly important to the testing practitioner.
They can provide a benchmark for writing documents like requirements, so
that testers and others doing verification have a framework for what they can
expect to find. More specifically, standards tell us what to put into key test
documents, such as a test plan.
Standards are not only practical from the development point of view,
but they are increasingly the basis for contracts and therefore also, when
things go wrong, for litigation. One of the issues that arise in litigation is
whether the software was developed according to known standards that are
prevalent in the industry today. This means we need to know not only what
the standards are, but to also see that they are applied.
ISO – International Organization for Standards
ISO 9000: addresses the quality management system of an organization.
ISO 9001: the base international standard for quality management
ISO 9000-3: Guidebook on how ISO 9000 applies to software.
TickIt: UK scheme for certifying organizations producing software according
to ISO 9001.
SPICE –Software Process Improvement and Capability Determination
WG10: Software Process Assessment working group for the ISO. SPICE was
created to develop a suite of related standards and guidebooks. The purpose
is to create a consistent standard for software process assessment that can
be used by different nations and different sectors. The SEI (Software
Engineering Institute) has worked closely with this group, including providing
the CMM (Capability Maturity Model) as input to the effort.
NIST – National Institute of Standards and Technology
A non-regulatory federal agency within the Commerce Department’s
Technology Administration. NIST's mission is to promote economic growth
by working with industry to develop and apply technology, measurements,
and standards. NIST carries out its mission through four interwoven
programs: NIST Laboratories, Baldrige National Quality Program,
Manufacturing Extension Partnership, and Advanced Technology Program.
275 http://www.certmagic.com
BH0-003
NIST programs are helping improve the quality and capabilities of software
used by businesses, research institutions, and consumers. As a result of our
programs (described below), many software packages are more efficient and
can exchange data with each other. More info at: http://www.nist.gov/
DoD – Department of Defense
As the DoD Executive Agent for Information Standards, CFS (Center for
Standards) influences, adopts, develops, promulgates, and maintains
standards for OSD, CINCs, Services, Agencies and the international defense
community. CFS leads DoD's IT standards activities, performs
interoperability assessments, and facilitates the interoperability of customer
IT systems.
GOALS
Identify, consolidate, and coordinate requirements for information
technology standards.
Advocate DoD requirements in standards bodies to permit DoD
adoption of commercial standards versus development of military
standards.
Actively pursue forming partnerships with technology providers to
promote timely delivery of products supporting approved standards.
Develop a framework and provide guidance for applying information
technology standards through updating and publishing the Joint
Technical Architecture (JTA).
Automate processes to the greatest extent possible in all
standardization efforts to expedite development and coordination of
standards and guidance.
Facilitate their use by program managers and engineers.
Chapter 1 - The Product
Overview
• Software Testing is designed and built by Software Testing engineers.
• Software Testing is used by virtually everyone in society.
• Software Testing engineers have a moral obligation to build reliable
Software Testing that does no harm to other people.
• Software Testing engineers view computer Software Testing, as being
made up of the programs, documents, and data required to design and
build the system.
• Software Testing users are only concerned with whether or not
Software Testing products meet their expectations and make their
tasks easier to complete.
Software Testing
276 http://www.certmagic.com
BH0-003
• Software Testing is both a product and a vehicle for developing a
product.
• Software Testing is engineered not manufactured.
• Software Testing does not wear out, but it does deteriorate.
• Currently, most Software Testing is still custom-built.
Software Testing Application Types
• System Software Testing
• Real-time Software Testing
• Business Software Testing
• Engineering and scientific Software Testing
• Embedded Software Testing
• Personal computer Software Testing
• Web-based Software Testing
• Artificial intelligence Software Testing
Software Testing Crisis
• Software Testing failures receive a lot more publicity than Software
Testing engineering success stories.
• The Software Testing crisis predicted thirty years ago has never
materialized and Software Testing engineering successes outnumber
the failures.
• The problems that afflict Software Testing development are associated
more with how to develop and support Software Testing properly, than
with simply building Software Testing that functions correctly.
Software Testing Myths
• Software Testing standards provide Software Testing engineers with all
the guidance they need. The reality is the standards may be outdated
and rarely referred to.
• People with modern computers have all the Software Testing
development tools. The reality is that CASE tools are more important
than hardware to producing high quality Software Testing, yet they are
rarely used effectively.
• Adding people is a good way to catch up when a project is behind
schedule. The reality is that adding people only helps the project
schedule when is it done in a planned, well-coordinated manner.
• Giving Software Testing projects to outside parties to develop solves
Software Testing project management problems. The reality is people
who can’t manage internal Software Testing development problems
277 http://www.certmagic.com
BH0-003
will struggle to manage or control the external development of
Software Testing too.
• A general statement of objectives from the customer is all that is
needed to begin a Software Testing project. The reality is without
constant communication between the customer and the developers it
is impossible to build a Software Testing product that meets the
customer's real needs.
• Project requirements change continually and change is easy to
accommodate in the Software Testing design. The reality is that every
change has far-reaching and unexpected consequences. Changes to
Software Testing requirements must be managed very carefully to
keep a Software Testing project on time and under budget.
• Once a program is written, the Software Testing engineer's work is
finished. The reality is that maintaining a piece of Software Testing is
never done, until the Software Testing product is retired from service.
• There is no way to assess the quality of a piece of Software Testing
until it is actually running on some machine. The reality is that one of
the most effective quality assurance practices (formal technical
reviews) can be applied to any Software Testing design product and
can serve as a quality filter very early in the product life cycle.
• The only deliverable from a successful Software Testing project is the
working program. The reality is the working program is only one of
several deliverables that arise from a well-managed Software Testing
project. The documentation is also important since it provides a basis
for Software Testing support after delivery.
• Software Testing engineering is all about the creation of large and
unnecessary documentation. The reality is that Software Testing
engineering is concerned with creating quality. This means doing
things right the first time and not having to create deliverables needed
to complete or maintain a Software Testing product. This practice
usually leads to faster delivery times and shorter development cycles.
278 http://www.certmagic.com
BH0-003
Chapter 2 - The Process
Overview
• The roadmap to building high quality Software Testing products is
Software Testing process.
• Software Testing processes are adapted to meet the needs of Software
Testing engineers and managers as they undertake the development
of a Software Testing product.
• A Software Testing process provides a framework for managing
activities that can very easily get out of control.
• Different projects require different Software Testing processes.
• The Software Testing engineer's work products (programs,
documentation, data) are produced as consequences of the activities
defined by the Software Testing process.
• The best indicators of how well a Software Testing process has worked
are the quality, timeliness, and long-term viability of the resulting
Software Testing product.
Software Testing Engineering
• Software Testing engineering encompasses a process, management
techniques, technical methods, and the use of tools.
Generic Software Testing Engineering Phases
• Definition phase - focuses on what (information engineering, Software
Testing project planning, requirements analysis).
• Development phase - focuses on how (Software Testing design, code
generation, Software Testing).
• Support phase - focuses on change (corrective maintenance, adaptive
maintenance, perfective maintenance, preventative maintenance).
Software Testing Engineering Umbrella Activities
• Software Testing project tracking and control
• Formal technical reviews
• Software Testing quality assurance
• Software Testing configuration management
• Document preparation and production
• Reusability management
279 http://www.certmagic.com
BH0-003
• Measurement
• Risk management
Common Process Framework
• Software Testing engineering work tasks
• Project milestones
• Work products
• Quality assurance points
Software Testing Engineering Institute (SEI) Capability Maturity Model (CMM)
• Level 1: Initial (ad hoc Software Testing processes)
• Level 2: Repeatable (able to repeat earlier successes)
• Level 3: Defined (management and engineering processes
documented, standardized, and integrated into organization-wide
Software Testing process)
• Level 4; Managed (Software Testing process and products are
quantitatively understood and controlled using detailed measures)
• Level 5: Optimizing (continuous process improvement is enabled by
quantitative feedback from the process and testing innovative ideas)
Software Testing Process Models
• Linear Sequential Model (old fashioned but reasonable approach when
requirements are well understood)
• Prototyping Model (good first step when customer has a legitimate
need, but is clueless about the details, developer needs to resist
pressure to extend a rough prototype into a production product)
• Rapid Application and Development (RAD) Model (makes heavy use of
reusable Software Testing components with an extremely short
development cycle)
• Incremental Model (delivers Software Testing in small but usable
pieces, each piece builds on pieces already delivered)
• Spiral Model (couples iterative nature of prototyping with the
controlled and systematic aspects of the linear sequential model)
• Win-Win Spiral Model (eliciting Software Testing requirements defined
through negotiation between customer and developer, where each
party attempts to balance technical and business constraints)
• Concurrent Development Model (similar to spiral model often used in
development of client/server applications)
280 http://www.certmagic.com
BH0-003
• Component-Based Development (spiral model variation in which
applications are built from prepackaged Software Testing components
called classes)
• Formal Methods Model (rigorous mathematical notation used to
specify, design, and verify computer-based systems)
• Fourth Generation (4GT) Techniques (Software Testing tool is used to
generate the source code for a Software Testing system from a high
level specification representation)
Chapter 3 - Project Management Concepts
Overview
• Project management involves the planning, monitoring, and control of
people, process, and events that occur during Software Testing
development.
• Everyone manages, but the scope of each person's management
activities varies according his or her role in the project.
• Software Testing needs to be managed because it is a complex
undertaking with a long duration time.
• Managers must focus on the fours P's to be successful (people,
product, process, and project).
• A project plan is a document that defines the four P's in such a way as
to ensure a cost effective, high quality Software Testing product.
• The only way to be sure that a project plan worked correctly is by
observing that a high quality product was delivered on time and under
budget.
Management Spectrum
• People (recruiting, selection, performance management, training,
compensation, career development, organization, work design,
team/culture development)
• Product (product objectives, scope, alternative solutions, constraint
tradeoffs)
• Process (framework activities populated with tasks, milestones, work
products, and QA points)
• Project (planning, monitoring, controlling)
People
• Players (senior managers, technical managers, practitioners,
customers, end-users)
• Team leadership model (motivation, organization, skills)
281 http://www.certmagic.com
BH0-003
• Characteristics of effective project managers (problem solving,
managerial identity, achievement, influence and team building)
Software Testing Team Organization
• Democratic decentralized (rotating task coordinators and group
consensus)
• Controlled decentralized (permanent leader, group problem solving,
subgroup implementation of solutions)
• Controlled centralized (top level problem solving and internal
coordination managed by team leader)
Factors Affecting Team Organization
• Difficulty of problem to be solved
• Size of resulting program
• Team lifetime
• Degree to which problem can be modularized
• Required quality and reliability of the system to be built
• Rigidity of the delivery date
• Degree of communication required for the project
Coordination and Communication Issues
• Formal, impersonal approaches (e.g. documents, milestones, memos)
• Formal interpersonal approaches (e.g. review meetings, inspections)
• Informal interpersonal approaches (e.g. information meetings,
problem solving)
• Electronic communication (e.g. e-mail, bulletin boards, video
conferencing)
• Interpersonal network (e.g. informal discussion with people other than
project team members)
The Product
• Software Testing scope (context, information objectives, function,
performance)
282 http://www.certmagic.com
BH0-003
• Problem decomposition (partitioning or problem elaboration - focus is
on functionality to be delivered and the process used to deliver it)
The Process
• Process model chosen must be appropriate for the: customers and
developers, characteristics of the product, and project development
environment
• Project planning begins with melding the product and the process
• Each function to be engineered must pass though the set of framework
activities defined for a Software Testing organization
• Work tasks may vary but the common process framework (CPF) is
invariant (project size does not change the CPF)
• The job of the Software Testing engineer is to estimate the resources
required to move each function though the framework activities to
produce each work product
• Project decomposition begins when the project manager tries to
determine how to accomplish each CPF activity
The Project
• Start on the right foot
• Maintain momentum
• Track progress
• Make smart decisions
• Conduct a postmortem analysis
W5HH Principle
• Why is the system being developed?
• What will be done by When?
• Who is responsible for a function?
• Where are they organizationally located?
• How will the job be done technically and managerially?
• How much of each resource is needed?
Critical Practices
• Formal risk management
283 http://www.certmagic.com
BH0-003
• Empirical cost and schedule estimation
• Metric-based project management
• Earned value tracking
• Defect tracking against quality targets
• People-aware program management
Chapter 4 - Software Testing Process and Project Metrics
Overview
• Software Testing process and project metrics are quantitative
measures that enable Software Testing engineers to gain insight into
the efficiency of the Software Testing process and the projects
conducted using the process framework. In Software Testing project
management, we are primarily concerned with productivity and quality
metrics. The four reasons for measuring Software Testing processes,
products, and resources (to characterize, to evaluate, to predict, and
to improve).
Measures, Metrics, and Indicators
• Measure - provides a quantitative indication of the size of some
product or process attribute
• Measurement - is the act of obtaining a measure
• Metric - is a quantitative measure of the degree to which a system,
component, or process possesses a given attribute
284 http://www.certmagic.com
BH0-003
Process and Project Indicators
• Metrics should be collected so that process and product indicators can
be ascertained
• Process indicators enable Software Testing project managers to:
assess project status, track potential risks, detect problem area early,
adjust workflow or tasks, and evaluate team ability to control product
quality
Process Metrics
• Private process metrics (e.g. defect rates by individual or module) are
known only to the individual or team concerned.
• Public process metrics enable organizations to make strategic changes
to improve the Software Testing process.
• Metrics should not be used to evaluate the performance of individuals.
• Statistical Software Testing process improvement helps and
organization to discover where they are strong and where are week.
Project Metrics
• Software Testing project metrics are used by the Software Testing
team to adapt project workflow and technical activities.
• Project metrics are used to avoid development schedule delays, to
mitigate potential risks, and to assess product quality on an on-going
basis.
• Every project should measure its inputs (resources), outputs
(deliverables), and results (effectiveness of deliverables).
Software Testing Measurement
• Direct measures of Software Testing engineering process include cost
and effort.
• Direct measures of the product include lines of code (LOC), execution
speed, memory size, and defects per reporting time period.
• Indirect measures examine the quality of the Software Testing product
itself (e.g. functionality, complexity, efficiency, reliability,
maintainability).
285 http://www.certmagic.com
BH0-003
Size-Oriented Metrics
• Derived by normalizing (dividing) any direct measure (e.g. defects or
human effort) associated with the product or project by LOC.
• Size oriented metrics are widely used but their validity and
applicability is widely debated.
Function-Oriented Metrics
• Function points are computed from direct measures of the information
domain of a business Software Testing application and assessment of
its complexity.
• Once computed function points are used like LOC to normalize
measures for Software Testing productivity, quality, and other
attributes.
• Feature points and 3D function points provide a means of extending
the function point concept to allow its use with real-time and other
engineering applications.
• The relationship of LOC and function points depends on the language
used to implement the Software Testing.
Software Testing Quality Metrics
• Factors assessing Software Testing quality come from three distinct
points of view (product operation, product revision, product
modification).
• Software Testing quality factors requiring measures include
correctness (defects per KLOC), maintainability (mean time to
change), integrity (threat and security), and usability (easy to learn,
easy to use, productivity increase, user attitude).
• Defect removal efficiency (DRE) is a measure of the filtering ability of
the quality assurance and control activities as they are applied through
out the process framework.
Integrating Metrics with Software Testing Process
• Many Software Testing developers do not collect measures.
• Without measurement, it is impossible to determine whether a process
is improving or not.
• Baseline metrics data should be collected from a large, representative
sampling of past Software Testing projects.
286 http://www.certmagic.com
BH0-003
• Getting this historic project data is very difficult, if the previous
developers did not collect data in an on-going manner.
Statistical Process Control
• It is important to determine whether the metrics collected are
statistically valid and not the result of noise in the data.
• Control charts provide a means for determining whether changes in
the metrics data are meaningful or not.
• Zone rules identify conditions that indicate out of control processes
(expressed as distance from mean in standard deviation units).
Metrics for Small Organizations
• Most Software Testing organizations have fewer than 20 Software
Testing engineers.
• Best advice is to choose simple metrics that provide value to the
organization and don’t require a lot of effort to collect.
• Even small groups can expect a significant return on the investment
required to collect metrics, if this activity leads to process
improvement.
Establishing a Software Testing Metrics Program
• Identify business goal
• Identify what you want to know
• Identify subgoals
• Identify subgoal entities and attributes
• Formalize measurement goals
• Identify quantifiable questions and indicators related to subgoals
• Identify data elements needed to be collected to construct the
indicators
• Define measures to be used and create operational definitions for them
• Identify actions needed to implement the measures
• Prepare a plan to implement the measures
287 http://www.certmagic.com
BH0-003
Chapter 5 - Software Testing Project Planning
Overview
• Software Testing planning involves estimating how much time, effort,
money, and resources will be required to build a specific Software
Testing system. After the project scope is determined and the problem
is decomposed into smaller problems, Software Testing managers use
historical project data (as well as personal experience and intuition) to
determine estimates for each. The final estimates are typically
adjusted by taking project complexity and risk into account. The
resulting work product is called a project management plan.
Estimation Reliability Factors
• Project complexity
• Project size
• Degree of structural uncertainty (degree to which requirements have
solidified, the ease with which functions can be compartmentalized,
and the hierarchical nature of the information processed)
• Availability of historical information
Project Planning Objectives
• To provide a framework that enables Software Testing manager to
make a reasonable estimate of resources, cost, and schedule.
• Project outcomes should be bounded by 'best case' and 'worst case'
scenarios.
• Estimates should be updated as the project progresses.
Software Testing Scope
• Describes the data to be processed and produced, control parameters,
function, performance, constraints, external interfaces, and reliability.
288 http://www.certmagic.com
BH0-003
• Often functions described in the Software Testing scope statement are
refined to allow for better estimates of cost and schedule.
Customer Communication and Scope
• Determine the customer's overall goals for the proposed system and
any expected benefits.
• Determine the customer's perceptions concerning the nature if a good
solution to the problem.
• Evaluate the effectiveness of the customer meeting.
Feasibility
• Technical feasibility is not a good enough reason to build a product.
• The product must meet the customer's needs and not be available as
an off-the-shelf purchase.
Estimation of Resources
• Human Resources (number of people required and skills needed to
complete the development project)
• Reusable Software Testing Resources (off-the-shelf components, full-
experience components, partial-experience components, new
components)
• Development Environment (hardware and Software Testing required to
be accessible by Software Testing team during the development
process)
Software Testing Project Estimation Options
• Delay estimation until late in the project.
• Base estimates on similar projects already completed.
• Use simple decomposition techniques to estimate project cost and
effort.
• Use empirical models for Software Testing cost and effort estimation.
• Automated tools may assist with project decomposition and
estimation.
289 http://www.certmagic.com
BH0-003
Decomposition Techniques
• Software Testing sizing (fuzzy logic, function point, standard
component, change)
• Problem-based estimation (using LOC decomposition focuses on
Software Testing functions, using FP decomposition focuses on
information domain characteristics)
• Process-based estimation (decomposition based on tasks required to
complete the Software Testing process framework)
Empirical Estimation Models
• Typically derived from regression analysis of historical Software
Testing project data with estimated person-months as the dependent
variable and KLOC or FP as independent variables.
• Constructive Cost Model (COCOMO) is an example of a static
estimation model.
• The Software Testing Equation is an example of a dynamic estimation
model.
Make-Buy Decision
• It may be more cost effective to acquire a piece of Software Testing
rather than develop it.
• Decision tree analysis provides a systematic way to sort through the
make-buy decision.
• As a rule, outsourcing Software Testing development requires more
skillful management than does in-house development of the same
product.
Automated Estimation Tool Capabilities
• Sizing of project deliverables
• Selecting project activities
• Predicting staffing levels
• Predicting Software Testing effort
• Predicting Software Testing cost
290 http://www.certmagic.com
BH0-003
• Predicting Software Testing schedule
Chapter 6 - Risk Analysis and Management
Overview
• Risks are potential problems that might affect the successful
completion of a Software Testing project. Risks involve uncertainty and
potential losses. Risk analysis and management are intended to help a
Software Testing team understand and manage uncertainty during the
development process. The important thing is to remember that things
291 http://www.certmagic.com
BH0-003
can go wrong and to make plans to minimize their impact when they
do. The work product is called a Risk Mitigation, Monitoring, and
Management Plan (RMMM).
Risk Strategies
• Reactive strategies - very common, also known as fire fighting, project
team sets resources aside to deal with problems and does nothing until
a risk becomes a problem
• Proactive strategies - risk management begins long before technical
work starts, risks are identified and prioritized by importance, then
team builds a plan to avoid risks if they can or minimize them if the
risks turn into problems
Software Testing Risks
• Project risks - threaten the project plan
• Technical risks - threaten product quality and the timeliness of the
schedule
• Business risks - threaten the viability of the Software Testing to be
built (market risks, strategic risks, management risks, budget risks)
• Known risks - predictable from careful evaluation of current project
plan and those extrapolated from past project experience
• Unknown risks - some problems simply occur without warning
Risk Identification
• Product-specific risks - the project plan and Software Testing
statement of scope are examined to identify any special characteristics
of the product that may threaten the project plan
• Generic risks - are potential threats to every Software Testing product
(product size, business impact, customer characteristics, process
definition, development environment, technology to be built, staff size
and experience)
Risk Impact
• Risk components - performance, cost, support, schedule
• Risk impact - negligible, marginal, critical, catastrophic
292 http://www.certmagic.com
BH0-003
• The risk drivers affecting each risk component are classified according
to their impact category and the potential consequences of each
undetected Software Testing fault or unachieved project outcome are
described
Risk Projection (Estimation)
• Establish a scale that reflects the perceived likelihood of each risk
• Delineate the consequences of the risk
• Estimate the impact of the risk on the project and product
• Note the overall accuracy of the risk projection to avoid
misunderstandings
Risk Table Construction
• List all risks in the first column of the table
• Classify each risk and enter the category label in column two
• Determine a probability for each risk and enter it into column three
• Enter the severity of each risk (negligible, marginal, critical,
catastrophic) in column four
• Sort the table by probability and impact value
• Determine the criteria for deciding where the sorted table will be
divided into the first priority concerns and the second priority concerns
• First priority concerns must be managed (a fifth column can be added
to contain a pointer into the RMMM)
Assessing Risk Impact
• Factors affecting risk consequences - nature (types of problems
arising), scope (combines severity with extent of project affected),
timing (when and how long impact is felt)
• If costs are associated with each risk table entry Halstead's risk
exposure metric can be computed (RE = Probability * Cost) and added
to the risk table.
Risk Assessment
• Define referent levels for each project risk that can cause project
termination (performance degradation, cost overrun, support difficulty,
schedule slippage).
293 http://www.certmagic.com
BH0-003
• Attempt to develop a relationship between each risk triple (risk,
probability, impact) and each of the reference levels.
• Predict the set of referent points that define a region of termination,
bounded by a curve or areas of uncertainty.
• Try to predict how combinations of risks will affect a referent level.
Risk Refinement
• Process of restating the risks as a set of more detailed risks that will
be easier to mitigate, monitor, and manage.
• CTC (condition-transition-consequence) format may be a good
representation for the detailed risks (e.g. given that <condition> then
there is a concern that (possibly) <consequence>).
Risk Mitigation, Monitoring, and Management
• Risk mitigation (proactive planning for risk avoidance)
• Risk monitoring (assessing whether predicted risks occur or not,
ensuring risk aversion steps are being properly applied, collect
information for future risk analysis, attempt to determine which risks
caused which problems)
• Risk management and contingency planning (actions to be taken in the
event that mitigation steps have failed and the risk has become a live
problem)
Safety Risks and Hazards
• Risks are also associated with Software Testing failures that occur in
the field after the development project has ended.
• Computers control many mission critical applications in modern times
(weapons systems, flight control, industrial processes, etc.).
• Software Testing safety and hazard analysis are quality assurance
activities that are of particular concern for these types of applications
and are discussed later in the text.
294 http://www.certmagic.com
BH0-003
Risk Information Sheets
• Alternative to RMMM in which each risk is documented individually.
• Often risk information sheets (RIS) are maintained using a database
system.
• RIS components - risk id, date, probability, impact, description,
refinement, mitigation/monitoring, management/contingency/trigger,
status, originator, assigned staff member.
Chapter 7 - Project Scheduling and Tracking
Overview
• The chapter describes the process of building and monitoring
schedules for Software Testing development projects. To build complex
Software Testing systems, many engineering tasks need to occur in
parallel with one another to complete the project on time. The output
from one task often determines when another may begin. It is difficult
to ensure that a team is working on the most appropriate tasks
without building a detailed schedule and sticking to it.
Software Testing Project Scheduling Principles
• Compartmentalization - the product and process must be decomposed
into a manageable number of activities and tasks
• Interdependency - tasks that can be completed in parallel must be
separated from those that must completed serially
• Time allocation - every task has start and completion dates that take
the task interdependencies into account
295 http://www.certmagic.com
BH0-003
• Effort validation - project manager must ensure that on any given day
there are enough staff members assigned to completed the tasks
within the time estimated in the project plan
• Defined Responsibilities - every scheduled task needs to be assigned
to a specific team member
• Defined outcomes - every task in the schedule needs to have a defined
outcome (usually a work product or deliverable)
• Defined milestones - a milestone is accomplished when one or more
work products from an engineering task have passed quality review
Relationship between People and Effort
• Adding people to a project after it is behind schedule often causes the
schedule to slip further
• The relationship between the number of people on a project and
overall productivity is not linear (e.g. 3 people do not produce 3 times
the work of 1 person, if the people have to work in cooperation with
one another)
• The main reasons for using more than 1 person on a project are to get
the job done more rapidly and to improve Software Testing quality.
Project Effort Distribution
Generally accepted guidelines are:
• 02-03 % planning
• 10-25 % requirements analysis
• 20-25 % design
• 15-20 % coding
• 30-40 % testing and debugging
Software Testing Project Types
• Concept development - initiated to explore new business concept or
new application of technology
• New application development - new product requested by customer
• Application enhancement - major modifications to function,
performance, or interfaces (observable to user)
• Application maintenance - correcting, adapting, or extending existing
Software Testing (not immediately obvious to user)
• Reengineering - rebuilding all (or part) of a legacy system
296 http://www.certmagic.com
BH0-003
Software Testing Process Degree of Rigor
• Casual - all framework activities applied, only minimum task set
required (umbrella activities minimized and documentation reduced)
• Structured - all framework and umbrella activities applied (SQA, SCM,
documentation, and measurement tasks are streamlined)
• Strict - full process and umbrella activities applied (high quality
products and robust documentation produced)
• Quick reaction - emergency situation, process framework used, but
only tasks essential to good quality are applied (back filling used to
develop documentation and conduct additional reviews after product is
delivered)
Rigor Adaptation Criteria
• Size of project
• Number of potential users
• Mission criticality
• Application longevity
• Requirement stability
• Ease of customer/developer communication
• Maturity of applicable technology
• Performance constraints
• Embedded/non-embedded characteristics
• Project staffing
• Reengineering factors
Task Set Selector Value
• Computed by scoring rigor adaptation criteria and adjusting the scores
using differential weighting based on project characteristics.
• Once computed the task selector value can be used to select the
appropriate task set (casual, structured, strict) for the project.
• It is OK to choose a less formal degree of rigor when the task selector
value falls in the overlap area between two levels of rigor, unless
project risk is high.
Concept Development Tasks
297 http://www.certmagic.com
BH0-003
• Concept scoping - determine overall project scope
• Preliminary concept planning - establishes development team's ability
to undertake the proposed work
• Technology risk assessment - evaluates the risk associated with the
technology implied by the Software Testing scope
• Proof of concept - demonstrates the feasibility of the technology in the
Software Testing context
• Concept implementation - concept represented in a form that can be
used to sell it to the customer
• Customer reaction to concept - solicits feedback on new technology
from customer
Scheduling
• Scheduling tools should be used to schedule any non-trivial project.
• PERT (program evaluation and review technique) and CPM (critical
path method) are quantitative techniques that allow Software Testing
planners to identify the chain of dependent tasks in the project work
breakdown structure that determine the project duration time.
• Timeline (Gantt) charts enable Software Testing planners to determine
what tasks will be need to be conducted at a given point in time
(based on estimates for effort, start time, and duration for each task).
• The best indicator of progress is the completion and successful review
of a defined Software Testing work product.
• Time-boxing is the practice of deciding a priori the fixed amount of
time that can be spent on each task. When the task's time limit is
exceeded, development moves on to the next task (with the hope that
a majority of the critical work was completed before time ran out).
Earned Value Analysis
• Earned value is a quantitative measure of percent of project completed
so far.
• The total hours to complete the entire project are estimated and each
task is given an earned value based on its estimated percentage
contribution to the total.
Error Tracking
• Allows comparison of current work to past projects and provides a
quantitative indication of the quality of the work being conducted.
• The more quantitative the approach to project tracking and control,
the more likely problems can be anticipated and dealt with in a
proactive manner.
298 http://www.certmagic.com
BH0-003
Chapter 8 - Software Testing Quality Assurance
Overview
• This chapter provides an introduction for Software Testing quality
assurance (SQA). SQA is the concern of every Software Testing
engineer to reduce cost and improve product time-to-market. A
Software Testing Quality Assurance Plan is not merely another name
for a test plan, though test plans are included in an SQA plan. SQA
activities are performed on every Software Testing project. Use of
metrics is an important part of developing a strategy to improve the
quality of both Software Testing processes and work products.
Quality Concepts
• Variation control is the heart of quality control (Software Testing
engineers strive to control the process applied, resources expended,
and end product quality attributes).
• Quality of design - refers to characteristics designers specify for the
end product to be constructed
• Quality of conformance - degree to which design specifications are
followed in manufacturing the product
• Quality control - series of inspections, reviews, and tests used to
ensure conformance of a work product to its specifications
• Quality assurance - consists of the auditing and reporting procedures
used to provide management with data needed to make proactive
decisions
Cost of Quality
• Prevention costs - quality planning, formal technical reviews, test
equipment, training
• Appraisal costs - in-process and inter-process inspection, equipment
calibration and maintenance, testing
• Failure costs - rework, repair, failure mode analysis
• External failure costs - complaint resolution, product return and
replacement, help line support, warranty work
299 http://www.certmagic.com
BH0-003
Chapter 8 - Software Testing Quality Assurance
Overview
• This chapter provides an introduction for Software Testing quality
assurance (SQA). SQA is the concern of every Software Testing
engineer to reduce cost and improve product time-to-market. A
Software Testing Quality Assurance Plan is not merely another name
for a test plan, though test plans are included in an SQA plan. SQA
activities are performed on every Software Testing project. Use of
metrics is an important part of developing a strategy to improve the
quality of both Software Testing processes and work products.
Quality Concepts
• Variation control is the heart of quality control (Software Testing
engineers strive to control the process applied, resources expended,
and end product quality attributes).
• Quality of design - refers to characteristics designers specify for the
end product to be constructed
• Quality of conformance - degree to which design specifications are
followed in manufacturing the product
• Quality control - series of inspections, reviews, and tests used to
ensure conformance of a work product to its specifications
• Quality assurance - consists of the auditing and reporting procedures
used to provide management with data needed to make proactive
decisions
Cost of Quality
• Prevention costs - quality planning, formal technical reviews, test
equipment, training
• Appraisal costs - in-process and inter-process inspection, equipment
calibration and maintenance, testing
• Failure costs - rework, repair, failure mode analysis
• External failure costs - complaint resolution, product return and
replacement, help line support, warranty work
300 http://www.certmagic.com
BH0-003
Total Quality Management
• Kaizen - develop a process that is visible, repeatable, and measurable
• Atarimae hinshitsu - examine the intangibles that affect the process
and work to optimize their impact on the process
• Kansei - examine the way the product is used by the customer with an
eye to improving both the product and the development process
• Miryokuteki hinshitsu - observe product use in the market place to
uncover new product applications and identify new products to develop
Software Testing Quality Assurance
• Conformance to Software Testing requirements is the foundation from
which Software Testing quality is measured.
• Specified standards are used to define the development criteria that
are used to guide the manner in which Software Testing is engineered.
• Software Testing must conform to implicit requirements (ease of use,
maintainability, reliability, etc.) as well as its explicit requirements.
SQA Group Activities
• Prepare SQA plan for the project.
• Participate in the development of the project's Software Testing
process description.
• Review Software Testing engineering activities to verify compliance
with the defined Software Testing process.
• Audit designated Software Testing work products to verify compliance
with those defined as part of the Software Testing process.
• Ensure that any deviations in Software Testing or work products are
documented and handled according to a documented procedure.
• Record any evidence of noncompliance and reports them to
management.
Software Testing Reviews
301 http://www.certmagic.com
BH0-003
• Purpose is to find defects (errors) before they are passed on to
another Software Testing engineering activity or released to the
customer.
• Software Testing engineers (and others) conduct formal technical
reviews (FTR) for Software Testing engineers.
• Using formal technical reviews (walkthroughs or inspections) is an
effective means for improving Software Testing quality.
Formal Technical Reviews
• Involves 3 to 5 people (including reviewers)
• Advance preparation (no more than 2 hours per person) required
• Duration of review meeting should be less than 2 hours
• Focus of review is on a discrete work product
• Review leader organizes the review meeting at the producer's request
• Reviewers ask questions that enable the producer to discover his or
her own error (the product is under review not the producer)
• Producer of the work product walks the reviewers through the product
• Recorder writes down any significant issues raised during the review
• Reviewers decide to accept or reject the work product and whether to
require additional reviews of product or not
Statistical Quality Assurance
• Information about Software Testing defects is collected and
categorized
• Each defect is traced back to its cause
• Using the Pareto principle (80% of the defects can be traced to 20% of
the causes) isolate the "vital few" defect causes
• Move to correct the problems that caused the defects
Software Testing Reliability
• Defined as the probability of failure free operation of a computer
program in a specified environment for a specified time period
• Can be measured directly and estimated using historical and
developmental data (unlike many other Software Testing quality
factors)
• Software Testing reliability problems can usually be traced back to
errors in design or implementation.
302 http://www.certmagic.com
BH0-003
Software Testing Safety
• Defined as a Software Testing quality assurance activity that focuses
on identifying potential hazards that may cause a Software Testing
system to fail.
• Early identification of Software Testing hazards allows developers to
specify design features to can eliminate or at least control the impact
of potential hazards.
• Software Testing reliability involves determining the likelihood that a
failure will occur, while Software Testing safety examines the ways in
which failures may result in conditions that can lead to a mishap.
Mistake-Proofing Software Testing
• Poka-yoke devices are mechanisms that lead to the prevention of a
potential quality problem before it occurs or to the rapid detection of a
quality problem if one is introduced
• Poka-yoke devices are simple, cheap, part of the engineering process,
and are located near the process task where the mistakes occur
ISO Quality Standards
• Quality assurance systems are defined as the organizational structure,
responsibilities, procedures, processes, and resources for
implementing quality management.
• ISO 9000 describes the quality elements that must be present for a
quality assurance system to be compliant with the standard, but it
does not describe how an organization should implement these
elements.
• ISO 9001 is the quality standard that contains 20 requirements that
must be present in an effective Software Testing quality assurance
system.
SQA Plan
• Management section - describes the place of SQA in the structure of
the organization
• Documentation section - describes each work product produced as part
of the Software Testing process
• Standards, practices, and conventions section - lists all applicable
standards/practices applied during the Software Testing process and
any metrics to be collected as part of the Software Testing engineering
work
303 http://www.certmagic.com
BH0-003
• Reviews and audits section - provides an overview of the approach
used in the reviews and audits to be conducted during the project
• Test section - references the test plan and procedure document and
defines test record keeping requirements
• Problem reporting and corrective action section - defines procedures
for reporting, tracking, and resolving errors or defects, identifies
organizational responsibilities for these activities
• Other - tools, SQA methods, change control, record keeping, training,
and risk management
304 http://www.certmagic.com
BH0-003
Chapter 9 - Software Testing Configuration Management
Overview
• Changes are inevitable when Software Testing is built. A primary goal
of Software Testing engineering is to improve the ease with which
changes can be made to Software Testing. Configuration management
is all about change control. Every Software Testing engineer has to be
concerned with how changes made to work products are tracked and
propagated throughout a project. To ensure that quality is maintained
the change process must be audited.
Software Testing Configuration Items
• Computer programs (both source and executable)
• Documentation (both technical and user)
• Data (contained within the program or external to it)
Fundamental Sources of Change
• New business or market conditions dictate changes to product
requirements or business rules
• New customer needs demand modification of data, functionality, or
services
• Business reorganization causes changes in project priorities or
Software Testing engineering team structure
• Budgetary or scheduling constraints cause system to be redefined
Baselines
• A work product becomes a baseline only after it is reviewed and
approved.
• A baseline is a milestone in Software Testing development that is
marked by the delivery of one or more configuration items.
• Once a baseline is established each change request must be evaluated
and verified by a formal procedure before it is processed.
305 http://www.certmagic.com
BH0-003
Software Testing Configuration Management Tasks
• Identification (tracking multiple versions to enable efficient changes)
• Version control (control changes before and after release to customer)
• Change control (authority to approve and prioritize changes)
• Configuration auditing (ensure changes made properly)
• Reporting (tell others about changes made)
Software Testing Configuration Objects
• To control and manage configuration items, each must be named and
managed using an object-oriented approach
• Basic objects are created by Software Testing engineers during
analysis, design, coding, or testing
• Aggregate objects are collections of basic objects and other aggregate
objects
• Configuration object attributes: unique name, description, list of
resources, and a realization (a pointer to a work product for a basic
object or null for an aggregate object)
• An entity-relationship (E-R) diagram can be used to show the
interrelationships among the objects
Version Control
• Combines procedures and tools to manage the different versions of
configuration objects created during the Software Testing process
• An entity is composed of objects at the same revision level
• A variant is a different set of objects at the same revision level and
coexists with other variants
• A new version is defined when major changes have been made to one
or more objects
Change Control
• Change request is submitted and evaluated to assess technical merit
and impact on the other configuration objects and budget
• Change report contains the results of the evaluation
• Change control authority (CCA) makes the final decision on the status
and priority of the change based on the change report
306 http://www.certmagic.com
BH0-003
• Engineering change order (ECO) is generated for each change
approved (describes change, lists the constraints, and criteria for
review and audit)
• Object to be changed is checked-out of the project database subject to
access control parameters for the object
• Modified object is subjected to appropriate SQA and testing procedures
• Modified object is checked-in to the project database and version
control mechanisms are used to create the next version of the
Software Testing
• Synchronization control is used to ensure that parallel changes made
by different people don’t overwrite one another
Software Testing Configuration Audit Questions
• Has the change specified by the ECO been made without
modifications?
• Has an FTR been conducted to assess technical correctness?
• Was the Software Testing process followed and Software Testing
engineering standards applied?
• Do the attributes of the configuration object reflect the change?
• Have the SCM standards for recording and reporting the change been
followed?
• Were all related SCI's properly updated?
Configuration Status Reporting Questions
• What happened?
• Who did it?
• When did it happen?
• What else will be affected by the change?
307 http://www.certmagic.com
BH0-003
Chapter 10 - System Engineering
Overview
• Before Software Testing can be engineered, the system it is part of
must be understood. The overall objective of the system must be
determined, the role of the system elements (hardware, Software
Testing, people, data, etc.) must be identified, and the operational
requirements must be created. A representation (i.e. prototype,
specification, symbolic model) of the system is produced as the result
of the system engineering process. It is important that system
engineering work products be managed using the quality assurance
techniques discussed in Chapter 8.
Systems
308 http://www.certmagic.com
BH0-003
• Don't take a "Software Testing-centric" view of the system; consider
all system elements before focusing on Software Testing.
• Good system engineering begins with a clear understanding of the
"world view" and progressively narrows until technical detail is
understood.
• Complex systems are actually a hierarchy of macro-elements that are
themselves systems.
Computer-Based System Elements
• Software Testing
• Hardware
• People
• Database
• Documentation
• Procedures
System Engineering Hierarchy
• World view
• Domain view
• Element view
• Detailed view
System Modeling
• Define the processes that serve the needs of the view under
consideration
• Represent the process behavior and the assumptions on which the
behavior is modeled
• Explicitly define the exogenous (links between constituents) and
endogenous (links between constituent components) input to the
model
• Represent all linkages (including outputs) required to understand the
view
System Model Restraining Factors
• Assumptions
• Simplifications
• Limitations
309 http://www.certmagic.com
BH0-003
• Constraints
• Preferences
System Simulation
• If simulation capability is not available for a reactive system, project
risk increases.
• Consider using an iterative process model that will allow the delivery
and testing of incrementally more complete products.
Business Process Engineering Hierarchy
• Information Strategy Planning (world view)
• Business Area Analysis (domain view)
• Business System Design (element view - Software Testing engineers)
• Construction and Integration (detailed view - Software Testing
engineers)
Business Process Engineering Architectures
• Data architecture - provides framework for information needs of a
business or business function
• Applications architecture - those system elements that transform
objects within the data architecture for some business purpose
• Technology infrastructure - provides foundation for the data and
application architectures
Product Engineering Hierarchy
• Requirements engineering (world view)
• Component engineering (domain view)
• Analysis and Design modeling (element view - Software Testing
engineers)
• Construction and Integration (detailed view - Software Testing
engineers)
Requirements Engineering
310 http://www.certmagic.com
BH0-003
• Requirements elicitation (find out from customers what the product
objectives are, what is to be done, how the product fits into business
needs, and how the product is used on a day to day basis)
• Requirements analysis and negotiation (requirements are categorized
and organized into subsets, relations among requirements identified,
requirements reviewed for correctness, requirements prioritized based
on customer needs)
• Requirements specification (work product produced describing the
function, performance, and development constraints for a computer-
based system)
• System modeling (system representation that shows relationships
among the system components)
• Requirements validation (examines the specification to ensure
requirement quality and that work products conform to agreed upon
standards)
• Requirements management (set of activities that help project team to
identify, control, and track requirements and changes as project
proceeds)
Traceability Tables
• Features traceability table
• Source traceability table
• Dependency traceability table
• Subsystem traceability table
• Interface traceability table
System Model Template
• User interface
• Input
• Process and control functions
• Output
• Maintenance and self test
Systems Modeling Process
• System Context Diagram (SCD) - top level node in system hierarchy
used to establish the boundary between the system being
implemented (system model template serves as its basis)
• System Flow Diagram (SFD) - refinement of the process and control
functions from SCD, derived by identifying the major subsystems and
311 http://www.certmagic.com
BH0-003
lines of information flow (precursor to Data Flow Diagram discussed in
Chapter 12)
• Initial SFD is becomes the top level node of a hierarchy of more
successively more detailed SFD's
• System Specification - developed by writing narrative description for
each subsystem and definitions for all data that flow between
subsystems
Chapter 11 - Analysis Concepts and Principles
Overview
After system engineering is completed, Software Testing engineers need to
look at the role of Software Testing in the proposed system. Software Testing
requirements analysis is necessary to avoid creating a Software Testing
product that fails to meet the customer's needs. Data, functional, and
behavioral requirements are elicited from the customer and refined to create
a specification that can be used to design the system. Software Testing
requirements work products must be reviewed for clarity, completeness, and
consistency.
Requirements Analysis
• Software Testing engineering task that bridges the gap between
system level requirements engineering and Software Testing design.
• Provides Software Testing designer with a representation of system
information, function, and behavior that can be translated to data,
architectural, and component-level designs.
• Expect to do a little bit of design during analysis and a little bit of
analysis during design.
Software Testing Requirements Analysis Phases
• Problem recognition
• Evaluation and synthesis (focus is on what not how)
• Modeling
• Specification
• Review
312 http://www.certmagic.com
BH0-003
Software Testing Requirements Elicitation
• Customer meetings are the most commonly used technique.
• Use context free questions to find out customer's goals and benefits,
identify stakeholders, gain understanding of problem, determine
customer reactions to proposed solutions, and assess meeting
effectiveness.
• If many users are involved, be certain that a representative cross
section of users is interviewed.
Facilitated Action Specification Techniques (FAST)
• Meeting held at neutral site, attended by both Software Testing
engineers and customers.
• Rules established for preparation and participation.
• Agenda suggested to cover important points and to allow for
brainstorming.
• Meeting controlled by facilitator (customer, developer, or outsider).
• Definition mechanism (flip charts, stickers, electronic device, etc.) is
used.
• Goal is to identify problem, propose elements of solution, negotiate
different approaches, and specify a preliminary set of solution
requirements.
Quality Function Deployment (QFD)
• Translates customer needs into technical Software Testing
requirements.
• Uses customer interviews, observation, surveys, and historical data for
requirements gathering.
• Customer voice table (contains summary of requirements)
• Normal requirements (must be present in product for customer to be
satisfied)
• Expected requirements (things like ease of use or reliability of
operation, that customer assumes will be present in a professionally
developed product without having to request them explicitly)
• Exciting requirements (features that go beyond the customer's
expectations and prove to be very satisfying when they are present)
• Function deployment (used during customer meetings to determine
the value of each function required for system)
• Information deployment (identifies data objects and events produced
and consumed by the system)
• Task deployment (examines the behavior of product within it
environment)
313 http://www.certmagic.com
BH0-003
• Value analysis (used to determine the relative priority of requirements
during function, information, and task deployment)
Use-Cases
• Scenarios that describe how the product will be used in specific
situations.
• Written narratives that describe the role of an actor (user or device) as
interaction with the system occurs.
• Use-cases are designed from the actor's point of view.
• Not all actors can be identified during the first iteration of
requirements elicitation, but it is important to identify the primary
actors before developing the use-cases.
Analysis Principles
• The information domain of the problem must be represented and
understood.
• The functions that the Software Testing is to perform must be defined.
• Software Testing behavior must be represented.
• Models depicting information, function, and behavior must be
partitioned in a hierarchical manner that uncovers detail.
• The analysis process should move from the essential information
toward implementation detail.
Information Domain
• Encompasses all data objects that contain numbers, text, images,
audio, or video.
• Information content or data model (shows the relationships among the
data and control objects that make up the system)
• Information flow (represents the manner in which data and control
objects change as each moves through the system)
• Information structure (representations of the internal organizations of
various data and control items)
Modeling
• Data model (shows relationships among system objects)
314 http://www.certmagic.com
BH0-003
• Functional model (description of the functions that enable the
transformations of system objects)
• Behavioral model (manner in which Software Testing responds to
events from the outside world)
Partitioning
• Process that results in the elaboration of data, function, or behavior.
• Horizontal partitioning is a breadth-first decomposition of the system
function, behavior, or information, one level at a time.
• Vertical partitioning is a depth-first elaboration of the system function,
behavior, or information, one subsytem at a time.
Software Testing Requirements Views
• Essential view - presents the functions to be accomplished and the
information to be processed without regard to implementation.
• Implementation view - presents the real world manifestation of
processing functions and information structures.
• Avoid the temptation to move directly to the implementation view,
assuming that the essence of the problem is obvious.
Software Testing Prototyping
• Throwaway prototyping (prototype only used as a demonstration of
product requirements, finished Software Testing is engineered using
another paradigm)
• Evolutionary prototyping (prototype is refined to build the finished
system)
• Customer resources must be committed to evaluation and refinement
of the prototype.
• Customer must be capable of making requirements decisions in a
timely manner.
Prototyping Methods and Tools
• Fourth generation techniques (4GT tools allow Software Testing
engineer to generate executable code quickly)
315 http://www.certmagic.com
BH0-003
• Reusable Software Testing components (assembling prototype from a
set of existing Software Testing components)
• Formal specification and prototyping environments (can interactively
create executable programs from Software Testing specification
models)
Specification Principles
• Separate functionality from implementation.
• Develop a behavioral model that describes functional responses to all
system stimuli.
• Define the environment in which the system operates and indicate how
the collection of agents will interact with it.
• Create a cognitive model rather than an implementation model.
• Recognize that the specification must be extensible and tolerant of
incompleteness.
• Establish the content and structure of a specification so that it can be
changed easily.
Specification Representation
• Representation format and content should be relevant to the problem.
• Information contained within the specification should be nested.
• Diagrams and other notational forms should be restricted in number
and consistent in use.
• Representations should be revisable.
Specification Review
• Conducted by customer and Software Testing developer.
• Once approved, the specification becomes a contract for Software
Testing development.
• The specification is difficult to test in a meaningful way.
• Assessing the impact of specification changes is hard to do.
Chapter 12 - Analysis Modeling
316 http://www.certmagic.com
BH0-003
The analysis model is the first technical representation of a system. Analysis
modeling uses a combination of text and diagrams to represent Software
Testing requirements (data, function, and behavior) in an understandable
way. Building analysis models helps make it easier to uncover requirement
inconsistencies and omissions. Two types of analysis modeling are commonly
used: structured analysis (discussed in this chapter) and object-oriented
analysis (discussed in Chapter 21). Data modeling uses entity-relationship
diagrams to define data objects, attributes, and relationships. Functional
modeling uses data flow diagrams to show how data are transformed inside
the system. Behavioral modeling uses state transition diagrams to show the
impact of events. Analysis work products must be reviewed for
completeness, correctness, and consistency. The SEPA web site contains
descriptions of several classical analysis techniques (DSSD, JSD, SADT).
Structured Analysis (DeMarco)
• Analysis products must be highly maintainable, especially the Software
Testing requirements specification.
• Problems of size must be dealt with using an effective method of
partitioning.
• Graphics should be used whenever possible.
• Differentiate between the logical (essential) and physical
(implementation) considerations.
• Find something to help with requirements partitioning and document
the partitioning before specification.
• Devise a way to track and evaluate user interfaces.
• Devise tools that describe logic and policy better than narrative text.
Analysis Model Objectives
• Describe what the customer requires.
• Establish a basis for the creation of a Software Testing design.
• Devise a set of requirements that can be validated once the Software
Testing is built.
Analysis Model Elements
• Data dictionary - contains the descriptions of all data objects
consumed or produced by the Software Testing
• Entity relationship diagram (ERD) - depicts relationships between data
objects
317 http://www.certmagic.com
BH0-003
• Data flow diagram (DFD) - provides an indication of how data are
transformed as they move through the system; also depicts functions
that transform the data flow (a function is represented in a DFD using
a process specification or PSPEC)
• State transition diagram (STD) - indicates how the system behaves as
a consequence of external events, states are used to represent
behavior modes. Arcs are labeled with the events triggering the
transitions from one state to another (control information is contained
in control specification or CSPEC)
Data Modeling Elements (ERD)
• Data object - any person, organization, device, or Software Testing
product that produces or consumes information
• Attributes - name a data object instance, describe its characteristics,
or make reference to another data object
• Relationships - indicate the manner in which data objects are
connected to one another
Cardinality and Modality (ERD)
• Cardinality - in data modeling, cardinality specifies how the number of
occurrences of one object are related to the number of occurrences of
another object (1:1, 1:N, M:N)
• Modality - zero (0) for an optional object relationship and one (1) for a
mandatory relationship
Functional Modeling and Information Flow (DFD)
• Shows the relationships of external entities, process or transforms,
data items, and data stores
• DFD's cannot show procedural detail (e.g. conditionals or loops) only
the flow of data through the Software Testing
• Refinement from one DFD level to the next should follow
approximately a 1:5 ratio (this ratio will reduce as the refinement
proceeds)
• To model real-time systems, structured analysis notation must be
available for time continuous data and event processing (e.g. Ward
and Mellor or Hately and Pirbhai)
Behavioral Modeling (STD)
318 http://www.certmagic.com
BH0-003
• State transition diagrams represent the system states and events that
trigger state transitions
• STD's indicate actions (e.g. process activation) taken as a
consequence of a particular event
• A state is any observable mode of behavior
• Hatley and Pirbhai control flow diagrams (CFD) can also be used for
behavioral modeling
Creating Entity Relationship Diagrams
• Customer asked to list "things" that application addresses, these
things evolve into input objects, output objects, and external entities
• Analyst and customer define connections between the objects
• One or more object-relationship pairs is created for each connection
• The cardinality and modality are determined for an object-relationship
pair
• Attributes of each entity are defined
• The entity diagram is reviewed and refined
Creating Data Flow Diagram
• Level 0 data flow diagram should depict the system as a single bubble
• Primary input and output should be carefully noted
• Refinement should begin by consolidating candidate processes, data
objects, and data stores to be represented at the next level
• Label all arrows with meaningful names
• Information flow must be maintained from one level to level
• Refine one bubble at a time
• Write a PSPEC (a "mini-spec" written using English or another natural
language or a program design language) for each bubble in the final
DFD
Creating Control Flow Diagrams
• Begin by stripping all the data flow arrows form the DFD
• Events (solid arrows) and control items (dashed arrows) are added to
the diagram
• Add a window to the CSPEC (contains an STD that is a sequential
specification of the behavior) for each bubble in the final CFD
319 http://www.certmagic.com
BH0-003
Data Dictionary Contents
• Name - primary name for each data or control item, data store, or
external entity
• Alias - alternate names for each data object
• Where-used/how-used - a listing of processes that use the data or
control item and how it is used (e.g. input to process, output from
process, as a store, as an external entity)
• Content description - notation for representing content
• Supplementary information - other data type information, preset
values, restrictions, limitations, etc.
Chapter 13 - Design Concepts and Principles
Overview
A Software Testing design is a meaningful engineering representation of
some Software Testing product that is to be built. A design can be traced to
the customer's requirements and can be assessed for quality against
predefined criteria. During the design process the Software Testing
requirements model is transformed into design models that describe the
details of the data structures, system architecture, interface, and
components. Each design product is reviewed for quality before moving to
the next phase of Software Testing development.
Design Specification Models
• Data design - created by transforming the analysis information model
(data dictionary and ERD) into data structures required to implement
the Software Testing
• Architectural design - defines the relationships among the major
structural elements of the Software Testing, it is derived from the
system specification, the analysis model, and the subsystem
interactions defined in the analysis model (DFD)
320 http://www.certmagic.com
BH0-003
• Interface design - describes how the Software Testing elements
communicate with each other, with other systems, and with human
users; the data flow and control flow diagrams provide much the
necessary information
• Component-level design - created by transforming the structural
elements defined by the Software Testing architecture into procedural
descriptions of Software Testing components using information
obtained from the PSPEC, CSPEC, and STD
Design Guidelines
A design should
• exhibit good architectural structure
• be modular
• contain distinct representations of data, architecture, interfaces, and
components (modules)
• lead to data structures that are appropriate for the objects to be
implemented and be drawn from recognizable design patterns
• lead to components that exhibit independent functional characteristics
• lead to interfaces that reduce the complexity of connections between
modules and with the external environment
• be derived using a reputable method that is driven by information
obtained during Software Testing requirements analysis
Design Principles
The design
• process should not suffer from tunnel vision
• should be traceable to the analysis model
• should not reinvent the wheel
• should minimize intellectual distance between the Software Testing
and the problem as it exists in the real world
• should exhibit uniformity and integration
• should be structured to accommodate change
• should be structured to degrade gently, even with bad data, events, or
operating conditions are encountered
• should be assessed for quality as it is being created
• should be reviewed to minimize conceptual (semantic) errors
321 http://www.certmagic.com
BH0-003
Fundamental Software Testing Design Concepts
• Abstraction - allows designers to focus on solving a problem without
being concerned about irrelevant lower level details (procedural
abstraction - named sequence of events, data abstraction - named
collection of data objects)
• Refinement - process of elaboration where the designer provides
successively more detail for each design component
• Modularity - the degree to which Software Testing can be understood
by examining its components independently of one another
• Software Testing architecture - overall structure of the Software
Testing components and the ways in which that structure provides
conceptual integrity for a system
• Control hierarchy or program structure - represents the module
organization and implies a control hierarchy, but does not represent
the procedural aspects of the Software Testing (e.g. event sequences)
• Structural partitioning - horizontal partitioning defines three partitions
(input, data transformations, and output); vertical partitioning
(factoring) distributes control in a top-down manner (control decisions
in top level modules and processing work in the lower level modules)
• Data structure - representation of the logical relationship among
individual data elements (requires at least as much attention as
algorithm design)
• Software Testing procedure - precise specification of processing (event
sequences, decision points, repetitive operations, data
organization/structure)
• Information hiding - information (data and procedure) contained within
a module is inaccessible to modules that have no need for such
information
Modular Design Method Evaluation Criteria
• Modular decomposability - provides systematic means for breaking
problem into subproblems
• Modular composability - supports reuse of existing modules in new
systems
• Modular understandability - module can be understood as a stand-
alone unit
• Modular continuity - side-effects due to module changes minimized
• Modular protection - side-effects due to processing errors minimized
Control Terminology
322 http://www.certmagic.com
BH0-003
• Span of control (number of levels of control within a Software Testing
product)
• Depth (distance between the top and bottom modules in program
control structure)
• Fan-out or width (number of modules directly controlled by a particular
module)
• Fan-in (number of modules that control a particular module)
• Visibility (set of program components that may be called or used as
data by a given component)
• Connectivity (set of components that are called directly or are used as
data by a given component)
Effective Modular Design
• Functional independence - modules have high cohesion and low
coupling
• Cohesion - qualitative indication of the degree to which a module
focuses on just one thing
• Coupling - qualitative indication of the degree to which a module is
connected to other modules and to the outside world
Design Heuristics for Effective Modularity
• Evaluate the first iteration of the program structure to reduce coupling
and improve cohesion.
• Attempt to minimize structures with high fan-out; strive for fan-in as
structure depth increases.
• Keep the scope of effect of a module within the scope of control for
that module.
• Evaluate module interfaces to reduce complexity, reduce redundancy,
and improve consistency.
• Define modules whose function is predictable and not overly restrictive
(e.g. a module that only implements a single subfunction).
• Strive for controlled entry modules, avoid pathological connection (e.g.
branches into the middle of another module)
323 http://www.certmagic.com
BH0-003
Chapter 14 - Architectural Design
Overview
Architectural design represents the structure of the data and program
components required to build a computer-based system. A number of
architectural "styles" exist. Architectural design begins with data design and
proceeds to the derivation of one or more representations of the architectural
structure of the system. The resulting architectural model encompasses both
the data architecture and the program structure. The architectural model is
subjected to Software Testing quality review like all other design work
products.
Software Testing architecture is a representation that enables a Software
Testing engineer to
• Analyze the effectiveness of the design in meeting stated requirements
• Consider architectural alternatives
• Reduce the risk associated with the construction of the Software
Testing
• Examine the system as a whole
Data Design Principles
• Systematic analysis principles applied to function and behavior should
also be applied to data.
• All data structures and the operations to be performed on each should
be identified.
• Data dictionary should be established and used to define both data and
program design.
• Low level design processes should be deferred until late in the design
process.
• Representations of data structure should be known only to those
modules that must make direct use of the data contained within in the
data structure.
• A library of useful data structures and operations should be developed.
• A Software Testing design and its implementation language should
support the specification and realization of abstract data types.
324 http://www.certmagic.com
BH0-003
Architectural Styles
• Data centered - data store (e.g. file or database) lies at the center of
this architecture and is accessed frequently by other components that
modify data
• Data flow - input data is transformed by a series of computational or
manipulative components into output data
• Call and return - program structure decomposes function into control
hierarchy with main program invokes several subprograms
• Object-oriented - components of system encapsulate data and
operations, communication between components is by message
passing
• Layered - several layers are defined, each accomplishing operations
that progressively become closer to the machine instruction set
Architecture Design Assessment Questions
• How is control managed within the architecture?
• Does a distinct control hierarchy exist?
• How do components transfer control within the system?
• How is control shared among components?
• What is the control topology?
• Is control synchronized or asynchronous?
• How are data communicated between components?
• Is the flow of data continuous or sporadic?
• What is the mode of data transfer?
• Do data components exist? If so what is their role?
• How do functional components interact with data components?
• Are data components active or passive?
• How do data and control interact within the system?
Architecture Trade-off Analysis Method
1. Collect scenarios
2. Elicit requirements, constraints, and environmental description
3. Describe architectural styles/patterns chosen to address scenarios and
requirements (module view, process view, data flow view)
4. Evaluate quality attributes independently (e.g. reliability, performance,
security, maintainability, flexibility, testability, portability, reusability,
interoperability)
5. Identify sensitivity points for architecture (any attributes significantly
affected by variation in the architecture)
6. Critique candidate architectures (from step 3) using the sensitivity
analysis (conducted in step 5)
325 http://www.certmagic.com
BH0-003
Architectural Complexity (similar to coupling)
• Sharing dependencies - represent dependence relationships among
consumers who use the same resource or producers who produce for
the same consumers
• Flow dependencies - represent dependence relationships between
producers and consumers of resources
• Constrained dependencies - represent constraints on the relative flow
among a set of components
Mapping Requirements to Software Testing Architecture in Structured Design
• Establish type of information flow (transform flow - overall data flow is
sequential and flows along a small number of straight line paths;
transaction flow - a single data item triggers information flow along
one of many paths)
• Flow boundaries indicated
• DFD is mapped into program structure
• Control hierarchy defined
• Resultant structure refined using design measures and heuristics
• Architectural description refined and elaborated
Transform Mapping
• Review fundamental system model
• Review and refine data flow diagrams for the Software Testing
• Determine whether the DFD has transform or transaction
characteristics
• Isolate the transform center by specifying incoming and outgoing flow
boundaries
• Perform first level factoring
• Perform second level factoring
• Refine the first iteration architecture using design heuristics for
improved Software Testing quality
Transaction Mapping
• Review fundamental system model
• Review and refine data flow diagrams for the Software Testing
326 http://www.certmagic.com
BH0-003
• Determine whether the DFD has transform or transaction
characteristics
• Identify the transaction center and flow characteristics along each
action path
• Map the DFD to a program structure amenable to transaction
processing
• Factor and refine the transaction structure and the structure of each
action path
• Refine the first iteration architecture using design heuristics for
improved Software Testing quality
Refining Architectural Design
• Processing narrative developed for each module
• Interface description provided for each module
• Local and global data structures are defined
• Design restrictions/limitations noted
• Design reviews conducted
• Refinement considered if required and justified
327 http://www.certmagic.com
BH0-003
Chapter 15 - User Interface Design
Overview
This chapter introduces the principles of user interface design as it relates to
the development of Software Testing products. Proper interface design
begins with careful analysis of the user, the task and the environment. Once
the user's tasks are identified, user scenarios are created and validated.
Good user interfaces are designed, they don't happen by chance. Prototyping
is a common approach to user interface design. Early involvement of the user
in the design process makes him or her more likely to accept the final
product. User interfaces must be field tested and validated prior to general
release.
Place User in Control
• Define interaction in such a way that the user is not forced into
performing unnecessary or undesired actions
• Provide for flexible interaction (users have varying preferences)
• Allow user interaction to be interruptible and reversible
• Streamline interaction as skill level increases and allow customization
of interaction
328 http://www.certmagic.com
BH0-003
• Hide technical internals from the casual user
• Design for direct interaction with objects that appear on the screen
Reduce User Memory Load
• Reduce demands on user's short-term memory
• Establish meaningful defaults
• Define intuitive short-cuts
• Visual layout of user interface should be based on a familiar real world
metaphor
• Disclose information in a progressive fashion
Make Interface Consistent
• Allow user to put the current task into a meaningful context
• Maintain consistency across a family of applications
• If past interaction models have created user expectations, do not
make changes unless there is a good reason to do so
User Interface Design Models
• Design model (incorporates data, architectural, interface, and
procedural representations of the Software Testing)
• User model (end user profiles - novice, knowledgeable intermittent
user, knowledgeable frequent users)
• User's model or system perception (user's mental image of system)
• System image (look and feel of the interface and supporting media)
User Interface Design Process (Spiral Model)
• User, task, and environment analysis and modeling
• Interface design
• Interface construction
• Interface validation
Task Analysis and Modeling
329 http://www.certmagic.com
BH0-003
• Software Testing engineer studies tasks human users must complete
to accomplish their goal in the real world without the computer and
map these into a similar set of tasks that are to be implemented in the
context of the user interface
• Software Testing engineer studies existing specification for computer-
based solution and derives a set of tasks that will accommodate the
user model, design model, and system perception
• Software Testing engineer may devise an object-oriented approach by
observing the objects and actions the user makes use of in the real
world and model the interface objects after their real world
counterparts
Interface Design Activities
• Establish the goals and intentions of each task
• Map each goal/intention to a sequence of specific actions (objects and
methods for manipulating objects)
• Specify the action sequence of tasks and subtasks (user scenario)
• Indicate the state of the system at the time the user scenario is
performed
• Define control mechanisms
• Show how control mechanisms affect the state of the system
• Indicate how the user interprets the state of the system from
information provided through the interface
Interface Design Issues
• System response time (time between the point at which user initiates
some control action and the time when the system responds)
• User help facilities (integrated, context sensitive help versus add-on
help)
• Error information handling (messages should be non-judgmental,
describe problem precisely, and suggest valid solutions)
• Command labeling (based on user vocabulary, simple grammar, and
have consistent rules for abbreviation)
User Interface Evaluation Cycle
1. Preliminary design
2. Build first interface prototype
3. User evaluates interface
4. Evaluation studied by designer
330 http://www.certmagic.com
BH0-003
5. Design modifications made
6. Build next prototype
7. If interface is not complete then go to step 3
User Interface Design Evaluation Criteria
• Length and complexity of written interface specification provide an
indication of amount of learning required by system users
• Number of user tasks and the average number of actions per task
provide an indication of interaction time and overall system efficiency
• Number of tasks, actions, and system states in the design model
provide an indication of the memory load required of system users
• Interface style, help facilities, and error handling protocols provide a
general indication of system complexity and the degree of acceptance
by the users
Chapter 16 - Component Level Design
Overview
The purpose of component level design is to translate the design model into
operational Software Testing. Component level design occurs after the data,
architectural, and interface designs are established. Component-level design
represents the Software Testing in a way that allows the designer to review it
for correctness and consistency, before it is built. The work product produced
331 http://www.certmagic.com
BH0-003
is the procedural design for each Software Testing component, represented
using graphical, tabular, or text-based notation.
Structured Programming
• Each block of code has a single entry at the top
• Each block of code has a single exit at the bottom
• Only three control structures are required: sequence, condition (if-
then-else), and repetition (looping)
• Reduces program complexity by enhancing readability, testability, and
maintainability
Design Notation
• Flowcharts (arrows for flow of control, diamonds for decisions,
rectangles for processes)
• Box diagrams (also known as Nassi-Scheidnerman charts - process
boxes subdivided to show conditional and repetitive steps)
• Decision table (subsets of system conditions and actions are
associated with each other to define the rules for processing inputs
and events)
• Program Design Language (PDL - structured English or pseudocode
used to describe processing details)
Program Design Language Characteristics
• Fixed syntax with keywords providing for representation of all
structured constructs, data declarations, and module definitions
• Free syntax of natural language for describing processing features
• Data declaration facilities for simple and complex data structures
• Subprogram definition and invocation facilities
Design Notation Assessment Criteria
• Modularity (notation supports development of modular Software
Testing)
• Overall simplicity (easy to learn, easy to use, easy to write)
• Ease of editing (easy to modify design representation when changes
are necessary)
332 http://www.certmagic.com
BH0-003
• Machine readability (notation can be input directly into a computer-
based development system)
• Maintainability (maintenance of the configuration usually involves
maintenance of the procedural design representation)
• Structure enforcement (enforces the use of structured programming
constructs)
• Automatic processing (allows the designer to verify the correctness
and quality of the design)
• Data representation (ability to represent local and global data directly)
• Logic verification (automatic logic verification improves testing
adequacy)
• Easily converted to program source code (makes code generation
quicker)
333 http://www.certmagic.com
BH0-003
Chapter 17 – Software Testing Techniques
Overview
The importance of Software Testing to Software Testing quality cannot be
overemphasized. Once source code has been generated, Software Testing
must be tested to allow errors to be identified and removed before delivery
to the customer. While it is not possible to remove every error in a large
Software Testing package, the Software Testing engineer’s goal is to remove
as many as possible early in the Software Testing development cycle. It is
important to remember that testing can only find errors, it cannot prove that
a program is bug free. Two basic test techniques involve testing module
input/output (black-box) and exercising internal logic of Software Testing
components (white-box). Formal technical reviews by themselves cannot find
allow Software Testing defects, test data must also be used. For large
Software Testing projects, separate test teams may be used to develop and
execute the set of test cases used in testing. Testing must be planned and
designed. The SEPA web site contains the template for a generic test plan.
Software Testing Objectives
• Testing is the process of executing a program with the intent of finding
errors.
• A good test case is one with a high probability of finding an as-yet
undiscovered error.
• A successful test is one that discovers an as-yet-undiscovered error.
Software Testing Principles
• All tests should be traceable to customer requirements.
• Tests should be planned long before testing begins.
334 http://www.certmagic.com
BH0-003
• The Pareto principle (80% of all errors will likely be found in 20% of
the code) applies to Software Testing.
• Testing should begin in the small and progress to the large.
• Exhaustive testing is not possible.
• To be most effective, testing should be conducted by an independent
third party.
Software Testing Testability Checklist
• Operability (the better it works the more efficiently it can be tested)
• Observabilty (what you see is what you test)
• Controllability (the better Software Testing can be controlled the more
testing can be automated and optimized)
• Decomposability (by controlling the scope of testing, the more quickly
problems can be isolated and retested intelligently)
• Simplicity (the less there is to test, the more quickly we can test)
• Stability (the fewer the changes, the fewer the disruptions to testing)
• Understandability (the more information known, the smarter the
testing)
Good Test Attributes
• A good test has a high probability of finding an error.
• A good test is not redundant.
• A good test should be best of breed.
• A good test should not be too simple or too complex.
Test Case Design Strategies
• Black-box or behavioral testing (knowing the specified function a
product is to perform and demonstrating correct operation based
solely on its specification without regard for its internal logic)
• White-box or glass-box testing (knowing the internal workings of a
product, tests are performed to check the workings of all independent
logic paths)
Basis Path Testing
• White-box technique usually based on the program flow graph
335 http://www.certmagic.com
BH0-003
• The cyclomatic complexity of the program computed from its flow
graph using the formula V(G) = E – N + 2 or by counting the
conditional statements in the PDL representation and adding 1
• Determine the basis set of linearly independent paths (the cardinality
of this set id the program cyclomatic complexity)
• Prepare test cases that will force the execution of each path in the
basis set.
Control Structure Testing
• White-box techniques focusing on control structures present in the
Software Testing
• Condition testing (e.g. branch testing) focuses on testing each decision
statement in a Software Testing module, it is important to ensure
coverage of all logical combinations of data that may be processed by
the module (a truth table may be helpful)
• Data flow testing selects test paths based according to the locations of
variable definitions and uses in the program (e.g. definition use
chains)
• Loop testing focuses on the validity of the program loop constructs
(i.e. simple loops, concatenated loops, nested loops, unstructured
loops), involves checking to ensure loops start and stop when they are
supposed to (unstructured loops should be redesigned whenever
possible)
Graph-based Testing Methods
• Black-box methods based on the nature of the relationships (links)
among the program objects (nodes), test cases are designed to
traverse the entire graph
• Transaction flow testing (nodes represent steps in some transaction
and links represent logical connections between steps that need to be
validated)
• Finite state modeling (nodes represent user observable states of the
Software Testing and links represent transitions between states)
• Data flow modeling (nodes are data objects and links are
transformations from one data object to another)
• Timing modeling (nodes are program objects and links are sequential
connections between these objects, link weights are required
execution times)
336 http://www.certmagic.com
BH0-003
Equivalence Partitioning
• Black-box technique that divides the input domain into classes of data
from which test cases can be derived
• An ideal test case uncovers a class of errors that might require many
arbitrary test cases to be executed before a general error is observed
• Equivalence class guidelines:
1. If input condition specifies a range, one valid and two invalid
equivalence classes are defined
2. If an input condition requires a specific value, one valid and two invalid
equivalence classes are defined
3. If an input condition specifies a member of a set, one valid and one
invalid equivalence class is defined
4. If an input condition is Boolean, one valid and one invalid equivalence
class is defined
Boundary Value Analysis
• Black-box technique that focuses on the boundaries of the input
domain rather than its center
• BVA guidelines:
1. If input condition specifies a range bounded by values a and b, test
cases should include a and b, values just above and just below a and b
2. If an input condition specifies and number of values, test cases should
be exercise the minimum and maximum numbers, as well as values
just above and just below the minimum and maximum values
3. Apply guidelines 1 and 2 to output conditions, test cases should be
designed to produce the minimum and maxim output reports
4. If internal program data structures have boundaries (e.g. size
limitations), be certain to test the boundaries
Comparison Testing
• Black-box testing for safety critical systems in which independently
developed implementations of redundant systems are tested for
conformance to specifications
• Often equivalence class partitioning is used to develop a common set
of test cases for each implementation
337 http://www.certmagic.com
BH0-003
Orthogonal Array Testing
• Black-box technique that enables the design of a reasonably small set
of test cases that provide maximum test coverage
• Focus is on categories of faulty logic likely to be present in the
Software Testing component (without examining the code)
• Priorities for assessing tests using an orthogonal array
1. Detect and isolate all single mode faults
2. Detect all double mode faults
3. Mutimode faults
Specialized Testing
• Graphical user interfaces (see Chapter 31 and the SEPA web checklist)
• Client/server architectures (see Chapter 28)
• Documentation and help facilities (see Chapter 8 and Chapter 15)
• Real-time systems
1. Task testing (test each time dependent task independently)
2. Behavioral testing (simulate system response to external events)
3. Intertask testing (check communications errors among tasks)
4. System testing (check interaction of integrated system Software
Testing and hardware)
Chapter 18 - Software Testing Strategies
Overview
338 http://www.certmagic.com
BH0-003
This chapter describes several approaches to testing Software Testing.
Software Testing must be planned carefully to avoid wasting development
time and resources. Testing begins "in the small" and progresses "to the
large". Initially individual components are tested using white box and black
box techniques. After the individual components have been tested and added
to the system, integration testing takes place. Once the full Software Testing
product is completed, system testing is performed. The Test Specification
document should be reviewed like all other Software Testing engineering
work products. A sample Test Specification document appears on the SEPA
web site.
Strategic Approach to Software Testing
• Testing begins at the component level and works outward toward the
integration of the entire computer-based system.
• Different testing techniques are appropriate at different points in time.
• The developer of the Software Testing conducts testing and may be
assisted by independent test groups for large projects.
• The role of the independent tester is to remove the conflict of interest
inherent when the builder is testing his or her own product.
• Testing and debugging are different activities.
• Debugging must be accommodated in any testing strategy.
• Make a distinction between verification (are we building the product
right?) and validation (are we building the right product?)
Strategic Testing Issues
• Specify product requirements in a quantifiable manner before testing
starts.
• Specify testing objectives explicitly.
• Identify the user classes of the Software Testing and develop a profile
for each.
• Develop a test plan that emphasizes rapid cycle testing.
• Build robust Software Testing that is designed to test itself (e.g. uses
anitbugging).
• Use effective formal reviews as a filter prior to testing.
• Conduct formal technical reviews to assess the test strategy and test
cases.
Unit Testing
• Black box and white box testing.
339 http://www.certmagic.com
BH0-003
• Module interfaces are tested for proper information flow.
• Local data are examined to ensure that integrity is maintained.
• Boundary conditions are tested.
• Basis path testing should be used.
• All error handling paths should be tested.
• Drivers and/or stubs need to be developed to test incomplete Software
Testing.
Integration Testing
• Top-down integration testing
1. Main control module used as a test driver and stubs are substitutes for
components directly subordinate to it.
2. Subordinate stubs are replaced one at a time with real components
(following the depth-first or breadth-first approach).
3. Tests are conducted as each component is integrated.
4. On completion of each set of tests and other stub is replaced with a
real component.
5. Regression testing may be used to ensure that new errors not
introduced.
• Bottom-up integration testing
1. Low level components are combined in clusters that perform a specific
Software Testing function.
2. A driver (control program) is written to coordinate test case input and
output.
3. The cluster is tested.
4. Drivers are removed and clusters are combined moving upward in the
program structure.
• Regression testing (check for defects propagated to other modules by
changes made to existing program)
1. Representative sample of existing test cases is used to exercise all
Software Testing functions.
2. Additional test cases focusing Software Testing functions likely to be
affected by the change.
3. Tests cases that focus on the changed Software Testing components.
• Smoke testing
1. Software Testing components already translated into code are
integrated into a build.
340 http://www.certmagic.com
BH0-003
2. A series of tests designed to expose errors that will keep the build
from performing its functions are created.
3. The build is integrated with the other builds and the entire product is
smoke tested daily (either top-down or bottom integration may be
used).
General Software Testing Test Criteria
• Interface integrity (internal and external module interfaces are tested
as each module or cluster is added to the Software Testing)
• Functional validity (test to uncover functional defects in the Software
Testing)
• Information content (test for errors in local or global data structures)
• Performance (verify specified performance bounds are tested)
Validation Testing
• Ensure that each function or performance characteristic conforms to its
specification.
• Deviations (deficiencies) must be negotiated with the customer to
establish a means for resolving the errors.
• Configuration review or audit is used to ensure that all elements of the
Software Testing configuration have been properly developed,
cataloged, and documented to allow its support during its maintenance
phase.
Acceptance Testing
• Making sure the Software Testing works correctly for intended user in
his or her normal work environment.
• Alpha test (version of the complete Software Testing is tested by
customer under the supervision of the developer at the developer’s
site)
• Beta test (version of the complete Software Testing is tested by
customer at his or her own site without the developer being present)
System Testing
• Recovery testing (checks the system’s ability to recover from failures)
341 http://www.certmagic.com
BH0-003
• Security testing (verifies that system protection mechanism prevent
improper penetration or data alteration)
• Stress testing (program is checked to see how well it deals with
abnormal resource demands – quantity, frequency, or volume)
• Performance testing (designed to test the run-time performance of
Software Testing, especially real-time Software Testing)
Debugging
• Debugging (removal of a defect) occurs as a consequence of
successful testing.
• Some people are better at debugging than others.
• Common approaches:
1. Brute force (memory dumps and run-time traces are examined for
clues to error causes)
2. Backtracking (source code is examined by looking backwards from
symptom to potential causes of errors)
3. Cause elimination (uses binary partitioning to reduce the number of
locations potential where errors can exist)
Bug Removal Considerations
• Is the cause of the bug reproduced in another part of the program?
• What "next bug" might be introduced by the fix that is being
proposed?
• What could have been done to prevent this bug in the first place?
342 http://www.certmagic.com
BH0-003
Chapter 19 - Technical Metrics for Software Testing
Overview
This chapter describes the use of technical metrics in the Software Testing
quality assurance process. Earlier in the text the use of metrics in project
management was discussed. Software Testing engineers use technical
metrics to help them assess the quality of the design and construction
Software Testing product being built. Technical metrics provide Software
Testing engineers with a basis to conduct analysis, design, coding, and
testing more objectively. Qualitative criteria for assessing Software Testing
quality are not always sufficient by themselves. The process of using
technical metrics begins by deriving the Software Testing measures and
metrics that are appropriate for the Software Testing representation under
consideration. Then data are collected and metrics are computed. The
metrics are computed and compared to pre-established guidelines and
historical data. The results of these comparisons are used to guide
343 http://www.certmagic.com
BH0-003
modifications made to work products arising from analysis, design, coding, or
testing.
Software Testing Quality Principles
• Conformance to Software Testing requirements is the foundation from
which quality is measured.
• Specified standards define a set of development criteria that guide the
manner in which Software Testing is engineered.
• Software Testing quality is suspect when a Software Testing product
conforms to its explicitly stated requirements and fails to conform to
the customer's implicit requirements (e.g. ease of use).
McCall's Quality Factors
• Product Operation
• Correctness
• Efficiency
• Integrity
• Reliability
• Usability
• Product Revision
• Flexibility
• Maintainability
• Testability
• Product Transition
• Interoperability
• Portability
• Reusability
McCall’s Software Testing Metrics
• Auditability
• Accuracy
• Communication commonality
• Completeness
344 http://www.certmagic.com
BH0-003
• Consistency
• Data commonality
• Error tolerance
• Execution efficiency
• Expandability
• Generality
• Hardware independence
• Instrumentation
• Modularity
• Operability
• Security
• Self-documentation
• Simplicity
• Software Testing system independence
• Traceability
• Training
FURPS Quality Factors
• Functionality
• Usability
• Reliability
• Performance
• Supportability
ISO 9126 Quality Factors
• Functionality
• Reliability
• Usability
• Efficiency
• Maintainability
• Portability
Measurement Process Activities
• Formulation – derivation of Software Testing measures and metrics
appropriate for Software Testing representation being considered
• Collection – mechanism used to accumulate the data used to derive
the Software Testing metrics
• Analysis – computation of metrics
345 http://www.certmagic.com
BH0-003
• Interpretation – evaluation of metrics that results in gaining insight
into quality of the work product
• Feedback – recommendations derived from interpretation of the
metrics is transmitted to the Software Testing development team
Formulation Principles for Technical Metrics
• The objectives of measurement should be established before collecting
any data.
• Each metric is defined in an unambiguous manner.
• Metrics should be based on a theory that is valid for the application
domain.
• Metrics should be tailored to accommodate specific products and
processes.
Software Testing Metric Attributes
• Simple and computable
• Empirically and intuitively persuasive
• Consistent and objective
• Consistent in use of units and measures
• Programming language independent
• Provides an effective mechanism for quality feedback
Representative Analysis Metrics
• Function-based metrics
• Bang metric (function strong or data strong)
• Specification quality metrics (Davis)
Representative Design Metrics
• Architectural design metrics
• Structural complexity (based on module fanout)
• Data complexity (based on module interface inputs and outputs)
• System complexity (sum of structural and data complexity)
• Morphology (number of nodes and arcs in program graph)
• Design structure quality index (DSQI)
346 http://www.certmagic.com
BH0-003
• Component-level design metrics
• Cohesion metrics (data slice, data tokens, glue tokens, superglue
tokens, stickiness)
• Coupling metrics (data and control flow, global, environmental)
• Complexity metrics (e.g. cyclomatic complexity)
• Interface design metrics (e.g. layout appropriateness)
Halstead’s Software Testing Science (Source Code Metrics)
• Overall program length
• Potential minimum algorithm volume
• Actual algorithm volume (number of bits used to specify program)
• Program level (Software Testing complexity)
• Language level (constant for given language)
Testing Metrics
• Metrics that predict the likely number of tests required during various
testing phases
• Metrics that focus on test coverage for a given component
Maintenance Metrics
• Software Testing Maturity Index (IEEE Standard 982.1-1988)
• SMI approaches 1.0 as product begins to stabilize
Chapter 20 Object-Oriented Concepts and Principles
Overview
This chapter provides an introduction to object-oriented programming and
management principles for object-oriented projects. Object-oriented
Software Testing engineering process is similar to that found in the rapid
prototyping or spiral paradigms. Even though, object-oriented Software
Testing engineering follows the same steps as the conventional approach
(analysis, design, implementation, and testing) it is harder to separate them
347 http://www.certmagic.com
BH0-003
into discrete activities. The next 3 chapters deal with the topics of object-
oriented analysis, object-oriented design, and object-oriented testing.
Evolutionary Object-Oriented Process Model
• Customer communication
• Planning
• Risk analysis
• Engineering construction and analysis
• Identify candidate classes
• Look-up classes in library
• Extract classes if available
• Engineer classes if not available
o Object-oriented analysis (OOA)
o Object-oriented design (OOD)
o Object-oriented programming (OOP)
o Object-oriented testing (OOT)
• Put new classes in library
• Construct Nth iteration of the system
• Customer evaluation
Object-Oriented Concepts
• Objects - encapsulates both data (attributes) and data manipulation
functions (called methods, operations, and services)
• Class - generalized description (template or pattern) that describes a
collection of similar objects
• Superclass - a collection of objects
• Subclass - an instance of a class
• Class hierarchy - attributes and methods of a superclass are inherited
by its subclasses
• Messages - the means by which objects exchange information with one
another
• Inheritance - provides a means for allowing subclasses to reuse
existing superclass data and procedures; also provides mechanism for
propagating changes
• Polymorphism - a mechanism that allows several objects in an class
hierarchy to have different methods with the same name (instances of
348 http://www.certmagic.com
BH0-003
each subclass will be free to respond to messages by calling their own
version of the method)
Advantages of Object-Oriented Architectures
• Implementation details of data and procedures and hidden from the
outside world (reduces the propagation of side effects when changes
are made).
• Data structures and operators are merged in single entity or class (this
facilitates reuse)
• Interfaces among encapsulated objects are simplified (system coupling
is reduced since object needs not be concerned about the details of
internal data structures)
Class Construction Options
• Build new class from scratch without using inheritance
• Use inheritance to create new class from existing class contains most
of the desired attributes and operations
• Restructure the class hierarchy so that the required attributes and
operations can be inherited by the newly created class
• Override some attributes or operations in an existing class and use
inheritance to create a new class with (specialized) private versions of
these attributes and operations.
349 http://www.certmagic.com
BH0-003
Chapter 21 Object-Oriented Analysis
Overview
This chapter describes the process of creating an object-oriented analysis
(OOA) model for Software Testing development projects. The OOA model is
composed of graphical or text-based representations that define class
attributes, relationships, behaviors, and inter-class communication. OOA
begins with scenario-based descriptions (use cases) of how actors (people,
machines, systems) in the problem space interact with the product to be
built. Class-Responsibility-Collaborator modeling translates use-case
information into representations of classes and their collaborations. An
object-relationship model can be built from the collaborator network. The
object-behavior model is represented using a state transition diagram. The
OOA model needs to be reviewed for quality like any other Software Testing
engineering product.
OOA Tasks
1. Basic user requirements must be communicated between the customer
and the Software Testing engineer.
2. Classes must be identified (e.g. define attributes and methods)
3. Specify class hierarchy
4. Represent object-to-object relationships
5. Model object behavior
6. Reapply 1 through 5 iteratively until model is complete
OOA Generic Steps
• Elicit customer requirements for system
• Identify scenarios or use cases
• Select classes and objects using basic requirements as a guide
• Identify attributes and operations for each system object
• Define structures and hierarchies that organize classes
• Build object-relationship model
• Build object-behavior model
350 http://www.certmagic.com
BH0-003
• Review OOA model against use-cases (scenarios)
Unified Modeling Language Perspectives
• User model view (describes usage scenarios from the end-user's
perspective)
• Structural model view (static structure of data and functionality is
modeled - classes, object, relationships)
• Behavioral model view (represents dynamic system aspects -
interactions or collaborations between structural elements in the user
and structural models)
• Implementation model view (representing the structural and
behavioral aspects of the system as they are to be built)
• Environment model view (representation of the structural and
behavioral aspects of the environment in which the system will be
implemented)
Domain Analysis Activities
• Define the domain to be investigated
• Categorize the items extracted from the domain
• Collect a representative sample of applications in the domain
• Analyze each application in the sample
• Identify candidate reusable objects
• Indicate reasons the objects may be reused
• Define adaptations of the objects that may be reused
• Estimate percentage of applications in the domain that might make
reuse of the objects
• Identify objects by name and use configuration management
techniques to control them
• Develop an analysis model for the objects
OOA Model Generic Components
• Static view of semantic classes (classes based on semantics of
customer requirements)
• Static view of attributes (attributes describe classes and suggest
operations relevant to classes)
351 http://www.certmagic.com
BH0-003
• Static view of relationships (represent relationships in a way that
allows identification of operations and the design of a messaging
approach)
• Static view of behaviors (behaviors accommodating system usage
scenarios implemented by sequences of operations)
• Dynamic view of communication (among objects based on events that
cause state transitions)
• Dynamic view of control and time (describe the nature and timing of
events causing state transitions)
Use Case Objectives
• Define the functional and operational requirements of system by
defining a scenario of usage agreed upon by the end-user and
Software Testing engineer
• Provide an unambiguous description of how the end-user and system
interact with one another
• Provide a basis for validation testing
Class-Responsibility-Collaborator (CRC) Modeling
• Develop a set of index cards that represent the system classes
• One class per card
• Cards are divide into three sections (class name, class responsibilities,
class collaborators)
• Once a complete CRC card set is developed it is reviewed examining
the usage scenarios
Criteria for Inclusion of a Class on a CRC Card
• Class information should be retained
• Provides needed services
• Contains multiple attributes
• Common set of attributes apply to all object occurrences
• Common set of operations apply to all object occurrences
• External entities that produce or consume information
Allocating Responsibilities to Classes
• Distribute system intelligence evenly
• State each responsibility as generally as possible
352 http://www.certmagic.com
BH0-003
• Information and its related behaviors should reside within the same
class
• Localize all information about one thing in a single class
• Share responsibilities among related classes when appropriate
Collaborators
• Any time a class cannot fulfill a responsibility on its own it needs to
interact with another class
• A server object interacts with a client object to fulfill some
responsibility
Reviewing CRC Models
• Each review participant is given a subset of the CRC cards
(collaborating cards must be separated)
• All use-case scenarios and use-case diagrams should be organized into
categories
• Review leader chooses a use-case scenario and begins reading it out
loud
• Each time a named object is read a token is passed to the reviewer
holding the object's card
• When the reviewer receives the token, he or she is asked to describe
the responsibilities listed on the card
• The group determines whether one of the responsibilities on the card
satisfy the use-case requirement or not
• If the responsibilities and collaborations on the index card cannot
accommodate the use-case requirements then modifications need to
be made to the card set
Deriving the Object-Relationship Model
• Using the CRC model a network of collaborators can be drawn
• Reviewing the CRC model index card, responsibilities and collaborators
are evaluated, each unlabeled connected line is named
• Once the named relationships are established each end is evaluated to
determine cardinality (0 to 1, 1 to 1, 0 to many, 1 to many)
• Continue the process until a complete object-relationship model has
been produced
353 http://www.certmagic.com
BH0-003
Object-Behavior Model Construction
• Evaluate all use-cases to understand the sequence of interaction within
the system
• Identify events that drive the interaction sequence and how events
relate to specific objects
• Create an event-trace for each use-case
• Build a state transition diagram for the system
• Review the object behavior model to verify accuracy and consistency
Chapter 22 Object-Oriented Design
Overview
This chapter discusses the steps required to develop an object-oriented
Software Testing design from an object-oriented analysis model. Object-
oriented design (OOD) is divided into two major activities: system design and
object design. System design defines the product architecture (the system
functions and classes encapsulated in the subsystems). System design
focuses on the specification of three components: the user interface, data
management functions, and task management facilities. Object design
focuses on the internal details of the individual classes and the messaging
354 http://www.certmagic.com
BH0-003
scheme. The design specification document form the SEPA web site is
applicable to OOD. The OOD projects must be reviewed to ensure quality.
Object-Oriented Design Layers
• Responsibilities layer (highest layer - contains data structure detail and
algorithmic detail for each object's attributes and operations)
• Message layer (establishes the internal and external interfaces for the
system, including the details of communication among object
collaborators)
• Class and object layer (contains class hierarchy including
representations of each object)
• Subsystem layer (lowest level - contains representations of each of the
subsystems and the necessary infrastructure that enable the Software
Testing to achieve the customer's requirements)
Object-oriented Design Issues
• Decomposability - facility of design method that allows the designer to
decompose the problem into easily solved subproblems
• Composability - degree to which design method ensures that modules
constructed for one project can be reused in another
• Understandability - ease with which a component can be understood
without examining other components
• Continuity - ability to isolate changes made in one module and restrict
the propagation of changes to other modules
• Protection - architectural characteristic that reduces the propagation of
side effects when errors occur
Generic Object-Oriented Design Steps
• Describe each subsystem and allocate it to processors and tasks
• Choose a design strategy for implementing data management,
interface support, and task management
• Design an appropriate system control mechanism
• Perform object design by creating a procedural representation for each
operation and data structures for each attribute
• Perform message design using collaborations between objects and
object-relationships
• Create a messaging model
• Review the design model and iterate as required
355 http://www.certmagic.com
BH0-003
Unified Approach to Object-Oriented Design
• System design - UML (unified modeling language) design activity that
focuses on Software Testing architecture and definition of subsystems
• Object design - UML design activity that focuses on describing object
and their interactions at a level of detail that will allow them to be
implemented in some programming language
Object-Oriented System Design Process
• Partition the analysis model into subsystems
• subsystems should have well defined communication interfaces
• with few exceptions classes should collaborate within their subsystem
• keep number of subsystems small
• partition subsystem internally to reduce complexity
• Identify concurrency dictated by the problem
• Allocate subsystems to processors and tasks
• allocate each subsystem to an independent processor (or)
• allocate subsystems to same processor and provide concurrency
support through operating system features
• Develop user interface design
• Choose basic strategy for implementing data management
• management of data critical to the application itself
• creation of infrastructure for storage and retrieval of objects
• Identify global resources and control mechanisms to access them
• Design control mechanism for system (including task management)
• Consider how subsystem boundary conditions should be handled
• list each request (contract) that can be made by subsystem
collaborators
• for each contract note the operations required to implement the
responsibilities implied by the contract
• for each contract create a table with these entries: type, collaborators,
class, operation, message format
• if subsystem interaction modes are complex create a subsystem-
collaboration diagram
• Review and consider trade-offs
356 http://www.certmagic.com
BH0-003
Object Design Process
• Object descriptions
• protocol description - object interface specified by defining each
message an object can receive and the operation triggered by
message (or)
• implementation description - shows implementation details for each
operation implied a message passed to the object
• Designing algorithms and data structures
• algorithm categories: data manipulation, computation, monitors
• refinement of operation programs defined during OOA
• Design optimization
• review object-relationship model to ensure implemented design leads
to efficient resource utilization, add redundancy where necessary
• revise attribute data structures and related operation algorithms to
improve processing efficiency
• create attributes to save derived information and avoid recomputation
• Modularity is important aspect of object-oriented design quality (the
program design language should support object definition)
Design Pattern Specification Components
• Name
• Intent
• Design forces motivating the pattern
• Solution that mitigates these design forces
• Classes required to implement the solution
• Responsibilities and collaborations among the solution classes
• Implementation suggestions
• Source code examples or templates
• Cross-references to related design patterns
Using Design Patterns in Object-Oriented Design
• Inheritance - makes use of an existing design pattern to create a
template for a new subclass
357 http://www.certmagic.com
BH0-003
• Composition - assembling complex objects or subsystems out of
selected design patterns using only interface information
Chapter 23 Object-Oriented Testing
Overview
This chapter discusses the testing of object-oriented systems. The process of
testing object-oriented systems begins with a review of the object-oriented
analysis and design models. Once the code is written object-oriented testing
(OOT) begins by testing "in the small" with class testing (class operations
and collaborations). As classes are integrated to become subsystems class
collaboration problems are investigated. Finally, use-cases from the OOA
model are used to uncover Software Testing validation errors. OOT similar to
testing conventional Software Testing in that test cases are developed to
exercise the classes, their collaborations, and behavior. OOT differs from
conventional Software Testing in that more emphasis is placed assessing the
completeness and consistency of the OOA and OOD models as they are built.
OOT tends to focus more on integration problems than on unit testing. The
test plan specification template from the SEPA web site is applicable to
object-oriented testing as well.
Object-Oriented Testing Activities
• Review OOA and OOD models
• Class testing after code is written
• Integration testing within subsystems
• Integration testing as subsystems are added to the system
• Validation testing based on OOA use-cases
Testing OOA and OOD Models
• Correctness of OOA and OOD models
• syntactic correctness judged by ensuring that proper modeling
conventions and symbolism have been used
• semantic correctness based on the model's conformance to the real
world problem domain
• Consistency of OOA and OOD models
358 http://www.certmagic.com
BH0-003
• assess the class-responsibility-collaborator (CRC) model and object-
relationship diagram
• review system design (examine the object-behavior model to check
mapping of system behavior to subsystems, review concurrency and
task allocation, use use-case scenarios to exercise user interface
design)
• test object model against the object relationship network to ensure
that all design object contain necessary attributes and operations
needed to implement the collaborations defined for each CRC card
• review detailed specifications of algorithms used to implement
operations using conventional inspection techniques
Assessing the Class Model
1. Revisit the CRC model and the object-relationship model
2. Inspect the description of each CRC card to determine if a delegated
responsibility is part of the collaborator's definition
3. Invert the connection to ensure that each collaborator that is asked for
service is receiving requests from a responsible source
4. Using the inverted connections from step 3, determine whether other
classes might be required or whether responsibilities are properly
grouped among classes
5. Determine whether widely requested responsibilities might be
combined into a single responsibility
6. Steps 1 to 5 are applied iteratively to each class and through the
evaluation of the OOA model
Object-Oriented Testing Strategies
• Unit testing in the OO context
• smallest testable unit is the encapsulated class or object
• similar to system testing of conventional Software Testing
• do not test operations in isolation from one another
• driven by class operations and state behavior, not algorithmic detail
and data flow across module interface
• Integration testing in the OO context
• focuses on groups of classes that collaborate or communicate in some
manner
• integration of operations one at a time into classes is often
meaningless
359 http://www.certmagic.com
BH0-003
• thread-based testing (testing all classes required to respond to one
system input or event)
• use-based testing (begins by testing independent classes first and the
dependent classes that make use of them)
• cluster testing (groups of collaborating classes are tested for
interaction errors)
• regression testing is important as each thread, cluster, or subsystem is
added to the system
• Validation testing in the OO context
• focuses on visible user actions and user recognizable outputs from the
system
• validation tests are based on the use-case scenarios, the object-
behavior model, and the event flow diagram created in the OOA model
• conventional black-box testing methods can be used to drive the
validation tests
Test Case Design for OO Software Testing
• Each test case should be uniquely identified and be explicitly
associated with a class to be tested
• State the purpose of each test
• List the testing steps for each test including:
• list of states to test for each object involved in the test
• list of messages and operations to exercised as a consequence of the
test
• list of exceptions that may occur as the object is tested
• list of external conditions needed to be changed for the test
• supplementary information required to understand or implement the
test
OO Test Design Issues
• White-box testing methods can be applied to testing the code used to
implement class operations, but not much else
• Black-box testing methods are appropriate for testing OO systems
• Fault-based testing
• best reserved for operations and the class level
• uses the inheritance structure
360 http://www.certmagic.com
BH0-003 61
• tester examines the OOA model and hypothesizes a set of plausible
defects that may be encountered in operation calls and message
connections and builds appropriate test cases
• misses incorrect specification and errors in subsystem interactions
• Object-oriented programming brings additional testing concerns
• classes may contain operations that are inherited from super classes
• subclasses may contain operations that were redefined rather than
inherited
• all classes derived from an previously tested base class need to be
thoroughly tested
• Scenario-based testing
• using the user tasks described in the use-cases and building the test
cases from the tasks and their variants
• uncovers errors that occur when any actor interacts with the OO
Software Testing
• concentrates on what the use does, not what the product does
• you can get a higher return on your effort by spending more time on
reviewing the use-cases as they are created, than spending more time
on use-case testing
• Testing surface structure (exercising the structure observable by end-
user, this often involves observing and interviewing users as they
manipulate system objects)
• Testing deep structure (exercising internal program structure - the
dependencies, behaviors, and communications mechanisms
established as part of the system and object design)
Class Level Testing Methods
• Random testing (requires large numbers data permutations and
combinations and can be inefficient)
• Partition testing (reduces the number of test cases required to test a
class)
• state-based partitioning (tests designed in way so that operations that
cause state changes are tested separately from those that do not)
361 http://www.certmagic.com
BH0-003
• attribute-based partitioning (for each class attribute, operations are
classified according to those that use the attribute, those that modify
the attribute, and those that do not use or modify the attribute)
• category-based partitioning (operations are categorized according to
the function they perform: initialization, computation, query,
termination)
Inter-Class Test Case Design
• Multiple class testing
• for each client class use the list of class operators to generate random
test sequences that send messages to other server classes
• for each message generated determine the collaborator class and the
corresponding server object operator
• for each server class operator (invoked by a client object message)
determine the message it transmits
• for each message, determine the next level of operators that are
invoked and incorporate them into the test sequence
• Tests derived from behavior models
• test cases must cover all states in the state transition diagram
• breadth first traversal of the state model can be used (test one
transition at a time and only make use of previously tested transitions
when testing a new transition)
• test cases can also be derived to ensure that all behaviors for the class
have been adequately exercised
362 http://www.certmagic.com
BH0-003
Chapter 24 Technical Metrics for Object-Oriented Systems
Overview
This chapter discusses the use of metrics to assess the quality o object-
oriented Software Testing. Software Testing engineers need objective criteria
to guide the OO system design and object design. Testers need quantitative
guidance to help in selecting test cases and their targets. OO technical
metrics can provide this guidance. As in the use of metrics in conventional
Software Testing development, the first step in the OO measurement process
is to derive measures and metrics that are appropriate to the Software
Testing representation under consideration. The next step is to collect the
necessary data and compute the metrics. Once computed, metrics are
analyzed based on pre-established guidelines and historical data. The results
of the analysis are interpreted to gain insight into the quality of the Software
Testing and modifications to work products from OOA, OOD, OOP, or OOT
may be made.
Goals for Using Object-Oriented Metrics
• To better understand product quality
• To assess process effectiveness
• To improve quality of the work performed at the project level
Distinguishing Characteristics of Object-Oriented Metrics
• Localization - OO metrics need to apply to the class as a whole and
should reflect the manner in which classes collaborate with one
another
• Encapsulation - OO metrics chosen need to reflect the fact that class
responsibilities, attributes, and operations are bound as a single unit
• Information hiding -OO metrics should provide an indication of the
degree to which information hiding has been achieved
• Inheritance - OO metrics should reflect the degree to which reuse of
existing classes has been achieved
• Abstraction - OO metrics represent abstractions in terms of measures
of a class (e.g. number of instances per class per application)
Object-Oriented Design Model Metrics
• Size(population, volume, length, functionality)
363 http://www.certmagic.com
BH0-003
• Complexity (how classes interrelate to one another)
• Coupling (physical connections between design elements)
• Sufficiency (how well design components reflect all properties of the
problem domain)
• Completeness (coverage of all parts of problem domain)
• Cohesion (manner in which all operations work together)
• Primitiveness (degree to which attributes and operations are atomic)
• Similarity (degree to which two or more classes are alike)
• Volatility (likelihood a design component will change)
Class-Oriented Metrics
• Chidamber and Kemerer (CK) Metrics Suite
• weighted metrics per class (WMC)
• depth of inheritance tree (DIT)
• number of children (NOC)
• coupling between object classes (CBO)
• response for a class(RFC)
• lack of cohesion in methods (LCOM)
• Lorenz and Kidd
• class size (CS)
• number of operations overridden by a subclass (NOO)
• number of operations added by a subclass (NOA)
• specialization index (SI)
• Harrison, Counsel, and Nithi (MOOD) Metrics Suite
• method inheritance factor (MIF)
• coupling factor (CF)
• polymorphism factor (PF)
Operation-Oriented Metrics
• Average operation size (OSavg)
• Operation complexity (OC)
• Average number of parameters per operation (NPavg)
Object-Oriented Testing Metrics
364 http://www.certmagic.com
BH0-003
• Encapsulation
• lack of cohesion in methods (LCOM)
• percent public and protected (PAP)
• public access to data members(PAD)
• Inheritance
• number of root classes (NOR)
• fan in (FIN)
• number of children (NOC)
• depth of inheritance tree (DIT)
• Class complexity
• weighted metrics per class(WMC)
• coupling between object classes (CBO)
• response for a class (RFC)
Object-Oriented Product Metrics
• Number of scenario scripts (NSS)
• Number of key classes (NKC)
• Number of subsystems(NSUB)
365 http://www.certmagic.com
BH0-003
Chapter 25 - Formal Methods
Overview
This chapter discusses the role of formal methods in Software Testing
engineering. Formal methods allow Software Testing engineers to create
specifications using mathematical notation that is more complete, more
consistent, and unambiguous. The mathematics used in formal Software
Testing engineering methods relies heavily on set theory and logic. In many
safety critical or mission critical systems, failures can have a high cost. Many
safety critical systems can not be completely tested without endangering the
lives of the people they are designed to protect. Use of formal methods
reduces the number of specification errors dramatically, which means that
the customer will encounter fewer errors when the product is deployed.
Using Formal Methods
• Define the data invariant, state, and operations for each system
function
• data invariant -condition true throughout execution of function that
contains a collection of data
• state- stored data accessed and altered by function
• operations - system actions that take place when data are read or
written to the state (a precondition and postcondition is associated
with each operation)
• Specification is represented in some set theoretic type notation from
some formal language (e.g. Z or VDM)
• Specification correctness can be verified using mathematical proofs
(set operations, logic operations, sequences, induction)
366 http://www.certmagic.com
BH0-003
Formal Specification Properties
• Unambiguous - formal syntax used by formal methods has only one
interpretation (unlike natural language statements)
• Consistency - ensuring through mathematical proof that initial facts
can be mapped (using inference rules)into later statements within the
specification
• Completeness - difficult to achieve in a large system even using formal
methods
Weaknesses of Less Formal Approaches
• Contradictions (statements do not agree with one another)
• Ambiguities(statements have more than one interpretation)
• Vagueness (specifications in large documents are often not written
precisely enough)
• Incompleteness (e.g. failing to list limitations and error handling
required of a function)
• Mixed levels of abstraction(occurs when very abstract statements are
intermixed randomly with statements written at lower levels of detail)
Necessary Mathematics
• Constructive set specification (also known as set builder notation)
• Set operations (membership, subset, union, intersection, difference,
crossproduct or Cartesian product, powerset)
• Logic operators (and, or, not, implies, universal quantification)
• Sequence properties (order, domain, range)
• Sequence operators(concatenation, head, tail, front, last)
Writing Formal Specifications
• Begin by defining state in terms of abstract items to be manipulated
by the function (similar to variable declaration in a programming
language)
• Define the data invariant by writing the data relations that will not
change during the execution of the function using mathematical
notation
• rite he precondition and postcondition for the function using
mathematical notation to show the system state before and after
function execution
367 http://www.certmagic.com
BH0-003
Formal Specification Language Components
• Syntax that defines the specific notation used to represent a
specification
• Semantics that help to define the objects used to define the system
• Set of relations that define the rules that indicate which objects
properly satisfy the specification
Ten Commandments of Formal Methods
1. Choose the appropriate notation
2. Do not over-formalize
3. Estimate costs
4. Have a formal methods guru on call
5. Do not abandon traditional development methods
6. Document sufficiently
7. Do not compromise quality standards
8. Do not be dogmatic in assuming formal specifications are flawless
9. Use of formal methods does not eliminate the need to test products
10.Reuse is still important
Chapter 26 - Cleanroom Software Testing Engineering
Overview
This chapter discusses the cleanroom approach to Software Testing
engineering. The philosophy behind cleanroom Software Testing engineering
is to develop code increments that are right the first time and verify their
correctness before testing, rather than relying on costly defect removal
processes. Cleanroom Software Testing engineering involves the integrated
use of Software Testing engineering modeling, program verification, and
368 http://www.certmagic.com
BH0-003
statistical Software Testing quality assurance. Under cleanroom Software
Testing engineering, the analysis and design models are created using a box
structure representation (black-box, state box, and clear box). A box
encapsulates some system component at a specific level of abstraction.
Correctness verification is applied once the box structure design is complete.
Once correctness has been verified for each box structure, statistical usage
testing commences. This involves defining a set of usage scenarios and
determining the probability of use for each scenario. Random data is
generated which conform to the usage probabilities. The resulting error
records are analyzed, and the reliability of the Software Testing is
determined for the Software Testing component.
Distinguishing Characteristics of Cleanroom Techniques
• Makes extensive use of statistical quality control
• Verifies design specification using mathematically-based proof of
correctness
• Relies heavily on statistical use testing to uncover high impact errors
Reasons Cleanroom Techniques Not Used Widely
• Some people believe cleanroom techniques are too theoretical, too
mathematical, and too radical for use in real Software Testing
development
• It does not advocate unit testing, relying instead on correctness
verification and statistical quality control (a major departure from the
way most Software Testing is developed today)
• Since most of the Software Testing industry is operating at the ad hoc
level of the Capability Maturity Model, most organizations do not make
rigorous use of the defined processes needed in all phases of the
Software Testing life cycle
It should be noted that all of the above roadblocks to cleanroom usage can
be overcome and that cleanroom Software Testing engineering offers
substantial benefits to those who do it.
Cleanroom Strategy
• Increment planning. The project plan is built around the incremental
strategy.
369 http://www.certmagic.com
BH0-003
• Requirements gathering. Using Chapter 11techniques, customer
requirements are refined for each increment.
• Box structure specification. Box structures isolate and separate the
definition of behavior, data, and procedures at each level of
refinement.
• Formal design. Specifications (black-boxes) are iteratively refined to
become architectural designs (state-boxes) and component-level
designs (clear boxes).
• Correctness verification. Correctness questions are asked and
answered and followed by formal mathematical verification when
required.
• Code generation, inspection, verification. Box structures are translated
into program language; inspections are used to ensure conformance of
code and boxes, as well as syntactic correctness of code; followed by
correctness verification of the code.
• Statistical test planning. A suite of test cases is created to match the
probability distribution of the projected product usage pattern.
• Statistical use testing. A statistical sample of all possible test cases is
used rather than exhaustive testing.
• Certification. Once verification, inspection, and usage testing are
complete and all defects removed, the increment is certified as ready
for integration.
Box Types
• Black box - specifies a set of transition rules that describe the behavior
of system components as responses to specific stimuli, makes use of
inheritance in a manner similar to classes
• State box - generalization of a state machine, encapsulates the data
and operations similar to an object, the inputs (stimuli) and outputs
(responses) are represented, data that must be retained between
transitions is encapsulated
• Clear box - contains the procedural design of the state box, in a
manner similar to structured programming
Design Verification Advantages
• Reduces verification to a finite process
• Improves quality
• Lets cleanroom teams verify every line of code
• Results in near zero levels of defects
• Scales up to larger systems and higher levels
• Produces better code than unit testing
370 http://www.certmagic.com
BH0-003
Certification Steps
• Usage scenarios must be created
• Usage profile is specified
• Test cases generated from the usage profile
• Tests are executed and failure data are recorded and analyzed
• Reliability is computed and recorded
Cleanroom Certification Models
• Sampling model - determines the number if random cases that need to
be executed to achieve a particular reliability level
• Component model - allows analyst to determine the probability that a
given component in a multi-component system fails prior to
completion
• Certification model - projected overall reliability of system
371 http://www.certmagic.com
BH0-003
Chapter 27 Component-Based Software Testing Engineering
Overview
The chapter describes component-based Software Testing engineering
(CBSE) as a process that emphasizes the design and construction of
computer-based systems using reusable Software Testing components. This
has the potential advantage of delivering highly reliable Software Testing
products in a very short time. CBSE encourages the use of predictable
architectural patterns and standard Software Testing infrastructures that
improve overall product quality. CBSE encompasses two parallel engineering
activities, domain engineering and component-based development. Domain
engineering explores the application domain with the specific intent of finding
functional, behavioral, and data components that are candidates for reuse
and places them in reuse libraries. Component-based development elicits
requirements from the customer and selects an appropriate architectural
style to meet the objectives of the system to be built. The next steps are to
select potential components for reuse, qualify the components to be sure
they fit the system architecture properly, adapt the components if they must
be modified to integrate them, then integrate the components into
subsystems within the application. Custom components are engineered only
when existing components cannot be reused. Formal technical reviews and
testing are applied to ensure the quality of the analysis model and the design
model. The resulting code is tested to uncover errors in the newly developed
Software Testing.
372 http://www.certmagic.com
BH0-003
Engineering of Component-Based Systems
• Software Testing team elicits requirements for system to be built
• Architectural design is established
• Team determines which of the requirements are amenable to
composition rather than construction
• Are commercial off-the-shelf (COTS) components available to
implement the requirement?
• Are internally developed reusable components available to implement
the requirement?
• Are the interfaces for available components compatible within in the
proposed system architecture?
• Team attempts to remove or modify requirements that cannot be
implemented with COTS or in-house components?
• For those requirements that can be addressed with available
components the following activities take place:
• component qualification - candidate components are identified based
on services provided and means by which consumers access them
• component adaptation - candidate components are modified to meet
the needs of the architecture or discarded
• component composition - architecture dictates the composition of the
end product from the nature of the connections and coordination
mechanisms
• component update - updating systems that include COTS is made
more complicated by the fact that a COTS developer must be involved
• Detailed design activities commence for remainder of the system
Domain Engineering
• Domain analysis
• define application domain to be investigated
• categorize items extracted from domain
• collect representative applications from the domain
• analyze each application from sample
• develop an analysis model for objects
• Domain model
• Software Testing architecture development
373 http://www.certmagic.com
BH0-003
• Structural model
• consists of small number of structural elements manifesting clear
patterns of interaction
• architectural style that can be reused across applications in the domain
• structure points are distinct constructs within the structural model
(e.g. interface, control mechanism, response mechanism)
• Reusable component development (new Software Testing components
must be developed and integrated with existing Software Testing
components)
• Repository of reusable artifacts or components
Structure Point Characteristics
• Abstractions with limited number of instances within an application and
recurs in applications in the domain
• The rules governing the use of a structure point should be easily
understood and structure point interface should be simple
• The structure point should implement information hiding by isolating
all complexity contained within the structure point itself
Component-Based Development
• Analysis
• Architectural design
• component qualification
• component adaptation
• component decomposition
• Component engineering
• Testing
• Iterative component update
Component Adaptation Techniques
• White-box wrapping (integration conflicts removed by making code-
level modifications to the code)
374 http://www.certmagic.com
BH0-003
• Grey-box wrapping (used when component library provides a
component extension language or API that allows conflicts to be
removed or masked)
• Black-box wrapping (requires the introduction of pre- and post-
processing at the component interface to remove or mask conflicts)
Component Composition Infrastructure Elements
• Data exchange model (similar to drag and drop type mechanisms
should be defined for all reusable components, allow human-to-
Software Testing and component-to-component transfer)
• Automation (tools, macros, scripts should be implemented to facilitate
interaction between reusable components)
• Structured storage (heterogeneous data should be organized and
contained in a single data structure rather several separate files)
• Underlying object model (ensures that components developed in
different languages are interoperable across computing platforms)
Representative Component Standards
• Object Management Group (OMG) CORBA (common object request
broker architecture)
• Microsoft COM (component object model)
• Sun Java Beans Component System
Classifying and Retrieving Components
• Describing reusable components
• concept - what the component does
• content - how the concept is realized
• context - specifies conceptual, operational, and implementation
features of the Software Testing component within its domain of
application
• Library indexing methods
• uncontrolled indexing vocabularies (syntax free, no restrictions)
• enumerated classification (hierarchical listing of the domain objects
grouped by class relations)
375 http://www.certmagic.com
BH0-003
• faceted classification (based on 1 to 8 basic descriptive features from
the application domain)
• attribute-value classification (similar to faceted classification using
unlimited number of fixed terms)
• Reuse environment elements
• component database capable of storing Software Testing components
and classification information to allow their retrieval
• library management system to allow access to database
• Software Testing component retrieval system that enables client
Software Testing to retrieve components and services from library
server
• CBSE tools that support integration of reused components into a new
design or implementation
Economics of Reuse
• Quality -- with each reuse additional component defects are identified
and removed which improves quality.
• Productivity -- since less time is spent on creating plans, models,
documents, code, and data the same level of functionality can be
delivered with less effort so productivity improves.
• Cost -- savings projected by estimating the cost of building the system
from scratch and subtracting the costs associated with reuse and the
actual cost of the Software Testing as delivered.
• Cost analysis using structure points -- can be computed based on
historical data regarding the costs of maintaining, qualification,
adaptation, and integrating each structure point.
• Reuse metrics can be computed for CBSE
376 http://www.certmagic.com
BH0-003
Chapter 28 Client/Server Software Testing Engineering
Overview
This chapter discusses client/server (C/S) Software Testing engineering.
Client/server Software Testing engineering blends conventional principles,
concepts, and methods discussed earlier in the text with elements of object-
oriented and component-based Software Testing engineering. C/S
architectures dominate the landscape of computer-based systems. In C/S
architectures, Software Testing residing on one computer (the client)
requests services or data from another computer (the server). The process
model used in C/S Software Testing engineering is evolutionary beginning
with requirements elicitation. Functionality is allocated to subsystems of
components that are assigned to either the client or the server side of the
C/S architecture. Design focuses on integration of existing components and
creation of new components. Implementation and testing must exercise both
the client and server functionality within the context of the component
integration standards and the C/S architecture. C/S Software Testing
engineering relies on the same SQA practices as other Software Testing
engineering processes. Formal technical reviews are used to assess the
quality of the analysis and design models. Specialized reviews consider issues
associated with component integration and middleware. Testing is used to
uncover errors at the component, subsystem, client, and sever levels.
Representative Client/Server Systems
• File servers (client requests selected records from a file, server
transmits records to client over the network)
• Database servers (client sends SQL requests to server, server
processes the request and returns the results to the client over the
network)
• Transaction servers (client sends requests that invokes remote
procedures on the server side, sever executes procedures invoked and
returns the results to the client)
377 http://www.certmagic.com
BH0-003
• Groupware servers (server provides set of applications that enable
communication among clients using text, images, bulletin boards,
video, etc.)
Software Testing Components for C/S Systems
• User interaction/presentation subsystem (handles all user events)
• Application subsystem (implements requires defined by the application
within the context of the operating environment, components may
reside on either the client or server side)
• Database management subsystem (performs data manipulation and
management for the application)
• Middleware (all Software Testing components that exist on both the
client and the server to allow exchange of information)
Representative C/S Configuration Options
• Distributed presentation - database and application logic remain on the
server, client Software Testing is used to reformat server data into GUI
format
• Remote presentation - similar to distributed presentation, primary
database and application logic remain on the server, data sent by the
server is used by the client to prepare the user presentation
• Distributed logic - client is assigned all user presentation tasks
associated with data entry and formulating server queries, server is
assigned data management tasks and updates information based on
user actions
• Remote data management - applications on server side create new
data sources, applications on client side process the new data returned
by the server
• Distributed databases - data is spread across multiple clients and
servers, requiring clients to support data management as well as
application and GUI components
• Fat server - most Software Testing functions for C/S system are
allocated to the server
• Thin clients - network computer approach relegating all application
processing to a fat server
Guidelines for Distributing Application Subsystems
• The presentation/interaction subsystem is generally placed on the
client.
378 http://www.certmagic.com
BH0-003
• If the database is to be shared by multiple users connected by a LAN,
the database is typically located on the server.
• Static data used for reference should be allocated to the client.
Linking C/S Software Testing Subsystems
• Pipes (permit messaging between different machines running different
operating systems)
• Remote procedure calls (permit process running on one machine to
invoke execution of process residing on another machine)
• Client/server SQL interaction (SQL requests passed from client to
server DBMS, this mechanism is limited to RDBMS)
Representative Middleware Architectures
• CORBA (ORB)
• COM (Microsoft)
• JavaBeans (Sun)
Design Issues for C/S Systems
• Data and architectural design - dominates the design process to be
able to effectively use the capabilities of RDBMS or OODMBS
• Event-driven paradigm - when used, behavioral modeling should be
conducted and the control-oriented aspects of the behavioral model
should translated into the design model
• Interface design - elevated in importance, since the user
interaction/presentation component implements all functions typically
associated with a GUI
• Object-oriented point of view - often chosen, since an object structure
is provides by events initiated in the GUI and their event handlers
within the client-based Software Testing
Architectural Design for Client/Server Systems
• Best described as communicating processes style architecture whose
goal is to achieve easy scalability when adding and arbitrary number of
clients
379 http://www.certmagic.com
BH0-003
• Since modern C/S systems tend to be component-based, an object
request broker (ORB) architecture is used for implementation
• Object adapters or wrappers provide service to facilitate
communication among client and server components
• component implementations are registered
• all component references are interpreted and reconciled
• component references are mapped to corresponding component
implementations
• objects are activated and deactivated
• operations are invoked when messages are transmitted
• security features are implemented
C/S Design Repository Information
• Entities (from ER diagram)
• Files (which implement entities)
• File-to-field relationship (establishes file layout)
• Fields (from data dictionary)
• File-to-file relationships (related files that may be joined together)
• Relationship validation
• Field type (used to permit inheritance from super classes)
• Data type (characteristics of field data)
• File type (used to identify file location)
• Field function (key, foreign key, attribute, etc.)
• Allowed values
• Business values (rules for editing, calculating derived fields, etc.)
Data Distribution and Management Techniques
• Relational data base management systems (RDBMS)
• Manual extract (user allowed to manually copy data from server to
client)
• Snapshot (automates manual extract by specifying a copy of the data
be transferred from the client to the server at predefined intervals)
• Replication (multiple copies of data are maintained at different sites)
• Fragmentation (system database is spread across several machines)
C/S Design Approach
380 http://www.certmagic.com
BH0-003
1. For each elementary business process, identify the files created,
updated, referenced, or deleted.
2. Use the files from step 1 as basis for defining components or objects.
3. For each component, retrieve the business rules and other business
object information that has been established for the relevant file.
4. Determine which rules are relevant to the process and decompose the
rules down to the method level.
5. As required, define any additional components that are needed to
implement the methods.
Process Design Entities
• Methods - describe how a business rule is to be implemented
• Elementary processes - business processes identified in the analysis
model
• Process/component link - identifies components that makeup the
solution for an elementary business process
• Components - describes components shown on structure chart
• Business rule/component link - identified components significant to
implementation of a given business rule
C/S Testing Strategy
• Application function tests (client applications tested in stand alone
manner)
• Sever tests (test coordination and management functions of server,
also measure performance of server)
• Database tests (check accuracy and integrity server data, examine
transactions posted by client, test archiving)
• Transaction testing (ensure each class of transactions is processed
correctly)
• Network communication testing (verify communication among network
nodes)
C/S Testing Tactics
• Begins with testing in the small and then proceeds to integration
testing using the non-incremental or big bang approach
381 http://www.certmagic.com
BH0-003
• Requires special attention to configuration testing and compatibility
testing
• OO testing tactics can be used for C/S systems (even when system
was not built using OO methodology)
• GUI testing requires special techniques in C/S systems (e.g. structured
capture/playback)
382 http://www.certmagic.com
BH0-003
Chapter 29 - Web Engineering
Overview
The chapter describes Web engineering (WebE) as the process used to create
high quality Web-based applications (WebApps). As WebApps become
increasingly integrated in business strategies (e.g. e-commerce) the need to
build reliable, usable, and adaptable systems grows in importance. Web
engineering is not a perfect clone of Software Testing engineering, but it
draws heavily on many of Software Testing engineering's principles and
management activities. The Web engineering process begins with the
formulation of the problem to be solved by the WebApp. The project is
planned and the WebApp requirements are analyzed. Architectural,
navigational, and interface design are conducted. The system is implemented
using specialized languages and tools associated with the Web. WebApps
tend to be highly evolutionary, so mechanisms for configuration
management, quality control, and maintenance must be established early.
Web engineering relies on formal technical reviews to assess the quality of
the analysis and design models. Specialized reviews are conducted to assess
the usability of the WebApp. Testing is applied to uncover errors in content,
functionality, and compatibility.
WebApp Attributes
• Network intensive
• Content-driven
• Continuous evolution
• Immediacy
• Security
• Aesthetics
WebE Application Categories
• Informational (read only content provided with simple navigation)
• Downloads (user downloads information from server)
383 http://www.certmagic.com
BH0-003
• Customizable (user customizes content to specific needs)
• Interaction (community of users communicate using chat rooms,
bulletin boards, or instant messaging)
• User input (users complete on-line forms to communicate need)
• Transaction-oriented (user makes request fulfilled by WebApp - places
an order)
• Service-oriented (application provides service to user, e.g. helps user
determine mortgage payment)
• Portal(application directs users to other web content or services)
• Database access (user queries a large database and extracts
information)
• Data warehousing (user queries large collection of databases and
extracts information)
Web Quality Requirements
• Usability
• Functionality
• Reliability
• Efficiency
• Maintainability
WebApp Enabling Technologies
• Component-based development (CORBA, COM/DCOM, JavaBeans)
• Security (encryption, firewalls, etc.)
• Internet standards (HTML, XML,SGML)
Evolutionary WebE Process Model
• Formulation (goals and objectives, scope for first increment)
• what is the motivation for the WebApp?
• why is the WebApp needed?
• who will use the WebApp?
• informational goals (user's intention for using the content)
• applicative goals (ability to perform tasks within the WebApp)
• Planning (estimates project cost, evaluates risks, defines finely
granulated schedule for first increment and coarser schedule for
subsequent increments)
384 http://www.certmagic.com
BH0-003
• Analysis (establishes requirements and identifies content items)
• content analysis (content provided by WebAppis identified)
• interaction analysis (use-cases developed to describe user interaction
with WebApp)
• functional analysis (usage scenarios used to define operations and
functions applied to the WebApp content)
• configuration analysis (WebApp environmental infrastructure is
described in detail)
• Engineering (content design and production tasks are one thread,
architectural design, navigation design, interface are the other thread)
• Page generation and testing (content and technical deigns are merged
to produce executable web pages; testing exercises WebApp
navigation, attempts to uncover errors in applets/scripts/forms, and
checks for environment incompatibilities)
• Customer evaluation (each increment of the WebApp is reviewed and
changes required by customer are applied to next increment)
Technical Elements for Web-Based Design
• Design principles and methods (high modularity, low coupling,
information hiding, stepwise refinement, OO design methods)
• Golden rules (design heuristics for hypermedia applications)
• Design Patterns (can be applied to WebApp functional elements,
documents, graphics, and general aesthetics)
• Templates (provide reusable skeletal frameworks for any design
pattern or document used within the WebApp)
Web App Architectural Structures
• Linear structures (text and graphics presented in fixed sequential
order)
• Grid structures (useful when WebApp content must be organized in
two pr more ways or dimensions)
• Hierarchical structures (not always traversed in strict depth-first
manner, branches are often inter-linked)
• Networked or "pure" web structure (every node is connected to every
other node)
WebE Design Patterns
• Cycle (user is returned to previously visited node)
385 http://www.certmagic.com
BH0-003
• Web ring (implements a grand cycle that links entire hypertext into a
tour of asubject)
• Contour (occurs when cycles are interconnected, allowing navigation
across paths defined by cycles)
• Counterpoint (hypertext commentary used to interrupt content
narrative to provide additional information or insight)
• Mirrorworld (content is presented using several threads, each with its
own perspective or point of view)
• Sieve(user guided through a series of decisions to direct user to
specific content indexed by the decisions)
• Neighborhood (uniform navigation is provided to user regardless of
position within the WebApp)
NavigationalDesign
• Identify the semantics of navigation for different users based on the
perceived roles (i.e. visitor, registered customer, or privileged user)
and the goals associated with their roles.
• Define the mechanics (syntax) of achieving navigation
WebApp Interface Guidelines
• Minor server errors are likely to cause user to leave WebApp and look
for an alternative site
• Reading speed on monitor is about 25% slower than for hardcopy
• Avoid "under construction" signs
• Users prefer not having to scroll to read content
• Navigation menus and headers should be designed consistently and be
available on all pages available to the user
• Do not rely on browser functions to assist in navigation
• Aesthetics should never take precedence over application functionality
• Navigation should be obvious to causal users
Testing WebApps for Errors
• WebApp content model is reviewed to uncover errors.
• Design model for WebApp is reviewed to uncover navigation errors.
• Selected processing components and web pages are unit tested.
• Architecture is constructed and integration tests are performed.
• Assembled WebApp is tested for overall functionality and content
delivery.
386 http://www.certmagic.com
BH0-003
• WebApp is implemented in a variety of different environmental
configurations and the compatibility of WebApp with each is assessed.
• WebAppis tested by a controlled and monitored group of end-users.
WebE Team Members
• Content developers and providers (focus on generation and/or
collection of WebApp content)
• Web publisher (liaison between technical staff who engineers WebApp
and non-technical content developers and providers)
• Web engineer (involved with WebApp requirements elicitation, analysis
modeling, architectural design, navigational design, interface design,
implementation, and testing)
• Support specialist (responsible for continuing WebApp maintenance
and support)
• Administrator or web master (responsible for daily operation of
WebApp)
Project Management Concerns Unique to WebE
• Many WebApps are out sourced to vendors specializing in the
development of web-based systems and applications.
• WebApp development is relatively new and there is little historical data
to use for estimation (e.g. there are few if any published WebE
metrics).
• The continuously evolving nature of WebApps make estimation, risk
analysis, and scheduling more complicated since project scope is less
clearly defined than in other Software Testing development projects.
WebE Project Management Guidelines
• Initiating a project
1. many of the analysis activities should be performed internally
2. a rough design for the WebApp should be developed internally
3. a rough delivery schedule including milestone dates and final delivery
dates should be developed
387 http://www.certmagic.com
BH0-003
4. the degree of oversight and interaction by the contractor with the
vendor should be identified
• Selection of candidate outsourcing vendors
1. Interview past clients to determine vendor's past performance
2. be certain the vendor's chief web engineer(s) from past successful
projects will involved with yours
3. carefully examine samples of the vendor's work on projects similar to
yours
• Assessing the validity of price quotes and reliability estimates
1. does the quoted cost of the WebApp provide a direct or indirect return-
on-investment that justifies the project?
2. does the vendor exhibit the required level of professionalism and
experience?
• Degree of project management you can expect or perform(directly
proportional to the size, cost, and complexity of WebApp, the larger
the more formal the management and SQA activities)
• Assessing the development schedule (short development times
suggest the use of fine granularity in the schedule, link minor
milestones scheduled on a daily timeline)
• Managing the scope (using an incremental process model allows the
development team to freeze the scope for one increment to allow an
operational WebApp release to be created)
WebE Software Testing Configuration Management Issues
• Content
• integrating heterogeneous media
• volatility
• People
• designers often are forced to create content
• content creators often have no Software Testing engineering
knowledge
• Scalability
• Politics
• Who owns a WebApp?
• Who assumes responsibility for accuracy?
• Who makes changes?
• Who pays for changes
388 http://www.certmagic.com
BH0-003
Chapter 30 - Reengineering
Overview
This chapter defines reengineering as the process of legacy Software Testing
products. The new Software Testing products often have increased
functionality, better performance, greater reliability, and are easier to
maintain than their predecessors. Business process reengineering (BPR)
defines business goals, evaluates existing business processes, and creates
revised business processes that better meet current goals. Software Testing
reengineering involves inventory analysis, document restructuring, reverse
engineering, program and data restructuring, and forward engineering. Many
reengineering work products are the same as those generated for any
Software Testing engineering process (analysis models, design models, test
procedures). The final product for any reengineering process is a
reengineered business process and/or the reengineered Software Testing to
support it. The same SQA practices are applied to Software Testing
reengineering as to they would to any other Software Testing development
process. Testing is used to uncover errors in content, functionality, and
interoperability.
Business Process Reengineering Principles
• Organize around outcomes, not tasks.
• Have those people who use the output of a process, perform the
process.
• Incorporate information processing work into the real work that
produces the raw information.
• Treat geographically dispersed resources as though they were
centralized.
• Link parallel activities instead of integrating their results.
• Put the decision point where the work is performed and build control
into the process.
• Capture the data once, at its source.
Business Process Reengineering Model
• Business redefinition (business goals are identified in the context of
four key drivers - cost reduction, time reduction, quality improvement,
empowerment)
• Process identification (processes critical to achieving business goals
are identified and prioritized)
389 http://www.certmagic.com
BH0-003
• Process evaluation (existing processes are analyzed and measured,
costs and time consumed by processes are noted, quality/performance
problems are isolated)
• Process specification and design(use-cases are prepared for each
process to be redesigned, these use-case scenarios deliver some
outcome to a customer, new tasks are design for each process)
• Prototyping (used to test processes before integrating them into the
business)
• Refinement and instantiation (based on feedback from the prototype,
business processes are refined and then instantiated within a business
system)
Software Testing Reengineering Process Model
• Inventory analysis (sorting active Software Testing applications by
business criticality, longevity, current maintainability, and other local
criteria helps to identify reengineering candidates)
• Document restructuring (need to decide to live with weak
documentation, update poor documents if they are used, or fully
rewrite the documentation for critical systems focusing on the
"essential minimum")
• Reverse engineering (process of design recovery - analyzing a
program in an effort to create a representation of the program at some
abstraction level higher than source code)
• Code restructuring (source code is analyzed and violations of
structured programming practices are noted and repaired, the revised
code also needs to be reviewed and tested)
• Data restructuring (usually requires full reverse engineering, current
data architecture is dissected and data models are defined, existing
data structures are reviewed for quality)
• Forward engineering (also called reclamation or renovation - recovers
design information from existing source code and uses this information
to reconstitute the existing system to improve its overall quality
and/or performance)
Reverse Engineering Concepts
• Abstraction level (ideally want to be able to derive design information
at the highest level possible)
• Completeness (level of detail provided at a given abstraction level)
390 http://www.certmagic.com
BH0-003
• Interactivity (degree to which humans are integrated with automated
reverse engineering tools)
• Directionality (one-way means the Software Testing engineer doing
the maintenance activity is given all information extracted from source
code, two-way means the information is fed to a reengineering tool
that attempts to regenerate the old program)
• Extract abstractions (meaningful specification of processing performed
is derived from old source code)
Reverse Engineering Activities
• Understanding process (source code is analyzed to at varying levels of
detail - system, program, component, pattern, statement - to
understand procedural abstractions and overall functionality)
• Understanding data
• internal data structures
• database structure
• User interfaces
• what are the basic actions (e.g. key strokes or mouse operations)
processed by the interface?
• what is a compact description of the system's behavioral response to
these actions?
• what concept of equivalence of interfaces is relevant here?
Restructuring Benefits
• Improved program and documentation quality
• Makes programs easier to learn, improves productivity, reduces
developer frustration
• Reduces effort required to maintain Software Testing
• Software Testing is easier to test and debug
Types of Restructuring
• Code restructuring
• program logic modeled using Boolean algebra and series of
transformation rules are applied to yield restructured logic
391 http://www.certmagic.com
BH0-003
• create resource exchange diagram showing data types, procedure and
variables shared between modules, restructure program architecture
to minimize module coupling
• Data restructuring
• analysis of source code
• data redesign
• data record standardization
• data name rationalization
• file or database translation
Forward Engineering
• Client/Server architectures
• application functionality migrates to each client computer
• new GUI interfaces implemented at client sites
• database functions allocated to servers
• specialized functionality may remain at server site
• new communications, security, archiving, and control requirements
must be established at both client and server sites
• Object-oriented architectures
• existing Software Testing is reverse engineered so that appropriate
data, functional, and behavioral models can be created
• use-cases are created if reengineered system extends functionality of
application
• data models created during reverse engineering are used with CRC
modeling as a basis to define classes
• create class hierarchies, object-relationship models, object-behavior
models and begin object-oriented design
• a component-based process model may be used if a robust component
library already exists
• where components must be built from scratch, it may be possible to
reuse algorithms and data structures from the original application
• User interfaces
• understand the original user interface and how the data moves
between the user interface and the remainder of the application
• remodel the behavior implied by the existing user interface into a
series of abstractions that have meaning in the context of a GUI
392 http://www.certmagic.com
BH0-003
• introduce improvements that make the mode of interaction more
efficient
• build and integrate the new GUI
Economics of Reengineering
• Cost of maintenance =cost annual of operation and maintenance over
application lifetime
• Cost of reengineering = predicted return on investment reduced by
cost of implementing changes and engineering risk factors
• Cost benefit = Cost of reengineering - Cost of maintenance
Chapter 31 Computer-Aided Software Testing Engineering
Overview
393 http://www.certmagic.com
BH0-003
This chapter discusses the use of computer-aided Software Testing
engineering (CASE) tools that can assist Software Testing engineering
managers and practitioners with every activity associated with the Software
Testing development process. CASE tools can automate management
activities and can manage work products. CASE tools can assist engineers
with analysis, design, coding, and testing work. Software Testing engineering
is hard work and tools that reduce the effort required to produce a work
product or accomplish a milestone have substantial benefits. CASE tools
assist the Software Testing engineer in producing high quality work products.
Tools can provide the Software Testing engineer with improved insight into
the Software Testing product and make decisions that lead to improved
Software Testing quality. Tools complement solid Software Testing
engineering practices. A good Software Testing process framework must be
established and Software Testing quality must be emphasized before tools
can be used effectively.
Prerequisites to Software Testing Tool Use
• Collection of useful tools that help in every step of building a product
• Organized layout that enables tools to be found quickly and used
efficiently
• Skilled craftsperson who understands how to use the tools effectively
CASE Building Blocks
• CASE tools
• Integration framework(specialized programs allowing CASE tools to
communicate with one another)
• Portability services (allow CASE tools and their integration framework
to migrate across different operating systems and hardware platforms
without significant adaptive maintenance)
• Operating system (database and object management services)
• Hardware platform
• Environmental architecture (hardware and system support)
CASE Tool Taxonomy
• Business process engineering tools - represent business data objects,
their relationships, and flow of the data objects between company
business areas
394 http://www.certmagic.com
BH0-003
• Process modeling and management tools - represent keyelements of
processes and provide links to other tools that provide support to
defined process activities
• Project planning tools - used for cost and effort estimation, and project
scheduling
• Risk analysis tools - help project managers build risk tables by
providing detailed guidance in the identification and analysis of risks
• Requirements tracing tools - provide systematic database-like
approach to tracking requirement status beginning with specification
• Metrics and management tools -management oriented tools capture
project specific metrics that provide an overall indication of
productivity or quality, technically oriented metrics determine metrics
that provide greater insight into the quality of design or code
• Documentation tools - provide opportunities for improved productivity
by reducing the amount of time needed to produce work products
• System Software Testing tools - network system Software Testing,
object management services, distributed component support, and
communications Software Testing
• Quality assurance tools -metrics tools that audit source code to
determine compliance with language standards or tools that extract
metrics to project the quality of Software Testing being built
• Database management tools - RDMS and OODMS serve as the
foundation for the establishment of the CASE repository
• Software Testing configuration management tools -uses the CASE
repository to assist with all SCM tasks (identification, version control,
change control, auditing, status accounting)
• Analysis and design tools -enable the Software Testing engineer to
create analysis and design models of the system to be built, perform
consistency checking between models
• PRO/SIM tools - prototyping and simulation tools provide Software
Testing engineers with ability to predict the behavior of real-time
systems before they are built and the creation of interface mockups for
customer review
• Interface design and development tools - toolkits of interface
components, often part environment with a GUI to allow rapid
prototyping of user interface designs
• Prototyping tools - enable rapid definition of screen layouts, data
design, and report generation
• Programming tools - compilers, editors, debuggers, OO programming
environments, fourth generation languages, graphical programming
environments, applications generators, and database query generators
• Web development tools - assist with the generation of web page text,
graphics, forms, scripts, applets, etc.
• Integration and testing tools
• Data acquisition (get data for testing)
• static measurement (analyze source code without using test cases)
• dynamic measurement (analyze source code during execution)
395 http://www.certmagic.com
BH0-003
• simulation (simulate function of hardware and other externals)
• test management (assist in test planning, development, and control)
• cross-functional (tools that cross test tool category boundaries)
• Static analysis tools - code-based testing tools, specialized testing
languages, requirements-based testing tools
• Dynamic analysis tools -intrusive tools modify source code by inserting
probes to check path coverage, assertions, or execution flow, non-
intrusive tools use a separate hardware processor running in parallel
with processor containing the program being tested
• Test management tools - coordinate regression testing, compare
actual and expected output, conduct batch testing, and serve as
generic test drivers
• Client/server testing tools - exercise the GUI and network
communications requirements for the client and server
• Reengineering tools
• reverse engineering to specification tools - generate analysis and
design models from source code, where used lists, and other design
information
• code restructuring and analysis tools -analyze program syntax,
generate control flow graph, and automatically generates a structured
program
• on-line system reengineering tools - used to modify on-line DBMS
Integrated CASE Environments
• Provide mechanism for sharing information among all tools contained
in the environment
• Enable changes to items to be tracked to other information items
• Provide version control and overall configuration management
• Allow direct access to any tool contained in the environment
• Establish automated support for the chosen Software Testing process
model, integrating CASE tools and SCI's into a standard work break
down structure
• Enable users of each tool to experience a consistent look and feel at
the human-computer interface
• Support communication among Software Testing engineers
• Collect both management and technical metrics to improve the process
and the product
Integration Architecture
• User interface layer
396 http://www.certmagic.com
BH0-003
• interface toolkit - contains Software Testing for UI management and
library of display objects
• common presentation protocol - guidelines that give all CASE tools the
same look and feel (icons, mouse behavior, menu names, object
names)
• Tools layer
• tools management services - control behavior of tools inside
environment
• CASE tools themselves
• Object management layer (OML) - performs the configuration
management function, working with the CASE repository OML provides
integration services
• Shared repository layer - CASE database and access control functions
enabling the OML to interact with the database
CASE Repository Functions
• Data integrity - includes functions to validate entries to the repository
and ensure consistency among related objects
• Information sharing - provides mechanism for sharing information
among multiple developers and multiple tools, controls modification of
information
• Data-tool integration - establishes shared data model and performs
configuration management functions
• Data-data integration - database management system allowing access
to related objects so functions can be achieved
• Methodology enforcement - the E-R model used to define steps needed
to be conducted to build the repository contents
• Document standardization - definition of objects in the database leads
directly to a standard approach for creation of engineering documents
Important DBMS Features Relevant to CASE Repositories
• Non-redundant data storage
• High-level access
• Data independence
• Transaction control
• Ad hoc data queries and reports
397 http://www.certmagic.com
BH0-003
• Openness
• Multi-user support
Special Features of CASE Repositories
• Storage of sophisticated data structures (diagrams, documents, files,
simple variables, information model describing relationships and
semantics of data stored in repository)
• Integrity enforcement (business rules, policies, constraints, and
requirements on the information being entered into repository, triggers
may be used to check the validity of the design models in real time)
• Semantic-rich tool interface (repository meta-model contains
semantics that enable a variety of tools to interpret meaning of data
stored in the repository)
• Process/project management(contains information about the Software
Testing application, the characteristics of each project, and the
organization's general process for Software Testing development -
phases, tasks, deliverables)
Software Testing Configuration Management Features Relevant to CASE
Repositories
• Versioning
• Dependency tracking and change management
• Requirements tracing
• Configuration management
• Audit trails
398 http://www.certmagic.com
BH0-003
Chapter 32 - The Road Ahead
Overview
The first 31 chapters of this text have explored a process for Software
Testing engineering. This chapter discusses some issues to consider when
trying understand how Software Testing and Software Testing engineering
will change in the future. Predicting future trends in any field requires
collecting data, organizing it, looking for subtle associations to extract
knowledge, and using this knowledge to suggest probable future
occurrences. A short-term prediction may or may not prove to be true and
yet, may still be true in the long term.
399 http://www.certmagic.com
BH0-003
Importance of Software Testing
• Software Testing as a differentiator (products, systems, services,
competitive advantages in the marketplace)
• Software Testing generates valuable information (programs,
documents, data)
• Mechanism for automating business, industry, and government
• Medium for transferring new technologies
• Means of capturing people's expertise for use by others
• Software Testing is a hidden technology, embedded in daily activities
and used without customers thinking about it
Scope of Change
• Changes in computing over the past 50 years have been driven by
advances in the hard sciences (physics, chemistry, materials science,
engineering)
• During the next several decades changes in computing are likely to be
driven by the soft sciences (human psychology, biology,
neurophysiology, sociology, philosophy)
• Changes in Software Testing engineering will be influenced by
• people who do the work
• processes they apply
• nature of information
• underlying computer technology
People and Systems Construction
• Systems are becoming more complex, requiring larger programs and
more people involved in their construction
• Communications between individual Software Testing engineers and
between specialized teams working on the same project will need to be
improved to avoid losing information
• The evolution of intelligent agents may change the work patterns of
Software Testing engineers by extending the capabilities of Software
Testing tools
• The World Wide Web has made many changes in the ways that people
acquire and access knowledge
400 http://www.certmagic.com
BH0-003
New Software Testing Engineering Process
• The first two decades of Software Testing engineering were
characterized by linear thinking, yet linear systems development runs
contrary to the ways in which most Software Testing systems are
actually built
• Evolutionary process models recognize that uncertainty dominates
most development activities
• Modern development time lines are impossibly short, iterative delivery
of partial products provides crucial functionality when complete
product delivery is not possible
• The Capability Maturity Model provides a good indicator about what
attributes should exist when solid Software Testing engineering is
practiced
• Object technologies coupled with component-based Software Testing
engineering are a natural outgrowth of the evolutionary process model
trend
• Customer involvement early in the design process is likely to be
observed more frequently
• Rapid growth in Web-based applications development is changing both
the Software Testing engineering process (focus in immediacy,
security, and aesthetics) and it participants (melding content non-
technical specialists with Software Testing designers)
New Modes for Representing Information
• Data processing has been replaced by the term information technology
• Emphasis is shifting from managing large quantities of data to
extracting meaningful information from this data
• Knowledge engineering techniques may begin migrating form the
artificial intelligence laboratories to the application domain as people
seeks ways to associate information from more than one context
• Software Testing systems may be viewed in the future as systems the
extract knowledge from data and information (many knowledge bases
have already been created)
The testing estimation process
In my opinion, one of the most difficult and critical activities in IT is
the estimation process. I believe that it occurs because when we say
that one project will be accomplished in such time by at such cost, it
must happen. If it does not happen, several things may follow: from
peers’ comments and senior management’s warnings to being fired
depending on the reasons and seriousness of the failure.
401 http://www.certmagic.com
BH0-003
Before even thinking of moving to Systems test at my organization, I
always heard from the development group members that the
estimations made by the Systems test group were too long and
expensive. Then, when I arrived at my new seat, I tried to
understand the testing estimation process.
The testing estimation process in place was quite simple. The inputs
for the process, provided by the development team, were: the size of
the development team and the number of working days needed for
building a solution before starting systems tests.
The testing estimation process said that the number of testing
engineers would be half of the number of development engineers and
one third of the number of development working days.
A spreadsheet was created in order to find out the estimation and
calculate the duration of tests and testing costs. They are based on
the following formulas:
Testing working days = (Development working days) / 3.
Testing engineers = (Development engineers) / 2.
Testing costs = Testing working days * Testing engineers *
person daily costs.
As the process was only playing with numbers, it was not necessary
to register anywhere how the estimation was obtained.
To exemplify how the process worked, if one development team said
that to deliver a solution for systems testing it would need 4
engineers and 66 working days then, the systems test would need 2
engineers (half) and 21 working days (one third). So, the solution
would be ready for delivery to the customer after 87 (66+21)
working days.
Just to be clear, in testing time, it was not included the time for
developing the testcases and preparing the testing environment.
Normally, it would need an extra 10 days for the testing team.
The new testing estimation process
Besides being simple, that process worked fine for different projects
and years. But, I was not happy with this approach and my
officemates from the development group were not, either. Metrics,
402 http://www.certmagic.com
BH0-003
project analogies, expertise, requirements, nothing were being used
to support the estimation process.
I mentioned my thoughts to the testing group. We could not stand
the estimation process for very long. I, myself, was not convinced to
support it any more. Then, some rules were implemented in order to
establish a new process.
Those rules are being shared below. I know that they are not
complete and it was not my intention for estimating but, from now, I
have strong arguments to discuss my estimation when someone
doubts my numbers.
T h e Ru l e s
1st Rule: Estimation shall be always based on the software requirements
All estimation should be based on what would be tested, i.e., the
software requirements.
Normally, the software requirements were only established by the
development team without any or just a little participation from the
testing team. After the specification have been established and the
project costs and duration have been estimated, the development
team asks how long would take for testing the solution. The answer
should be said almost right away.
Then, the software requirements shall be read and understood by the
testing team, too. Without the testing participation, no serious
estimation can be considered.
2nd Rule: Estimation shall be based on expert judgment
Before estimating, the testing team classifies the requirements in the
following categories:
♦ Critical: The development team has little knowledge in how to
implement it;
♦ High: The development team has good knowledge in how to
implement it but it is not an easy task;
♦ Normal: The development team has good knowledge in how to
implement.
The experts in each requirement should say how long it would take
for testing them. The categories would help the experts in
estimating the effort for testing the requirements.
3rd Rule: Estimation shall be based on previous projects
403 http://www.certmagic.com
BH0-003
All estimation should be based on previous projects. If a new project
has similar requirements from a previous one, the estimation is
based on that project.
4th Rule: Estimation shall be based on metrics
My organization has created an OPD, Organization Process Database, where
the project metrics are recorded. We have recorded metrics from three
years ago obtained from dozens of projects.
The number of requirements is the basic information for estimating a testing
project. From it, my organization has metrics that guide us to estimate a
testing project. The table below shows the metrics used to estimate a
testing project. The team size is 01 testing engineer.
Metric Value
1 Number of testcases created for each requirement 4,53
2 Number of testcases developed by Working day 14,47
3 Number of testcases executed by Working day 10,20
4 Number of ARs for testcase 0,77
5 Number of ARs verified by Working day 24,64
For instance, if we have a project with 70 functional requirements and a
testing team size of 2 engineers, we reach the following estimates:
Metric Value
Number of testcases – based on metric 317,10
1
Preparation phase – based on metric 2 11 working days
Execution phase – based on metric 3 16 working days
Number of ARs – based on metric 4 244 ARs
Regression phase – based on metric 5 6 working days
The testing duration is estimated in 22 (16+6) working days. Plus, 11
working days for preparing it.
5th Rule: Estimation shall never forget the past
I have not sent away the past. The testing team continues using the old
process and the spreadsheet. After the estimation is done following the new
rules, the testing team estimates again using the old process in order to
compare both results.
404 http://www.certmagic.com
BH0-003
Normally, the results from the new estimate process are cheaper and faster
than the old one in about 20 to 25%. If the testing team gets a different
percentage, the testing team returns to the process in order to understand if
something was missed.
6th Rule: Estimation shall be recorded
All decisions should be recorded. It is very important because if
requirements change for any reason, the records would help the testing team
to estimate again. The testing team would not need to return for all steps
and take the same decisions again. Sometimes, it is an opportunity to adjust
the estimation made earlier.
7th Rule: Estimation shall be supported by tools
A new spreadsheet has been created containing metrics that help to reach
the estimation quickly. The spreadsheet calculates automatically the costs
and duration for each testing phase.
There is also a letter template that contains some sections such as: cost
table, risks, and free notes to be filled out. This letter is sent to the
customer. It also shows the different options for testing that can help the
customer decides which kind of test he needs.
8th Rule: Estimation shall always be verified
Finally, All estimation should be verified. I’ve created another spreadsheet for
recording the estimations. The estimation is compared to the previous ones
recorded in a spreadsheet to see if they have similar trend. If the estimation
has any deviation from the recorded ones, then a re-estimation should be
made.
Test Design
To specify refinements of the test approach and to identify the features to be
covered by the design and its associated tests. It also identifies the test
cases and test procedures, if any, required to accomplish the testing and
specifies the feature pass/fail procedure.
405 http://www.certmagic.com
BH0-003
Outline:
-Test design specification identifier.
-Features to be tested.
-Approach refinements.
-Test case identification.
-Feature pass/fail criteria.
--------------------------------------------------------------------------------------
----------------------
When does test design begin? It begins in the requirements stage. Most
bugs are introduced in the requirements and design phase. What this means
is the we need to be involved with the whole product life cycle from the
beginning.
The typical stage of test design starts with requirements generation. It is at
this point that a high level test design is done, plus a determination can be
made of the components that would need to be tested, as well as an overall
integration.
This continues with the design stage, where more detailed test plans and
details can be created.
When designing tests, there are some things to keep in mind: the main
objectives.
Objective in test design:
1. Detect as many defects as possible.
2. Minimize test development costs.
3. Minimize test execution costs.
4. Minimize test maintenance costs.
Detailed design considerations:
1. Satisfying test development objectives.
2. Conforming to the test architecture.
3. Design of each test case.
The steps to detailed design:
1. Identifying the items that should be tested.
2. Assigning priorities to those items based on risk.
3. Designing high level test designs for similar groups of items.
4. Designing individual test cases based on the high level designs.
406 http://www.certmagic.com
BH0-003
The determination must be made which of the following are highest to lowest
priority in the testing process to determine what is the critical path with
regards to test design.
Types of system test:
1. Volume
2. Usability
3. Performance
4. Configuration
5. Compatibility
6. Reliability
7. Load/Stress
8. Security
9. Resource Usage
10.Installability
11.Recovery
12.Serviceability
The other things involved with test design involve determining the metrics
that will be used to determine when the testing group will sign off on the
software for release. Also, what type of physical resources will be necessary
(lab space, hardware, other related tools, or any specific software and
licenses).
A very useful book was “Software Testing in the Real World” by Ed Kit.
1. What is the percent of the total gross of sales that come from product
failure?
20% (Manufacturing Business)
30% (Service Sector)
3. What is the cost of quality?
Prevention Cost + Appraisal Cost + Failure Cost
(Total cost in multiple iterations – cost of doing things right first time)
4. What is management by fact?
Management by fact is using quantitative measures and metrics to
manage the planning, execution, and reporting of software testing.
5. What are the three types of interfaces?
• Interfaces
407 http://www.certmagic.com
BH0-003
o Person/Machine - Interfaces that include the operating
system and the development languages that are available, as
well as the input/output facilities.
o Communications Interfaces - Interfaces that include
transmission of information between computers and remote
equipment (e.g., transmission of computer data over networks.)
o Program Interfaces - Interfaces for the exchange of
information, whether on the same computer, or distributed
across multiple tiers of the application architecture.
6. What three rules should be followed for all reviews?
a. Review the Product not the author
b. Focus on Defect
c. Overall responsibility as a team.
7. What is boundary value testing?
The kind of testing in which test data or test cases are selected by
identified the boundaries that separate valid and invalid conditions.
Tests are conducted to test the inside and outside of these boundaries,
in addition to these boundary points. Experience suggest that these
test have higher payoff than selecting the random values for testing.
8. What is decision/branch coverage strategy?
Branch Coverage Testing seeks to ensure that every branch has been
executed. Branch Coverage can be tested by probes inserted at points
in the program that represent arcs from branch points in the flow
graphs.
9. Which of the following is not one of the 6 Structural Test Approaches?
Options not present … These are six Structural Testing
o Load/Stress .
o Execution
o Recovery
o Operations
o Compliance (to process)
o Security -
10.Which of the following is not one of the 8 Functional Test Approaches?
o Requirements
o Regression
o Error Handling
o Manual Support.
o Interfaces/Intersystem
o Control
o Parallel –
408 http://www.certmagic.com
BH0-003
o Acceptance Testing
11. Which of the following is not a perspective of quality?
a.
transcendent
b.
product-based
c.
translucent
d.
user-based
e.
value-based
f.
manufacturing based Translucent
12. True or False. Effectiveness is doing things right and efficiency is
doing the right things. False
13. Which of the following is not one of Deming's 14 points for
management?
a. Adopt a new philosophy
b. Eliminate slogans, exhortations, and targets for the work force
c. Mobility of management
d. Create constancy of purpose
14. True or False. The largest cost of quality is from production failure.
True
15. Defects are least costly to correct at what stage of the development
cycle?
a. Requirements
b. Analysis & Design
c. Construction
d. Implementation
16. A review is what category of cost of quality?
a. Preventive
b. Appraisal
c. Failure
17. True or False. A defect is related to the term fault. Not sure
18. What type of change do you need before you can obtain a behavior
change?
a. Lifestyle
b. Vocabulary
c. Internal
d. Management
19. Software testing accounts for what percent of software development
costs?
a. 10-20
b. 40-50
c. 70-80
d. 5-10
20. The purpose of software testing is to:
a. Demonstrate that the application works properly
b. Detect the existence of defects
c. Validate the logical design
409 http://www.certmagic.com
BH0-003
21. True or False. One of the key concepts of a task force is that the
leader be an expert in leading groups as opposed to an expert in a topical
area. True
22. Match the following terms with their definitions:
a. Black box testing
b. White box testing
c. Conversion testing
d. Thread testing
e. Integration testing
Definitions not found :
Black Box Testing : A test techniques that focuses on testing the
functionality of the program , component, or application against its
specifications without the knowledge of how the system is constructed ;
usually data or business process driven. Logic or path is unknown.
White Box Testing : A test technique that assumes that path of the logic of
the program is known. White box testing is usually consists of testing paths,
branch by branch, to produce predictable results. This test technique is
usually used by the development team in testing units and components.
Conversion Testing : A test technique that validates the effectiveness of data
conversion process, includes fields to field mapping, and data transaction.
Thread Testing : Test technique in which a string of unit which accomplish a
specific function is tested.
Integration Testing: Testing two or more programs or components that
interact with each other to validate the quality and correct data transfer and
interfaces.
Chapter 1: The Product
Chapter 1 Quiz
1. What factor has precipitated more sophisticated and complex computer-
based systems?
a. Vast increases in computer memory and storage capacity.
b. Greater variety of exotic input/output options.
c. Profound changes in computer architectures.
d. All of the above.
410 http://www.certmagic.com
BH0-003
2. Which question no longer concerns the modern software engineer?
a. Why does computer hardware cost so much?
b. Why does software take a long time to finish?
c. Why does it cost so much to develop a piece of software?
d. Why can't software errors be removed from products prior to delivery?
3. Today the increased power of the personal computer has brought about an
abandonment of the practice of team development of software.
a. True
b. False
4. Software is a product and can be manufactured using the same
technologies used for other engineering artifacts.
a. True
b. False
5. Software deteriorates rather than wears out because
a. Software suffers from exposure to hostile environments
b. Defects are more likely to arise after software has been used often
c. Multiple change requests introduce errors in component interactions
d. Software spare parts become harder to order
6. Most software continues to be custom built because
a. Component reuse is common in the software world
b. Reusable components are too expensive to use
c. Software is easier to build without using someone else's components.
d. Off the shelf software components are not commonly available
7. The nature of software applications can be characterized by their
information
a. complexity
b. content
c. determinacy
d. choices "b" and "c"
8. Modern software applications are so complex that it is hard to develop
mutually exclusive category names.
a. True
b. False
411 http://www.certmagic.com
BH0-003
9. The current software crisis was caused by the Y2K problem whose seeds
were first sown by careless programmers in the early 1970's.
a. True
b. False
10. Software developers succeed more often than they fail, but software
failures receive more press coverage.
a. True
b. False
11. Adding more people to a project that is already behind schedule is a good
way to catch up.
a. True
b. False
12. Modern CASE tools are more important than the newest hardware for
achieving good software quality and productivity.
a. True
b. False
13. Change cannot be easily accommodated in most software systems,
unless a system was designed with change in mind.
a. True
b. False
14. A general statement of objectives is all that is needed to begin
developing a piece of software.
a. True
b. False
15. The formal technical review is an inadequate substitute for testing
regardless of nature of the software defect.
a. True
b. False
16. Documentation is no longer a necessary part of the software
development process because no one reads it.
a. True
b. False
412 http://www.certmagic.com
BH0-003
Chapter 2: The Process
Chapter 2 Self-Check Quiz
1. Which of the items listed below is not one of the software engineering
layers?
a. Process
b. Manufacturing
c. Methods
d. Tools
2. What are the three generic phases of software engineering?
a. definition, development, support
b. what, how, where
c. programming, debugging, maintenance
d. analysis, design, testing
3. Which of these terms is a level name in the Capability Maturity Model?
a. Ad hoc
b. Repeatable
c. Reusable
d. Organized
4. Which of these items should be used to select a software process
framework?
a. People
b. Product
c. Project
d. All of the above
413 http://www.certmagic.com
BH0-003
5. In which software development problem solving stage are the results
delivered?
a. Status quo
b. Problem definition
c. Technical development
d. Solution integration
6. Software development activities are easy to compartmentalize into four
non-overlapping phases?
a. True
b. False
7. The linear sequential model of software development is
a. A reasonable approach when requirements are well defined.
b. A good approach when a working program is required quickly.
c. The best approach to use for projects with large development teams.
d. An old fashioned model that is rarely used any more.
8. The linear sequential model of software development is also known as the
a. Classical life cycle model
b. Fountain model
c. Spiral model
d. Chaos model
9. The prototyping model of software development is
a. A reasonable approach when requirements are well defined.
b. A useful approach when a customer cannot define requirements clearly.
c. The best approach to use for projects with large development teams.
d. A risky model that rarely produces a meaningful product.
10. The rapid application development model is
a. Another name for component-based development.
b. A useful approach when a customer cannot define requirements clearly.
c. A high-speed adaptation of the linear sequential model.
d. All of the above.
11. Evolutionary software process models
a. Are iterative in nature
b. Can easily accommodate product requirements changes
c. Do not generally produce throwaway systems
d. All of the above
414 http://www.certmagic.com
BH0-003
12. The incremental model of software development is
a. A reasonable approach when requirements are well defined.
b. A good approach when a working core product is required quickly.
c. The best approach to use for projects with large development teams.
d. A revolutionary model that is not used for commercial products.
13. The spiral model of software development
a. Ends with the delivery of the software product
b. Is more chaotic than the incremental model
c. Includes project risks evaluation during each iteration
d. All of the above
14. The WINWIN spiral model of software development is
a. A used when requirements must be defined by customer negotiation.
b. Useful when a customer is able to provide requirements completely.
c. The best approach to use for projects with large development teams.
d. Like the spiral model without the risk assessment step.
15. The concurrent development model is
a. Another name for the rapid application development model.
b. Often used for the development of client/server applications.
c. Only used for development of parallel or distributed systems.
d. Used whenever a large number of change requests are anticipated.
16. The component-based development model is
a. Only appropriate for computer hardware design.
b. Not able to support the development of reusable components.
c. Dependent on object technologies for support.
d. Not cost effective by known quantifiable software metrics.
17. The formal methods model of software development makes use of
mathematical methods to
a. Define the specification for computer-based systems
b. Develop defect free computer-based systems
c. Verify the correctness of computer-based systems
d. All of the above
18. Fourth generation techniques
a. Allow software to be developed without any testing.
b. Eliminate the need for costly requirements gathering activities.
c. Can reduce the time required to develop software.
d. Are best used by non-programmers to build small systems.
415 http://www.certmagic.com
BH0-003
Chapter 3: Project Management Concepts
Chapter 3 Self-Check Quiz
1. Effective software project management focuses on
a. people, performance, payoff, product
b. people, product, performance, process
c. people, product, process, project
d. people, process, payoff, product
2. Organizations that achieve high levels of maturity in people management
have a higher likelihood of implementing effective software engineering
processes.
a. True
b. False
3. The first step in project planning is to
a. determine the budget.
b. select a team organizational model.
c. determine the project constraints.
d. establish the objectives and scope.
4. Process framework activities are populated with
a. milestones
b. work products
c. QA points
d. All of the above
5. Project management is less important for modern software development
since most projects are successful and completed on time.
a. True
b. False
6. Which of the following is not considered a player in the software process?
a. customers
b. end-users
c. project managers
416 http://www.certmagic.com
BH0-003
d. sales people
7. The best person to hire as a project team leader is the most competent
software engineering practitioner available.
a. True
b. False
8. The best project team organizational model to use when tackling
extremely difficult problems is the
a. controlled centralized model
b. controlled decentralized model
c. democratic decentralized model
d. chief programmer team model
9. Which factor is the least important when choosing the organizational
structure for a software team?
a. degree of communication desired
b. predicted size of the resulting program
c. rigidity of the delivery date
d. size of the project budget
10. One of the best ways to avoid frustration during the software
development process is to
a. give team members more control over process and technical decisions.
b. give team members less control over process and technical decisions.
c. hide bad news from the project team members until things improve.
d. reward programmers based on their productivity.
11. Which of these software characteristics is not a factor contributing to
project coordination difficulties?
a. interoperability
b. performance
c. scale
d. uncertainty
12. Which of these software characteristics are used to determine the scope
of a software project?
a. context, lines of code, function
b. context, function, communication requirements
c. information objectives, function, performance
d. communications requirements, performance, information objectives
417 http://www.certmagic.com
BH0-003
13. The major areas of problem decomposition during the project scoping
activity are the
a. customer workflow
b. functionality to be delivered
c. process used to deliver functionality
d. answers b and c
14. Product and process decomposition occurs simultaneously as the project
plan evolves.
a. True
b. False
15. When can selected common process framework activities be omitted
during process decomposition?
a. when the project is extremely small in size
b. any time the software is mission critical
c. rapid prototyping does not require their use
d. never, the activities are invariant
16. What activity does a software project manager need to perform to
minimize the risk of software failure?
a. double the project team size
b. request a large budget
c. allow absolutely no schedule slippage
d. define milestones and track progress
17. Which of the following questions is not addressed when the W5HH
principle is applied?
a. Why is the system being developed?
b. What will be done by whom?
c. Where are they organizationally located?
d. How much of each resource is required?
18. Which of these is not a critical practice for performance-based project
management?
a. assessing product usability
b. defect tracking against quality targets
c. empirical cost estimation
d. formal risk management
418 http://www.certmagic.com
BH0-003
Chapter 4: Software Process and Project Metrics
Chapter 4 Self-Check Quiz
1. Which of these is not a valid reason for measuring software processes,
products, and resources?
a. to characterize them
b. to evaluate them
c. to price them
d. to improve them
2. The terms measure, measurement, and metric all share the same
definition according to the IEEE Standard Glossary of Software Engineering
Terms.
a. True
b. False
3. Process indicators enable a software project manager to
a. assess the status of an on-going project
b. track potential risks
c. adjust work flow or tasks
d. all of the above
4. Public metrics are used
a. to make strategic changes to the software process.
b. to make tactical changes during a software project.
c. to evaluate the performance of software development teams.
d. answers a and b
5. Which of the following items are not measured by software project
metrics?
a. inputs
b. markets
c. outputs
d. results
6. Software quality and functionality must be measured indirectly.
a. True
b. False
419 http://www.certmagic.com
BH0-003
7. Which of following are advantages of using LOC (lines of code) as a size-
oriented metric?
a. LOC is easily computed.
b. LOC is a language dependent measure.
c. LOC is a language independent measure.
d. LOC can be computed before a design is completed.
8. Which of the following is an advantage of using function points (FP) as a
measure of the functionality delivered by a software application?
a. FP is easily computed.
b. FP is a language dependent measure.
c. FP is a function of LOC.
d. FP can be computed before a design is completed.
9. Extended function point metrics are designed to be applied to
a. business information systems applications
b. PC applications
c. real-time or embedded applications
d. computer graphics applications
10. Backfiring is the best way to compute function point measures once a
software application is completed.
a. True
b. False
11. Which of the following software quality factors is most likely to be
affected by radical changes to computing architectures?
a. operation
b. transition
c. revision
d. none of the above
12. Which of the following provide useful measures of software quality?
a. correctness, performance, integrity, usability
b. reliability, maintainability, integrity, sales
c. correctness, maintainability, size, satisfaction
d. correctness, maintainability, integrity, usability
13. A software quality metric that can be used at both the process and
project levels is defect removal efficiency (DRE).
a. True
b. False
420 http://www.certmagic.com
BH0-003
14. Why is it important to measure the process of software engineering and
software it produces?
a. It is really not necessary unless the project is extremely complex.
b. To determine costs and allow a profit margin to be set.
c. To determine whether a software group is improving or not.
d. To make software engineering more like other engineering processes.
15. To be an effective aid in process improvement the baseline data used
must be:
a. based on reasonable guesstimates from past projects
b. measured consistently across similar projects
c. collected over the past 6 months
d. based on all previously completed projects
16. Baseline data must be collected in an on-going manner and cannot be
computed by formal study of historical project data.
a. True
b. False
17. One graphical technique for determining whether a process exhibits out-
of-control change behavior is a
a. control chart
b. fishbone diagram
c. Pareto diagram
d. process diagram
18. Zone rules may be used to
a. allocate software functions to team members
b. determine a marketing strategy for a product
c. identify an out-of-control process
d. validate a set of software process metrics
19. Small software organizations are not likely to see any economic return
from establishing software metrics program.
a. True
b. False
20. The software metrics chosen by an organization are driven by the
business or technical goals an organization wishes to accomplish.
a. True
b. False
421 http://www.certmagic.com
BH0-003
Chapter 5: Software Project Planning
Chapter 5 Self-Check Quiz
1. The only reason an estimate is unreliable is lack of experience with the
application on the part of the estimator.
422 http://www.certmagic.com
BH0-003
a. True
b. False
2. Since project estimates are not completely reliable, they can be ignored
once a software development project begins.
a. True
b. False
3. The objective of software project planning is to
a. convince the customer that a project is feasible.
b. make use of historical project data.
c. enable a manager to make reasonable estimates of cost and schedule.
d. determine the probable profit margin prior to bidding on a project.
4. The project scope is defined as a means of bounding the system
a. Functionality and performance
b. Staffing and skills
c. Costs and resources
d. Schedule and milestones
5. The most common way to determine the information needed to define
project scope is to
a. conduct a preliminary meeting with the customer.
b. examine historical project data from similar applications.
c. build a software prototype and show it to the customer.
d. perform a market analysis to determine potential customers.
6. Software feasibility is based on which of the following
a. business and marketing concerns
b. scope, constraints, market
c. technology, finance, time, resources
d. technical prowess of the developers
7. A consideration of software scope must include an evaluation of all
external interfaces.
a. True
b. False
8. The number of people required for a software project is determined
a. after an estimate of the development effort is made.
b. by the size of the project budget.
423 http://www.certmagic.com
BH0-003
c. from an assessment of the technical complexity of the system.
d. all of the above
9. Reusable software components must be
a. catalogued for easy reference.
b. standardized for easy application
c. validated for easy integration
d. all of the above
10. The software engineering environment (SEE) consists of which of the
following?
a. customers and users
b. developers and managers
c. hardware platforms and software tools
d. none of the above
11. The hardware required for most computer-based systems is more costly
to purchase than the software.
a. True
b. False
12. Which of the following is a broad classification of software project
estimation techniques?
a. automated processes
b. white-box methods
c. empirical models
d. regression models
13. The size estimate for a software product to be built must be based on a
direct measure like LOC.
a. True
b. False
14. An expected value estimate is determined by computing the weighted
average of
a. estimates from three different estimators
b. three different size estimates for the same project
c. three different size estimates from similar projects
d. none of the above
424 http://www.certmagic.com
BH0-003
15. Problem-based estimation is based on problem decomposition that
focuses on
a. information domain values and software functions
b. project schedule and milestones
c. LOC and FP counts
d. process activities
16. LOC-based estimation techniques require problem decomposition based
on
a. information domain values
b. project schedule
c. software functions
d. process activities
17. FP-based estimation techniques require problem decomposition based on
a. information domain values
b. project schedule
c. software functions
d. process activities
18. Process-based estimation techniques require problem decomposition
based on
a. information domain values and data objects
b. project schedule and milestones
c. software functions and process activities
d. none of the above
19. Empirical estimation models are typically based on
a. expert judgement based on past project experiences
b. refinement of expected value estimation
c. regression models derived from historical project data
d. trial and error determination of the parameters and coefficients
20. COCOMO II is an example of a suite of modern empirical estimation
models that require sizing information expressed as:
a. function points
b. lines of code
c. object points
d. any of the above
425 http://www.certmagic.com
BH0-003
21. Putnam's software equation is a dynamic empirical model that has two
independent parameters: a size estimate and an indication of project
duration in calendar months or years.
a. True
b. False
22. Using a statistical technique like decision tree analysis can provide some
assistance in sorting out the true costs associated with the make-buy
decision.
a. True
b. False
23. Outsourcing always provides a simple means of acquiring software at
lower cost than on site development of the same product.
a. True
b. False
24. A weakness of the current generation of automated estimation tools is
the
a. high cost of acquisition and use
b. inability of these tools to take software reuse into account when making
an estimate
c. inability to integrate LOC and FP data
d. significant differences between tool estimates and actual values on
several projects
Chapter 6: Risk Analysis & Management
Chapter 6 Self-Check Quiz
426 http://www.certmagic.com
BH0-003
1. Proactive risk management is sometimes described as fire fighting.
a. True
b. False
2. Software risk always involves two characteristics
a. fire fighting and crisis management
b. known and unknown risks
c. uncertainty and loss
d. staffing and budget
3. Three categories of risks are
a. business risks, personnel risks, budget risks
b. project risks, technical risks, business risks
c. planning risks, technical risks, personnel risks
d. management risks, technical risks, design risks
4. Generic risks require far more attention than product-specific risks.
a. True
b. False
5. Which of these categories would not be likely to be contained within a risk
item checklist?
a. product size
b. development environment
c. staff size
d. product profitability
6. Which question would be irrelevant when assessing the overall software
project risk?
a. Have top managers formally committed to support the project?
b. Are end-users committed to the project and proposed system being
built?
c. Are requirements fully understood by development team and
customers?
d. Does the proposed budget have time allocated for marketing?
7. Software risk impact assessment should focus on consequences affecting
a. planning, resources, cost, schedule
b. marketability, cost, personnel
c. business, technology, process
d. performance, support, cost, schedule
427 http://www.certmagic.com
BH0-003
8. Risk projection attempts to rate each risk in two ways
a. likelihood and cost
b. likelihood and impact
c. likelihood and consequences
d. likelihood and exposure
9. Risk tables are sorted by
a. probability and cost
b. probability and impact
c. probability and consequences
d. probability and exposure
10. Individual team members can make their own estimate for a risk
probability and then develop a consensus value.
a. True
b. False
11. Which factor(s) likely affect the probable consequences if a risk does
occur?
a. risk cost
b. risk timing and scope
c. risk resources
d. all of the above
12. A risk referent level is a risk component value (performance, cost,
support, schedule) or combination of values that cause a project to be
terminated.
a. True
b. False
13. The reason for refining risks is to break them into smaller units having
different consequences.
a. True
b. False
14. An effective risk management plan will need to address which of the
following issues?
428 http://www.certmagic.com
BH0-003
a. risk avoidance
b. risk monitoring
c. contingency planning
d. all of the above
15. Risk monitoring involves watching the risk indicators defined for the
project and not determining the effectiveness of the risk mitigation steps
themselves.
a. True
b. False
16. Hazard analysis focuses on the identification and assessment of potential
hazards that can cause
a. project termination
b. schedule slippage
c. cost overruns
d. an entire system to fail
17. Risk information sheets (RIS) are never an acceptable substitute for a full
risk mitigation, monitoring, and management (RMMM) plan.
a. True
b. False
Chapter 7: Project Scheduling and Tracking
Chapter 7 Self-Check Quiz
1. Software projects are inevitably late and there is nothing that can explain
why.
a. True
b. False
2. It is unethical to undertake a project that you know in advance can not be
completed by the customer's deadline, unless you inform the customer of the
risk and establish a project plan that can deliver the needed system
incrementally.
a. True
429 http://www.certmagic.com
BH0-003
b. False
3. Which of the following is not one of the guiding principles of software
project scheduling?
a. compartmentalization
b. market assessment
c. time allocation
d. effort validation
4. If you must add people to a late project, be certain that they are assigned
to highly compartmentalized tasks.
a. True
b. False
5. Doubling the size of your software project team is guaranteed to cut
project completion time in half.
a. True
b. False
6. The software equation can be used to show that by extending the project
deadline slightly
a. fewer people are required
b. you are guaranteed to meet the deadline
c. more lines of code can be produced
d. none of the above
7. The 40-20-40 rule suggests that the least amount of development effort
be spent on
a. estimation and planning
b. analysis and design
c. coding
d. testing
8. A task set is a collection of
a. engineering work tasks, milestones, deliverables
b. task assignments, cost estimates, metrics
c. milestones, deliverables, metrics
d. responsibilities, milestones, documents
430 http://www.certmagic.com
BH0-003
9. A task set will grow in size and complexity as the degree of rigor
a. shrinks
b. changes
c. grows
d. all of the above
10. The term back-filling refers to writing additional documentation and
conducting additional reviews after a project is delivered.
a. True
b. False
11. Adaptation criteria are used to determine the
a. costs of product maintenance
b. adjustments to the project schedule
c. best project type classification for a software process
d. recommended degree of rigor for software process
12. A task selector value is most appropriately used to determine whether to
accept or reject a given task for inclusion in the project task set.
a. True
b. False
13. If the task selector value is in an overlap area it may be OK to choose a
less formal degree of rigor for project with low risk levels.
a. True
b. False
14. For purposes of determining the major engineering tasks and distributing
them on the project time line, the project manager should assume that the
process model used is
a. linear sequential
b. iterative
c. evolutionary
d. any of the above
15. The only means in accomplishing task refinement is to make use of a
process design language approach.
a. True
b. False
16. The task (activity) network is a useful mechanism for
431 http://www.certmagic.com
BH0-003
a. computing the overall effort estimate
b. detecting inter-task dependencies
c. specifying the task set to the customer
d. none of the above
17. Two tools for computing critical path and project completion times from
activity networks are
a. CPM and PERT
b. DRE and SQA
c. FP and LOC
d. ASD and BSD
18. Timeline charts assist project managers in determining what tasks will be
conducted at a given point in time.
a. True
b. False
19. The best indicator of progress on a software project is the completion
a. of a defined engineering activity task
b. of a successful budget review meeting on time
c. and successful review of a defined software work product
d. and successful acceptance of project prototype by the customer
20. The purpose of earned value analysis is to
a. determine how to compensate developers based on their productivity
b. provide a quantitative means of assessing software project progress
c. provide a qualitative means of assessing software project progress
d. set the price point for a software product based on development effort
21. Error tracking provides a quantitative means of assessing the quality of
the individuals implementing a software product.
a. True
b. False
22. The software plan is not a static document, it is frequently adjusted to
make the project appear on track to meet all deadlines and quality targets.
a. True
b. False
432 http://www.certmagic.com
BH0-003
Chapter 8: Software Quality Assurance
Chapter 8 Self-Check Quiz
1. Variation control in the context of software engineering involves
controlling variation in the
a. process applied
b. resources expended
c. product quality attributes
d. all of the above
2. There is no need to assess customer satisfaction when trying to determine
the quality of a piece of software.
a. True
b. False
3. A key concept of quality control is that all work products
433 http://www.certmagic.com
BH0-003
a. are delivered on time and under budget
b. have complete documentation
c. have measurable specifications for process outputs
d. are thoroughly tested before delivery to the customer
4. The goal of quality assurance is to provide management with the data
needed to determine which software engineers are producing the most
defects.
a. True
b. False
5. Quality costs may be divided into costs associated with
a. prevention, appraisal, and failure
b. people, process, and product
c. customers, developers, and maintenance
d. all of the above
6. Until a mature software process has been achieved an organization would
be wise to spend most of its efforts on which TQM step
a. developing a visible, repeatable, measurable process
b. examining the ways in which customers use their products
c. observing the use of their products in the marketplace
d. optimizing the impact of intangibles on their current process
7. Software quality might be defined as conformance to explicitly stated
requirements and standards, nothing more and nothing less.
a. True
b. False
8. People who perform software quality assurance must look at the software
from the customer's perspective.
a. True
b. False
9. Which of these activities is not one of the activities recommended to be
performed by an independent SQA group?
a. prepare SQA plan for the project
b. review software engineering activities to verify process compliance
c. report any evidence of noncompliance to senior management
d. serve as the sole test team for any software produced
434 http://www.certmagic.com
BH0-003
10. The purpose of software reviews is to uncover errors and defects in work
products so they can be removed before moving on to the next phase of
development.
a. True
b. False
11. In general the earlier a software defect is discovered and corrected the
less costly to the overall project budget.
a. True
b. False
12. Defect amplification models can be used to illustrate the costs associated
with using software from its initial deployment to its retirement.
a. True
b. False
13. Which of the following are objectives for formal technical reviews?
a. allow senior staff members to correct errors
b. assess programmer productivity
c. par determining who introduced an error into a program
d. uncover errors in software work products
14. At the end of a formal technical review all attendees can decide to
a. accept the work product without modification
b. modify the work product without further review
c. reject the product due to severe errors
d. all of the above
15. A review summary report answers which three questions?
a. terminate project, replace producer, request a time extension
b. what defects were found, what caused defects, who was responsible
c. what was reviewed, who reviewed it, what were the findings
d. none of the above
16. In any type of technical review, the focus of the review is on the product
and not the producer.
a. True
b. False
17. Statistical quality assurance involves
a. using sampling in place of exhaustive testing of software
435 http://www.certmagic.com
BH0-003
b. surveying customers to find out their opinions about product quality
c. tracing each defect to its underlying cause, isolating the"vital few" causes,
and moving to correct them
d. tracing each defect to its underlying causes and using the Pareto principle
to correct each problem found
18. Software reliability problems can almost always be traced to
a. errors in requirements gathering
b. errors in design and implementation
c. human error
d. errors in operation
19. Software safety is a quality assurance activity that focuses on hazards
that
a. affect the reliability of a software component
b. may cause an entire system to fail
c. may result from user input errors
d. prevent profitable marketing of the final product
20. Poka-yoke devices are mechanisms that lead to the
a. creation of quality processes with minimal resources
b. determining causes of software defects
c. prevention of potential quality problems
d. none of the above
21. The ISO quality assurance standard that applies to software engineering
is
a. ISO 9000
b. ISO 9001
c. ISO 9002
d. ISO 9003
22. Which of the following is not a section in the standard for SQA plans
recommended by IEEE?
a. budget
b. documentation
c. reviews and audits
d. test
436 http://www.certmagic.com
BH0-003
Chapter 9: Software Configuration Management
Chapter 9 Self-Check Quiz
1. Which of these are valid software configuration items?
a. case tools
b. documentation
c. executable programs and test data
d. all of the above
2. Once a software engineering work product becomes a baseline it can not
be changed again.
a. True
b. False
3. Which configuration objects would not typically be found in the project
database?
a. design specification
b. marketing data
c. source code
d. test plans
437 http://www.certmagic.com
BH0-003
4. Which of the following tasks is not part of software configuration
management?
a. change control
b. reporting
c. statistical quality control
d. version control
5. A basic configuration object is a __________created by a software
engineer during some phase of the software development process.
a. program data structure
b. hardware driver
c. unit of text
d. all of the above
6. An E-R diagram can be used to show the interrelationships among
configuration objects.
a. True
b. False
7. A(n) __________ is composed of objects at the same revision level.
a. entity
b. item
c. variant
d. version
8. A(n)__________ is a different collection of objects at the same revision
level.
a. entity
b. item
c. variant
d. version
9. A new__________ is defined when major changes have been made to one
or more objects.
a. entity
b. item
c. variant
d. version
438 http://www.certmagic.com
BH0-003
10. Change control is not necessary if a development group is making use of
an automated project database tool.
a. True
b. False
11. The check-in and check-out process helps with which element of change
control?
a. budget control
b. object control
c. synchronization control
d. version control
12. Every customer change request is transformed into an engineering
change order, regardless of its impact on the project.
a. True
b. False
13. Configuration audits are needed even if you make use of formal technical
reviews as part of your software engineering process.
a. True
b. False
14. When software configuration management is a formal activity the
software configuration audit is conducted by the
a. development team
b. quality assurance group
c. senior managers
d. testing specialists
15. The primary purpose of configuration status reporting is to
a. allow revision of project schedule and cost estimates by project managers
b. evaluate the performance of software developers and organizations
c. make sure that change information is communicated to all affected parties
d. none of the above
439 http://www.certmagic.com
BH0-003
Chapter 10: System Engineering
Chapter 10 Self-Check Quiz
1. Software engineers do not need to consider hardware when designing a
computer-based system.
a. True
b. False
2. Which of the following can be elements of computer-based systems?
a. Documentation and data
b. hardware and software
c. people
d. all of the above
3. The system engineering process usually begins with the
a. detailed view
b. domain view
c. element view
d. world view
4. To construct a system model the engineer should consider one of the
following restraining factors?
a. assumptions and constraints
b. budget and expenses
c. data objects and operations
d. schedule and milestones
5. By following modern system engineering practices simulation of reactive
systems is no longer necessary.
a. True
b. False
440 http://www.certmagic.com
BH0-003
6. During business process engineering, three different architectures are
examined.
a. applications, data, technology infrastructure
b. communications, organization, financial infrastructure
c. network,database, reporting structure
d. systems, requirements, data structure
7. Which elements of business processing engineering are the responsibilities
of the software engineer?
a. business area analysis
b. business system design
c. product planning
d. information strategy planning
8. The goal of product engineering is to translate the customer's desire for a
set of defined capabilities into a working product.
a. True
b. False
9. The architecture components for product engineering are
a. data, hardware, software, people
b. data, documentation, hardware, software
c. data, hardware, software, procedures
d. documentation, hardware, people, procedures
10. What makes requirements elicitation difficult?
a. bounding scope
b. understanding user needs
c. requirements volatility
d. all of the above
11. It is not possible to consider overall feasibility until the detailed
requirements have been elicited from the customer.
a. True
b. False
12. It is relatively common for different customers to propose conflicting
requirements, each arguing that his or her version is the right one.
a. True
b. False
441 http://www.certmagic.com
BH0-003
13. The system specification describes the
a. function and behavior of a computer-based system
b. implementation of each allocated system element
c. algorithmic detail and data structures
d. time required for system simulation
14. System models are built to allow the system engineer to evaluate the
system components in relationship to one another,
a. True
b. False
15. The best way to conduct a requirements validation review is to
a. examine the system model for errors
b. have the customer look over the requirements
c. send them to the design team and see if they have any concerns
d. use a checklist of questions to examine each requirement
16. The use of traceability tables helps to
a. debug programs following the detection of run-time errors
b. determine the performance of algorithm implementations
c. identify, control, and track requirements changes
d. none of the above
17. The top level of the hierarchical model of a system is known as the
a. AFD
b. DFD
c. SCD
d. SFD
18. The system model template contains which of the following elements
a. input c. user interface
b. output d. all of the above
Chapter 11: Analysis Concepts and Principles
Chapter 11 Self-Check Quiz
1. One of the most difficult parts of software requirements analysis is
ensuring the developer understands the customer's needs.
a. True
b. False
442 http://www.certmagic.com
BH0-003
2. What task is not performed as part of software requirements analysis?
a. evaluation and synthesis
b. modeling and problem recognition
c. planning and scheduling
d. specification and review
3. The use of context free questions by themselves provides an effective
means of eliciting requirements information from the customer.
a. True
b. False
4. The goal of facilitated application specification techniques(FAST) is to have
the developer and customer
a. construct a software prototype quickly
b. learn each other's jobs
c. work together to develop a preliminary set of requirements
d. work together to develop the technical software specification
5. Which of these people would not be likely to part of the FAST team?
a. hardware and software engineers
b. manufacturing representative
c. marketing representatives
d. senior financial officers
6. Which of these requirements are considered during quality function
deployment(QFD)?
a. exciting requirements
b. expected requirement
c. normal requirements
d. technology requirements
7. Value analysis is conducted as part of quality function deployment to
determine the
a. cost of project quality assurance activities.
b. relative cost of requirements during function, task, and information
deployment.
c. relativepriority of requirements during function, task, and information
deployment.
d. size of the customer voice table.
443 http://www.certmagic.com
BH0-003
8. Use-cases are scenarios that describe
a. how software is to be used in a given situation.
b. how CASE tools will be used to construct the system.
c. the build plan for a software product.
d. the test cases for a software product.
9. The actors described in use-cases are the people who are the intended
software product users.
a. True
b. False
10. Information content represents the individual data and control objects
that comprise the information that is
a. necessary to lay out all output.
b. required for error handling.
c. demanded for operating system interfacing.
d. transformed by the software.
11. Information flow represents the manner in which data and control
a. are related to one another.
b. changeas each moves through the system.
c. will be implemented in the final design.
d. none of the above.
12. Information structure represents the internal organization of the
a. data structures used to represent data types.
b. project staffing model.
c. project communications model.
d. various data and control items.
13. What types of models are created during software requirements analysis?
a. functional and behavioral
b. algorithmic and data structure
c. architectural and structural
d. usability and reliability
14. In the context of requirements analysis, two types of problem
partitioning are
a. bottom-up and top-down
b. horizontal and vertical
c. subordinate and superordinate
d. none of the above
444 http://www.certmagic.com
BH0-003
15. In the context of requirements analysis, partitioning results in the
elaboration of data, function, or behavior.
a. True
b. False
16. Which view should be consider first during software requirements
analysis?
a. actor view
b. data view
c. essential view
d. implementation view
17. Evolutionary prototyping is generally preferred to throw away prototyping
because it
a. allows reuse of the initial prototype.
b. does not require as much customer involvement.
c. is easier to quickly.
d. is more reliable.
18. For software prototyping to be effective technique, tools are required to
develop prototypes rapidly to keep the schedule on track.
a. True
b. False
19. Which of the following is not a principle that should be followed when
creating a specification?
a. create a cognitive model rather than a design model
b. make sure the specification dots every "i" and crosses every "t"
c. recognize that the specification must be augmentable
d. separate functionality from implementation
20. Which of the following is not a guideline for representing requirements?
a. diagrams should be restricted in number and consistent in use
b. representation format and content should be relevant to the content
c. representations should be revisable
d. use no more than 7 plus or minus 2 colors in any diagrams
21. Once the software requirements specification document is approved by
both the customer and developer it becomes an unchangeable baseline
document.
445 http://www.certmagic.com
BH0-003
a. True
b. False
Chapter 12: Analysis Modeling
Chapter 12 Self-Check Quiz
1. Structured analysis is only useful for modeling information systems work,
not real-time engineering problems.
a. True
b. False
446 http://www.certmagic.com
BH0-003
2. Which of the following is not an objective for building an analysis model?
a. define set of software requirements
b. describe customer requirements
c. develop an abbreviated solution for the problem
d. establish basis for software design
3. The data flow diagram
a. depicts relationships between data objects
b. depicts functions that transform the data flow
c. specified major logical decisions as they occur
d. indicates system reactions to external events
4. The entity relationship diagram
a. depicts relationships between data objects
b. depicts functions that transform the data flow
c. indicates how data are transformed by the system
d. indicates system reactions to external events
5. The state transition diagram
a. depicts relationships between data objects
b. depicts functions that transform the data flow
c. indicates how data are transformed by the system
d. indicates system reactions to external events
6. The data model consists of three pieces of interrelated information
a. attributes
b. data objects
c. relationships
d. all of the above
7. The relationships shown in a data model must be classified to show their
a. Width and depth
b. Directionality and reliability
c. cardinality and modality
d. probability and risk
8. The primary purpose of an entity relationship diagram in the data model is
to allow normalization of relationship tables.
a. True
b. False
447 http://www.certmagic.com
BH0-003
9. The data flow diagram must be augmented by descriptive text in order to
describe the functional requirements for a software product.
a. True
b. False
10. It is not possible to use ordinary data flow diagrams to model the
functional requirements of real-time systems.
a. True
b. False
11. The Ward-Mellor extensions to data flow diagrams use
a. dashed lines to show control flow
b. double headed arrows for time-continuous flow
c. boldline to represent real-time operations
d. solid bars as windows into the CSPEC
12. Control flow diagrams use
a. dashed lines to show control flow
b. solid bars as windows into the CSPEC
c. answers a and b
d. single headed arrows for discrete data flow
13. For purposes of behavior modeling a state is any
a. consumer or producer of data.
b. data object hierarchy.
c. observable mode of behavior.
d. well defined process.
14. The states shown in a state transition diagram do not necessarily
correspond to the processes shown in a control flow diagram for the same
system.
a. True
b. False
15. Performing a grammatical parse of the processing narrative is the good
first step to take in producing a(n)
a. data dictionary
b. data flow diagram
c. entity relationship diagram
d. state transition diagram
448 http://www.certmagic.com
BH0-003
16. Control flow diagrams are
a. needed to model event driven systems.
b. required for all systems.
c. used in place of data flow diagrams.
d. useful for modeling user interfaces
17. The process specification used to describe all flow processes that appear
in the final DFD must be written using a program design language.
a. True
b. False
18. The data dictionary contains descriptions of each software
a. configuration item
b. data object
c. diagram
d. notation
19. The process activation table (PAT) provides a process view of the
information contained in a state transition diagram (STD).
a. True
b. False
449 http://www.certmagic.com
BH0-003
Chapter 13: Design Concepts and Principles
Chapter 13 Self-Check Quiz
1. Which of the following is not an area of concern in the design model?
a. architecture
b. data
c. interfaces
d. project scope
2. The importance of software design can be summarized in a single word
a. accuracy
b. complexity
c. efficiency
d. quality
3. Which of these is a characteristic of a good design?
450 http://www.certmagic.com
BH0-003
a. exhibits strong coupling between its modules
b. implements all requirements in the analysis model
c. includes test cases for all components
d. incorporates source code for descriptive purposes
4. Which of the following is not a characteristic common to all design
methods?
a. configuration management
b. functional component notation
c. quality assessment guidelines
d. refinement heuristics
5. A set of design rules should be established before work begins to ensure
design consistency and uniformity.
a. True
b. False
6. What types of abstraction are used in software design?
a. control
b. data
c. procedural
d. all of the above
7. When using structured design methodologies the process of stepwise
refinement is unnecessary.
a. True
b. False
8. Since modularity is an important design goal it is not possible to have too
many modules in a proposed design.
a. True
b. False
9. Which of these model types does not represent a software architecture?
a. data
b. dynamic
c. process
d. structural
451 http://www.certmagic.com
BH0-003
10. The control hierarchy represents the
a. decision order
b. organization of modules
c. repetition of operations
d. sequence of processes
11. Horizontal partitioning defines separate branches for major program
functions, while vertical partitioning distributes control in a top-down
manner.
a. True
b. False
12. Data structure design takes less time than algorithm design, so it may be
saved for last.
a. True
b. False
13. Software procedure focuses on the
a. control hierarchy in a more abstract sense.
b. processing details of each module individually.
c. processing details of each the set of modules collectively.
d. relationship between control and procedure.
14. Information hiding makes program maintenance easier by hiding data
and procedure from unaffected parts of the program.
a. True
b. False
15. To achieve high modularity of software components you need
a. high coupling and high cohesion
b. high coupling and low cohesion
c. low coupling and high cohesion
d. low coupling and low cohesion
16. Cohesion is a qualitative indication of the degree to which a module
a. can be written more compactly.
b. focuses on just one thing.
c. is able to complete its function in a timely manner.
d. is connected to other modules and the outside world.
17. Coupling is a qualitative indication of the degree to which a module
452 http://www.certmagic.com
BH0-003
a. can be written more compactly.
b. focuses on just one thing.
c. is able to complete its function in a timely manner.
d. is connected to other modules and the outside world.
18. Design heuristics are typically only used by students and not needed by
experienced software engineers.
a. True
b. False
19. The reason it is a mistake to do component level design before data
design is that
a. component design is language dependent and data design is not.
b. data design is easier to do.
c. data design is hard to do.
d. the structure of the data usually affects the way in which component-level
design is conducted.
20. The purpose of the requirements cross reference (matrix) in the design
document is to
a. allow managers to monitor the productivity of the design team.
b. establish that all requirements are accounted for by the design.
c. indicate costs associated with each requirement
d. provide implementers with the names of designers for each requirement.
453 http://www.certmagic.com
BH0-003
Chapter 14: Architectural Design
Chapter 14 Self-Check Quiz
1. Which of the following is not part of software architecture?
a. algorithm details
b. databases
c. data design
d. program structure
2. The architectural model provides the software engineer with a view of the
system as a whole.
a. True b. False
3. Which of these characteristics are true of a data warehouse, but not a
typical database?
a. business level orientation and large size
b. currency and correctness of information
c. integration and non volatility
d. all of the above
454 http://www.certmagic.com
BH0-003
4. Data mining and database knowledge discovery are distinct processes.
a. True b. False
5. Data design actually begins during the creation of the analysis model, not
the architectural model.
a. True b. False
6. An architectural style encompasses which of the following elements?
a. constraints
b. set of components
c. semantic models
d. all of the above
7. To determine the architectural style or combination of styles that best fits
the proposed system, requirements engineering is used to uncover
a. algorithmic complexity
b. characteristics and constraints
c. control and data
d. design patterns
8. The criteria used to assess the quality of an architectural design should be
based on system
a. accessibility and reliability
b. data and control
c. functionality
d. implementation details
9. In the architecture trade-off analysis method the architectural style should
be described using the
a. data flow view
b. module view
c. process view
d. all of the above
10. Quantitative methods for assessing the quality of proposed architectural
designs are readily available.
a. True
b. False
11. A useful technique for evaluating the overall complexity of a proposed
architecture is to look at the component
a. number and size of components
455 http://www.certmagic.com
BH0-003
b. flow dependencies and sharing dependencies
c. size and cost
d. none of the above
12. When the overall flow in a segment of a data flow diagram is largely
sequential and follows straight-line paths _________ is present.
a. low coupling
b. good modularity
c. transaction flow
d. transform flow
13. When the information flow in a segment of a data flow diagram is
characterized by a single item that triggers other data flow along one of
many paths _________ is present.
a. high coupling
b. poor modularity
c. transaction flow
d. transform flow
14. When refining the DFD during transform mapping the goal is to strive to
derive bubbles showing high cohesion.
a. True
b. False
15. When you encounter both transform flow and transaction flow in the
same DFD the flow is partitioned and the appropriate mapping technique is
used on each part of the DFD.
a. True
b. False
16. In refining the DFD during transaction mapping it is unnecessary to
create a PSPEC since only the CSPEC is relevant to this type of architectural
style.
a. True
b. False
17. In transaction mapping the first level factoring results in the
a. creation of a CFD
b. derivation of the control hierarchy
c. distribution of worker modules
d. refinement of the module view
456 http://www.certmagic.com
BH0-003
18. A necessary supplement to transform or transaction mapping needed to
create a complete architectural design is
a. entity relationship diagrams
b. the data dictionary
c. processing narratives for each module
d. test cases for each module
Chapter 15: User Interface Design
Chapter 15 Self-Check Quiz
1. Which of the following interface design principles does not allow the user
to remain in control of the interaction with a computer?
a. allow interaction to interruptible
b. allow interaction to be undoable
c. hide technical internals from casual users
d. only provide one rigidly defined method for accomplishing a task
2. Which of the following interface design principles reduce the user's
memory load?
a. define intuitive shortcuts
b. disclose information in a progressive fashion
c. establish meaningful defaults
d. all of the above
3. The reason for reducing the user's memory load is to make his or her
interaction with the computer quicker to complete.
a. True
b. False
4. Interface consistency implies that
a. input mechanisms remain the same throughout the application
b. each application should have its own distinctive look and feel
c. navigational methods are context sensitive
d. answers a and b
5. If past interactive models have created certain user expectations it is not
generally good to make changes to the model.
a. True
b. False
457 http://www.certmagic.com
BH0-003
6. Which model depicts the profile of the end users of a computer system?
a. design model
b. user model
c. user's model
d. system perception
7. Which model depicts the image of a system that an end user creates in his
or her head?
a. design model
b. user model
c. system image
d. system perception
8. Which model depicts the look and feel of the user interface along with all
supporting information?
a. user model
b. user's model
c. system image
d. systems perception
9. Which of these framework activities is not normally associated with the
user interface design processes?
a. cost estimation
b. interface construction
c. interface validation
d. user and task analysis
10. Which approach to user task analysis can be useful in user interface
design?
a. have users indicate their preferences on questionnaires
b. rely on the judgement of experienced programmers
c. study related automated systems
d. observe users performing tasks manually
11. Object-oriented analysis techniques can be used to identify and refine
user task objects and actions.
a. True
b. False
12. The computer's display capabilities are the primary determinant of the
order in which user interface design activities are completed.
a. True
458 http://www.certmagic.com
BH0-003
b. False
13. One means of defining user interface objects and actions is to conduct a
grammatical parse of the user scenario.
a. True
b. False
14. The following common design issues surface for almost every user
interface:
a. adaptive user profiles and functional shortcuts
b. error handling and system response time
c. resolution of graphics displays and design of icons
d. none of the above
15. Add-on help facilities are almost always better received by users than
integrated help facilities.
a. True
b. False
16. User interface development systems typically provide the following
mechanism for building interface prototypes including
a. code generation
b. drawing tools
c. input validation
d. all of the above
17. Usability questionnaires are most meaningful to the interface designers
when completed by
a. customers
b. experienced programmers
c. product users
d. project managers
18. Several usability measures can be collected while observing users
interacting with a computer system including
a. down time for the application
b. number of software defects
c. software reliability
d. time spent looking at help materials
459 http://www.certmagic.com
BH0-003
Chapter 16: Component-Level Design
Chapter 16 Self-Check Quiz
1. Which of the following is not a fundamental structured programming
construct?
a. recursion
b. condition
c. repetition
d. sequence
2. Which of these is a graphical notation for depicting procedural detail?
a. box diagram
b. decision table
c. ER diagram
d. graph matrix
3. In general, box diagrams and flowcharts should
a. be used in place of programming design languages
b. be used to document the entire design or not at all
c. only be used to document or evaluate design in specific instances
d. none of the above
4. A decision table should be used
a. to document all conditional statements
b. to guide the development of the project management plan
c. only when building an expert system
d. when a complex set of conditions and actions appears in a component
5. A program design language (PDL) is often a
a. combination of programming constructs and narrative text
b. legitimate programming language in its own right
c. machine readable software development language
d. useful way to represent software architecure
460 http://www.certmagic.com
BH0-003
6. Since a program design language is not a real programming language the
designer is free to write the procedural design without worrying about syntax
errors.
a. True
b. False
7. Modern software engineers believe that the only design notation useful for
procedural representation is pseudocode.
a. True
b. False
8. Which of these criteria are useful in assessing the effectiveness of a
particular design notation?
a. maintainability
b. modularity
c. simplicity
d. all of the above
Chapter 17: Software Testing Techniques
Chapter 17 Self-Check Quiz
1. With thorough testing it is possible to remove all defects from a program
prior to delivery to the customer.
a. True
b. False
2. Which of these are objectives for software testing?
a. determine the productivity of programmers
b. eliminate the need for future program maintenance
c. eliminate every error prior to release
d. uncover software errors
3. Test cases should be designed long before testing begins.
a. True
b. False
4. Which of the following are characteristics of testable software?
a. observability
461 http://www.certmagic.com
BH0-003
b. simplicity
c. stability
d. all of the above
5. The testing technique that requires devising test cases to demonstrate
that each program function is operational is called
a. black-box testing
b. glass-box testing
c. grey-box testing
d. white-box testing
6. The testing technique that requires devising test cases to exercise the
internal logic of a software module is called
a. behavioral testing
b. black-box testing
c. grey-box testing
d. white-box testing
7. What types of errors are missed by black-box testing and can be
uncovered by white-box testing?
a. behavioral errors
b. subtle logic errors
c. performance errors
d. input error
8. Program flow graphs are identical to program flowcharts.
a. True
b. False
9. The cyclomatic complexity metric provides the designer with information
regarding the number of
a. cycles in the program
b. errors in the program
c. independent logic paths in the program
d. statements in the program
10. The cyclomatic complexity of a program can be computed directly from a
PDL representation of an algorithm without drawing a program flow graph.
a. True
b. False
462 http://www.certmagic.com
BH0-003
11. Condition testing is a control structure testing technique where the
criteria used to design test cases is that they
a. rely on basis path testing
b. exercise the logical conditions in a program module
c. select test paths based on the locations and uses of variables
d. focus on testing the validity of loop constructs
12. Data flow testing is a control structure testing technique where the
criteria used to design test cases is that they
a. rely on basis path testing
b. exercise the logical conditions in a program module
c. select test paths based on the locations and uses of variables
d. focus on testing the validity of loop constructs
13. Loop testing is a control structure testing technique where the criteria is
used to design test cases so that they
a. rely basis path testing
b. exercise the logical conditions in a program module
c. select test paths based on the locations and uses of variables
d. focus on testing the validity of loop constructs
14. Black-box testing attempts to find errors in which of the following
categories
a. incorrect or missing functions
b. interface errors
c. performance errors
d. all of the above
15. Graph-based testing methods can only be used for object-oriented
systems
a. True
b. False
16. Equivalence testing divides the input domain into classes of data from
which test cases can be derived to reduce the total number of test cases that
must be developed.
a. True
b. False
17. Boundary value analysis can only be used during white-box testing.
a. True
463 http://www.certmagic.com
BH0-003
b. False
18. Comparison testing is typically done to test two competing products as
part of customer market analysis prior to product release.
a. True
b. False
19. Orthogonal array testing enables the test designer to maximize the
coverage of the test cases devised for relatively small input domains.
a. True
b. False
20. Real-time applications add a new and potentially difficult element to the
testing mix
a. performance
b. reliability
c. security
d. time
464 http://www.certmagic.com
BH0-003
Chapter 18: Software Testing Strategies
Chapter 18 Self-Check Quiz
1. In software quality assurance work there is no difference between
software verification and software validation.
a. True
b. False
2. The best reason for using Independent software test teams is that
a. software developers do not need to do any testing
b. strangers will test the software mercilessly
c. testers do not get involved with the project until testing begins
d. the conflicts of interest between developers and testers is reduced
3. What is the normal order of activities in which software testing is
organized?
a. unit, integration, system, validation
b. system, integration, unit, validation
c. unit, integration, validation, system
d. none of the above
4. By collecting software metrics and making use of existing software
reliability models it is possible to develop meaningful guidelines for
determining when software testing is done.
a. True
465 http://www.certmagic.com
BH0-003
b. False
5. Which of the following strategic issues needs to be addressed in a
successful software testing process?
a. conduct formal technical reviews prior to testing
b. specify requirements in a quantifiable manner
c. consider using independent test teams
d. all of the above
6. Which of the following need to be assessed during unit testing?
a. algorithmic performance
b. code stability
c. execution and error handling
d. all of the above
7. Units and stubs are not needed for unit testing because the modules are
tested independently of one another.
a. True
b. False
8. Top-down integration testing has as it's major advantage(s) that
a. low level modules never need testing
b. major decision points are tested early
c. no stubs need to be written
d. none of the above
9. Bottom-up integration testing has as it's major advantage(s) that
a. major decision points are tested early
b. no drivers need to be written
c. no stubs need to be written
d. regression testing is not required
10. Regression testing should be a normal part of integration testing because
as a new module is added to the system new
a. control logic and data flow paths are invoked
b. memory size increases
c. drivers require testing
d. a and b
11. Smoke testing might best be described as
466 http://www.certmagic.com
BH0-003
a. bulletproofing shrink-wrapped software
b. rolling integration testing
c. testing that hides implementation errors
d. unit testing for small programs
12. Sandwich testing involves the use of
a. bottom-up testing for subordinate modules
b. schedule compression techniques to reduce testing time
c. testing within tight data boundaries
d. two competitive test teams
13. Which test criteria should be applied in a phase of testing?
a. functional validity
b. interface integrity
c. correctness
d. all of the above
14. The focus of validation testing is to uncover places that users will be able
to observe failure of the software to conform to its requirements.
a. True
b. False
15. Configuration reviews are not needed if regression testing has been
rigorously applied during software integration.
a. True
b. False
16. Acceptance tests are normally conducted by the
a. developer
b. end users
c. test team
d. systems engineers
17. Recovery testing is a system test that forces the software to fail in a
variety of ways and verifies that software is able to continue execution
without interruption.
a. True
b. False
18. Security testing attempts to verify that protection mechanisms built into
a system protect it from improper penetration.
467 http://www.certmagic.com
BH0-003
a. True
b. False
19. Stress testing examines the pressures placed on the user during system
use in extreme environments.
a. True
b. False
20. Performance testing is only important for real-time or embedded
systems.
a. True
b. False
21. Debugging is not testing, but always occurs as a consequence of testing.
a. True
b. False
22. Which of the following is an approach to debugging?
a. backtracking
b. brute force
c. cause elimination
d. all of the above
468 http://www.certmagic.com
BH0-003
Chapter 19: Technical Metrics for Software
Chapter 19 Self-Check Quiz
1. Conformance to implicit requirements and customer expectations has no
place in modern software quality assurance work.
a. True
b. False
2. Failure to conform to explicitly stated requirements and development
standards is cause of most software quality problems.
a. True
b. False
3. Which of the following is not one of three software product aspects
addressed by McCall's software quality factors?
a. ability to undergo change
b. adaptability to new environments
c. operational characteristics
d. production costs and scheduling
4. Which of the following are FURPS quality factors?
a. flexibility
b. complexity
c. reusability
d. usability
5. The ISO 9126 quality standards for computer software are useful because
they lend themselves to direct measurement of software attributes.
a. True
b. False
469 http://www.certmagic.com
BH0-003
6. Most technical software metrics described in this chapter represent indirect
measures software attributes that are useful in the quantitative assessment
of software quality.
a. True
b. False
7. Which of these are reasons for using technical product measures during
software development?
a. large body of scientific evidence supports their use
b. provides software engineers with an objective mechanism for assessing
software quality
c. they allow all quality software and quality information to be expressed
unambiguously as a single number
d. all of the above
8. Which measurement activity is missing from the list below?
Formulation
Collection
Analysis
Interpretation
a. design
b. feedback
c. measurement
d. quantification
9. One of the most important attributes for a technical software metric is that
it should be
a. easy to compute
b. qualitative in nature
c. reliable over time
d. widely applicable
10. The function point metric is an example of project metric that can be
used to assist with technical decision-making based on the analysis model
information, without making use of historical project data.
a. True
b. False
11. The computation of DeMarco's bang metric requires the developer to
distinguish between function-strong and data-strong applications.
a. True
470 http://www.certmagic.com
BH0-003
b. False
12. Which two characteristics of the software requirementsare the
specification metrics proposed by the Davis address?
a. functionality and performance
b. performance and completeness
c. specificity and completeness
d. specificity and functionality
13. Architectural design metrics focus on
a. architectural structure
b. data structural relationships
c. internal module complexity
d. module complexity
14. Component-level metrics include measures of
a. complexity
b. coupling
c. module cohesion
d. all of the above
15. Interface metrics are used to assess the complexity of the module's input
and output relationships with external devices.
a. True
b. False
16. Halstead's source code metrics are based on the number of
a. modules in the program
b. operators and operands in the program
c. number of Boolean conditions in the program
d. volume elements in the program
17. Technical testing metrics fall into which broad category(s)
a. metrics that focus on defect removal effectiveness
b. metrics that focus on test coverage
c. metrics that predict the number of test cases required
d. answers b and c
18. The IEEE software maturity index is used to provide a measure of the
a. maintainability of a software product based on its availability
b. relative age of a software product being considered for retirement
471 http://www.certmagic.com
BH0-003
c. reliability of a software product following regression testing
d. stability of a software product as it is modified during maintenance
472 http://www.certmagic.com
BH0-003
Chapter 20: Object-Oriented Concepts and Principles
Chapter 20 Self-Check Quiz
1. The object-oriented view demands a revolutionary approach to software
engineering.
a. True
b. False
2. Encapsulation of attributes and operations within an object
a. allows for easy reuse of this information.
b. increases the cost of program maintenance.
c. is a poor programming practice.
d. none of the above
3. A generalized description of a collection of similar objects is a
a. class
b. instance
c. subclass
d. super class
4. The values are assigned to an object's attributes make that object unique.
a. True
b. False
5. An object's operations are activated by ordinary function calls.
a. True
b. False
6. Operations are object procedures that are invoked when an object
receives a message.
a. True
b. False
473 http://www.certmagic.com
BH0-003
7. Which of these is not one of the primary benefits of object-oriented
architectures?
a. easy component reuse
b. improved execution performance
c. information hiding
d. simplified interfaces
8. Inheritance provides a mechanism by which changes to lower level classes
can be propagated to all super classes quickly.
a. True
b. False
9. Polymorphism reduces the effort required to extend an object system by
a. coupling objects together more tightly.
b. enabling a number of different operations to share the same name.
c. making objects more dependent on one another.
d. removing the barriers imposed by encapsulation.
10. Which of the following should be considered as candidate objects in a
problem space?
a. events
b. people
c. structures
d. all of the above
11. Attributes are chosen for an object by examining the data dictionary and
identifying the entities that appear to be related.
a. True
b. False
12. Which of the following is not one of the broad categories used to classify
operations?
a. computation
b. data manipulation
c. event monitors
d. transformers
13. Consideration of an object's life history and messages passed among
system objects may suggest additional operations that need to be added to
an object definition.
a. True
474 http://www.certmagic.com
BH0-003
b. False
14. Object-oriented projects require less management planning and
oversight effort than conventional software projects.
a. True
b. False
15. The common process framework suggested for object-oriented software
development consists of a(n)
a. analysis part and a design part.
b. recursive part and an iterative part
c. recursive part and a parallel part
d. quality process and a software reuse part
16. Because an overriding goal for object-oriented projects should be reuse,
LOC estimates are good project metrics.
a. True
b. False
17. Two metrics that may be useful in scheduling object-oriented projects are
a. number of key classes and number of support classes
b. number of major iterations and number of completed contracts
c. number of scenario scripts and number of subsystems
d. all of the above
18. Which of the following may be considered a major milestone during an
object-oriented software development project?
a. object-oriented analysis completed
b. object-oriented design completed
c. object-oriented programming completed
d. all of the above
Chapter 21: Object-Oriented Analysis
Chapter 21 Self-Check Quiz
1. Unlike structured analysis, top-down decomposition and consideration of
end-to-end processing sequences are not present when OOA is used.
a. True
475 http://www.certmagic.com
BH0-003
b. False
2. The first step in any OOA process model are to
a. build an object-relationship model
b. define collaborations between objects
c. elicit customer requirements
d. select a representation language
3. UML (unified modeling language) analysis modeling focuses on the
a. behavioral model and environment model
b. behavioral model and implementation model
c. user model and environmental model
d. user model and structural model
4. Assume team A has access to a robust class library and team B does not.
If both teams were building the same software product you would expect
a. team A to complete the product faster and at a lower cost
b. team B to complete the product faster and at a lower cost
c. team A to spend more team testing and debugging their code
d. team B to produce a better product programming from scratch
5. Object-oriented domain analysis is concerned with the identification and
specification of reusable capabilities within an application domain.
a. True
b. False
6. Static components of an OOA model are
a. focus on control
b. not reusable
c. sensitive to timing and event processing
d. structural in nature
7. Dynamic components of an OOA model are
a. not reusable
b. sensitive to timing and event processing
c. stable throughout the operational life of an application
d. structural in nature
8. Which of these is not an objective for use-case creation?
a. define the functional and operational system requirements
b. define the object hierarchy for the system
476 http://www.certmagic.com
BH0-003
c. provide a basis for validation testing
d. provide a description of end-user and system interaction
9. Which of the following items does not appear on a CRC card?
a. class collaborators
b. class name
c. class reliability
d. class responsibilities
10. Class responsibilities are defined by
a. its attributes only
b. its collaborators
c. its operations only
d. both its attributes and operations
11. Which of these is not a generic relationship that helps an analyst define
potential class collaborators?
a. comes-before
b. depends-upon
c. has-knowledge-of
d. is-part-of
12. To review a complete CRC model the reviewers only need to walk
through one or two representative use-cases.
a. True
b. False
13. Once the classes and objects have been identified using the CRC model
the analyst should focus on the structure and hierarchy of the class model.
a. True
b. False
14. When a group of classes collaborate among themselves to accomplish a
cohesive set of responsibilities they are referred to as a(n)
a. collaboratory
b. object hierarchy
c. package
d. subclass unit
15. The CRC model defines the relationships between the objects, but unlike
the object-relationship model it does not specify the
a. cardinality of the relationships
477 http://www.certmagic.com
BH0-003
b. collaboration among the classes
c. direction of the relationships
d. number of relationships
16. The object-behavior model indicates how the system
a. functions in the operating environment
b. objects collaborate with one another
c. responds to external stimuli
d. responds to internal stimuli
17. Events occur whenever a(n)
a. actor and the OO system exchange information
b. class operation is invoked
c. messages are passed between objects
d. all of the above
18. The passive state of an object is
a. only observable from outside the system
b. simply the current status of all its attributes
c. the resting behavior of the class
d. none of the above
19. An state transition can only occur when triggered by a(n)
a. actor
b. collaboration attempt
c. event
d. none of the above
478 http://www.certmagic.com
BH0-003
Chapter 22: Object-Oriented Design
Chapter 22 Self-Check Quiz
1. The four layers defined for object-oriented design are the same as design
layers used for conventional software design.
a. True
b. False
2. Which of the following criteria appears in object-oriented design models,
but not conventional design models?
a. representation of module hierarchy
b. specification of data definitions
c. specification of message connections
d. specification of procedural logic
479 http://www.certmagic.com
BH0-003
3. Object-oriented design achieves low module coupling and provides better
information hiding than other approaches.
a. True
b. False
4. The same generic steps are applied in object-oriented design, regardless
of the particular design method that is chosen.
a. True
b. False
5. The UML (unified modeling language) approach to object-oriented design
has two major activities.
a. architectural design and object design
b. interface design and message design
c. message design and system design
d. system design and object design
6. Which of the following activities is part of the system design activity of the
UML approach to OOD?
a. choose a strategy for data management
b. partition analysis model into subsystems
c. user interface design
d. all of the above
7. The first step of system design in OOD is partitioning the analysis model
into cohesive collections of classes, relationships, and behaviors called
a. class hierarchies
b. client/server links
c. subsystems
d. system layers
8. When subsystems are concurrent they must be allocated to separate
processors.
a. True
b. False
9. User interfaces are frequently built from toolkits containing reusable
classes so that the implementer only needs to instantiate objects appropriate
to problem domain.
a. True
b. False
480 http://www.certmagic.com
BH0-003
10. Which of these areas is considered part of the data management
component of OOD system design?
a. creation of an infrastructure for object storage and retrieval
b. management of data critical to the application
c. normalization of the class data attributes
d. both a and b
11. A guardian object
a. controls access to a resource and moderates conflicting requests for it
b. is a multimedia database object
c. provides security to a networked or web-based system
d. none of the above
12. Every contract between subsystems is manifested by exactly one
message that moves between an object in each subsystem.
a. True
b. False
13. The design description of an object can take one of two forms
a. object template or pseudocode
b. operator sequences or attribute graphs
c. protocol description or object description
d. subsystem collaboration graph or protocol graph
14. In OOD operations are refined by
a. isolating new operations at lower abstraction levels
b. performing a grammatical parse
c. writing a processing narrative
d. all of the above
15. Design patterns are not applicable to the design of object-oriented
sofware?
a. True
b. False
16. Two design patterns that can be used in object-oriented systems are
a. inheritance and composition
b. inheritance and reuse
c. polymorphism and composition
d. polymorphism and reuse
481 http://www.certmagic.com
BH0-003
17. Object-oriented designs do not need to be implemented using object-
oriented programming techniques.
a. True
b. False
Chapter 23: Object-Oriented Testing
Chapter 23 Self-Check Quiz
1. Which of these should every object-oriented model be tested for?
a. completeness
b. consistency
c. correctness
d. all of the above
2. Because if the evolutionary nature of OO software development, there is
no cost savings attributed to early detection of errors.
a. True
b. False
3. The correctness of the OOA and OOD models is judged based on the
a. model's conformance to real world problem domain
b. review of the connections between classes
c. review of the modeling conventions used
d. both a and c
4. To facilitate assessing the consistency of the OOA and OOD models, each
class and its connections to other classes should be examined by reviewing
the
a. CRC model
b. object-relationship model
c. state transition function
d. both a and b
482 http://www.certmagic.com
BH0-003
5. Class testing for OO software is equivalent to unit testing in conventional
software testing.
a. True
b. False
6. The OO testing integration strategy involves testing
a. groups of classes that collaborate or communicate in some way
b. single operations as they are added to the evolving class implementation
c. operator programs derived from use-case scenarios
d. none of the above
7. To develop validation tests for OO software the tester should
a. focus on user visible actions
b. derive test cases from the object-behavior model
c. look at use-cases from the analysis model
d. all of the above
8. Test case design for OO software is driven by the algorithmic detail of the
individual operations.
a. True b. False
9. Encapsulation of attributes and operations inside objects makes it easy to
obtain object state information during testing.
a. True b. False
10. Use-cases can provide useful input into the design of black-box and
state-based tests of OO software.
a. True
b. False
11. Fault-based testing is best reserved for
a. conventional software testing
b. operations and classes that are critical or suspect
c. use-case validation
d. white-box testing of operator algorithms
12. Testing OO class operations is made more difficult by
a. encapsulation
b. inheritance
c. polymorphism
d. both b and c
483 http://www.certmagic.com
BH0-003
13. It is not necessary to test operators inherited by other objects.
a. True
b. False
14. Scenario-based testing
a. concentrates on actor and software interaction
b. misses errors in specifications
c. misses errors in subsystem interactions
d. both a and b
15. Deep structure testing is not designed to
a. object behaviors
b. communication mechanisms
c. exercise object dependencies
d. exercise structure observable by the user
16. Random order tests are conducted to exercise different class instance life
histories.
a. True
b. False
17. Which of these techniques is not useful for partition testing at the class
level
a. attribute-based partitioning
b. category-based partitioning
c. equivalence class partitioning
d. state-based partitioning
18. Multiple class testing is too complex to be tested using random test
cases.
a. True
b. False
19. Test derived from behavioral class models should be based on the
a. data flow diagram
b. object-relation diagram
c. state transition diagram
d. use-case diagram
Chapter 24: Technical Metrics for Object-Oriented Systems
484 http://www.certmagic.com
BH0-003
Chapter 24 Self-Check Quiz
1. The primary objectives for using object-oriented metrics are no different
than those for metrics derived from conventional software.
a. True
b. False
2. For OO systems the localization mechanism is based on the
a. data structures
b. internal structure of functions
c. module coupling
d. objects
3. For OO systems encapsulation encompasses
a. attributes
b. class states
c. operations
d. all of the above
4. Metrics that provide an indication that information hiding has been avoided
should provide an indication of the high quality for an OO system design.
a. True
b. False
5. Examples of inheritance metrics for OO systems would be
a. number of children
b. nesting level in the hierarchy
c. both a and b
d. none of the above
6. OO metrics represent abstractions in measures of a(n)
a. attribute
b. class
c. object
d. operation
7. Experienced OO designers derive no benefit from having access to design
metrics.
a. True
485 http://www.certmagic.com
BH0-003
b. False
8. Which of the following is not a measurable characteristic of an object-
oriented design?
a. completeness
b. efficiency
c. size
d. volatility
9. The depth of inheritance tree (DIT) metric can give an OO software
designer a reading on the
a. attributes required for each class
b. completion time required for system implementation
c. complexity of the class hierarchy
d. level of object reusability achieved
10. For OO software it is important to keep class coupling low and operation
cohesion high.
a. True
b. False
11. If you encounter a class with a large responsibility (large class size or CS
value) you should consider
a. making it a base class
b. making it a subclass
c. partitioning the class
d. starting a new class hierarchy
12. Because the class is the dominant unit in OO systems there is no call for
the definition of class-oriented metrics.
a. True
b. False
13. OO design metrics provide an indication of design quality and also
provide a general indication of the amount of testing effort required for the
OO system.
a. True
b. False
14. OO testing metrics can help you in targeting suspect
a. clusters
486 http://www.certmagic.com
BH0-003
b. scenarios
c. threads
d. all of the above
15. OO project metrics may be combined with historical project data to
compute
a. design metrics
b. process metrics
c. productivity metrics
d. testing metrics
16. Which of the following metrics can provide the software planner with
insight into software size?
a. number of key classes
b. number of scenario scripts
c. number of subsystems
d. all of the above
Chapter 25: Formal Methods
Chapter 25 Self-Check Quiz
1. Which of the following is not one of the desired properties of a formal
specification?
a. completeness
b. consistency
c. unambiguous
d. all of the above
2. Which of the following is a deficiency of natural language specification of
software products?
487 http://www.certmagic.com
BH0-003
a. contradictions
b. vagueness
c. mixed abstraction levels
d. all of the above
3. Effective use of formal methods will eliminate all defects that would
normally appear during design, coding, and testing.
a. True
b. False
4. It is not realistic to expect that a complex software system could be
specified using a single mathematical expression.
a. True
b. False
5. A data invariant is a set of conditions that are true during the execution of
any function.
a. True
b. False
6. In formal methods work, stored data that the system accesses and alters
is called a(n)
a. attribute
b. data structure
c. state
d. variant
7. In formal methods work, an action that reads or writes data to a state is
called a(n)
a. actor
b. event
c. invariant
d. operation
8. What defines the circumstances in which a particular operation is valid?
a. data invariant
b. precondition
c. postcondition
d. state
9. Constructive set specification is preferable to enumeration because it
488 http://www.certmagic.com
BH0-003
a. allows succinct definition of large sets
b. only works for software products
c. uses mathematical notation
d. all of the above
10. Knowledge of _______ is indispensable if a software engineer intends to
make use of formal methods.
a. calculus
b. differential equations
c. set operations
d. statistical methods
11. Universal quantification is a way of making a statement about
a. all the elements of a set
b. particular elements of a set
c. quality of an operations input set
d. use of metrics in the design process
12. Which of the following is not an operator that may be applied to
sequences?
a. head
b. front
c. rear
d. tail
13. A common notational convention in many formal methods is to write the
variable with a prime in the postcondition for an operator.
a. True
b. False
14. Which of these are components of a formal specification language?
a. semantics that defines the objects used to describe system
b. set of relations defining the object rules
c. syntax that defining the notation
d. all of the above
15. Using formal methods eliminates the need to write natural language
commentary in the specification document.
a. True
b. False
489 http://www.certmagic.com
BH0-003
16. Using formal methods absolves a software engineer from having to test
any of the library components used in a software design.
a. True
b. False
17. Formal specification techniques are not widely used in industry yet.
a. True
b. False
Chapter 26: Cleanroom Software Engineering
Chapter 26 Self-Check Quiz
1. The cleanroom strategy is based on the ________software process model.
a. evolutionary
b. incremental
c. revolutionary
d. spiral
2. The cleanroom strategy relies on
a. exhaustive testing
b. extensive unit testing of all modules
c. tests that exercise the software as it is really used
d. white box testing strategies
3. Use of formal program correctness proofs as part of the cleanroom process
eliminates the need do any testing for software defects.
a. True
b. False
4. Which of the following characteristics distinguish cleanroom software
engineering from conventional software engineering?
a. explicit use of statistical quality control
b. relies heavily on statistical use testing
c. use of formal proof methods for design verification
d. all of the above
490 http://www.certmagic.com
BH0-003
5. In cleanroom software engineering a box encapsulates some system
aspect at a particular level of detail.
a. True
b. False
6. This box specification describes an abstraction, stimuli, and response.
a. black box
b. clear box
c. state box
d. white box
7. This box specification describes the architectural design for some system
component.
a. black box
b. clear box
c. state box
d. white box
8. This box specification is closely aligned with procedural design and
structured programming.
a. black box
b. clear box
c. state box
d. white box
9. In cleanroom software engineering the structured programming approach
is used to
a. refine data design
b. refine function design
c. refine usage test cases
d. both a and b
10. To prove a design correct you must identify all conditions and then prove
a random statistical sample of these conditions are correct.
a. True
b. False
11. By using only structured programming constructs as you create a
procedural design, you make the work of proving design correctness much
easier.
a. True
491 http://www.certmagic.com
BH0-003
b. False
12. Which of the following is not an advantage of using rigorous correctness
verification of each refinement of the clear box design.
a. improves performance of code
b. produces better code than unit testing
c. reduces verification effort
d. results in near zero defect levels
13. Statistical use testing relies on probability distributions based on
a. mixture of control structures used in the program
b. order in which the module execute
c. the way software will actually be used
d. user interface design standards
14. Certification of an increment is complete once it has passed the formal
verification process.
a. True
b. False
15. Which of the following models is part of the clean room certification
process?
a. component model
b. sampling model
c. both a and b
d. none of the above
492 http://www.certmagic.com
BH0-003
Chapter 27: Component-Based Software Engineering
Chapter 27 Self-Check Quiz
1. In component-based software engineering, the development team
examines the requirements to see which are amenable to composition, rather
than construction, before beginning detailed design tasks.
a. True
b. False
2. Which of the following is not one of the CBSE activities that take place for
requirements that can be addressed with commercial off-the-shelf (COTS)
components?
a. component adaptation
b. component composition
c. component design
d. component qualification
3. What are the two parallel engineering activities found in the CBSE process
model?
a. component-based development and library development
b. domain engineering and component-based development
c. domain engineering and process development
d. none of the above
4. Which of the following is not one of the major activities of domain
engineering?
a. analysis
b. construction
c. dissemination
d. validation
5. Domain analysis is only applicable to CBSE or object-oriented software
engineering.
a. True
b. False
6. The purpose of a domain characterization function is to determine
a. a basis for estimating development costs
b. the approximate size of the application domain information
493 http://www.certmagic.com
BH0-003
c. whether an existing function can be reused in a particular application
d. all of the above
7. A structure point is a(n)
a. distinct construct within a structural model
b. element within the reuse library
c. similar to a feature point in structured design
d. all of the above
8. Which of the following is an example of structure point for some software
domain?
a. bounds setting mechanism
b. control mechanism
c. user interface
d. all of the above
9. Which of the following factors would not be considered during component
qualification?
a. application programming interface (API)
b. development and integration tools required
c. exception handling
d. testing equipment required
10. Which of the following is not a technique used for component wrapping?
a. black-box wrapping
b. clear-box wrapping
c. gray-box wrapping
d. white-box wrapping
11. Which of the following should be part of an infrastructure for effective
component integration?
a. automation
b. data exchange model
c. underlying object model
d. all of the above
12. Which of the following is not one of the issues that form a basis for
design for reuse?
a. object-oriented programming
b. program templates
c. standard data
d. standard interface protocols
494 http://www.certmagic.com
BH0-003
13. Which of the following is not one of the classification schemes used for
software components?
a. attribute-value classification
b. domain classification
c. enumerated classification
d. faceted classification
14. In a reuse environment, library queries are often characterized using the
________ element of the 3C Model.
a. concept
b. content
c. context
d. all of the above
15. Which of the following is not improved by the effective use of CBSE?
a. cost
b. performance
c. productivity
d. quality
16. The effort required to qualify, adapt, and integrate structure points into
new systems must be based on historical data collected for qualification,
adaptation, and integration of these reusable components in other
applications.
a. True
b. False
17. It is impossible to develop effective metrics for software reuse.
a. True
b. False
Chapter 28: Client/Server Software Engineering
Chapter 28 Self-Check Quiz
1. Which of the following is an example of client/server system?
a. database servers
b. file servers
c. transaction servers
d. all of the above
495 http://www.certmagic.com
BH0-003
2. Which of the following is not a subsystem typically found in a client/server
system?
a. application subsystem
b. database subsystem
c. message passing subsystem
d. user interaction subsystem
3. Which of the following is an example of a fat server design?
a. distributed presentation
b. local logic
c. remote presentation
d. none of the above
4. In most client/server systems the presentation system is placed on the
client and any shared databases are located on the server.
a. True
b. False
5. Remote procedure calls permit server operations to execute on local client
machines.
a. True
b. False
6. An ORB (object request broker) is middleware that enables an object
residing
a. on a client to send a message to a method encapsulated by a server
object
b. on a database to send a message to a method encapsulated by a server
object
c. on a LAN to send a message to a method encapsulated in an Internet
object
d. on a server to send a message to a method encapsulated by a client
object
7. Two process models that are especially well-suited for C/S software
engineering are
a. object-oriented and component-based software engineering
b. rapid prototyping and database design
c. revolutionary design and component-based software engineering
d. structured design and event-based software engineering
496 http://www.certmagic.com
BH0-003
8. Requirements modeling activities for C/S systems are quite similar to the
analysis modeling methods used for more conventional architectures.
a. True
b. False
9. The design approach used for C/S systems rarely requires modification to
accommodate the hardware architecture.
a. True b. False
10. To accommodate the differences between COTS components supplied by
several vendors and in-house components, the ORB architecture must be
designed to
a. achieve interoperability among components
b. eliminate incompatible components from the system
c. function only with in-house components
d. replace the functionality of incompatible components
11. In the C/S context an elementary business process can be defined as a
set of tasks performed fully by one user at a client site.
a. True
b. False
12. In the design repository a business object is defined as
a. client information
b. information visible to system developers
c. information visible to system users
d. server information
13. Which of the following techniques may be used for data distribution and
management in C/S systems?
a. fragmentation
b. replication
c. snapshot
d. all of the above
14. Which symbol would not be found in the structure chart for an
elementary business process?
a. application object
b. cardinality link
c. control couple
d. database object
15. Which entities do not reside in the design repository?
497 http://www.certmagic.com
BH0-003
a. business rule/component links
b. components
c. methods
d. none of the above
16. With the exception of integration testing, C/S systems pose no new
testing concerns for software engineers.
a. True
b. False
17. Which of the following testing approaches is commonly used to test C/S
systems?
a. database tests
b. transaction testing
c. network communication testing
d. all of the above
18. The tactics used in object-oriented testing are useless for testing a C/S
system that has been implemented using an imperative programming
language.
a. True
b. False
Chapter 29: Web Engineering
Chapter 29 Self-Check Quiz
1. Which of the following is not a characteristic of a WebApp?
a. content driven
b. continuously evolving
c. easily measurable
d. network intensive
2. WebApps must be developed and deployed quickly, making the application
of software engineering processes exceptionally difficult.
a. True
b. False
3. Which of the following characteristics is least important when assessing
the quality of a WebApp?
498 http://www.certmagic.com
BH0-003
a. usability
b. reliability
c. maintainability
d. aesthetics
4. Which of the following technologies is important to web engineers?
a. component-based development
b. internet standards
c. security
d. all of the above
5. Which process model best describes WebE?
a. customer driven design
b. evolutionary design
c. structured design
d. all of the above
6. The engineering activity in the WebE process incorporates two parallel
tasks
a. content design and production
b. content design and programming
c. page generation and evaluation
d. none of the above
7. During the formulation step of the WebE process two types of goals need
to be defined
a. applicative goals and aesthetic goals
b. applicative goals and informational goals
c. information goals and performance goals
d. aesthetic goals and performance goals
8. Which type of analysis is not conducted during the WebE process?
a. content analysis
b. functional analysis
c. interaction analysis
d. market analysis
9. Which technical elements should a web engineer try to reuse during web-
based design?
a. design methods
b. design patterns
c. templates
d. all of the above
499 http://www.certmagic.com
BH0-003
10. Which of the following is not one of the architectural structures used by
web engineers in architectural design?
a. linear
b. grid
c. hierarchical
d. parallel
11. Which of the following is a design pattern used during web-based design?
a. cycle
b. counterpoint
c. sieve
d. all of the above
12. Web navigational design involves creating a semantic navigational unit
for each goal associated with each defined user role.
a. True b. False
13. Interface design for WebApps is identical to interface design for any other
piece of interactive software.
a. True b. False
14. Because WebApps are constantly evolving, testing is an on-going activity
conducted by support staff using regression testing techniques.
a. True
b. False
15. Unit testing and integration testing are not needed when testing
WebApps.
a. True b. False
16. Which of these roles is not usually assigned to members of the WebE
team?
a. content developer
b. marketing specialist
c. web master
d. web publisher
500 http://www.certmagic.com
BH0-003
17. Although outsourcing WebApp development is common practice, it is
important to perform thorough analysis of the application and even create a
rough design internally before selecting a vendor.
a. True b. False
18. Which of these issues require special consideration when considering
tactics for WebApp configuration management?
a. content
b. politics
c. scalability
d. all of the above
Chapter 30: Reengineering
Chapter 30 Self-Check Quiz
1. Which of the following is not an example of a business process?
a. designing a new product
b. hiring an employee
c. purchasing services
d. testing software
2. Which of the following is not a principle that should guide business process
reengineering?
a. capture data at each source
b. fully redocument legacy processes
c. organize around outcomes
d. put decision point where work is performed
3. Business process reengineering has no start or end, it is an evolutionary
process.
a. True
b. False
4. Business process reengineering is just another silver bullet fad with no real
benefits to anyone.
a. True
b. False
5. How much of software maintenance work involves fixing errors?
a. 20 percent
b. 40 percent
501 http://www.certmagic.com
BH0-003
c. 60 percent
d. 80 percent
6. Which of the following activities is not part of the software reengineering
process model?
a. forward engineering
b. inventory analysis
c. prototyping
d. reverse engineering
7. The software reengineering process model includes restructuring activities
for which of the following work items?
a. code
b. documentation
c. data
d. all of the above
8. Which of the following is not an issue to consider when reverse
engineering?
a. abstraction level
b. completeness
c. connectivity
d. directionality
9. The first reverse engineering activity involves seeking to understand
a. data
b. processing
c. user interfaces
d. none of the above
10. Reverse engineering of data focuses on
a. database structures
b. internal data structures
c. both a and b
d. none of the above
11. Reverse engineering should precede the reengineering of any user
interface.
a. True b. False
502 http://www.certmagic.com
BH0-003
12. Which of these benefits can be achieved when software is restructured?
a. higher quality programs
b. reduced maintenance effort
c. software easier to test
d. all of the above
13. Code restructuring is a good example of software reengineering.
a. True b. False
14. Which of these is not an example of data redesign?
a. data analysis
b. data name rationalization
c. data record standardization
d. none of the above
15. Forward engineering is not necessary if an existing software product is
producing the correct output.
a. True b. False
16. Reengineering client/server systems begins with a thorough analysis of
the business environment that encompasses the existing computing system.
a. True b. False
17. The only time reengineering enters into work with a legacy system is
when its components will be implemented as objects.
a. True b. False
18. Which of these activities would not be part of a process model for
reengineering a user interface?
a. correct ergonomic failings of the interface
b. measure the interface performance in the marketplace
c. remodel the interface behavior
d. understanding the original interface
19. The cost benefits derived from reengineering are realized largely due to
decreased maintenance and support costs for the new software product.
a. True b. False
Chapter 31: Computer-Aided Software Engineering
503 http://www.certmagic.com
BH0-003
Chapter 31 Self-Check Quiz
1. The primary purpose of computer-aided software engineering tools (CASE)
is to allow direct development of applications by end-users.
a. True
b. False
2. Which of the following characteristics are essential to having an effective
CASE environment?
a. collection of useful tools
b. organized tool layout
c. skilled craftsperson
d. all of the above
3. The most valuable CASE tools are those that
a. automatically debug source code
b. build analysis and design diagrams
c. contribute information to the development process
d. all of the above
4. The primary purpose of the CASE integration framework is to
a. allow communication among CASE tools
b. allow communication among software developers and customers
c. eliminate the need for integration testing
d. provide facilities for component-based design
5. CASE tools are commonly classified by their
a. environment architecture
b. function
c. use in the software development process
d. all of the above
6. Most CASE tools are limited to supporting specific programming languages
or specific technical/management methods and require some degree of
interaction with the software engineer.
a. True
b. False
7. Which of the following is not considered one of the benefits derived from
integrated CASE?
a. increased project control
504 http://www.certmagic.com
BH0-003
b. reduction in effort to perform umbrella activities
c. smooth information transfer between tools
d. testing of work products is eliminated
8. CASE tool integration demands the use of
a. component-based design
b. database technology
c. object-oriented software engineering
d. structured design
9. Which of the following is not one of the layers in the architectural model
for the CASE integration framework?
a. object management
b. portability
c. shared repository
d. user interface
10. The primary purpose of the object management layer in the CASE
integration framework is to
a. allow the use of component-base software engineering
b. perform configuration management
c. support object-oriented design
d. none of the above
11. The integrated CASE repository is a database that acts as the center for
accumulation and storage of software engineering information.
a. True
b. False
12. Which of the following would not be one of the functions performed by
the integrated CASE repository?
a. data integrity
b. document standardization
c. information sharing
d. project scheduling
13. Which database management technology is used to support today's CASE
repositories?
a. hierarchical
b. object-oriented
c. relational
d. both b and c
505 http://www.certmagic.com
BH0-003
14. Which of these integrated CASE repository features is not commonly
found in commercial database management systems?
a. integrity enforcement
b. process/project management capability
c. storage of sophisticated data structures
d. all of the above
15. One of the most important features of the CASE repository is its ability to
a. print document automatically
b. test software products as they are accessed
c. track daily code production by individual programmers
d. track relationships among configuration objects
Chapter 32: The Road Ahead
Chapter 32 Self-Check Quiz
1. Software is important commercially because it can function as a(n)
a. Automation
b. Differentiator
c. Information generator
d. Both b and c
2. The changes that will affect software engineering are not likely to be
influenced by
a. new programming languages
b. people doing the work
c. processes selected
d. underlying computing technology
506 http://www.certmagic.com
BH0-003
3. As software projects grow in complexity, adding people can actually
reduce productivity of team members.
a. True b. False
4. The World Wide Web is not likely to change the way software engineers
acquire information on applications domains.
a. True b. False
5. Evolutionary software process models are likely to dominate the software
development process as time lines become shorter because they
a. can deliver partial solutions when complete solutions cannot be delivered
within the available time
b. do not require developers to understand the customer's needs fully even
when tackling complex projects
c. require less time devoted to testing than linear software process models
d. all of the above
6. Involving customers early in the design process is likely to increase end-
user satisfaction and improve overall product quality.
a. True b. False
7. There is no difference between data processing and information
processing.
a. True b. False
8. Information collected on a variety of topics can be connected to form a
body of facts called
a. data
b. knowledge
c. wisdom
d. none of the above
9. It is likely that hardware will continue to serve as the technology driver in
computing for the next 30 years.
a. True b. False
1)What types of errors are missed by black box testing and can
be
uncovered by white box testing?
a.behaviroal testing
507 http://www.certmagic.com
BH0-003
b.black box testing
c.grey box testing
d.white box testing
-Answers are not RIGHT. correct answer is missing.
2)Program flow graphs are identical to prgram
flowcharts...true/false ?
3)Data flow testing is a control structure testing technique
where the
criteria used to design test cases is that they-
a.rely on basis path testing
b.exercise the logical conditions in a program module
c.select test paths based on the locations and uses of
variables
d.focus on testing the validity of loop constructs.
4)Condition testing is a control structure testing technique
where the
criteria used to design test cases is that they-
a.rely on basis path testing
b.exercise the logical conditions in a program module
c.select test paths based on the locations and uses of
variables
d.focus on testing the validity of loop constructs.
5)Loop testing is a control structure testing technique where
the
criteria is used to design test cases so that they-
a.rely on basis path testing
b.exercise the logical conditions in a program module
c.select test paths based on the locations and uses of
variables
d.focus on testing the validity of loop constructs.
6)Graph based testing methods can only be used for
object-oriented
systems...true/false ?
refer roger pressman book for questions 2-6. refer topics
related to testing.
7)Which of the following need to be assessed during unit
testing?
a.algorithmic performance
b.code stability
c.execution and error handling
d.all of the above
answer: d
8)Units and Stubs are not needed for unit testing bcase the
modules are
tested independently of one another..true/false ?
answer: false
9)Configuration reviews are not needed for if regression testing
508 http://www.certmagic.com
BH0-003
has
been rigorously applied during software
integration....true/false..?
answer: false
10)which question no longer concerns the modern s/w engineer ?
a.why does computer hardware cost so much ?
b.why does software take a long time to finish ?
c.why does it cost so much to develop a piece of software ?
d.why can’t software errors be removed from products prior to
delivery
?
???? :)
11)Effectiveness is doing things right and efficiency is doing
the
right things..true/false and justify your answer.
Refer domain 1 in cbok(last few pages in domain 1). you will
find answer there
Questions And Answers –
1] What are the risk mitigation factors in e-Commerce Application?
2] What is Phase Containment.[ Requires details and w.r.t Defect
Prevention. I know the basic analogy]
1)Define and explain ant three aspects of code review
Ans : In my opinion Review for code is nothing different than review
of any other artifact. So following aspects are important
1) Participant should come with preparation and Review meeting
must not long more that two hours
2) Review the product not the producer
3) Concentrate on finding the defects not on solution.
2)prepare a check list for the developers on unit
testing before the application comes to testing
department
General
1. All Programs are Compiled successfully
2. All Programs are commented properly.
3. All Programs do not have spelling mistakes and grammar
errors in comments.
4. All program names in Build are same as in Release Notes.
5. Online Help exists for each program unit.(for each page if
applicable)
509 http://www.certmagic.com
BH0-003
6. User Manual does contain all program names and
associated text.
GUI Interface
• Background and Foreground color combination suits the
eyes
• Program works in all screen resolution modes
• Program does not contain orphan/broken web links
• Application does not raise any critical error message ( It
should be returned to developer if continuously severe
errors encountered)
3)List what you think are the primary goals of testing
Ans : In my opinion first primary goal of testing is to find defects.
Dynamic Testing is the last opportunity to fix the problems before
they
become exponentially expensive as they enter in production after
that.
Other primary goal – User Satisfaction and Cost Reduction.
4)Write any attributes which will impact the testing
process
• Maintainability
• Reliability
• Correctness
• Flexibility
5)What are the product standards for test plan ,Test
script and Test Report
Test Plan
1. Preface
2. Executive Summary
3. Introduction of the Project
4. Organization of Document
5. Test Organization
6. Test Techniques
7. Test Tools
8. Acceptance Criteria
9. Glossary
510 http://www.certmagic.com
BH0-003
Test Script
• Date of Test Conducted
• Type of Test Conducted
• Serial Number
• Statement to Execute
• Expected Result
• Actual Result
• Remark
(Make this in table format)
Test Report
• Name of the Project
• Name of the Module/Subsytem
• Type of the Testing
• In the Heading Present the total number of Defects
• Graphical Analysis of Test Result visa Histogram or so….
6)what is the activity done in acceptance Testing
which is not done in System Testing to ensure the
customer requirements
• Customers are involved in Acceptance Testing which is
normally not done in System Testing. Usually in Ideal cases
Acceptance Testing is Customer’s responsibility , though most
of the time it is done with the support of developers.
•
7)What 3-tools would you purchase for your company for
use in testing and justify why you would want them
o I will buy these tools
Microsoft Word ( To make Test Plan , Test Script,
Test Case)
Microsoft Excel ( For Defect Metrics )
Microsoft PowerPoint ( For Presentations)
•
8)Write a Sample Test Policy that will be used in
hiring qualified testers
1. Definition of Testing – Claims Management System is HIPAA
compliant.
2. Testing System – Static and Dynamic Testing will be done using
the tools of MS Office, Empirics etc
3. Evaluation- Overall Failure Cost
4. Standards – Application has to follow all company standards.
511 http://www.certmagic.com
BH0-003
•
9)What qualities a Tester should have
o Good Communication Skills (Includes Good Listener)
o Test to Break Attitude
o Critical Eye
o Programming Fundamental knowledge
o Good Aptitude
o Convincing Skills
o Diplomatic
o Friendly
o Good Error Guessing
o Tools Knowledge
o Latest Industry Trend Knowledge
o Certified as Testers
o Conflict Resolution Technique
o
10)If your company is going to conduct a review Meeting
what position would U select the review committee and
why?
o I will like to take the position of moderator. That will give
me full control on the review before the Review, During
the Review and after the review.
Questions
Difference between test case and test condition documents
Q: can any one tell me the difference between test case and test condition
documents in the sense what each one is made of what i think is that a test
condition document is prepared before the test case document is prepared,
from what i understand i will write the test condition
for a login screen as follows:
1. enter valid data into the username and password fields and press login
button. and the test case for the above test condition as I would write is:
1. enter "suman" in user name
2. enter "passwrd" in the password field
3. press the login button.
this is what i understand of test condition and test case. as i do not rely on
my source of knowledge I want the experts to correct me if i am wrong in
any way....
Ans:
Your answers are ok with small corrections.
Enter "suman" in login field is a test condition with a test case.
Enter alphanumeric charaters (or whatever according to the requirements) of
512 http://www.certmagic.com
BH0-003
length between x & y is a test condition. A table containing input to this field
including -ve test data like junk characters etc...
(which are used for validations) form the data-driven test cases for that
particular condition alone.
Again, Enter "password" and click on "Login" button are 2 separate test
conditions. For password entry, you can create a table for test cases while
(similar to the one mentioned above), for the 3rd condition - clicking on the
button, using keyboard shortcut, using enter key if the
button is made "default", etc.. are the test cases...
Test Cases : defines the steps to execute a test
condition/scenarios.
Test Condition: specific conditions/scenarios in the functionality of the sw
that could also be treated as the V&V. The Test case document may include
the verification points (conditions) or
we can put these V&V's in a separate document, in case of automated
testing we put both together Can some one throw some light on the basic
differences between Incremental testing and Unit Testing???
Also give some idea about "thread testing".
Ans: in my knowledge unit testing is the form of testing you perform on the
smallest modules of the software written. when each module is written
seperately by a developer and tested for correctness, that is called unit
testing. some times we need to write stubs to test one module to emulate
other modules that may not yet been completed but reqiured by this module
under test for input or output. and about incremental testing....i have read
about incremental integration testing. if you mean the latter one, its like
combining 2-3 modules and perform integration testing on this combined
module,...then add again 2-3 modules and then test this whole big
module...and sooo on. this is integration testing. and about thread testing is
it????
well hmmmmm havent heard of......
these views are what i understood by reading the definitions of the terms.
please make sure from other sources too.
You're right about Unit and Incremental Testing. In Incremental Testing there
are two types, Top-down and Bottom-up. Thread testing is a technique that
is used to test string of units that accomplish a specific function in the
application. An example of thread testing would be the testing of a business
transaction through the integrated client, server and network. I'm using the
terminology from the Study Guide.
Unit testing involves testing the code - path, condition, branch, etc.. each
must be executed at least once...
Incremental testing is a an integration test type in which u test collection
of units in increments.
Top-down & Bottom up integration are again, types of Integration test
which use "drivers" and "stubs". here a vertical chain of units are tested.
513 http://www.certmagic.com
BH0-003
Thread Testing uses strings of units in one horizontal line for testing a
certain functionality...
For best info, please refer "Study Guide" and / or "An Integrated Approach to
S/w Engg" by Roger Pressman..
Can anybody give some points to go for Test Automation or How you decide
upon a Test Case?, you want to go for Manual Test or use Automation Tools.
Ans: At the outset, u should know the basics of Test Automation and how
the whole thing works... U should also know for most applications, the use of
Test Script Language (which if u know C and C++) should get u going on
that..
see this sites :
the primer examples and the language description in the faq and the primers
will give u a good look-in as to the possible areas and scenarios of use and
then u can do the following! :
[step 1]
check for the application type
- web based application
- database application
- COM based application (VB ,VC)
[step 2]
identify the areas for the automation , this depends upon how u are going
to automate the application
- by recording the steps
- by writing the test scripts
Now if you have the test cases ready with you , you can just convert the test
cases into the test scripts and test the application. in some of the cases you
can not automate every component of your applciation , so only remaining
option is manual testing. Automation plays an important role in the all
cycles/phases of testing and it promotes re-use! like the same way u can re-
use test cases atleast 50% of the time!
Function Point Analysis –
Q.: I have some questions on Functional Point Analysis.
1. What is the purpose of Functional Point Analysis ?
2. What are the tools available to do Functional Point Analysis ?
3. Does there exists any constant/standard "Functional Points" specific
for the programming language used to built, that particular application ?
4. When should Functional Point Analysis is done in the Software
Development Life Cycle ?
5. What would the benefits of Functional Point Analysis ?
Ans :
514 http://www.certmagic.com
BH0-003
This link can be answer user questions.
http://www.sei.cmu.edu/activities/str/descriptions/fpa_body
--
I found the following description with respect to FPA in a document "
SDLC models.doc" that
Costing:
FPA method:
Costing in software development is done using function points and person
hours. Function points are a unit measure for software much like an hour is
to measuring time, miles are to measuring distance. According to industry
sources, if you are developing a project in COBOL 85, 175 LOC of COBOL
code translates to 1 FP. Or if it is C++, 54 lines of code translate to 1 FP.
The average productivity for COBOL code for 1 person 1 day is 1 to 1.5 FP.
With this you can arrive at the effort for the project. FPA is a method to
break systems into smaller components, so they can be better understood
and analyzed. It is difficult to use the FP method while doing a prototype in
multimedia projects as LOC do not play any role in the project and it is
difficult to measure coding in Flash. From the above description, I thought
that FP would be constant/standard for particular language. If so, then 1 FP
equates to how many lines of code in Java ?. i.e. my requirememt is to
calculate FP's in Java code. And if the above description is incorrect, how to
calculate the FP
manually.
-
What is the purpose of Functional Point Analysis ?
- The main purpose of the FP analysis is to estimate size of the work product
and thereby the estimated man hours will be calcualted.
2. What are the tools available to do Functional Point Analysis ?
- I have no Idea. But most of the times FP will be calculated manually.
3. Does there exists any constant/standard "Functional Points" specific for
the programming language used to built, that particular application ?
FP analysis is not specific to the programming language. It is universal and
can be applied to any s/w product.
4. When should Functional Point Analysis is done in the Software
Development Life Cycle ?
It should be done during the estimation i.e., before submitting the project
proposal. Should be re-evaluated at the requirements stage.
5. What would the benefits of Functional Point Analysis ?
Major advantage is you will gain the control over the SDLC as this is one of
the most reliable and realistic approach for size estimations.
You would find a plethora of information on FPA in this free manual(.doc)
from www.softwaremetrics.com.
515 http://www.certmagic.com
BH0-003
This link would lead to download the same .
http://www.softwaremetrics.com/freemanual.htm
This manual should be of great help and would answer most of your
questions.
This may not be "The Bible" for FPA , but should be a great Reading material
for starters.
Does anyone have sofytcopy of the user guides for Winrunner and Load
runner? If so, can anyone please pass it on?
Ans: Can u just get into www.wilsonmar.com and in that click at "Related
Topics - TSL Coding"
Hope u can get there all the user guides...
Integration Testing -
Ans: It is necessary to write seperate test cases for Integration Testing. The
test case includes to test,
1. Communication between components/modules
2. Functionality which uses the components/modules
etc.
Seperate Test case template is not necessary.
--
As we all know, Integration testing mainly concentrates on Data Input from
other screens/modules and Data Transfer to other modules. In that case I
generally follow a template for Integration testing. Please find the attached
sample document. Hope this will give a better idea.
difference between manual support testing and operations If U mean OAT
....then
OAT (Operational Acceptance Testing) means the testing will be carried out
in client environment using scamble production data. Everything will be
controlled by the Client. U can say this is same as Beta Testing. It is part of
UAT (User Acceptance Testing). In Alpha Testing (one other form of UAT) the
environment is a controlled environment (Developer environment). The
testing will be in presence of Client. But the control is with the Development
Team and the testing team of that project.
Manual Support Testing: I am not clear abt the question. Do u mean Manual
Testing.....
What is Test Strategy?
What is difference between Test Plan and Test Strategy?
What is Internationalisation Testing?
Ans: Test Strategy : This is one of the item falls under the Test Plan, and
test strategy will give full insight to the counterpart that how will be the
516 http://www.certmagic.com
BH0-003
Testing carried out, viz, for preparation of test data we will use boundary ,
equivalence partitioning, and we will be using this % of white box and this %
of black box........
2. Test Strategy is the sub set of the Test Plan.
3. Internationalisation Testing : The same application in various Languages,
i.e. multi language testing, u may have to test the application selecting
various language.
As far as I'm concerned, Internationalisation testing consist of testing in
multilingual browsers(like
french, japenese etc) with respective plug-in if required.
Questions Set 1
1) What is the percent of the total cost of quality that comes from rework?
2) What is the percent of the total gross of sales that come from product
failure?
3) What are the three types of interfaces?
4) True or False. The largest cost of quality is from production failure.
5) True or False. A defect is related to the term fault.
Ans:
For 1st three there is no specific answer. U need to have lot of experience
before answering the questions. Cost of quality involves many thing. It
depends on the product. For last 2, the answer is yes and no. If u think
deeply in to that u will get the answer why it is yes and No.
Knowledge Domain 1: TEST PRINCIPLES AND CONCEPTS
1. Name the causes from which most of the testing problems occur.
2. In _______________ situation, the cost of testing is less than the
resultant loss from undetected defects.
3. As the cost of testing _________________, the number of undetected
defects ______________ .
a) decreases ; decreases
b) increases ; increases
c) increases ; decreases
d) decreases ; remains same
e) remains constant ; decreases
4. __________________ point in a testing cost curve (cost-effectiveness of
testing curve) separates the __________________ condition and
___________________ condition.
5. At the optimum point condition in a testing cost curve, the cost of testing
to uncover defects _________ the losses from those defects.
6. Define cost-effective perspective in testing.
7. Name the criteria involved in a testing policy.
517 http://www.certmagic.com
BH0-003
8. Testing cost curve is a graph plotted between _________________ and
__________________.
9. Draw the Testing Cost Curve and the Optimum Test (where it occurs).
10. What is The Domino Effect ?
11. Name the technological developments that caused organizations to revise
their approach to testing.
12. What is testing policy?
13. Name the two components of the testing strategy and define them.
14. ________________ method, which is a true organizational policy, is
used to establish a testing policy.
a) Management Directive
b) Information Services Consensus Policy
c) Users Meeting
d) All the three above
e) None of the above
15. ________________ is an organizational responsibility.
16. ________________ (method) is an economical and effective method to
write a testing policy.
17. Testing after coding is the only _________________ technique used to
determine the adequacy of the system.
18. Mention the stages of a traditional software development life cycle.
19. It is not unusual to hear of testing consuming 50% of the development
budget. (True / False)
20. Studies have shown that the majority of system errors occur in the
_______________ phase.
a) Requirements
b) Design
c) Code
d) Testing
e) Operation & Maintenance
21. Coding errors amount to __________ % of all detected system errors.
a) 20 b) 25 c) 36 d) 50 e) 64
22. Studies show that approximately two-thirds of all detected system errors
can be attributed to errors made during the design phase. (True / False)
23. Over _________ % of the life cycle costs of a software system are spent
on maintenance.
a) 40 b) 50 c) 60 d) 30 e) None of the above
Those planning to appear for exam can go through these and prepare
answers for themselves. As and when I get time, I will post series of
questions on various topics covered in the examination.
In designing a test strategy, _______________ become the basis or
objective of testing.
25. List the considerations in Developing Testing Methodologies.
26. Strategic risks are the low-level business risks faced by the software
system. (True / False)
518 http://www.certmagic.com
BH0-003
27. _____________ risks are the high-level business risks faced by the
software system.
28. Tactical risks are the high-level business risks faced by the software
system. (True / False)
29. Tactical risks are subsets at a lower level of the strategic risks. (True /
False)
30. What is the reason in decomposing the strategic risks into tactical risks?
31. Name the categories into which tactical risks can be divided.
32. _________________ must be developed to describe when and how
testing will occur.
33. What is incremental testing?
34. It is difficult to create test scenarios for high-level risks. (True / False)
35. What are the different types of incremental testing?
36. ________________ testing assumes that the path of logic in a unit or
program is known.
a) White Box b) Black Box c) Static d) Performance
e) None of the above
37. Modules are added in descending hierarchical order in _______________
type of incremental testing.
38. During software acceptance (acceptance testing), _______________
testing technique is relied upon for more accurate results (than other testing
techniques listed below).
a) White Box b) Black Box c) Incremental d) Thread
39. When evaluating the paybacks received from various test techniques,
_________________ testing produces a higher defect yield than other
dynamic techniques when planned and executed correctly.
40. What qualities must an individual possess to test effectively a software
application?
41. Due to a change in design, the requirements for an already coded (built)
software component (i.e., 1 software component among 10 software
components in an application) got modified in its entirety. The developer
had to modify the code based on the new requirements. Your role as a test
manager is to choose the appropriate type of regression test to minimize
the impact to the project schedule. What are the types and which type of
regression test would you choose?
42. At a minimum, the developer should always execute Unit Regression
Testing when a change is made. (True / False)
43. What are the rules that should be followed for all reviews?
44. What is ‘stage containment’ as referred to reviews?
Someone please review the answers and provide the answer of those
questions which are unanswered...
As per guide book this answer
20. Studies have shown that the majority of system errors occur in
the _Requirement phase.
a) Requirements
519 http://www.certmagic.com
BH0-003
b) Design
c) Code
d) Testing
e) Operation & Maintenance
is nor correct. Guide says that it is Design Phase.
-
Your answer is correct as most of the errors found down the line in SDLC
phases are there because requirements are not freezed correctly in
Requirement phase.
--
I agree with guide that most of the error are due to problem in design phase.
Suppose we have req. X eg."abc function should work". and at the design
dependencies(functional & parameter) are not correctly defined, result is that
req. X is working. I wud like to say that at the time of designing if approach
is clear and defined then req X will work fine but due to design issue it is not
working. Basically req. are defined because application need those function.
I do not mean to offend you But i think that most of the erros are due to the
problem in "Requirement stage" the several reasons for this is :
1)In correct requirements
2)Requirements not communicated properly.
3)" " not understood properly.A developer may have implemented a
reuirement incorrectly.
4)A changed requirement implemented withoput communicating it..
-
"Studies have shown that the majority of system errors occur in "
I think that the word error is doing the trick here.
According to CBOK it says DESIGN is the answer and I think that it is right,
because there can be wrong designing of architecture inspite of
understanding the requirements correctly whih can cause error in system.
This question is indirectly related to Verfication and Validation part which we
have discussed in detail in previous mail.
Here are some objective/subjective questions for Exam.
Topics: Software Testing Strategies - alpha and beta testing; criteria for
software testing completion; debugging; ITG; various testing techniques.
1. Testing begins at the component level and works “outward” towards the
integration of the entire computer-based system. (True/False)
2. For object-oriented systems, testing begins at the __________________
level.
3. Testers get involved with the project only when the testing steps are
about to begin. (True/False)
520 http://www.certmagic.com
BH0-003
4. What is the role of an independent test group (ITG)?
5. Model of software failures (uncovered during testing) as a function of
execution time is a graph resulting in _______________ shape.
a) linear b) parabolic c) exponential
d) logarithmic
6. A software failure model is a graph between _____________ (y-axis) and
_____________ (x-axis).
a) Number of software failures; Execution Time
b) Failures per test hour; Execution Time
c) Execution Time; Failures per test hour
d) Failures per test hour; Time taken to fix the failures
7. Define antibugging.
8. A ‘bathtub’ curve can be applicable to
a) hardware b) software c) hardware and software
d) None of the above
9. Alpha test is conducted at one or more customer sites by the end-user of
the software. (True/False)
10. Alpha tests are conducted in a ________________ environment.
a) controlled b) non-controlled
11. Beta tests are conducted in a controlled environment. (True/False)
12. ________________ test is conducted at the developer’s site by a
customer.
a) system b) alpha c) beta
d) None of the above
13. Name the three categories for debugging approaches.
14. What is meant by error seeding and fault injection?
15. Distinguish between isolation and debugging.
--
Preparing for The Exam –
521 http://www.certmagic.com
BH0-003
Read CBOK 2 times. I feel first start from topic 5 (Risk analysis) onwards first
as these topics are important from exam point of view.
Read testing by Willayam perry for topics like web based testing, client
server testing etc.
Solve questions in the file section of various groups.
Discuss case studies with in group.
Read vocabulary twice carefully as some objectives are directly from it.
Use bulleted points in the exam for subjectives
Generally in part1& 3- 50 objectives each for 45 min each
Part2 & 4- 8 subjectives for 75 min. each
Each paper is of 100 marks For passing u have to score 75 marks in each.
--
Skill Category:10 - Acceptance Testing
1. What is acceptance testing?
2. Software acceptance is ______________ process of approving or rejecting
software systems during development or maintenance, according to how well
the software satisfies predefined criteria.
3. Acceptance test is _______________ step in the "V" concept testing
process model.
4. Formal final software acceptance testing must occur at the end of the
____________ process.
5. What are the features/items that software acceptance, in a life cycle
process, has to care for?
6. If you happen to be the customer/user of the software, what are (your)
the responsibilities for software acceptance?
7. ________________ testing is designed to determine whether the software
is "fit" for the user to use.
8. The concept of "fit" in acceptance testing is important in
_____________ and _______________.
9. Mention the four components of fit in acceptance testing.
10. What are the concerns that a software user faces in making the
acceptance decision?
11. The acceptance testing workbench begins with software
__________________ for the system specifications.
12. What is a 'Use Case'?
13. What is a System Boundary Diagram? Give an example.
522 http://www.certmagic.com
BH0-003
14. 'Test Cases' cannot be developed with system users and designers as the
'Use Cases' are being developed. (True/False)
15. What are the acceptance requirements that a system must meet in
acceptance testing?
16. Name the inputs for the acceptance testing workbench.
17. There should be _____________ relationship between use case
definitions and test cases.
18. ________________ is the input to the test case work paper.
19. Software acceptance is a ____________ process during which users and
developers identify criteria for the acceptance of software systems.
20. What is the objective of acceptance testing?
Skill Category:10 - Acceptance Testing
1. What is acceptance testing?
Acceptance Testing is formal testing conducted to determine whether a
software system satisfied its acceptance criteria and to enable the buyer to
determine whether to accept the system. Software acceptance testing is
usually the final opportunity of the customer to examine the software and to
object the developer for insufficient or incorrect software.
2. Software acceptance is incremental process of approving or rejecting
software systems during development or maintenance, according to how well
the software satisfies predefined criteria.
3. Acceptance test is Final step in the "V" concept testing process model.
4. Formal final software acceptance testing must occur at the end of the
Developmental (Software Development Life Cycle ) process.
5. What are the features/items that software acceptance, in a life cycle
process, has to care for?
In Software Acceptance Testing Contract and Acceptance Criteria are most
important things those are required to be taken care of.
6. If you happen to be the customer/user of the software, what are (your)
the responsibilities for software acceptance?
1) Creation (or Update/review) of Acceptance Test Plan
2) Resource and Schedule for Acceptance Testing
3) Special Data collection for Acceptance Testing
4) Training Logistics for Acceptance Testing
5) Following Standard Practices while carrying out any tasks.
7. Acceptance testing is designed to determine whether the software is "fit"
for the user to use.
523 http://www.certmagic.com
BH0-003
8. The concept of "fit" in acceptance testing is important in Design_ and
Testing .
9. Mention the four components of fit in acceptance testing.
1) Data
2) People
3) Structure
4) Rules
10. What are the concerns that a software user faces in making the
acceptance decision?
If a critical requirement is not met the decision is Reject but if a non critical
requirement is not met the decision may be taken in favor of passing the
project.
11. The acceptance testing workbench begins with software __contract for
the system specifications.
12. What is a 'Use Case'?
Use case is a set of sequence of steps describing an interaction between user
and system.
13. What is a System Boundary Diagram? Give an example.
The system Boundary diagram is a diagram which represents the system in a
way which separates the Actors (Users) of the system from Use cases
(Events of the system ) thru a boundary.
14. 'Test Cases' cannot be developed with system users and designers as the
'Use Cases' are being developed. (True/False)
False
15. What are the acceptance requirements that a system must meet in
acceptance testing?
All critical requirements , performance requirements and functional
requirements must be met.
16. Name the inputs for the acceptance testing workbench.
Contract , and Acceptance Criteria.
17. There should be one to many relationship between use case definitions
and test cases.
18. Contract_ is the input to the test case work paper.
19. Software acceptance is a (n)Ongoing process during which users and
developers identify criteria for the acceptance of software systems.
524 http://www.certmagic.com
BH0-003
20. What is the objective of acceptance testing?
The objective of acceptance testing is to ensure that throughout the
development cycle all aspects of development process meet the user’s need.
--
Suppose we have four level of priority(P1,P2,P3,P4) and
severity(S1,S2,S3,S4).
What should be priority and severity of the validation related bugs.
Any relevant info/doc/links would be helpful.
Ans:
If u detect any Defect, u shud assign Priority and Severity for that Defect.
Severity : is with respect to the application
Priority : is with respect to the Delivery of the application
eg. For a defect, priority may be very high and seviority may be very less or
visa versa.
-
I am also preparing for exam. This question has been raised many
times that how to prepare for this exam. Here is my answer
1) Become a member of (which you have done already)
2)Become a member of CSTEHelpDesk yahoo Group
3)Actively participate in both the Groups.
(Read all e-mails. Save all good e-mails. Go thru all previous e-
mails also... you will find lots of your questions answered )
4)Read the contents of CSTE_Preparation.zip which is available under
file section of this group.
5)Go to www.softwarecertification.com and collect the latest CBOK
(Common body ok Knowledge, avaliable on File section too. But going
thru the site by yourself will give you more confidence )
6) Arrange Guide Book and study that as many times as you can.
7)Solve all questions available under File section.
8)Read some more books Example ...... Effective methods of software
Testing Testing Computer Software etc.
9) Try to read and understand all documents available under file section of
yahoo group.
10) If you still have some extra time, become a member of CSQA yahoo
group also. Sometimes testing concepts are discussed in this group also and
some members are really experienced in this group.
525 http://www.certmagic.com
BH0-003
Skill Category: 6 - Test Planning Process
1. What is the purpose/objective of a test plan?
2. "Testers are in a lose-lose situation". What does this mean?
3. What is it that tester(s) complain usually?
4. Test plan is a ________________ between the testers and the project
team / users describing the role of testing in the project.
5. The test plan should provide: [SELECT ALL
THAT APPLY] a) background
information on the software being tested b) details of each
individual testing the software c) test
objective d) risks that need
to be taken care of e) specific tests to be
performed
6. List the concerns that testers face in assuring that the test plan will be
complete.
7. The concern "Us-versus-Them Mentality" occurs/arises a)
when developers and testers are on the same side of testing issue b)
between testers and end-users c) when
developers and testers are on opposite sides of testing issue d) between
developers and end-users
8. Developing the test plan can be shown to be _________ step on the 11-
step software testing process.
9. Which of the following tester's concern leads to the syndrome "throw it
over the wall"? a) Us-
versus-Them Mentality b) Lack of
customer and user involvement c) Not enough
time for testing d) Overreliance on
independent testers e) Having to say no
10. Test planning is an activity that commences after the product
requirements analysis documents are ready. (True/False)
11. List the input(s) used in developing the test plan.
12. Match the "TEST FACTOR" with the appropriate "TEST":
TEST FACTOR TEST (a)
RELIABILITY (i) Recovery Testing (b) FILE
INTEGRITY (ii) Manual, Regression & Functional Testing (c) ACCESS
526 http://www.certmagic.com
BH0-003
CONTROL (iii) Stress Testing (d) CONTINUITY OF
PROCESSING (iv) Compliance Testing (e) SERVICE
LEVEL (v) Functional Testing (f)
PORTABLE (vi) Operations Test (g) EASE
OF OPERATION (vii) Diter Testing
Pick ONE from the given choices:
A. (a)-(v); (b)-(iv); (c)-(i); (d)-(ii); (e)-(vii); (f)-(iii); (g)-(vi)
B. (a)-(ii); (b)-(v); (c)-(i); (d)-(iii); (e)-(iv); (f)-(vii); (g)-(vi)
C. (a)-(ii); (b)-(v); (c)-(iv); (d)-(i); (e)-(iii); (f)-(vii); (g)-(vi)
D. (a)-(v); (b)-(iv); (c)-(ii); (d)-(i); (e)-(vi); (f)-(vii); (g)-(iii)
13. Choose the one from the list of TEST FACTORS given below that doesn't
map to Functional Testing.
a) Reliability b) File Integrity c) Audit Trail d)
Correctness e) Performance f) Coupling
14. What are the contents of a Software/Test Matrix?
15. Name any 5 structural attributes of the software, which you might have
worked upon, that may be impacted and thus require testing.
--
Answers to Skill Category: 6 - Test Planning Process
Skill Category: 6 - Test Planning Process
1. What is the purpose/objective of a test plan?
The main purpose is to organize and manage the Test Efforts and maximize
the resource utilization to carry out testing tasks.
Detailed Objectives
Improve Testing Coverage
Avoid unnecessary repetitions
Analyze the program and spot good test cases quickly
527 http://www.certmagic.com
BH0-003
Improve test efficiency
2. "Testers are in a lose-lose situation". What does this mean?
If they find more bugs – They are blamed for delaying the project,
Developers are not happy and developers become defensive too.
If they find less bugs – They are blamed for poor quality, Test manager is
not happy , Upper Management is not happy.
3. What is it that tester(s) complain usually?
Less testing Time, Low Priority to test
4. Test plan is a __bridge___ between the testers and the project team /
users describing the role of testing in the project.
5. The test plan should provide: [SELECT ALL
THAT APPLY]
a) background information on the software being tested b)
details of each individual testing the software c) test
objective
d) risks that need to be taken care of e)
specific tests to be performed
6. List the concerns that testers face in assuring that the test plan will be
complete.
Not Enough Training
Us versus Them mentality
Lack of Test Tools
Lack of Management Understanding/Support of testing
Lack of Customer and User Involvement
Not Enough Time for testing
Over Reliance On Independent Testers
Rapid Changes
528 http://www.certmagic.com
BH0-003
Testers Are in a Lose-Lose Situations
Having to Say ‘No’
7. The concern "Us-versus-Them Mentality" occurs/arises a)
when developers and testers are on the same side of testing issue b)
between testers and end-users
c) when developers and testers are on opposite sides of testing
issue d) between developers and end-users
8. Developing the test plan can be shown to be _second___ step on the 11-
step software testing process.
9. Which of the following tester's concern leads to the syndrome "throw it
over the wall"?
a) Us-versus-Them Mentality
b) Lack of customer and user involvement
c) Not enough time for testing
d) Overreliance on independent testers
e) Having to say no
10. Test planning is an activity that commences after the product
requirements analysis documents are ready. (True/False)
11. List the input(s) used in developing the test plan.
Software Contract
Software Project Plan ( Which contains Project Schedule)
Requirements Document
Design Document
529 http://www.certmagic.com
BH0-003
12. Match the "TEST FACTOR" with the appropriate "TEST":
TEST FACTOR TEST
(a) RELIABILITY (i) Recovery Testing
(b) FILE INTEGRITY (ii) Manual, Regression & Functional Testing
(c) ACCESS CONTROL (iii) Stress Testing
(d) CONTINUITY OF PROCESSING (iv) Compliance Testing
(e) SERVICE LEVEL (v) Functional Testing
(f) PORTABLE (vi) Operations Test
(g) EASE OF OPERATION (vii) Diter Testing
Pick ONE from the given choices:
A. (a)-(v); (b)-(iv); (c)-(i); (d)-(ii); (e)-(vii); (f)-(iii); (g)-(vi)
B. (a)-(ii); (b)-(v); (c)-(i); (d)-(iii); (e)-(iv); (f)-(vii); (g)-(vi)
C. (a)-(ii); (b)-(v); (c)-(iv); (d)-(i); (e)-(iii); (f)-(vii); (g)-(vi)
D. (a)-(v); (b)-(iv); (c)-(ii); (d)-(i); (e)-(vi); (f)-(vii); (g)-(iii)
13. Choose the one from the list of TEST FACTORS given below that doesn't
map to Functional Testing.
a) Reliability b) File Integrity c) Audit Trail d)
Correctness e) Performance f) Coupling
14. What are the contents of a Software/Test Matrix?
Test Matrix is a Table (or an arrangement), which contains different Modules
to be tested as Rows and Types of Testing techniques to be conducted as
Columns. One or more kind of testing techniques can be conducted for one
module.
15. Name any 5 structural attributes of the software, which you might have
worked upon, that may be impacted and thus require testing.
530 http://www.certmagic.com
BH0-003
Maintainability
Reliability
Complexity
Efficiency
Usability
--
I need some input from you from your experience regarding " Data
varification and Data Validation " Testing.
If u have done similar kind of testing or if u any input on this topic just let
me know.
We have Seibel Database .We have two other databases one is duplicate of
this original Seibel database and other one partial data fetched from the
original database.
So I want to do " Data varification and Data Validation " Testing on this
And varify the data in the database.
And to this kind of testing what I should I need?? i mean Reqtments???
So how to go about it kindly suggest.....
Ans:
In my earlier project we had similar kind of data related testing also.
we had AC Nielson Data, POS Data and other various data's coming into our
system
and based upon that data application enacts. We had Data Pulling and Data
Loading.
So what we did was we just divided into 2 parts,
1. We just concentrated about Data Pulling and Data Loading.
2. We did the validation on the application with the data available in the
database.
let me expain in detail about this.
1. Data Pulling : We had our application which will have some batch jobs and
schedulars which gets triggered automatically
531 http://www.certmagic.com
BH0-003
and request file is sent to the NT Server and that will send the request file to
the AC Nielson Mainframe Server, and
AC Nielson Server will send the response file to the NT Server and NTserver
will redirect to the application.
So this covers the Data Pulling.
Data UpLoading : We first validate whether the data which we have received
is the valid and correct data which we have requested
for and this validation of the data is done manually. and once we feel that
the data is correct and we have received the complete
data which we have requested for the we start doing the data Uploading. so
the new data gets replicated into the local database server.
Here we validate the application uploads the data properly, means first it
should upload one table and that triggers to another table(eg. first temp
table
gets updated and then data will be moving to the concern tables) this Data is
also will be validated manually.
so here we end with the Data Pulling, Data Validation and Data Uploading.
2. Application Validation : We assume that the data available data in our
database is correct and then we run the application whether that
is behaving in the proper way.
Here we end with the application Testing/Validation.
Here we can corelate the above with u'r application,
a) Main Siebel Server is here depicted as AC Nielson Mainframe Server
b) Local Siebel Replication -- Local Database
c) Both aplications contacting local Databased for processing.
--
This email message is a notification to let you know that
a file has been uploaded to the Files area of the
group.
File : /SoftwareEngineeringQuiz.doc
Uploaded by : rkavirayani <rkavirayani@y...
Description : Student Quiz from the book "SOFTWARE ENGINEERING - A
Practitioner's Approach" 5/e :)- By Roger S. Pressman. This has been
compiled
chapterwise from the book. Use those chapters as relevant for CSTE
Examination. The same can be obtained at http://www.mhhe.com/pressman
web site
and click on "Student Center" to see the quiz questions chapterwise. This
document is compiled for the benefit of all those aspirants who plan to get
certified in various Software Engineering/Software QA exams.
532 http://www.certmagic.com
BH0-003
You can access this file at the URL
http://groups.yahoo.com/group/CSTE/files/SoftwareEngineeringQuiz.doc
To learn more about file sharing for your group, please visit
http://help.yahoo.com/help/us/groups/files
--
look at the files section in help desk. u can get many objective
questions
Chapter 1: The Product
Chapter 1 Self-Check Quiz
1. What factor has precipitated more sophisticated and complex computer-
based systems?
a. Vast increases in computer memory and storage capacity.
b. Greater variety of exotic input/output options.
c. Profound changes in computer architectures.
d. All of the above.
2. Which question no longer concerns the modern software engineer?
a. Why does computer hardware cost so much?
b. Why does software take a long time to finish?
c. Why does it cost so much to develop a piece of software?
d. Why can't software errors be removed from products prior to delivery?
3. Today the increased power of the personal computer has brought about an
abandonment of the practice of team development of software.
a. True
b. False
4. Software is a product and can be manufactured using the same
technologies used for other engineering artifacts.
a. True
b. False
533 http://www.certmagic.com
BH0-003
5. Software deteriorates rather than wears out because
a. Software suffers from exposure to hostile environments
b. Defects are more likely to arise after software has been used often
c. Multiple change requests introduce errors in component interactions
d. Software spare parts become harder to order
6. Most software continues to be custom built because
a. Component reuse is common in the software world
b. Reusable components are too expensive to use
c. Software is easier to build without using someone else's components.
d. Off the shelf software components are not commonly available
7. The nature of software applications can be characterized by their
information
a. complexity
b. content
c. determinacy
d. choices "b" and "c"
( I am not sure about this )
8. Modern software applications are so complex that it is hard to develop
mutually exclusive category names.
a. True
b. False
9. The current software crisis was caused by the Y2K problem whose seeds
were first sown by careless programmers in the early 1970's.
a. True
b. False
10. Software developers succeed more often than they fail, but software
failures receive more press coverage.
a. True
b. False
11. Adding more people to a project that is already behind schedule is a good
way to catch up.
a. True
b. False
12. Modern CASE tools are more important than the newest hardware for
achieving good software quality and productivity.
534 http://www.certmagic.com
BH0-003
a. True
b. False
13. Change cannot be easily accommodated in most software systems,
unless a system was designed with change in mind.
a. True
b. False
14. A general statement of objectives is all that is needed to begin
developing a piece of software.
a. True
b. False
(I am not sure about this)
15. The formal technical review is an inadequate substitute for testing
regardless of nature of the software defect.
a. True
b. False
16. Documentation is no longer a necessary part of the software
development process because no one reads it.
a. True
b. False
-
You can get the answers by solving the questions directly online at
http://www.mhhe.com/engcs/compsci/pressman/student/olc/chap01quiz.mh
tm
l
You have the choice to select the chapter and then solve it. Once you
submit the page, it evaluates your performance. Must say its a very
cool site.
Can anyone explain me about the exact difference of the following queries
1) Difference between Sanity and Testing and Smoke Testing?
2) DIfference between Verification and Validation
3) TEst strategy, Test Methdology , Testing techniques
Ans:
535 http://www.certmagic.com
BH0-003
The CBOK Domain 1 simplifies Verification as the answer to 'Did we
build the right system?' and Validation as 'Did we build the system
right?'. Verification would mostly be done before the system is
built, while validation is the testing during and after the system is
built. Does that sound right to others?
No offense but I guess it should be exactly the opposite of what u
typed.
Verification is comparison between actual characteristics of something
(can be a product of a s/w project) and the specified characteristics
It is checking that you have built the system right.
Validation is comparison between actual charateristics of something
(can be a product of a s/w project) and the expected charateristics.
It is checking that you have built the right system
But now I am really confused, I thought verification
was performed during requirements review, so how could it test if the
system is built right if it isn't even built yet? Is it verifying
that the system WILL be built right according to the requirements?
Verification and validation are technically synonyms, so how much
emphasis on the exam is put on the QAI definitions of these words?
I hate to reply to my own message, but I researched this further and
am still confused since even archived message posts contradict the
CBOK. The CBOK Domain 1 Part 2 at the bottom of page 49 does state
the definition as I posted earlier, 'Verification answers the
question, "Did we build the right system?" while validation
addresses, "Did we build the system right?". So, any thoughts as to
why the CBOK states the opposite of what other people tend to think?
On the exam, which way would you answer the question?
Please find the enclosed Doc on Verification &
Validation differences..
--
536 http://www.certmagic.com
BH0-003
Verification is the process of ensuring the accuracy of a project based on
Written Requirements and Specifications.
Where as
Validation is the certification at the end of audit / review that states of our
Findings.
--
Verification:-
- Make sure you have done the thing right.
- Determine whether product of each phase follow blueprint of last
phase.
Validation:-
- Make sure you have done the thing right.
- Determine whether software as built meets current user needs.
-
- Inspection, walkthroughs, reviews (Verification, Validation)
- Unit testing (Verification)
- Integration testing (Verification)
- System testing (Validation)
- Acceptance testing (Validation)
- Maintenance testing (Verification, Validation)
- Regression testing (Verification, Validation)
Q: Can anyone explain me about the exact difference of the following
queries
1) Difference between Sanity and Testing and Smoke Testing?
2) DIfference between Verification and Validation
3) TEst strategy, Test Methdology , Testing techniques
Smoke test describes an initial set of tests that determine if a new
version of application performs well enough for further testing
Sanity tests are subsets of the confidence test and are used only to
validate high-level functionality.
--
Since many of you have been asking me the same question of how to get the
answers for the Objective questions that I compiled from the above book:)-
537 http://www.certmagic.com
BH0-003
1. Go the web site http://www.mhhe.com/pressman and click on
"Student Center". Choose the chapter that you wish to answer
and click on "Student Quiz" which opens up in a new window.
After you answer the questions and click "Submit" button, the
answers are displayed and any explanation for incorrect
answer(s) (those attempted by you) is also given. If you can
get hold of the book, it's good - else, go through the Student
Outlines available on or CSTEHelpdesk yahoo groups, which
should help you in answering most of the questions.
The testing method that uses statistical techniques to find out
how faults in the program affect it's failure rate is:
Your answer:
cyclomatic complexity
fault estimation
error guessing
statistical testing
2. faults that cause an input to be associated with the wrong path
domain are called
Your answer:
process flow inconsistencies
domain faults
computation faults
path errors
3. If you know that a developer tends to have extra errors in
date-processing code, and decide to test dates harder than
usual as a result, you are doing:
538 http://www.certmagic.com
BH0-003
Your answer:
risk based testing
static testing
error based testing
black box testing
4. Which of these is NOT an example of stress testing?
Your answer:
entering transactions to determine that sufficient disk space has
been allocated to the application
Ensuring that the communication capacity is sufficient to handle
the volume of work by attempting to overload the network with
transactions
Inducing a failure on one of the systems such that the program
terminates.
testing system overflow condtions by entering more transactions
than can be accommodated by tables, queues, internal storage
facilities, and so on
5. Fault seeding is instrumentation designed to
Your answer:
populate an audit trail
estimate the number of bugs left in the code by measuring how
many "bait" errors have not been found.
measure how many branches have been tested
test error handling code
539 http://www.certmagic.com
BH0-003
6. _______ attempts to decide what constitutes a sufficient set of
paths to test.
Your answer:
boolean anal
0 comments
Post a comment