The document discusses software testing techniques. It notes that often only final code is tested and there are many testing techniques. It then provides an overview of testing preliminaries, different test techniques, comparing techniques, and software reliability. Key points include that errors made early in development are often discovered late and are costly to fix, and it is beneficial to start testing early. The document discusses various manual and automated testing techniques like inspections, coverage-based testing, and requirements-based testing.
Chapter 10 Testing and Quality Assurance1Unders.docxketurahhazelhurst
Chapter 10:
Testing and Quality
Assurance
1
Understand quality & basic techniques for software verification and validation.
Analyze basics of software testing and testing techniques.
Discuss the concept of “inspection” process.
Objectives
2
Quality assurance (QA): activities designed
to measure and improve quality in a product— and process.
Quality control (QC): activities designed to validate and verify the quality of the product through detecting faults and “fixing” the defects.
Need good techniques, process, tools,
and team.
Testing Introduction
similar
3
Two traditional definitions:
Conforms to requirements.
Fit to use.
Verification: checking software conforms to
its requirements (did the software evolve
from the requirements properly; does the software “work”?)
Is the system correct?
Validation: checking software meets user requirements (fit to use)
Are we building the correct system?
What Is “Quality”?
4
Testing: executing program in a controlled environment and “verifying/validating” output.
Inspections and reviews.
Formal methods (proving software correct).
Static analysis detects “error-prone conditions.”
Some “Error-Detection” Techniques (finding errors)
5
Error: a mistake made by a programmer or software engineer that caused the fault, which in turn may cause a failure
Fault (defect, bug): condition that may cause a failure in the system
Failure (problem): inability of system to perform a function according to its spec due to some fault
Fault or failure/problem severity (based on consequences)
Fault or failure/problem priority (based on importance of developing a fix, which is in turn based
on severity)
Faults and Failures
6
Activity performed for:
Evaluating product quality
Improving products by identifying defects and having them fixed prior to software release
Dynamic (running-program) verification of program’s behavior on a finite set of test cases selected from execution domain.
Testing can NOT prove product works 100%—even though we use testing to demonstrate that parts of the software works.
Testing
Not always
done!
7
Who tests?
Programmers
Testers/Req. Analyst
Users
What is tested?
Unit code testing
Functional code testing
Integration/system testing
User interface testing
Testing (cont.)
Why test?
Acceptance (customer)
Conformance (std, laws, etc.)
Configuration (user vs. dev.)
Performance, stress, security, etc.
How (test cases designed)?
Intuition
Specification based (black box)
Code based (white box)
Existing cases (regression)
8
Progression of Testing
Equivalence Class Partitioning
Divide the input into several groups, deemed “equivalent” for purposes of finding errors.
Pick one “representative” for each class used for testing.
Equivalence classes determined by req./design specifications and some intuition
Example: pick “larger” of
two integers and . . .
Lessen duplication.
Complete coverage.
10
Suppose we have n distinct functional requirements.
Su ...
Problem Statement:One of the common concerns from the customers is that how to effectively optimize the testing given the
multiple integration points in a distributed/composite system environments, which does expose at least the below
pain points:
1. Avoid Exhausted testing
2. Meet all the boundary conditions
3. Limited time to execute 100% test execution
4. Include all the critical business functions
5. Efficient Regression Testing
and the list goes on...
Resolution: The solution is detailed in the attachment and have effectively implemented in various client places.
A presenetation on basics of software testing, explaining the software development life cycle and steps invovled in it and detials about each step from the testing point of view.
A presenetation on basics of software testing, explaining the software development life cycle and steps invovled in it and detials about each step from the testing point of view.
Normal Labour/ Stages of Labour/ Mechanism of LabourWasim Ak
Normal labor is also termed spontaneous labor, defined as the natural physiological process through which the fetus, placenta, and membranes are expelled from the uterus through the birth canal at term (37 to 42 weeks
A Strategic Approach: GenAI in EducationPeter Windle
Artificial Intelligence (AI) technologies such as Generative AI, Image Generators and Large Language Models have had a dramatic impact on teaching, learning and assessment over the past 18 months. The most immediate threat AI posed was to Academic Integrity with Higher Education Institutes (HEIs) focusing their efforts on combating the use of GenAI in assessment. Guidelines were developed for staff and students, policies put in place too. Innovative educators have forged paths in the use of Generative AI for teaching, learning and assessments leading to pockets of transformation springing up across HEIs, often with little or no top-down guidance, support or direction.
This Gasta posits a strategic approach to integrating AI into HEIs to prepare staff, students and the curriculum for an evolving world and workplace. We will highlight the advantages of working with these technologies beyond the realm of teaching, learning and assessment by considering prompt engineering skills, industry impact, curriculum changes, and the need for staff upskilling. In contrast, not engaging strategically with Generative AI poses risks, including falling behind peers, missed opportunities and failing to ensure our graduates remain employable. The rapid evolution of AI technologies necessitates a proactive and strategic approach if we are to remain relevant.
Macroeconomics- Movie Location
This will be used as part of your Personal Professional Portfolio once graded.
Objective:
Prepare a presentation or a paper using research, basic comparative analysis, data organization and application of economic information. You will make an informed assessment of an economic climate outside of the United States to accomplish an entertainment industry objective.
A review of the growth of the Israel Genealogy Research Association Database Collection for the last 12 months. Our collection is now passed the 3 million mark and still growing. See which archives have contributed the most. See the different types of records we have, and which years have had records added. You can also see what we have for the future.
Introduction to AI for Nonprofits with Tapp NetworkTechSoup
Dive into the world of AI! Experts Jon Hill and Tareq Monaur will guide you through AI's role in enhancing nonprofit websites and basic marketing strategies, making it easy to understand and apply.
This slide is special for master students (MIBS & MIFB) in UUM. Also useful for readers who are interested in the topic of contemporary Islamic banking.
Synthetic Fiber Construction in lab .pptxPavel ( NSTU)
Synthetic fiber production is a fascinating and complex field that blends chemistry, engineering, and environmental science. By understanding these aspects, students can gain a comprehensive view of synthetic fiber production, its impact on society and the environment, and the potential for future innovations. Synthetic fibers play a crucial role in modern society, impacting various aspects of daily life, industry, and the environment. ynthetic fibers are integral to modern life, offering a range of benefits from cost-effectiveness and versatility to innovative applications and performance characteristics. While they pose environmental challenges, ongoing research and development aim to create more sustainable and eco-friendly alternatives. Understanding the importance of synthetic fibers helps in appreciating their role in the economy, industry, and daily life, while also emphasizing the need for sustainable practices and innovation.
June 3, 2024 Anti-Semitism Letter Sent to MIT President Kornbluth and MIT Cor...Levi Shapiro
Letter from the Congress of the United States regarding Anti-Semitism sent June 3rd to MIT President Sally Kornbluth, MIT Corp Chair, Mark Gorenberg
Dear Dr. Kornbluth and Mr. Gorenberg,
The US House of Representatives is deeply concerned by ongoing and pervasive acts of antisemitic
harassment and intimidation at the Massachusetts Institute of Technology (MIT). Failing to act decisively to ensure a safe learning environment for all students would be a grave dereliction of your responsibilities as President of MIT and Chair of the MIT Corporation.
This Congress will not stand idly by and allow an environment hostile to Jewish students to persist. The House believes that your institution is in violation of Title VI of the Civil Rights Act, and the inability or
unwillingness to rectify this violation through action requires accountability.
Postsecondary education is a unique opportunity for students to learn and have their ideas and beliefs challenged. However, universities receiving hundreds of millions of federal funds annually have denied
students that opportunity and have been hijacked to become venues for the promotion of terrorism, antisemitic harassment and intimidation, unlawful encampments, and in some cases, assaults and riots.
The House of Representatives will not countenance the use of federal funds to indoctrinate students into hateful, antisemitic, anti-American supporters of terrorism. Investigations into campus antisemitism by the Committee on Education and the Workforce and the Committee on Ways and Means have been expanded into a Congress-wide probe across all relevant jurisdictions to address this national crisis. The undersigned Committees will conduct oversight into the use of federal funds at MIT and its learning environment under authorities granted to each Committee.
• The Committee on Education and the Workforce has been investigating your institution since December 7, 2023. The Committee has broad jurisdiction over postsecondary education, including its compliance with Title VI of the Civil Rights Act, campus safety concerns over disruptions to the learning environment, and the awarding of federal student aid under the Higher Education Act.
• The Committee on Oversight and Accountability is investigating the sources of funding and other support flowing to groups espousing pro-Hamas propaganda and engaged in antisemitic harassment and intimidation of students. The Committee on Oversight and Accountability is the principal oversight committee of the US House of Representatives and has broad authority to investigate “any matter” at “any time” under House Rule X.
• The Committee on Ways and Means has been investigating several universities since November 15, 2023, when the Committee held a hearing entitled From Ivory Towers to Dark Corners: Investigating the Nexus Between Antisemitism, Tax-Exempt Universities, and Terror Financing. The Committee followed the hearing with letters to those institutions on January 10, 202
2. 2
Nasty question
Suppose you are being asked to lead the team to
test the software that controls a new X-ray
machine. Would you take that job?
Would you take it if you could name your own
price?
What if the contract says you’ll be charged with
murder in case a patient dies because of a mal-
functioning of the software?
4. 4
State-of-the-Art
30-85 errors are made per 1000 lines of source
code
extensively tested software contains 0.5-3 errors
per 1000 lines of source code
testing is postponed, as a consequence: the later
an error is discovered, the more it costs to fix it.
error distribution: 60% design, 40%
implementation. 66% of the design errors are not
discovered until the software has become
operational.
5. 5
Relative cost of error correction
1
2
5
10
20
50
100
RE design code test operation
6. 6
Lessons
Many errors are made in the early phases
These errors are discovered late
Repairing those errors is costly
⇒ It pays off to start testing real early
7. 7
How then to proceed?
Exhaustive testing most often is not feasible
Random statistical testing does not work either if
you want to find errors
Therefore, we look for systematic ways to proceed
during testing
8. 8
Classification of testing techniques
Classification based on the criterion to measure
the adequacy of a set of test cases:
coverage-based testing
fault-based testing
error-based testing
Classification based on the source of information
to derive test cases:
black-box testing (functional, specification-based)
white-box testing (structural, program-based)
9. 9
Some preliminary questions
What exactly is an error?
How does the testing process look like?
When is test technique A superior to test
technique B?
What do we want to achieve during testing?
When to stop testing?
10. 10
Error, fault, failure
an error is a human activity resulting in software
containing a fault
a fault is the manifestation of an error
a fault may result in a failure
11. 11
When exactly is a failure a failure?
Failure is a relative notion: e.g. a failure w.r.t. the
specification document
Verification: evaluate a product to see whether it
satisfies the conditions specified at the start:
Have we built the system right?
Validation: evaluate a product to see whether it
does what we think it should do:
Have we built the right system?
14. 14
Test adequacy criteria
Specifies requirements for testing
Can be used as stopping rule: stop testing if 100%
of the statements have been tested
Can be used as measurement: a test set that
covers 80% of the test cases is better than one
which covers 70%
Can be used as test case generator: look for a test
which exercises some statements not covered by
the tests so far
A given test adequacy criterion and the
associated test technique are opposite sides of
the same coin
15. 15
What is our goal during testing?
Objective 1: find as many faults as possible
Objective 2: make you feel confident that the
software works OK
16. 16
Example constructive approach
Task: test module that sorts an array A[1..n]. A
contains integers; n < 1000
Solution: take n = 0, 1, 37, 999, 1000. For n = 37,
999, take A as follows:
A contains random integers
A contains increasing integers
A contains decreasing integers
These are equivalence classes: we assume that
one element from such a class suffices
This works if the partition is perfect
17. 17
Testing models
Demonstration: make sure the software satisfies
the specs
Destruction: try to make the software fail
Evaluation: detect faults in early phases
Prevention: prevent faults in early phases
time
18. 18
Testing and the life cycle
requirements engineering
criteria: completeness, consistency, feasibility, and testability.
typical errors: missing, wrong, and extra information
determine testing strategy
generate functional test cases
test specification, through reviews and the like
design
functional and structural tests can be devised on the basis of
the decomposition
the design itself can be tested (against the requirements)
formal verification techniques
the architecture can be evaluated
19. 19
Testing and the life cycle (cnt’d)
implementation
check consistency implementation and previous documents
code-inspection and code-walkthrough
all kinds of functional and structural test techniques
extensive tool support
formal verification techniques
maintenance
regression testing: either retest all, or a more selective retest
20. 20
Test-Driven Development (TDD)
First write the tests, then do the
design/implementation
Part of agile approaches like XP
Supported by tools, eg. JUnit
Is more than a mere test technique; it subsumes
part of the design work
21. 21
Steps of TDD
1. Add a test
2. Run all tests, and see that the system fails
3. Make a small change to make the test work
4. Run all tests again, and see they all run properly
5. Refactor the system to improve its design and
remove redundancies
22. 22
Test Stages
module-unit testing and integration testing
bottom-up versus top-down testing
system testing
acceptance testing
installation testing
23. 23
Test documentation (IEEE 928)
Test plan
Test design specification
Test case specification
Test procedure specification
Test item transmittal report
Test log
Test incident report
Test summary report
24. 24
Overview
Preliminaries
All sorts of test techniques
manual techniques
coverage-based techniques
fault-based techniques
error-based techniques
Comparison of test techniques
Software reliability
25. 25
Manual Test Techniques
static versus dynamic analysis
compiler does a lot of static testing
static test techniques
reading, informal versus peer review
walkthrough and inspections
correctness proofs, e.g., pre-and post-conditions: {P} S {Q}
stepwise abstraction
26. 26
(Fagan) inspection
Going through the code, statement by statement
Team with ~4 members, with specific roles:
moderator: organization, chairperson
code author: silent observer
(two) inspectors, readers: paraphrase the code
Uses checklist of well-known faults
Result: list of problems encountered
27. 27
Example checklist
Wrong use of data: variable not initialized,
dangling pointer, array index out of bounds, …
Faults in declarations: undeclared variable,
variable declared twice, …
Faults in computation: division by zero, mixed-
type expressions, wrong operator priorities, …
Faults in relational expressions: incorrect
Boolean operator, wrong operator priorities, .
Faults in control flow: infinite loops, loops that
execute n-1 or n+1 times instead of n, ...
28. 28
Overview
Preliminaries
All sorts of test techniques
manual techniques
coverage-based techniques
fault-based techniques
error-based techniques
Comparison of test techniques
Software reliability
29. 29
Coverage-based testing
Goodness is determined by the coverage of the
product by the test set so far: e.g., % of
statements or requirements tested
Often based on control-flow graph of the program
Three techniques:
control-flow coverage
data-flow coverage
coverage-based testing of requirements
30. 30
Example of control-flow coverage
procedure bubble (var a: array [1..n] of integer; n: integer);
var i, j: temp: integer;
begin
for i:= 2 to n do
if a[i] >= a[i-1] then goto next endif;
j:= i;
loop: if j <= 1 then goto next endif;
if a[j] >= a[j-1] then goto next endif;
temp:= a[j]; a[j]:= a[j-1]; a[j-1]:= temp; j:= j-1; goto loop;
next: skip;
enddo
end bubble;
input: n=2, a[1] = 5, a[2] = 3
√
√
√
√
√
√
√
√
√
√
√
√
31. 31
Example of control-flow coverage (cnt’d)
procedure bubble (var a: array [1..n] of integer; n: integer);
var i, j: temp: integer;
begin
for i:= 2 to n do
if a[i] >= a[i-1] then goto next endif;
j:= i;
loop: if j <= 1 then goto next endif;
if a[j] >= a[j-1] then goto next endif;
temp:= a[j]; a[j]:= a[j-1]; a[j-1]:= temp; j:= j-1; goto loop;
next: skip;
enddo
end bubble;
input: n=2, a[1] = 5, a[2] = 3
√
√
√
√
√
√
√
√
√
√
√
√
a[i]=a[i-1]
32. 32
Control-flow coverage
This example is about All-Nodes coverage,
statement coverage
A stronger criterion: All-Edges coverage, branch
coverage
Variations exercise all combinations of
elementary predicates in a branch condition
Strongest: All-Paths coverage (≡ exhaustive
testing)
Special case: all linear independent paths, the
cyclomatic number criterion
33. 33
Data-flow coverage
Looks how variables are treated along paths
through the control graph.
Variables are defined when they get a new value.
A definition in statement X is alive in statement Y
if there is a path from X to Y in which this variable
is not defined anew. Such a path is called
definition-clear.
We may now test all definition-clear paths
between each definition and each use of that
definition and each successor of that node: All-
Uses coverage.
34. 34
Coverage-based testing of requirements
Requirements may be represented as graphs,
where the nodes represent elementary
requirements, and the edges represent relations
(like yes/no) between requirements.
And next we may apply the earlier coverage
criteria to this graph
35. 35
Example translation of requirements to a graph
A user may order new
books. He is shown a
screen with fields to fill
in. Certain fields are
mandatory. One field is
used to check whether
the department’s budget
is large enough. If so,
the book is ordered and
the budget reduced
accordingly.
Enter fields
All mandatory
fields there?
Check budget
Order book
Notify
user
Notify user
36. 36
Similarity with Use Case success scenario
1. User fills form
2. Book info checked
3. Dept budget checked
4. Order placed
5. User is informed
Enter fields
All mandatory
fields there?
Check budget
Order book
Notify
user
Notify user
37. 37
Overview
Preliminaries
All sorts of test techniques
manual techniques
coverage-based techniques
fault-based techniques
error-based techniques
Comparison of test techniques
Software reliability
38. 38
Fault-based testing
In coverage-based testing, we take the structure
of the artifact to be tested into account
In fault-based testing, we do not directly consider
this artifact
We just look for a test set with a high ability to
detect faults
Two techniques:
Fault seeding
Mutation testing
40. 40
Mutation testing
procedure insert(a, b, n, x);
begin bool found:= false;
for i:= 1 to n do
if a[i] = x
then found:= true; goto leave endif
enddo;
leave:
if found
then b[i]:= b[i] + 1
else n:= n+1; a[n]:= x; b[n]:= 1
endif
end insert;
2
-
n-1
41. 41
Mutation testing (cnt’d)
procedure insert(a, b, n, x);
begin bool found:= false;
for i:= 1 to n do
if a[i] = x
then found:= true; goto leave endif
enddo;
leave:
if found
then b[i]:= b[i] + 1
else n:= n+1; a[n]:= x; b[n]:= 1
endif
end insert;
n-1
42. 42
How tests are treated by mutants
Let P be the original, and P’ the mutant
Suppose we have two tests:
T1 is a test, which inserts an element that equals a[k] with
k<n
T2 is another test, which inserts an element that does not
equal an element a[k] with k<n
Now P and P’ will behave the same on T1, while
they differ for T2
In some sense, T2 is a “better” test, since it in a
way tests this upper bound of the for-loop, which
T1 does not
43. 43
How to use mutants in testing
If a test produces different results for one of the
mutants, that mutant is said to be dead
If a test set leaves us with many live mutants, that
test set is of low quality
If we have M mutants, and a test set results in D
dead mutants, then the mutation adequacy score
is D/M
A larger mutation adequacy score means a better
test set
44. 44
Strong vs weak mutation testing
Suppose we have a program P with a component
T
We have a mutant T’ of T
Since T is part of P, we then also have a mutant P’
of P
In weak mutation testing, we require that T and T’
produce different results, but P and P’ may still
produce the same results
In strong mutation testing, we require that P and
P’ produce different results
45. 45
Assumptions underlying mutation testing
Competent Programmer Hypothesis: competent
programmers write programs that are
approximately correct
Coupling Effect Hypothesis: tests that reveal
simple fault can also reveal complex faults
46. 46
Overview
Preliminaries
All sorts of test techniques
manual techniques
coverage-based techniques
fault-based techniques
error-based techniques
Comparison of test techniques
Software reliability
47. 47
Error-based testing
Decomposes input (such as requirements) in a
number of subdomains
Tests inputs from each of these subdomains, and
especially points near and just on the boundaries
of these subdomains -- those being the spots
where we tend to make errors
In fact, this is a systematic way of doing what
experienced programmers do: test for 0, 1, nil, etc
48. 48
Error-based testing, example
Example requirement:
Library maintains a list of “hot” books. Each new
book is added to this list. After six months, it is
removed again. Also, if book is more than four
months on the list, and has not been borrowed
more than four times a month, or it is more than
two months old and has been borrowed at most
twice, it is removed from the list.
50. 50
Strategies for error-based testing
An ON point is a point on the border of a subdomain
If a subdomain is open w.r.t. some border, then an
OFF point of that border is a point just inside that
border
If a subdomain is closed w.r.t. some border, then an
OFF point of that border is a point just outside that
border
So the circle on the line age=6 is an ON point of both
A and B
The other circle is an OFF point of both A and B
51. 51
Strategies for error-based testing (cnt’d)
Suppose we have subdomains Di, i=1,..n
Create test set with N test cases for ON points of
each border B of each subdomain Di, and at least
one test case for an OFF point of each border
This set is called N∗1 domain adequate
53. 53
Overview
Preliminaries
All sorts of test techniques
manual techniques
coverage-based techniques
fault-based techniques
error-based techniques
Comparison of test techniques
Software reliability
54. 54
Comparison of test adequacy criteria
Criterion A is stronger than criterion B if, for all
programs P and all test sets T, X-adequacy
implies Y-adequacy
In that sense, e.g., All-Edges is stronger that All-
Nodes coverage (All-Edges “subsumes” All-
Nodes)
One problem: such criteria can only deal with
paths that can be executed (are feasible). So, if
you have dead code, you can never obtain 100%
statement coverage. Sometimes, the subsumes
relation only holds for the feasible version.
56. 56
Experimental results
There is no uniform best test technique
The use of multiple techniques results in the
discovery of more faults
(Fagan) inspections have been found to be very
cost effective
Early attention to testing does pay off
57. 57
Overview
Preliminaries
All sorts of test techniques
manual techniques
coverage-based techniques
fault-based techniques
error-based techniques
Comparison of test techniques
Software reliability
58. 58
Software reliability
Interested in expected number of failures (not
faults)
… in a certain period of time
… of a certain product
… running in a certain environment
60. 60
Failure behavior
Subsequent failures are modeled by a stochastic
process
Failure behavior changes over time (e.g. because
errors are corrected) ⇒ stochastic process is
non-homogeneous
µ(τ) = average number of failures until time τ
λ(τ) = average number of failures at time τ (failure
intensity)
λ(τ) is the derivative of µ(τ)
62. 62
Operational profile
Input results in the execution of a certain sequence
of instructions
Different input ⇒ (probably) different sequence
Input domain can thus be split in a series of
equivalence classes
Set of possible input classes together with their
probabilities
63. 63
Two simple models
Basic execution time model (BM)
Decrease in failure intensity is constant over time
Assumes uniform operational profile
Effectiveness of fault correction is constant over time
Logarithmic Poisson execution time model (LPM)
First failures contribute more to decrease in failure intensity
than later failures
Assumes non-uniform operational profile
Effectiveness of fault correction decreases over time
65. 65
Summary
Do test as early as possible
Testing is a continuous process
Design with testability in mind
Test activities must be carefully planned,
controlled and documented.
No single reliability model performs best
consistently
Editor's Notes
As an exercise, refer the students to the Therac-case, discussed in chapter 1.
It thus really pays off to start testing early. See also following picture
Old picture from Boehm’s book. It shows that errors discovered during operation might cost 100 times as much as errors discovered during requirements engineering.
As a simple illustration of why exhaustive testing does not work: take a simple loop with an if statement in it. Exhaustive testing if the loop is executed 100 times takes 2 100 test cases Random testing does work if you want to achieve reliability (see later sections/slides).
Coverage-based: e.g. how many statements or requirements have been tested so far Fault-based: e.g., how many seeded faults are found Error-based: focus on error-prone points, e.g. off-by-one points Black-box: you do not look inside, but only base yourself on the specification/functional description White-box: you do look inside, to the structure, the actual program/specification. This classification is mostly used at the module level.
For example, I may accidentally assume a procedure is only called with a positive argument (the error). So I forget to test for negative values (the fault). Now if the procedure is actually called with a negative argument, something may go wrong (wrong answer, abortion): the failure Note that the relation between errors, faults and failures need not be 1-1.
But even with this definition, things may be subtle. Suppose a program contains a fault which never shows up, say because a certain piece of the code never gets executed. Is this “latent” fault actually a fault? If not, does it become a fault if we reuse this part of the program in another context? See also next slide.
The Ariane 5 took off and exploded within 40 seconds. Ultimate cause: overflow in conversion of some variable; this case was not tested. In the Ariane 4, this did not cause any problem. This variable related to the horizontal speed of the rocket. The piece of software in question only served to speed up the restart of the launching process in case something went wrong and one had to stop the launch prematurily. The software ran for about a minute after launching. The Ariane 4 is much slower than the Ariane 5, so within this one minute, the rocket was still going up, and the variable in question had a small value. In the Ariane 5, by this time horizontal speed was much higher. So, failure to specify boundary conditions for this software? Reuse failure?
If exhausting testing does not work, we have to select a good subset. But how do we determine the quality of such a test set? This is a very crucial step, and the various test techniques all address this issue in one way or another.
Note that the stopping rule view is a special case of the measurement view. We use these adequacy criteria to decide whether one testing technique is better than another. A number of such relations between test techniques is given later on.
Objective 1 is the kind of objective used in all kinds of functional and structural test techniques. These try to systematically exercise the software so as to make sure we test “everything”. The idea behind objective 2 is that we might not be interested in faults that never show up, but we really want to find those that hae a large probability of manifesting themselves. So we pursue a high reliability. Random testing then works, provided the test cases profile matches the operational profile, I.e. the distribution of test cases mimics actual use of the system. An example development method where this objective is applied is Cleanroom.
This as example approach where we want to find as many faults as possible. The partition is perfect iff the paths in the program follow the equivalence classes chosen. For instance, we assume that the sorting module treat all arrays of length 1 < n < 999 the same. Probably, those of length 1 an 999 are also treated in the same way, but just to make sure we test these boundary cases separately. Now if the sorting program treats, say, arrays with negative numbers differently from those with positive numbers, this equivalence class partitioning is not perfect, and a fault in the program may go unnoticed because we may happen to use a test case that, say, only has positive numbers, and none that has negative numbers.
The first two models are phase models; testing is a phase following coding. The demonstration mode is often used when testing one’s own software. This model also applies when the test set is not carefully/systematically constructed. All kinds of structural and functional techniques follow the destructive mode of operation. The last two models acknowledge that testing is something that has to be done in every development phase. For instance, requirements can be reviewed too. And by making sure that there is a test for every requirement, including every non-functional requirement, you can even prevent errors from being made in the first place. Over the years, a gradual shift can be observed, from demonstration to prevention.
Correctness proofs: complex, not one very often Stepwise abstraction: opposite of stepwise refinement, so you develop pre and post-conditions of a module by working backwards from the individual statements
Much of the real value of this type of technique is in the learning process that the peole get involved in.
In branch coverage, both branches of an if-statement are tested, even if one is empty In normal branch coverage, a combined condition like a = 1 and b = 2 requires two tests. We may also test al four combinations of the two simple predicates. The cyclomatic number criterion is related to the cyclomatic complexity metric of McCabe
We have to include each successor to enforce that all branches following a P-use are taken. Further variations differentiate between uses in a predicate (C-use) and uses elsewhere (computations, C-use). This leads to criteria like All-C-uses/Some-P-uses and the like.
Kinds of variations in program testing: seed faults founded by one group into the program tested by another group.
In each variation, mutant, one simple change is made.
Note that if we happen to insert an element of a that occurs before the final element, we won’t notice a difference
This is a graphical view of this same requirement. It shows the two-dimensional (age, average number of loans per month) domain. The subdomains are bordered by lnes such as age=6, or (age=4, 0<= av<= 5). For each border, it is indicated which of the adjacent subdomains is closed by putting a hachure at that side; a subdomain is closed at some border iff that border belongs to the subdomain; otherwise it is open.
This yields the same picture, with the same borders, and can be used with the same test set.
Usually, stronger metrics induce more costs
These properties relate to program-based criteria. The first four are rather general and should apply to any test adequacy criterion. E.G. the applicability criterion says:for every program, there is an adequate test set. This is not true for the All-Nodes and Al-Edges criteria, e.g., since they may have dead code, so that you cannot achieve 100% coverage. Anticomposition: if components have been tested adequately, this does not mean their composition is also tested adequately (re Ariane 5 disaster). This does not hold for the All-Nodes and All-Edges.