Modern Software Testing Techniques: A Practical Guide for Developers and Testers

Congratulations on purchasing this ebook!
Here's an exclusive list of the Top Crypto Exchanges for you to consider when
registering. These platforms offer excellent opportunities to dive into the exciting
world of cryptocurrencies. Check out the shortened links below and get started
with ease!
1. Binance: the world's leading cryptocurrency exchange, offers a wide range of
trading options and low fees. Registration link: https://mas.so/binance
2. OKX: a popular choice for derivatives trading, provides advanced features
and high liquidity. Registration link: https://mas.so/okx
3. Bybit: known for its copy trading capabilities, allows users to follow
successful traders. Registration link: https://mas.so/bybit
4. Coinbase: a user-friendly US-based exchange, prioritizes security and
compliance. Registration link: https://mas.so/coinbase
5. WhiteBIT: a rapidly growing European exchange, offers a diverse selection of
trading pairs. Registration link: https://mas.so/whitebit
6. Bitforex: a reputable Asian exchange, provides competitive fees and margin
trading. Registration link: https://mas.so/bitforex
7. Bitmart: a popular platform for altcoin trading, offers staking opportunities.
Registration link: https://mas.so/bitmart
8. Gate: an established exchange with advanced features, supports margin
lending and borrowing. Registration link: https://mas.so/gate
9. MEXC: known for its fast matching engine and deep liquidity, caters to
experienced traders. Registration link: https://mas.so/mexc
10. Hotcoin: a user-friendly platform with a focus on emerging markets, offers a
seamless trading experience. Registration link: https://mas.so/hotcoin
11. Huobi (HTX): a renowned Asian exchange, provides diverse trading options
and high liquidity. Registration link: https://mas.so/htx
12. Digifinex: a rising star in the crypto world, offers competitive fees and a
growing user base. Registration link: https://mas.so/digi
Don't wait any longer! Take advantage of these opportunities and embark on
your crypto journey with confidence. Remember, always do your research and
choose the exchange that best suits your needs. Happy trading!
*Disclaimer: All links are provided for convenience and are not endorsed or affili-
ated with this ebook. Users should exercise due diligence and caution when par-
ticipating in any crypto-related activities.*

1
© István Forgács and Attila Kovács 2024
I. Forgács and A. Kovács, Modern Software Testing Techniques,
https://doi.org/10.1007/978-1-4842-9893-0_1
CHAPTER 1
Software Testing
Basics
This chapter overviews the basics of software testing from the point of view
of bugs: lifetime, classifications, pursuing processes, and various pesticides
against bugs.
Estimated Time
• Beginners: 100 minutes.
• Intermediates: 80 minutes.
• Experts: We suggest reading sections “Pesticided
Against Bugs”
, “Classification of bugs”
, “Fault-based
testing”
, and “Testing principles”; the rest may be
skipped. It takes 30 minutes.

Bugs and Other Software Quality Destroyers
Bugs and other software quality destroyers refer to issues or factors that
can negatively impact the quality, reliability, and performance of software.
To mitigate these software quality destroyers, it’s crucial to follow best

2
practices in software development, including thorough testing, proper
design and architecture, effective documentation, security considerations,
performance optimization, user-centric design, and ongoing maintenance
and updates. This section overviews the quality from the perspective of
the bugs.

Lifetime of Bugs: From Cradle to Coffin
Software is implemented by developers or generated by models designed
by software engineers. Non-considering AI created software, the developer
is the one who creates the bugs. Why? There can be many reasons.
First, because developers are humans. Successful and scalable software
products need professional architecting, designing, and coding. There are
many places and reasons where and when bugs can arise. Professional
developers need to have programming skills, need to know architecture
and design patterns, and need to have some domain knowledge and skills
for handling databases, networks, hardware architectures, algorithms, etc.
Note that the developers build up the quality of the software. The testers
support the developers via quality control.
At present, software is produced mainly manually with the help of
some artificial intelligence applications. One of the most important levels
for building an application is the programming language level. There are
plenty of programming languages. Note that in each situation, there can
be many factors determining the “best” programming language. Complex
applications require applying more programming languages, frameworks,
and libraries together.
Programming language paradigms can be imperative or declarative.
In the first case, the developer focuses on how the underlying machine
will execute statements. Programs define control flow and usually the
way how the program states are changed. Imperative programming
Chapter 1 Software Testing Basics

3
is a programming paradigm that uses statements that change the
program’s states. “Declarative programming is a style of building the
structure and elements of computer programs that expresses the logic
of a computation without describing its control flow” (Lloyd 1994). The
imperative classification can be broken down into multiple paradigms
such as structured, procedural, and object-oriented (however, there are
declarative object-oriented languages like Visual Prolog or QML), while
declarative programming is an umbrella term of constraint, functional,
and logic programming including domain-specific languages. Other
paradigms, orthogonal to the imperative or declarative classification, may
include concurrent and generative programming. Independently from the
chosen language, any programming model describes some kind of logic
and data.
This book does not contain the details of programming (procedures,
objects, classes, type systems, generics, pattern matching, etc.); however,
we use some abstract programming elements (pseudocode) and
sometimes Python.
An error is a mistake, misconception, or misunderstanding during the
SDLC. Errors can arise in different places: in the requirement specification,
in the architecture, in the code, in the data structure, in the documents,
etc. Due to different errors, software bugs are inserted into the code.
“A software bug is a flaw or fault in a computer program or system
that causes it to produce an incorrect or unexpected result, or to behave in
unintended ways” (Wikipedia). Note that we use the terms “bug,” “defect,”
and “fault” interchangeably according to the traditional phrasing in the
literature. Bugs affect program functionality and may result in incorrect
output, referred to as failure. Later in this chapter, we overview the existing
bug classification and describe a new one. Bugs can arise in different
software life cycle phases. Bugs can celebrate their dawn as a result of
faulty requirements analysis, software design, coding, testing, and even
erroneous maintenance. Bugs can slip through the quality gate of unit,

4
integration, system, or acceptance testing. Bugs can survive the regression
testing, and new bugs may arise during any change or correction.
Developers and testers share the common goal of reducing the occurrence
of bugs and, if any exist, detecting them through the creation of “killer”
tests before the software release.
Technically, the very first step to avoiding the rise of bugs is to
surmount the chosen programming language. Second, the requirements
must be well-defined, unambiguous, complete, and well-understood.
Hence, developers (and testers) need to understand the requirements
(business, product, process, transition, etc.) and be able to understand the
various requirements models. In the following, we overview the possible
pesticides against software bugs.

Pesticides Against Bugs
Unfortunately, the unified theory of testing does not exist and will not
exist. A good SDLC includes various activities to minimize the bugs such as
defect prevention, detection, and correction.
In the early phases of implementing a system, subsystem, or feature,
the business rules and requirements that are incomplete or ambiguous will
lead to defects during development. Complex code extended or modified
many times without refactoring will also lead to avoidable defects.
Some people think that defect prevention is bullshit. “How can we
prevent bugs that are already there?” Well, defect prevention is a process
that emphasizes the need for early staged quality feedback for avoiding
defects in the later software products. It stresses the need for quality gates
and reviews and encourages learning from the previous defects. In other
words, defect prevention is a quality improvement process aiming at
identifying common causes of defects and changing the relevant processes
to prevent reoccurrence. The common process of defect prevention is
(1) classifying and analyzing the identified defects, (2) determining and

5
analyzing the root causes, and (3) feedback on the results for process
improvement (see Forgács et al. 2019). There are numerous techniques for
preventing bugs:
• Apply specification or test models.
• Apply specification by examples.
• Apply test-first methods.
• Manage complexity by divide and conquer.
• Apply the right form of reviews.
• Apply checklists.
• Apply automatic static analyzers.
• Refactor the code.
The first two are related to improving the requirements, the others
are related to improving the code, moreover, the third one improves both.
The first four are “real” preventions as they happen before coding, the
others just before testing. Refactoring is a common prevention technique
used during maintenance. Defect prevention is a cheap solution while
defect correction is more expensive. That’s why defect prevention is valid;
moreover, it is obligatory. Clearly, the main target of any quality assurance
task is to prevent defects.
Fault detection can be made ad hoc and can be semistructured
or structured. The most common structured ways are the black-box
(specification-based) and white-box (structure-based) methods.
In black-box testing, the tester doesn’t need to be aware of how the
software has been implemented, and in many cases, the software source
is not even available. Equivalently, the tester knows only the specification.
What matters is whether the functionality follows the specification or
not. Black-box testing usually consists of functional tests where the tester
enters the input parameters and checks whether the application behaves

6
correctly and properly handles normal and abnormal events. The most
important step for producing test cases in black-box testing is called
test design.
In contrast, white-box testing is performed based on the structure of
the test object, or more specifically, the tester knows and understands
the code structure of the program. Regarding the code, white-box testing
can be done by testers, but it’s more often done by the developers on their
own. The process of producing white-box testing is called test creation (test
generation).
We note that both black-box and white-box testing can be applied at
any test level. At the unit level, a set of methods or functions implementing
a single functionality should be tested against the specification. At the
system or acceptance level, the whole application is tested. Black-box
testing can be successfully performed without programming knowledge
(however, domain knowledge is an advantage), but white-box testing
requires a certain level of technical knowledge and developers’
involvement. Writing automated test code for black-box and white-box
testing needs programming skills. Nowadays, codeless test automation
tools allow any tester to automate tests. There are plenty of tools
supporting test automation (both free and proprietary). Test automation is
essential for continuous integration and DevOps.
When testing is performed based on the tester’s experience, in the ad
hoc case, we speak about error guessing; in the semistructured case, we
speak about exploratory testing. A special case of the latter is called session-
based testing (Bach 2000).
Besides the mentioned software testing types, there are various
methods for fault detection, such as graph-based approaches (searching
for erroneous control flow dynamic), classifiers (based on machine
learning or Bayesian aiming at identifying abnormal events), and
data-trace pattern analyzers. But none of these methods have been proven
to be efficient in practice yet (however, in some “limited” situations, they
can be applied).

7
In this book, we primarily focus on the first and most important step
in the fight against bugs: test design. We consider test design as a defect
prevention strategy. Considering the “official” definition, test design is the
“activity of deriving and specifying test cases from test conditions,” where
a test condition is a “test aspect of a component or system identified as a
basis for testing.” Going forward, the test basis is “the body of knowledge
used as the basis for test analysis and design.”
Let’s make it clearer. Requirements or user stories with acceptance
criteria determine what you should test (test objects and test conditions),
and from this, you have to figure out the way of testing; that is, design the
test cases.
One of the most important questions is the following: what are the
requirements and prerequisites of successful test design? If you read
different blogs, articles, or books, you will find the following:
• The time and budget that are available for testing
• Appropriate knowledge and experience of the people
involved
• The target coverage level (measuring the
confidence level)
• The way the software development process is organized
(for instance, waterfall vs. agile)
• The ratio of the test execution methods (e.g., manual vs.
automated), etc.
Do you agree? If you don’t have enough time or money, then you will
not design the tests. If there is no testing experience, then no design is
needed, because “it doesn’t matter anyway.” Does everyone mean the
same thing when they use the terms “coverage” and “confidence level”? If
you are agile, you don’t need to spend time designing tests anymore. Is it
not necessary to design, maintain, and then redesign automated tests?

8
We rather believe that good test design involves three prerequisites:
1. Complete specification (clear and managed
test bases)
2. Risk and complexity analysis
3. Historical data of your previous developments
Some explanation is needed. A complete specification unfortunately
doesn’t mean error-free specification and during test design, lots of
problems can be found and fixed (defect prevention). It only means that
we have all the necessary requirements, or in agile development, we have
all the epics, themes, and user stories with acceptance criteria.
We have that there is an optimum value to be gained if we consider
the testing costs and the defect correcting costs together (see Figure 1-2),
and the goal of good test design is to select appropriate testing techniques
that will approach this optimum. This can be achieved by complexity and
risk analysis and using historical data. Thus, risk analysis is inevitable
to define the thoroughness of testing. The more risk the usage of the
function/object has, the more thorough the testing that is needed. The
same can be said for code complexity. For more risky or complex code,
we should first apply more linear test design techniques instead of a
single combinatorial one.
Our (we think proper) view on test design is that if you have the
appropriate specification (test basis) and reliable risk and complexity
analysis, then knowing the historical data, you can optimally perform
test design. At the beginning of your project, you have no historical data,
and you will probably not reach the optimum. It is no problem, make
an initial assessment. For example, if the risk and complexity are low,
then use only exploratory testing. If they are a little bit higher, then use
exploratory testing and simple specification-based techniques such
as equivalence partitioning with boundary value analysis. If the risk is

9
high, you can use exploratory testing, combinative testing, state-based
testing, defect prevention, static analysis, and reviews. We note, however,
that regardless of the applied development processes or automation
strategies, for given requirements, you should design the same tests.
This remains valid even for exploratory testing as you can apply it in
arbitrary models.
Have you ever thought about why test design is possible at all? Every
tester knows that lots of bugs can be found by applying appropriate test
design techniques though the number of test cases is negligible compared
to all the possible test cases. The reason is the Coupling Effect hypotheses.
This hypothesis states that a test set that can detect the presence of
single faults in the implementation is also likely to detect the presence of
multiple faults. Thus, we only have to test the application to separate it
from the alternative specifications which are very close to the one being
implemented (see section “Fault-Based Testing”).
Classification of Bugs
Software faults can be classified into various categories based on their
nature and characteristics.
Almost 50 years ago, Howden (1976) published his famous paper
“Reliability of the path analysis testing strategy.” He showed that
“there is no procedure which, given an arbitrary program P and output
specification, will produce a nonempty finite test set T, subset of the input
domain D, such that if P is correct on T, then P is correct on all of D. The
reason behind this result is that the nonexistent procedure is expected to
work for all programs, and thus, the familiar noncomputability limitations
are encountered.” What does it mean? In simpler terms, the sad reality is

10
that, apart from exhaustive testing, there is no universal method to create a
reliable test set that guarantees finding all bugs for all programs. Therefore,
it is impossible to definitively state that all bugs have been discovered after
testing. However, this does not mean that testing should be neglected as
it is still possible to find most of the bugs. Howden introduced a simple
software fault classification scheme. According to his classification, three
types of faults exist:
• Computation fault: This type of fault relates to errors
or faults in calculations or computations performed
by the implementation. It encompasses issues such as
incorrect arithmetic operations, mathematical errors,
or flaws in algorithmic implementations.
• Domain fault: Domain faults involve faults in the
control flow or logic of the implementation such as
problems with loops, conditionals, or branching,
resulting in incorrect control flow, unintended
behavior, or faulty decision-making.
• Subcase fault: Subcase faults refer to situations where
something is missing or not properly implemented
within the software. This can include missing or
incomplete functionality, unhandled edge cases, or
gaps in the implementation that lead to incorrect or
unexpected behavior.
However, this classification is based on the implemented code, but
when we design test cases, we do not have any code. Thus, we should start
from the functional specification/requirements. The requirements should
consist of two main elements:
1. What the system should do.
2. In which conditions the systems should do that.

11
The system’s computation represents what it should do, while
the conditions under which the computation occurs fall within the
domain of the given computation. Both components are susceptible to
implementation errors. Therefore, when considering requirements, we
encounter two types of errors:
1. Domain error
2. Computation error
The only distinction concerning Howden’s classification is that the
subcase error is nonexistent from the specification’s perspective since
the specification should encompass everything to be implemented. If
something is missing, it is not a subcase error but rather a requirement for
incompleteness, which can be addressed using defect prevention methods
as discussed in the section “Pesticides Against Bugs.” A comprehensive
specification includes all the conditions the system should fulfill, resulting
in test cases for each specific domain. These test cases thoroughly examine
potential subcase errors. If a predicate is missing or not implemented, the
related test case follows a different path, leading to a faulty computation,
which is then manifested as a computation error (excluding coincidental
correctness, as discussed in the next paragraph). Therefore, we can
consider this situation as a computation error as well.
Therefore, we are left with only these two types of errors, and based
on them, we can enhance our test design. In this book, we introduce one
test design technique for computation errors and another for domain
errors. Our technique for detecting domain errors is weak-reliable,
meaning that the input value used to identify the error is “one dimension
higher” than the one where the bug remains hidden. The reason for this
is that even if the code follows an incorrect control flow, the computation
may still yield the same result for certain inputs. This phenomenon
is known as coincidental correctness. For instance, if the correct path
involves the computation y = y * x and the incorrect path has y = y + x,

12
when both y and x are equal to 2, the result will be 4 for both paths. Our
technique for finding the computation errors is not (weak) reliable;
however, in practice, it can find most of the bugs.
Software Testing

Testing Life Cycle
This subsection is a review. If you are an experienced software tester, you
can skip it, except “Test Analysis”
. If you are a developer, we suggest reading
it to get acquainted with the viewpoints and tasks of a tester.
You cannot design your tests if you don’t understand the whole test
process. We mentioned that the selected test design techniques strongly
depend on the results of the risk analysis. Similarly, test creation at the
implementation phase is an extension of the test design. Figure 1-1 shows
the relevant entities of the traditional testing life cycle including the test
design activities.

13
Figure 1-1. The testing life cycle
Test Planning
The test planning process determines the scope, objective, approach,
resources, and schedule of the intended test activities. During test
planning – among others – the test objectives, test items, features to be
tested, the testing tasks, the test approach, human and other resources, the
degree of tester independence, the test environment, entry and exit criteria
to be used, and any risks requiring contingency planning are identified.
A test policy defines the testing philosophy and the goals that the
organization wishes to achieve through testing activities, selecting the
frames that testing parties should adhere to and follow. It should apply to
both new projects and maintenance work.

14
The purpose of the corporate test strategy is to standardize and simplify
the creation of test plans and test schedules, gather best practices, and
provide them for future projects. The project test strategy is defined as a set
of guiding principles that exposes the test design and regulates how testing
would be carried out.
The test approach defines how (in which way) testing is carried out,
that is, how to implement the project test strategy. It can be proactive – that
is, the test process is initiated as early as possible to find and fix the faults
before the build (preferable, if possible) – or reactive, which means that
the quality control process begins after the implementation is finished.
The test approach can be of different (not necessarily disjoint) types, such
as specification-based, structure-based, experience-based, model-based,
risk-based, script-based, fault-based, defect-based, standard-compliant,
test-first, etc., or a combination of them.

Test Monitoring and Control
Test monitoring is an ongoing comparison of the actual and the planned
progress. Test control involves the actions necessary to meet the objectives
of the plan.
Although monitoring and control are activities that belong to the test
manager, it’s important to ensure that the appropriate data/metrics from
the test design activities are collected, validated, and communicated.
Test Analysis
The test engineering activity in the fundamental test process begins mainly
with the test analysis. Test analysis is the process of looking at something that
can be used to derive quality information for the software product. The test
analysis process is based on appropriate project documents or knowledge,
called the test basis, on which the tests are based. The most important thing
in test analysis is the opportunity to better understand the problem we are
working on and to anticipate possible problems that may occur in the future.

15
The test analysis phase has three main steps before the review:
1. The first step is to analyze the test basis thoroughly:
Business requirement documents, system
requirement documents, functional design
specifications, technical specifications, user manual,
source code, etc.
2. The second step is to identify the test objects
(features, scenarios) and conditions by defining what
should be tested.
A test condition is a statement referring to the test
object, which can be true or false. Test conditions
can be stated for any part of a component (or
system) that could be verified by some tests, for
example, for a function, transaction, feature, quality
attribute, or structural element.
3. The third step of the test analysis phase is risk
analysis. For each elicited (mainly high level) test
object, the risk analysis process determines and
records the following risk attributes:
• The impact of malfunctioning (how important the
appropriate functioning is)
• The likelihood of malfunctioning (how likely it is
to fail)
Based on the preceding attributes, various risk-scoring techniques and
risk scales exist. The risk level is usually determined by the multiples of the
computed scores.
The question is why risk analysis is necessary?

16
The cost of projects is strongly influenced by two factors. One of them
is the cost of testing (designing and executing the tests, building the testing
environment, etc.), and the other is the cost of defect correction (through
the SDLC).
The cost of testing increases when we want to find more bugs. In the
very beginning, the increase is approximately linear; that is, by executing
twice as many test cases, the number of detected bugs will be doubled.
However, it shortly becomes over-linear. The reason is that after a certain
level, we need to combine (and test) the elements of the input domain by
combinatorial methods. For example, we first test some partitions, then
their boundaries, then pairs or triples of boundary values, etc. Suppose
that the number of test cases is not negligible in the project. There is no
reason to design and execute too many tests since we are unable to find
twice as many bugs with twice as many tests.
The other factor is the cost of defect correction (bug fixing). The later
we find a bug in the software life cycle, the more costly its correction.
According to some publications, the correction cost is “exponential in
time” but clearly over-linear, most likely showing polynomial growth.
Therefore, if we can find faults early enough in the life cycle, then the
total correcting costs can drastically be reduced. As you can see in
Figure 1-2, considering these two factors together, the total cost has an
optimum value.

17
Figure 1-2. Optimum of the correcting cost and the testing cost
This means that it’s not a good strategy to reduce testing and testing
costs as you should consider these two factors together. The main question
is the following. Does this cost optimum result in an acceptable production
code quality? Suppose that you reached the optimum, yet the quality of
the code is not acceptable. Then you should correct some annoying bugs
by which the total correcting cost increases. You do not have any choice
as you should fix the bugs until the quality of the code is acceptable;
otherwise, your company will be negatively impacted. Consequently, if
you reach the optimum, then your code is fine. However, reaching this
optimum is difficult.
Now we can answer the question of why risk analysis is needed. Let’s
consider two implemented functions and assume that the first one is
more complex (in some aspect). Hence, it contains faults with a higher
probability. If we test both at the same expense and thoroughness, then
after testing, more faults will be left in the first code. It means that the total
correcting cost will be higher.
We can use similar arguments for other test objects. Suppose that
we have two functions having the same complexity and testing cost (the
distribution of faults is similar), but the first one is riskier. For example,
this function is used more often, or the function is used in a “critical”

18
flow. Therefore, more bugs will be detected and fixed in this function with
a higher correcting cost during the life cycle. Note that we are speaking
about the detection of bugs here, not about the presence of bugs.
Roughly speaking, more complex code and higher risk raise the
bug-fixing costs if the testing effort/cost remains unchanged. Since this
additional bug fixing occurs later in the life cycle, the code quality at
release will be poorer. Therefore, it is important that a riskier or more
complex code part should be tested with more thoroughness, that is, with
higher testing costs. This higher testing cost is inevitable to achieve the
optimal total cost as shown in Figure 1-3.
Figure 1-3. Converging to the optimal cost. The total cost optimum
of a high-risk function (HRF) is above the total cost optimum of a
low-risk function (LRF). However, this optimum can be reached via
spending more money on testing (multiple designs, etc.)
It means that for riskier or more complex code, you should select
stronger test design techniques. OK, but how can you reach this optimum?
This is not an easy task, and we have no complete answer. But we do
think this is possible. First, risk and complexity analyses must be done for
each software element, and the assessed risk and complexity data should
be stored. Second, the company should collect and store all the testing

19
and defect correction costs. Third, the company must define and apply
different strategies for optimizing the costs based on the measured data on
different levels of risks and complexities. This selection can be validated
and improved periodically to approach the optimum. We suggest building
the proposed model into the company’s process improvement strategy.
Finally, you should apply the most efficient techniques. Don’t use
combinatorial testing unless a cheaper choice is appropriate. Use more
techniques in parallel as they can find different bugs. A good solution
for risky code is to apply use case testing or state-based testing with
“cheap” test selection criterion, equivalence partitioning, boundary value
analysis, and combinative testing together. They are not expensive, and
you will find most of the bugs.
In the following two chapters, we introduce new test design techniques
that shift the optimum right and down. The optimum strongly depends on
the applied test design techniques.
Test Design
Test design can be performed in seven steps (see Figure 1-1):
1. First, the technical test approaches are worked out,
which means that it should be planned how the
product quality together with the cost optimization
goals can be achieved based on risk analysis (see the
previous section).
2. After working out the test approach, the test design
techniques are selected that meet the (1) testing
objectives and (2) the result of risk and complexity
analysis. In general, it is advisable to select test design
techniques understandable by other stakeholders
and supported by the test design automation tools of
the organization. Real systems usually require using
more techniques in combination.

20
3. The next step is to determine the test case selection
criteria (for simplicity, we refer to this as test selection
criteria). The test selection criteria determine when
to stop designing more test cases or how many test
cases must be designed for a given situation. The
optimal test selection criterion for the boundary value
analysis, for example, results in designing four test
cases for each linear border. However, usually, it is
possible to define different (“weaker” or “stronger”)
test selection criteria for a given condition and design
technique. We note that determining the test selection
criteria is not always necessary as sometimes there is
a unique optimum (see Chapter 3).
There is an important notion closely related to test
selection criteria, namely, the test data adequacy
criteria. While test selection criteria are defined
regarding the test design, that is, independently of
the implementation, test data adequacy criteria
are defined regarding program execution. Both
test selection and test data adequacy criteria,
however, provide a way to define a notion of
“thoroughness” for test case sets. By applying
these criteria, we can check whether our test set is
adequate, and no additional testing is needed. We
can call both criteria “adequacy criteria concerning
thoroughness.”
It is important to note that some authors define
test selection coverage as being simply the same
as code coverage. However, although test selection
coverage and code coverage are both useful to
assess the quality of the application code, code

21
coverage is a term to describe which application
code is exercised when the test is running, while test
selection coverage relates to the percentage of the
designed test cases concerning the requirements
in the selection criterion. Please keep in mind that
neither test selection nor test data adequacy criteria
tell anything about the quality of the software, only
about the expected quality of the test cases. When
deriving the test cases, be aware that one test case
may exercise more than one test condition, and
thus, there is the opportunity to optimize the test
case selection by combining multiple test coverage
items in a single test case.
There is another important notion related to test
selection. In a multivariate domain, we can apply
different fault models to predict the consequences
of faults. The single fault assumption relies on the
statistic that failures are only rarely the product of
two or more simultaneous faults. Here we assume
that fault in a program occurs due to a single
program statement (or the absence of it). On the
contrary, the multiple fault assumption means that
more than one component leads to the cause of the
problem. Here we assume that fault in the program
occurs due to the values of more than one variable.
Single fault assumption also means that we test one
input domain by one test case only, and if it fails,
then we know the location of the fault. By applying
the multiple fault assumption, we can design fewer
test cases; however, in case of a failure, we should
make additional test cases for bug localization.
There is no rule for which method is better.

22
Finally, in this book, we consider the following
prerequisites for black-box test design: (1) determine
the risk level and complexity; (2) determine the
applicable library elements and other applicable
artifacts (do not forget the traces); (3) based on
the first two steps, determine the defensiveness of
testing, the fault localization model, and the test
design techniques to be used; and (4) adopt the
testing process and the automated test cases into the
CI chain. For risk-level determination, see Forgács
and Kovács (2019). Test defensiveness means the
thoroughness of the input validation.
A test design technique can be reliable or unreliable
for a defect. A test design technique is reliable for
a bug, if for any test set fulfilling the test design
technique, there is at least one error-revealing test
case. Executing an error-
revealing test case means
that there is at least one program point where the
intermediate control/state/value of some variable
is faulty (excluding coincidental correctness). For
example, assume that a bug where the correct
predicate x 200 is erroneously programmed as
x ≥ 200 and is tested by BVA. Here BVA is reliable
as it should contain a test x = 200 and the program
follows a wrong path. Note that this doesn’t mean
that the bug will be detected as the wrong value may
not reach a validated output.

23
4. The next step is to establish the test data. Test data
are used to execute the tests and can be generated
by testers or by any appropriate automation tool
(can be produced systematically or by using
randomization models, simulators, emulators). Test
data may be recorded for reuse (e.g., in automated
regression testing) or maybe thrown away after
usage (e.g., in error guessing).
The time, cost, and effectiveness of producing
adequate test data are extremely important. Some
data may be used for positive and others for
negative testing. Typically, test data are created
together with the test case they are intended to be
used for. Test data can be generated manually by
copying from production or legacy sources into the
test environment or by using automated test data
generation tools. Test data creation might take many
pre-steps or test very time-consuming environment
configurations. Note that concrete test cases may
take a longer time to create and may require a lot
of maintenance. Note as well that test data (both
input and environment data) are crucial for the
reproducibility of the tests.
5. Now we can finalize the test case design. A test case
template contains a test case ID, a trace mapped
to the respective test condition, test case name,
test case description, precondition, postcondition,
dependencies, test data, test steps, environment
description, expected result, actual result, priority,
status, expected average running time, comments,
etc. For some software, it may be difficult to

24
compute the proper outcome of a test case. In
these cases, test oracles (sources of information for
determining whether a test has passed or failed) are
useful. Test management tools are a great help to
manage test cases.
Where appropriate, the test cases should be recorded
in the test case specification document. In this
case, the traceability between the test basis, feature
sets, test conditions, test coverage items, and test
cases should be explicitly described. It is advisable
here that the content of the test case specification
document should be approved by the stakeholders.
A test suite (or test set) is a collection of test cases
or test procedures that are used to test software to
show that it fulfills the specified set of behaviors. It
contains detailed instructions for each set of test
cases and information on the system configuration.
Test suites are executed in specific test cycles. In this
book, we use a simplified notation for a test case,
for example, TC = ([1, “a”
, TRUE]; 99), where the test
case TC here has three input data parameters 1, “a”
,
and TRUE and expected value of 99. Sometimes,
when the expected outcome is irrelevant concerning
a given example, we omit it from the test case. The
notation TS = {([1, “a”
, TRUE]; 17), ([2, “b”
, FALSE];
35)} means that the test suite TS contains two
test cases.
6. The next step is to design the test environment. The
test environment consists of items that support test
execution with software, hardware, and network
configuration. The test environment design is

25
based on entities like test data, network, storage,
servers, and middleware. The test environment
management has organizational and procedural
aspects: designing, building, provisioning, and
cleaning up test environments requires a well-
established organization.
7. Finally, all the important artifacts produced during
the test design should be validated (by reviews)
including the control of the existence of the
bidirectional traceability between the test basis, test
conditions, test cases, and procedures.

Test Implementation and Execution
During implementation and execution, the designed test cases are
implemented and executed. After implementing the test cases, the test
scripts are developed. A test script (or test procedure specification) is a
document specifying a sequence of actions and data needed to carry out
a test. A script typically has steps that describe how to use the application
(which items to select from a menu, which buttons to press, and in which
order) to perform an action on the test object. In some sense, a test script is
an extended test case with implementation details.
When test automation is determined to be a useful option, this stage
also contains implementing the automation scripts. Test execution
automation is useful when the project is long term and repeated regression
testing provides a positive cost-benefit.
The next step is to set up the test environment. In some projects (that
lack constraints imposed by prior work, like brand-new or legacy projects),
extra time should be allowed for experiencing and learning. Even in a stable
environment, organizing communication channels and communicating
security issues and software evolution are challenging and require time.

26
Next, the finalization of the approach comes via test case
implementation. At this point, everything is prepared for starting the
execution, manually and/or automatically, which can be checked with the
entry criteria. Some documents may help to localize the items that will be
tested (test item transmittal documents or release notes).
The existence of a continuous integration environment is important
for test execution (depending on the SDLC and the type of project
and product). The execution process is iterative, and in many cases, it
is the longest step of the fundamental test process (design once, run
many times).
Test Closure
The test closure is a complete test report that gives a summary of the test
project. It formally closes the project, collates all the test results, provides
a detailed analysis, presents metrics to clients, and adjudicates the risks
concerning the software.
Fault-Based Testing
Both developers and testers need technical and nontechnical skills.
Moreover, those skills are equally important. To get and control their
technical skills, however, developers can exercise coding on many different
online platforms. There are tasks at various levels, and the courses involve
hints and help when you need it. On the other hand, by implementing the
code, the developer can easily check the expected result, that is, comparing
the output of the code with the requirements.
This is not true for testing. In some sense, testing is more difficult than
coding as validating the efficiency of the test cases (i.e., the goodness of
your tests) is much harder than validating code correctness. In practice,
the tests are just executed without any validation (with just a few
counterexamples, see Kovács and Szabados 2016). On the contrary, the

27
code is (hopefully) always validated by testing. By designing and executing
the test cases, the result is that some tests have passed and some others
have failed. Testers know nothing about how many bugs remain in the
code, nothing about their bug-revealing efficiency.
Unfortunately, there are no online courses available where after
entering the test cases, the test course platform tells which tests are
missing (if any) and why. Thus, testers cannot measure their technical
abilities (ISTQB offers to measure nontechnical skills only). The
consequence is that testers have only a vague assumption about their test
design efficiency.
That was the reason for establishing the platform https://test-
design.org/practical-exercises/, where testers can check their
technical knowledge and can improve it. This platform also helped the
authors introduce a new test design technique, referred to as action-state
testing.
Fortunately, the efficiency of the tests can be measured. Let us assume
that you want to know how efficient your test cases are. You can insert
100 artificial yet realistic bugs into your application. If the test cases find
80 bugs, then you can think that the test case efficiency is about 80%.
Unfortunately, the bugs influence each other; that is, a bug can suppress
some others. Therefore, you should make 100 alternative applications
with a single-seeded bug in each, then execute them. Now, if you find 80
artificial bugs, your efficiency is close to 80% if the bugs are realistic. This is
the way how mutation testing, a form of fault-based testing works.
This is the basic concept of fault-based testing, that is, selecting test
cases that would distinguish the program under test from alternative
programs that contain hypothetical faults. If the program code contains a
fault, then executing it, the output (behavior) must be different. Therefore,
to be able to distinguish the correct code from all its alternatives, test cases
should be designed in a way that some output be different with respect to
the correct code and all its faulty alternatives. Each alternative is a textual
modification of the code. However, there is an unmanageable number of

28
alternatives, and thus we cannot validate the “goodness” of our test set.
Fortunately, it was shown that by testing a certain restricted class of faults,
a wide class of faults can also be found (Offutt 1992). The set of faults
is commonly restricted by two principles: the competent programmer
hypothesis and the coupling effect. We suggest reading a nice survey on
mutation testing (Jia and Harman 2011).
The competent programmer hypothesis (CPH) was introduced by
(Hamlet 1977) and (DeMillo et al. 1978), who observed that “Programmers
have one great advantage that is almost never exploited: they create
programs that are close to being correct.” Developers do not implement
software randomly. They start from a specification, and the software will
be very similar to their expectations, hence, close to the specification.
Coupling effect hypothesis means that complex faults are coupled to
simple faults in such a way that a test data set detecting all simple faults
in a program will detect a high percentage of the complex faults as well.
A simple fault is a fault that can be fixed by making a single change to a
source statement. A complex fault is a fault that cannot be fixed by making
a single change to a source statement. If this hypothesis holds, then it is
enough to consider simple faults, that is, faults where the correct code is
modified (mutated) by a single change (mutation operator). Thus, we can
apply mutation testing to get an efficient test design technique with an
appropriate test selection criterion.
As mentioned, mutation testing is the most common form of fault-
based testing, in which by slightly modifying the original code, we create
several mutants. A reliable test data set should then differentiate the
original code from the well-selected mutants. In mutation testing, we
introduce faults into the code to see the reliability of our test design.
Therefore, mutation testing is actually not testing, but “testing the tests.”
A reliable test data set must “kill” all of them. A test kills a mutant if the
original code and the mutant behave differently. For example, if the code
is y = x and the mutant is y = 2 * x, then a test case x = 0 does not kill the
mutant while x = 1 does.

29
If a mutant hasn’t been killed, then the reasons can be the following:
1. Our test case set is not good enough, and we should
add a test that will kill the mutant.
2. The mutant is equivalent to the original code. It’s
not an easy task to decide the equivalence. And
what is more, the problem is undecidable in the
worst case.
In the case of a first-order mutant, the code is modified in one place. In
the case of a second-order mutant, the code is modified in two places, and
during the execution, both modifications will be executed. Offutt showed
that the coupling effect holds for first and second-order mutants; that
is, only a very small percentage of second-order mutants were not killed
when all the first-order mutants were killed. Moreover, Offutt showed
(1989, 1992) that the second-order mutant killing efficiency is between
99.94% and 99.99%. This means that if we have a test set that will kill all
the first-order mutants, then it will also kill the second-order mutants by
99.94%–99.99%. Thus, it is enough to consider only simple mutants.
The real advantage is that if we have a test design technique that
kills the first-order mutants, then this technique kills the second and
higher-order mutants as well. Anyway, we can assume that the bigger the
difference between the correct and the incorrect code is, the higher the
possibility is to find the bug. The minimum difference is the set of first-
order mutants, and therefore, we can assume we’ll find almost all the bugs
if we can find the bugs that are created by the first-order mutant operators.
Excellent, we have a very strong hypothesis that if we have a good test
design technique to find the first-order mutants, we can find almost all the
bugs, and our software becomes very high quality. Consider the numbers
again. Based on the comprehensive book of Jones and Bonsignour (2011),
we know that the number of potential source code bugs in 1000 lines of

30
(Java, JavaScript, or similar) code is about 35–36 on average. Thus, a code
with one million lines of code may contain 35000–36000 bugs. A test design
technique that finds at least 99.94% of the bugs would not detect only 22
faults in this huge code base.
The next step is to introduce a test design technique that theoretically
finds the bugs being first-order mutants. However, finding 99.94%–99.99%
of the bugs requires applying the test design technique without any
mistakes. Nobody works without making mistakes; therefore, this high
percentage is only a theoretical possibility. However, there is a huge
opportunity and responsibility in the testes’ hands. By applying the new
technique in the following in a professional way, extremely high code
quality can be achieved.
We note that fault-based testing differs from defect-based testing since
the latter is a test technique in which test cases are developed from what is
known about a specific defect type (see ISTQB glossary, term defect-based
test technique).
Requirements and Testing
The origin of most software bugs can be attributed to the requirements
phase. Therefore, the most effective approach to reducing the number
of newly discovered bugs in a project is to incorporate a requirements
analysis stage that teams must undertake prior to commencing coding. Any
requirements engineering handbook explains to you why completeness,
clearness, correctness, consistency, and measurability are key attributes
in defining and checking the right requirements. The granularity of
requirements determines the extent to which the requirements capture the
desired functionality, behavior, data, and other characteristics of the system
being developed. Requirements may vary in granularity, ranging from high-
level abstract statements to more detailed and specific descriptions. The
granularity depends on the nature of the project, the needs of stakeholders,
and the development methodology being followed.

31
At a high level, requirements may be expressed as broad objectives
or goals, providing an overall vision of what the system should achieve.
These high-level requirements are often captured in documents such as
a project vision statement or a system-level requirements specification.
As the development process progresses, the requirements can be further
refined and decomposed into more detailed and specific statements.
These detailed requirements provide a more precise description about
what the system should do and in which conditions the systems should do
that. These detailed requirements may include functional requirements
and nonfunctional requirements. The choice of granularity depends on
factors such as project complexity, stakeholder needs, and development
approach. In some cases, a coarse-grained level of requirements may
be sufficient to guide the development process, while in others, a more
fine-grained and detailed set of requirements may be necessary to ensure
accurate implementation.
Finding the right balance in requirements granularity is crucial. Too
much granularity can lead to excessive detail, making the requirements
difficult to understand, implement, and manage. Conversely, insufficient
granularity can result in vague or ambiguous requirements, leading to
misunderstandings and potential gaps in the final system. Ultimately,
the granularity of requirements should be tailored to the specific project,
striking a balance between providing clear guidance for development
while allowing flexibility and adaptability as the project evolves.
The granularity of requirements plays a significant role in the test design
process. The level of detail and specificity in requirements directly impacts
the test selection and adequacy, test case design, and overall testing
strategy. Here’s how requirements granularity influences test design:
• Test design: The granularity of requirements influences
the number of tests. Higher-level, coarse-grained
requirements lead to high-level test design, focusing on
the overall system behavior and major functionalities

32
resulting in less test cases. On the other hand, more
fine-grained, detailed requirements allow detailed test
design, resulting in more test cases.
• Test prioritization: The level of requirements
granularity helps prioritize testing efforts. If
requirements are broken down into smaller,
detailed units, it becomes easier to identify critical
functionalities that require higher testing priority. This
ensures that the most important aspects of the system
are thoroughly tested, reducing the risk of overlooking
critical areas.
• Test execution efficiency: Requirements granularity
influences the efficiency of test execution.
Coarse-
grained requirements may lead to higher-level,
scenario-based tests that require fewer test cases
to cover a wider scope. In contrast, more granular
requirements may require a larger number of test
cases to achieve comprehensive coverage, potentially
increasing the overall effort and time required for test
execution.
• Traceability: Granular requirements facilitate
traceability between requirements and test cases.
When requirements are well-defined and specific, it
becomes easier to establish a clear mapping between
individual requirements and the corresponding test
cases. This improves traceability, making it easier
to track the progress of testing and ensure that all
requirements are adequately covered.

33
• Test maintenance: The granularity of requirements
impacts the maintenance of test artifacts. If
requirements change or evolve over time, having more
granular requirements allows for targeted updates to
affected test cases. Coarser requirements may require
more extensive modifications to test cases, potentially
increasing the effort required for test maintenance.
In summary, requirements granularity influences test design,
prioritization, execution efficiency, traceability, and test maintenance.
Striking the right balance in requirements granularity is crucial to ensure
that the test design aligns with the specific characteristics and objectives of
the software being developed.
There are several ways, from informal to formal, expressing the
requirements adequately. In this book, we apply the simplest textual form
for describing the requirements.
Testing Principles
Testing principles are fundamental guidelines and concepts that serve as
the foundation of effective and efficient software testing. These principles
help ensure that software testing is conducted systematically and
thoroughly to identify defects, verify software functionality, and control
overall software quality. In the following, we encounter the key testing
principles and their significance:
1. Testing is Possible
Software testing is possible due to the coupling effect hypothesis (CEH).
Without CEH, only exhaustive testing could be applied. However, it is
impossible to test all possible input combinations and scenarios for a
complex software system. Testing efforts should focus primarily on critical
and high-risk areas to maximize the likelihood of identifying significant

34
defects. Howden (1976) proved that there is no general algorithm to create
a reliable test set that guarantees finding all bugs for all programs. On the
other hand, just because testing didn’t find any defects in the software,
it doesn’t mean that the software is perfect or ready to be shipped.
Thoroughly testing all the specified requirements and fixing all the defects
found could still produce a system that does not fulfill the users’ needs and
expectations. Bug-free software can be produced only by applying formal
methods; however, these methods are hard to scale.
2. Early and Balanced Testing
Start testing activities as early as possible in the software development
life cycle. Early testing helps in detecting defects at their source, reducing
the cost of fixing issues later in the development process. The test-first
methodology is a crucial aspect of early testing, wherein the test design is
made before implementation. This allows the software development team
to identify flaws at an earlier stage rather than waiting until the project is
complete. Early testing, especially requirements validation, is the answer
to the “absence of error is a fallacy,” stating that bug-free software still
can be unusable. The fallacy is violated if the requirements are efficiently
validated involving the users/product owners. The client’s requirements
and expectations are as important as the quality of the product.
Early testing may speed up the time to market, supports identifying
problems in the earliest stages of product development, results in greater
efficiency in the testing process, preserves software quality, and allows
continuous feedback. To achieve an effective early testing strategy, it should
be identified what it means for the testing team, defect prevention should
be applied, developers should be incorporated into the testing activities,
and test design automation tools must be applied. Testing activities should
be balanced: testers should allocate testing efforts across various levels and
types of testing, such as unit testing, integration testing, system testing, and
acceptance testing, to achieve a balanced testing approach.

35
3. Testing is Independent and Context Dependent
From a managerial aspect, testing and development processes should
be independent; hence, testers should remain independent from the
development process to ensure unbiased evaluations of the software. This
independence helps identify issues that developers might overlook due to
familiarity with the code. On the other hand, testing is context dependent;
that is, the corporate testing approaches and techniques should be tailored
based on the specific requirements, goals, and constraints of the project.
There is no one-size-fits-all solution for testing; the testing project cannot
be copy-pasted.
4. Continuity of Testing
Continuous development requires continuous testing and continuous
feedback. Clearly, running the same set of tests for changed requirements
will not find new defects. Testers must review and update test cases according
to the new or modified requirements. This involves the deletion of obsolete
test cases. Fortunately, modern methods such as modeling the test design
make it possible. Continuous development necessitates continuous testing
and feedback as an integral part of its iterative and agile nature. Embracing
these practices enhances software quality, responsiveness to change, and
collaboration between development teams and stakeholders, ultimately
leading to better software products and improved customer satisfaction.
Continuous feedback, which includes input from end users and stakeholders,
helps shape the software according to customer needs and expectations.
Integrating customer feedback into the development process results
in more customer-centric products. Continuous testing is an essential
component of CI/CD pipelines. It enables automated testing at every stage
of the deployment process, ensuring that only thoroughly tested and verified
code is released to production. Continuous testing aids in risk mitigation by
continuously assessing the impact of new changes on the existing code base.
It reduces the chances of introducing unintended side effects. Continuous

36
testing and feedback promote a culture of innovation: developers can
experiment, iterate, and refine their ideas rapidly, leading to more innovative
and competitive software solutions. Continuous testing and feedback
provide valuable insights into development practices, allowing the team to
continuously improve their processes, tools, and methodologies over time.
5. Defect Clustering
Defects tend to cluster around specific modules or functionalities of the
software. By identifying and addressing these high-defect areas, testing
efforts can be prioritized effectively. In the following, we explain the
reasons behind this. The uninterested reader can skip this section.
• Recall that in probability theory, a lognormal
distribution is a continuous probability distribution
of a random variable whose logarithm is normally
distributed (Gaussian distribution). The lognormal
distribution plays a significant role in nature and
information technology. In nature, it often arises as
a result of multiplicative processes. For example,
the growth of populations, the sizes of biological
organisms, and the occurrence of earthquakes or
financial returns can exhibit lognormal distributions.
In information technology, the lognormal distribution
finds applications in various areas. One important
application is in modeling the behavior of data transfer
rates in computer networks, where it captures the
inherent variability and fluctuations in network traffic.
Overall, the lognormal distribution serves as a useful
tool in understanding and analyzing phenomena in
both natural and technological systems, providing a
mathematical framework to describe and model their
inherent variability and probabilistic properties.

37
The Pareto distribution is commonly associated with the “80–20
rule” or the principle of “vast majority, rare few.” The Pareto distribution
describes phenomena where extreme events or values are more prevalent.
Lognormal and Pareto distributions are very similar, and there is no
evidence which can be applied for defect clustering. However, the message
of this principle is that defects are not equally distributed, and if the tester
detects more bugs in a feature, then this feature should be tested more
carefully. Lognormal or Pareto distributions can play significant roles in
software testing in various ways, particularly in identifying and prioritizing
software defects or issues. Here are a few ways in which these distributions
can be applied:
1. Defect distribution: Scientific research support that
in human-
developed software systems, large parts
of the problems or defects are caused by small parts
of the underlying issues.
2. Defect prioritization: By analyzing defect data and
categorizing them based on their frequency or
impact, software testers can identify the most critical
defects that contribute to most problems. This
prioritization helps allocate resources efficiently to
address the most impactful issues.
3. Root cause analysis: Pareto-type distributions can
aid in identifying the root causes behind software
defects. By analyzing defect data and classifying
them based on their root causes, testers can
identify the few underlying causes that contribute
to a significant portion of the defects. This allows
for focused efforts on addressing the root causes,
leading to more effective and efficient defect
resolution.

38
4. Test case prioritization: Software testers can
prioritize test cases based on their impact and/or
likelihood of detecting critical defects. By focusing
testing efforts on the subset of test cases that cover
the most critical functionalities or areas of the
software, testers can maximize the effectiveness of
their testing efforts and increase the likelihood of
uncovering important defects.
5. Performance optimization: In performance testing,
lognormal or Pareto-type distributions can be used
to analyze system performance metrics, such as
response times or resource utilization. By identifying
the few performance bottlenecks or areas that
contribute to many performance issues, testers can
prioritize optimization efforts and allocate resources
to address the most impactful areas, thereby
improving overall system performance.
Overall, the mentioned distributions can guide decision-making in
software testing by helping testers prioritize their efforts, localize, and
identify critical defects or root causes and optimize system performance.
By focusing on the vital few rather than the trivial many, testers can
enhance the quality and reliability of software systems.
AI and big data analysis are essential drivers of economic growth,
transforming industries and business operations. AI-powered technologies
enable businesses to analyze vast amounts of data quickly, leading to
informed decision-making and improved efficiency. In the context
of AI-generated software, lognormal or Pareto distributions may be
observed in data distribution, fault analysis, and model optimization.
These distributions highlight the significant impact of a small subset of
data, bugs, or hyperparameters on the system’s performance. Although

39
specific evidence for defect clustering in AI-generated software is lacking,
identifying critical factors remains relevant in various scenarios. The
applicability of these distributions may vary based on factors like problem
domain, data quality, and AI system complexity.
Two Misconceptions
It is NOT an applicable principle that testing shows the presence, not the
absence of defects quited by Dijstra, see (Buxton at al 1969). It is only for a
good excuse for lazy testers. Even if testing doesn’t result in the absence of
defects, it may assure the absence of a class of defects. Properly structured
tests and formal methods can demonstrate the absence of errors
(Goodenough, Gerhart 1975). However, it is unknown how to scale formal
software engineering. In this book, we provide a technique, by which for an
error class, a reliable test set can be constructed (apart from coincidental
correctness).
Most testers know and believe in the pesticide paradox as it’s often
some questions at ISTQB exams. Originally, Beizer (1990) wrote: “Every
method you use to prevent or find bugs leaves a residue of subtler bugs
against which those methods are ineffectual.” This paradox can be
rewritten as “If the same set of repetitive tests is conducted, the method
will be useless for discovering new defects,” or “if the same set of test cases
are executed again and again over the period of time, then these set of tests
are not capable enough to identify new defects in the system.” In simple
words: tests wear out. The meaning of these descriptions is significantly
different from what Beizer stated, but the main problem is that it is wrong.
The truth is that if you have a good test set, it remains good and you
can use it after years. Let’s assume that we have the correct software and a
reliable test set detecting any potential defect. Without loss of generality,
we can assume that the software is deterministic. Imagine that we make
all the possible modifications for the software and each modified version
remains deterministic. Some modifications introduce new bugs into the

40
software. Now let’s use our test set for all the modified pieces of code.
There are two possibilities:
1. The test for the modified software fails.
2. The test for the modified software passes.
Let’s consider a scenario where the software’s code changes after some
time and a bug is introduced. Because the test set detected the bug earlier
it will continue to do so since both the test set and the modified software
remain the same and the software operates deterministically. This is true
without doing all the possible modifications. Consequently, the test set
never becomes outdated or less effective than when it was initially created.
On the other hand, if the functional requirements changed, then new tests
should be designed for the modified functionality. However, this is a quite
different problem.
Comparison of the Existing and Our Principles
Compared with other widespread testing principles, we omit the
following ones:
• Pesticide paradox.
• Testing shows the presence of defects.
• Absence of errors fallacy.
We substituted the negative principle “exhaustive testing is not
possible” with a positive one “testing is possible.” Finally, we extended
the principle of “Testing is context dependent.” Here is the list of our
principles:
1. Testing is possible.
2. Early and balanced testing.

41
3. Continuity of testing.
4. Testing is independent and context dependent.
5. Defect clustering in human developed software.
Summary
In this chapter, we first looked at bugs and explained why the coupling
effect hypothesis makes it possible to use test design for detecting bugs. We
have shown the requirements for a good test design through which most
bugs can be detected. We have introduced a new classification of bugs,
where a bug is either a control flow bug or a computational bug from the
requirements specification point of view.
The next part is devoted to the fundamentals of software testing,
where we briefly describe the software testing life cycle. We looked at test
planning and test analysis and showed the importance of risk analysis.
Then we moved on to test design, test implementation and execution, and
finally test closure.
We showed how mutation tests verify our designed test sets and
how to measure test efficiency. We described the connection between
requirements and software testing. Finally, we renewed the principles
of software testing, removing some and adding new ones. We have also
shown why some of the original principles are wrong or not useful.

43
© István Forgács and Attila Kovács 2024
I. Forgács and A. Kovács, Modern Software Testing Techniques,
https://doi.org/10.1007/978-1-4842-9893-0_2
CHAPTER 2
Test Design
Automation by
Model-Based Testing
In this chapter, you’ll discover the significance of automating test design,
a crucial aspect of test automation. The primary approach for automated
test design is called model-based testing (MBT), which is widely used.
MBT methods are divided into one-phase and two-phase model-based
testing. We explain the advantages of the latter approach over the
traditional one. Additionally, we categorize models into three types:
stateless, stateful, and mixed models. We illustrate that stateless models
are less effective in identifying defects. On the other hand, the stateful
solution can be challenging to apply to certain requirements. As a result,
the mixed model solution emerges as the most favorable option. This
content should be understood within the reading time.
• Beginners: 5 hours
• Intermediates: 4 hours
• Experts: 3 hours

44
Higher-Order Bugs
A system (or software) is called stateless if it has no memory of the previous
interactions. It means computational independence on any preceding
events in a sequence of interactions. A stateless application decouples the
computations from the states; it is dependent only on the input parameters
that are supplied. Simple examples are
• A search function in a text editor with a given
search pattern
• A sorting function with given items to be sorted
• A website that serves up a static web page (each request
is executed independently without any knowledge of
the previous requests)
In these cases, a stateless system/function returns the same value
for the same arguments in the same environment. The computations are
replaceable without changing the behavior of the application.
On the other hand, stateful applications have internal states. These
internal states need some place for storage (memory, database, other
variables). When a stateful function is called several times, then it may
behave differently. From the test design point of view, such systems
are more difficult to test: the system must be treated together with the
preceding events. An operating system is stateful. A traditional web
application is stateful. Most applications we use nowadays are stateful.
Considering bug detection, the simplest case is when a single data
parameter triggers a failure whenever a program execution reaches a
certain point along any program path from the data entry point. There
may be a subset of data values of this parameter for which the software
always fails; otherwise, it always passes assuming that there is no other
bug in the code. Simple and cheap test design techniques can find this
Chapter 2 Test Design Automation by Model-Based Testing

45
type of bug, which will be referred to as first-order bugs. Examples of test
design techniques concentrating on finding first-order bugs are equivalent
partitioning and boundary value analysis.
It can occur, however, that the failure only happens if more parameters
have specific values in common. Here, the single fault assumption cannot
be applied. If the number of parameters causing the failure together is
two, then the bug is a second-order bug, and so on. Fortunately, there are
fewer (n+1)-order bugs than n-order bugs (see Kuhn et al. 2004, Forgács
et al. 2019).
Here is an example for demonstrating first and second-order bugs.
Online Shop
Company RedShoe is selling shoes in its online shop.
OS-R1 If the total ordering price is below EUR 100, then no price reduction is given.
OS-R2 The customer gets a 4% reduction when reaching or exceeding a total price
of EUR 100. Over a value of EUR 200, the customer gets an 8% reduction.
OS-R3 If the customer is a premium VIP, then she gets an extra 3% reduction. If
the customer is a normal VIP, they get a 1% extra reduction. Normal VIPs must be
registered, and the customer is a premium VIP if in the past year, the amount of their
purchases has reached a certain limit. The system automatically calculates the VIP
status.
OS-R4 If the customer pays immediately at the end of the order, she gets an
additional 3% reduction in price.
OS-R5 The output is the reduced price to be paid. The lowest price difference
(accuracy) is 10 euro cents.

46
Here is a correct and faulty Python implementation.
# Correct python implementation
def webshop(price, vip, prepay):
reduction = 0
if price = 200:
reduction = 8

if price = 100 and price 200:
reduction = 4
if vip == 'normal':
reduction = reduction + 1
if vip == 'premium':
if prepay == True:

return(price*(100-reduction)/100)
# Buggy python implementation
def webshop(price, vip, prepay):
reduction = 0

if price 200: #FIRST-ORDER BUG HERE
reduction = 8

if price = 100 and price 200:
reduction = 4
if vip == 'normal':
if vip == 'premium':

if prepay == True: #SECOND-ORDER BUG

return(price*(100-reduction)/100)
For the first-order bug, only the variable price is responsible: if your test
contains the value price = 200, then the bug will be detected; otherwise,
it isn’t. Boundary value analysis is reliable for this bug. Recall that a test
selection technique is reliable for a bug; if applied, it certainly detects
the bug.
For the second-order bug, you should set two variables; that is, the
variables vip and prepay must be set to find the bug. The bug is only
detected if vip ≠ premium and prepay = True. See Table 2-1.

47
Table 2-1. Decision table for the Online Shop application. You can
read the table as “if VIP status is ‘None’ and Prepay is ‘True,’ then the
reduction is 3% in case of the correct implementation earlier and zero
in case of the incorrect implementation.” The other columns can be
read similarly
VIP Status None Normal Premium
Prepay True False True False True False
Reduction value (correct implementation) 3 0 4 1 6 3
Reduction value (incorrect
implementation)
0 0 1 1 6 3
Unfortunately, boundary value analysis is not reliable for this bug. You
can see that a second-order bug can occur even in the case of a single data
fault. Having a second-order bug only means that it can be revealed by
setting two variables.
Consider an application where a free bike is given when you rent three
cars, but the bike is withdrawn if you delete one of them. A simple bug
happened by changing a parameter from False to True during a function
call. You can find the bug if you add three cars, delete one of them, and
finally add a bike. But the bug remains undetected when you add a bike
first, then three cars, and finally delete a car. This is clearly a higher-order
bug; however, here, not the input values but the order of the inputs is the
clue for detecting the bug. We note here that to detect higher-order bugs by
applying white-box methods, different dataflow-based adequacy criteria
are available in the literature (Rapps et al. 1982, Korel et al. 1988).
In the following, we enumerate some real examples of higher-order
bugs found by the authors.
The first is an email service. When first signing in, it works correctly.
However, by adding another account and signing in for the second time,
the screen is frozen (some workaround had to be performed for signing in).

48
The second example is a video editor. When inserting an mp3 audio
file, it’s fine. But when you modify the audio, delete the original, and insert
the modified, the length of the audio remains the original, and some parts
of the new one may be cut.
These are examples of second-order bugs. With a high probability, the
testing was not good enough to detect these annoying bugs. To summarize,
a higher-order bug occurs when it can only be revealed by setting more
variables or executing the code passes for an input sequence, but it fails for
some other input sequence, where the sequence consists of more than one
input. It’s a surprise that even simple code may contain third-, fourth-, or
even higher-order bugs.
We’re sure you can find similarly annoying bugs in your domain. The
reason is that application providers believe that detecting these bugs
would be too expensive. But this is not true. We will show how to use
effective and efficient methods for finding higher-order bugs easily.
The most widely used test design automation technique is model-
based testing (MBT). The essence of MBT is that instead of creating test
cases one by one manually, we create an appropriate test model from
which an MBT tool can generate test cases based on appropriate test
selection criteria. Models represent test design techniques. For example,
statecharts may represent state transition testing. There are excellent
papers about MBT. Here we consider MBT in a slightly different way.
Model-Based Testing
Model-based testing (MBT) has become more and more popular in recent
years. MBT is used for validating requirements, that is, for software defect
prevention, for shared understanding, for generating executable test
cases, and so on. In this book, we restrict our attention to automated test
design for functional testing, and we use MBT and automated test design
interchangeably.

49
Nowadays almost everybody uses Agile and DevOps, where
automation is a must for all phases of the software life cycle. Test
design cannot be an exception. The result of test design automation is
automatically generated test cases. Whenever the requirements change, so
does the model. A big advantage of MBT is that there are no obsolete test
cases since MBT tools regenerate the test cases after each modification.
Models are believed to be abstract representations of objects or
systems. In software testing, a model should consist of all the necessary
elements of the system, but only those elements, not more. Unfortunately,
in software testing, most modeling techniques are borrowed from
computer science and software engineering. The result is that the defect
detection capability of these models is not efficient enough. In this book,
we introduce a new MBT approach, improving existing techniques and
introducing a new modeling technique to address this issue.
Model-based testing is about 25 years old, see Dalal et al. (1998). A
basic overview of MBT and its practice is the well-known book by Utting
et al. (2010). A recent book guide to the ISTQB-certified model-based
testers is due by Kramer et al. (2016).
One-Phase (Traditional)
Model-Based Testing
Various MBT tools employ diverse models, yet they share a common
feature: all these models are considered computer-readable. This means
that the model should be understandable for a computer program that
reads, scans, and processes the model and generates executable test cases.
The traditional model-based testing process is the following:
Step 1. Creation of the MBT model

50
In the first step, modelers create MBT models from requirements/user
stories and from acceptance criteria. The model should contain everything
to detect functional defects but should exclude all other unnecessary
information. The models can be graphical or textual. Textual models can
be transformed into graphs. As such, use cases can also be converted
to graphs.
As mentioned, testers use mainly system design models without any
tailoring. Frequently used flowcharts include conditions and gateways
that are superfluous for test models. The reason is that both outcomes
of a condition should be covered by a test; thus, the input should be set
to cover both branches of a decision. In this book, we use a simple and
understandable model, where the edges are actions and the nodes are the
system responses. Figure 2-1 is a simplified model for account creation. The
specification is well-known; we omitted it.

51
Figure 2-1. Flowchart for account creation

52
This chart is oversimplified avoiding backward edges to illustrate our
concepts. The chart should contain implementation-dependent actions
such as “select create account” and “press create your account.” The model
should contain those cases, where after an error, the customer can correct
the faulty event.
Let’s consider the validation steps in the models. Modeling outputs is
not an easy task. At some model points, the output value could be different
along different paths from the starting point. This is the basic behavior of
stateful systems even if the model is stateless. When the model is created,
we don’t know which test paths will be generated. If there is a long e2e
test, then a faulty output results in subsequent faulty output. The solution
is that validation is moved to Step 3 when the test cases are generated.
Another method is to add some code to the model to compute the outputs
correctly. However, this is not a good solution as the goal is to use the
model for validation and not to implement the output.
Step 2. Choosing appropriate test selection criteria
Test selection criteria are created based on the application to be
implemented and on product risks. The criteria can be anything, but
usually, they are based on model traversing. The minimum test selection
criterion is that each edge in the graph should be covered, that is, the all-
edge criterion. We use the notions “all-edge criterion” and “all-transition
criterion” interchangeably. As requirements should also be tested, it’s
reasonable to make the model in a way that covering each edge involves
covering all the requirements as well. Surprisingly, for several systems
when applying stateful models, this minimum test selection criterion
extended with appropriate BVA techniques is almost reliable. In the
section “Test Selection Criteria for Action-State Testing” we introduce a
slightly stonger criterion. In some cases, stronger test selection criteria,
such as the all-transition pairs criterion, are needed. This means that each
adjacent edge pair should be covered. In our model earlier, fulfilling both
criteria requires the same two test cases. An even stronger criterion is
when all the (different) paths in the graph are covered. In our model, this

53
results in four test cases, but for larger systems, this criterion would lead to
too many test cases.
Step 3. Generation of semi-abstract test cases
The third step automatically generates test cases from MBT models
based on the test selection criterion. Most articles refer to these test cases
as “abstract” as these test cases haven’t been executable yet. However,
these test cases are computer-readable. That’s why we will call them
semi-abstract test cases. These test cases involve steps that contain enough
information to be able to convert them into executable tests. Here is an
example of a test step for a web application:
When add Coke is #pressed
Here “When” and “is” are keywords for humans to understand test
steps better. add Coke is the selector, that is, the identifier of the GUI
object, and #pressed is a command or action word. The selector connects
the test case and the application under test; the action word maps the
action to be done to the code to be executed.
The test cases consist of implementation-dependent steps such as
pressing a button or validating that some field is disabled. These model
elements are difficult to include before the implementation as the
developers usually have some freedom regarding the implementation.
Therefore, traditional modeling is usually started when the
implementation is ready. In some cases, if screens for the features are
planned, then models can be created, and tests can be generated in
parallel with the implementation.
Step 4. Generation of executable test cases
With the help of a test adaptation layer, the test cases can be concretized.
In this step, the test design automation tool generates the test code. The
commands are replaced by the test code that is available before the model is
created. However, mapping the test code and the software under test (SUT)
usually happens only when the application is ready. Here is an example:
When add Coke is #pressed - selector(add Coke).click()

54
Here the command #pressed is replaced by click(), and the abstract
selector “add Coke” is replaced by the concrete selector that can be a
CSS selector. This is a simple replacement; however, in other cases, the
implementation is more difficult:
Consider the greater than ‘’ operator:
#gt string
- invoke('text').then(parseInt).should('gt', Number(string))
In these examples, the code is JavaScript for the Cypress runner,
while the commands are examples from the Harmony test design
automation tool.
We also should map the UI object in the code and the test. The
simplest case is when the developer uses special attributes, and therefore
the selector name in the code identifies the UI element, such as “add
Coke” in our former example. Without special attributes or identifiers,
the selector can be very long and nonunderstandable. In this case, an
understandable abstract selector name should be mapped to the original.
In our example the CSS selector is
div[id=render-root] div form ul li:nth-child(4)
button
that has been mapped to add Coke.
The semi-abstract test cases are computer-readable; hence, software
tools can parse them, and based on the selectors and the command - code
mapping, the executable tests are generated.
Note that the command - code mapping can easily be changed. In this
way, different test codes for different test runners can be generated.
Lots of tools can generate test automation code and selectors while
parsing the application. The GUI objects are parsed and inserted for later
use. During the adaptation, the selectors and some code are available, and
these ones with input/output values should be mapped to the model/test

55
step. Note that this is not a capture-and-replay technique as the model and
the semi-abstract test cases are ready. During capture and replay, the test
cases are generated during the test execution, and not before.
Step 5. Test execution and test result analysis
The executable test cases are executed against the SUT by a test runner.
The result of the test design automation is the executable test code. Test
execution is the next step of the test automation. Test result analysis is
outside of the entire test automation as it is made by a team to improve the
application and the whole process.

Two-Phase Model-Based Testing
Traditional MBT requires computer-readable models. These models
should contain detailed information to become executable. A tester or
business analyst usually has a special domain knowledge of the system.
Therefore, higher-level test cases would be enough that make it possible
to create a more compact model. That’s why we introduced two-phase
modeling when the modeling process is divided into two parts:
1. High-level modeling
2. Low-level modeling
The high-level model is human-readable and can be done before
implementation. The low-level model is computer-readable, and it’s
generated from the high-level model during manual test execution. The
two-phase MBT process is the following:
Step 1. Choosing appropriate test selection criteria
Here, the process changed as the first step is to select the appropriate
test selection criterion. The reason is that the modeling is driven by the test
selection criterion. It means that the missing model steps are displayed
and offered to insert.

56
Step 2. Creation of the MBT model
In this step, modelers create high-level models from requirements/
user stories and acceptance criteria. The high-level model consists of high-
level steps. These steps are implementation-independent and as abstract
as possible. For example, an action can be as
add items so that the price remains just below EUR 15
A tester can do it in different ways such as
1. add item for EUR 11.
or
The model elements can be as high-level descriptions as possible.
The only requirement is that the tester can execute them. In this way,
the model will be more compact and remains more understandable. For
example, a single step for modeling a password change can be
modify password = password is successfully modified
In contrast, a low-level model requires the following steps:
1. Select password modification.
2. Insert existing password.
3. Insert new password.
4. Insert new password again.
5. Submit password modification = password
successfully modified.
Let’s consider our account creation example. You can see that the steps
are higher level, and the graph is more understandable. Only five actions
remained; originally there were nine (see Figure 2-2).

57
Figure 2-2. High-level model for account creation
Let’s consider the validations in the models. How to model the outputs
during high-level modeling? It’s much simpler since outputs are also
abstract. For example, if we should validate a whole screen as a response
to an action, then the tester can do it later when the test is executed. The
model should only contain “= check the screen appears.” When the test
is executed, the tester can check everything similarly to use exploratory
testing (see Step 3). However, in some cases, concrete output values can be
added to the model.
Step 3. Generating the abstract test cases
Based on the model, the abstract test cases are generated. These
test cases are only executable by humans and can be considered as “live
documentation.” The abstract tests are also examples of how the system
works, and with them, the requirement specification can be validated.
This is very important as requirement specification is usually not complete
and error-prone leading to some false or missing implementation. The
cheapest way to fix the specification problems is the validation of the
high-level model or abstract test cases. We will see concrete examples for
improving the specification later. This is an excellent defect prevention
method as an incomplete or faulty requirement leads to faulty code, but
these test cases can be used against implementation leading to fewer
defects. This is not only an efficient defect prevention but also cheap as
it’s a side effect of test design automation. It usually cannot be done by

58
applying the one-phase approach as the models are created after the
implementation; otherwise, lots of remodeling work should be done.
Step 4. Test execution and low-level model generation plus test
execution
The prerequisite of this step is the implemented SUT. The tester
executes the abstract test cases one by one, while the test design
automation tool generates the low-level model step. For example, if the
high-level model step is “add pizzas to reach EUR 40,” the tester first selects
the “shopping” feature, then selects a Pizza Chicken for 12 euros twice and
a Pizza Smoked Salmon for 16 euros. While clicking the selection and then
the add buttons (three times), the system generates model steps such as
When Shopping is #pressed
When add button for Pizza Chicken is #pressed
When add button for Pizza Chicken is #pressed
When add button for Pizza Smoked Salmon is #pressed
These model steps are the same as in the traditional model steps in
Step 3 and are computer-readable. The output is also generated on the fly.
In most cases, the output is a value or the visibility of some UI elements.
Therefore, the validation step is usually an additional selection, where
the output value or the visible UI object is selected. Other validation
commands such as “non-visible” or “active” can also be selected.
Besides the generation of the low-level model, the executable test
code is also generated, and the test step is executed immediately. If a step
fails, it can be fixed and executed again. When the test execution has been
finished, the test case is automated, and no debugging of the test steps
is needed.
The whole process is in Figure 2-3.

59
Figure 2-3. Two-phase MBT process
The two-phase modeling is a test-first solution as high-level models
can be created before implementation. It can be used for stateful and
stateless cases (see the subsequent chapters). The models are more
compact and therefore more understandable. Executing the abstract tests
is comfortable as testers don’t need to calculate the results in advance.
They just need to check them, which is much easier. Using this method,
the focus is on test design instead of test code creation.
Stateless Modeling
As previously mentioned, models can be categorized as stateful, stateless,
or a combination of both. A stateless model primarily comprises user
actions (inputs) and system responses (outputs) but does not incorporate
any states. In the realm of software test design automation, stateless
modeling stands out as the most prevalent technique due to its simplicity
and applicability to a wide range of systems. This approach involves
modeling business processes to depict the dynamic aspects of the systems

60
being tested. One of the closely related test design techniques is known
as use case testing. Additionally, there exist several other techniques for
describing the system’s behavior, which can also be employed in software
testing similar to use cases. This involves navigating through the diagrams
based on a test selection criterion. Examples of such models include
BPMN, UML activity diagrams, and so forth. Although these solutions
employ different notations, they share substantial similarities in terms of
the information they encompass.
A crucial consideration lies in the fact that developing a stateless
model for test design automation should significantly deviate from the
process of modeling for system design. In this context, we aim to illustrate
these distinctions by presenting examples of both an incorrect and a
well-constructed model.
We show a simple example to demonstrate all the mentioned modeling
techniques. In this way, we can compare the error-reveling capability of
the different models.
Our example in the following is a simplified specification of our car
rental exercise from the website test-design.org.
A rental company loans cars (EUR 300) and bikes (EUR 100) for a week.
R1 The customer can add cars or bikes one by one to the rental order.
R2 The customer can remove cars or bikes one by one from the rental order.
R3 If the customer rents cars for more than EUR 600, they become eligible to
receive a complimentary bike rental as part of a discount offer:
R3a If the customer has selected some bikes previously, then one of them
becomes free.
R3b If the customer hasn’t selected any bike previously, then one free bike is
added.

Modern Software Testing Techniques: A Practical Guide for Developers and Testers

Modern Software Testing Techniques: A Practical Guide for Developers and Testers

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Modern Software Testing Techniques: A Practical Guide for Developers and Testers

Similar to Modern Software Testing Techniques: A Practical Guide for Developers and Testers (20)

More from Lucky Gods

More from Lucky Gods (20)

Recently uploaded

Recently uploaded (20)

Modern Software Testing Techniques: A Practical Guide for Developers and Testers