Foundations Of Software Testing

Foundations of  
Software Testing
ECOOP/ISSTA’21 Summer School
Marcel Böhme (Monash University, Australia)
Soon @ Max Planck Institute for Security and Privacy, Germany.

Marcel Böhme, Monash University · ECOOP/ISSTA’21 Summer School · Foundations of Software Testing
• Fuzzing for Automatic Vulnerability Discovery

• Making machines attack other machines.

• Focus on scalability, eﬃciency, and eﬀectiveness.

• Foundations of Software Security

• Assurances in Software Security

• Fundamental limitations of existing approaches

• Drawing from multiple disciplines (information theory, biostatistics)
whoami
Marcel Böhme
ARC DECRA Fellow

Senior Lecturer (A/Prof)

Monash University, Australia
Looking for PhD & PostDocs 
at Max Planck Institute 
Bochum, Germany
1

software testing
Input Process Output
2

software testing
Test Case Program
Pass 
or

Fail
2

software testing
Problem:

Generate at least one failing test case for each bug in the program.

• You’ve been generating test cases for your program.

• No bugs found! 👍

• Is your program free of bugs?

• Probably not. 😆

• Is your test case generation technique eﬀective?

• Maybe? 😅

•
🤔
• How do you even measure eﬀectiveness if there are no bugs?
questions for today
3

• How does the test case know if something is a bug or a feature?

• What is the difference between effectiveness and efficiency?

• When is the most effective technique (whitebox fuzzing)  
more efficient than random test generation (blackbox fuzzing)?

• How does greybox fuzzing work? Why is it so successful?

• What is the relationship btw. #bugs found and #machines available?
questions for today
3

• Let’s start at the beginning.

• A test case consists of a test input and at least one test oracle.

• A test case passes if no test oracle detects a bug for the test input.
test case = test input + test oracle
4
test case
test oracle
test input
effectiveness
efficiency
scalability
^

Test Input
Expected 
Output?
Program
test case » system testing
5
test case
test oracle
test input
effectiveness
efficiency
scalability
^

$ ./gifbuild -d crashing.PoC.gif
#
# GIF information from ./crashing.PoC.gif
screen width 0
screen height 0
screen colors 2
screen background 0
pixel aspect byte 232
image # 1
image left 0
image top 0
ASAN:DEADLYSIGNAL
=================================================================
==18392==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000  
(pc 0x000000403d84 bp 0x7fc122903708 sp 0x7ffcac6ff150 T0)
#0 0x403d83 in Gif2Icon /home/root/giflib-asan/gifbuild.c:877
#1 0x401c3c in main /home/root/giflib-asan/gifbuild.c:100
#2 0x7fc12255e82f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
#3 0x4020b8 in _start (/home/root/giflib-asan/gifbuild+0x4020b8)
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV /home/root/giflib/gifbuild.c:877 in Gif2Icon
==18392==ABORTING
Test Input
Test
Oracle

Unit
test case » unit testing
6
test case
test oracle
test input
effectiveness
efficiency
scalability
^

Unit
Test Harness
Input Expected?
6
test case
test oracle
test input
effectiveness
efficiency
scalability
^

Test Input
Test Oracle
Test Case
http://scala-ide.org/docs/2.0.x/testingframeworks.html
6
test case
test oracle
test input
effectiveness
efficiency
scalability
^

How does the test case know if the
test input exposes a bug?
test case » test oracle
7
test case
test oracle
test input
effectiveness
efficiency
scalability
^

How does the test case know if the
test input exposes a bug?
The test oracle flags it as a bug.
test case » test oracle
test oracle
7
test case
test oracle
test input
effectiveness
efficiency
scalability
^

test oracle
Question:
What kind of test oracles do you know?
7
test case
test oracle
test input
effectiveness
efficiency
scalability
^

Assertion
Test Input
test oracle » assertion-based testing
8
test case
test oracle
test input
effectiveness
efficiency
scalability
^

Test input
Satisfies 
Postcondition?
Satisfies 
Precondition?
If not, ignore.
test oracle » property-based testing
9
test case
test oracle
test input
effectiveness
efficiency
scalability
^

Question:
Can we have a test oracle that tells for
every input what is the expected output?
test oracle
10
test case
test oracle
test input
effectiveness
efficiency
scalability
^

Question:
Can we have a test oracle that tells for
every input what is the expected output?
test oracle
No. This is called the oracle problem.
10
test case
test oracle
test input
effectiveness
efficiency
scalability
^

• We may know how outputs relate to each other for all inputs.

• Mathematical laws:

• sin(π - x) = sin(x)

• x + y = y + x

• Round-trip properties:

• x = unzip(zip(x))

• x = uncompress(compress(x))

• x = decrypt(encrypt(x))

• x = pickle.dump(pickle.load(x)) # serialization / parsing
test oracle » metamorphic testing
11
test case
test oracle
test input
effectiveness
efficiency
scalability
^

• We may know how to change inputs, expecting the same output.

• Compiler / interpreter testing

• If you add unreachable code, the compiled binary should give the same output.

• Constraint solver testing

• If you change a constraint and guarantee that the resulting constraint 
is logically equivalent, then the solver should produce the same result.

• Fairness testing

• If you change a sensitive field (gender or race), then the classifier  
should produce the same result.
test oracle » EMI testing
12
test case
test oracle
test input
effectiveness
efficiency
scalability
^

• We may know how to change the program, expecting the same output.

• Regression testing ensures that future program versions are  
at least as correct as the current version.

• When a regression test case fails, the bug is either  
in the program or in the regression test oracle.
test oracle » regression testing
13
test case
test oracle
test input
effectiveness
efficiency
scalability
^

• We may have many implementations that can cast their vote.

• For instance, Guido Vranken’s cryptofuzz @ OSS-Fuzz continuously tests  
cryptographic protocol implementations:

• OpenSSL, BoringSSL, LibreSSL, BearSSL, MBedTLS,  
EverCrypt, Crypto++, cppcrypto, crypto-js, libgcrypt, libtomcrypt,  
symcrypt, wolfcrypt, veracrypt, libtomath + 40 more
test oracle » differential testing
14
test case
test oracle
test input
effectiveness
efficiency
scalability
^

• Implicit test oracles detect “non-semantic”, functional bugs.

• Examples: buﬀer overflows, memory leaks, data races, integer overflows,  
null pointers, type confusion,

• Test oracles: crashes, exceptions, kernel panic, runtime monitors, 
instrumentation from code sanitizers,..
test oracle » implicit oracles
15
test case
test oracle
test input
effectiveness
efficiency
scalability
^

• Examples of non-functional requirements

• Higher performance, low energy consumption, good user ratings

• Often checked via A/B testing or canary testing

• Deploy a new feature to a small user base first.

• Deploy old version to your remaining user base.

• Compare your non-functional measures.

• Have the values changed for the worse?
test oracle » non-functional testing
16
test case
test oracle
test input
effectiveness
efficiency
scalability
^

• Examples of non-functional requirements

• Higher performance, low energy consumption, good user ratings

• Quantify side channel leakage

• High accuracy and robustness of floating-point arithmetic code
16
test oracle » non-functional testing
test case
test oracle
test input
effectiveness
efficiency
scalability
^

assuming the perfect test oracle.
software testing
Problem:

test input
*
*

Question:
What are ways to generate test inputs?
test input
17
test case
test oracle
test input
effectiveness
efficiency
scalability
^

• Manually construct test inputs

• Assertion (e.g., JUnit)

• Setup concrete program state.

• Assert a property of that state.

• Record & Replay (e.g., Selenium)

• Record a user interaction.

• Replay the recorded interaction.

• Assert the same behaviour.
test input » manual generation
18
test case
test oracle
test input
effectiveness
efficiency
scalability
^

• Automatically construct test inputs

• Blackbox: Random test input generation. No program information.

• Greybox: Guided random test input generation. Program feedback.

• Whitebox: Systematic test input generation. Analyze program code.

• Structure-aware (guided) random test input generation
test input » automatic generation
test case
test oracle
test input
effectiveness
efficiency
scalability
^





• Software testing as

• Optimization problem (Search-based Software Testing)

• Constraint satisfaction problem (Symbolic Execution)
19
test case
test oracle
test input
effectiveness
efficiency
scalability
^

test input » no bugs found 🤔
After generating a few test inputs, what
does it mean if no bugs have been found?
20
test case
test oracle
test input
effectiveness
efficiency
scalability
^

• Is your program free of bugs? Probably not 😆
https://www.cs.utexas.edu/users/EWD/ewd02xx/EWD249.PDF 21
test case
test oracle
test input
effectiveness
efficiency
scalability
^


• However, we can estimate the residual risk for

• whitebox fuzzing (Filieri, Pāsāreanu, and Wisser, “Reliability Analysis in Symbolic Pathfinder”, ICSE’13)

• blackbox fuzzing (Böhme; “STADS: Software Testing as Species Discovery”; TOSEM’18)

• greybox fuzzing (Böhme, Liyanage, and Wüstholz; “Estimating Residual Risk in Greybox Fuzzing”; ESEC/FSE’21)
21
test case
test oracle
test input
effectiveness
efficiency
scalability
^


• Is your test input generator eﬀective? Maybe? 😅
Recall, we call a test input generator as effective if it generates
at least one test input for each bug in the program. 21
test case
test oracle
test input
effectiveness
efficiency
scalability
^


• Is your test input generator eﬀective? Maybe? 😅
Recall, we call a test input generator as effective if it generates
at least one test input for each bug in the program.
How do we measure effectiveness
if there are no bugs? 🤔
21
test case
test oracle
test input
effectiveness
efficiency
scalability
^

effectiveness
How do we know if we are the best at fishing
if we never catch any fish in our lake?

• Catch fish at representative lakes (Fuzzbench)

• If we are the best at fishing in many lakes, 
then we might be the best in fishing in our lake.
effectiveness » benchmarking
22
test case
test oracle
test input
effectiveness
efficiency
scalability
^


• If we are the best at fishing in many lakes, 
then we might be the best in fishing in our lake.

• Problem: Too many lakes which may also have no fishes.
22
test case
test oracle
test input
effectiveness
efficiency
scalability
^


• Catch fish predominantly at lakes we know have catchable fish
22
test case
test oracle
test input
effectiveness
efficiency
scalability
^


• Catch fish predominantly at lakes we know have catchable fish.

• Problem 1: We don’t learn how good we are at catching fish 
that others don’t know how to catch.

• Problem 2: We still need to find and “curate” those fishes.
22
test case
test oracle
test input
effectiveness
efficiency
scalability
^

• Catch fish at representative lakes (Fuzzbench).


• Catch artificial fish to representative lakes (Lava-M, rode0day).
Looks like fish.
Swims like fish. 22
test case
test oracle
test input
effectiveness
efficiency
scalability
^




• Problem: Are artificial fish “realistic”? That is, is our performance of  
catching artificial fish indicative of our performance of catching real fish?
Looks like fish.
Swims like fish. 22
test case
test oracle
test input
effectiveness
efficiency
scalability
^




• Catch more realistic artificial fish in representative lakes (SemSeed).
22
test case
test oracle
test input
effectiveness
efficiency
scalability
^




• Catch more realistic artificial fish in representative lakes (SemSeed).
Problem: The best fuzzer for most programs
may not be the best fuzzer for my program.
😣 22
test case
test oracle
test input
effectiveness
efficiency
scalability
^

• Hypothesis: Can’t catch fish living in parts of the lake we don’t cover.

• Coverage-based evaluation: The more we covered, the better we are.
effectiveness » coverage
23
test case
test oracle
test input
effectiveness
efficiency
scalability
^



• Types of coverage:

• Code coverage, e.g, statement, branch, def-use pair, MCDC, path coverage.
23
test case
test oracle
test input
effectiveness
efficiency
scalability
^





• Input coverage, e.g, grammar or protocol coverage; pairwise testing
23
test case
test oracle
test input
effectiveness
efficiency
scalability
^





• Input coverage, e.g, grammar or protocol coverage; pairwise testing

• Requirements coverage, e.g., specification (pre-/post-condition) coverage.
23
test case
test oracle
test input
effectiveness
efficiency
scalability
^

• Hypothesis: Can’t catch real fish if we can’t catch artificial fish.

• Mutation-based evaluation: The more artificial fish we catch,  
the better we are.
effectiveness » artificial faults
24
test case
test oracle
test input
effectiveness
efficiency
scalability
^

software testing problem
test input
*
*
assuming more coverage is correlated with better bug finding.
coverage element
*
*
v

So, you are saying: Achieve 100% code coverage!
That should be easy, right?
effectiveness » cover all the things
25
test case
test oracle
test input
effectiveness
efficiency
scalability
^

So, you are saying: Achieve 100% code coverage!
That should be easy, right?
wrong.
25
test case
test oracle
test input
effectiveness
efficiency
scalability
^

Marcel Böhme, Monash University · ECOOP/ISSTA’21 Summer School · Foundations of Software Testing 25
test case
test oracle
test input
effectiveness
efficiency
scalability
^

• Problem: We don’t know how much coverage *can* be achieved 🤔
26
test case
test oracle
test input
effectiveness
efficiency
scalability
^


• We cannot compute the asymptote.
?
26
test case
test oracle
test input
effectiveness
efficiency
scalability
^



• Determining whether an element can 
be covered is as hard as determining 
whether an assertion can be violated.
if (unexpected_behavior) {
fail();
}
Can this be covered?
26
test case
test oracle
test input
effectiveness
efficiency
scalability
^



• However, we can estimate the  
asymptote during testing!
26
test case
test oracle
test input
effectiveness
efficiency
scalability
^



• However, we can estimate the  
asymptote during testing!

• Consider test input gen. as sampling.

• Cast as species discovery problem.
ACM TOSEM’18
26
test case
test oracle
test input
effectiveness
efficiency
scalability
^

test input
*
*
coverage element
*
*
v
as many as possible
v

So, it should be better to systematically
generate test inputs that cover most
of the coverage elements rather than
to randomly generate inputs, right?
effectiveness
27
test case
test oracle
test input
effectiveness
efficiency
scalability
^

So, it should be better to systematically
generate test inputs that cover most
of the coverage elements rather than
to randomly generate inputs, right?
effectiveness
wrong.
27
test case
test oracle
test input
effectiveness
efficiency
scalability
^

effectiveness
27
test case
test oracle
test input
effectiveness
efficiency
scalability
^

efficiency
Consider time!*
*[TSE’15] “A Probabilistic Analysis of the Efficiency of Automated Software Testing”
Böhme and Paul 28
test case
test oracle
test input
effectiveness
efficiency
scalability
^

efficiency
When is the most effective technique (whitebox fuzzing)
more efficient than random test generation (blackbox fuzzing)?
Consider time!*
*[TSE’15] “A Probabilistic Analysis of the Efficiency of Automated Software Testing”
Böhme and Paul 28
test case
test oracle
test input
effectiveness
efficiency
scalability
^

• Our whitebox fuzzer generates one test input per path.

• Most eﬀective! Covers all statements, branches, paths, and bugs!
efficiency » example
void crashme(char s[4]) {
if (s[0] == 'b')
if (s[1] == 'a')
if (s[2] == 'd')
if (s[3] == '!')
abort();
}
29
test case
test oracle
test input
effectiveness
efficiency
scalability
^



• Discovers the bug after 5 inputs.
if (s[0] == 'b')
if (s[1] == 'a')
if (s[2] == 'd')
if (s[3] == '!')
abort();
}
This program has five paths:

1. ****: false
2. b***: true, false
3. ba**: true, true, false
4. bad*: true, true, true, false
5. bad!: true, true, true, true, abort();
29
test case
test oracle
test input
effectiveness
efficiency
scalability
^



• Our generational blackbox fuzzer generates a random input of length 4.

• Discovers the bug after ((2-8)4)-1 ≈ 4 billion inputs, in expectation.

if (s[0] == 'b')
if (s[1] == 'a')
if (s[2] == 'd')
if (s[3] == '!')
abort();
}
29
test case
test oracle
test input
effectiveness
efficiency
scalability
^





• On my machine, this takes 6.3 seconds. On 100 machines, it takes 63 milliseconds.
if (s[0] == 'b')
if (s[1] == 'a')
if (s[2] == 'd')
if (s[3] == '!')
abort();
}
29
test case
test oracle
test input
effectiveness
efficiency
scalability
^





If our whitebox fuzzer takes too long
per input, our blackbox fuzzer outperforms!
» There is a maximum time per test input!
if (s[0] == 'b')
if (s[1] == 'a')
if (s[2] == 'd')
if (s[3] == '!')
abort();
}
29
test case
test oracle
test input
effectiveness
efficiency
scalability
^



• Our mutational blackbox fuzzer mutates a random character in a seed.

• Started with the seed bad?

• Discovers the bug after ((4-1)*(2-8))-1 ≈ 1024 inputs, in expectation.
if (s[0] == 'b')
if (s[1] == 'a')
if (s[2] == 'd')
if (s[3] == '!')
abort();
}
29
test case
test oracle
test input
effectiveness
efficiency
scalability
^



• Our mutational blackbox fuzzer mutates a random character in a seed.

• Started with the seed bad?

• Discovers the bug after ((4-1)*(2-8))-1 ≈ 1024 inputs, in expectation.
Where do we get that seed?
Discover it!
if (s[0] == 'b')
if (s[1] == 'a')
if (s[2] == 'd')
if (s[3] == '!')
abort();
}
29
test case
test oracle
test input
effectiveness
efficiency
scalability
^

[CCS’16] “Coverage-based Greybox Fuzzing as Markov Chain”
Böhme Pham, and Roychoudhury
• Our greybox fuzzer is mutational but adds inputs that increase coverage.
**** b*** (1✕ 4-1 ✕ 2-8)-1 
= 1024
****
b***
ba** (1/2 ✕ 4-1 ✕ 2-8)-1 
= 2048
****
b***
ba**
bad* (1/3 ✕ 4-1 ✕ 2-8)-1 
= 3072
****
b***
ba**
bad*
bad! (1/4 ✕ 4-1 ✕ 2-8)-1 
= 4096
Total: 10240
Seed corpus
Expected #inputs
“Interesting” 
Input
if (s[0] == 'b')
if (s[1] == 'a')
if (s[2] == 'd')
if (s[3] == '!')
abort();
}
test case
test oracle
test input
effectiveness
efficiency
scalability
^


• Started with an random test input ****

**** b*** (1✕ 4-1 ✕ 2-8)-1 
= 1024
****
b***
ba** (1/2 ✕ 4-1 ✕ 2-8)-1 
= 2048
****
b***
ba**
bad* (1/3 ✕ 4-1 ✕ 2-8)-1 
= 3072
****
b***
ba**
bad*
bad! (1/4 ✕ 4-1 ✕ 2-8)-1 
= 4096
Total: 10240
if (s[0] == 'b')
if (s[1] == 'a')
if (s[2] == 'd')
if (s[3] == '!')
abort();
}
test case
test oracle
test input
effectiveness
efficiency
scalability
^



• Discovers the bug after generating 10k inputs.

• On my machine, 150 milliseconds.

**** b*** (1✕ 4-1 ✕ 2-8)-1 
= 1024
****
b***
ba** (1/2 ✕ 4-1 ✕ 2-8)-1 
= 2048
****
b***
ba**
bad* (1/3 ✕ 4-1 ✕ 2-8)-1 
= 3072
****
b***
ba**
bad*
bad! (1/4 ✕ 4-1 ✕ 2-8)-1 
= 4096
Total: 10240
if (s[0] == 'b')
if (s[1] == 'a')
if (s[2] == 'd')
if (s[3] == '!')
abort();
}
test case
test oracle
test input
effectiveness
efficiency
scalability
^



• Discovers the bug after generating 10k inputs.

• On my machine, 150 milliseconds.

• If we prefer seeds on low-probability paths, 
it only takes 4k inputs (55 ms).

**** b*** (1✕ 4-1 ✕ 2-8)-1 
= 1024
****
b***
ba** (1 ✕ 4-1 ✕ 2-8)-1 
= 1024
****
b***
ba**
bad* (1 ✕ 4-1 ✕ 2-8)-1 
= 1024
****
b***
ba**
bad*
bad! (1 ✕ 4-1 ✕ 2-8)-1 
= 1024
Total: 4096
if (s[0] == 'b')
if (s[1] == 'a')
if (s[2] == 'd')
if (s[3] == '!')
abort();
}
test case
test oracle
test input
effectiveness
efficiency
scalability
^

• Limitation of “smarter” testing: 
If your test input generation is not fast enough,  
even a simple (guided) random test input generation  
will find more bugs in your limited time budget.
efficiency
test case
test oracle
test input
effectiveness
efficiency
scalability
^

scalability
test case
test oracle
test input
effectiveness
efficiency
scalability
Okay. Let’s take the most popular technique
and distribute it across many machines.
How does bug finding scale with #machines?
[ESEC/FSE’20] “Fuzzing: On the Exponential Cost of Vulnerability Discovery”
Böhme and Falk
^

• Google has been fuzzing OSS for about 4 years

• 25k machines; 11k+ bugs in 160+ OSS; 16k+ bugs in Chrome browser

• Discovery rate reduces. As more bugs are fixed, less new bugs found.
scalability
Böhme and Falk
test case
test oracle
test input
effectiveness
efficiency
scalability
^

• Google has been fuzzing OSS for about 4 years.

• Suppose, Google now employs 100x more machines.

• In 1 month on 2.5 million machines they find 100 vulns more.
scalability
Böhme and Falk
test case
test oracle
test input
effectiveness
efficiency
scalability
^



How long do you expect it would take
to find all of these known vulnerabilities on
*250 million* machines?
scalability
Böhme and Falk
test case
test oracle
test input
effectiveness
efficiency
scalability
^



• In 1 month on 2.5 million machines they find 100 vulns more
Given the same non-deterministic fuzzer,
finding the same bugs linearly faster
requires linearly more machines.
Do you agree?
scalability
Böhme and Falk
test case
test oracle
test input
effectiveness
efficiency
scalability
^



Now, how many undiscovered vulns
do you expect to find in 1 month
on 250 million machines?
scalability
Böhme and Falk
test case
test oracle
test input
effectiveness
efficiency
scalability
^



Given the same non-deterministic fuzzer,
finding linearly more new bugs in c months,
requires exponentially more machines.
scalability
Böhme and Falk
test case
test oracle
test input
effectiveness
efficiency
scalability
^

1 2 4 8 16 32 64 128 256 512
machines
new bugs 1 2 3 4 5 6 7 8 9
24 hrs
scalability

test input
*
*
coverage element
*
*
v
as many as possible
v






19
test case
test oracle
test input
effectiveness
efficiency
scalability



• x + y = y + x





11
test case
test oracle
test input
effectiveness
efficiency
scalability
Test Input
Expected 
Output?
Program
5
test case
test oracle
test input
effectiveness
efficiency
scalability
Unit
Test Harness
Input Expected?
6
test case
test oracle
test input
effectiveness
efficiency
scalability

test input
*
*
coverage element
*
*
v
as many as possible
v






19
test case
test oracle
test input
effectiveness
efficiency
scalability



• x + y = y + x





11
test case
test oracle
test input
effectiveness
efficiency
scalability
Test Input
Expected 
Output?
Program
5
test case
test oracle
test input
effectiveness
efficiency
scalability
Unit
Test Harness
Input Expected?
6
test case
test oracle
test input
effectiveness
efficiency
scalability
test case
test oracle
test input
effectiveness
efficiency
scalability
test case
test oracle
test input
effectiveness
efficiency
scalability
x
?

efficiency
test case
test oracle
test input
effectiveness
efficiency
scalability
^
test input
*
*
coverage element
*
*
v
as many as possible
v






19
test case
test oracle
test input
effectiveness
efficiency
scalability



• x + y = y + x





11
test case
test oracle
test input
effectiveness
efficiency
scalability
Test Input
Expected 
Output?
Program
5
test case
test oracle
test input
effectiveness
efficiency
scalability
Unit
Test Harness
Input Expected?
6
test case
test oracle
test input
effectiveness
efficiency
scalability
test case
test oracle
test input
effectiveness
efficiency
scalability
test case
test oracle
test input
effectiveness
efficiency
scalability
x
?

efficiency
test case
test oracle
test input
effectiveness
efficiency
scalability
^
test input
*
*
coverage element
*
*
v
as many as possible
v






19
test case
test oracle
test input
effectiveness
efficiency
scalability



• x + y = y + x





11
test case
test oracle
test input
effectiveness
efficiency
scalability
Test Input
Expected 
Output?
Program
5
test case
test oracle
test input
effectiveness
efficiency
scalability
Unit
Test Harness
Input Expected?
6
test case
test oracle
test input
effectiveness
efficiency
scalability
test case
test oracle
test input
effectiveness
efficiency
scalability
1 2 4 8 16 32 64 128 256 512
machines
new bugs 1 2 3 4 5 6 7 8 9
24 hrs
scalability
test case
test oracle
test input
effectiveness
efficiency
scalability
x
?

efficiency
test case
test oracle
test input
effectiveness
efficiency
scalability
^
1 2 4 8 16 32 64 128 256 512
machines
new bugs 1 2 3 4 5 6 7 8 9
24 hrs
scalability
test input
*
*
coverage element
*
*
v
as many as possible
v






19
test case
test oracle
test input
effectiveness
efficiency
scalability



• x + y = y + x





11
test case
test oracle
test input
effectiveness
efficiency
scalability
Test Input
Expected 
Output?
Program
5
test case
test oracle
test input
effectiveness
efficiency
scalability
Unit
Test Harness
Input Expected?
6
test case
test oracle
test input
effectiveness
efficiency
scalability
test case
test oracle
test input
effectiveness
efficiency
scalability
test case
test oracle
test input
effectiveness
efficiency
scalability
x
?
If you want to take a deeper dive:
* Attend workshops and talks @ ECOOP/ISSTA’21 this week!

* Read our interactive text book: The Fuzzing Book

* Read our IEEE Software article: “Fuzzing: Challenges and Reflections”

* Apply for PhD / PostDoc in my group at MPI-SP, Bochum, Germany.

Web: https://mboehme.github.com Twitter: @mboehme_

Foundations Of Software Testing

Recommended

Recommended

More Related Content

Similar to Foundations Of Software Testing

Similar to Foundations Of Software Testing (20)

More from mboehme

More from mboehme (7)

Recently uploaded

Recently uploaded (20)

Foundations Of Software Testing