1.
Universiteit Antwerpen
Reproducible Crashes:
Fuzzing Pharo by Mutating the Test
Methods
Mehrdad Abdi — Henrique Rocha — Serge Demeyer
VST 2021 (March 9th)

2.
VST2021 © Serge Demeyer
• Background
- Fuzzing
- Mutation Coverage
- Test Amplification
• Fuzzing by amplification
- Motivating examples
- Proposed approach
- Challenge
• Conclusions
Outline
2

3.
Fuzz
Testing
Mutation
Coverage
Test
Amplification

4.
VST2021 © Serge Demeyer
Testing
4
Program
Under
Test
Valid Input
Expected output
Software Testing is the process of executing a program or system with
the intent of finding errors.
(Myers, Glenford J., The art of software testing. Wiley, 1979

5.
VST2021 © Serge Demeyer
Fuzz Testing
5
Program
Under
Test
Unexpected Input
Crash/Freeze
Fuzzing (or Fuzz Testing) is an automated testing technique to verify the
robustness of a program-under-test.
Valid input is replaced by random values with the goal to force the
program-under-test into unexpected exceptional behavior.

6.
VST2021 © Serge Demeyer
Black Box Fuzzing
6
Access to
execution
Mutating valid
inputs

7.
VST2021 © Serge Demeyer
White Box Fuzzing
7
Access to code,
models, specs, …
Program Analysis

8.
VST2021 © Serge Demeyer
Grey Box Fuzzing
8
Limited Access
(bytecode, trace, …)
Reverse
Engineering

9.
VST2021 © Serge Demeyer
Code Coverage
9
Program
Under
Test
Valid Input Expected output
coverage

10.
VST2021 © Serge Demeyer
Line Coverage (Statement / Branch / Path / …)
10
c++
java

11.
VST2021 © Serge Demeyer
Mutation Testing
11
int compare(int v1,
int v2)
{if (v1 <v2)
return 1;
return -1
;
}
int compare(int v1,
int v2)
{if (v1 >=v2)
return 1;
return -1
;
}
🙂
🙁

12.
VST2021 © Serge Demeyer
Mutation Coverage
12

13.
VST2021 © Serge Demeyer
Test Amplification
13
Program
Under
Test
Valid Input Expected output
coverage
+coverage
Extra Input
Extra Input
+Extra Input +Extra output

14.
VST2021 © Serge Demeyer
Test amplification Definition
14
[Danglot-19] Benjamin Danglot, Oscar Vera-Perez, Zhongxing Yu, Andy Zaidman, Martin Monperrus and Benoit
Baudry. 2019. A snowballing literature study on test ampli
fi
ca
ti
on. Journal of Systems and Software.
Test amplification consists of exploiting the knowledge of a large
number of test cases, in which developers embed meaningful
input data and expected properties in the form of oracles, in
order to enhance these manually written tests with respect to
an engineering goal. [Danglot-19]
mutation coverage

15.
VST2021 © Serge Demeyer
Fuzzing by Amplification
15
[Danglot-19] Benjamin Danglot, Oscar Vera-Perez, Zhongxing Yu, Andy Zaidman, Martin Monperrus and Benoit
Baudry. 2019. A snowballing literature study on test ampli
fi
ca
ti
on. Journal of Systems and Software.
Test amplification consists of exploiting the knowledge of a large
number of test cases, in which developers embed meaningful
input data and expected properties in the form of oracles, in
order to enhance these manually written tests with respect to
an engineering goal. [Danglot-19]
Detecting unexpected terminations
(crashes, freezes, …)

16.
VST2021 © Serge Demeyer
Fuzzing by Amplification
16
Image from: https://www.enwild.com/snow-peak-colored-titanium-spork.html
Fuzzing?
Amplification?

17.
VST2021 © Serge Demeyer
Motivating example (1/2)
17

18.
VST2021 © Serge Demeyer
Motivating example (2/2)
18

19.
VST2021 © Serge Demeyer
Proposed approach
19
Profile Test Method

20.
VST2021 © Serge Demeyer
Proposed approach
20
(2) Assertion Removal

21.
VST2021 © Serge Demeyer
Proposed approach
21
(2) Input Amplification
Input Amplification Operators
• Literal mutation
• Nullify objects
• Remove statements
• Duplicate statements
• Add new statements
• Change helper methods
• …

22.
VST2021 © Serge Demeyer
Proposed approach
22
(3) Create new tests
+ install in the test suite

23.
VST2021 © Serge Demeyer
Proposed approach
23
(4) Execute

24.
VST2021 © Serge Demeyer
Proposed approach
24
(5) Sandboxing

25.
VST2021 © Serge Demeyer
Proposed approach
25
Observation

26.
VST2021 © Serge Demeyer
Proposed approach
26
(6) Reporting

27.
VST2021 © Serge Demeyer
Challenge
27
If it looks like a duck, swims like a duck, and quacks like a duck, then it probably is a duck.

28.
VST2021 © Serge Demeyer
Challenge
28
Explicit
Types
Profiling
White Box Fuzzing

29.
VST2021 © Serge Demeyer
Smalltalk Community
29
Cool dude!
Mehrdad Abdi, Henrique Rocha, and Serge Demeyer. Reproducible crashes: Fuzzing pharo
by mutating the test methods. In Proceedings IWST 2020 (International Workshop on
Smalltalk Technologies). ESUG, 2020.

30.
VST2021 © Serge Demeyer
Testing Community
30
Related Work?
• Fuzzing by mutating test code
• Fuzzing / Test Amplification
- dynamically typed languages
(Python, Javascript, …)
feedback and/or suggestions
serge.demeyer@uantwerpen.be