The document discusses the effectiveness, efficiency, and scalability of fuzzing for automated software testing, highlighting its role in vulnerability discovery. It emphasizes the correlation between code coverage and bug finding, and critiques existing approaches to measuring the effectiveness of testing without identified bugs. Additionally, it suggests that maximizing coverage is crucial for increasing bug discovery and that simpler techniques may outperform more complex methods if they are faster.

1. The Curious Case of Fuzzing
for Automated Software Testing
Marcel Böhme
Software Security
MPI-SP & Monash
Keywords: Vulnerability Discovery,
Automated Software Testing,
E
ff
ectiveness, E
ffi
ciency,
Scalability, Guarantees

2. • Fuzzing for Automatic Vulnerability Discovery
• Making machines attack other machines.
• Focus on scalability, e
ffi
ciency, and e
ff
ectiveness.
• Foundations of Software Security
• Assurances in Software Security
• Fundamental limitations of existing approaches
• Drawing from multiple disciplines (information theory, biostatistics)
whoami
2
* Looking for PhD students :)

3. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
Test Input
Expected
Output?
Program
software testing

4. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
Test Input
Expected
Output?
Program
software testing

5. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
Test Input
Expected
Output?
Program
The Oracle
checks whether
software testing

6. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
Test Input
Expected
Output?
Program
software testing

7. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
Test Input
Expected
Output?
Program
software testing

8. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
Test Input
Expected
Output?
Program
The Fuzzer
auto-generates
software testing

9. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
software testing
Test Input
Expected
Output?
Program
The Fuzzer
auto-generates

10. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
software testing
Test Input
Expected
Output?
Program
The Fuzzer
auto-generates
fuzzing == automated

11. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
fuzzing :: properties
• e
ff
ectiveness
• e
ffi
ciency
• scalability

12. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
fuzzing :: properties
• e
ff
ectiveness
• e
ffi
ciency
• scalability

13. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
fuzzing :: properties
• e
ff
ectiveness
• e
ffi
ciency
• scalability

14. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
• You’ve been generating test cases for your program.
• No bugs found! 👍
• Is your program free of bugs?
fuzzing :: effectiveness
[ESEC/FSE’21] “Estimating Residual Risk in Greybox Fuzzing
”
Böhme, Liyanage, and Wüstholz

15. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
• You’ve been generating test cases for your program.
• No bugs found! 👍
• Is your program free of bugs?
• Probably not. 😆
fuzzing :: effectiveness
[ESEC/FSE’21] “Estimating Residual Risk in Greybox Fuzzing
”
Böhme, Liyanage, and Wüstholz

16. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
• You’ve been generating test cases for your program.
• No bugs found! 👍
• Is your program free of bugs?
• Probably not. 😆
• Is your test case generation technique e
ff
ective?
fuzzing :: effectiveness
[ESEC/FSE’21] “Estimating Residual Risk in Greybox Fuzzing
”
Böhme, Liyanage, and Wüstholz

17. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
• You’ve been generating test cases for your program.
• No bugs found! 👍
• Is your program free of bugs?
• Probably not. 😆
• Is your test case generation technique e
ff
ective?
• Maybe? 😅
fuzzing :: effectiveness
[ESEC/FSE’21] “Estimating Residual Risk in Greybox Fuzzing
”
Böhme, Liyanage, and Wüstholz

18. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
• You’ve been generating test cases for your program.
• No bugs found! 👍
• Is your program free of bugs?
• Probably not. 😆
• Is your test case generation technique e
ff
ective?
• Maybe? 😅
•
🤔
• How do you even measure e
ff
ectiveness if there are no bugs?
fuzzing :: effectiveness
[ESEC/FSE’21] “Estimating Residual Risk in Greybox Fuzzing
”
Böhme, Liyanage, and Wüstholz

19. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
• How do you even measure e
ff
ectiveness if there are no bugs?
fuzzing :: effectiveness

20. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
• How do you even measure e
ff
ectiveness if there are no bugs?
fuzzing :: effectiveness
Code Coverage!

21. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
• How do you even measure e
ff
ectiveness if there are no bugs?
fuzzing :: effectiveness
Code Coverage!

22. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
• How do you even measure e
ff
ectiveness if there are no bugs?
fuzzing :: effectiveness
Code Coverage!
So, you are saying: Achieve 100% coverage.
That should be easy, right?

23. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
• How do you even measure e
ff
ectiveness if there are no bugs?
fuzzing :: effectiveness
Code Coverage!
So, you are saying: Achieve 100% coverage.
That should be easy, right?
Wrong.

24. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
• How do you even measure e
ff
ectiveness if there are no bugs?
fuzzing :: effectiveness
Code Coverage!

25. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
• How do you even measure e
ff
ectiveness if there are no bugs?
fuzzing :: effectiveness
Code Coverage!

26. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
• How do you even measure e
ff
ectiveness if there are no bugs?
fuzzing :: effectiveness
Code Coverage!
We cannot compute S!
As hard as software verification.

27. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
• How do you even measure e
ff
ectiveness if there are no bugs?
fuzzing :: effectiveness
Code Coverage!

28. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
• How do you even measure e
ff
ectiveness if there are no bugs?
fuzzing :: effectiveness
Code Coverage!

29. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
• How do you even measure e
ff
ectiveness if there are no bugs?
fuzzing :: effectiveness
Code Coverage!
So, you are saying: As a fuzzer achieves
more coverage, it also finds more bugs, right?
[ICSE’22] “On the Reliability of Coverage-based Fuzzer Benchmarking
”
Böhme, Szekeres, and Metzmann

30. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
• How do you even measure e
ff
ectiveness if there are no bugs?
fuzzing :: effectiveness
Code Coverage!
So, you are saying: As a fuzzer achieves
more coverage, it also finds more bugs, right?
Right!
[ICSE’22] “On the Reliability of Coverage-based Fuzzer Benchmarking
”
Böhme, Szekeres, and Metzmann

31. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
• How do you even measure e
ff
ectiveness if there are no bugs?
fuzzing :: effectiveness
Code Coverage!

32. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
• How do you even measure e
ff
ectiveness if there are no bugs?
fuzzing :: effectiveness
Very strong correlation with bug finding!
Code Coverage!

33. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
• How do you even measure e
ff
ectiveness if there are no bugs?
fuzzing :: effectiveness
Code Coverage!
So, you are saying: We can compare fuzzers in terms
of coverage achieved and declare the winner as the
most effective at bug finding, right?
[ICSE’22] “On the Reliability of Coverage-based Fuzzer Benchmarking
”
Böhme, Szekeres, and Metzmann

34. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
• How do you even measure e
ff
ectiveness if there are no bugs?
fuzzing :: effectiveness
Code Coverage!
So, you are saying: We can compare fuzzers in terms
of coverage achieved and declare the winner as the
most effective at bug finding, right?
Wrong.
[ICSE’22] “On the Reliability of Coverage-based Fuzzer Benchmarking
”
Böhme, Szekeres, and Metzmann

35. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
fuzzing :: effectiveness
Worst in Coverage
Best in Bug Finding
[ICSE’22] “On the Reliability of Coverage-based Fuzzer Benchmarking
”
Böhme, Szekeres, and Metzmann

36. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
fuzzing :: effectiveness
So, it should be better to systematically
generate test inputs that cover most
of the coverage elements rather than
to randomly generate inputs, right?

37. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
fuzzing :: effectiveness
So, it should be better to systematically
generate test inputs that cover most
of the coverage elements rather than
to randomly generate inputs, right?
Wrong.

38. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
fuzzing :: effectiveness

39. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
fuzzing :: efficiency
[ESEC/FSE’14] “On the Ef
fi
ciency of Automated Software Testing
”
Böhme and Paul

40. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
fuzzing :: efficiency
time budget
[ESEC/FSE’14] “On the Ef
fi
ciency of Automated Software Testing
”
Böhme and Paul

41. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
fuzzing :: efficiency
time budget
[ESEC/FSE’14] “On the Ef
fi
ciency of Automated Software Testing
”
Böhme and Paul

42. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
fuzzing :: efficiency
time budget
[ESEC/FSE’14] “On the Ef
fi
ciency of Automated Software Testing
”
Böhme and Paul

43. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
fuzzing :: efficiency
time budget
As we increase the test input
generation time for the most effective
technique, it will achieve less and less
coverage within the time budget.
[ESEC/FSE’14] “On the Ef
fi
ciency of Automated Software Testing
”
Böhme and Paul

44. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
fuzzing :: efficiency
Even the most effective technique
is outperformed by random testing,
if it takes too much time
to generate the test inputs.
[ESEC/FSE’14] “On the Ef
fi
ciency of Automated Software Testing
”
Böhme and Paul

45. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
fuzzing :: scalability
[ESEC/FSE’20] “Fuzzing: On the Exponential Cost of Vulnerability Discovery
”
Böhme and Falk

46. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
fuzzing :: scalability
[ESEC/FSE’20] “Fuzzing: On the Exponential Cost of Vulnerability Discovery
”
Böhme and Falk
If we increase the #machines exponentially,
we find the same bugs exponentially faster,
right?

47. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
fuzzing :: scalability
[ESEC/FSE’20] “Fuzzing: On the Exponential Cost of Vulnerability Discovery
”
Böhme and Falk
If we increase the #machines exponentially,
we find the same bugs exponentially faster,
right?
Right!

48. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
fuzzing :: scalability
[ESEC/FSE’20] “Fuzzing: On the Exponential Cost of Vulnerability Discovery
”
Böhme and Falk
If we increase the #machines exponentially,
the number of new bugs found in the same time
increases exponentially, right?

49. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
fuzzing :: scalability
[ESEC/FSE’20] “Fuzzing: On the Exponential Cost of Vulnerability Discovery
”
Böhme and Falk
If we increase the #machines exponentially,
the number of new bugs found in the same time
increases exponentially, right?
Wrong.

50. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
fuzzing :: scalability
1 2 4 8 16 32 64 128 256 512
machines
new bugs 1 2 3 4 5 6 7 8 9
24 hrs

51. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
fuzzing :: summary

52. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
fuzzing :: summary
• Maximize coverage to increase #bugs found.
Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
• How do you even measure effectiveness if there are no bugs?
fuzzing :: effectiveness
Very strong correlation with bug finding!
Code Coverage!

53. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
fuzzing :: summary
• Maximize coverage to increase #bugs found.
• Compare fuzzers in terms of *both*, coverage and bug
fi
nding.
Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
fuzzing :: effectiveness
Worst in Coverage
Best in Bug Finding
[ICSE’22] “On the Reliability of Coverage-based Fuzzer Benchmarking”
Böhme, Szekeres, and Metzmann

54. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
fuzzing :: summary
• Maximize coverage to increase #bugs found.
• Compare fuzzers in terms of *both*, coverage and bug
fi
nding.
• Dumb and fast is better than smart and slow.
Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
fuzzing :: efficiency
time budget
As we increase the test input
generation time for the most effective
technique, it will achieve less and less
coverage within the time budget.
[ESEC/FSE’14] “On the Efficiency of Automated Software Testing”
Böhme and Paul
Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
fuzzing :: efficiency
time budget
As we increase the test input
generation time for the most effective
technique, it will achieve less and less
coverage within the time budget.
[ESEC/FSE’14] “On the Efficiency of Automated Software Testing”
Böhme and Paul

55. Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
fuzzing :: summary
• Maximize coverage to increase #bugs found.
• Compare fuzzers in terms of *both*, coverage and bug
fi
nding.
• Dumb and fast is better than smart and slow.
• Bug
fi
nding comes at an exponential cost.
Marcel Böhme, Max Planck Institute for Security and Privacy · RUB Tag der Informatik · The Curious Case of Fuzzing for Automated Software Testing
fuzzing :: scalability
1 2 4 8 16 32 64 128 256 512
machines
new bugs 1 2 3 4 5 6 7 8 9
24 hrs
* Looking for PhD students :)