This document provides an overview and introduction to fuzzing performance. It discusses how fuzzing tools from Codenomicon, like Defensics, can scale testing through parallel and distributed processing. Metrics like test cases per second are discussed as measures of fuzzing performance. The document demonstrates how Defensics is able to generate a high volume of test cases and load through features like multi-threaded execution and running fuzzers in parallel. It provides an example showing Defensics was able to generate over 50 times the load of a single client.

1. WWW.CODENOMICON.COM
Fuzzing 101:
Fuzzing Performance
Ari Takanen,
CTO of Codenomicon
Nov 10th, 2010

2. WWW.CODENOMICON.COM
2
 Fuzzing 101:
• The webcast series for
fuzzing industry
• Vendor neutral
presentations on
fuzzing technologies
and use-cases
• Includes invited
speakers from the
industry
 Codenomicon:
• Fuzzing research since
1996
• 2001, Spinoff from
University of Oulu
• 50-100% annual growth
in number of customers
and revenues in fuzzing
industry
About Fuzzing 101 and Codenomicon

3. WWW.CODENOMICON.COM
3
About Ari Takanen
 The Past: Researcher and Lecturer
• 1998-2002
• University of Oulu
• OUSPG/PROTOS research group
• Software Quality related lectures
 The Present: Entrepreneur and Evangelist
• 2001-today
• CTO of Codenomicon
• Evangelist: 10 conference talks every year
• Author of two books:
• VoIP Security
• Fuzzing

4. WWW.CODENOMICON.COM
4
Agenda
 Intro:
• Fuzzing Overview
• Fuzzing Performance
 Demo in Theory
• Multi-threaded execution
• Test execution optimization
• Command-line use
 And if time allows: Demo in Practice for the
interested

5. WWW.CODENOMICON.COM
Fuzzing Overview
Positive results from negative testing!

6. WWW.CODENOMICON.COM
6
What is Testing - Three Flavors
 Feature/Conformance tests validate positive
requirements with use cases and test cases
 Performance/Load testing repeats one of the
feature tests to validate performance limits
 Robustness/Fuzz testing uses misuse cases to
verify software reliability with unexpected inputs

7. WWW.CODENOMICON.COM
7
Security Vulnerability = Just A Bug

8. WWW.CODENOMICON.COM
8
Codenomicon Labs Test Results
http://www.codenomicon.com/labs/results

9. WWW.CODENOMICON.COM
9
Fuzz Test Effectiveness against WiFi

10. WWW.CODENOMICON.COM
10
Some Helpful Definitions
 Vulnerability – a weakness in software, a bug
 Threat/Attack – exploit/worm/virus against a
specific vulnerability
 Protocol Modeling – Technique for explaining
interface message sequences and message
structures
 Fuzzing – process and technique for security
testing
 Anomaly – abnormal or unexpected input
 Failure – crash, busy-loop, memory corruption,
or other indication of a bug in software

11. WWW.CODENOMICON.COM
11
Fuzzing In Short
 Fuzzing means crash-testing
 Also called:
• Negative testing
• Robustness testing
• Grammar testing
 Based on sending systematically broken
(rarely random) inputs to a software, in order
to crash it
 We will ignore random mutator fuzzers for now
 Two techniques of smart model-based fuzzers:
• Template-based
• Specification-based

12. WWW.CODENOMICON.COM
12
Model Based Fuzzing Techniques
 Template Based Fuzzing
• Quality of tests is based on the used seed and
modeling technique
• Very quick to develop, but slow to run
• Editing requires deep protocol know-how
• Good for testing around known vulnerabilities
 Specification Based Fuzzing
• Full test coverage
• Always repeatable
• Short test cycle, more optimized tests
• Easy to edit and add tests

13. WWW.CODENOMICON.COM
13
 Precision is about attack
surface/protocol coverage
 All interfaces/protocols
tested?
 All message sequences
tested?
 All message structures
tested?
 All data definitions tested?
 All “tags” (values) tested?
 Accuracy is about
anomaly coverage
 Anomaly categories?
SQL? Buffer overflow?
 All values: 0..65k, a..z,
0x00..0x255 ?
 Combinations of
anomalies?
Coverage

14. WWW.CODENOMICON.COM
Anomaly Coverage Selection

15. WWW.CODENOMICON.COM
15
Traffic Capture Fuzzing

16. WWW.CODENOMICON.COM
16
Traffic Capture Fuzzing Results
 Test against samba seems to find zero-day

17. WWW.CODENOMICON.COM
17
Key Questions in regards to Fuzzing
 Test Coverage:
• Q: Which interfaces and protocols are tested?
• A: Understanding of threat models and attack surface
are the most critical starting point to fuzzing
• Q: How well something is tested?
• A1: Specification coverage explains how deep
knowledge the fuzzer has on a tested interface, and is
typically represented by protocol models or protocol
templates
• A2: Anomaly coverage explains how well each protocol
element is tested, and is the hardest metric to measure

18. WWW.CODENOMICON.COM
Fuzzing Performance
Faster, Better, More!

19. WWW.CODENOMICON.COM
Fuzzing Scalability
 Software-based Fuzzers scale for all testing needs
• The throughput depends only on the available hardware
• The entire Network Under Test (NUT) can be virtualized
• Software fuzzers can attack every interface, and against all
protocol layers
• XML
• HTTP
• TLS
• TCP/IP

20. WWW.CODENOMICON.COM
20
Why Fast Speed Fuzzing?
 “One of the most important aspects of
fuzzing is how fast you can execute test
cases. The faster you can execute test
cases, the more test cases you can run
and the more vulnerabilities you will find.”
- Dr. Charlie Miller from Independent Security
Evaluators. "

21. WWW.CODENOMICON.COM
Generating Load with Defensics
 Full model-based message
sequences
 Options for monitoring,
instrumentation, fuzzing, …
 Amount of available CPUs and
Logging Level impact performance
 Some speed records (test cases per
second):
• TLS: 2.400 tc/s
• IPv6: 4.500 tc/s
• HTTP: 16.000 tc/s
• DNS: 41.000 tc/s (with capture replay)

22. WWW.CODENOMICON.COM
22
Motivation for our Performance Study
 Test our test generation efficiency for different
type of protocol fuzzers using high-end
servers
 Analyze benefits (test efficiency, ROI) of
software-based fuzzers to appliances

23. WWW.CODENOMICON.COM
23
Raw Fuzzing Performance Metrics
 Packets or bytes per second does not really
work as each fuzz test case can consist of
several packets back and forth
 Test cases, or full protocol transactions,
generated by each suite provides more insight
 Performance is generated by having fuzzers
running in parallel
 These result in “test cases per second” or tc/s

24. WWW.CODENOMICON.COM
24
Defensics Benefits
 No hardware constraints
 No operating system constraints
 Parallel processes and threads

25. WWW.CODENOMICON.COM
25
Performance Bottlenecks and Metrics
 Network bandwidth:
• Can be measured with number of full protocol
transactions, packets, and data volume
 CPU usage by different components of
Defensics
• Test generator(s)
• Test driver(s)
 Memory usage of Defensics components
 Most interesting metric:
• New type of bugs related to performance?

26. WWW.CODENOMICON.COM
26
Fuzz Test Can Generate 50x Load

27. WWW.CODENOMICON.COM
Generating Load with Dell R910

28. WWW.CODENOMICON.COM
Quick-Start to Fast-Speed Fuzzing
Demo

29. WWW.CODENOMICON.COM
29
Test Plans Using Command-Line
 With some fuzzers, you can run the tools
faster on command-line
 With Defensics, the GUI is sometimes even
faster
 E.g. with a laptop with dual-core Intel T7500
processor (with default tool settings):
• 10.000 x valid-case in loop
• using GUI: 30 seconds
• using command-line: 35 seconds

30. WWW.CODENOMICON.COM
30
Running Fuzzers in Parallel
 Defensics GUI has options to run several
suites in parallel
• 2 x 5.000 x valid-case in loop
• using GUI: 22 seconds (about 50-100% faster)

31. WWW.CODENOMICON.COM
31
Using Multi-threaded Mode
 When CPU resources are available, multi-
treaded mode is often best way to speed up
testing
 Uses less memory
 Note that all parallel execution creates more
false positives
 Let’s look at that and other speed optimization
hints now in live demo...

32. WWW.CODENOMICON.COM
Conclusions
Not so fast!

33. WWW.CODENOMICON.COM
33
Performance Value
 Users can now…
• Increase test coverage through parallel tests
• Speed up testing
• Find performance bottle-necks
• Integrate Defensics also in other functional tests
 Remember that Fuzzing ROI is measured in:
• TCO vs. Value of Security Findings / Cost of Security
Compromise
• Reduced maintenance costs, failure reproduction costs
• Better customer confidence, higher retention rate of
customers
• In performance: software-based tools can be significantly
cheaper (and faster) than appliances

34. WWW.CODENOMICON.COM
34
More News from Codenomicon
 Facebook:
• Become fan of Codenomicon and Fuzzing
 Twitter:
• CodenomiconLTD
 Codenomicon Website:
• Newsletter every second month
 Customer portal
• The Backstage! Contact us for access!

35. WWW.CODENOMICON.COM
35
Our Book On Fuzzing!
 http://www.fuzz-test.com/book/
 Takanen, DeMott and Miller:
“Fuzzing for Software Security
Testing and Quality Assurance”
 Aimed at the general public, you do
not need to be a security specialist
to read this book
 Purpose of the book is to teach
next-gen testing approaches to:
• Software practitioners
• Security engineers
• Academics

36. WWW.CODENOMICON.COM
PROACTIVE SECURITY AND ROBUSTNESS SOLUTIONS
THANK YOU – QUESTIONS?
“Thrill to the excitement of the chase!
Stalk bugs with care, methodology,
and reason. Build traps for them.
....
Testers!
Break that software (as you must) and
drive it to the ultimate
- but don’t enjoy the programmer’s
pain.”
[from Boris Beizer]