Fuzzing is a software testing technique that involves providing invalid, unexpected, or random data as inputs to a computer program to detect bugs. Coverage-guided fuzzing uses genetic algorithms to generate inputs that maximize code coverage. It is effective at finding security bugs like overflows, memory errors, and crashes. The presenter demonstrates finding 13 bugs in Boost regex in 30 minutes using libFuzzer. Fuzzing is widely used at Google to test critical software like browsers, libraries, and the Linux kernel due to its ability to find many bugs without requiring test cases.
1. Fuzzing:
The New Unit Testing
C++ Russia 2017, Moscow, Feb 25
Dmitry Vyukov, dvyukov@, Google
2. Agenda
● What is fuzzing
● Coverage-guided fuzzing
● Small tutorial
● How to write effective fuzzers
● Fuzzing@Google
3. What is Fuzzing?
wikipedia.org/wiki/Fuzz_testing:
Fuzz testing or fuzzing is a software testing technique, often automated or
semi-automated, that involves providing invalid, unexpected, or random data to
the inputs of a computer program.
4. Who cares?
- We are not testing/checking anything!
- Random data will not trigger any bugs!
5. Fuzzing can find lots of bugs
- With the help of sanitizers:
- Use-after-free, buffer overflows
- Uses of uninitialized memory
- Memory leaks
- Data races, deadlocks
- Int/float overflows, bitwise shifts by invalid amount (other UB)
- Plain crashes:
- NULL dereferences, uncaught exceptions, div-by-zero
- Resource usage bugs:
- Memory exhaustion, hangs or infinite loops, infinite recursion (stack overflows)
- Logical bugs (lots of, see below)
6. Data is not necessary "white noise"
- There is number of tricks to generate "not so random" data
- May or may not require some human help
- If used correctly achieves very impressive code coverage
7. What can be fuzzed?
Anything that consumes complex inputs:
● Parsers of any kind (xml, json, asn.1, pdf, truetype, ...)
● Media codecs (audio, video, raster & vector images, etc)
● Network protocols (HTTP, RPC, SMTP, MIME...)
● Crypto (boringssl, openssl)
● Compression (zip, gzip, bzip2, brotli, ...)
● Formatted output (sprintf, template engines)
● Compilers and interpreters (Javascript, PHP, Perl, Python, Go, Clang, ...)
● Regular expression matchers (PCRE, RE2, libc’s regcomp)
● Text/UTF processing (icu)
● Databases (SQLite)
● Browsers, text editors/processors (Chrome, vim, OpenOffice)
● OS Kernels (Linux), drivers, supervisors and VMs
Must have for everything that consumes untrusted inputs, open to internet or otherwise security sensitive.
8. Types of Fuzzers
- Grammar-based generation
- Generate random inputs according to grammar rules
- Peach, packetdrill, csmith, gosmith, syzkaller
- Blind mutation
- Requires a corpus of representative inputs, apply random mutations to them
- ZZUF, Radamsa
- Grammar reverse-engineering
- Learn grammar from existing inputs using algorithmic approach of machine learning
- Sequitur algorithm, go-fuzz
- Symbolic execution + SAT solver
- Synthesize inputs with maximum coverage using black magic
- KLEE
- Coverage-guided fuzzers
- Genetic algorithm that strives to maximize code coverage
- libFuzzer, AFL, honggfuzz, syzkaller
- Hybrid
9. Coverage-guided fuzzing
Build the program with code coverage instrumentation;
Collect initial corpus of inputs (optional);
while (true) {
Choose a random input from corpus and mutate it;
Run the target program on the input, collect code coverage;
If the input gives new coverage, add mutation back to the corpus;
}
10. Coverage-guiding in action
if input[0] == '{' {
if input[1] == 'i' && input[2] == 'f' {
if input[3] == '(' {
input[input[4]] = input[5]; // potential OOB write
}
}
}
Requires "{if(" input to crash, ~2^32 guesses to crack when blind.
Coverage-guiding:
Guess "{" in ~2^8, add to corpus.
Guess "{i" in ~2^8, add to corpus.
Guess "{if" in ~2^8, add to corpus.
Guess "{if(" in ~2^8, add to corpus.
Total: ~2^10 guesses.
See: AFL: Pulling JPEGs out of thin air
12. Coverage flavours
Basic blocks:
... (A)
if (...) {
... (B)
}
... (C)
-fsanitize-coverage=bb
Edges:
... (A)
if (...) {
... (B)
}
... (C)
-fsanitize-coverage=trace-pc-guard
Gives better feedback signal.
Counters:
for (...) {
... (hit N times)
}
-fsanitize-coverage=8bit-counters
Gives better feedback signal
for loops and recursion.
13. Cracking hashes
What about more complex cases?
if (*(uint32_t*)input == crc32(input+4, size-4)) {...}
if (*(uint64_t*)input == 0xBCEBC041BADBALL) {...}
14. Cracking hashes
Intercept comparison operations:
● compiler intercepts int comparisons (-fsanitize-coverage=trace-cmp)
● runtime intercepts strcmp/memcmp and friends
Several possibilities:
● extract int/string literals and insert them into inputs
● find one comparison operand in the input and replace with the other operand
● use PC^POPCNT(op1^op2) as "coverage" signal (Hamming distance)
15. Dictionaries
● User-provided
○ e.g. for HTTP: "HTTP/1.1", "Host", "Accept-Encoding"
● Automatically extracted from program
○ memcpy(input, "HTTP/1.1", 8)
16. Tutorial
"...one of the most highly regarded and expertly designed C++ library projects in the world"
boost.regex
(latest version 1.63, in boost since 1.18)
17. Tutorial: fuzzing function
As simple as:
int LLVMFuzzerTestOneInput(const uint8_t * Data, size_t Size) {
try {
std::string str((char*)Data, Size);
boost::regex e( str);
boost::match_results<std::string::const_iterator> what;
boost::regex_match(str, what, e, boost::match_default);
} catch (const std::exception&) {}
return 0;
}
18. Tutorial: building (the hard part)
1. Build boost with coverage and AddressSanitizer:
./b2 cxxflags="-fsanitize-coverage=trace-pc-guard -fsanitize=address" toolset=clang install
2. Build fuzzer with coverage, AddressSanitizer and libFuzzer:
clang++ fuzzer.cc -fsanitize-coverage=trace-pc-guard -fsanitize=address libFuzzer.a
The rest is at tutorial.libfuzzer.info
20. 30 minutes, 13 bugs (ticket/12818):
AddressSanitizer: heap-buffer-overflow perl_matcher.hpp:132 in re_skip_past_null
AddressSanitizer: heap-buffer-overflow basic_regex_parser.hpp:2599 in parse_perl_extension
AddressSanitizer: heap-buffer-overflow perl_matcher.hpp:221 in re_is_set_member
AddressSanitizer: heap-buffer-overflow perl_matcher.hpp:166 in re_is_set_member
AddressSanitizer: heap-buffer-overflow interceptors.inc:278 in strlen
AddressSanitizer: stack-overflow basic_regex_creator.hpp:1054 in create_startmap
AddressSanitizer: SEGV on unknown address 0x0000000016e0
MemorySanitizer: use-of-uninitialized-value perl_matcher.hpp:166 in re_is_set_member
basic_regex_parser.hpp:904: runtime error: shift exponent 325804978 is too large for 32-bit type 'unsigned int'
basic_regex_parser.hpp:2599: runtime error: load of value 56794092, which is not a valid value for type 'syntax_element_type'
a.out: perl_matcher_common.hpp:606: Assertion `r.first != r.second' failed
Direct leak of 4096 byte(s) in 1 object(s) allocated in get_mem_block regex.cpp:204
ALARM: working on the last Unit for 17 seconds
Will find more when these are fixed!
Results
21. Finding logical bugs
Not only security/stability
- But we don't know the right result!
- Use your imagination!
22. Finding logical bugs
● sanity checks on results
○ uncompressesed image decoder: 100 byte input -> 100 MB output?
○ function returns both error and object, or no error and no object
○ know that some substring must present in output, but it is not
○ encrypt, check that decryption with wrong key fails
● sometimes we do know the right result
○ any sorting: check that each element is present, check that it's not descending
○ building a trie: check size, all elements are present
● asserts
○ assert(a == b)
23. Finding logical bugs
Round-trip:
● encode-decode
● serialize-deserialize
● compress-decompress
● encrypt-decrypt
● assemble-disassemble
Checks:
● decode-encode: check that encode don't fail
● decode-encode-decode: check that second decode don't fail
● decode-encode-decode: check that decode results are equal
● encode-decode-encode: check that encode results are equal
Very powerful technique.
24. Finding logical bugs
Comparing two (or more) implementations gives phenomenal results:
● check that output is equal
● or at least check that ok/fail result is the same
○ e.g. gcc and clang both accept or reject the code
But I don't want to write the second impl!
● there can be several libraries implementing the same (libxmlFoo vs libxmlBar)
● implementation in a different language (re2 vs Go's regexp)
● compare "fast but complex" with "slow but dumb" (sometimes easy to write)
● compare different functions (marshalBinary vs marshalText)
25. Quick Quiz: how to fuzz clang-format?
clang-format: shuffles whitespaces in a source file.
Let's imaging destiny of mankind depends on correctness of clang-format!
How would you fuzz test it?
26. Quick Quiz: how to fuzz clang-format?
● run with asan/msan/ubsan
● format twice, compare results (e.g. relies on unordered_map order)
● format, then format result (must be idempotent)
● strip all whitespaces, compare before/after
● check violations of max line length
● compile before/after (formatting breaks/unbreaks code)
27. Regression testing
Normally you run fuzzer for a long time.
But any guided fuzzer accumulates corpus of inputs with max coverage.
And that's perfect for regression testing! Just run it once on every change!
28. Fuzzing@Google Why?
- faster and faster development
- more and more code
- correctness is important
- stability is still important
- security is super important
- want to move fast, but keep development costs under control
Traditional testing is not enough anymore!
29. Fuzzing@Google How?
- Developers can write "fuzz tests"
- picked up by automatic large-scale fuzzing system
- but also work as regression unit tests
- OSS-Fuzz: continuous fuzzing for OSS
- 50+ projects, 190 fuzzers
- libFuzzer, radamsa, AFL (coming)
- 5000 cores
- ClusterFuzz: automated fuzzing for Chromium
- 350 fuzzers
- libFuzzer, radamsa, AFL, custom fuzzers
- 12000 cores
- Automatically files bugs and verifies fixes
- syzkaller: continuous fuzzing of Linux kernel
- several upstream branches + android/chromeos
- 100+ VMs + physical devices
31. Sales pitch
● Fuzzing is complimentary to any other testing technique
● Fuzzing is mandatory for anything security-related
● Fuzzing finds LOTS of bugs
● Fuzzing is easy to use
Call to action:
● choose 1 library that uses complex inputs (important or you suspect for bugs)
● write a fuzzer
● run locally with ASAN
