Fuzzing is a software testing technique that involves providing invalid, unexpected, or random data as inputs to a computer program to detect bugs. Coverage-guided fuzzing uses genetic algorithms to generate inputs that maximize code coverage. It is effective at finding security bugs like overflows, memory errors, and crashes. The presenter demonstrates finding 13 bugs in Boost regex in 30 minutes using libFuzzer. Fuzzing is widely used at Google to test critical software like browsers, libraries, and the Linux kernel due to its ability to find many bugs without requiring test cases.

1. Fuzzing:
The New Unit Testing
C++ Russia 2017, Moscow, Feb 25
Dmitry Vyukov, dvyukov@, Google

2. Agenda
● What is fuzzing
● Coverage-guided fuzzing
● Small tutorial
● How to write effective fuzzers
● Fuzzing@Google

3. What is Fuzzing?
wikipedia.org/wiki/Fuzz_testing:
Fuzz testing or fuzzing is a software testing technique, often automated or
semi-automated, that involves providing invalid, unexpected, or random data to
the inputs of a computer program.

4. Who cares?
- We are not testing/checking anything!
- Random data will not trigger any bugs!

5. Fuzzing can find lots of bugs
- With the help of sanitizers:
- Use-after-free, buffer overflows
- Uses of uninitialized memory
- Memory leaks
- Data races, deadlocks
- Int/float overflows, bitwise shifts by invalid amount (other UB)
- Plain crashes:
- NULL dereferences, uncaught exceptions, div-by-zero
- Resource usage bugs:
- Memory exhaustion, hangs or infinite loops, infinite recursion (stack overflows)
- Logical bugs (lots of, see below)

6. Data is not necessary "white noise"
- There is number of tricks to generate "not so random" data
- May or may not require some human help
- If used correctly achieves very impressive code coverage

7. What can be fuzzed?
Anything that consumes complex inputs:
● Parsers of any kind (xml, json, asn.1, pdf, truetype, ...)
● Media codecs (audio, video, raster & vector images, etc)
● Network protocols (HTTP, RPC, SMTP, MIME...)
● Crypto (boringssl, openssl)
● Compression (zip, gzip, bzip2, brotli, ...)
● Formatted output (sprintf, template engines)
● Compilers and interpreters (Javascript, PHP, Perl, Python, Go, Clang, ...)
● Regular expression matchers (PCRE, RE2, libc’s regcomp)
● Text/UTF processing (icu)
● Databases (SQLite)
● Browsers, text editors/processors (Chrome, vim, OpenOffice)
● OS Kernels (Linux), drivers, supervisors and VMs
Must have for everything that consumes untrusted inputs, open to internet or otherwise security sensitive.

8. Types of Fuzzers
- Grammar-based generation
- Generate random inputs according to grammar rules
- Peach, packetdrill, csmith, gosmith, syzkaller
- Blind mutation
- Requires a corpus of representative inputs, apply random mutations to them
- ZZUF, Radamsa
- Grammar reverse-engineering
- Learn grammar from existing inputs using algorithmic approach of machine learning
- Sequitur algorithm, go-fuzz
- Symbolic execution + SAT solver
- Synthesize inputs with maximum coverage using black magic
- KLEE
- Coverage-guided fuzzers
- Genetic algorithm that strives to maximize code coverage
- libFuzzer, AFL, honggfuzz, syzkaller
- Hybrid

9. Coverage-guided fuzzing
Build the program with code coverage instrumentation;
Collect initial corpus of inputs (optional);
while (true) {
Choose a random input from corpus and mutate it;
Run the target program on the input, collect code coverage;
If the input gives new coverage, add mutation back to the corpus;
}

10. Coverage-guiding in action
if input[0] == '{' {
if input[1] == 'i' && input[2] == 'f' {
if input[3] == '(' {
input[input[4]] = input[5]; // potential OOB write
}
}
}
Requires "{if(" input to crash, ~2^32 guesses to crack when blind.
Coverage-guiding:
Guess "{" in ~2^8, add to corpus.
Guess "{i" in ~2^8, add to corpus.
Guess "{if" in ~2^8, add to corpus.
Guess "{if(" in ~2^8, add to corpus.
Total: ~2^10 guesses.
See: AFL: Pulling JPEGs out of thin air

11. Mutations
● erase/insert/change/shuffle bit/byte/bytes
● crossover/splice 2 inputs
● insert token from a dictionary
● insert magic numbers (2^10±1, 2^16±1, 2^31±1, 2^32±1)
● change an ASCII integer (e.g. "123" => "2465357635")
● ...

12. Coverage flavours
Basic blocks:
... (A)
if (...) {
... (B)
}
... (C)
-fsanitize-coverage=bb
Edges:
... (A)
if (...) {
... (B)
}
... (C)
-fsanitize-coverage=trace-pc-guard
Gives better feedback signal.
Counters:
for (...) {
... (hit N times)
}
-fsanitize-coverage=8bit-counters
Gives better feedback signal
for loops and recursion.

13. Cracking hashes
What about more complex cases?
if (*(uint32_t*)input == crc32(input+4, size-4)) {...}
if (*(uint64_t*)input == 0xBCEBC041BADBALL) {...}

14. Cracking hashes
Intercept comparison operations:
● compiler intercepts int comparisons (-fsanitize-coverage=trace-cmp)
● runtime intercepts strcmp/memcmp and friends
Several possibilities:
● extract int/string literals and insert them into inputs
● find one comparison operand in the input and replace with the other operand
● use PC^POPCNT(op1^op2) as "coverage" signal (Hamming distance)

15. Dictionaries
● User-provided
○ e.g. for HTTP: "HTTP/1.1", "Host", "Accept-Encoding"
● Automatically extracted from program
○ memcpy(input, "HTTP/1.1", 8)

16. Tutorial
"...one of the most highly regarded and expertly designed C++ library projects in the world"
boost.regex
(latest version 1.63, in boost since 1.18)

17. Tutorial: fuzzing function
As simple as:
int LLVMFuzzerTestOneInput(const uint8_t * Data, size_t Size) {
try {
std::string str((char*)Data, Size);
boost::regex e( str);
boost::match_results<std::string::const_iterator> what;
boost::regex_match(str, what, e, boost::match_default);
} catch (const std::exception&) {}
return 0;
}

18. Tutorial: building (the hard part)
1. Build boost with coverage and AddressSanitizer:
./b2 cxxflags="-fsanitize-coverage=trace-pc-guard -fsanitize=address" toolset=clang install
2. Build fuzzer with coverage, AddressSanitizer and libFuzzer:
clang++ fuzzer.cc -fsanitize-coverage=trace-pc-guard -fsanitize=address libFuzzer.a
The rest is at tutorial.libfuzzer.info

19. Demo

20. 30 minutes, 13 bugs (ticket/12818):
AddressSanitizer: heap-buffer-overflow perl_matcher.hpp:132 in re_skip_past_null
AddressSanitizer: heap-buffer-overflow basic_regex_parser.hpp:2599 in parse_perl_extension
AddressSanitizer: heap-buffer-overflow perl_matcher.hpp:221 in re_is_set_member
AddressSanitizer: heap-buffer-overflow perl_matcher.hpp:166 in re_is_set_member
AddressSanitizer: heap-buffer-overflow interceptors.inc:278 in strlen
AddressSanitizer: stack-overflow basic_regex_creator.hpp:1054 in create_startmap
AddressSanitizer: SEGV on unknown address 0x0000000016e0
MemorySanitizer: use-of-uninitialized-value perl_matcher.hpp:166 in re_is_set_member
basic_regex_parser.hpp:904: runtime error: shift exponent 325804978 is too large for 32-bit type 'unsigned int'
basic_regex_parser.hpp:2599: runtime error: load of value 56794092, which is not a valid value for type 'syntax_element_type'
a.out: perl_matcher_common.hpp:606: Assertion `r.first != r.second' failed
Direct leak of 4096 byte(s) in 1 object(s) allocated in get_mem_block regex.cpp:204
ALARM: working on the last Unit for 17 seconds
Will find more when these are fixed!
Results

21. Finding logical bugs
Not only security/stability
- But we don't know the right result!
- Use your imagination!

22. Finding logical bugs
● sanity checks on results
○ uncompressesed image decoder: 100 byte input -> 100 MB output?
○ function returns both error and object, or no error and no object
○ know that some substring must present in output, but it is not
○ encrypt, check that decryption with wrong key fails
● sometimes we do know the right result
○ any sorting: check that each element is present, check that it's not descending
○ building a trie: check size, all elements are present
● asserts
○ assert(a == b)

23. Finding logical bugs
Round-trip:
● encode-decode
● serialize-deserialize
● compress-decompress
● encrypt-decrypt
● assemble-disassemble
Checks:
● decode-encode: check that encode don't fail
● decode-encode-decode: check that second decode don't fail
● decode-encode-decode: check that decode results are equal
● encode-decode-encode: check that encode results are equal
Very powerful technique.

24. Finding logical bugs
Comparing two (or more) implementations gives phenomenal results:
● check that output is equal
● or at least check that ok/fail result is the same
○ e.g. gcc and clang both accept or reject the code
But I don't want to write the second impl!
● there can be several libraries implementing the same (libxmlFoo vs libxmlBar)
● implementation in a different language (re2 vs Go's regexp)
● compare "fast but complex" with "slow but dumb" (sometimes easy to write)
● compare different functions (marshalBinary vs marshalText)

25. Quick Quiz: how to fuzz clang-format?
clang-format: shuffles whitespaces in a source file.
Let's imaging destiny of mankind depends on correctness of clang-format!
How would you fuzz test it?

26. Quick Quiz: how to fuzz clang-format?
● run with asan/msan/ubsan
● format twice, compare results (e.g. relies on unordered_map order)
● format, then format result (must be idempotent)
● strip all whitespaces, compare before/after
● check violations of max line length
● compile before/after (formatting breaks/unbreaks code)

27. Regression testing
Normally you run fuzzer for a long time.
But any guided fuzzer accumulates corpus of inputs with max coverage.
And that's perfect for regression testing! Just run it once on every change!

28. Fuzzing@Google Why?
- faster and faster development
- more and more code
- correctness is important
- stability is still important
- security is super important
- want to move fast, but keep development costs under control
Traditional testing is not enough anymore!

29. Fuzzing@Google How?
- Developers can write "fuzz tests"
- picked up by automatic large-scale fuzzing system
- but also work as regression unit tests
- OSS-Fuzz: continuous fuzzing for OSS
- 50+ projects, 190 fuzzers
- libFuzzer, radamsa, AFL (coming)
- 5000 cores
- ClusterFuzz: automated fuzzing for Chromium
- 350 fuzzers
- libFuzzer, radamsa, AFL, custom fuzzers
- 12000 cores
- Automatically files bugs and verifies fixes
- syzkaller: continuous fuzzing of Linux kernel
- several upstream branches + android/chromeos
- 100+ VMs + physical devices

30. Fuzzing@Google
~15'000bugs
GLIBC MUSL LIBC pugixml PCRE ICU Freetype ffmpeg Harfbuzz SQLite Python OpenSSL BoringSSL libxml2 BPF Capstone file Radare2 gRPC
WOFF2 LLVM Tensorflow libav FreeType2 Foxit libtiff Go Linux libexif libFLAC Little CMS Adobe Reader Adobe Flash Player Adobe DNG SDK
ESET NOD32 ClamAV BitDefender poppler ghostscript dcraw qcms libwebp libwebm libvpx gipfeli libots Snapseed Dart VM IJG libjpeg-turbo libpng
mozjpeg PHP Firefox Internet Explorer Apple Safari LibreOffice GnuTLS GnuPG OpenSSH bash tcpdump JavaScriptCore pdfium libmatroska
libarchive wireshark ImageMagick lcms libbpg lame libsndfile less lesspipe strings dpkg rcs systemd-resolved libyaml Info-Zip unzip libtasn pfctl
mandoc IDA Pro clamav nasm ctags mutt procmail fontconfig pdksh wavpack redis cmsgpack taglib privoxy perl libxmpradare2 SleuthKit X.Org
exifprobe jhead Xerces-C metacam exiv btrfs Knot DNS curl wpa_supplicant dnsmasq imlib2 libraw libwmf uudecode MuPDF libbson libsass boost

31. Sales pitch
● Fuzzing is complimentary to any other testing technique
● Fuzzing is mandatory for anything security-related
● Fuzzing finds LOTS of bugs
● Fuzzing is easy to use
Call to action:
● choose 1 library that uses complex inputs (important or you suspect for bugs)
● write a fuzzer
● run locally with ASAN

32. Thanks!
Q&A
http://tutorial.libfuzzer.info
Dmitry Vyukov, dvyukov@

33. References
https://en.wikipedia.org/wiki/Fuzz_testing
https://github.com/google/sanitizers
http://llvm.org/docs/LibFuzzer.html
http://tutorial.libfuzzer.info
http://lcamtuf.coredump.cx/afl/
https://github.com/google/oss-fuzz
https://github.com/google/syzkaller
https://github.com/dvyukov/go-fuzz