American Fuzzy Lop (AFL) is a black box fuzzing tool used to find crashes in software by generating random inputs. It operates on instrumented binaries without needing knowledge of the problem domain, proving effective in discovering elusive bugs across numerous applications. The document specifically discusses using AFL to test the encode/decode functions of the Python library PyHocon, highlighting its ease of use and potential in identifying complex issues.

1. Black Box Fuzzing
2016-10-24

2. • American Fuzzy Lop (AFL) is a fuzzing tool
• Also, a breed of rabbit
• Written by Michał Zalewski (`lcamtuf`)
• Used as part of a bunch of DEFCON presentations (including CGC)
• Big in the computer security field
• Finds crashes in programs
• Works on (instrumented) binaries
• Has no understanding of problem domain
• Has found A LOT of tricky, impressive bugs:
• http://lcamtuf.coredump.cx/afl/
• Firefox, OpenSSL, clang, glibc, perl, screen, Redis…
American Fuzzy Lop

3. • Fuzz testing is the generation of random, hopefully invalid inputs
• Meant to catch the edge cases that you didn’t anticipate / test for
Aside: Fuzz Testing

4. 1. Create a binary that reads from stdin, returns non-zero on exception
2. Give AFL the binary and a few sample inputs
3. Profit! (analyze crash logs)
Usage

5. • We want to check for crashes in the encode/decode functions of pyhocon
• HOCON (Human-Optimized Config Object Notation)
• https://github.com/typesafehub/config/blob/master/HOCON.md
• pyhocon is a python library for HOCON SerDes
• https://github.com/chimpler/pyhocon
Example

6. • We want to check for crashes in the encode/decode functions of pyhocon
1. Write a simple wrapper program
2. Since we’re in Python, we also use the python-afl library
3. Add the expected Exceptions
4. Run!
5. Wait…
Example

7. The code

8. It’s running!

9. • Bit + Byte flips
• Arithmetic offsets
• Troublesome values (0, 1, INT_MAX, etc)
• Random overwrites + appends
• Inserts/Deletes/Splices of inputs at random offsets
Checks

10. • Bit + Byte flips
• Arithmetic offsets
• Troublesome values (0, 1, INT_MAX, etc)
• Random overwrites + appends
• Inserts/Deletes/Splices of inputs at random offsets
Checks

11. • https://github.com/chimpler/pyhocon/issues/103
• It happens when you try to append to an list that is nested in a dictionary
• Internally, a boolean was being passed in when it should have been a string
• While type checking would have also found this, a person manually testing likely
would not (and did not) find it
Bug found!

12. • American Fuzzy Lop is a very good tool for black box fuzz testing of software.
• Very easy to use (nothing to learn, no domain knowledge)
• Especially useful for code:
• That is complex
• That you didn’t write
• That you don’t have the source code for
• ie. Code you don’t understand
• Further reading:
• https://github.com/mirrorer/afl/blob/master/docs/technical_details.txt
• “10/10; would crash again”
Conclusions