![Who cares?
4
A team of hackers won $2 million by
building a machine that could hack better
than they could
Read more at
DARPA Cyber Grand
Challenge
Automation of Security
[detecting and fixing
vulnerabilities in binaries
automatically]
Fuzzing
Talk,
2022](https://image.slidesharecdn.com/fuzzing-220617065914-c2a5c01c/85/Fuzzing-pptx-4-320.jpg)
![Lack of specifications
• Many programs take in structured inputs
PDF Reader, library for manipulating TIFF, PNG images
Compilers which take in programs as input
Web-browsers, ...
• Generating a completely random input will likely crash the application with little
insight gained about the underlying vulnerability.
• Instead take a legal well-formed PDF file and mutate it!
Does not depend on program at all [nature of BB fuzzing]
Does not even depend on input structure.
Yet can leverage complex input structure by starting with a well-formed seed and
minimally modifying it.
Fuzzing
Talk,
2022
6](https://image.slidesharecdn.com/fuzzing-220617065914-c2a5c01c/85/Fuzzing-pptx-6-320.jpg)
![Prioritize low probability paths
15
Use grey-box fuzzer which keeps track of path id for a test.
Find probabilities that fuzzing a test t which exercises π leads to an
input which exercises π’
Higher weightage to low probability paths discovered, to gravitate to
those -> discover new paths with minimal effort.
π π'
1 void crashme (char* s) {
2 if (s[0] == ’b’)
3 if (s[1] == ’a’)
4 if (s[2] == ’d’)
5 if (s[3] == ’!’)
6 abort ();
7 }
p
Fuzzing
Talk,
2022](https://image.slidesharecdn.com/fuzzing-220617065914-c2a5c01c/85/Fuzzing-pptx-15-320.jpg)
Uploaded byAbhik Roychoudhury
Fuzzing.pptx
The document discusses advances in fuzz testing, focusing on different techniques such as grey-box and directed fuzzing to enhance vulnerability detection in software. It highlights the importance of using structured inputs and various mutation strategies to improve the efficiency of fuzzing, as exemplified by the success of tools like AFL and its variants. The presentation emphasizes the integration of symbolic execution elements with fuzz testing to achieve better coverage and detection of vulnerabilities in software systems.
More Related Content
![[Wroclaw #4] Fuzzing - underestimated method of finding hidden bugs](https://cdn.slidesharecdn.com/ss_thumbnails/fuzzing4pdfexported-161029144418-thumbnail.jpg?width=640&height=640&fit=bounds)
More from Abhik Roychoudhury
Fuzzing.pptx
- 1.
- 2. APR-talk, 2022 Working in Program Analysis and Software Security(2001-) Singapore Cybersecurity Consortium (2016) National of Excellence in Trustworthy Software Systems (2019) 2 Comprehensive university with Science, Arts, Engineering, medical, Law, Business, Public Policy, Music, Computing, … Public university 30K undergraduate students, 10K+ graduate students overall. 2500+ faculty members overall, 100+ in Computing (two departments CS and IS). http://www.nus.edu.sg/about#corporate-information
- 3. Fuzz Testing 3 Springfield Project- Fuzzing as a service OSS-Fuzz - Continuous fuzzing for open-source projects Term coined by Barton Miller at University of Wisconsin in 1988 Biased random search over inputs to find crashes / hangs, tools existed until 2016. And then, since 2016 … research in looking inside the tools
- 4. Who cares? 4 A teamof hackers won $2 million by building a machine that could hack better than they could Read more at DARPA Cyber Grand Challenge Automation of Security [detecting and fixing vulnerabilities in binaries automatically] Fuzzing Talk, 2022
- 5. Presented by ThuanPham Black-box Fuzzing 5
- 6. Lack of specifications •Many programs take in structured inputs PDF Reader, library for manipulating TIFF, PNG images Compilers which take in programs as input Web-browsers, ... • Generating a completely random input will likely crash the application with little insight gained about the underlying vulnerability. • Instead take a legal well-formed PDF file and mutate it! Does not depend on program at all [nature of BB fuzzing] Does not even depend on input structure. Yet can leverage complex input structure by starting with a well-formed seed and minimally modifying it. Fuzzing Talk, 2022 6
- 7.
- 8. Grey-box Fuzzing 8 Mutators Test suite Mutatedfiles Input Queue Enqueue Dequeue Fuzzing Talk, 2022
- 9. Grey-box Fuzzing Algorithm 9 •Input: Seed Inputs S • 1: T✗ = ∅ • 2: T = S • 3: if T = ∅ then • 4: add empty file to T • 5: end if • 6: repeat • 7: t = chooseNext(T) • 8: p = assignEnergy(t) • 9: for i from 1 to p do • 10: t0 = mutate_input(t) • 11: if t0 crashes then • 12: add t0 to T✗ • 13: else if isInteresting(t0 ) then • 14: add t0 to T • 15: end if • 16: end for • 17: until timeout reached or abort-signal • Output: Crashing Inputs T✗ Fuzzing Talk, 2022
- 10. Space of Problemsin SW Reliability • Fuzz Testing Feed semi-random inputs to find hangs and crashes • Continuous fuzzing Incrementally find new “problems” in software • Crash reproduction Re-construct a reported crash, crashing input not included due to privacy • Reaching nooks and corners • Localizing reported observable errors • Patching reported errors from input-output examples 10 Fuzzing Talk, 2022
- 11. Space of Techniques Search(Fuzzing) • Random • Biased-random … • … with mutations (AFL Fuzzer) • … • Low set-up overhead • Fast, less accurate • Use objective function to steer Symbolic Execution • Dynamic Symbolic execution • Concolic Execution • Cluster paths based on symbolic expressions of variables • .... • High set-up overhead • Slow, more accurate • Use logical formula to steer 11 Fuzzing Talk, 2022
- 12. In this(?) talk… Search • Enhance the effectiveness of search techniques, with symbolic execution and model checking as inspiration • Think: Symbolic (Formal) • Act: Fuzzing (Informal) 12 Fuzzing Talk, 2022
- 13. Grey-box Fuzzing Algorithm 13 •Input: Seed Inputs S • 1: T✗ = ∅ • 2: T = S • 3: if T = ∅ then • 4: add empty file to T • 5: end if • 6: repeat • 7: t = chooseNext(T) • 8: p = assignEnergy(t) • 9: for i from 1 to p do • 10: t0 = mutate_input(t) • 11: if t0 crashes then • 12: add t0 to T✗ • 13: else if isInteresting(t0 ) then • 14: add t0 to T • 15: end if • 16: end for • 17: until timeout reached or abort-signal • Output: Crashing Inputs T✗ Fuzzing Talk, 2022
- 14. Programming by experienced people Schematic if (condition1) return// short path, frequented by many inputs else if (condition2) exit // short paths, frequented by many inputs else …. Fuzzing Talk, 2022 14
- 15. Prioritize low probabilitypaths 15 Use grey-box fuzzer which keeps track of path id for a test. Find probabilities that fuzzing a test t which exercises π leads to an input which exercises π’ Higher weightage to low probability paths discovered, to gravitate to those -> discover new paths with minimal effort. π π' 1 void crashme (char* s) { 2 if (s[0] == ’b’) 3 if (s[1] == ’a’) 4 if (s[2] == ’d’) 5 if (s[3] == ’!’) 6 abort (); 7 } p Fuzzing Talk, 2022
- 16. Power-Schedules 16 Constant: AFL uses thisschedule (fuzzing ~1 minute) (i) .. how AFL judges fuzzing time for the test exercising path i Cut-off Exponential: p(i) = (i) p(i) = 0, if f(i) > µ min( ((i)/β)*2s(i), M) otherwise β is a constant s(i) #times the input exercising path i has been chosen from queue f(i) # generated inputs exercising path i (path-frequency) µ mean #fuzz exercising a discovered path (avg. path-frequency) M maximum energy expendable on a state Fuzzing Talk, 2022
- 17. Independent Evaluation andDeployment • An independent evaluation by team Codejitsu found that AFLFast exposes errors in the benchmark binaries of the DARPA Cyber Grand Challenge 19x faster than AFL. • Picked up by Zalewski@AFL, with following observations, paraphrased AFLFAST assigns substantially less energy in the beginning of the fuzzing campaign. Most of the cycles that AFLFAST carries out, are in fact very short. This causes the queue to be cycled very rapidly, which in turn causes new retained inputs to be fuzzed almost immediately. In other words, because AFLFAST assigns less energy, it can process the complete queue substantially faster. We say it starts by exploration rather than by exploitation • Implemented inside AFL (version 2.33b, FidgetyAFL), and distributed approximately within one year of publication 17 There remain differences between the two in terms of path discovered. More experiments may be needed.
- 18. In this talk… • Greybox Fuzzing is frequently used, daily in corporations State-of-the-art in automated vulnerability detection Extremely efficient coverage-based input generation All program analysis before/at instrumentation time. Start with a seed corpus, choose a seed file, fuzz it. Add to corpus only if new input increases coverage. Cannot be directed, unlike symbolic execution! • Enhance the effectiveness of search techniques, with symbolic execution & model checking as inspiration – Enhance coverage, how to make it directed? Fuzzing Talk, 2022 18
- 19. (Earlier) View-point 19 DirectedFuzzing: classical constraint satisfaction prob. Program analysis to identify program paths that reach given program locations. Symbolic Execution to derive path conditions for any of the identified paths. Constraint Solving to find an input that satisfies the path condition and thus reaches a program location that was given. φ1 = (x>y)∧(x+y>10) φ2 = ¬(x>y)∧(x+y>10) x > y a = x a = y x+y>10 b = a return b Fuzzing Talk, 2022
- 20. (Later) View-point 20 DirectedFuzzing as optimization problem! 1. Instrumentation Time: • Instrument program to aggregate distance values. 2. Runtime, for each input • decide how long to be fuzzed based on distance. • If input is closer to the targets, it is fuzzed for longer. • If input is further away from the targets, it is fuzzed for shorter. Fuzzing Talk, 2022
- 21. Power Schedules -Recap 21 • Input: Seed Inputs S • 1: T✗ = ∅ • 2: T = S • 3: if T = ∅ then • 4: add empty file to T • 5: end if • 6: repeat • 7: t = chooseNext(T) • 8: p = assignEnergy(t) • 9: for i from 1 to p do • 10: t0 = mutate_input(t) • 11: if t0 crashes then • 12: add t0 to T✗ • 13: else if isInteresting(t0 ) then • 14: add t0 to T • 15: end if • 16: end for • 17: until timeout reached or abort-signal • Output: Crashing Inputs T✗ Fuzzing Talk, 2022
- 22. Instrumentation 22 Function-level targetdistance using call graph (CG) BB-level target distance using control-flow graph (CFG) 1. Identify target BBs and assign distance 0 2. Identify BBs that call functions and assign 10*FLTD 3. For each BB, compute harmonic mean of (length of shortest path to any function-calling BB + 10*FLTD). CFG for function b 8.7 11 10 30 13 12 N/A Fuzzing Talk, 2022
- 23. Directed fuzzing asoptimization 23 Integrating Simulated Annealing as power schedule In the beginning (t = 0min), assign the same energy to all seeds. Later (t=10min), assign a bit more energy to seeds that are closer. At exploitation (t=80min), assign maximal energy to seeds that are closest. Fuzzing Talk, 2022
- 24. Outcomes 24 • Directed greyboxfuzzer (AFLGo) outperforms symbolic execution-based directed fuzzers (KATCH & BugRedux) • in terms of reaching more target locations and • in terms of detecting more vulnerabilities, • on their own, original benchmark sets. • Integrated as OSS-Fuzz fork (AFLGo for Continuous Fuzzing) • Tool AFLGo publicly available, follow-up works, survey by community. Fuzzing Talk, 2022
- 25. Beyond Symb. Exec.… Verification? Search • Enhance the effectiveness of search techniques, with symbolic execution as inspiration Enhance coverage Achieve directed search More advanced properties than crashes! Get close to the effect of verification as in model checking 25 84 139 59 AFLGo KLEE Fuzzing Talk, 2022 Synergy
- 26. M |= ⏀ ModelChecking via Dir. Fuzzing 26 Bug finding search in model checking via directed fuzzing Cover the whole specification language of properties for a well- known and popular temporal logic – LTL No state explosion problem as in model checking. Fuzzing for more advanced oracles than simply crashes and hangs! Most uses of Software Model Checking are for bug finding Restricted set of properties for software model checking Mostly restricted to proving / disproving of invariants due to nature of state abstractions Unnecessary state savings and state explosion problem. Fuzzing Talk, 2022
- 27. Software Model Checking FuzzingTalk, 2022 27 Consider . None of the exec. traces of M should satisfy . Construct a finite-state automata A such that Language(A ) = Traces satisfying Construct the synch product M A Check whether any exec trace of M is an exec trace of the product M A i.e. check Language(M A ) = empty-set? Yes: Violation of found, report counterexample No: Property holds for all exec traces of M. Used in SPIN Model checker
- 28. Linear-time Temporal Logic 28 …..….. ….. Satisfies j1 Satisfies j1 Satisfies j1 Satisfies j1 U j2 Satisfies j2 Satisfies j1 R j2 Satisfies j2 Satisfies j2 Satisfies j2 ….. ….. Satisfies j1 R j2 ….. ….. Satisfies j2 Satisfies j2 Satisfies j2 Satisfies j1 Ùj2 Fuzzing Talk, 2022
- 29. Product Automata (i) SystemModel M (iii) Product Automata M A p s1 s2 p p true q1 q2 (s1,q1) (s1,q2) (s2,q1) (s2,q2) true p p true true p (ii) Property Automata A M |= GF p 29 Fuzzing Talk, 2022
- 30. Temporal Logic guidedFuzzing Fuzzing Talk, 2022 30 Find • Find acceptance states reachable from initial states (DFS). • Conduct fuzzing to reach an accepting state s from initial state 1 Find • Find all such acceptance states which are reachable from itself (DFS). • Conduct fuzzing to reach state s back from state s 2 Counter-example • Counter-example evidence (if any) obtained by simply concatenating the two DFS stacks. • Construct violating input from the two fuzzing runs 3
- 31. Büchi Automata GuidedFuzzing 31 Save Progress Prefix Selection Target Selection Trace Evaluation Progress Saving 31 Fuzzing Talk, 2022
- 32. Finding Zero-day Bugs 32 FuzzingTalk, 2022
- 33. Stateful Fuzzing Input 3 Fuzzing Generate testcases Check output Existing works: Manual specification (IJON) Return code as states (AFLNet) Challenge: To cover the state space without an explicit specification of the protocol.
- 34. IDLE RECV_HEADERS RECV_BODY REQ_PENDING SEND_HEADERS SEND_BODY BODY_IS_FINAL END_STREAM In fuzzing, wemonitor the changes of enumeration variables’ values to construct the state transition tree on the fly. State Transition Tree END_STREAM END_STREAM END_STREAM REQ_PENDING SEND_HEADERS SEND_BODY BODY_IS_FINAL END_STREAM Monitor the variable: stream->state
- 35. IDLE RECV_HEADERS RECV_BODY REQ_PENDING SEND_HEADERS SEND_BODY BODY_IS_FINAL END_STREAM State Machine Comparewith the official document. https://datatracker.ietf.org/doc/html/rfc7540#section-5.1
- 36. State Fuzzing Algorithm IDLE RECV_HEADERS RECV_BODY REQ_PENDING SEND_HEADERS SEND_BODY BODY_IS_FINAL END_STRE AM 1.Give all state transitions equal opportunities to be fuzzed. 2. Assign more energy on the valuable state sequence. 3. Save the inputs that trigger new state transitions. 4. Correlate input bytes and state transitions, giving more opportunities on mutating these bytes. More effort to mutate this part
- 37.
- 38. Handling Structured Data? MakeGreybox Fuzzing input-structure aware by 1. Changing the input representation (structured files) Use tree-like representation instead of bit string 2. Adding new mutation operators working at chunk level (e.g., chunk deletion, insertion and splicing) 3. Prioritizing more valid seed inputs More valid seeds are assigned higher fuzzing “energy” 4. Applying optimizations to retain fuzzing efficiency
- 39. Mutators Test suite Mutated files InputQueue Enqueue Dequeue Fuzzing Talk, 2022 39 AFLSmart File Cracker root chunk 1 … … chunk 2 Seed input validity score (0->100) 100: the whole seed is fully cracked/parsed XML-based input model. One input model for each file format. (e.g., Peach pits)
- 40.
- 41. Pointers Fuzzing Talk, 2022 41 Directed Greybox Fuzzing( PDF ) 24th ACM Conference on Computer and Communications Security (CCS) 2017. Coverage-based Greybox Fuzzing as Markov Chain ( PDF ) 23rd ACM Conference on Computer and Communications Security (CCS) 2016, Also in IEEE Transactions in Software Engineering (TSE) 2019. NUS Fuzzing page: https://abhikrc.com/projects/Fuzz/ “All” fuzzing papers: https://wcventure.github.io/FuzzingPaper/ ACKNOWLEDGEMENT: National Cyber Security Research program from NRF Singapore Smart Greybox Fuzzing ( PDF ) IEEE Transactions on Software Engineering (TSE), September 2021. Linear-time Temporal Logic guided Greybox Fuzzing International Conference on Software Engineering (ICSE), 2022. Fuzzing: Challenges and Reflections ( PDF ) IEEE Software, 38(3), pages 79-86, 2021, Outcome from a 2019 Shonan Meeting. Stateful Greybox Fuzzing Usenix Security Symposium, 2022.
- 42. Looking forward Fuzzing Talk, 2022 42 Protocol Implementation Lot ofpast works on fuzzing have focused on parsers or file format processors. Stateful system fuzzing could be the next step – internet facing protocols. Model Checking efficacy, without guarantees, and without state caching. Protocol Spec Fuzzer Advanced Oracle Bugs Validate True Bugs