Fault Models and Fuzzing


Published on

Abstract from StarEast:

Fuzzing and Fault Modeling for Product Evaluation

Test environments are often maintained under favorable conditions, providing a stable, reliable configuration for testing. However, once the product is released to production, it is subject to the stresses of low resources, noisy–even malicious–data, unusual users, and much more. Will real-world use destroy the reliability of your software and embarrass your organization? Shmuel Gershon describes fuzzing and fault modeling, techniques used to simulate worst-case run-time scenarios in his testing. Fuzzing explores the limits of a system interface with high volumes of random input data. Fault models examine and highlight system resource usage, pointing out potential problems. These techniques help raise important questions about product quality–even for conditions that aren’t explicit in requirements. Shmuel shares practical principles for getting started with fuzzing and fault modeling, and demonstrates catastrophic failures on real-world applications. Come and learn how, using free tools, you can push the limits of your software.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Fault Models and Fuzzing

  1. 1. Fault Models and Fuzz Techniques<br />Shmuel Gershon<br />STAREAST 2011<br />Copyright © CC:BY-NC-SA 2007-11, Shmuel Gershon.<br />
  2. 2. About...<br /><ul><li>Shmuel Gershon
  3. 3. Testing Engineer
  4. 4. http://testing.gershon.info
  5. 5. Creator of Rapid Reporter
  6. 6. Twitter: @sgershon, Skype: sgershon, shmuel@gershon.info</li></ul>Disclaimer:<br /><ul><li>Names and Brands referenced herein may be claimed as property of third parties
  7. 7. Views expressed in this presentation are solely my own, and do not in any manner represent the views of my employer
  8. 8. Information in this presentation is provided 'as is' without any warranties or representations of any kind</li></ul>Copyright © CC:BY-NC-SA 2007-11<br />
  9. 9. Fault Models and FuzzTalk Objectives<br />Understand the principles of the techniques<br />Meet tools available for both techniques<br />Learn to apply the methods (in different app types)<br />Perceive drawbacks and difficulties<br />Lay down a foundation for further research<br />Have at least one new idea for your tests :)<br />Testing is questioning a product in order to evaluate its value to a person that matters<br />Adapted from Jerry Weinberg + James Bach + Michael Bolton<br />Copyright © CC:BY-NC-SA 2007-11<br />
  10. 10. Putting Concepts into Context<br />Availability<br />Robustness<br />Dependability<br />Security<br />Reliability<br />Stability<br />Copyright © CC:BY-NC-SA 2007-11<br />
  11. 11. Fault Models and FuzzTalk Outline<br />Fault Models<br />Overview<br />Examples + Demos<br />Pitfalls and Tricks<br />Fuzzing<br />Overview<br />Examples + Demos<br />Pitfalls and Tricks<br />Summary and Questions<br />Copyright © CC:BY-NC-SA 2007-11<br />
  12. 12. Fault Models OverviewConceptual Diagram<br />OS – Disk Storage<br />Injection<br />Runtime<br />Fault<br />Layer<br />Application Under Test<br />OS – User I/O<br />OS - Memory<br />OS - Network<br />Copyright © CC:BY-NC-SA 2007-11<br />
  13. 13. Fault Models Overview – Definition<br />Definitions:<br />Introducing faults in order to test (error handling) code paths, that might otherwise rarely be followed<br />Assess the robustness of software by checking it's reaction to adverse events<br />Purpose<br />Validation of: Robustness, Dependability (Availability) and Security<br />Also known as: Recovery Code test, Fault Injection, Negative test, Error Handling, Stress test...<br />Copyright © CC:BY-NC-SA 2007-11<br />
  14. 14. Fault Models Overview – Approaches<br />Random Runtime Fault Injections:<br />A tool control the type, time and location to ‘attack’. May modify the coverage of a set of tests (issues found may be harder to reproduce or debug).<br />Initiated Runtime Fault Injection:<br />Specific tests in which the faults are controlled at specific point, aiming a clear error handling flow.<br />Bugs found in this approach may be easier to reproduce, accept and fix.<br />(This is the approach we'll focus on).<br />Copyright © CC:BY-NC-SA 2007-11<br />
  15. 15. Fault Models Overview – Reactions<br />Reactions against Runtime Fault Bugs<br />It will never happen in real life<br />A user will not do that<br />It should fail in such conditions<br />It is an unsupported scenario<br />We can’t fix this<br />It is a third-party problem<br />How would you reply?<br />Whose decision it ultimately is?<br />Copyright © CC:BY-NC-SA 2007-11<br />
  16. 16. Fault Injection – Example #1<br />Memory Starvation:<br />Refuses to allocate a memory upon request (Insufficient Memory).<br /><ul><li>Pinball:</li></ul>Silently skips the action,no harm done.<br /><ul><li>WordPad:</li></ul>Disappears with your most valuable work<br />Copyright © CC:BY-NC-SA 2007-11<br />
  17. 17. Fault Injection – Example #2<br />Network Errors:<br />Simulates an error responses for common network requests and resources.<br />Browser Wars:<br /><ul><li>One browser survives
  18. 18. The other crashes!</li></ul>Copyright © CC:BY-NC-SA 2007-11<br />
  19. 19. Fault Injection – Example #3<br />Low Resources / Slow or Clogged CPU<br />Computer behaves as a busy or old computer<br />(CPU is busy on other tasks / time scheduler starvation...)<br /><ul><li>Playing to win:</li></ul>Can we escape from death<br /> by simply slowing down the rest?<br />Copyright © CC:BY-NC-SA 2007-11<br />
  20. 20. Fault Models – Notes<br />Finding bugs can take a long time (but you record important data during the process)<br />Abstract knowledge of the internal flows of the software is imperative for good results<br />Bugs are received with the comments seen earlier<br />It is difficult to assess the exposure of such bugs<br />These points can make it harder to adopt the practices in the company...<br />...so be sure to address them<br />Copyright © CC:BY-NC-SA 2007-11<br />
  21. 21. Fault Injection Tools Examples<br />Canned Heat is free & easy, but buggy<br />Limitations:<br />Does not support .NET apps or Services<br />Bug: In some systems it will not load or needs to be reloaded between applications.<br />Others:<br />Verifier, AppVerifier<br />SlowProc, HeavyLoad<br />Holodeck - not free.<br />PIN and (soon) the random malloc blocker<br />In-House or Brute-Force tools<br />Copyright © CC:BY-NC-SA 2007-11<br />
  22. 22. Fuzz<br />Fuzz<br />Fuzz<br />Fuzz<br />Fuzz<br />Fuzz<br />Fuzz TestingConceptual Diagram<br />OS – Disk Storage<br />Application Under Test<br />OS – User I/O<br />OS - Memory<br />OS - Network<br />Copyright © CC:BY-NC-SA 2007-11<br />
  23. 23. Fuzz Testing – Definition<br />Definitions:<br />Providing random data which is free of preconceptions to the inputs of a program, in order to reach unexpected states<br />Fuzz explores the points that programmers and testers leave out due to assumption<br />(Data that does not necessarily map to harmful inputs, or to valid inputs)<br />Purpose<br />Built-in code assertions, Coding and state assumptions<br />Validation of: Robustness, Dependability (Availability) and Security (many times it finds buffer overflows)<br />Copyright © CC:BY-NC-SA 2007-11<br />
  24. 24. Be only as smart as you have to <br />Fuzz Testing – Approaches<br />Smart Fuzzers:<br />Organize the input data so it would pass initial filters on the software (or the environments)<br />Dumb Fuzzers:<br />Data is close to random and chaos, in order to avoid assumption pitfalls<br />Test approach:<br />Simple, automated, no assumptions or objective<br />Copyright © CC:BY-NC-SA 2007-11<br />
  25. 25. Fuzz Testing – Reactions<br />Contrary Reactions are similar, but usually in less extent<br />It will never happen in real life<br />A user will not do that<br />It should fail in such conditions<br />We can not fix it<br />When fuzz finally finds a bug it can be very severe, (crashes? overflows? data loss?)<br />Can take little active time from testers, adoption mostly requires initial investment.<br />Copyright © CC:BY-NC-SA 2007-11<br />
  26. 26. Fuzz Testing – Failures to look for<br />Crashes<br />Failing built-in Code Assertions<br />Undesired (or impossible) states<br />Wrong Error messages<br />Absence of error messages<br />Copyright © CC:BY-NC-SA 2007-11<br />
  27. 27. Fuzz Testing – Example #1<br />GUI Fuzzer:<br />Sends random keyboard and mouse events all over the application.<br /><ul><li>Solitaire
  28. 28. MS Paint</li></ul>Copyright © CC:BY-NC-SA 2007-11<br />
  29. 29. Fuzz Testing – Example #2<br />File Fuzzer:<br />Manipulates a file in order to create a set of randomly modified files.<br /><ul><li>File Fuzzerdemo
  30. 30. WordPad:</li></ul>Gives an error message.<br /><ul><li>Open Office:</li></ul>Crashes!<br /> and MS Word too!!<br />Copyright © CC:BY-NC-SA 2007-11<br />
  31. 31. Fuzz Testing – Examples #3<br /><ul><li>CD-Rom driver:</li></ul>Interface is easy, the Peach Frameworkmakes it automatic<br /><ul><li>Web HTTP:
  32. 32. POST Fuzzing
  33. 33. Web Fuzzer Fuzzing</li></ul>Web fuzzers are very diverse, and are available in many forms<br />Copyright © CC:BY-NC-SA 2007-11<br />
  34. 34. Comments on Fuzz Testing<br />Finding bugs can take a long time<br />Knowledge of the internal flows of the software is no necessary, and it can even hurt<br />Does not necessarily find software weaknesses in the shortest amount of time<br />Not every issue found is exploitable<br />Developing a smart framework is important in order to achieve efficiency.<br />Copyright © CC:BY-NC-SA 2007-11<br />
  35. 35. Fuzz Testing Tools Examples<br />Free tools:<br />GUI and Command line:<br />Fuzzer by the University of Wisconsin<br />File manipulation:<br />FileFuzz<br />Fuzz Framework:<br />Peach Fuzzer<br />List of fuzzers:<br />http://www.infosecinstitute.com/blog/2005/12/fuzzers-ultimate-list.html<br />http://www.computerdefense.org/2007/01/15/fuzzing-tools/<br />Commercial tools:<br />Network Protocols:<br />Codenomicon’sDefensics<br />Ixia<br />Copyright © CC:BY-NC-SA 2007-11<br />
  36. 36. Fault Models and Fuzz Summary<br /><ul><li>We can reach hard-to-reach-in-lab scenarios with these techniques
  37. 37. These tests are a good way to prevent(some) surprises
  38. 38. Impact of such bugs can be very high!
  39. 39. There are tools available, many of them free
  40. 40. You can build your own
  41. 41. this is not a way of doing,</li></ul> It is a way of thinking<br />Copyright © CC:BY-NC-SA 2007-11<br />
  42. 42. Fault Models and Fuzz Techniques - Learn More<br />Wikipedia:<br />Fuzz: http://en.wikipedia.org/wiki/Fuzz_testing<br />Fault Injection: http://en.wikipedia.org/wiki/Fault_injection<br />University of Wisconsin-Madison Fuzz Articles and Software:<br />http://www.cs.wisc.edu/~bart/fuzz/<br />Look out! It’s the fuzz:<br />http://iac.dtic.mil/iatac/download/Vol10_No1.pdf<br />Fault Injection:<br />“How to Break Software” book by James Whitaker<br />A study on fuzzing effectiveness:<br />http://www.docstoc.com/docs/53958850/Fuzz-By-Number<br />Fuzzing Examples at the Open Wen App Security Project:<br />http://www.owasp.org/index.php/Fuzzing<br />Attack your programs before someone else does:<br />http://www.whitestar.linuxbox.org/pipermail/fuzzing/2006-November/000168.html<br />Copyright © CC:BY-NC-SA 2007-11<br />
  43. 43. Fault Models and Fuzz<br />Credits:<br /><stripped in web version><br />Questions?<br />?<br />Copyright © CC:BY-NC-SA 2007-11<br />