Fault Models and Fuzzing

Fault Models and Fuzz Techniques
Shmuel Gershon
STAREAST 2011
Copyright © CC:BY-NC-SA 2007-11, Shmuel Gershon.

About...
Shmuel Gershon
Testing Engineer
http://testing.gershon.info
Creator of Rapid Reporter
Twitter: @sgershon, Skype: sgershon, shmuel@gershon.info
Disclaimer:
Names and Brands referenced herein may be claimed as property of third parties
Views expressed in this presentation are solely my own, and do not in any manner represent the views of my employer
Information in this presentation is provided 'as is' without any warranties or representations of any kind
Copyright © CC:BY-NC-SA 2007-11

Fault Models and FuzzTalk Objectives
Understand the principles of the techniques
Meet tools available for both techniques
Learn to apply the methods (in different app types)
Perceive drawbacks and difficulties
Lay down a foundation for further research
Have at least one new idea for your tests :)
Testing is questioning a product in order to evaluate its value to a person that matters
Adapted from Jerry Weinberg + James Bach + Michael Bolton

Putting Concepts into Context
Availability
Robustness
Dependability
Security
Reliability
Stability

Fault Models and FuzzTalk Outline
Fault Models
Overview
Examples + Demos
Pitfalls and Tricks
Fuzzing
Overview
Examples + Demos
Pitfalls and Tricks
Summary and Questions

Fault Models OverviewConceptual Diagram
OS – Disk Storage
Injection
Runtime
Fault
Layer
Application Under Test
OS – User I/O
OS - Memory
OS - Network

Fault Models Overview – Definition
Definitions:
Introducing faults in order to test (error handling) code paths, that might otherwise rarely be followed
Assess the robustness of software by checking it's reaction to adverse events
Purpose
Validation of: Robustness, Dependability (Availability) and Security
Also known as: Recovery Code test, Fault Injection, Negative test, Error Handling, Stress test...

Fault Models Overview – Approaches
Random Runtime Fault Injections:
A tool control the type, time and location to ‘attack’. May modify the coverage of a set of tests (issues found may be harder to reproduce or debug).
Initiated Runtime Fault Injection:
Specific tests in which the faults are controlled at specific point, aiming a clear error handling flow.
Bugs found in this approach may be easier to reproduce, accept and fix.
(This is the approach we'll focus on).

Fault Models Overview – Reactions
Reactions against Runtime Fault Bugs
It will never happen in real life
A user will not do that
It should fail in such conditions
It is an unsupported scenario
We can’t fix this
It is a third-party problem
How would you reply?
Whose decision it ultimately is?

Fault Injection – Example #1
Memory Starvation:
Refuses to allocate a memory upon request (Insufficient Memory).
Pinball:
Silently skips the action,no harm done.
WordPad:
Disappears with your most valuable work

Network Errors:
Simulates an error responses for common network requests and resources.
Browser Wars:
One browser survives
The other crashes!

Low Resources / Slow or Clogged CPU
Computer behaves as a busy or old computer
(CPU is busy on other tasks / time scheduler starvation...)
Playing to win:
Can we escape from death
by simply slowing down the rest?

Fault Models – Notes
Finding bugs can take a long time (but you record important data during the process)
Abstract knowledge of the internal flows of the software is imperative for good results
Bugs are received with the comments seen earlier
It is difficult to assess the exposure of such bugs
These points can make it harder to adopt the practices in the company...
...so be sure to address them

Fault Injection Tools Examples
Canned Heat is free & easy, but buggy
Limitations:
Does not support .NET apps or Services
Bug: In some systems it will not load or needs to be reloaded between applications.
Others:
Verifier, AppVerifier
SlowProc, HeavyLoad
Holodeck - not free.
PIN and (soon) the random malloc blocker
In-House or Brute-Force tools

Fuzz
Fuzz
Fuzz
Fuzz
Fuzz
Fuzz
Fuzz TestingConceptual Diagram
OS – Disk Storage
Application Under Test
OS – User I/O
OS - Memory
OS - Network

Fuzz Testing – Definition
Definitions:
Providing random data which is free of preconceptions to the inputs of a program, in order to reach unexpected states
Fuzz explores the points that programmers and testers leave out due to assumption
(Data that does not necessarily map to harmful inputs, or to valid inputs)
Purpose
Built-in code assertions, Coding and state assumptions
Validation of: Robustness, Dependability (Availability) and Security (many times it finds buffer overflows)

Be only as smart as you have to 
Fuzz Testing – Approaches
Smart Fuzzers:
Organize the input data so it would pass initial filters on the software (or the environments)
Dumb Fuzzers:
Data is close to random and chaos, in order to avoid assumption pitfalls
Test approach:
Simple, automated, no assumptions or objective

Fuzz Testing – Reactions
Contrary Reactions are similar, but usually in less extent
It will never happen in real life
A user will not do that
It should fail in such conditions
We can not fix it
When fuzz finally finds a bug it can be very severe, (crashes? overflows? data loss?)
Can take little active time from testers, adoption mostly requires initial investment.

Fuzz Testing – Failures to look for
Crashes
Failing built-in Code Assertions
Undesired (or impossible) states
Wrong Error messages
Absence of error messages

Fuzz Testing – Example #1
GUI Fuzzer:
Sends random keyboard and mouse events all over the application.
Solitaire
MS Paint

Fuzz Testing – Example #2
File Fuzzer:
Manipulates a file in order to create a set of randomly modified files.
File Fuzzerdemo
WordPad:
Gives an error message.
Open Office:
Crashes!
 and MS Word too!!

Fuzz Testing – Examples #3
CD-Rom driver:
Interface is easy, the Peach Frameworkmakes it automatic
Web HTTP:
POST Fuzzing
Web Fuzzer Fuzzing
Web fuzzers are very diverse, and are available in many forms

Comments on Fuzz Testing
Finding bugs can take a long time
Knowledge of the internal flows of the software is no necessary, and it can even hurt
Does not necessarily find software weaknesses in the shortest amount of time
Not every issue found is exploitable
Developing a smart framework is important in order to achieve efficiency.

Fuzz Testing Tools Examples
Free tools:
GUI and Command line:
Fuzzer by the University of Wisconsin
File manipulation:
FileFuzz
Fuzz Framework:
Peach Fuzzer
List of fuzzers:
http://www.infosecinstitute.com/blog/2005/12/fuzzers-ultimate-list.html
http://www.computerdefense.org/2007/01/15/fuzzing-tools/
Commercial tools:
Network Protocols:
Codenomicon’sDefensics
Ixia

Fault Models and Fuzz Summary
We can reach hard-to-reach-in-lab scenarios with these techniques
These tests are a good way to prevent(some) surprises
Impact of such bugs can be very high!
There are tools available, many of them free
You can build your own
this is not a way of doing,
It is a way of thinking

Fault Models and Fuzz Techniques - Learn More
Wikipedia:
Fuzz: http://en.wikipedia.org/wiki/Fuzz_testing
Fault Injection: http://en.wikipedia.org/wiki/Fault_injection
University of Wisconsin-Madison Fuzz Articles and Software:
http://www.cs.wisc.edu/~bart/fuzz/
Look out! It’s the fuzz:
http://iac.dtic.mil/iatac/download/Vol10_No1.pdf
Fault Injection:
“How to Break Software” book by James Whitaker
A study on fuzzing effectiveness:
http://www.docstoc.com/docs/53958850/Fuzz-By-Number
Fuzzing Examples at the Open Wen App Security Project:
http://www.owasp.org/index.php/Fuzzing
Attack your programs before someone else does:
http://www.whitestar.linuxbox.org/pipermail/fuzzing/2006-November/000168.html

Fault Models and Fuzz
Credits:
<stripped in web version>
Questions?
?

Fault Models and Fuzzing

by sgershon on Apr 12, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

3 Embeds 8

Statistics

Fault Models and Fuzzing — Presentation Transcript

http://us-w1.rockmelt.com	6
https://twitter.com	1
http://twitter.com	1