Your SlideShare is downloading. ×
0
Fuzzing101 - webinar on Fuzzing Performance
Fuzzing101 - webinar on Fuzzing Performance
Fuzzing101 - webinar on Fuzzing Performance
Fuzzing101 - webinar on Fuzzing Performance
Fuzzing101 - webinar on Fuzzing Performance
Fuzzing101 - webinar on Fuzzing Performance
Fuzzing101 - webinar on Fuzzing Performance
Fuzzing101 - webinar on Fuzzing Performance
Fuzzing101 - webinar on Fuzzing Performance
Fuzzing101 - webinar on Fuzzing Performance
Fuzzing101 - webinar on Fuzzing Performance
Fuzzing101 - webinar on Fuzzing Performance
Fuzzing101 - webinar on Fuzzing Performance
Fuzzing101 - webinar on Fuzzing Performance
Fuzzing101 - webinar on Fuzzing Performance
Fuzzing101 - webinar on Fuzzing Performance
Fuzzing101 - webinar on Fuzzing Performance
Fuzzing101 - webinar on Fuzzing Performance
Fuzzing101 - webinar on Fuzzing Performance
Fuzzing101 - webinar on Fuzzing Performance
Fuzzing101 - webinar on Fuzzing Performance
Fuzzing101 - webinar on Fuzzing Performance
Fuzzing101 - webinar on Fuzzing Performance
Fuzzing101 - webinar on Fuzzing Performance
Fuzzing101 - webinar on Fuzzing Performance
Fuzzing101 - webinar on Fuzzing Performance
Fuzzing101 - webinar on Fuzzing Performance
Fuzzing101 - webinar on Fuzzing Performance
Fuzzing101 - webinar on Fuzzing Performance
Fuzzing101 - webinar on Fuzzing Performance
Fuzzing101 - webinar on Fuzzing Performance
Fuzzing101 - webinar on Fuzzing Performance
Fuzzing101 - webinar on Fuzzing Performance
Fuzzing101 - webinar on Fuzzing Performance
Fuzzing101 - webinar on Fuzzing Performance
Fuzzing101 - webinar on Fuzzing Performance
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Fuzzing101 - webinar on Fuzzing Performance

423

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
423
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
10
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. WWW.CODENOMICON.COM Fuzzing 101: Fuzzing Performance Ari Takanen, CTO of Codenomicon Nov 10th, 2010
  • 2. WWW.CODENOMICON.COM 2  Fuzzing 101: • The webcast series for fuzzing industry • Vendor neutral presentations on fuzzing technologies and use-cases • Includes invited speakers from the industry  Codenomicon: • Fuzzing research since 1996 • 2001, Spinoff from University of Oulu • 50-100% annual growth in number of customers and revenues in fuzzing industry About Fuzzing 101 and Codenomicon
  • 3. WWW.CODENOMICON.COM 3 About Ari Takanen  The Past: Researcher and Lecturer • 1998-2002 • University of Oulu • OUSPG/PROTOS research group • Software Quality related lectures  The Present: Entrepreneur and Evangelist • 2001-today • CTO of Codenomicon • Evangelist: 10 conference talks every year • Author of two books: • VoIP Security • Fuzzing
  • 4. WWW.CODENOMICON.COM 4 Agenda  Intro: • Fuzzing Overview • Fuzzing Performance  Demo in Theory • Multi-threaded execution • Test execution optimization • Command-line use  And if time allows: Demo in Practice for the interested
  • 5. WWW.CODENOMICON.COM Fuzzing Overview Positive results from negative testing!
  • 6. WWW.CODENOMICON.COM 6 What is Testing - Three Flavors  Feature/Conformance tests validate positive requirements with use cases and test cases  Performance/Load testing repeats one of the feature tests to validate performance limits  Robustness/Fuzz testing uses misuse cases to verify software reliability with unexpected inputs
  • 7. WWW.CODENOMICON.COM 7 Security Vulnerability = Just A Bug
  • 8. WWW.CODENOMICON.COM 8 Codenomicon Labs Test Results http://www.codenomicon.com/labs/results
  • 9. WWW.CODENOMICON.COM 9 Fuzz Test Effectiveness against WiFi
  • 10. WWW.CODENOMICON.COM 10 Some Helpful Definitions  Vulnerability – a weakness in software, a bug  Threat/Attack – exploit/worm/virus against a specific vulnerability  Protocol Modeling – Technique for explaining interface message sequences and message structures  Fuzzing – process and technique for security testing  Anomaly – abnormal or unexpected input  Failure – crash, busy-loop, memory corruption, or other indication of a bug in software
  • 11. WWW.CODENOMICON.COM 11 Fuzzing In Short  Fuzzing means crash-testing  Also called: • Negative testing • Robustness testing • Grammar testing  Based on sending systematically broken (rarely random) inputs to a software, in order to crash it  We will ignore random mutator fuzzers for now  Two techniques of smart model-based fuzzers: • Template-based • Specification-based
  • 12. WWW.CODENOMICON.COM 12 Model Based Fuzzing Techniques  Template Based Fuzzing • Quality of tests is based on the used seed and modeling technique • Very quick to develop, but slow to run • Editing requires deep protocol know-how • Good for testing around known vulnerabilities  Specification Based Fuzzing • Full test coverage • Always repeatable • Short test cycle, more optimized tests • Easy to edit and add tests
  • 13. WWW.CODENOMICON.COM 13  Precision is about attack surface/protocol coverage  All interfaces/protocols tested?  All message sequences tested?  All message structures tested?  All data definitions tested?  All “tags” (values) tested?  Accuracy is about anomaly coverage  Anomaly categories? SQL? Buffer overflow?  All values: 0..65k, a..z, 0x00..0x255 ?  Combinations of anomalies? Coverage
  • 14. WWW.CODENOMICON.COM Anomaly Coverage Selection
  • 15. WWW.CODENOMICON.COM 15 Traffic Capture Fuzzing
  • 16. WWW.CODENOMICON.COM 16 Traffic Capture Fuzzing Results  Test against samba seems to find zero-day
  • 17. WWW.CODENOMICON.COM 17 Key Questions in regards to Fuzzing  Test Coverage: • Q: Which interfaces and protocols are tested? • A: Understanding of threat models and attack surface are the most critical starting point to fuzzing • Q: How well something is tested? • A1: Specification coverage explains how deep knowledge the fuzzer has on a tested interface, and is typically represented by protocol models or protocol templates • A2: Anomaly coverage explains how well each protocol element is tested, and is the hardest metric to measure
  • 18. WWW.CODENOMICON.COM Fuzzing Performance Faster, Better, More!
  • 19. WWW.CODENOMICON.COM Fuzzing Scalability  Software-based Fuzzers scale for all testing needs • The throughput depends only on the available hardware • The entire Network Under Test (NUT) can be virtualized • Software fuzzers can attack every interface, and against all protocol layers • XML • HTTP • TLS • TCP/IP
  • 20. WWW.CODENOMICON.COM 20 Why Fast Speed Fuzzing?  “One of the most important aspects of fuzzing is how fast you can execute test cases. The faster you can execute test cases, the more test cases you can run and the more vulnerabilities you will find.” - Dr. Charlie Miller from Independent Security Evaluators. "
  • 21. WWW.CODENOMICON.COM Generating Load with Defensics  Full model-based message sequences  Options for monitoring, instrumentation, fuzzing, …  Amount of available CPUs and Logging Level impact performance  Some speed records (test cases per second): • TLS: 2.400 tc/s • IPv6: 4.500 tc/s • HTTP: 16.000 tc/s • DNS: 41.000 tc/s (with capture replay)
  • 22. WWW.CODENOMICON.COM 22 Motivation for our Performance Study  Test our test generation efficiency for different type of protocol fuzzers using high-end servers  Analyze benefits (test efficiency, ROI) of software-based fuzzers to appliances
  • 23. WWW.CODENOMICON.COM 23 Raw Fuzzing Performance Metrics  Packets or bytes per second does not really work as each fuzz test case can consist of several packets back and forth  Test cases, or full protocol transactions, generated by each suite provides more insight  Performance is generated by having fuzzers running in parallel  These result in “test cases per second” or tc/s
  • 24. WWW.CODENOMICON.COM 24 Defensics Benefits  No hardware constraints  No operating system constraints  Parallel processes and threads
  • 25. WWW.CODENOMICON.COM 25 Performance Bottlenecks and Metrics  Network bandwidth: • Can be measured with number of full protocol transactions, packets, and data volume  CPU usage by different components of Defensics • Test generator(s) • Test driver(s)  Memory usage of Defensics components  Most interesting metric: • New type of bugs related to performance?
  • 26. WWW.CODENOMICON.COM 26 Fuzz Test Can Generate 50x Load
  • 27. WWW.CODENOMICON.COM Generating Load with Dell R910
  • 28. WWW.CODENOMICON.COM Quick-Start to Fast-Speed Fuzzing Demo
  • 29. WWW.CODENOMICON.COM 29 Test Plans Using Command-Line  With some fuzzers, you can run the tools faster on command-line  With Defensics, the GUI is sometimes even faster  E.g. with a laptop with dual-core Intel T7500 processor (with default tool settings): • 10.000 x valid-case in loop • using GUI: 30 seconds • using command-line: 35 seconds
  • 30. WWW.CODENOMICON.COM 30 Running Fuzzers in Parallel  Defensics GUI has options to run several suites in parallel • 2 x 5.000 x valid-case in loop • using GUI: 22 seconds (about 50-100% faster)
  • 31. WWW.CODENOMICON.COM 31 Using Multi-threaded Mode  When CPU resources are available, multi- treaded mode is often best way to speed up testing  Uses less memory  Note that all parallel execution creates more false positives  Let’s look at that and other speed optimization hints now in live demo...
  • 32. WWW.CODENOMICON.COM Conclusions Not so fast!
  • 33. WWW.CODENOMICON.COM 33 Performance Value  Users can now… • Increase test coverage through parallel tests • Speed up testing • Find performance bottle-necks • Integrate Defensics also in other functional tests  Remember that Fuzzing ROI is measured in: • TCO vs. Value of Security Findings / Cost of Security Compromise • Reduced maintenance costs, failure reproduction costs • Better customer confidence, higher retention rate of customers • In performance: software-based tools can be significantly cheaper (and faster) than appliances
  • 34. WWW.CODENOMICON.COM 34 More News from Codenomicon  Facebook: • Become fan of Codenomicon and Fuzzing  Twitter: • CodenomiconLTD  Codenomicon Website: • Newsletter every second month  Customer portal • The Backstage! Contact us for access!
  • 35. WWW.CODENOMICON.COM 35 Our Book On Fuzzing!  http://www.fuzz-test.com/book/  Takanen, DeMott and Miller: “Fuzzing for Software Security Testing and Quality Assurance”  Aimed at the general public, you do not need to be a security specialist to read this book  Purpose of the book is to teach next-gen testing approaches to: • Software practitioners • Security engineers • Academics
  • 36. WWW.CODENOMICON.COM PROACTIVE SECURITY AND ROBUSTNESS SOLUTIONS THANK YOU – QUESTIONS? “Thrill to the excitement of the chase! Stalk bugs with care, methodology, and reason. Build traps for them. .... Testers! Break that software (as you must) and drive it to the ultimate - but don’t enjoy the programmer’s pain.” [from Boris Beizer]

Ă—