Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Testing web application firewalls (waf) accuracy

1,894 views

Published on

This presentation discusses how to properly measure the accuracy of Web Application Firewalls. The presentation explain the 4 attributes that must be measured (FP, FN, TP, TN) and how to properly calculate a WAF's accuracy.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Testing web application firewalls (waf) accuracy

  1. 1. WAF Accuracy Testing Done Properly Introducing AWT framework Ory Segal, Director of Threat Research
  2. 2. ©2015 AKAMAI | FASTER FORWARDTM WAF Accuracy Lingo • Imagine a WAF that protects against 100% of all possible attack vectors …by blocking 100% of all HTTP requests • Accurate WAF testing requires you to measure: • How many real attacks got blocked (TP) • How much valid requests were allowed through (TN) • How much valid traffic was inappropriately blocked (FP) • How many attacks were allowed through (FN) • Lets talk about Precision, Recall, Accuracy, MCC…
  3. 3. ©2015 AKAMAI | FASTER FORWARDTM Precision, Recall, Accuracy, MCC % of blocked requests that were actual attacks % of attacks that were actually blocked % of decisions that were good decisions * MCC: http://en.wikipedia.org/wiki/Matthews_correlation_coefficient Correlation between WAF decisions and actual nature of requests Precision = tp tp+ fp Recall = tp tp+ fn Accuracy = tp+tn tp+tn+ fp+ fn MCC = tp×tn (tp+ fp)(tp+ fn)(tn+ fp)(tn+ fn)
  4. 4. ©2015 AKAMAI | FASTER FORWARDTM Lets Look at Some Examples A WAF’s accuracy needs to be measured both in its ability to block attacks, as well as it’s ability to allow good traffic through… WAF Type Requests Valid Attacks Blocked TP TN FP FN P R A MCC Real 1000 990 10 11 8 987 3 2 0.73 0.8 0.995 0.76 Off 1000 990 10 0 0 990 0 10 N/A 0 0.99 0 Always Block 1000 990 10 1000 10 0 990 0 0.01 1 0.01 0 Noisy 1000 990 10 31 8 967 23 2 0.26 0.8 0.975 0.45 Conservative 1000 990 10 2 2 990 0 8 1.00 0.2 0.992 0.45
  5. 5. ©2015 AKAMAI | FASTER FORWARDTM WAF Testing Framework Requirements • A tool that will send both valid traffic and real attacks • Easy addition of test cases (both valid & attacks) • Accuracy statistics gathering – FP, FN, TP, TN, P, R, A, MCC • Rich info about each test that was sent – full request, response, expected behavior, request nature • Reporting capabilities
  6. 6. ©2015 AKAMAI | FASTER FORWARDTM Introducing: Akamai WAF Testing Framework
  7. 7. ©2015 AKAMAI | FASTER FORWARDTM Akamai WAF Testing (AWT) Framework • Written in Python • Test cases are represented as textual files (.awt) • Options to create or add new test cases: • Write text files • Use a “Burp Extender” to record web interaction (meaningful requests only) • Transform Wireshark .pcap files (only ports HTTP traffic) • Multithreaded – can be very fast, or very “considerate” • Configurable and can work with any WAF • Intuitive XML & HTML reports • Easy debugging of FP/FN
  8. 8. ©2015 AKAMAI | FASTER FORWARDTM AWT Built-In Test Cases In order to accurately assess WAF, we collected test cases from the following sources: Retrieved valid traffic from Akamai’s Cloud Security Intelligence big data platform Recorded manual interaction with top “problematic” web sites Ported known “false positive” test cases from other tools Commercial web scanners Popular SQLi tools Exploits from the internet (fuzzers, exploit-db, … Traffic database is divided to 95% / 5% Automatic crawling of Alexa Top 100 internet sites Malicious traffic from Akamai’s Cloud Security Intelligence big data platform
  9. 9. ©2015 AKAMAI | FASTER FORWARDTM AWT Reports - Example

×