Analyzing the Effectivess of Web Application Firewalls

Larry Suto – November 2011 Analyzing The Effectiveness of Web Application Firewalls Analyzing the Effectiveness of Web Application FirewallsA benchmark study of eight web application firewalls and intrusion prevention systems by Larry Suto Application Security Consultant San Francisco November 2011 1

Larry Suto – November 2011 Analyzing The Effectiveness of Web Application Firewalls Table Of Contents Analyzing the Effectiveness of Web Application Firewalls Table Of Contents 1. Executive Summary 2. Key Findings Overall Findings Findings: Baseline Tuned Findings: DAST/Scanner Tuned 3. Evaluation Criteria & Results Evaluation Notes Barracuda Networks: Barracuda 660 Citrix: NetScaler NS9.3 Build 48.6.nc DenyAll: rWeb 4.0 - Core manager v0.3.0.220 F5: ASM 3900 Imperva: SecureSphere 8.0 Trustwave: ModSecurity 2.6.2 SourceFire: SourceFire 3D Sensor 2100 - 4.9.1.6 build 150 Unnamed IPS 4. Conclusions 5. Challenges False Positive Risk WAF Evasion 6. Background 7. Methodology Overview Constraints and Assumptions Targets of Attack (TOA) Testing Tools Experimental Tests Setup TEST BL — Baseline Test Vulnerabilities Included Configuration Notes 8. Future Directions 9. References 10. Bibliography 11. Acknowledgements Appendix A: Definition of Terms and Acronyms Appendix B: Test sites and vulnerability data 2

Larry Suto – November 2011 Analyzing The Effectiveness of Web Application Firewalls 1. Executive SummaryThis study evaluated the effectiveness of Web Application Firewalls (WAFs) and Intrusion PreventionSystem (IPS) devices at protecting web applications against external attacks in comparison to eachother. In addition, this study provides general guidelines about the ease of use, and factors affecting theeffectiveness of web application protection solutions.The study examined the following cross-section of modern WAFs and IPSs both proprietary and opensource. (in alphabetical order): ● WAF’s: Barracuda 660, Citrix NetScaler, DenyAll rWeb, F5 ASM, Imperva SecureSphere, ModSecurity ● IPS’s: Sourcefire 3D, Unnamed IPSEach of the eight systems was evaluated in two separate tests. Each of the tests leveraged the devices indifferent configurations. The tests were as follows: 1. Baseline Tuned: Tuned configuration requiring 1 day or less of an experienced expert’s time 2. DAST : Trained solutions by application generated filters from Dynamic Application Security Testing (DAST) software products.DAST tools perform automated web application vulnerability scans. Some of the DAST solutions nowgenerate filters which can be imported into WAF and IPS protection policies.The NTODefend DASTtuning solution was used in this study to generate filters for five of the eight tested platforms. The otherthree platforms are not yet supported the DAST tool used in this study. There are other DAST vendorsthat have WAF integration such as Cenzic and Whitehat Security but they were not used due to time andlogistical considerations. Future testing should compare these solutions as well.The following graph (Figure 1) shows a high-level summary of the test results - the % vulnerabilities foundwith baseline tuning versus the % vulnerabilities found when tuned with DAST generated filters. Figure 1 3

Larry Suto – November 2011 Analyzing The Effectiveness of Web Application FirewallsThe following table shows the same results of both tests in a table format and includes both the actualnumeric results as well as the percentages. Figure 2 2. Key Findings Overall FindingsThe most significant conclusions of the study are as follows: ● To maximize effectiveness, WAF solutions must be tuned by a trained professional. The study found that basic tuning takes an average 3.5 hours by a informed user or expert. (see Figure 6) ● For the solutions supported, there was an average improvement of 36% more vulnerabilities blocked when the DAST generated filters were applied. (see Figure 3 below) ● IPS solutions are not designed for and are not very effective at blocking web application vulnerabilities with their default settings. However; when tuned by DAST tool generated filters, IPS solutions can be as or more effective than several of the WAF’s. (see Figure 3 below)NTODefend does not yet support for Barracuda, Citrix, or F5 Figure 3Additional general conclusions from this study are as follows: ● A web application firewall is a recommended element of the security defense strategy. ● WAF and IPS solutions are both capable of providing effective protection, but neither solution is a sufficient application defense strategy in isolation. They need to be accompanied by solutions & processes such as DAST and Static Analysis Security Testing (SAST) tools, risk management etc. ● Organizations should understand and test the differences of their solution’s detection and blocking effectiveness before and after training with DAST generated filters. ● Rule creation requires detailed knowledge of both the application and the WAF or IPS solution. It is not practical for a general user to produce effective filters in a reasonable amount of time. Using an automated solution made the rules generation part of the experiment manageable, otherwise any manual attempt at generation of such rules would require a deep understanding of regular expressions, several target systems such as SQL databases, and a deep consistency across all rules to ensure that they do not violate each other, etc. ● Training with a DAST tool as only as effective as the tool’s output. Care should be taken that the tool has a quality ruleset and that false positives have been remediated. 4

Larry Suto – November 2011 Analyzing The Effectiveness of Web Application Firewalls ● DAST tuning solutions can significantly improve the effectiveness of WAFs if the WAFs allow the DAST to create new customer rules (e.g. Sourcefire and ModSecurity). Solutions that only allow DAST solutions to turn on existing proprietary WAF rules (e.g. Imperva) do not benefit from such improvement. The improvement of the WAFs and IPS solutions is, however, dependent on the quality of the DAST tuning system. Findings: Baseline TunedFrom the first test, it became clear that considerable vendor-specific knowledge and understanding isrequired to properly configure and tune the WAF and IPS solutions. The key findings from the first testwhere the solutions were tuned by an experienced expert were as follows: ● The WAFs are fairly effective at detecting and defending web attacks on their own; the most effective solution found 88% of the vulnerabilities known in the test applications while the average effectiveness across all WAF solutions was 62%. ● The IPS solutions are not designed to protect web applications and were not very effective at defending them during this test. Findings: DAST/Scanner TunedTest two yielded the following results when WAF and IPS solutions were trained by DAST generatedfilters. (See Figure 3 below) ● The IPS solutions improved by an average of 60% bringing up their performance at-par or better than the trained/configured WAFs; with their overall blocking effectiveness averaging 82%. ● In this study, the supported WAF’s that were tuned with DAST solution improved an average of 19% from their baseline tuned state. 3. Evaluation Criteria & ResultsThroughout the study, results and information were captured according to the study’s methodology. Threecriteria were defined by which to score the solutions, then the weighting and specific scoring guidelineswere defined for each criteria. The following table details the evaluation criteria and their weightings: Figure 4 5

Larry Suto – November 2011 Analyzing The Effectiveness of Web Application FirewallsThe results closely matched the results from the percentage blocked scores Figure 5The following table details the summary of the results from both tests: Figure 6 Evaluation NotesList of the defensive products used and experiences using them for the testing. Barracuda Networks: Barracuda 660 ● Pros: The WAF is fairly full featured and has a responsive and well laid out web GUI. Basic signatures are easy to enable. It offers above average protection against attacks. With some work, advanced signatures can be added. ● Cons: Advanced tuning and signature enablement is a bit counterintuitive and not easy to enable or execute. The signature strategy seems to be focused mostly on regular expressions which sometimes are difficult to get right. The default signatures/basic tuning processes were fairly effective but the strict mode did not seem to improve blocking accuracy. Learning mode is not recommended out of the box because of performance issues. ● Support: Barracuda’s support is very responsive and adequately addressed all operational issues. They assisted in turning on all the default necessary signatures and demonstrated how to enable and apply the stricter signatures as well. 6

Larry Suto – November 2011 Analyzing The Effectiveness of Web Application Firewalls Citrix: NetScaler NS9.3 Build 48.6.nc ● Pros: The evaluation version of the Netscaler platinum is full featured and includes the web application firewall feature. The setup was straightforward. Accuracy of blocking was decent in the tests. ● Cons: Management interface could be a little more intuitive. Policy configuration and management could be better. ● Support: Relatively little support was required to perform the setup. DenyAll: rWeb 4.0 - Core manager v0.3.0.220 ● Pros: The web interface is easy to use, and the default blacklist is easy to configure. ● Cons: While the interface does allow for custom rules, it does not have a method to optimally import a custom blacklist such as from NTODefend. ● Support: DenyAll support was very helpful in providing instructions to install a custom blacklist using the command line interface. F5: ASM 3900 ● Pros: Excellent appliance quality. Great intuitive and snappy web based interface. Once basic configuration is accomplished, there are a lot of options for tuning the protection policies. With basic tuning F5 performed very well in the tests. The F5 sits on a very advance load balancing platform and provides other advanced security features such as DDOS protection. It is supposed to do well against HTTP resource exhaustion attacks. ● Cons: For users not familiar with the F5 product line, the setup can be quite challenging. ● Support: F5 was very helpful and gave a comprehensive walk-through of the tool including the basic setup and policy configuration. Imperva: SecureSphere 8.0 ● Pros: Imperva has a high quality protection engine with a very robust set of basic policies. It is also one of the easier WAFs to configure and provides the most value straight out of the box. ● Cons: The Imperva UI while providing a lot of information is a bit difficult to navigate. Modifying existing policies to handle exceptions can be unwieldy. Creating custom policies can be difficult. System can be be chatty with false positives during learning. ● Support: Imperva was very helpful in assisting with setting up the appliance and ensuring that everything was properly configured.The recommended implementation is to allow the appliance to collect traffic to develop and application profile before stricter rules are applied. Thus there is a bit of a grey area between what a default policy and a stricter applied policy might be. Nevertheless, after turning on a reasonable amount of the policies, it became clear that it was a very effective solution. Trustwave: ModSecurity 2.6.2 ● Pros: Free open sourced software makes it lowest acquisition cost. Easy to integrate with Apache. An expert user can modify filters to accomplish nearly any task ● Cons: Limited only to Apache users, and is installed on the application server, thereby limiting scalability options. There is no user interface. Requires command-line activity and numerous config files. ● Support: Did not use anything other than the projects forums and mailing lists. Live support is purchased separately from 3rd party consulting companies. 7

Larry Suto – November 2011 Analyzing The Effectiveness of Web Application Firewalls SourceFire: SourceFire 3D Sensor 2100 - 4.9.1.6 build 150 ● Pros: The web interface is very full featured, and importing/enabling custom rules is easy to accomplish. SourceFire supports the SourceFire/NTODefend solution as a WAF for PCI section 6.6 compliance. ● Cons: Not useful on its own to function as a WAF. Had some troubles with clearing rules and reloading them, but their support team was able to clear up the problem quickly. ● Support: Sourcefire was very helpful when needed. There is also a very active community of support around the snort project which includes volunteers as well as Sourcefire paid staff. Unnamed IPSThis IPS vendor did not give authorization to be named in this study. The IPS solution supports Snort 2.8rules, which was used for this testing. ● Pros: The UI was very compact, appealing, and visually oriented. ● Cons: Not useful on its own to function as a WAF. ● Support: Their support team was very helpful in overcoming the few configuration difficulties encountered. 4. ConclusionsWAFs are a controversial technology. While some security professionals argue that they are easilyavoided and the only remediation to web application vulnerabilities is to fix the code [7], others argue theopposite [6]. Manual code reviews and code fixes are essential, however they take considerable amountof time: days, weeks or even months once a vulnerability becomes known to the developers. In casewhere users do not have access to source code for any number of reasons, (3rd party software, legacysoftware, external service etc.) there is no manual recourse for fixing bugs. For some organizations,contractual issues and elaborate change control processes means that known vulnerable applicationscan and do long periods without being remediated. This problem, where for a period of time a vulnerabilityis known, but not fixed at the source, is addressed by "well-trained" defensive tools like WAFs and IPS’swhere a vulnerability can be effectively patched while the code is being repaired.This study clearly demonstrated that using WAFs/IPS devices in default or minimal mode to protect webapplications are often not useful, and WAFs configured with the recommended policies and tuned canoften be bypassed by a moderately skilled hacker or an automated hacking tool. This has been welldocumented (WAF Evasion section in this document). What is interesting is that training these tools witha DAST filter generator can dramatically increase their effectiveness and make them a far more usefulpart of an enterprises application security strategy. Additionally, trained IPS devices can be effectiveat protecting Web Applications. And finally are we staring at the abyss of a flawed model at detectingattacks? If WAFs could be made more intelligent somehow the future is bright for this technology. 5. ChallengesThe WAF vendors have spent a significant amount of time making claims of one sort or another as to whytheir technology is better, and this data may not speak to all of the various claims and may not be fair tosome of the individual strengths of each technology. However, it is the fairest and most impartial modelthat the study could come up with to test these technologies in a vendor-neutral way. It is also importantto note that while the tests are complete as far as they go, that this test is not an exhaustive study andonly begins to scratch the surface of the problem.In particular, WAF evasion research was completely left out, due to time constraints and to keep thescope of the study reasonable. Even with this limited level of testing, several conclusions can be madewithout doubt. For instance, it is almost certain that even with a reasonable amount of learning and tuningmost WAFs will miss well-know and well understood vulnerabilities. Suffice it to say that the problemescalates polynomially once we start throwing in complications due to AJAX/JSON type sites or variousJava, ActiveX and Flash plugins. 8

Larry Suto – November 2011 Analyzing The Effectiveness of Web Application Firewalls False Positive RiskAlthough there was not time to complete a comprehensive study of false positives (the blocking of goodtraffic) A limited amount of effort was used to try an eliminated false positives, but in future tests this areashould have more attention. The primary focus in this study was blocking of known vulnerabilities. WAF EvasionAfter a cursory review it can be discovered that there are numerous examples in the field where itbecomes obvious that the status-quo solutions to run-time attack detection and containment technologyare routinely by-passed. At the time of the writing of this paper, the following examples were retrievedfrom the Web: ● Example 1: ModSecurity SQL Injection challenge held by Trustwaves SpiderLabs. In this challenge the ModSecurity WAF was tuned and set up to protect a set of four of the scanner vendor vulnerable websites. Coincidentally, this was a subset of the flawed websites used in the current study. So lets take a look at what happened. In one of the successful bypasses of the ModSecurity inbound CRS SQL Injection rules, an individual by the name of Johannes Dahse used a combination of MySQL comments and new line characters to evade the CRS SQL Injection filters. The problem was not in the actual filter itself, which used regular expressions by the way, but in the transformation function -t:replaceComments (Modsecurity transformation functions are a pipeline of normalization functions that modify the input so that the rule can detect the attack). In this case -treplaceComments altered the input (by ignoring standalone termination */ and simultaneously replacing unterminated /* comments with a space( 0x20)) so that the filter could not recognize it as an attack. In the end it was not possible to create a rule to remove SQL comments reliably so another rule with a complex regular expression was created instead to detect the presence of SQL comments themselves. ● Example 2: The second example comes of failed context-free matching techniques comes to us from a paper from the recent Defcon 19. The talk discussed the paper titled "Bypassing PHPIDS 0.6.5." One of the attacks presented bypassed "ALL" the PHPIDS rules as of 0.6.5. How could of this happened you ask? The flaw was the incorrect assumption that "attacks can never be repetitive". The heart of the problem was the following bad expression: Line 90 in ./lib/IDS/Converter.php (?:(.{2, }) 1{32,}) The regular expression matches any two characters which are repeated 33 or more times.The first match is a back reference created with the 1 and then 32 additional matches are required to trigger the regex. Even though the attack string “<script>alert(1)</script> is matched by 4 filter sets, if it is repeated 33 times it will bypass PHPIDS entirely. ● Example 3: JSReg issues has had issues as well. The details are here "http://code.google.com/ p/jsreg/wiki/Exploits", Not notice that one of them is a regular expression rewrite error: "the failure to strip the single line comment which then fools the regex rule into thinking that the code is a regex object and not function calls".The commonality between these examples is that all them are using pattern matching—in particularregular expression type of context free matching—with a little bit of AI and boundary restriction ofparameter values thrown in to detect attacks. We have already seen that pattern matching does not yieldoptimum results elsewhere in the industry, I refer to Anti-virus applications of signature detection, wherethe upper bound of successful detection seems unable to cross the 60% mark. I think that a defense 9

Larry Suto – November 2011 Analyzing The Effectiveness of Web Application Firewallsmechanism that can only protect against 70-80% of the attacks is like a house that’s missing a wall. Wehave a long way to go, and while the state-of-art against attackers is not where we need it to be, I thinkthat next section discusses some future directions that have promise. 6. BackgroundWhen comparing web application security solutions amongst each other for the purpose of assessingwhich one to choose and why consumers are faced with several difficulties. These difficulties arisefrom the inability to compare apples to apples when it comes to comparing Web Application Firewalls(WAFs), network Intrusion Prevention Systems (IPSs), and similar defense technologies that purportto safeguard against Web attacks [1]. Each Web application firewall vendor delivers different strengthsand weaknesses, and presents varying philosophies of how Web attacks should be defended against[2]. Security certification standards such as Common Criteria, PCI DSS 6.6, DIACAP etc. focus on thetheoretical for the most part, and do not provide a reasonable guideline for comparison, despite theirscorecard nature [1, 2, 3].Security pundits are at odds with each other on whether the problem can be solved using fully automatictechniques such as WAFs and IPSs or is it only solvable using partial automated tools such as static-analysis tools in conjunction with manual code review and remediation strategies. There is a list ofdo’s and don’ts; with advice like “DO consider WAF for performance monitoring”; “DON’T think it’s setand forget” [1]. Non-profit consortia provide an invaluable resource to the consumer, however they arevery sensitive to vendor-neutrality, and their core approach of staying that way is to keep out of thecomparative-studies field all together. It remains difficult to find standards that provide direct comparativequantitative measures, and especially difficult to find ones that can explain it to average users of thistechnology. So, as things stand, it falls on to the consumer to fund an internal study on some standardsbasis.The most promising guide for end-users seems to conduct such a study seems to be The WebApplication Firewall Evaluation Criteria (WAFEC) and its corresponding WAFEC Evaluation ResponseMatrix. However, WAFEC only provides only qualitative comparative analysis between potentialcandidate solutions. They still require extensive knowledge of each aspect of web technology and thisauthor found that Evaluation Response Matrix has no mention of relative importance of each criterion, norany method of how to measure evaluate over-all effectiveness of the targets of evaluation [5,6].Previous research on WAFs have been confined to vendor organizations and customer evaluationengagements. There have been some publications discussing the functionality of WAFs but most havebeen limited to discussions on defining WAF functionality and providing advice on what keep in mindwhen conducting private evaluations and deployments. See the appendix for some references.To the best of the author’s knowledge this is the first study to conduct a general purpose evaluation ofWAF and IPS systems’ ability to protect web applications using a uniform testbed with a simple and easilyreproducible methodology.An interesting thing to keep in mind is that in the literature a WAF is referred to as security mechanismwith an intimate understanding of the nature of HTTP traffic and web application vulnerabilities. Theconsensus is that a WAF is an HTTP focused implementation of general purpose IPS technology.This study is also one of the first to provide clear empirical evidence that without specialized configurationand tuning, a general purpose IPS provides limited protection against web application attacks. 10

Larry Suto – November 2011 Analyzing The Effectiveness of Web Application Firewalls7. Methodology OverviewThe study tested each solution against the same set of websites and web application prototypes toensure that the experiments were instantiated against well-known and well understood vulnerabilities.This provided a uniform baseline for comparison.This experimental study provides a quantitative comparison between various web application securitysolutions that include both Web Application Firewalls and Intrusion Prevention Systems and evaluatestheir relative effectiveness in detecting, reporting and thwarting web attacks. The purpose of this study isto create a foundation for such comparative studies. Constraints and AssumptionsThe experiments in the study were designed with following constraints and assumptions: ● A typical end-user (hereinafter user) wants to protect a vulnerable web application or website, hereinafter called the security target. ● The user can detect vulnerabilities within the security target, using an automated tool. ● The user, while knowledgeable about the Web application security principles, is not familiar with any vendor specific details for the products being evaluated. ● The user does not fully understand the underlying technological details such-as deep understanding of HTTP protocol negotiations. ● The user only has limited time and budget to perform these evaluations. ● This study is not an exhaustive dissertation of various advanced capabilities of the evaluated web application firewalls and intrusion prevention systems. Targets of Attack (TOA)The study is a comparative analysis of primary results from a set of controlled tests conducted on acollection of vulnerable site. The vulnerable site used is called Target of Attack (TOA). The followingTOAs were used in these experiments: ● Acunetix AspNet - http://testaspnet.vulnweb.com ● Acunetix TestPHP - http://testphp.vulnweb.com ● Cenzic Crackme - http://crackme.cenzic.com ● HP zerowebappsecurity - http://zero.webappsecurity.com ● IBM Testfire - http://demo.testfire.net ● NTO Web scan test - http://www.webscantest.comThese sites demonstrate manifestations of well known and well understood vulnerabilities. Testing ToolsThe study required use of at least one DAST product capable of creating filters for as many as possible ofthe above listed defensive products.The primary DAST products (vulnerability scanners) used were NT OBJECTive’s NTOSpider andMavituna’s Netsparker. NT OBJECTive’s NTODefend product was used for filter generation.There are other DAST vendors that have WAF integration such as Cenzic and Whitehat Security, andstand alone applications such as ThreadFix (previously called Vulnerability Manager) from Denim Group. 11

Larry Suto – November 2011 Analyzing The Effectiveness of Web Application FirewallsNT OBJECTive’s NTODefend product includes a QuickScan feature which tests for false positives (theblocking of good traffic) by sending both legitimate and attack payloads to confirm that only attack trafficis being blocked this was only used on two sites as a sanity check but expanding false positive would beimportant in future tests.The other tools and features were not used due to time and logistical considerations. Future testingshould compare these solutions as well. Experimental Tests SetupThe evaluation tests used the DAST scanners independently, as shown in Figure 1 to detect exploitablevulnerabilities in the TOAs. The study is comprised of the following tests: TEST BL — Baseline TestA security scan the security targets of vulnerable systems for bugs using two These Scanners are theused for subsequent tests for each of the Evaluation Target WAF or IPS. Figure 7Figure 1 (above) depicts the WAF setup as it relates to the scanner and protected application location forTests I and II.TEST I Scanning the security targets with a WAF/IPS in front of the vulnerable applications. In this testthe WAF/IPS is configured to protect the application with approximately of one half day of effort. Vendorrequested assistance is provided as necessary to ensure that the recommended protection policies are inplace. This is the Tuned/Trained mode.TEST II Scanning the security targets with a WAF in front of the vulnerable applications. In this test theWAF is configured to protect the application with a minimum of one half day effort. Vendor requestedassistance is provided as necessary to ensure that the recommended protection policies are in place. Athird-party rule generator is then used to import policies in order to improve the blocking capability of thesystem. This is the DAST or Scanner Trained mode.Vulnerabilities IncludedEach WAF/IPS is different in its specific methodology for finding and preventing attacks. For the purposesof this report, the types of vulnerabilities that were counted were those that are useful against customapplications, and which most users care about. These are: ● SQL Injection / Blind SQL Injection ● Cross Site Scripting / Persistent Cross Site Scripting ● Command Injection ● CSRF / HTTP Response Splitting ● Arbitrary File Upload attacks ● Remote File Include (PHP Code Injection) 12

Larry Suto – November 2011 Analyzing The Effectiveness of Web Application FirewallsThe vulnerabilities were recorded to cross reference which WAFs blocked which vulnerabilities in asimple format to track and compare the results.A complete listing of vulnerabilities blocked by each WAF was recorded to track the effectiveness of eachtechnology. This resulted in a fairly high degree of confidence that the extent of Blocked and Missednumbers are complete.Configuration NotesFor tuning and training the basic policy was enabled plus any settings that were recommended by thevendor in order to optimally configure the system in a reasonably effective protection mode. For many ofthe systems, I received direct assistance from the vendor in order to accomplish this.To then see how well customized training from an outside source would improve the protection, customfilters/rules were generated using the DAST tuning solution for each supported WAF and loaded them inand repeated the test. Again, this was done mostly when it was obvious that the basic tuning process ofthe WAF was going to involve a significant effort. The assumption was that protection would improve withtraining, and this was clearly proven during testing. The collected data is being made freely available foranyone to review and re-create.Additionally, it should be understood that more manual configuration effort could likely be used to improvethe results of the WAFs further, but time constraints are always going to be an issue in real world use, aswell as when compiling a study like this. 8. Future DirectionsSo what are the future directions? Surely we cannot do manual defense. Even today, there is a strongdebate on whether or not programmers can be taught to create flawlessly secure programs [6]. As wehave seen, automated defense systems that perform strictly signature or pattern based detection onlyachieve partial success at best, and will probably reach an upper bound that is well below the requiredthreshold of risk mitigation. So what does it all mean? Is run-time automated defense doomed to failure?I don’t think so. There are entities and companies doing research in interesting directions in computer-sciences, mathematics, artificial intelligence and other related areas that may prove to be much morepowerful weapons in fight against attacks. I think that the true solutions lies not just in ever intractableconstruction of more and more complex pattern- matching systems, but in approaches that fundamentallyredefine the problem and by doing so discover innovative approaches to solving this very real, veryimportant issue of protecting Web Application Assets. 13

Larry Suto – November 2011 Analyzing The Effectiveness of Web Application Firewalls9. References[1] Web Application Firewalls; What’s in a name?; Ramon Krikken, Copyright 2009 Burton Group; Presentation atInfoSec 2009[2] Web App Firewalls: How to Evaluate, Buy, Implement; Mary Brandel; 2009-06-11; http://www.cio.com.au/article/307044/web_app_firewalls_how_evaluate_buy_implement[3] Brian Soliman Blog, Technical Articles, 2011-07-25; Common Criteria (CC) for Information TechnologySecurity Evaluation; http://bryansoliman.wordpress.com/2011/07/25/common-criteria-cc-for-information-technology-security-evaluation/[4] WebAppSec.org; 2006; Web Application Firewall Evaluation Criteria (WAFEC) 1.0; Project Coordinator: OfreShezaf; http://projects.webappsec.org/f/wasc-wafec-v1.0.pdf[5] WebAppSec.org; 2006; WAFEC Response Matrix; http://projects.webappsec.org/f/WAFEC_Evaluation_Response_Matrix_1.0.xls[6] OWASP AppSec USA 2010 Keynote; Application Security — It’s a jungle out there[7] Time to Automate Web Defenses?; Robert Lemos; 2011-10-25;http://www.darkreading.com/vulnerability-management/167901026/security/attacks-breaches/231901651/time-to-automate-web-defenses.html10. BibliographyBrian Soliman Blog, Technical Articles, 2011-07-25; Common Criteria (CC) for Information Technology SecurityEvaluation http://bryansoliman.wordpress.com/2011/07/25/common-criteria-cc-for-information-technology-security-evaluation/Brickman, J. (2010) Common Criteria – a good concept in transformation http://community.ca.com/blogs/iam/archive/2010/03/25/common-criteria-a-good-concept-in-transformation.aspxF5 Networks and WhiteHat Security Solutions (2008); Vulnerability Assessment Plus Web Application Firewall(VA+WAF)11. AcknowledgementsThanks to all the WAF and IPS vendors for all their graciousness and patience. Things always seem totake longer than you expect. The more scrutiny we place our solutions under, the better for everyone.This is a nascent space with a lot of potential. Special thanks to:Kasey Cross at ImpervaIdo Berger and Tom Spector at F5 NetworksGrant Murphy at Barracuda NetworksRenaud Bidou at DenyAllAlso I would like to thank NT OBJECTives and Mavituna Security for DAST scanner and filter support.In addition I would like to thank Amir Khan for helping host the systems and Ahmed Masud for valuableeditorial commentary. 14

Larry Suto – November 2011 Analyzing The Effectiveness of Web Application FirewallsAppendix A: Definition of Terms and AcronymsBlacklist - The list of items, that when matched by a pattern matching system that decides to authorizean operation, will result in rejection of such an authorization.Brute-force attack - An attack that simply tries all possible permutations of a given scenario to discoverand exploit any possible vulnerabilities.Cross-site scripting - The ability to include executable code (typically Javascript code) in non Same-origin contexts.Origin - Defined as the tuple (domain name, application layer protocol, port number) for purpose ofresource access control. Resources (objects, methods, instances) can access other resources createdfrom the same origin. See Same-origin contextSame-origin context - A programmatic boundary in browser-side programming languages that restrictsaccess to methods, properties and objects across different domains (origins). See also: OriginDeny-by-default - An authorization paradigm where conditions, input, output and processing actionsmust be explicitly allowed (using a Whitelist). The rationale behind deny-by-default paradigm is tosignificantly reduce the possibilities of unexpected or malicious behavior.Denial of Service attack - Any attack that affects the availability and normal usage of a service.DoS - Denial of Service: See also Denial of Service AttackDDoS - Distributed Denial of Service. This variation uses multiple clients to perform attacks: See alsoDenial of Service AttackFuzzing - See fuzz testingFuzz Testing - Fuzz testing or fuzzing is a software testing technique that applies random unexpecteddata to inputs of a computer program. The program is then monitored for unanticipated behavior.HTTP Response Splitting - An attackers ability to send a single HTTP request that forces the webserver to form an output stream, which is then interpreted by the target as two HTTP responses insteadof one response. This type of vulnerability can be exploited to perform several web application basedattacks.OS Command Injection - An injection attack that utilizes the lack of input validation on web applicationsthat execute out to command line applications. An attacker can use such a vulnerability to executearbitrary programs on the host operating system.SQL injection attack - SQL injection attack utilizes the lack of input validation vulnerability of web datathat an application uses to build a SQL query string. An attacker can use such a vulnerability construct aninput in such a way as to execute arbitrary code on the back-end database.Whitelist - See Deny-by-defaultXSS - See Cross-site Scripting 15

Larry Suto – November 2011 Analyzing The Effectiveness of Web Application FirewallsAppendix B: Test sites and vulnerability data Summary of raw results 16

Larry Suto – November 2011 Analyzing The Effectiveness of Web Application Firewalls Acunetix AspNet - http://testaspnet.vulnweb.com 17

Larry Suto – November 2011 Analyzing The Effectiveness of Web Application Firewalls Acunetix TestPHP - http://testphp.vulnweb.com 18

Larry Suto – November 2011 Analyzing The Effectiveness of Web Application Firewalls Cenzic Crackme - http://crackme.cenzic.com 19

Larry Suto – November 2011 Analyzing The Effectiveness of Web Application Firewalls HP zerowebappsecurity - http://zero.webappsecurity.com 20

Larry Suto – November 2011 Analyzing The Effectiveness of Web Application Firewalls IBM Testfire - http://demo.testfire.net 21

Larry Suto – November 2011 Analyzing The Effectiveness of Web Application Firewalls NTO Web Scan Test - http://www.webscantest.com 22

Analyzing the Effectivess of Web Application Firewalls

by Larry Suto on Nov 14, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

Embed Views

Statistics

Analyzing the Effectivess of Web Application Firewalls — Document Transcript