Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Why Johnny Still Can’t Pentest: A Comparative Analysis of Open-source Black-box Web Application Vulnerability Scanners

31 views

Published on

Slides for my Master in Computer Science thesis defense.

Published in: Education
  • Be the first to comment

  • Be the first to like this

Why Johnny Still Can’t Pentest: A Comparative Analysis of Open-source Black-box Web Application Vulnerability Scanners

  1. 1. Why Johnny Still Can’t Pentest: A Comparative Analysis of Open-source Black-box Web Vulnerability Scanners Rana Khalil Master Thesis Defense, 20/11/2018 School of EECS University of Ottawa Committee: Carlisle Adams (Supervisor) Guy-Vincent Jourdan Anil Somayaji (Carleton University) 1
  2. 2. Roadmap 1. Introduction 2. Methodology 3. Results 4. Conclusion 2
  3. 3. Roadmap 1. Introduction 2. Methodology 3. Results 4. Conclusion 3
  4. 4. Introduction • We use websites for everything: e-commerce, online banking, social networking, social media, etc. • Web security has become a major concern 4
  5. 5. How to Secure a Web Application? • A combination of techniques are used to secure web applications: • Static code analysis • Web application firewalls • Secure coding practices • Web application vulnerability scanners • Etc. • Focus of this research: Performing a comprehensive comparative analysis of the performance of six chosen scanners 5
  6. 6. Previous Work 6 • Suto’s case studies [10][11] • 2007 paper evaluated scanners in PaS mode • 2010 paper evaluated scanners in PaS and Trained modes • Benchmark applications: • Web Input Vector Extractor Teaser (WIVET) created in 2009 by Tatli et al. [12] • Web Application Vulnerability Scanner Evaluation Project (WAVSEP) created in 2010 by Chen [13] • Doupé et al.’s work on evaluating WAVS in both PaS and Trained modes on the WackoPicko application [15] • Several other studies include [14], [16] and [17]
  7. 7. Roadmap 1. Introduction 2. Methodology 3. Results 4. Conclusion 7
  8. 8. Methodology 8 Figure 2.1: Methodology Process Tool Selection Benchmark Selection Environment Setup Feature and Metric Selection Result Analysis
  9. 9. Tool Selection 9 • Chen’s evaluation [18] • Consultation with professional ethical hackers Table 2.1: Characteristics of the Scanners Evaluated
  10. 10. Benchmark Selection 10 • Benchmark applications • WIVET - contains 56 test cases that utilize both Web 1.0 and Web 2.0 technologies • WAVSEP - consists of a total of 1220 true positive (TP) test cases and 40 false positive (FP) test cases covering a range of vulnerability categories • WackoPicko - intentionally vulnerable realistic web application • Contains 16 vulnerabilities covering several of the OWASP Top 10 • Contains crawling challenges: HTML parsing, multi-step process, infinite web site, authentication, client-side code, etc.
  11. 11. Environment Setup 11 • Each scanner was run in two modes: • Default - default configuration setting • Configured 1. Maximize crawling coverage – changing configuration 2. Maximize crawling coverage – use of proxy 3. Maximize attack vector • WackoPicko test scans were further divided into two subcategories: • INITIAL – without authentication / publicly accessible • CONFIG - valid username/password combination • In total, each scanner was run eight times. Note: Tests performed in a VM that was restored to its initial state before every test run. Table 2.2: Steps Included in Configured Scan
  12. 12. Feature Selection 12 • Crawling coverage: • % of passed test cases on the WIVET application • Crawling challenges in the WackoPicko application • Vulnerability detection accuracy: • TP, FN and FP on the WAVSEP and WackoPicko applications • Speed: • Scan time on the WAVSEP and WackoPicko appliations • Reporting: • Vulnerability detected • Vulnerability location • Exploit performed • Usability: • Efficiency • Product documentation • Community support Crawling Coverage Detection Accuracy Speed WIVET WackoPicko WAVSEP Features Applications Figure 2.2: Feature Measurement
  13. 13. Metric Selection 13Table 2.3 Vulnerability Scores • Final ranking was calculated based on the crawling coverage and vulnerability detection on the WackoPicko application
  14. 14. Roadmap 1. Introduction 2. Methodology 3. Results 4. Conclusion 14
  15. 15. Vulnerability Detection Accuracy – FN 1/2 15 FNs in WackoPicko Reason(s) 1. Weak password - admin interface with credentials admin/admin • Scanners did not attempt to guess username/password • Scanners did attempt to guess username/password but failed 2. Session id - vulnerability in the admin interface • Scanners did not guess the admin credentials and therefore never reached this vulnerability 3. Parameter manipulation - userid of sample user functionality • Most scanners did not attempt to manipulate the userid field • Arachni manipulated the userid field but failed to enter a valid number • Skipfish successfully manipulated the userid field but did not report it as a vulnerability
  16. 16. Vulnerability Detection Accuracy – FN 2/2 16 FNs in WackoPicko Reason(s) 4. Stored SQL injection - required registering a user 5. Directory traversal - required photo upload 6. Multi-step stored XSS - required completing a multi-step process • Crawling challenges – discussed later • Lack of detection for these types of vulnerabilities 7. Forceful browsing: - link to a high quality version of a picture 8. Logic flaw – coupon management system • Application specific vulnerabilities • Require understanding business logic of the application Note: WAVSEP FNs not listed
  17. 17. Vulnerability Detection Accuracy – TP 1/2 17 Table 3.1: WackoPicko Default and Configured Scan Detection Results Name RXSS XSS Stored SQLi Reflected Command line injection File Inclusion File Exposure RXSS behind JS RXSS behind Flash Arachni INITIAL INITIAL INITIAL INITIAL INITIAL INITIAL Burp Pro INITIAL INITIAL INITIAL INITIAL INITIAL INITIAL INITIAL CONFIG Skipfish INITIAL INITIAL INITIAL INITIAL INITIAL Vega INITIAL INITIAL INITIAL INITIAL Wapiti INITIAL INITIAL INITIAL INITIAL ZAP INITIAL INITIAL INITIAL INITIAL INITIAL INITIAL CONFIG Default Configured • All scanners missed at least 50% of the vulnerabilities • Running the scanners in trained mode increased the overall detection
  18. 18. 18 Figure 3.1: WAVSEP Overall TP Detection Arachni Burp Skipfish Wapiti Vega ZAP Default 60.16% 27.87% 4.02% 25.41% 71.31% 60.74% Configured 60.16% 42.54% 62.62% 24.43% 71.31% 79.26% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% %ofWAVSEPTestsDetected Key Observations: • WAVSEP results were better than WackoPicko: • Vulnerability categories in the application • Integrating WAVSEP in the SDLC of the scanner • ZAP achieved highest score, followed by Vega and Skipfish • Vulnerability category detection varied with scanner • Arachni discovered 100% of SQLi, RFI, unvalidated redirect, but had a low detection rate for LFI vulnerabilities Vulnerability Detection Accuracy – TP 2/2
  19. 19. Crawling Coverage 1/2 19 Table 3.2: Account Creation Scanner # of Accounts Arachni 202 Burp Pro 113 Skipfish 364 Vega 117 Wapiti 0 ZAP 111 Features that scanners found difficult to crawl in WackoPicko: • Uploading a picture • All scanners were not able to upload a picture in Default mode • Burp and ZAP were able to in Configured mode • Authentication • All scanners except for Wapiti successfully created accounts • Multi-step processes • All scanners were not able to complete the process in Default mode • Burp and ZAP were able to in Configured mode
  20. 20. Crawling Coverage 2/2 20 Figure 3.2: WIVET Results Arachni Burp Skipfis h Wapiti Vega ZAP Default 94 50 50 50 16 42 Configured 94 50 50 50 16 78 0 10 20 30 40 50 60 70 80 90 100 %ofWIVETTestsPassed Features that scanners found difficult to crawl in WackoPicko: • Infinite websites • All scanners recognized the infinite loop except Arachni • Client-side code • Flash applications • Dynamic JavaScript • Ajax Requests
  21. 21. Scanning Speed 21 Figure 3.3: WackoPicko Default Scanning Speed Figure 3.4: WackoPicko Configured Scanning Speed Arachni Burp Skipfish Vega Wapiti ZAP INITIAL 0.3 0.1 0.05 0.08 0.04 0.07 CONFIG 0.32 0.12 0.1 0.1 0.05 0.18 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 ScanTime(Hours) Arachni Burp Skipfish Vega Wapiti ZAP INITIAL 0.3 0.17 0.05 0.12 1.47 0.2 CONFIG 0.32 0.35 0.1 0.22 1.62 1.31 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 ScanTime(Hours)
  22. 22. Reporting Features Features tested for: 1) List of all the vulnerabilities detected 2) Locations of all the detected vulnerabilities 3) Exploits performed to detect these vulnerabilities All six scanners generate reports that include the above three features 22
  23. 23. Usability Features Features tested for: 1) Efficiency 2) Product documentation 3) Community support 23 Table 3.3: Usability Features
  24. 24. Final Ranking 24 Figure 3.4: Final Ranking Name Score Burp Pro 26 ZAP 23 Arachni 15 Wapiti 10 Skipfish 10 Vega 8
  25. 25. Comparison to Previous Research 25 • Suto’s 2010 study showed there is no significant increase in vulnerability detection in PaS vs Trained modes • Our results show there is a significant increase for several of the scanners • Possible reasons for difference in conclusion – different benchmark applications and scanners were used • Doupé et al.’s 2010 study showed that scanners had difficulty crawling through common web technologies such as Flash applications and dynamic JavaScript measured on the WackoPicko and WIVET applications • Our results show similar conclusion • Doupé et al.’s study showed that scanners missed several categories of vulnerabilities such as weak authentication, stored vulnerabilities and logic specific vulnerabilities • Our results show similar conclusion • Doupé et al.’s study showed that commercial scanners are not significantly better than open-source scanners • Our results show similar conclusion
  26. 26. Roadmap 1. Introduction 2. Methodology 3. Results 4. Conclusion 26
  27. 27. Conclusion • Scanners are far from being used as PaS tools only • Several classes of vulnerabilities were not detected • Scanners had difficulty crawling through common web technologies such as dynamic JavaScript and Flash applications • Different scanners have different strengths/weaknesses • Open-source scanner performance is comparable to commercial scanner performance and in several cases better 27
  28. 28. References [1] Internet Live Stats, “Internet Users.” http://www.internetlivestats.com/ internet-users/, 2018. Accessed Aug. 4, 2018. [2] InternetLiveStats,“Total number of Websites.”http://www.internetlivestats. com/total-number-of-websites/, 2018. Accessed Aug. 4, 2018. [3] Braga, M., “100,000 Canadian victims: What we know about the Equifax breach - and what we don’t.” https://www.cbc.ca/news/technology/equifax-canada- breach-sin-cybersecurity-what-we-know-1.4297532, Sept. 2017. Accessed Aug. 4, 2018. [4] Trustwave, “2018 Trustwave Global Security Report.” https://www2.trustwave. com/GlobalSecurityReport.html, 2018. Accessed Aug. 4, 2018. [5] “The OWASP Foundation.” https://www.owasp.org/index.php/Main_Page, 2018. Accessed Aug. 3, 2018. [6] “Category:OWASP Top Ten Project.” https://www.owasp.org/index.php/ Category:OWASP_Top_Ten_Project#tab=Main, 2018. Accessed Aug. 3, 2018. [7] “OWASP Top 10 - 2017: The Ten Most Critical Web Application Security Risks.” https://www.owasp.org/images/7/72/OWASP_Top_10- 2017_%28en%29.pdf.pdf, 2018. Accessed Aug. 3, 2018. [8] J. Jive, “XSS Vectors Cheat Sheet.” https://gist.github.com/kurobeats/9a613c9ab68914312cbb415134795b45, 2017. Accessed Aug. 3, 2018. [9] B. Shura, R. Auger, and R. Gaucher, “Web Application Security Scanner Evaluation Criteria.” http://projects.webappsec.org/w/page/13246986/Web% 20Application%20Security%20Scanner%20Evaluation%20Criteria, 2014. Accessed Aug. 4, 2018. 28
  29. 29. References [10] L. Suto, “Analyzing the Effectiveness and Coverage of Web Application Security Scanners,” Case Study, Oct. 2007. [11] L. Suto, “Analyzing the Accuracy and Time Costs of Web Application Security Scanners,” Feb. 2010. [12] E. I. Tatli and B. Urgun, “Web Input Vector Extractor Teaser.” https://github. com/bedirhan/wivet, 2014. Accessed Aug. 1, 2018. [13] S. Chen, “The Web Application Vulnerability Scanner Evaluation Project.” https: //github.com/sectooladdict/wavsep, 2014. Accessed Aug. 2, 2018. [14] D. Gupta, J. Bau, J. Mitchell, and E. Bursztein, “State of the art: Automated black- box web application vulnerability testing,” in 2010 IEEE Symposium on Security and Privacy (SP), pp. 332–345, May 2010. [15] A. Doupé, M. Cova, and G. Vigna, “Why Johnny Can’t Pentest: An Analysis of Black-Box Web Vulnerability Scanners,” in Detection of Intrusions and Malware, and Vulnerability Assessment (C. Kreibich and M. Jahnke, eds.), (Berlin, Heidelberg), pp. 111–131, Springer Berlin Heidelberg, 2010. [16] K. McQuade, “Open Source Web Vulnerability Scanners: The Cost Effective Choice?,” in Conference for Information Systems Applied Research, 2014. [17] S. ElIdressi, N. Berbiche, F. Guerouate, and M. Sbihi, “Performance Evaluation of Web Application Security Scanners for Prevention and Protection against Vulnerabilities,” in International Journal of Applied Engineering Research, vol. 12, pp. 11068– 11076, 2017. [18] S. Chen, “The Prices vs. Features of Web Application Vulnerability Scanners.” http://sectoolmarket.com/price-and-feature-comparison-of-web- application-scanners-opensource-list.html, 2016. Accessed Aug. 1, 2018. 29
  30. 30. References [19] “Home - Arachni - Web Application Security Scanner Framework.” http://www. arachni-scanner.com/, 2017. Accessed Aug. 4, 2018. [20] L. Kuppan, “IronWASP - Iron Web application Advanced Security testing Platform.” https://ironwasp.org/index.html, 2014. Accessed Aug. 4, 2018. [21] “OWASP Zed Attack Proxy Project.” https://www.owasp.org/index.php/ OWASP_Zed_Attack_Proxy_Project, 2018. Accessed Aug. 4, 2018. [22] M. Zalewski, “skipfish(1) - Linux man page.” https://linux.die.net/man/1/ skipfish. Accessed Aug. 4, 2018. [23] N. Surribas, “Wapiti : a Free and Open-source Web-application Vulnerability Scanner.” http://wapiti.sourceforge.net/, 2018. Accessed Aug. 4, 2018. [24] “Vega Vulnerability Scanner - Subgraph OS.” https://subgraph.com/vega/, 2014. Accessed Aug. 4, 2018. [25] “Burp Suite Scanner | PortSwigger.” https://portswigger.net/burp, 2018. Ac- cessed Aug. 4, 2018. [26] “AppScan - Application Security | IBM.” https://www.ibm.com/security/ application-security/appscan, 2018. Accessed Aug. 3, 2018. [27] R. Siles and S. Bennetts, “OWASP Vulnerable Web Applications Direc- tory Project.” https://www.owasp.org/index.php/OWASP_Vulnerable_Web_ Applications_Directory_Project, 2018. Accessed Aug. 1, 2018. [28] A. Doupe, “WackoPicko Vulnerable Website.” https://github.com/adamdoupe/ WackoPicko, 2018. Accessed Aug. 2, 2018. 30
  31. 31. References [29] C. Willis, “OWASP Broken Web Applications Project.” https://www.owasp.org/index.php/OWASP_Broken_Web_Applications_Project, 2016. Accessed Aug. 3, 2018. [30] A. Riancho, “WIVET - Web Input Vector Extractor Teaser.” https://hub.docker. com/r/andresriancho/wivet/, 2015. Accessed Aug. 2, 2018. [31] “The Web Application Vulnerability Scanner Evaluation Project.” https://hub. docker.com/r/owaspvwad/wavsep/, 2016. Accessed Aug. 2, 2018. [32] R. Khalil, “Thesis-Test-Results.” https://github.com/rkhal101/Thesis-Test- Results, 2018. [33] “Crawler Coverage and Vulnerability Detection.” http://www.arachni-scanner. com/features/framework/crawl-coverage-vulnerability- detection/. Accessed Aug. 4, 2018. [34] R. Khalil, “Arachni does not maintain session across scan #986.” https://github. com/Arachni/arachni/issues/986, 2018. Accessed Aug. 4, 2018. [35] R.Khalil,“Sitemap does not contain all crawled links #987.”https://github.com/Arachni/arachni/issues/987, 2018. Accessed Aug. 4, 2018. [36] R. Khalil, “Run Vega on WIVET #157.” https://github.com/subgraph/Vega/ issues/157, 2018. Accessed Aug. 4, 2018. [37] R.Khalil,“"Alerts for this node” does not display high alerts of risk type High #4899.” https://github.com/zaproxy/zaproxy/issues/4899, 2018. Accessed Aug. 4, 2018. 31
  32. 32. Thank you! 32

×