This thesis compares the performance of six open-source web vulnerability scanners: Arachni, Burp Pro, Skipfish, Vega, Wapiti, and ZAP. The researcher evaluated the scanners using three benchmark applications and analyzed their crawling coverage, vulnerability detection accuracy, speed, and usability. The results showed that no scanner detected all vulnerabilities and they struggled with features like authentication, file uploads, and multi-step processes. ZAP and Burp Pro achieved the highest scores overall, while all scanners had room for improvement in fully automating the assessment of complex web applications.

1. Why Johnny Still Can’t Pentest:
A Comparative Analysis of Open-source Black-box Web
Vulnerability Scanners
Rana Khalil
Master Thesis Defense, 20/11/2018
School of EECS
University of Ottawa
Committee:
Carlisle Adams (Supervisor)
Guy-Vincent Jourdan
Anil Somayaji (Carleton University)
1

2. Roadmap
1. Introduction
2. Methodology
3. Results
4. Conclusion
2

3. Roadmap
1. Introduction
2. Methodology
3. Results
4. Conclusion
3

4. Introduction
• We use websites for everything: e-commerce, online banking, social networking,
social media, etc.
• Web security has become a major concern
4

5. How to Secure a Web Application?
• A combination of techniques are used to secure web applications:
• Static code analysis
• Web application firewalls
• Secure coding practices
• Web application vulnerability scanners
• Etc.
• Focus of this research: Performing a comprehensive comparative analysis of the
performance of six chosen scanners
5

6. Previous Work
6
• Suto’s case studies [10][11]
• 2007 paper evaluated scanners in PaS mode
• 2010 paper evaluated scanners in PaS and Trained modes
• Benchmark applications:
• Web Input Vector Extractor Teaser (WIVET) created in 2009 by Tatli et al. [12]
• Web Application Vulnerability Scanner Evaluation Project (WAVSEP) created in 2010 by Chen [13]
• Doupé et al.’s work on evaluating WAVS in both PaS and Trained modes on the WackoPicko
application [15]
• Several other studies include [14], [16] and [17]

7. Roadmap
1. Introduction
2. Methodology
3. Results
4. Conclusion
7

8. Methodology
8
Figure 2.1: Methodology Process
Tool
Selection
Benchmark
Selection
Environment
Setup
Feature and
Metric
Selection
Result
Analysis

9. Tool Selection
9
• Chen’s evaluation [18]
• Consultation with professional ethical hackers
Table 2.1: Characteristics of the Scanners Evaluated

10. Benchmark Selection
10
• Benchmark applications
• WIVET - contains 56 test cases that utilize both Web 1.0 and Web 2.0 technologies
• WAVSEP - consists of a total of 1220 true positive (TP) test cases and 40 false
positive (FP) test cases covering a range of vulnerability categories
• WackoPicko - intentionally vulnerable realistic web application
• Contains 16 vulnerabilities covering several of the OWASP Top 10
• Contains crawling challenges: HTML parsing, multi-step process, infinite web site,
authentication, client-side code, etc.

11. Environment Setup
11
• Each scanner was run in two modes:
• Default - default configuration setting
• Configured
1. Maximize crawling coverage – changing
configuration
2. Maximize crawling coverage – use of proxy
3. Maximize attack vector
• WackoPicko test scans were further divided into two
subcategories:
• INITIAL – without authentication / publicly accessible
• CONFIG - valid username/password combination
• In total, each scanner was run eight times.
Note: Tests performed in a VM that was restored to its initial
state before every test run.
Table 2.2: Steps Included in Configured
Scan

12. Feature Selection
12
• Crawling coverage:
• % of passed test cases on the WIVET application
• Crawling challenges in the WackoPicko application
• Vulnerability detection accuracy:
• TP, FN and FP on the WAVSEP and WackoPicko
applications
• Speed:
• Scan time on the WAVSEP and WackoPicko appliations
• Reporting:
• Vulnerability detected
• Vulnerability location
• Exploit performed
• Usability:
• Efficiency
• Product documentation
• Community support
Crawling
Coverage
Detection
Accuracy
Speed
WIVET
WackoPicko
WAVSEP
Features Applications
Figure 2.2: Feature Measurement

13. Metric Selection
13Table 2.3 Vulnerability Scores
• Final ranking was calculated
based on the crawling coverage
and vulnerability detection on
the WackoPicko application

14. Roadmap
1. Introduction
2. Methodology
3. Results
4. Conclusion
14

15. Vulnerability Detection Accuracy – FN 1/2
15
FNs in WackoPicko Reason(s)
1. Weak password - admin interface
with credentials admin/admin
• Scanners did not attempt to guess username/password
• Scanners did attempt to guess username/password but failed
2. Session id - vulnerability in the
admin interface
• Scanners did not guess the admin credentials and therefore
never reached this vulnerability
3. Parameter manipulation - userid
of sample user functionality
• Most scanners did not attempt to manipulate the userid field
• Arachni manipulated the userid field but failed to enter a
valid number
• Skipfish successfully manipulated the userid field but did not
report it as a vulnerability

16. Vulnerability Detection Accuracy – FN 2/2
16
FNs in WackoPicko Reason(s)
4. Stored SQL injection - required
registering a user
5. Directory traversal - required
photo upload
6. Multi-step stored XSS - required
completing a multi-step process
• Crawling challenges – discussed later
• Lack of detection for these types of vulnerabilities
7. Forceful browsing: - link to a
high quality version of a picture
8. Logic flaw – coupon
management system
• Application specific vulnerabilities
• Require understanding business logic of the application
Note: WAVSEP FNs not listed

17. Vulnerability Detection Accuracy – TP 1/2
17
Table 3.1: WackoPicko Default and Configured Scan Detection Results
Name RXSS XSS Stored SQLi
Reflected
Command line
injection
File
Inclusion
File
Exposure
RXSS
behind JS
RXSS behind
Flash
Arachni INITIAL INITIAL INITIAL INITIAL INITIAL INITIAL
Burp Pro INITIAL INITIAL INITIAL INITIAL INITIAL INITIAL INITIAL CONFIG
Skipfish INITIAL INITIAL INITIAL INITIAL INITIAL
Vega INITIAL INITIAL INITIAL INITIAL
Wapiti INITIAL INITIAL INITIAL INITIAL
ZAP INITIAL INITIAL INITIAL INITIAL INITIAL INITIAL CONFIG
Default
Configured
• All scanners missed at least 50% of the vulnerabilities
• Running the scanners in trained mode increased the overall detection

18. 18
Figure 3.1: WAVSEP Overall TP Detection
Arachni Burp Skipfish Wapiti Vega ZAP
Default 60.16% 27.87% 4.02% 25.41% 71.31% 60.74%
Configured 60.16% 42.54% 62.62% 24.43% 71.31% 79.26%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
%ofWAVSEPTestsDetected
Key Observations:
• WAVSEP results were better than WackoPicko:
• Vulnerability categories in the application
• Integrating WAVSEP in the SDLC of the
scanner
• ZAP achieved highest score, followed by Vega and
Skipfish
• Vulnerability category detection varied with scanner
• Arachni discovered 100% of SQLi, RFI,
unvalidated redirect, but had a low detection
rate for LFI vulnerabilities
Vulnerability Detection Accuracy – TP 2/2

19. Crawling Coverage 1/2
19
Table 3.2: Account Creation
Scanner # of Accounts
Arachni 202
Burp Pro 113
Skipfish 364
Vega 117
Wapiti 0
ZAP 111
Features that scanners found difficult to crawl in WackoPicko:
• Uploading a picture
• All scanners were not able to upload a picture in Default mode
• Burp and ZAP were able to in Configured mode
• Authentication
• All scanners except for Wapiti successfully created accounts
• Multi-step processes
• All scanners were not able to complete the process in Default
mode
• Burp and ZAP were able to in Configured mode

20. Crawling Coverage 2/2
20
Figure 3.2: WIVET Results
Arachni Burp
Skipfis
h
Wapiti Vega ZAP
Default 94 50 50 50 16 42
Configured 94 50 50 50 16 78
0
10
20
30
40
50
60
70
80
90
100
%ofWIVETTestsPassed
Features that scanners found difficult to
crawl in WackoPicko:
• Infinite websites
• All scanners recognized the infinite
loop except Arachni
• Client-side code
• Flash applications
• Dynamic JavaScript
• Ajax Requests

21. Scanning Speed
21
Figure 3.3: WackoPicko Default Scanning Speed Figure 3.4: WackoPicko Configured Scanning Speed
Arachni Burp Skipfish Vega Wapiti ZAP
INITIAL 0.3 0.1 0.05 0.08 0.04 0.07
CONFIG 0.32 0.12 0.1 0.1 0.05 0.18
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
ScanTime(Hours)
Arachni Burp Skipfish Vega Wapiti ZAP
INITIAL 0.3 0.17 0.05 0.12 1.47 0.2
CONFIG 0.32 0.35 0.1 0.22 1.62 1.31
0
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
ScanTime(Hours)

22. Reporting Features
Features tested for:
1) List of all the vulnerabilities detected
2) Locations of all the detected vulnerabilities
3) Exploits performed to detect these vulnerabilities
All six scanners generate reports that include the above three features
22

23. Usability Features
Features tested for:
1) Efficiency
2) Product documentation
3) Community support
23
Table 3.3: Usability Features

24. Final Ranking
24
Figure 3.4: Final Ranking
Name Score
Burp Pro 26
ZAP 23
Arachni 15
Wapiti 10
Skipfish 10
Vega 8

25. Comparison to Previous Research
25
• Suto’s 2010 study showed there is no significant increase in vulnerability detection in PaS vs Trained modes
• Our results show there is a significant increase for several of the scanners
• Possible reasons for difference in conclusion – different benchmark applications and scanners were used
• Doupé et al.’s 2010 study showed that scanners had difficulty crawling through common web technologies
such as Flash applications and dynamic JavaScript measured on the WackoPicko and WIVET applications
• Our results show similar conclusion
• Doupé et al.’s study showed that scanners missed several categories of vulnerabilities such as weak
authentication, stored vulnerabilities and logic specific vulnerabilities
• Our results show similar conclusion
• Doupé et al.’s study showed that commercial scanners are not significantly better than open-source scanners
• Our results show similar conclusion

26. Roadmap
1. Introduction
2. Methodology
3. Results
4. Conclusion
26

27. Conclusion
• Scanners are far from being used as PaS tools only
• Several classes of vulnerabilities were not detected
• Scanners had difficulty crawling through common web technologies such as dynamic
JavaScript and Flash applications
• Different scanners have different strengths/weaknesses
• Open-source scanner performance is comparable to commercial scanner performance and in
several cases better
27

28. References
[1] Internet Live Stats, “Internet Users.” http://www.internetlivestats.com/ internet-users/, 2018. Accessed Aug. 4, 2018.
[2] InternetLiveStats,“Total number of Websites.”http://www.internetlivestats. com/total-number-of-websites/, 2018. Accessed Aug. 4, 2018.
[3] Braga, M., “100,000 Canadian victims: What we know about the Equifax breach - and what we don’t.”
https://www.cbc.ca/news/technology/equifax-canada- breach-sin-cybersecurity-what-we-know-1.4297532, Sept. 2017. Accessed Aug. 4, 2018.
[4] Trustwave, “2018 Trustwave Global Security Report.” https://www2.trustwave. com/GlobalSecurityReport.html, 2018. Accessed Aug. 4,
2018.
[5] “The OWASP Foundation.” https://www.owasp.org/index.php/Main_Page, 2018. Accessed Aug. 3, 2018.
[6] “Category:OWASP Top Ten Project.” https://www.owasp.org/index.php/ Category:OWASP_Top_Ten_Project#tab=Main, 2018. Accessed
Aug. 3, 2018.
[7] “OWASP Top 10 - 2017: The Ten Most Critical Web Application Security Risks.” https://www.owasp.org/images/7/72/OWASP_Top_10-
2017_%28en%29.pdf.pdf, 2018. Accessed Aug. 3, 2018.
[8] J. Jive, “XSS Vectors Cheat Sheet.” https://gist.github.com/kurobeats/9a613c9ab68914312cbb415134795b45, 2017. Accessed Aug. 3, 2018.
[9] B. Shura, R. Auger, and R. Gaucher, “Web Application Security Scanner Evaluation Criteria.”
http://projects.webappsec.org/w/page/13246986/Web% 20Application%20Security%20Scanner%20Evaluation%20Criteria, 2014. Accessed
Aug. 4, 2018.
28

29. References
[10] L. Suto, “Analyzing the Effectiveness and Coverage of Web Application Security Scanners,” Case Study, Oct. 2007.
[11] L. Suto, “Analyzing the Accuracy and Time Costs of Web Application Security Scanners,” Feb. 2010.
[12] E. I. Tatli and B. Urgun, “Web Input Vector Extractor Teaser.” https://github. com/bedirhan/wivet, 2014. Accessed Aug. 1, 2018.
[13] S. Chen, “The Web Application Vulnerability Scanner Evaluation Project.” https: //github.com/sectooladdict/wavsep, 2014. Accessed Aug.
2, 2018.
[14] D. Gupta, J. Bau, J. Mitchell, and E. Bursztein, “State of the art: Automated black- box web application vulnerability testing,” in 2010
IEEE Symposium on Security and Privacy (SP), pp. 332–345, May 2010.
[15] A. Doupé, M. Cova, and G. Vigna, “Why Johnny Can’t Pentest: An Analysis of Black-Box Web Vulnerability Scanners,” in Detection of
Intrusions and Malware, and Vulnerability Assessment (C. Kreibich and M. Jahnke, eds.), (Berlin, Heidelberg), pp. 111–131, Springer Berlin
Heidelberg, 2010.
[16] K. McQuade, “Open Source Web Vulnerability Scanners: The Cost Effective Choice?,” in Conference for Information Systems Applied
Research, 2014.
[17] S. ElIdressi, N. Berbiche, F. Guerouate, and M. Sbihi, “Performance Evaluation of Web Application Security Scanners for Prevention and
Protection against Vulnerabilities,” in International Journal of Applied Engineering Research, vol. 12, pp. 11068– 11076, 2017.
[18] S. Chen, “The Prices vs. Features of Web Application Vulnerability Scanners.”
http://sectoolmarket.com/price-and-feature-comparison-of-web- application-scanners-opensource-list.html, 2016. Accessed Aug. 1, 2018.
29

30. References
[19] “Home - Arachni - Web Application Security Scanner Framework.” http://www. arachni-scanner.com/, 2017. Accessed Aug. 4, 2018.
[20] L. Kuppan, “IronWASP - Iron Web application Advanced Security testing Platform.” https://ironwasp.org/index.html, 2014. Accessed Aug.
4, 2018.
[21] “OWASP Zed Attack Proxy Project.” https://www.owasp.org/index.php/ OWASP_Zed_Attack_Proxy_Project, 2018. Accessed Aug. 4,
2018.
[22] M. Zalewski, “skipfish(1) - Linux man page.” https://linux.die.net/man/1/ skipfish. Accessed Aug. 4, 2018.
[23] N. Surribas, “Wapiti : a Free and Open-source Web-application Vulnerability Scanner.” http://wapiti.sourceforge.net/, 2018. Accessed Aug.
4, 2018.
[24] “Vega Vulnerability Scanner - Subgraph OS.” https://subgraph.com/vega/, 2014. Accessed Aug. 4, 2018.
[25] “Burp Suite Scanner | PortSwigger.” https://portswigger.net/burp, 2018. Ac- cessed Aug. 4, 2018.
[26] “AppScan - Application Security | IBM.” https://www.ibm.com/security/ application-security/appscan, 2018. Accessed Aug. 3, 2018.
[27] R. Siles and S. Bennetts, “OWASP Vulnerable Web Applications Direc- tory Project.”
https://www.owasp.org/index.php/OWASP_Vulnerable_Web_ Applications_Directory_Project, 2018. Accessed Aug. 1, 2018.
[28] A. Doupe, “WackoPicko Vulnerable Website.” https://github.com/adamdoupe/ WackoPicko, 2018. Accessed Aug. 2, 2018.
30

31. References
[29] C. Willis, “OWASP Broken Web Applications Project.” https://www.owasp.org/index.php/OWASP_Broken_Web_Applications_Project,
2016. Accessed Aug. 3, 2018.
[30] A. Riancho, “WIVET - Web Input Vector Extractor Teaser.” https://hub.docker. com/r/andresriancho/wivet/, 2015. Accessed Aug. 2, 2018.
[31] “The Web Application Vulnerability Scanner Evaluation Project.” https://hub. docker.com/r/owaspvwad/wavsep/, 2016. Accessed Aug. 2,
2018.
[32] R. Khalil, “Thesis-Test-Results.” https://github.com/rkhal101/Thesis-Test- Results, 2018.
[33] “Crawler Coverage and Vulnerability Detection.” http://www.arachni-scanner. com/features/framework/crawl-coverage-vulnerability-
detection/. Accessed Aug. 4, 2018.
[34] R. Khalil, “Arachni does not maintain session across scan #986.” https://github. com/Arachni/arachni/issues/986, 2018. Accessed Aug. 4,
2018.
[35] R.Khalil,“Sitemap does not contain all crawled links #987.”https://github.com/Arachni/arachni/issues/987, 2018. Accessed Aug. 4, 2018.
[36] R. Khalil, “Run Vega on WIVET #157.” https://github.com/subgraph/Vega/ issues/157, 2018. Accessed Aug. 4, 2018.
[37] R.Khalil,“"Alerts for this node” does not display high alerts of risk type High #4899.” https://github.com/zaproxy/zaproxy/issues/4899,
2018. Accessed Aug. 4, 2018.
31

32. Thank you!
32