Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Hackfest 2019 Talk

425 views

Published on

Hackfest 2019 talk

Published in: Engineering
  • Be the first to comment

Hackfest 2019 Talk

  1. 1. Why Johnny Still Can’t Pentest: A Comparative Analysis of Open-source Web Application Vulnerability Scanners @rana__khali By: Rana Khalil
  2. 2. whoami @rana__khali • Security assessment analyst • Master in Computer Science under the supervision of Dr. Carlisle Adams • Speaker at BSides, ISSA and OWASP Ottawa, and WiCyS
  3. 3. Let me weave you a tale…. @rana__khali
  4. 4. This is Johnny @rana__khali
  5. 5. @rana__khali Company X You need to develop secure software!
  6. 6. @rana__khali
  7. 7. @rana__khali “Commercial scanner X ensures web application security by securing your website and web applications against hacker attacks.” “Commercial scanner Y dead accurate web vulnerability scanner to identify vulnerabilities in your websites…” “Open-source scanner Z provides first-class coverage, vulnerability detection and accuracy for modern web application technologies.” “Open-source scanner W can help you find and validate SQL Injection, Cross-Site Scripting (XSS), inadvertently disclosed sensitive information, and other vulnerabilities.”
  8. 8. @rana__khali Vulnerabilities? Develop code Run web application vulnerability scanner on application Yes Yaaay! No
  9. 9. @rana__khali I’m freaking amazing! I can do it all! And that worked out great for Johnny
  10. 10. @rana__khali And that worked out great for Johnny….until it didn’t Vulnerability #1 Vulnerability #2 Vulnerability #3 ………… Vulnerability ∞ Pentest Findings
  11. 11. @rana__khali • How do these tools work? • Do they require special configuration? • How much coverage do they achieve? • What vulnerabilities can they find? • What vulnerabilities can they NOT find?
  12. 12. Let the research begin…. @rana__khali
  13. 13. How do these scanners work? @rana__khali Web Application Vulnerability Scanners have three modules: Crawler Attacker Analysis *XSS found* *SQLi found* *LFI found* *RFI found*
  14. 14. How are these scanners used? @rana__khali Option #1: Point and Shoot (PaS) Option #2: Trained/Configured domain-name • Scanner is given only root URL • Default configuration unchanged • Minimal human interference • Manually visit every page of the application in proxy mode. • Change configuration & train scanner
  15. 15. Tool Selection @rana__khali • Chen’s evaluation • Consultation with penetration testers Name Version License Price Arachni 1.5.1-0.5.12 Arachni Public Source v1.0 N/A Burp Pro 1.7.35 Commercial $349/year Skipfish 2.10b Apache v2.0 N/A Vega 1.0 MIT N/A Wapiti 3.0.1 GNU GPL v2 N/A ZAP 2.7.0 Apache v2.0 N/A
  16. 16. Environment Setup @rana__khali Tools Applications * VM restored to initial state before every test run
  17. 17. If it sounds too good to be true, it probably is.…. @rana__khali
  18. 18. Vulnerability Detection @rana__khali Vulnerabilities in WackoPicko that were not detected by any scanners: 1. Weak authentication credentials • admin/admin
  19. 19. Vulnerability Detection @rana__khali Vulnerabilities in WackoPicko that were not detected by any scanners: 2. Parameter Manipulation Sample user: WackoPicko/users/sample.php?userid=1 Real user: WackoPicko/users/sample.php?userid=2
  20. 20. Vulnerability Detection @rana__khali Vulnerabilities in WackoPicko that were not detected by any scanners: 3. Forceful Browsing • Access to a link that contains a high quality version of a picture without authentication
  21. 21. Vulnerability Detection @rana__khali Vulnerabilities in WackoPicko that were not detected by any scanners: 4. Logic Flaw • Coupon management functionality
  22. 22. @rana__khali Note: This slide is shamelessly stolen from David Caissy’s 2017 Appsec Talk. Can scanners catch everything?
  23. 23. Vulnerability Detection @rana__khali On average scanners found only 40% of the vulnerabilities. 0 10 20 30 40 50 60 70 80 90 100 Arachni Burp Skipfish Wapiti Vega ZAP %ofDetectedVulnerabilities
  24. 24. Crawling Challenges @rana__khali Features that scanners found difficult to crawl in WackoPicko: 1. Uploading a file • All scanners were not able to upload a picture in PaS mode • Burp and ZAP were able to in Trained mode
  25. 25. Crawling Challenges @rana__khali Features that scanners found difficult to crawl in WackoPicko: 2. Authentication • All scanners except for Wapiti successfully created accounts • None of the scanners used the created accounts to authenticate Scanner # of Accounts Arachni 202 Burp 113 Skipfish 364 Vega 117 Wapiti 0 ZAP 111
  26. 26. Crawling Challenges @rana__khali Features that scanners found difficult to crawl in WackoPicko: 3. Multi-step processes • All scanners were not able to complete the process in PaS mode • Burp and ZAP were able to in Trained mode
  27. 27. Crawling Challenges @rana__khali Features that scanners found difficult to crawl in WackoPicko: 4. Infinite websites • All scanners recognized the infinite loop except Arachni ….. /calendar.php?date=1541454543 /calendar.php?date=1541540943 /calendar.php?date=1541627343
  28. 28. Crawling Challenges @rana__khali Features that scanners found difficult to crawl in WackoPicko: 5. State awareness • In PaS mode none of the scanners discovered any of the vulnerabilities that require authentication • Vulnerabilities that require authentication were only discovered in Trained mode
  29. 29. Crawling Challenges @rana__khali Features that scanners found difficult to crawl: 6. Client-side Code • Standard anchor links • Links created dynamically using JavaScript • Multi-page forms • Links in comments • Links embedded in Flash objects • Links within AJAX requests 0 10 20 30 40 50 60 70 80 90 100 Arachni Burp Skipfish Wapiti Vega ZAP %ofWIVETTestsPassed
  30. 30. What now? Should Johnny even bother using an automated scanner? @rana__khali
  31. 31. Johnny definitely should! @rana__khali • Scanners DO NOT replace a skilled pentester, but can aid the pentester • Vulnerability scan is NOT EQUIVALENT to a vulnerability assessment • Using a vulnerability scanner requires skill • A fool with a tool is still a fool • Configure your scanner! Never run your scanner in PaS • Specify the target • Set Login / logout conditions • Set the scanner in proxy mode and visit every page of the application • Configure scenarios (business flows) and cleanup b/w scenarios • Monitor and review the requests of your scan • After all that work, you’re only protected against script kiddies
  32. 32. That’s why Johnny still can’t pentest. @rana__khali
  33. 33. Get in Touch! @rana__khali https://medium.com/@ranakhalil101 https://www.linkedin.com/in/ranakhalil1/ https://twitter.com/rana__khalil https://ruor.uottawa.ca/handle/10393/38595

×