skipfish
10 November 2011
Ernst & Young, Sydney Australia


Previously presented at:
OWASP NL
30 June 2010
Overview
Not an OWASP Project
By Michal Zalewski
Major contributions to webappsec with Google
  RatProxy;
  Browser Security Handbook;
  “Rise of the Robots” i.e. The inspiration for the OWASP
  “Google Hacking” Project
Overview


Fast webappsec scanner which“spiders” using word lists
  Could be used to test www DOS
Overview

Fast webappsec scanner which“spiders” using word lists
  Similar to Burp Scanner, etc

  Does not satisfy WASC Security Scanner Evaluation Criteria
    I don’t think lcamtuf intends too either :)
Overview

3.Fast webappsec scanner which“spiders” using word lists
     Similar to DirBuster maybe Nikto, etc

     “2007 entries resulting in about 42K HTTP Requests”
       Based on the recommended *minimal* Word List
         i.e. bigger wordlist = bigger number of HTTP Requests
Build/Install
From Source Code
   Doesn’t build on OpenBSD (issue noted)
   Dependency on libidn

     Builds on backtrack
Release Cycle


lcamtuf rapidly updates via minor releases
   i.e. RatProxy followed same development

                            Insert http://
                            vis.cs.ucdavis.edu/
                            ~ogawa/codeswarm/
Build/Install


http://www.shortinfosec.net/2010/03/compiling-latest-skipfish-for-windows.html


   Not mantained with each release i.e. v1.29b
   No mention of support on code.google.com
      i.e. Use at your own risk
Spidering



./skipfish -W /dev/null -LV ...
Word List


keywords and extensions

type hits total_age last_age keyword
Supplied Word Lists


1. Empty
2. extensions-only.wl
   Must be used in conjunction with ./skipfish -Y
Word List
The following all contain 1.7K keywords:
Word List
minimal.wl
~50,000 HTTP Requests
medium.wl
~50,000 HTTP Requests x 2
complete.wl
~50,000 HTTP Requests x 3
Word List

                                         Insert sh script


1.Select wordlist from ./dictionaries/
2.Copy as ../skipfish.wl

   *copy* .wl as skipfish may append skipfish.wl
   may depends on cmd line i.e. ./skipfish -V ...
Wordlist

Custom Wordlist
  ./skipfish -W custom_wl ...
Suppress Automatic Learning
  ./skipfish -L ...
Suppress Amending Wordlist
  ./skipfish -V ...
Lightweight Brute Force



~1,700 HTTP Requests
cp ./dictionaries/complete.wl dictionary.wl

./skipfish -W dictionary.wl -Y ...
Word List

Limit Keyword Guess Size Jar
  ./skipfish -G ...
Drop Old Dictionary Entries
  ./skipfish -R ...
Don’t fuzz $keyword.$extension
  ./skipfish -Y ...
Basic Usage

Output Directory
  ./skipfish -o output_dir URL ...
Suppress Real-Time Statistics
  ./skipfish -u ...
Usage - Scheduling

Percentage of links and directories
  ./skipfish -p percentage ...
Repeat previous scan
  ./skipfish -q seed ...
Usage - Authentication

HTTP Auth
   ./skipfish -A user:pass ...
Cookie
   ./skipfish -C name=value ...
Autocomplete Forms
   ./skipfish -T form_field=value ...
Usage - Cookie
Cookie
    ./skipfish -C name=value ...
Ignore new set-cookies from specific locations
i.e. prevent URIs from being fetched, such as logout.aspx
    ./skipfish -X ...
Ignore new set-cookies from all locations
    ./skipfish -N ...
Usage - HTTP Headers

User Agent
  ./skipfish -b ffox or ie or phone...
Custom HTTP Header
  ./skipfish -H Header ...
Usage - Scoping
Spider from
  ./skipfish -I URI ...
Parameters not to Fuzz, such as SessionID
  ./skipfish -K SessionID_parameter ...
Include Domain
  ./skipfish -D FQDN...
Exclude URI
  ./skipfish -S URI or -X URI ...
Usage - Scoping

Limit crawl depth to number of sub directories/folders
  ./skipfish -d number ...
Limit the number of child directories per parent
  ./skipfish -c number ...
Limit Total HTTP Requests
  ./skipfish -r number ...
Usage - Scoping

No parsing of Form
  ./skipfish -O ...
No parsing of HTML
  ./skipfish -P ...
Usage - Low Impact
Mixed TLS/SSLv3 and HTTP (i.e. Cleartext)
  ./skipfish -M ...
  Low severity i.e. images are out of scope
Caching Directives of HTTP 1.0 vs 1.1
  ./skipfish -E ...
Information Leakage i.e. E-mail Addresses and URL
  ./skipfish -U ...
Usage - Reporting

Suppress reporting of duplicates hosts
  ./skipfish -Q ...
Suppress warning of “trusted” domains
  ./skipfish -B ...
Purge binary content without affecting report quality
  ./skipfish -e ...
Delta Reporting


sfscandiff
  non-destructively annotated by adding red
  background to all new or changed nodes; and blue
  background to all new or changed issues found
Issues


Won’t detect common low risks, such as:
  cookie without HTTPonly or secure flags
  autocomplete enabled Forms
Issues (Credit ‘FX’)

High Number of False Positives
  ASCII txt interpreted as JSON reply with XSSI
  Deviation between charset and MIME type
    Note ./skipfish -J ...

  No wordlist generation based on robots.txt
Issues (Credit ‘FX’)

                                                Resolved
Does not write output while the tool is executing
Total Size of HTTP Request vs File System Image
Issues


Does not support intercepting web proxy
  No supporting log entires that skipfish was used
  Use wireshark instead i.e. TCP/80 and TCP/443
Benefits (Credit ‘FX’)


Will display the source of CGI script
Can detect IPS
  HTTP 500 for ASP.NET HttpRequestValidationException
Performance Tuning
Number of connections to all hosts
  ./skipfish -g ...
  Recommended to be < 50
Per IP
  ./skipfish -m number ...
     2 - 4 localhost
     4 - 8 local network
     10 - 20 external
     30 - 50 hosts which lag or slow connections
Performance Tuning
I/O Timeout
  ./skipfish -w number ...
Total Request Timeout
  ./skipfish -t number ...
Number of HTTP Errors before Terminating
  ./skipfish -f number ...
Truncate HTTP Response
  ./skipfish -s number ...
Q&A
Thanks Wouter - Ernst & Young


Latest slides available from
  http://slideshare.net/cmlh
  http://github.com/cmlh/skipfish


http://cmlh.id.au/contact

skipfish

  • 1.
    skipfish 10 November 2011 Ernst& Young, Sydney Australia Previously presented at: OWASP NL 30 June 2010
  • 2.
    Overview Not an OWASPProject By Michal Zalewski Major contributions to webappsec with Google RatProxy; Browser Security Handbook; “Rise of the Robots” i.e. The inspiration for the OWASP “Google Hacking” Project
  • 3.
    Overview Fast webappsec scannerwhich“spiders” using word lists Could be used to test www DOS
  • 4.
    Overview Fast webappsec scannerwhich“spiders” using word lists Similar to Burp Scanner, etc Does not satisfy WASC Security Scanner Evaluation Criteria I don’t think lcamtuf intends too either :)
  • 5.
    Overview 3.Fast webappsec scannerwhich“spiders” using word lists Similar to DirBuster maybe Nikto, etc “2007 entries resulting in about 42K HTTP Requests” Based on the recommended *minimal* Word List i.e. bigger wordlist = bigger number of HTTP Requests
  • 6.
    Build/Install From Source Code Doesn’t build on OpenBSD (issue noted) Dependency on libidn Builds on backtrack
  • 7.
    Release Cycle lcamtuf rapidlyupdates via minor releases i.e. RatProxy followed same development Insert http:// vis.cs.ucdavis.edu/ ~ogawa/codeswarm/
  • 8.
    Build/Install http://www.shortinfosec.net/2010/03/compiling-latest-skipfish-for-windows.html Not mantained with each release i.e. v1.29b No mention of support on code.google.com i.e. Use at your own risk
  • 9.
  • 10.
    Word List keywords andextensions type hits total_age last_age keyword
  • 11.
    Supplied Word Lists 1.Empty 2. extensions-only.wl Must be used in conjunction with ./skipfish -Y
  • 12.
    Word List The followingall contain 1.7K keywords:
  • 13.
    Word List minimal.wl ~50,000 HTTPRequests medium.wl ~50,000 HTTP Requests x 2 complete.wl ~50,000 HTTP Requests x 3
  • 14.
    Word List Insert sh script 1.Select wordlist from ./dictionaries/ 2.Copy as ../skipfish.wl *copy* .wl as skipfish may append skipfish.wl may depends on cmd line i.e. ./skipfish -V ...
  • 15.
    Wordlist Custom Wordlist ./skipfish -W custom_wl ... Suppress Automatic Learning ./skipfish -L ... Suppress Amending Wordlist ./skipfish -V ...
  • 16.
    Lightweight Brute Force ~1,700HTTP Requests cp ./dictionaries/complete.wl dictionary.wl ./skipfish -W dictionary.wl -Y ...
  • 17.
    Word List Limit KeywordGuess Size Jar ./skipfish -G ... Drop Old Dictionary Entries ./skipfish -R ... Don’t fuzz $keyword.$extension ./skipfish -Y ...
  • 18.
    Basic Usage Output Directory ./skipfish -o output_dir URL ... Suppress Real-Time Statistics ./skipfish -u ...
  • 19.
    Usage - Scheduling Percentageof links and directories ./skipfish -p percentage ... Repeat previous scan ./skipfish -q seed ...
  • 20.
    Usage - Authentication HTTPAuth ./skipfish -A user:pass ... Cookie ./skipfish -C name=value ... Autocomplete Forms ./skipfish -T form_field=value ...
  • 21.
    Usage - Cookie Cookie ./skipfish -C name=value ... Ignore new set-cookies from specific locations i.e. prevent URIs from being fetched, such as logout.aspx ./skipfish -X ... Ignore new set-cookies from all locations ./skipfish -N ...
  • 22.
    Usage - HTTPHeaders User Agent ./skipfish -b ffox or ie or phone... Custom HTTP Header ./skipfish -H Header ...
  • 23.
    Usage - Scoping Spiderfrom ./skipfish -I URI ... Parameters not to Fuzz, such as SessionID ./skipfish -K SessionID_parameter ... Include Domain ./skipfish -D FQDN... Exclude URI ./skipfish -S URI or -X URI ...
  • 24.
    Usage - Scoping Limitcrawl depth to number of sub directories/folders ./skipfish -d number ... Limit the number of child directories per parent ./skipfish -c number ... Limit Total HTTP Requests ./skipfish -r number ...
  • 25.
    Usage - Scoping Noparsing of Form ./skipfish -O ... No parsing of HTML ./skipfish -P ...
  • 26.
    Usage - LowImpact Mixed TLS/SSLv3 and HTTP (i.e. Cleartext) ./skipfish -M ... Low severity i.e. images are out of scope Caching Directives of HTTP 1.0 vs 1.1 ./skipfish -E ... Information Leakage i.e. E-mail Addresses and URL ./skipfish -U ...
  • 27.
    Usage - Reporting Suppressreporting of duplicates hosts ./skipfish -Q ... Suppress warning of “trusted” domains ./skipfish -B ... Purge binary content without affecting report quality ./skipfish -e ...
  • 28.
    Delta Reporting sfscandiff non-destructively annotated by adding red background to all new or changed nodes; and blue background to all new or changed issues found
  • 29.
    Issues Won’t detect commonlow risks, such as: cookie without HTTPonly or secure flags autocomplete enabled Forms
  • 30.
    Issues (Credit ‘FX’) HighNumber of False Positives ASCII txt interpreted as JSON reply with XSSI Deviation between charset and MIME type Note ./skipfish -J ... No wordlist generation based on robots.txt
  • 31.
    Issues (Credit ‘FX’) Resolved Does not write output while the tool is executing Total Size of HTTP Request vs File System Image
  • 32.
    Issues Does not supportintercepting web proxy No supporting log entires that skipfish was used Use wireshark instead i.e. TCP/80 and TCP/443
  • 33.
    Benefits (Credit ‘FX’) Willdisplay the source of CGI script Can detect IPS HTTP 500 for ASP.NET HttpRequestValidationException
  • 34.
    Performance Tuning Number ofconnections to all hosts ./skipfish -g ... Recommended to be < 50 Per IP ./skipfish -m number ... 2 - 4 localhost 4 - 8 local network 10 - 20 external 30 - 50 hosts which lag or slow connections
  • 35.
    Performance Tuning I/O Timeout ./skipfish -w number ... Total Request Timeout ./skipfish -t number ... Number of HTTP Errors before Terminating ./skipfish -f number ... Truncate HTTP Response ./skipfish -s number ...
  • 36.
    Q&A Thanks Wouter -Ernst & Young Latest slides available from http://slideshare.net/cmlh http://github.com/cmlh/skipfish http://cmlh.id.au/contact

Editor's Notes

  • #2 \n
  • #3  http://lcamtuf.coredump.cx/\nhttp://lcamtuf.blogspot.com/\n\nhttp://twitter.com/lcamtuf\n\nEmployed by Google\nImage Attribution http://www.knackery.net/hackers.php and http://lcamtuf.coredump.cx\n
  • #4 webappsec scanner e.g. Burp Scanner, IBM Rational AppScan, etc\nspider e.g. DirBuster, Nikto, etc\n\n&amp;#x201C;2007 entries (recommended by lcamtuf) resulting in about 42K HTTP Requests&amp;#x201D; quoted from Felix &amp;#x201C;FX&amp;#x201D; Linder article\n\nWASC applicable standard is Web Application Security Scanner Evaluation Criteria - quoted from lcamtuf documentation.\n\n
  • #5 webappsec scanner e.g. Burp Scanner, IBM Rational AppScan, etc\nspider e.g. DirBuster, Nikto, etc\n\n&amp;#x201C;2007 entries (recommended by lcamtuf) resulting in about 42K HTTP Requests&amp;#x201D; quoted from Felix &amp;#x201C;FX&amp;#x201D; Linder article\n\nWASC applicable standard is Web Application Security Scanner Evaluation Criteria - quoted from lcamtuf documentation.\n\n
  • #6 webappsec scanner e.g. Burp Scanner, IBM Rational AppScan, etc\nspider e.g. DirBuster, Nikto, etc\n\n&amp;#x201C;2007 entries (recommended by lcamtuf) resulting in about 42K HTTP Requests&amp;#x201D; quoted from Felix &amp;#x201C;FX&amp;#x201D; Linder article\n\nWASC applicable standard is Web Application Security Scanner Evaluation Criteria - quoted from lcamtuf documentation.\n\n
  • #7 RatProxy had a similar release cycle\n\nhttps://gist.github.com/1321223\n
  • #8 \n
  • #9 http://www.shortinfosec.net/2010/03/compiling-latest-skipfish-for-windows.html\n
  • #10 http://lcamtuf.blogspot.com/2010/11/understanding-and-using-skipfish.html\n
  • #11  extensions (a s extensions (a subset of keywords) ubset of keywords)\n
  • #12  -Y is &amp;#x201C;don&amp;#x2019;t fuzz $keyword.$extension&amp;#x201D;\n\n
  • #13 \n
  • #14 \n
  • #15 \n\n\n
  • #16 \n
  • #17 http://lcamtuf.blogspot.com/2010/11/understanding-and-using-skipfish.html\n
  • #18 \n
  • #19 \n\n
  • #20 \n\n
  • #21 -C is cookie, can you curl to determine cookie\n
  • #22 -C is cookie, can you curl to determine cookie\n\nhttp://lcamtuf.blogspot.com/2010/11/understanding-and-using-skipfish.html\n
  • #23 -H Custom HTTP Header\n\n
  • #24 -I i.e. capital &amp;#x201C;i&amp;#x201D;\n-S or -X i.e. Exclude locations\n\n\n
  • #25 -c Limits the number of child directories per parent - not clear in Google Code documentation\nNeed to read this -F Bypass the IP Address resolver - need to confirm that is the refer header or something else\n\n
  • #26 -c Limits the number of child directories per parent - not clear in Google Code documentation\n-F Bypass the IP Address resolver - need to confirm that is the refer header or something else\n\n
  • #27 \n
  • #28 -B suppress warning of trusted domains i.e. Cross Domain Content Inclusion\n-Q Suppress the reporting of duplicate nodes i.e. might miss something in report\n-p Used to perform a percentage of the scan (i.e. periodic scanning) supplement with -q\n-e http://lcamtuf.blogspot.com/2010/11/understanding-and-using-skipfish.html\n\n\n
  • #29 http://code.google.com/p/skipfish/wiki/SkipfishDoc\n
  • #30 These low risk are quoted from the documentation hosted on Google Code\n
  • #31 FX is Felix Lindner http://www.h-online.com/security/features/Testing-Google-s-Skipfish-1001315.html?view=print\n\n&amp;#x201C;some regular ASCII text files were interpreted as JSON responses without XSSI (Cross Site Script Inclusion) protection&amp;#x201D;\n\nskipfish -J was not mentioned by FX\n
  • #32 http://www.h-online.com/security/features/Testing-Google-s-Skipfish-1001315.html?view=print\n
  • #33 \n
  • #34 http://www.h-online.com/security/features/Testing-Google-s-Skipfish-1001315.html?view=print\n
  • #35 \n
  • #36 \n
  • #37 \n