Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Shmoocon 2015 - httpscreenshot

1,617 views

Published on

httpscreenshot is a tool developed internally over the past year and a half. It has become one of our go to tools for the reconnaissance phase of every penetration test. The tool itself takes a list of addresses, domains, URLs, and visits each in a browser, parses SSL certificates to add new hosts, and captures a screenshot/HTML of the browser instance. Similar tools exist but none met our needs with regards to speed (threaded), features (JavaScript support, SSL auto detection and certificate scraping), and reliability.

The cluster portion of the tool will go through and group "similar" websites together, where "similar" is determined by a fuzzy matching metric.

This tool can be used by both blue and red teams. The blue teams can use this tool to quickly create an inventory of applications and devices they have running in their environments. This inventory will allow them to quickly see if there is anything running in their environment that they may not know about which should be secured or in many cases removed.

The red teams can use this tool to quickly create the same inventory as part of our reconnaissance, which is often very effective in identifying potential target assets.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Shmoocon 2015 - httpscreenshot

  1. 1. httpscreenshot A Tool for Both Teams
  2. 2. Outline • Who we are • The problem • Our solution • Demo • How we’ve used it • Q&A
  3. 3. Who We Are – Steve Breen • Senior penetration tester • Former “Enterprise” developer – Current hacky script developer • Vulnerability and exploit development hobbyist by night @breenmachine breenmachine.blogspot.com
  4. 4. Who we are – Justin Kennedy • Lifelong security hobbyist, actively for ~15 years • Intern -> Computer Tech -> Help Desk -> SOC -> Network Security -> Junior PT -> Senior PT -> Team Lead • Terrible at making slides look pretty… if you came to see pretty slides, you may be in the wrong talk. If you came to see an awesome tool that is available (and is OSS) as of today, you’re in the right place. • @jstnkndy / juken (freenode)
  5. 5. Before we get started _o/
  6. 6. Blue Team challenge Let’s start off with a question… Can you account for every device or application on your network? Why not?
  7. 7. Blue Team challenge reasons 1. You’ve inherited an infrastructure that you didn’t build and (of course) not everything was documented by your predecessor. 2. You work in an environment where business units don't necessarily communicate as much as they should and another business unit has spun up some demo or test application without telling you. 3. You forgot about that old NT4 or tomcat box that no one has touched in the past 10 years. 4. Or someone just plugged some shit into a network jack.
  8. 8. Red Team challenge 1. We are constantly attempting to compromise organizations that we don't know anything about (besides our recon). 2. It's our job to identify what the target attack surface looks like. 3. Anyone in here ever masscan a /8 for common web ports? 4. Let’s face it, we don’t always have as much time as we’d like on an assessment.
  9. 9. *Disclaimer: Some of these may look familiar
  10. 10. root calvin
  11. 11. Our Solution HTTPScreenshot/Cluster • HTTPScreenshot: A python script to screenshot thousands of websites really quickly (and reliably) • Cluster: A script to do “fuzzy matching” on HTML pages. Produce immediately usable output with “similar” pages grouped together.
  12. 12. HTTPScreenshot.py • Goals: Fast, Thorough, Automagic • Challenges: Code was hacked together during assessments – needs some TLC • Fun features: – Input is nmap/masscan output – Javascript parsed and executed – SSL autodetect – SSL Certificate domain scraping for vhosts – Headless (configurable fail-over to FireFox) – Threaded – Saves PNG and HTML (good for grep’ing) – Attempts tls 1.0 and falls back to sslv3 when necessary
  13. 13. Cluster.py • Identify “similar” websites and group them together • Displays the resulting groups in a useful way (HTML output with JS “hoverzoom”) • Algorithm - Reduces to DBSCAN: – Needed a clustering algorithm that didn’t require definition of “k” – Uses HTML tag/attr values – computes a “similarity” score for two sites • href, name, src, id, class, title, h1 – Works fairly well – could DEFINITELY be improved upon • Supports “diff” reports. Sites that have been changed/added/removed since the last scan
  14. 14. Demo!
  15. 15. How we’ve used it
  16. 16. Story 1 - SQLi
  17. 17. Story 2 – WTF is SuperGoose?
  18. 18. Story 2 – WTF is SuperGoose?
  19. 19. Story 3 – Bug Bounties
  20. 20. Cheat Sheet: Usage • masscan --iL scope.txt --oG out.gnmap –rate 100000 • httpscreenshot.py -i out.gnmap --headless -- workers 50 --timeout 45 --autodetect --vhosts -- retries 1 • cluster.py –d <httpscreenshot directory>
  21. 21. masshttp #!/bin/sh ~/tools/masscan –p1-65535 -iL scope.txt -oG out.gnmap --rate 100000 mkdir httpscreenshot cd httpscreenshot python ~/tools/httpscreenshot.py -i ../out.gnmap -p -t 50 -w 45 - a -vH -r 1 python ~/tools/httpscreenshot.py -i ../out.gnmap -p -t 10 -w 5 -a -vH -r 1 cd .. python ~/tools/cluster.py -d httpscreenshot/
  22. 22. Questions / Suggestions? Code: github.com/breenmachine/

×