Analyzing Robots.txt for Fun and Profit

3,075 views

Published on

goto http://www.securitytube.net for a detailed video

Published in: Economy & Finance, Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,075
On SlideShare
0
From Embeds
0
Number of Embeds
1,143
Actions
Shares
0
Downloads
55
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Analyzing Robots.txt for Fun and Profit

  1. 1. Mining Robots.txt for Fun and Profit Vivek Ramachandran http://www.SecurityTube.Net
  2. 2. SecurityTube.Net www.SecurityTube.Net - the YouTube for Computer Networking and Security!
  3. 3. What is Robots.Txt ? <ul><li>A plain text file placed in the wwwroot of a website </li></ul><ul><li>It serves as a way to instruct automated bots such as search engine bots (Googlebot , Yahoo! Slurp etc ) about what to mine from the site and what not to </li></ul><ul><li>It is written in what is referred to as a Robots Exclusion Protocol </li></ul>
  4. 4. A Sample Robots.txt <ul><li>User-Agent: Googlebot </li></ul><ul><li>Disallow: /images </li></ul><ul><li>Disallow: /archive </li></ul><ul><li>Disallow: /xyz </li></ul><ul><li>… </li></ul><ul><li>User-Agent: * </li></ul><ul><li>Disallow: /images </li></ul>
  5. 5. What Robots.txt should not be used for! <ul><li>It should never be used to hide important directories </li></ul><ul><li>Should be never used as some form of security </li></ul><ul><li>Reason: </li></ul><ul><li>The file is world readable </li></ul><ul><li>Anyone can disobey the rules written there </li></ul>
  6. 6. Badly written robots.txt file <ul><li>User-agent: * </li></ul><ul><li>Disallow: /partner_extranet/ </li></ul><ul><li>Disallow: /faq/ </li></ul><ul><li>Disallow: /ftp_download/ </li></ul><ul><li>Disallow: /protected/ </li></ul><ul><li>Disallow: /scripts/ </li></ul><ul><li>Disallow: /CVS/ </li></ul>
  7. 7. Lets surf to /ftp_download An Attacker could easily use a brute force Authentication cracker to get through this screen!
  8. 8. Demo! <ul><li>We will look at the Robots.txt file of some common websites and analyze them for possible vulnerabilities </li></ul>

×