Your SlideShare is downloading. ×
0
Mining Robots.txt for Fun and Profit Vivek Ramachandran http://www.SecurityTube.Net
SecurityTube.Net www.SecurityTube.Net   - the YouTube for Computer Networking and Security!
What is Robots.Txt ? <ul><li>A plain text file placed in the wwwroot of a website </li></ul><ul><li>It serves as a way to ...
A Sample Robots.txt <ul><li>User-Agent: Googlebot </li></ul><ul><li>Disallow: /images </li></ul><ul><li>Disallow: /archive...
What Robots.txt should not be used for! <ul><li>It should never be used to hide important directories </li></ul><ul><li>Sh...
Badly written robots.txt file <ul><li>User-agent: *  </li></ul><ul><li>Disallow: /partner_extranet/  </li></ul><ul><li>Dis...
Lets surf to /ftp_download An Attacker could easily use a brute force Authentication cracker to get through this screen!
Demo! <ul><li>We will look at the Robots.txt file of some common websites and analyze them for possible vulnerabilities </...
Upcoming SlideShare
Loading in...5
×

Analyzing Robots.txt for Fun and Profit

2,720

Published on

goto http://www.securitytube.net for a detailed video

Published in: Economy & Finance, Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,720
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
48
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Transcript of "Analyzing Robots.txt for Fun and Profit"

  1. 1. Mining Robots.txt for Fun and Profit Vivek Ramachandran http://www.SecurityTube.Net
  2. 2. SecurityTube.Net www.SecurityTube.Net - the YouTube for Computer Networking and Security!
  3. 3. What is Robots.Txt ? <ul><li>A plain text file placed in the wwwroot of a website </li></ul><ul><li>It serves as a way to instruct automated bots such as search engine bots (Googlebot , Yahoo! Slurp etc ) about what to mine from the site and what not to </li></ul><ul><li>It is written in what is referred to as a Robots Exclusion Protocol </li></ul>
  4. 4. A Sample Robots.txt <ul><li>User-Agent: Googlebot </li></ul><ul><li>Disallow: /images </li></ul><ul><li>Disallow: /archive </li></ul><ul><li>Disallow: /xyz </li></ul><ul><li>… </li></ul><ul><li>User-Agent: * </li></ul><ul><li>Disallow: /images </li></ul>
  5. 5. What Robots.txt should not be used for! <ul><li>It should never be used to hide important directories </li></ul><ul><li>Should be never used as some form of security </li></ul><ul><li>Reason: </li></ul><ul><li>The file is world readable </li></ul><ul><li>Anyone can disobey the rules written there </li></ul>
  6. 6. Badly written robots.txt file <ul><li>User-agent: * </li></ul><ul><li>Disallow: /partner_extranet/ </li></ul><ul><li>Disallow: /faq/ </li></ul><ul><li>Disallow: /ftp_download/ </li></ul><ul><li>Disallow: /protected/ </li></ul><ul><li>Disallow: /scripts/ </li></ul><ul><li>Disallow: /CVS/ </li></ul>
  7. 7. Lets surf to /ftp_download An Attacker could easily use a brute force Authentication cracker to get through this screen!
  8. 8. Demo! <ul><li>We will look at the Robots.txt file of some common websites and analyze them for possible vulnerabilities </li></ul>
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×