Analyzing Robots.txt for Fun and Profit

Mining Robots.txt for Fun and Profit Vivek Ramachandran http://www.SecurityTube.Net

SecurityTube.Net www.SecurityTube.Net - the YouTube for Computer Networking and Security!

What is Robots.Txt ?
A plain text file placed in the wwwroot of a website
It serves as a way to instruct automated bots such as search engine bots (Googlebot , Yahoo! Slurp etc ) about what to mine from the site and what not to
It is written in what is referred to as a Robots Exclusion Protocol

A Sample Robots.txt
User-Agent: Googlebot
Disallow: /images
Disallow: /archive
Disallow: /xyz
…
User-Agent: *
Disallow: /images

What Robots.txt should not be used for!
It should never be used to hide important directories
Should be never used as some form of security
Reason:
The file is world readable
Anyone can disobey the rules written there

Badly written robots.txt file
User-agent: *
Disallow: /partner_extranet/
Disallow: /faq/
Disallow: /ftp_download/
Disallow: /protected/
Disallow: /scripts/
Disallow: /CVS/

Lets surf to /ftp_download An Attacker could easily use a brute force Authentication cracker to get through this screen!

Demo!
We will look at the Robots.txt file of some common websites and analyze them for possible vulnerabilities

http://www.securitytube.net	579
http://securitytube.net	340
http://blog.security4all.be	197
http://security4all.blogspot.com	38
http://translate.googleusercontent.com	1

Analyzing Robots.txt for Fun and Profit

by SecurityTube.Net on Jun 13, 2008

Accessibility

Categories

Tags

Upload Details

Usage Rights

5 Embeds 1,155

Statistics

Analyzing Robots.txt for Fun and Profit — Presentation Transcript