Mining Robots.txt for Fun and Profit Vivek Ramachandran http://www.SecurityTube.Net

SecurityTube.Net www.SecurityTube.Net - the YouTube for Computer Networking and Security!

What is Robots.Txt ? A plain text file placed in the wwwroot of a website It serves as a way to instruct automated bots such as search engine bots (Googlebot , Yahoo! Slurp etc ) about what to mine from the site and what not to It is written in what is referred to as a Robots Exclusion Protocol

A plain text file placed in the wwwroot of a website

It serves as a way to instruct automated bots such as search engine bots (Googlebot , Yahoo! Slurp etc ) about what to mine from the site and what not to

It is written in what is referred to as a Robots Exclusion Protocol

A Sample Robots.txt User-Agent: Googlebot Disallow: /images Disallow: /archive Disallow: /xyz … User-Agent: * Disallow: /images

User-Agent: Googlebot

Disallow: /images

Disallow: /archive

Disallow: /xyz

…

User-Agent: *

Disallow: /images

What Robots.txt should not be used for! It should never be used to hide important directories Should be never used as some form of security Reason: The file is world readable Anyone can disobey the rules written there

It should never be used to hide important directories

Should be never used as some form of security

Reason:

The file is world readable

Anyone can disobey the rules written there

Badly written robots.txt file User-agent: * Disallow: /partner_extranet/ Disallow: /faq/ Disallow: /ftp_download/ Disallow: /protected/ Disallow: /scripts/ Disallow: /CVS/

User-agent: *

Disallow: /partner_extranet/

Disallow: /faq/

Disallow: /ftp_download/

Disallow: /protected/

Disallow: /scripts/

Disallow: /CVS/

Lets surf to /ftp_download An Attacker could easily use a brute force Authentication cracker to get through this screen!

Demo! We will look at the Robots.txt file of some common websites and analyze them for possible vulnerabilities

We will look at the Robots.txt file of some common websites and analyze them for possible vulnerabilities