SlideShare a Scribd company logo
1 of 8
Mining Robots.txt for Fun and Profit Vivek Ramachandran http://www.SecurityTube.Net
SecurityTube.Net www.SecurityTube.Net   - the YouTube for Computer Networking and Security!
What is Robots.Txt ? ,[object Object],[object Object],[object Object]
A Sample Robots.txt ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
What Robots.txt should not be used for! ,[object Object],[object Object],[object Object],[object Object],[object Object]
Badly written robots.txt file ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]
Lets surf to /ftp_download An Attacker could easily use a brute force Authentication cracker to get through this screen!
Demo! ,[object Object]

More Related Content

More from SecurityTube.Net (8)

Network Attacks
Network AttacksNetwork Attacks
Network Attacks
ย 
TCP/IP basics
TCP/IP basicsTCP/IP basics
TCP/IP basics
ย 
Wireless Security Basics
Wireless Security BasicsWireless Security Basics
Wireless Security Basics
ย 
Microsoft OS Vulnerabilities
Microsoft OS VulnerabilitiesMicrosoft OS Vulnerabilities
Microsoft OS Vulnerabilities
ย 
Cryptography Lecture by Sam Bowne
Cryptography Lecture by Sam BowneCryptography Lecture by Sam Bowne
Cryptography Lecture by Sam Bowne
ย 
Active Https Cookie Stealing
Active Https Cookie StealingActive Https Cookie Stealing
Active Https Cookie Stealing
ย 
Black Hat Dc 09 Marlinspike Defeating Ssl
Black Hat Dc 09 Marlinspike Defeating SslBlack Hat Dc 09 Marlinspike Defeating Ssl
Black Hat Dc 09 Marlinspike Defeating Ssl
ย 
SSL MITM Attack Over Wireless
SSL MITM Attack Over WirelessSSL MITM Attack Over Wireless
SSL MITM Attack Over Wireless
ย 

Recently uploaded

From Luxury Escort Service Kamathipura : 9352852248 Make on-demand Arrangemen...
From Luxury Escort Service Kamathipura : 9352852248 Make on-demand Arrangemen...From Luxury Escort Service Kamathipura : 9352852248 Make on-demand Arrangemen...
From Luxury Escort Service Kamathipura : 9352852248 Make on-demand Arrangemen...
From Luxury Escort : 9352852248 Make on-demand Arrangements Near yOU
ย 
call girls in Sant Nagar (DELHI) ๐Ÿ” >เผ’9953056974 ๐Ÿ” genuine Escort Service ๐Ÿ”โœ”๏ธโœ”๏ธ
call girls in Sant Nagar (DELHI) ๐Ÿ” >เผ’9953056974 ๐Ÿ” genuine Escort Service ๐Ÿ”โœ”๏ธโœ”๏ธcall girls in Sant Nagar (DELHI) ๐Ÿ” >เผ’9953056974 ๐Ÿ” genuine Escort Service ๐Ÿ”โœ”๏ธโœ”๏ธ
call girls in Sant Nagar (DELHI) ๐Ÿ” >เผ’9953056974 ๐Ÿ” genuine Escort Service ๐Ÿ”โœ”๏ธโœ”๏ธ
9953056974 Low Rate Call Girls In Saket, Delhi NCR
ย 
โžฅ๐Ÿ” 7737669865 ๐Ÿ”โ–ป Malda Call-girls in Women Seeking Men ๐Ÿ”Malda๐Ÿ” Escorts Ser...
โžฅ๐Ÿ” 7737669865 ๐Ÿ”โ–ป Malda Call-girls in Women Seeking Men  ๐Ÿ”Malda๐Ÿ”   Escorts Ser...โžฅ๐Ÿ” 7737669865 ๐Ÿ”โ–ป Malda Call-girls in Women Seeking Men  ๐Ÿ”Malda๐Ÿ”   Escorts Ser...
โžฅ๐Ÿ” 7737669865 ๐Ÿ”โ–ป Malda Call-girls in Women Seeking Men ๐Ÿ”Malda๐Ÿ” Escorts Ser...
amitlee9823
ย 
Call Girls in New Ashok Nagar, (delhi) call me [9953056974] escort service 24X7
Call Girls in New Ashok Nagar, (delhi) call me [9953056974] escort service 24X7Call Girls in New Ashok Nagar, (delhi) call me [9953056974] escort service 24X7
Call Girls in New Ashok Nagar, (delhi) call me [9953056974] escort service 24X7
9953056974 Low Rate Call Girls In Saket, Delhi NCR
ย 
VIP Independent Call Girls in Mumbai ๐ŸŒน 9920725232 ( Call Me ) Mumbai Escorts ...
VIP Independent Call Girls in Mumbai ๐ŸŒน 9920725232 ( Call Me ) Mumbai Escorts ...VIP Independent Call Girls in Mumbai ๐ŸŒน 9920725232 ( Call Me ) Mumbai Escorts ...
VIP Independent Call Girls in Mumbai ๐ŸŒน 9920725232 ( Call Me ) Mumbai Escorts ...
dipikadinghjn ( Why You Choose Us? ) Escorts
ย 
Call Girls Service Pune โ‚น7.5k Pick Up & Drop With Cash Payment 9352852248 Cal...
Call Girls Service Pune โ‚น7.5k Pick Up & Drop With Cash Payment 9352852248 Cal...Call Girls Service Pune โ‚น7.5k Pick Up & Drop With Cash Payment 9352852248 Cal...
Call Girls Service Pune โ‚น7.5k Pick Up & Drop With Cash Payment 9352852248 Cal...
roshnidevijkn ( Why You Choose Us? ) Escorts
ย 
Call Girls Banaswadi Just Call ๐Ÿ‘— 7737669865 ๐Ÿ‘— Top Class Call Girl Service Ban...
Call Girls Banaswadi Just Call ๐Ÿ‘— 7737669865 ๐Ÿ‘— Top Class Call Girl Service Ban...Call Girls Banaswadi Just Call ๐Ÿ‘— 7737669865 ๐Ÿ‘— Top Class Call Girl Service Ban...
Call Girls Banaswadi Just Call ๐Ÿ‘— 7737669865 ๐Ÿ‘— Top Class Call Girl Service Ban...
amitlee9823
ย 
CBD Belapur Expensive Housewife Call Girls Number-๐Ÿ“ž๐Ÿ“ž9833754194 No 1 Vipp HIgh...
CBD Belapur Expensive Housewife Call Girls Number-๐Ÿ“ž๐Ÿ“ž9833754194 No 1 Vipp HIgh...CBD Belapur Expensive Housewife Call Girls Number-๐Ÿ“ž๐Ÿ“ž9833754194 No 1 Vipp HIgh...
CBD Belapur Expensive Housewife Call Girls Number-๐Ÿ“ž๐Ÿ“ž9833754194 No 1 Vipp HIgh...
priyasharma62062
ย 
VIP Call Girl in Mumbai Central ๐Ÿ’ง 9920725232 ( Call Me ) Get A New Crush Ever...
VIP Call Girl in Mumbai Central ๐Ÿ’ง 9920725232 ( Call Me ) Get A New Crush Ever...VIP Call Girl in Mumbai Central ๐Ÿ’ง 9920725232 ( Call Me ) Get A New Crush Ever...
VIP Call Girl in Mumbai Central ๐Ÿ’ง 9920725232 ( Call Me ) Get A New Crush Ever...
dipikadinghjn ( Why You Choose Us? ) Escorts
ย 

Recently uploaded (20)

7 tips trading Deriv Accumulator Options
7 tips trading Deriv Accumulator Options7 tips trading Deriv Accumulator Options
7 tips trading Deriv Accumulator Options
ย 
(Sexy Sheela) Call Girl Mumbai Call Now ๐Ÿ‘‰9920725232๐Ÿ‘ˆ Mumbai Escorts 24x7
(Sexy Sheela) Call Girl Mumbai Call Now ๐Ÿ‘‰9920725232๐Ÿ‘ˆ Mumbai Escorts 24x7(Sexy Sheela) Call Girl Mumbai Call Now ๐Ÿ‘‰9920725232๐Ÿ‘ˆ Mumbai Escorts 24x7
(Sexy Sheela) Call Girl Mumbai Call Now ๐Ÿ‘‰9920725232๐Ÿ‘ˆ Mumbai Escorts 24x7
ย 
Navi Mumbai Cooperetive Housewife Call Girls-9833754194-Natural Panvel Enjoye...
Navi Mumbai Cooperetive Housewife Call Girls-9833754194-Natural Panvel Enjoye...Navi Mumbai Cooperetive Housewife Call Girls-9833754194-Natural Panvel Enjoye...
Navi Mumbai Cooperetive Housewife Call Girls-9833754194-Natural Panvel Enjoye...
ย 
From Luxury Escort Service Kamathipura : 9352852248 Make on-demand Arrangemen...
From Luxury Escort Service Kamathipura : 9352852248 Make on-demand Arrangemen...From Luxury Escort Service Kamathipura : 9352852248 Make on-demand Arrangemen...
From Luxury Escort Service Kamathipura : 9352852248 Make on-demand Arrangemen...
ย 
call girls in Sant Nagar (DELHI) ๐Ÿ” >เผ’9953056974 ๐Ÿ” genuine Escort Service ๐Ÿ”โœ”๏ธโœ”๏ธ
call girls in Sant Nagar (DELHI) ๐Ÿ” >เผ’9953056974 ๐Ÿ” genuine Escort Service ๐Ÿ”โœ”๏ธโœ”๏ธcall girls in Sant Nagar (DELHI) ๐Ÿ” >เผ’9953056974 ๐Ÿ” genuine Escort Service ๐Ÿ”โœ”๏ธโœ”๏ธ
call girls in Sant Nagar (DELHI) ๐Ÿ” >เผ’9953056974 ๐Ÿ” genuine Escort Service ๐Ÿ”โœ”๏ธโœ”๏ธ
ย 
(INDIRA) Call Girl Mumbai Call Now 8250077686 Mumbai Escorts 24x7
(INDIRA) Call Girl Mumbai Call Now 8250077686 Mumbai Escorts 24x7(INDIRA) Call Girl Mumbai Call Now 8250077686 Mumbai Escorts 24x7
(INDIRA) Call Girl Mumbai Call Now 8250077686 Mumbai Escorts 24x7
ย 
โžฅ๐Ÿ” 7737669865 ๐Ÿ”โ–ป Malda Call-girls in Women Seeking Men ๐Ÿ”Malda๐Ÿ” Escorts Ser...
โžฅ๐Ÿ” 7737669865 ๐Ÿ”โ–ป Malda Call-girls in Women Seeking Men  ๐Ÿ”Malda๐Ÿ”   Escorts Ser...โžฅ๐Ÿ” 7737669865 ๐Ÿ”โ–ป Malda Call-girls in Women Seeking Men  ๐Ÿ”Malda๐Ÿ”   Escorts Ser...
โžฅ๐Ÿ” 7737669865 ๐Ÿ”โ–ป Malda Call-girls in Women Seeking Men ๐Ÿ”Malda๐Ÿ” Escorts Ser...
ย 
Call Girls in New Ashok Nagar, (delhi) call me [9953056974] escort service 24X7
Call Girls in New Ashok Nagar, (delhi) call me [9953056974] escort service 24X7Call Girls in New Ashok Nagar, (delhi) call me [9953056974] escort service 24X7
Call Girls in New Ashok Nagar, (delhi) call me [9953056974] escort service 24X7
ย 
Kopar Khairane Russian Call Girls Number-9833754194-Navi Mumbai Fantastic Unl...
Kopar Khairane Russian Call Girls Number-9833754194-Navi Mumbai Fantastic Unl...Kopar Khairane Russian Call Girls Number-9833754194-Navi Mumbai Fantastic Unl...
Kopar Khairane Russian Call Girls Number-9833754194-Navi Mumbai Fantastic Unl...
ย 
VIP Independent Call Girls in Mumbai ๐ŸŒน 9920725232 ( Call Me ) Mumbai Escorts ...
VIP Independent Call Girls in Mumbai ๐ŸŒน 9920725232 ( Call Me ) Mumbai Escorts ...VIP Independent Call Girls in Mumbai ๐ŸŒน 9920725232 ( Call Me ) Mumbai Escorts ...
VIP Independent Call Girls in Mumbai ๐ŸŒน 9920725232 ( Call Me ) Mumbai Escorts ...
ย 
(INDIRA) Call Girl Srinagar Call Now 8617697112 Srinagar Escorts 24x7
(INDIRA) Call Girl Srinagar Call Now 8617697112 Srinagar Escorts 24x7(INDIRA) Call Girl Srinagar Call Now 8617697112 Srinagar Escorts 24x7
(INDIRA) Call Girl Srinagar Call Now 8617697112 Srinagar Escorts 24x7
ย 
Vasai-Virar Fantastic Call Girls-9833754194-Call Girls MUmbai
Vasai-Virar Fantastic Call Girls-9833754194-Call Girls MUmbaiVasai-Virar Fantastic Call Girls-9833754194-Call Girls MUmbai
Vasai-Virar Fantastic Call Girls-9833754194-Call Girls MUmbai
ย 
Cybersecurity Threats in Financial Services Protection.pptx
Cybersecurity Threats in  Financial Services Protection.pptxCybersecurity Threats in  Financial Services Protection.pptx
Cybersecurity Threats in Financial Services Protection.pptx
ย 
Q1 2024 Conference Call Presentation vF.pdf
Q1 2024 Conference Call Presentation vF.pdfQ1 2024 Conference Call Presentation vF.pdf
Q1 2024 Conference Call Presentation vF.pdf
ย 
Call Girls Service Pune โ‚น7.5k Pick Up & Drop With Cash Payment 9352852248 Cal...
Call Girls Service Pune โ‚น7.5k Pick Up & Drop With Cash Payment 9352852248 Cal...Call Girls Service Pune โ‚น7.5k Pick Up & Drop With Cash Payment 9352852248 Cal...
Call Girls Service Pune โ‚น7.5k Pick Up & Drop With Cash Payment 9352852248 Cal...
ย 
Call Girls Banaswadi Just Call ๐Ÿ‘— 7737669865 ๐Ÿ‘— Top Class Call Girl Service Ban...
Call Girls Banaswadi Just Call ๐Ÿ‘— 7737669865 ๐Ÿ‘— Top Class Call Girl Service Ban...Call Girls Banaswadi Just Call ๐Ÿ‘— 7737669865 ๐Ÿ‘— Top Class Call Girl Service Ban...
Call Girls Banaswadi Just Call ๐Ÿ‘— 7737669865 ๐Ÿ‘— Top Class Call Girl Service Ban...
ย 
CBD Belapur Expensive Housewife Call Girls Number-๐Ÿ“ž๐Ÿ“ž9833754194 No 1 Vipp HIgh...
CBD Belapur Expensive Housewife Call Girls Number-๐Ÿ“ž๐Ÿ“ž9833754194 No 1 Vipp HIgh...CBD Belapur Expensive Housewife Call Girls Number-๐Ÿ“ž๐Ÿ“ž9833754194 No 1 Vipp HIgh...
CBD Belapur Expensive Housewife Call Girls Number-๐Ÿ“ž๐Ÿ“ž9833754194 No 1 Vipp HIgh...
ย 
Mira Road Memorable Call Grls Number-9833754194-Bhayandar Speciallty Call Gir...
Mira Road Memorable Call Grls Number-9833754194-Bhayandar Speciallty Call Gir...Mira Road Memorable Call Grls Number-9833754194-Bhayandar Speciallty Call Gir...
Mira Road Memorable Call Grls Number-9833754194-Bhayandar Speciallty Call Gir...
ย 
VIP Call Girl in Mumbai Central ๐Ÿ’ง 9920725232 ( Call Me ) Get A New Crush Ever...
VIP Call Girl in Mumbai Central ๐Ÿ’ง 9920725232 ( Call Me ) Get A New Crush Ever...VIP Call Girl in Mumbai Central ๐Ÿ’ง 9920725232 ( Call Me ) Get A New Crush Ever...
VIP Call Girl in Mumbai Central ๐Ÿ’ง 9920725232 ( Call Me ) Get A New Crush Ever...
ย 
Call Girls Koregaon Park Call Me 7737669865 Budget Friendly No Advance Booking
Call Girls Koregaon Park Call Me 7737669865 Budget Friendly No Advance BookingCall Girls Koregaon Park Call Me 7737669865 Budget Friendly No Advance Booking
Call Girls Koregaon Park Call Me 7737669865 Budget Friendly No Advance Booking
ย 

Analyzing Robots.txt for Fun and Profit