OWASP “Google Hacking” Project
           Download Indexed Cache

              Christian Heinrich
              christian...
Copyright Notice


Slides and Notes Licensed as:
 AU Creative Commons 2.5
    Attribution-Non Commercial-No Derivative W...
Updates to Slides

Incorporates all previous slides from:
    OWASP USA Conference 2008
    ToorCon X (USA)
    SecTor ...
Latest (SFW) Slides



Published on
http://www.slideshare.net/cmlh




 OWASP “Google Hacking” Project             4
Published as Separate PPT Presentations

Recommended Delivery:

1. OWASP “Google Hacking” Project
   1.1 “Search Engine Re...
Slide References and Further Info



Refer to the Notes Page of each Slide

Some slides are hidden due to time limit




O...
Christian Heinrich aka “cmlh”



Experience Since 1996:

Penetration Tester
Web Application Security
Reverse Engineer
...
Christian Heinrich aka “cmlh”



.gov.au Procurement Panels:

Federal Attorney General’s CNVA Program
NSW Government 231...
Christian Heinrich aka “cmlh”




 Wireless Network
 https://twitter.com/ruxcon




OWASP “Google Hacking” Project      ...
Christian Heinrich aka “cmlh”



Presented at:
OWASP Conferences
   Australia, Europe and USA.
ToorCon (San Diego, USA)...
Christian Heinrich aka “cmlh”



“End User” Experience Since 1996:

Security Thought Leader within AU Media:
   Former C...
Christian Heinrich aka “cmlh”



“End User” Experience Since 1996:

Federal .gov.au
   DSD Certified Gateway Service Pro...
Christian Heinrich aka “cmlh”

Contributions to OWASP:

OWASP Testing Guide v3
 4.2.1 “Spiders/Robots/Crawlers”
 4.2.2 “...
OWASP “Google Hacking” Project
           Download Indexed Cache

              Christian Heinrich
              christian...
Download Indexed Cache



Supports OWASP Testing Guide v3
4.2.2 “Search Engine Reconnaissance”
   Provides Evidence of C...
Command Line Arguments



Google SOAP Search API related:

 -key                   API Key
                         demo ...
Results 1 to 10
cmlh$ /usr/bin/perl dic.pl –key “demo” -query “site:owasp.org" -start 1

"Download Indexed Cache" Proof of...
Results 11 to …
cmlh$ /usr/bin/perl dic.pl –key demo -query “site:owasp.org" -start 11

"Download Indexed Cache" Proof of ...
Google Search Results - 1 to 1000




#!/usr/bin/perl –w
for (my $result=0; $result < 990; $result = $result + 10) {
   sy...
Exploiting Page Rank



Page Rank Orders “Less Public” Results Last

Descending $start of doGoogleSearch:
 e.g. –start:99...
Google Search Results - 1000 to 1




#!/usr/bin/perl –w
for (my $result=990; $result >= 1; $result = $result - 10) {
   s...
Generated Output
cmlh$ /usr/bin/perl dic.pl –key “demo” -query “site:owasp.org" -start 1

"Download Indexed Cache" Proof o...
Generated Output

Directory:
Name Stripped of “:” from Google Operator
/dic sub-directory


Files in Directory:
x.html
...
1.html Example

cmlh$ cd siteowasp.org/dic/
cmlh$ head –n 25 1.html

<meta http-equiv="Content-Type" content="text/html;
c...
[SearchQuery].csv Example

cmlh$ cat siteowasp.org.csv
1,http://www.owasp.org/
2,http://www.owasp.org/download/
3,http://w...
DataDumper.txt Example

$VAR1 = bless( {
  'searchTime' => '0.136083‘
  'endIndex' => '10',
  'searchComments' => '',
  'd...
Google SOAP Search API in Perl

doGoogleSearch
 $key
 $q
 $start -1 subtracted for Zero Index


doGoogleSearchResponse
...
Google SOAP Search API in Perl

doGetCachedPage
 $key
 $URL


doGetCachedPageResponse
 … xsi:type="ns2:base64">




OWA...
Google SOAP Search API Limitations

Search Query limited to:
10 Words
2048 Bytes


1K Search Queries Per Day
Limited to...
“10K Possible Results from 10 Different Queries”



Specific each FQDN over 10 site: -queries

For example:
 … -query “si...
Google SOAP Search API Limitations

Issuing of API Keys Discontinued 5 Dec 2006




OWASP “Google Hacking” Project        ...
Google SOAP Search API Limitations

Will be Deprecated on 31 August 2009




OWASP “Google Hacking” Project             32
dic Roadmap

PoC v0.1
Previewed at OWASP USA, ToorCon and SecTor (CA)
Released at RUXCON 2K8 in Sydney, AU, Nov 2008


P...
dic Roadmap

PoC v0.3
Specify Range of Google Search Results to 1000
   Code Sync with “TCP Input Text”
   Consider Net...
Call for Project Reviewers



Perl – CPAN Modules

SOAP::Lite
Net::Google




Interested? christian.heinrich@owasp.org
O...
Call for Project Reviewers



Perl – Quality Assurance:
Perl::Critic CPAN Module
perltidy


Code Contribution Licensed a...
Call for Project Reviewers



Development

Eclipse
   EPIC Plug-in
   Subclipse Plug-in
Subversion Repository
   code...
Call for Project Reviewers



OWASP Alpha Project Reviewers:

pdp @ GNUCITIZEN
Chris Gates @ Carnal0wnage
Glenn Roberts...
OWASP Project


Project Endorsers
 Justin Derry (OWASP AU Conference Chair)
 Dinis Cruz (OWASP Board)


OWASP Project Ma...
Project Controversy

              - OWASP “Google Hacking” Role:
2. Someone in an Engineering Function at Google
3. Compl...
Project Controversy

code.google.com denies “Google Hacking” labels




But permits project names of “Google Hacking”
http...
Closing Remarks


Mitigation strategies are in the following slides:
“Spiders/Robots/Crawlers”
“Continuous Improvement”
...
Closing Remarks


Upcoming Presentations:
http://snipurl.com/cmlh_speaking_schedule


E-mail:
christian.heinrich@owasp.org...
Upcoming SlideShare
Loading in …5
×

Download Indexed Cache

3,057 views

Published on

Slides for https://code.google.com/p/dic

Published in: Technology, News & Politics
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
3,057
On SlideShare
0
From Embeds
0
Number of Embeds
15
Actions
Shares
0
Downloads
43
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Download Indexed Cache

  1. 1. OWASP “Google Hacking” Project Download Indexed Cache Christian Heinrich christian.heinrich@owasp.org OWASP “Google Hacking” Project Lead Last Updated 2 July 2009 Copyright © The OWASP Foundation Permission is granted to copy, distribute and/or modify this document under the terms of the OWASP License. The OWASP Foundation http://www.owasp.org
  2. 2. Copyright Notice Slides and Notes Licensed as:  AU Creative Commons 2.5  Attribution-Non Commercial-No Derivative Works OWASP “Google Hacking” Project 2
  3. 3. Updates to Slides Incorporates all previous slides from:  OWASP USA Conference 2008  ToorCon X (USA)  SecTor 2K8 (Canada)  RUXCON 2K8 (Australia)  OWASP Australian Conference 2009  OWASP European Conference 2009  5th CONFidence 2009 (Poland)  OWASP London Chapter Meeting May 2009  SyScan’09 Singapore Lasted Updated 2 July 2009 OWASP “Google Hacking” Project 3
  4. 4. Latest (SFW) Slides Published on http://www.slideshare.net/cmlh OWASP “Google Hacking” Project 4
  5. 5. Published as Separate PPT Presentations Recommended Delivery: 1. OWASP “Google Hacking” Project 1.1 “Search Engine Recon/Discovery” 1.2. “Download Indexed Cache” 2. “TCP Input Text” 3. OWASP “Google Hacking” Project 3.1 “Spiders/Robots/Crawlers” 3.2 “Continuous Improvement” OWASP “Google Hacking” Project 5
  6. 6. Slide References and Further Info Refer to the Notes Page of each Slide Some slides are hidden due to time limit OWASP “Google Hacking” Project 6
  7. 7. Christian Heinrich aka “cmlh” Experience Since 1996: Penetration Tester Web Application Security Reverse Engineer Crypto Analyst Governance (i.e. PCI, ISO, etc) OWASP “Google Hacking” Project 7
  8. 8. Christian Heinrich aka “cmlh” .gov.au Procurement Panels: Federal Attorney General’s CNVA Program NSW Government 2319/2020 OWASP “Google Hacking” Project 8
  9. 9. Christian Heinrich aka “cmlh”  Wireless Network  https://twitter.com/ruxcon OWASP “Google Hacking” Project 9
  10. 10. Christian Heinrich aka “cmlh” Presented at: OWASP Conferences Australia, Europe and USA. ToorCon (San Diego, USA) SecTor (Toronto, Canada) CONFidence (Poland, Europe) SyScan (Singapore) RUXCON (Sydney, Australia) OWASP “Google Hacking” Project 10
  11. 11. Christian Heinrich aka “cmlh” “End User” Experience Since 1996: Security Thought Leader within AU Media: Former CSO of FOXTEL Former CSO of News Limited (AU part of News Corp) OWASP “Google Hacking” Project 11
  12. 12. Christian Heinrich aka “cmlh” “End User” Experience Since 1996: Federal .gov.au DSD Certified Gateway Service Provider  ASIO Web Hosting Government Endorsed Business (GEB) State .nsw.gov.au Critical Infrastructure OWASP “Google Hacking” Project 12
  13. 13. Christian Heinrich aka “cmlh” Contributions to OWASP: OWASP Testing Guide v3  4.2.1 “Spiders/Robots/Crawlers”  4.2.2 “Search Engine Reconnaissance” OWASP “Google Hacking” Project  “Download Indexed Cache” PoC Presentations at OWASP Conferences:  Australia, Europe and USA OWASP “Google Hacking” Project 13
  14. 14. OWASP “Google Hacking” Project Download Indexed Cache Christian Heinrich christian.heinrich@owasp.org OWASP “Google Hacking” Project Lead Last Updated 2 July 2009 Copyright © The OWASP Foundation Permission is granted to copy, distribute and/or modify this document under the terms of the OWASP License. The OWASP Foundation http://www.owasp.org
  15. 15. Download Indexed Cache Supports OWASP Testing Guide v3 4.2.2 “Search Engine Reconnaissance” Provides Evidence of Cached Page during Fieldwork Repository at: http://code.google.com/p/dic OWASP “Google Hacking” Project 15
  16. 16. Command Line Arguments Google SOAP Search API related:  -key API Key demo is embedded API Key  -query Google Search Query  -start Starting Google Search Result (Zero Based Index i.e. 1=0) OWASP “Google Hacking” Project 16
  17. 17. Results 1 to 10 cmlh$ /usr/bin/perl dic.pl –key “demo” -query “site:owasp.org" -start 1 "Download Indexed Cache" Proof of Concept (PoC) 0.1 (Released at RUXCON 2K8) Copyright 2009 Christian Heinrich Licensed under the Apache License, Version 2.0 Creating ./siteowasp.org 1. Downloading https://www.owasp.org/ from Google Cache [46k] as 1.html 2. Downloading http://www.owasp.org/ from Google Cache [46k] as 2.html [SNIP] 8. Downloading http://www.owasp.org/index.php/Session_Management from Google Cache [88k] as 8.html 9. Downloading http://www.owasp.org/index.php/Testing_for_file_extensions handling from Google Cache [24k] as 9.html 10. Downloading http://www.owasp.org/index.php/OWASP_SoC_2008_ASDR_Reviewers from Google Cache [20k] as 10.html OWASP “Google Hacking” Project 17
  18. 18. Results 11 to … cmlh$ /usr/bin/perl dic.pl –key demo -query “site:owasp.org" -start 11 "Download Indexed Cache" Proof of Concept (PoC) 0.1 [SNIP] Copyright 2008 Christian Heinrich Licensed under the Apache License, Version 2.0 Appending ./siteowasp.org 11. Downloading https://www.owasp.org/index.php/System_Information_Leak from Google Cache [26k] as 11.html 12. Downloading http://www.owasp.org/index.php/Buffer_overflows from Google Cache [34k] as 12.html [SNIP] 18. Downloading http://www.owasp.org/index.php/Testing_Guide_Introduction from Google Cache [111k] as 18.html 19. Downloading http://www.owasp.org/index.php/OWASP_Java_Project from Google Cache [28k] as 19.html 20. Downloading https://www.owasp.org/index.php/Insecure_Temporary_File from Google Cache [26k] as 20.html OWASP “Google Hacking” Project 18
  19. 19. Google Search Results - 1 to 1000 #!/usr/bin/perl –w for (my $result=0; $result < 990; $result = $result + 10) { system (“./dic.pl -key “[key]" -query “[query]" -start $resultn"); } OWASP “Google Hacking” Project 19
  20. 20. Exploiting Page Rank Page Rank Orders “Less Public” Results Last Descending $start of doGoogleSearch:  e.g. –start:990, -start:980, etc  Remember $start – 1 i.e. 0 OWASP “Google Hacking” Project 20
  21. 21. Google Search Results - 1000 to 1 #!/usr/bin/perl –w for (my $result=990; $result >= 1; $result = $result - 10) { system (“./dic.pl -key “[key]" -query “[query]" -start $resultn"); } OWASP “Google Hacking” Project 21
  22. 22. Generated Output cmlh$ /usr/bin/perl dic.pl –key “demo” -query “site:owasp.org" -start 1 "Download Indexed Cache" Proof of Concept (PoC) 0.1 (Released at RUXCON 2K8) Copyright 2009 Christian Heinrich Licensed under the Apache License, Version 2.0 Creating ./siteowasp.org 1. Downloading https://www.owasp.org/ from Google Cache [46k] as 1.html 2. Downloading http://www.owasp.org/ from Google Cache [46k] as 2.html [SNIP] 8. Downloading http://www.owasp.org/index.php/Session_Management from Google Cache [88k] as 8.html 9. Downloading http://www.owasp.org/index.php/Testing_for_file_extensions handling from Google Cache [24k] as 9.html 10. Downloading http://www.owasp.org/index.php/OWASP_SoC_2008_ASDR_Reviewers from Google Cache [20k] as 10.html OWASP “Google Hacking” Project 22
  23. 23. Generated Output Directory: Name Stripped of “:” from Google Operator /dic sub-directory Files in Directory: x.html x is Search Result Number [SearchQuery].csv SearchResultNumber, URL OWASP “Google Hacking” Project 23
  24. 24. 1.html Example cmlh$ cd siteowasp.org/dic/ cmlh$ head –n 25 1.html <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"><base href="https://www.owasp.org/index.php /Main_Page"><div style="margin:-1px - 1px 0;padding: 0;border:1px solid #999;background:#fff"><div style= "margin:12px;p adding:8px;border:1px solid #999;background:#ddd;font:13px arial,sans- serif;color:#000;font-weight:normal;text-align:left">This is Google's cache of <a href="https://www.owasp.org/" style="text decoration:underline;color:#00c">https:// www.owasp.org/</a>. It is a snapshot of the page as it appeared on 17 Feb 2009 17:00:03 [snip] OWASP “Google Hacking” Project 24
  25. 25. [SearchQuery].csv Example cmlh$ cat siteowasp.org.csv 1,http://www.owasp.org/ 2,http://www.owasp.org/download/ 3,http://www.owasp.org:443/ 4,https://www.owasp.org/images/b/b1/OWASP_gr_newsle [snip] 5,http://www.owasp.org/images/0/06/Dublin_Sponsorsh [snip] 6,https://www.owasp.org/images/2/21/OWASP_gr_newsle [snip] 7,http://www.owasp.org/index.php/Cincinnati 8,http://www.owasp.org/index.php/Testing_for_file_e [snip] 9,http://www.owasp.org/index.php/OWASP_SoC_2008_ASD [snip] 10,http://www.owasp.org/index.php/OWASP_Taiwan_Tran [snip] OWASP “Google Hacking” Project 25
  26. 26. DataDumper.txt Example $VAR1 = bless( { 'searchTime' => '0.136083‘ 'endIndex' => '10', 'searchComments' => '', 'documentFiltering' => 0, 'searchTips' => '', 'estimatedTotalResultsCount' => '41100', 'searchQuery' => 'site:owasp.org', 'startIndex' => '1', 'resultElements' => [ bless( { [SNIP] OWASP “Google Hacking” Project 26
  27. 27. Google SOAP Search API in Perl doGoogleSearch  $key  $q  $start -1 subtracted for Zero Index doGoogleSearchResponse  URL  cachedSize OWASP “Google Hacking” Project 27
  28. 28. Google SOAP Search API in Perl doGetCachedPage  $key  $URL doGetCachedPageResponse  … xsi:type="ns2:base64"> OWASP “Google Hacking” Project 28
  29. 29. Google SOAP Search API Limitations Search Query limited to: 10 Words 2048 Bytes 1K Search Queries Per Day Limited to Search Results within 0…999 10K Possible Results from 10 Different Queries OWASP “Google Hacking” Project 29
  30. 30. “10K Possible Results from 10 Different Queries” Specific each FQDN over 10 site: -queries For example:  … -query “site:www.google.com” …  … -query “site:video.google.com” … 3. … 9. [snip]  … -query “code.google.com” … OWASP “Google Hacking” Project 30
  31. 31. Google SOAP Search API Limitations Issuing of API Keys Discontinued 5 Dec 2006 OWASP “Google Hacking” Project 31
  32. 32. Google SOAP Search API Limitations Will be Deprecated on 31 August 2009 OWASP “Google Hacking” Project 32
  33. 33. dic Roadmap PoC v0.1 Previewed at OWASP USA, ToorCon and SecTor (CA) Released at RUXCON 2K8 in Sydney, AU, Nov 2008 PoC v0.2 Moving repository to code.google.com/p/dic Records the Timestamp from Google Cache Previewed at OWASP AU/EU 2009, SyScan09SG OWASP “Google Hacking” Project 33
  34. 34. dic Roadmap PoC v0.3 Specify Range of Google Search Results to 1000 Code Sync with “TCP Input Text” Consider Net::Google CPAN Perl Module PoC v0.4 Maintenance Release Released approx 31 August 2009 Once Google deprecates SOAP Search API OWASP “Google Hacking” Project 34
  35. 35. Call for Project Reviewers Perl – CPAN Modules SOAP::Lite Net::Google Interested? christian.heinrich@owasp.org OWASP “Google Hacking” Project 35
  36. 36. Call for Project Reviewers Perl – Quality Assurance: Perl::Critic CPAN Module perltidy Code Contribution Licensed as: Apache License, Version 2.0 Interested? christian.heinrich@owasp.org OWASP “Google Hacking” Project 36
  37. 37. Call for Project Reviewers Development Eclipse EPIC Plug-in Subclipse Plug-in Subversion Repository code.google.com Interested? christian.heinrich@owasp.org OWASP “Google Hacking” Project 37
  38. 38. Call for Project Reviewers OWASP Alpha Project Reviewers: pdp @ GNUCITIZEN Chris Gates @ Carnal0wnage Glenn Roberts @ Solutionary Interested? christian.heinrich@owasp.org OWASP “Google Hacking” Project 38
  39. 39. OWASP Project Project Endorsers  Justin Derry (OWASP AU Conference Chair)  Dinis Cruz (OWASP Board) OWASP Project Manager  Paulo Coimbra OWASP “Google Hacking” Project 39
  40. 40. Project Controversy - OWASP “Google Hacking” Role: 2. Someone in an Engineering Function at Google 3. Complaint Received by Tom Brennan (OWASP) Facts:  Not an Google or OWASP Summer of Code  Does not violate Google’s Terms of Service  Contacted for Sec. Role at Google Sydney AU  Google SOAP API perl code related to tit  Separation with OWASP Project due to new scope OWASP “Google Hacking” Project 40
  41. 41. Project Controversy code.google.com denies “Google Hacking” labels But permits project names of “Google Hacking” http://code.google.com/p/googlehacking OWASP “Google Hacking” Project 41
  42. 42. Closing Remarks Mitigation strategies are in the following slides: “Spiders/Robots/Crawlers” “Continuous Improvement” OWASP “Google Hacking” Project 42
  43. 43. Closing Remarks Upcoming Presentations: http://snipurl.com/cmlh_speaking_schedule E-mail: christian.heinrich@owasp.org Slides available from: http://www.slideshare.net/cmlh OWASP “Google Hacking” Project 43

×