The document discusses techniques for performing specialized searches and information extraction from web pages and documents. It describes using regular expressions and custom scripts to search for patterns like phone numbers. It also provides examples of using APIs and libraries like JavaScript::V8 to execute JavaScript and extract hidden content from web pages. The goal is to go beyond typical keyword searches to recognize structures and patterns to refine search results and find embedded information.

1. Christopher M. Frenz

2.  Information is being generated at a faster
rate than ever before
 The speed at which information can be
generated is continually increasing
 Continuous improvements in computers,
storage, and networking make much of this
information readily available to indviduals

3. 54,000 hits

5.  Most search engines use a keyword based
approach
 If a document contains all of the keywords
specified it is returned as a match
 Ranking algorithms (e.g. PageRank) are used
to put the most relevant results at the top of
the list and the least relevant at the bottom

6.  Not everything can be easily expressed as a
keyword
 Suppose you want to search for unknown
phone numbers? How can you do this with
keywords?
 How do we recognize a phone number when
we see one?

8.  We recognize a phone number by recognizing
the pattern of digits
◦ (XXX) XXX-XXXX
 While it is hard to express such a pattern in
the form of a keyword, it is really easy to
express it in the form of a regular expression
 (s?(?d{3})?[-s.]?d{3}[-.]d{4})

9. #!usr/bin/perl
use strict;
use warnings;
(my $string=<<'LIST');
John (555) 555-5555 fits pattern
Bob 234 567-8901
Mary 734-234-9873
Tom 999 999-9999
Harry 111 111 1111 does not fit pattern
LIST
while($string=~/(s?(?d{3})?[-s.]?d{3}[-.]d{4})/g){
print "$1n";
}

10.  Conduct a broad key word search using an
existing search engine
 Use your custom coded application to take
the returned search results and perform
regular expression based pattern matching
 The results that match your regular
expression are your refined search results

11. General Search APIs Specialized Search APIs
 Bing
 Yahoo BOSS
 Blekko
 Yandex
 Twitter
 Medicine – Pubmed
 Physics –Arxiv
 Government –
GovTrack
 Finance – Yahoo
Finance
 etc

12. Seeking to Extract: DFN [A-Z]d+
Script described in:
http://www.biomedcentral.com/1472-6947/7/32

13.  #!usr/bin/perl
 use LWP;
 use strict;
 use warnings;
 #sets query and congress session
 my $query='fracking';
 my $congress=112;
 my $ua = LWP::UserAgent->new;
 my $url="http://www.govtrack.us/api/v2/bill?q=$query&congress=$congress";
 my $response=$ua->get($url);
 my $result=$response->content;
 print $result;
Returns JSON formatted
output

14.  #!usr/bin/perl

 use LWP;
 use XML::LibXML;
 use strict;
 use warnings;

 my $ua=LWP::UserAgent->new();
 my $query='perl programming';
 my $url="http://blekko.com/ws/?q=$query+/rss";
 my $response=$ua->get($url);
 my $results=$response->content; die unless $response->is_success;

 my $parser=XML::LibXML->new;
 my $domtree=$parser->parse_string($results);
 my @Records=$domtree->getElementsByTagName("item");
 my $i=0;
 foreach(@Records){
 my $link=$Records[$i]->getChildrenByTagName("link");
 print "$i $linkn";
 my $description=$Records[$i]->getChildrenByTagName("description");
 print "$descriptionnn";
 $i++;
 }

16.  Allows programmers to extract code samples
pertaining to a set of keywords
 Recognizes the patterns associated with
CC++ functions and CC++ Control
structures
int myfunc ( ){
//code here
}
while ( ) {
//code here
}

17.  use Text::Balanced qw(extract_codeblock);

 #delimiter used to distinguish code blocks for use with Text::Balanced
 $delim='{}';

 #regex used to match keywords/patterns that precede code blocks
 my $regex='(((int|long|double|float|void)s*?w{1,25})|if|while|for)';

 foreach $link(@links){
 $response=$request->get("$link"); # gets Web page
 $results=$response->content;
 while($results=~s/<script.*?>.*?</script>//gsi){}; # filters out Javascript
 pos($results)=0;
 while($results=~s/.*?($regexs*?(.*?)s*?){/{/s){
 $code=$1 . extract_codeblock($results,$delim);
 print OFile "<h3><a href="$link">$link</a></h3> n";
 print OFile "$code" . "n" . "n";
 }
 }

19.  A common challenge to performing
information extraction and text mining on
many Web pages or parts of Web pages is
that the content is served up by JavaScript
 This can be dealt with by putting the
JavaScript that serves up the content through
a JavaScript Engine like V8

20.  <title>Contact XYZ inc</title>
<H1>Contact XYZ inc</H1><br>
<p>For more information about XYZ inc, please contact us at the following Email address</p>
<script type="text/javascript" language="javascript">
<!--
// Email obfuscator script 2.1 by Tim Williams, University of Arizona
// Random encryption key feature by Andrew Moulden, Site Engineering Ltd
// This code is freeware provided these four comment lines remain intact
// A wizard to generate this code is at http://www.jottings.com/obfuscator/
{ coded = "OKUxkq@KwtoO2K.0ko"
key = "l7rE9B41VmIKiFwOLq2uUGYCQaWoMfzNASycJj3Ds8dtRkPv6XTHg0beh5xpZn"
shift=coded.length
link=""
for (i=0; i<coded.length; i++) {
if (key.indexOf(coded.charAt(i))==-1) {
ltr = coded.charAt(i)
link += (ltr)
}
else {
ltr = (key.indexOf(coded.charAt(i))-shift+key.length) % key.length
link += (key.charAt(ltr))
}
}
document.write("<a href='mailto:"+link+"'>"+link+"</a>")
}
//-->
</script><noscript>Sorry, you need Javascript on to email me.</noscript>

22.  #!usr/bin/perl
use JavaScript::V8;
use LWP;
use Text::Balanced qw(extract_codeblock);
use strict;
use warnings;
#delimiter used to distinguish code blocks for use with Text::Balanced
my $delim='{}';
#downloads Web page
my $ua=LWP::UserAgent->new;
my $response=$ua->get('http://localhost/email.html');
my $result=$response->content;
#print "$resultnn";
#extracts JavaScript
my $js;
if($result=~s/.*?http://www.jottings.com/obfuscator/s*{/{/s){
$js=extract_codeblock($result,$delim);
}
#modified JS to make it processable by V8 module
$js=~s/document.write/write/;
$js=~s/'/'/g;
#print "$jsnn";
#processes JS
my $context = JavaScript::V8::Context->new();
$context->bind_function(write => sub { print @_ });
my $mail=$context->eval("$js");
print "$mailnn";

24.  cfrenz@gmail.com
 http://www.linkedin.com/in/christopherfrenz/