Web::Scraper for SF.pm LT

Practical Web Scraping with Web::Scraper Tatsuhiko Miyagawa [email_address] Six Apart, Ltd. / Shibuya Perl Mongers SF.pm Lightning Talk

How many of you
have done
screen-scraping w/ Perl?

How many of you
have used
LWP::Simple and regexp?

<td>Current UTC (or GMT/Zulu)-time used: Monday, August 27, 2007 at 12:49:46

<td>Current UTC (or GMT/Zulu)-time used: Monday, August 27, 2007 at 12:49:46 > perl -MLWP::Simple -le '$c = get("http://timeanddate.com/worldclock/"); $c =~ m@(.*?)@ and print $1' Monday, August 27, 2007 at 12:49:46

It works!

WWW::MySpace 0.70

WWW::Search::Ebay 2.231

There are
3 problems
(at least)

(1)
Fragile
Easy to break even with slight HTML changes
(like newlines, order of attributes etc.)

(2)
Hard to maintain
Regular expression based scrapers are good
Only when they're used in write-only scripts

(3)
Improper
HTML & encoding
handling

I &hearts; Shibuya > perl –e '$c =~ m@(.*?)@ and print $1' I &hearts; Shibuya

Web::Scraper
to the rescue

Web scraping toolkit
inspired by scrapi.rb
DSL-ish

Example
#!/usr/bin/perl
use strict;
use warnings;
use Web::Scraper;
use URI;
my $s = scraper {
process "strong#ctu", time => 'TEXT';
result 'time';
};
my $uri = URI->new("http://timeanddate.com/worldclock/");
print $s->scrape($uri);

Basics
use Web::Scraper;
my $s = scraper {
# DSL goes here
};
my $res = $s->scrape($uri);

process
process $selector,
$key => $what,
… ;

$selector:
CSS Selector
or
XPath (start with /)

CSS Selector: strong#ctu
XPath: //strong[@id="ctu"]
<td>Current UTC (or GMT/Zulu)-time used: Monday, August 27, 2007 at 12:49:46

$key:
key for the result hash
append "[]" for looping

$what:
'@attr'
'TEXT'
'RAW'
Web::Scraper
sub { … }
Hash reference

<ul class="sites"> <li><a href="http://vienna.openguides.org/">OpenGuides</a></li> <li><a href="http://vienna.yapceurope.org/">YAPC::Europe</a></li> </ul>

process "ul.sites > li > a",
'urls[]' => ' @href ';
# { urls => [ … ] }
<ul class="sites"> <li><a href=" http://vienna.openguides.org/ ">OpenGuides</a></li> <li><a href=" http://vienna.yapceurope.org/ ">YAPC::Europe</a></li> </ul>

process '//ul[@class="sites"]/li/a',
'names[]' => ' TEXT ';
# { names => [ 'OpenGuides', … ] }
<ul class="sites"> <li><a href="http://vienna.openguides.org/"> OpenGuides </a></li> <li><a href="http://vienna.yapceurope.org/"> YAPC::Europe </a></li> </ul>

process "ul.sites > li > a",
'sites[]' => {
link => '@href', name => 'TEXT';
};
# { sites => [ { link => …, name => … },
# { link => …, name => … } ] };
<ul class="sites"> <li><a href="http://vienna.openguides.org/">OpenGuides</a></li> <li><a href="http://vienna.yapceurope.org/">YAPC::Europe</a></li> </ul>

> cpan Web::Scraper
comes with 'scraper' CLI

> scraper http://example.com/
scraper> process "a", "links[]" => '@href';
scraper> d
$VAR1 = {
links => [
'http://example.org/',
'http://example.net/',
],
};
scraper> y
---
links:
- http://example.org/
- http://example.net/

> scraper /path/to/foo.html
> GET http://example.com/ | scraper

Thank you
http://search.cpan.org/dist/Web-Scraper
http://www.slideshare.net/miyagawa/webscraper

http://coderwall.com	243
http://www.codeordie.org	34
http://www.slideshare.net	26
http://www.hanrss.com	20
file://	3
http://nanotechplastics.com	2
http://nashvilletnrealestate.org	1
http://codeordie.org	1
http://nanotechnologyenergy.org	1
http://geneticexams.com	1
http://localhost:3000	1

Web::Scraper for SF.pm LT

by Tatsuhiko Miyagawa on Nov 27, 2007

Accessibility

Categories

Tags

Upload Details

Usage Rights

11 Embeds 333

Statistics

Web::Scraper for SF.pm LT — Presentation Transcript