Web::Scraper for SF.pm LT

Practical Web Scraping with Web::Scraper Tatsuhiko Miyagawa [email_address] Six Apart, Ltd. / Shibuya Perl Mongers SF.pm Lightning Talk

How many of you

have done

screen-scraping w/ Perl?

How many of you

have used

LWP::Simple and regexp?

<td>Current UTC (or GMT/Zulu)-time used: Monday, August 27, 2007 at 12:49:46

<td>Current UTC (or GMT/Zulu)-time used: Monday, August 27, 2007 at 12:49:46 > perl -MLWP::Simple -le '$c = get("http://timeanddate.com/worldclock/"); $c =~ m@(.*?)@ and print $1' Monday, August 27, 2007 at 12:49:46

It works!

WWW::MySpace 0.70

WWW::Search::Ebay 2.231

There are

3 problems

(at least)

(1)

Fragile

Easy to break even with slight HTML changes

(like newlines, order of attributes etc.)

(2)

Hard to maintain

Regular expression based scrapers are good

Only when they're used in write-only scripts

(3)

Improper

HTML & encoding

handling

I &hearts; Shibuya > perl –e '$c =~ m@(.*?)@ and print $1' I &hearts; Shibuya

Web::Scraper

to the rescue

Web scraping toolkit

inspired by scrapi.rb

DSL-ish

Example

#!/usr/bin/perl

use strict;

use warnings;

use Web::Scraper;

use URI;

my $s = scraper {

process "strong#ctu", time => 'TEXT';

result 'time';

};

my $uri = URI->new("http://timeanddate.com/worldclock/");

print $s->scrape($uri);

Basics

use Web::Scraper;

my $s = scraper {

# DSL goes here

};

my $res = $s->scrape($uri);

process

process $selector,

$key => $what,

… ;

$selector:

CSS Selector

or

XPath (start with /)

CSS Selector: strong#ctu

XPath: //strong[@id="ctu"]

<td>Current UTC (or GMT/Zulu)-time used: Monday, August 27, 2007 at 12:49:46

$key:

key for the result hash

append "[]" for looping

$what:

'@attr'

'TEXT'

'RAW'

Web::Scraper

sub { … }

Hash reference

<ul class="sites"> <li><a href="http://vienna.openguides.org/">OpenGuides</a></li> <li><a href="http://vienna.yapceurope.org/">YAPC::Europe</a></li> </ul>

process "ul.sites > li > a",

'urls[]' => ' @href ';

# { urls => [ … ] }

<ul class="sites"> <li><a href=" http://vienna.openguides.org/ ">OpenGuides</a></li> <li><a href=" http://vienna.yapceurope.org/ ">YAPC::Europe</a></li> </ul>

process '//ul[@class="sites"]/li/a',

'names[]' => ' TEXT ';

# { names => [ 'OpenGuides', … ] }

<ul class="sites"> <li><a href="http://vienna.openguides.org/"> OpenGuides </a></li> <li><a href="http://vienna.yapceurope.org/"> YAPC::Europe </a></li> </ul>

process "ul.sites > li > a",

'sites[]' => {

link => '@href', name => 'TEXT';

};

# { sites => [ { link => …, name => … },

# { link => …, name => … } ] };

<ul class="sites"> <li><a href="http://vienna.openguides.org/">OpenGuides</a></li> <li><a href="http://vienna.yapceurope.org/">YAPC::Europe</a></li> </ul>