Web::Scraper

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Practical Web Scraping with Web::Scraper Tatsuhiko Miyagawa [email_address] Six Apart, Ltd. / Shibuya Perl Mongers YAPC:...

<ul><li>Tatsuhiko Miyagawa </li></ul>

<ul><li>CPAN: MIYAGAWA </li></ul>

abbreviation Acme::Module::Authors Acme::Sneeze Acme::Sneeze::JP Apache::ACEProxy Apache::AntiSpam Apache::Clickable Apach...

http://code.sixapart.com/

<ul><li>Practical Web Scraping </li></ul><ul><li>with Web::Scraper </li></ul>

Web pages are built using text-based mark-up languages ( HTML and XHTML ), and frequently contain a wealth of useful da...

Web pages are built using text-based mark-up languages ( HTML and XHTML ), and frequently contain a wealth of useful da...

<ul><li>"Screen-scraping </li></ul><ul><li>is so 1999!" </li></ul>

<ul><li>RSS is a metadata </li></ul><ul><li>not a complete </li></ul><ul><li>HTML replacement </li></ul>

<ul><li>Practical Web Scraping </li></ul><ul><li>with Web::Scraper </li></ul>

<ul><li>What's wrong with </li></ul><ul><li>LWP & Regexp? </li></ul>

<td>Current <strong>UTC</strong> (or GMT/Zulu)-time used: <strong id="ctu">Monday, August 27, 2007 at 12:49:46<...

<td>Current <strong>UTC</strong> (or GMT/Zulu)-time used: <strong id="ctu">Monday, August 27, 2007 at 12:49:46<...

<ul><li>It works! </li></ul>

WWW::MySpace 0.70

WWW::Search::Ebay 2.231

WWW::Mixi 0.50

<ul><li>It works … </li></ul>

<ul><li>There are </li></ul><ul><li>3 problems </li></ul><ul><li>(at least) </li></ul>

<ul><li>(1) </li></ul><ul><li>Fragile </li></ul><ul><li>Easy to break even with slight HTML changes </li></ul><ul><li>(lik...

<ul><li>(2) </li></ul><ul><li>Hard to maintain </li></ul><ul><li>Regular expression based scrapers are good </li></ul><ul...

<ul><li>(3) </li></ul><ul><li>Improper </li></ul><ul><li>HTML & encoding </li></ul><ul><li>handling </li></ul>

<span class="message">I &hearts; Vienna</span> > perl –e '$c =~ m@<span class="message">(.*?)</span>@ ...

<span class="message">I &hearts; Vienna</span> > perl –MHTML::Entities –e '$c =~ m@<span class="message&q...

<span class="message"> ウィーンが大好き！ </span> > perl –MHTML::Entities –MEncode –e '$c =~ m@<span class="messa...

<ul><li>The "right" way </li></ul><ul><li>of screen-scraping </li></ul>

<ul><li>(1), (2) </li></ul><ul><li>Maintainable </li></ul><ul><li>Less fragile </li></ul>

<ul><li>Use XPath </li></ul><ul><li>and CSS Selectors </li></ul>

<ul><li>XPath </li></ul><ul><li>HTML::TreeBuilder::XPath </li></ul><ul><li>XML::LibXML </li></ul>

XPath <td>Current <strong>UTC</strong> (or GMT/Zulu)-time used: <strong id="ctu">Monday, August 27, 2007 at 12:...

<ul><li>CSS Selectors </li></ul><ul><li>"XPath for HTML coders" </li></ul><ul><li>"XPath for people who hat...

CSS Selectors <ul><li>body { font-size: 12px; } </li></ul><ul><li>div.article { padding: 1em } </li></ul><ul><li>span#coun...

<ul><li>XPath: </li></ul><ul><li>//strong[@id="ctu"] </li></ul><ul><li>CSS Selector: </li></ul><ul><li>strong#...

CSS Selectors <td>Current <strong>UTC</strong> (or GMT/Zulu)-time used: <strong id="ctu">Monday, August 27, 200...

Complete Script #!/usr/bin/perl use strict; use warnings; use Encode; use LWP::UserAgent; use HTTP::Response::Encoding; us...

<ul><li>Robust, </li></ul><ul><li>Maintainable, </li></ul><ul><li>and </li></ul><ul><li>Sane character handling </li></ul>

Exmaple (before) <td>Current <strong>UTC</strong> (or GMT/Zulu)-time used: <strong id="ctu">Monday, August 27, ...

Example (after) #!/usr/bin/perl use strict; use warnings; use Encode; use LWP::UserAgent; use HTTP::Response::Encoding; us...

<ul><li>but … </li></ul><ul><li>long and boring </li></ul>

<ul><li>Practical Web Scraping </li></ul><ul><li>with Web::Scraper </li></ul>

<ul><li>Web scraping toolkit </li></ul><ul><li>inspired by scrapi.rb </li></ul><ul><li>DSL-ish </li></ul>

Example (before) #!/usr/bin/perl use strict; use warnings; use Encode; use LWP::UserAgent; use HTTP::Response::Encoding; u...

Example (after) <ul><li>#!/usr/bin/perl </li></ul><ul><li>use strict; </li></ul><ul><li>use warnings; </li></ul><ul><li>us...

Basics <ul><li>use Web::Scraper; </li></ul><ul><li>my $s = scraper { </li></ul><ul><li># DSL goes here </li></ul><ul><li>}...

process <ul><li>process $selector, </li></ul><ul><li>$key => $what, </li></ul><ul><li>… ; </li></ul>

<ul><li>$selector: </li></ul><ul><li>CSS Selector </li></ul><ul><li>or </li></ul><ul><li>XPath (start with /) </li></ul>

<ul><li>$key: </li></ul><ul><li>key for the result hash </li></ul><ul><li>append "[]" for looping </li></ul>

<ul><li>$what: </li></ul><ul><li>'@attr' </li></ul><ul><li>'TEXT' </li></ul><ul><li>Web::Scraper </li></ul><ul><li>sub { …...

<ul class="sites"> <li><a href="http://vienna.openguides.org/">OpenGuides</a></li> <li><a href="h...

<ul><li>process "ul.sites > li > a", </li></ul><ul><li>'urls[]' => ' @href '; </li></ul><ul><li># { urls => [ ...

<ul><li>process '//ul[@class="sites"]/li/a', </li></ul><ul><li>'names[]' => ' TEXT '; </li></ul><ul><li># { nam...

<ul><li>process "ul.sites > li", </li></ul><ul><li>'sites[]' => scraper { </li></ul><ul><li>process 'a', </li><...

$<ul><li>process "ul.sites > li > a", </li></ul><ul><li>'sites[]' => sub { </li></ul><ul><li># $_ is HTML::Eleme...$

<ul><li>process "ul.sites > li > a", </li></ul><ul><li>'sites[]' => { </li></ul><ul><li>link => '@href', name =...

result <ul><li>result; </li></ul><ul><li># get stash as hashref (default) </li></ul><ul><li>result @keys; </li></ul><ul><...

<ul><li>More Examples </li></ul>

Thumbnail URLs on Flickr set <ul><li>#!/usr/bin/perl </li></ul><ul><li>use strict; </li></ul><ul><li>use Data::Dumper; </l...

<span class="vcard"> <a href="http://twitter.com/iamcal" class="url" rel="contact"...

Twitter Friends <ul><li>#!/usr/bin/perl </li></ul><ul><li>use strict; </li></ul><ul><li>use Web::Scraper; </li></ul><ul><l...

Twitter Friends (complex) <ul><li>#!/usr/bin/perl </li></ul><ul><li>use strict; </li></ul><ul><li>use Web::Scraper; </li><...

<ul><li>Tools </li></ul>

<ul><li>> cpan Web::Scraper </li></ul><ul><li>comes with 'scraper' CLI </li></ul>

<ul><li>> scraper http://example.com/ </li></ul><ul><li>scraper> process "a", "links[]" => '@href'; ...

<ul><li>> scraper /path/to/foo.html </li></ul><ul><li>> GET http://example.com/ | scraper </li></ul>

<ul><li>TODO </li></ul>

<ul><li>Web::Scraper </li></ul><ul><li>Needs documentation </li></ul>

<ul><li>More examples </li></ul><ul><li>to put in eg/ directory </li></ul>

<ul><li>integrate with </li></ul><ul><li>WWW::Mechanize </li></ul><ul><li>and Test::WWW::Declare </li></ul>

<ul><li>XPath Auto-suggestion </li></ul><ul><li>off of DOM + element </li></ul><ul><li>DOM + XPath => Element </li></ul><u...

<ul><li>Questions? </li></ul>

<ul><li>Thank you </li></ul><ul><li>http://search.cpan.org/dist/Web-Scraper </li></ul><ul><li>http://www.slideshare.net/mi...

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

Almost Scraping: Web Scraping witho... by Michelle Minkoff 9859 views
Scraper by Veerendra Kumar 8508 views
Web Scraping and Its Business Benefits by Eminenture 588 views
Web Scraper Shibuya.pm tech talk #8 by Tatsuhiko Miyagawa 17131 views
Relevance Assessment Tool by Dirk Lewandowski 818 views
When RSS Fails: Web Scraping with HTTP by Matthew Turland 8276 views

View on Slideshare

1 of 79 Ad

View on Slideshare

1 of 79 Ad