Plagger the duct tape of internet

Plagger the duct tape of the Web Tatsuhiko Miyagawa [email_address] Six Apart, Ltd. / Shibuya Perl Mongers XML Developers' Day #9

IRC

#plagger-ja

chat.freenode.net

(iso-2022-jp)

アウェイっぽいので

自己紹介

Tatsuhiko Miyagawa

http://www.vox.com/

What is Plagger?

Pl uggable

RSS/Atom

Agg regato r

Why Pluggable?

Just for a feed aggregation?

History

2002 Apr.

baseball2rss

http://search.cpan.org/dist/WWW-Baseball-NPB/

2003 Oct.

rss2javascript

http://blog.bulknews.net/cookbook/blosxom/rss/rss2js.html

2004 Sep.

bloglines2ipod

http://bulknews.net/lib/utils/bloglines2ipod/

2004 Oct.

rss2audiobook

http://bulknews.net/lib/utils/rss2audiobook/

2005 Aug.

bloglines2gmail

http://svn.bulknews.net/repos/public/bloglines2email/trunk/

#!/usr/bin/perl use strict; use warnings; use LWP::UserAgent; use XML::RSS; my $url = "http://example.com/rss.xml"; my $agent = LWP::UserAgent->new; my $xml = $agent->get($url)->content; my $rss = XML::RSS->new; $rss->parse($xml); for my $item (@{$rss->items}) { # do something with $item }

"HTTP リクエストが

エラーになったら ?"

#!/usr/bin/perl use strict; use warnings; use LWP::UserAgent; use XML::RSS; my $url = "http://example.com/rss.xml"; my $agent = LWP::UserAgent->new; my $res = $agent->get($url); if ($res->is_error) { die "Bah." } my $xml = $res->content; my $rss = XML::RSS->new; $rss->parse($xml); for my $item (@{$rss->items}) { # do something with $item }

"Atom も読みたい !"

#!/usr/bin/perl use strict; use warnings; use LWP::UserAgent; use XML::RSS; use XML::Atom::Feed; my $url = "http://example.com/rss.xml"; my $agent = LWP::UserAgent->new; my $res = $agent->get($url); if ($res->is_error) { die "Bah." } my $xml = $res->content; if ($res->content_type =~ /atom/) { my $feed = XML::Atom::Feed->new($xml); } else { my $rss = XML::RSS->new; $rss->parse($xml); }

If-Modified-Since で

帯域節約したい！

#!/usr/bin/perl use strict; use warnings; use LWP::UserAgent; use XML::RSS; my $url = "http://example.com/rss.xml"; my $agent = LWP::UserAgent->new; my $local = cache_path_for($url); my $res = $agent->mirror($url, $local); if ($res->is_error) { die "Bah." } my $xml = $res->content; …

" 壊れたフィードも

パースしたい！ "

Etc., etc.

他にも

rss2opml

http://aruntx.com/software/rss2opml/

rss2pdf

http://rss2pdf.com/

rss2atom

brian.wanamaker.com/mybicycle/2004/02/rss2atom.html

atom2rss

http://www.2rss.com/software.php?page=atom2rss

rss2ical

http://bura-bura.com/blog/archives/2004/06/22/rss2ical/

Bloglines2opml

http://mycvs.org/wp/wp-content/wp-transform.php

rss2gmail

http://www.cs.utexas.edu/~karu/gmailrss/

rss2imap

http://rss2imap.sourceforge.jp/

ebay2rss

http://www.2rss.com/software.php?page=ebay2rss

svn2rss

http://twiki.org/cgi-bin/view/Codev/Svn2rss

<any>2<any>

Where either of <any> is RSS|Atom|OPML

This is ridiculous.

Different Languages,

Different Bugs.

No hackability

via http://www.atmarkit.co.jp/fnetwork/rensai/5minplagger/02.html

IRC, Eject, Growl MSAgent, SSTP … Filter Publish StripRSSAd TruePermalink EntryFullText Pipe Thumbnail FindEnclosures FetchEnclosure SpamAssassin RSSLiberalDateTime URLBL ResolveRelativeLink … Gmail Delicious PDF MT Feed Planet Speech … Notify Bloglines Config OPML, XOXO File, DBI, FOAF … Mixi, Frepa POP3, iCal iTunes, Amazon YouTube … Subscription CustomFeed

IRC, Eject, Growl MSAgent, SSTP … Filter Publish StripRSSAd TruePermalink EntryFullText Pipe Thumbnail FindEnclosures FetchEnclosure SpamAssassin RSSLiberalDateTime URLBL ResolveRelativeLink … Gmail Delicious PDF MT Feed Planet Speech … Notify Bloglines Config OPML , XOXO File, DBI, FOAF … Mixi, Frepa POP3, iCal iTunes, Amazon YouTube … Subscription CustomFeed

Just like UNIX pipe

Subscribe OPML | StripRSSAd | ResolveRelativeLink | Publish Feed --type=Atom

I believe RSS has the potential to be the “UNIX pipe of the internet” … Ray Ozzie CTO of Microsoft http://rayozzie.spaces.live.com/blog/cns!FB3017FBB9B2E142!285.entry

"the Unix shell for Web 2.0"

組み合わせの数

{CustomFeed,Subscription}/*.pm: 35

{Publish,Notify}/*.pm: 37

35 * 37 = 1295

Plagger

Core features

RSS/Atom

Auto-Discovery

Support Feed formats

RSS 0.91 to Atom 1.0

(XML::Feed + hacks)

Support parsing

Broken feeds

(XML::Liberal)

HTTP optimizations

If-Modified-Since / gzip

(URI::Fetch)

Podcast / Videocast

Support

(RSS 2.0 & Atom 1.0)

Photocast

Media RSS

iTunes RSS*

非同期ダウンロード

cURL, wget, HTTP::Parallel & HTTP::Async*

完全な国際化

Unicode & Timezone

Access to

browser Cookies

IE, Safari, Firefox and w3m

Screen-scraping

Via CutomFeed::*

Stackable Plugins

Rule-based

Dispatch of Plugins

Plagger の

インストール

省略

http://plagger.org/trac/wiki/PlaggerQuickStart

Plagger

クイックチュートリアル

Email クライアントで

RSS フィードを読みたい！

(Gmail, Thunderbird)

rss2email.yaml plugins: - module: Subscription::Config config: feed: - http://bulknews.vox.com/library/posts/atom.xml - http://bulknews.typepad.com/blog/ - module: Publish::Gmail config: mailto: miyagawa@gmail.com

RSS in Gmail

HTML + Images

Feed Image (Logo / Buddy Icon)

Search

Auto grouping (“Conversations”)

Diff

オフライン（飛行機）でも

フィードを読みたい！

rss2email.yaml plugins: - module: Subscription::Config config: feed: - http://bulknews.vox.com/library/posts/atom.xml - http://bulknews.typepad.com/blog/ - module: Filter::FindEnclosures - module: Filter::FetchEnclosure config: dir: /tmp - module: Publish::Gmail config: mailto: miyagawa@gmail.com attach_enclosures: 1

Offline Mode POP3 + Thunderbird

" フィードのリストを

YAML で管理するのメンドウ "

opml2email.yaml plugins: - module: Subscription::OPML config: url: http://example.com/subscription.opml - module: Publish::Gmail config: mailto: miyagawa@gmail.com # subscription.opml <?xml version="1.0"?> <opml> <outline title="Subscriptions"> <outline title="miyagawa" type="rss" xmlUrl="http://bulknews.typepad.com/blog/atom.xml" /> <outline title="miyagawa on Vox" type="rss" htmlUrl="http://bulknews.vox.com/" /> </outline> </opml>

"OPML を手動で編集するのはテラダルス "

filesub2email.yaml plugins: - module: Subscription::File config: url: file:///path/to/subscription.txt - module: Publish::Gmail config: mailto: miyagawa@gmail.com > cat subscription.txt http://bulknews.typepad.com/blog/atom.xml http://bulknews.vox.com/ >

"1000 フィード以上読んでると自分のマシンで読むのは帯域のムダ。 Bloglines にクロールさせたい。 "

bloglines2email.yaml plugins: - module: Subscription::Bloglines config: username: YOU@example.com password: blahblahblah - module: Publish::Gmail config: mailto: miyagawa@gmail.com

" それ Livedoor Reader で "

bloglines2email.yaml plugins: - module: Subscription::LivedoorReader config: username: YOU@example.com password: blahblahblah - module: Publish::Gmail config: mailto: miyagawa@gmail.com

"Gmail / Thunderbird から

del.icio.us にブクマしたい !"

bloglines2email.yaml plugins: - module: Subscription::Bloglines config: username: YOU@example.com password: blahblahblah - module: Widget::Simple config: widget: delicious - module: Publish::Gmail config: mailto: miyagawa@gmail.com

" それはてなブックマークで "

bloglines2email.yaml plugins: - module: Subscription::Bloglines config: username: YOU@example.com password: blahblahblah - module: Widget::Simple config: widget: hatena_bookmark_users - module: Publish::Gmail config: mailto: miyagawa@gmail.com

"RSS 広告ウザス "

bloglines2email.yaml plugins: - module: Subscription::Bloglines config: username: YOU@example.com password: blahblahblah - module: Widget::Simple config: widget: delicious - module: Filter::StripRSSAd - module: Publish::Gmail config: mailto: miyagawa@gmail.com

Quick tour

For more plugins

Plugin phases (types)

Subscription

Aggregator

CustomFeed

Filter

Publish

Search

Notify

Subscription

Aggregator

CustomFeed

Filter

Publish

Search

Notify

Subscription

load subscriptions

(list the feeds/URLs to aggregate)

Subscription::Config

- module: Subscription::Config config: feed: - http://www.yapcchicago.org/feed/ - http://tokyo.yapcasia.org/blog/

Subscription::OPML

- module: Subscription::OPML config: url: http://www.example.com/subs.opml # subs.opml <opml> <outline xmlUrl="http://www.yapcchicago.org/feed/" /> <outline htmlUrl="http://tokyo.yapcasia.org/blog/" /> </opml>

Subscription::File

- module: Subscription::File config: url: file:///path/to/subscription.txt % cat subscription.txt http://www.yapcchicago.org/feed/ http://tokyo.yapcasia.org/blog/ %

Subscription::XOXO

- module: Subscription::XOXO config: url: http://www.example.com/subscription.html # subscription.html <ul class="xoxo"> <li><a href="http://www.yapcchicago.org/feed/">YAPC::NA</a></li> <li><a href="http://tokyo.yapcasia.org/blog/">YAPC::NA</a></li> </ul>

Subscription::Bookmarks

Read bookmarks file of IE, Firefox and Safari

Subscription

Aggregator

CustomFeed

Filter

Publish

Search

Notify

Filter

Normalize / Repair feed metadata

Upgrade feed content

Filter feed content using text filters

Invoke some action on entries

Filter::EntryFullText

本文なしのフィードをアップグレード

個別 HTML を取得して正規表現 / XPath

Filter::TruePermalink

リダイレクト URL などを Canonicalize

(e.g. http://…/go.php?url=….)

Filter::FindEnclosures

コンテンツからエンクロージャを抽出

<a href="http://…./foo.mp3">episode #1</a>

Subscription

Aggregator

CustomFeed

Filter

Publish

Search

Notify

Publish

Publish aggregated entry to online services

reBlogging

Convert feeds to other formats

Publish::Feed

Republish feed in RSS/Atom

Good to use with scrapers

Publish::MT

Reblog entries using MT XML-RPC

Publish::MTWidget

Publish::Email

text/plain, multipart/alternative

Pluggable email protocols

(SMTP, SMTP Auth, IMAP, Maildir …)

Publish::iCal

Publish iCal feeds out of RSS/Atom

Publish::Excel

仕事の合間に !

Subscription

Aggregator

CustomFeed

Filter

Publish

Search

Notify

Search

Index aggregated entries on search engines

Search::Spotlight

Search::Estraier

Uses HyperEstraier XMLRPC node API

Search::Lucene*

Use Lucene WebService API

(OpenSearch 1.1 and Atom)

Subscription

Aggregator

CustomFeed

Filter

Publish

Search

Notify

Notify

Notify feed updates in various ways

Notify::Growl

Notify::Balloon

Notify::MSAgent

Notify::Eject Supports: Windows, Linux, FreeBSD and Mac OSX!

Notify::Pizza

Notify::Pizza

Now it does Sushi too!

Plagger に

対する誤解

http://d.hatena.ne.jp/sugarcut/20061117/p1 それ Pla 脳それ Plagger でできるよそれプラズマで説明できるよ

何でもできる

… わけじゃない

Plagger is a

"Feed" aggregator

Plagger::Subscription Plagger::Feed author id link title tags url entries Plagger::Feed author id link title tags url entries Plagger::Feed author id link title tags url entries Plagger::Entry author id link permalink title tags enclosures

ピザ Pla の意義

"the Unix shell for Web 2.0"

"RSS is the Standard IO for the Web"

Feed formats RSS 0.91 RSS 2.0 RSS1.0 / RDF Atom 1.0 JSON iCal OPML XOXO XBEL Sitemaps attention.xml Amazon API Google API OpenSearch AtomPP GData

Feed Vocabulary / Extensions rvw: Enclosures Photocast iTunes RSS Media RSS Dublin Core FOAF microformats

Plagger = The duct tape of the web.

Example:

Location metadata

N 37.7782 W 122.3973

GeoRSS <georss:point>37.7782 -122.3973</georss:point> <georss:where> <gml:Point> <gml:pos>37.7782 -122.3973</gml:pos> </gml:Point> </georss:where> xmlns:georss="http://www.georss.org/georss" xmlns:gml="http://www.opengis.net/gml"

RDF geo vocabulary <foaf:based_near> <geo:Point> <geo:lat>35.678</geo:lat> <geo:long>139.770</geo:long> </geo:Point> </foaf:based_near> xmlns:geo="http://www.w3.org/2003/01/geo/wgs84_pos#"

Flickr geo tag <media:category scheme="urn:flickr:tags"> geo:lat=37.7782 geo:lon=-122.3973 </media:category> xmlns:media="http://search.yahoo.com/mrss/"

GeoURL <meta name="ICBM" content="37.7782, -122.3973" />

geo microformats <div class="geo"> <span class="latitude">37.7782</span> <span class="longitude">-122.3973</span> </div>

Links to Google Maps <a href="http://maps.google.com/maps?q=37.7782,-122.3973&z=16"> Link to Google Maps</a>

Eznavi mail

Photo EXIF

adr <div class="adr"> <div class="street-address">548 4th St.</div> <span class="locality">San Francisco</span>, <span class="region">CA</span> <span class="postal-code">94107</span> <div class="country-name">U.S.A.</div> </div>

We don't care

about format diffs.

GeoRSS <georss:point>37.7782 -122.3973</georss:point> <georss:where> <gml:Point> <gml:pos>37.7782 -122.3973</gml:pos> </gml:Point> </georss:where> Namespace::GeoRSS

RDF geo vocabulary <foaf:based_near> <geo:Point> <geo:lat>35.678</geo:lat> <geo:long>139.770</geo:long> </geo:Point> </foaf:based_near> Namespace::Geo

Flickr geo tag <media:category scheme="urn:flickr:tags"> geo:lat=37.7782 geo:lon=-122.3973 </media:category> Filter::geotagged

GeoURL <meta name="ICBM" content="37.7782, -122.3973" /> Filter::GeoURL

geo microformats <div class="geo"> <span class="latitude">37.7782</span> <span class="longitude">-122.3973</span> </div> Filter::Microformats::geo

Links to Google Maps <a href="http://maps.google.com/maps?q=37.7782,-122.3973&z=16"> Link to Google Maps</a> Filter::ExtractMapsLinks

Eznavi mail Filter::ExtractMapsLinks

Photo EXIF Filter::FetchEnclosure + Filter::ExtractEXIF

adr <div class="adr"> <div class="street-address">548 4th St.</div> <span class="locality">San Francisco</span>, <span class="region">CA</span> <span class="postal-code">94107</span> <div class="country-name">U.S.A.</div> </div> Filter::Microformats::adr + Filter::Geocoding::US