Flyover Beenleigh Qld

Plagger – RSS/Atom remixing platform Tatsuhiko Miyagawa miyagawa@gmail.com Six Apart, Ltd. / Shibuya Perl Mongers YAPC::NA 2006 Chicago

What is Plagger? Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Pluggable RSS/Atom Aggregator Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Platform for Aggregation / remixing Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Whatever Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Speaking of RSS/Atom aggregator Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Who here is using Bloglines? (or any other web-based aggregators) Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Who here thinks that it sucks? Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Plagger is for you. see Bloglines2gmail Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Who here has ever written a tool using XML::RSS? Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Welcome aboard. There's a chance that you can transform your script into a Plagger plugin. And I can give you a svn commit bit! Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Why Pluggable? Just for a feed aggregation? Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

2002 Apr. baseball2rss http://search.cpan.org/dist/WWW-Baseball-NPB/ Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

2003 Oct. rss2javascript http://blog.bulknews.net/cookbook/blosxom/rss/rss2js.html Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

2004 Sep. bloglines2ipod http://bulknews.net/lib/utils/bloglines2ipod/ Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

2004 Oct. rss2audiobook http://bulknews.net/lib/utils/rss2audiobook/ Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

2005 Aug. bloglines2gmail http://svn.bulknews.net/repos/public/bloglines2email/trunk/ Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Looks like It's not only me doing these things. Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

rss2opml http://aruntx.com/software/rss2opml/ Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

rss2pdf http://rss2pdf.com/ Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

rss2atom brian.wanamaker.com/mybicycle/2004/02/rss2atom.html Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

atom2rss http://www.2rss.com/software.php?page=atom2rss Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

rss2ical http://bura-bura.com/blog/archives/2004/06/22/rss2ical/ Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Bloglines2opml http://mycvs.org/wp/wp-content/wp-transform.php Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

rss2gmail http://www.cs.utexas.edu/~karu/gmailrss/ Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

rss2imap http://rss2imap.sourceforge.jp/ Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

ebay2rss http://www.2rss.com/software.php?page=ebay2rss Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

svn2rss http://twiki.org/cgi-bin/view/Codev/Svn2rss Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

<anything>2<anything> Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Being sick of writing the same code again and again Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Why not creating A pluggable platform Instead? Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

With reusable Parsers / Emitters / Filters? Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

That's what Plagger is. Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Subscription Filter Publish Bloglines StripRSSAd Gmail TruePermalink Delicious Config EntryFullText Pipe PDF OPML, XOXO MT Thumbnail File, DBI, FOAF … FindEnclosures Feed FetchEnclosure Planet SpamAssassin RSSLiberalDateTime Speech URLBL … CustomFeed ResolveRelativeLink … Mixi, Yahoo360JP Notify POP3, iCal iTunes, Amazon IRC, Eject, Growl YouTube … MSAgent, SSTP … Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Just like Lego™ Block Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Create an app With combo of Plugins! Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Example App #1 Bloglines to Gmail Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

bloglines2gmail.yaml plugins: - module: Subscription::Bloglines config: username: you@example.com password: foobar mark_read: 1 - module: Publish::Gmail config: mailto: example@gmail.com mailfrom: miyagawa@example.com mailroute: via: smtp host: smtp.example.com Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Run it on crontab % ./plagger –c bloglines2gmail.yaml Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

RSS in Gmail Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

HTML + Images Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Feed Image (Logo / Buddy Icon) Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Search Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Auto grouping (“Conversations”) Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Diff Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Tips: Filter Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Tips: is:unread Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Example App #2 RSS to ircbot Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

RSS bot in action #plagger on freenode Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Config for RSS bot (1/3) plugins: - module: Subscription::Config config: feed: # Trac's feed for changesets - http://plagger.org/…/rss Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Config for RSS bot (2/3) # I don't like to be notified of same items # more than once! - module: Filter::Rule rule: module: Fresh mtime: path: /tmp/rssbot.time autoupdate: 1 Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Config for RSS bot (3/3) - module: Notify::IRC config: daemon_port: 9999 nickname: plaggerbot server_host: chat.freenode.net server_channels: - #plagger-ja - #plagger Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

See more in examples/irc.yaml Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Example App #3 Planet Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

http://planet.yapcchicago.org/ Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

planet-yapcna.yaml (1/4) plugins: - module: Subscription::Config config: feed: - http://yapcchicago.org/feed/ - http://use.perl.org/search.pl?query=YAPC… - http://del.icio.us/rss/tag/yapcna2006 - http://feeds.technorati.com/feed/posts/… - http://bloglines.com/search?q=YAPC+NA&… # etc, etc … Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

planet-yapcna.yaml (2/4) # Normalize feed title and permalinks - module: Filter::FeedBurnerPermalink - module: Filter::TruePermalink - module: Filter::StripTagsFromTitle Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

planet-yapcna.yaml (3/4) # Create a smartfeed for all the entries merged - module: SmartFeed::All rule_op: AND rule: - module: Fresh duration: 10080 # seven days - module: URLBL dnsbl: rbl.bulkfeeds.jp config: title: Planet YAPC::NA Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

planet-yapcna.yaml (4/4) # Generate nice XHTML out of the SmartFeed - module: Publish::Planet rule: expression: $args->{feed}->id eq 'smartfeed:all' config: dir: /path/to/htdocs skin: sixapart-std template: members_list: 1 style_url: http://example.com/style.css Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

(I admit this Planet config is so clumsy and I'll work on that to make it suck less.) Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Example App #4 YouTube downloader Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

youtube.yaml plugins: - module: Subscription::Config config: feed: - http://www.youtube.com/rss/tag/yapc.rss # discover real .flv URLs on YouTube.com - module: Filter::FindEnclosures # fetch them to local directory - module: Filter::FertchEnclosure config: dir: path/to/save Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Coming soon … Filter::ffmpeg, Sync::PSP, Sync::iPodVideo Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Plagger Core features Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

RSS/Atom Auto-Discovery Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Support various Feed formats RSS 0.91 to Atom 1.0 Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Support parsing Broken XML feeds (XML::Liberal) Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Podcast / Videocast Support (RSS 2.0 & Atom 1.0) Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Photocast Media RSS iTunes RSS* Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Sane I18N impl. Unicode & Timezone Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Access to browser's Cookies IE, Safari, Firefox and w3m Thanks to brian d foy Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Quick tour On available plugins Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Plugin phases (types) Subscription Aggregator CustomFeed Filter Publish Notify Search Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Subscription load subscriptions (list the feeds/URLs to aggregate) Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Subscription::Config - module: Subscription::Config config: feed: - http://www.yapcchicago.org/feed/ - http://tokyo.yapcasia.org/blog/ Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Subscription::OPML - module: Subscription::OPML config: url: http://www.example.com/subs.opml # subs.opml <opml> <outline xmlUrl=quot;http://www.yapcchicago.org/feed/quot; /> <outline htmlUrl=quot;http://tokyo.yapcasia.org/blog/quot; /> </opml> Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Subscription::File - module: Subscription::File config: url: file:///path/to/subscription.txt % cat subscription.txt http://www.yapcchicago.org/feed/ http://tokyo.yapcasia.org/blog/ % Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Subscription::XOXO - module: Subscription::XOXO config: url: http://www.example.com/subscription.html # subscription.html <ul class=quot;xoxoquot;> <li><a href=quot;http://www.yapcchicago.org/feed/quot;>YAPC::NA</a></li> <li><a href=quot;http://tokyo.yapcasia.org/blog/quot;>YAPC::NA</a></li> </ul> Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Subscription::Bookmarks Read bookmarks file of IE, Firefox and Safari Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Aggregator Aggregate and parse the feeds listed in subscription(s) Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Aggregator::Simple The quot;dumbquot; aggregator Using LWP and XML::Feed Might be okay < 20 feeds Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Aggregator::Xango The quot;fastquot; aggregator using Xango.pm the POE based scalable web crawler For > 100 feeds Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

CustomFeed Feed formats other than RSS/Atom Scrapers Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

CustomFeed::POP3 Each email is a feed. Attachments are enclosures. Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

CustomFeed::MySpace* Your friends journal as feed (* indicates it's not developed yet) Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

CustomFeed::FlickrSearch Search results as feed Each photo found is an entry (with enclosures). Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Filter Normalize / Repair feed metadata Upgrade feed content Filter feed content using text filters Invoke some action on entries Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Filter::StripRSSAd Supports: Google AdSense, FeedBurner, Pheedo Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Filter::EntryFullText Upgrade content-less feed to fulltext feed by fetching individual HTML and extracting the content body Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Filter::TruePermalink Resolves nasty redirection URL to the quot;truequot; permalink (e.g. http://…/go.php?url=….) Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Filter::FindEnclosures Find enclosures from content body <a href=quot;http://…./foo.mp3quot;>episode #1</a> Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Filter::RSSLiberalDateTime Deal with broken rss datetime format <pubDate>2006/06/27 01:45:22 +0900</pubDate> Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Publish Publish aggregated entry to online services Convert feeds to other formats Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Publish::Feed Republish feed in RSS/Atom Good to use with scrapers Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Publish::Delicious Auto-post entries to your del.icio.us using its REST API Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Publish::iCal* Publish iCal feeds out of RSS/Atom Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Publish::MTWidget Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Publish::Excel If your boss is unhappy your reading blogs on browsers. Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Search Index aggregated entries on search engines Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Search::Spotlight Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Search::Estraier Uses HyperEstraier XMLRPC node API Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Notify Notify feed updates in various ways Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Notify::Campfire Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Notify::Growl Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Notify::MSAgent Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Notify::Eject Supports: Windows, Linux, FreeBSD and Mac OSX! Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

So far, Plagger rocks  Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Actually, Plagger sucks  Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

No good Documentation (Not a big deal if you can read Perl code) Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Horrible lots of CPAN deps. % grep requires Makefile.PL | wc –l 25 % grep recommends Makefile.PL | wc –l 70 Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

cpan Plagger Doesn't work (partially) Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

144 open tickets On Trac http://plagger.org/trac/query (Not a bad sign. I use it as a Wishlist) Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

The way it de-dupes entries is clumsy Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

No database Backend (yet) Planned to be in core of 0.8 Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Plugin invocations Can be rule-based but undocumented Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Different Plugin functionalities On the same namespaces (CustomFeed, Filter, Publish) Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Publish::Gmail was badly named Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

I want you To fix & improve it. Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Plagger dev. Status Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Version 0.7.3 Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Coming Soon … Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

iTunes RSS support Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Enclosure processors ffmpeg, Sync::PSP, Sync::iPodVideo Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Rich Media metadata ID3 tag in enclosures Links to imdb.com / amazon.com hReview microformats Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Database Storage & Server API branches/plagger-server Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Calendar Support iCal parser & emitter hCalendar microformats .ics attached in emails Sync::SyncML Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

How's the dev going on? Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

31 authors 128 plugins (most of them are from Japan) Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Buzz in Japan Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

I am Happy With quot;the Buzzquot; Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

I am !Happy With quot;In Japanquot; Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Help spreading the words Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Doc Contribution Is highly welcome Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

http://plagger.org/ Planet, Mailing List, IRC Bug Tracking, SVN repository Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

#plagger on freenode Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Join Us! Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Thank you Questions? Tatsuhiko Miyagawa 2006/06/27 YAPC::NA 2006

Flyover Beenleigh Qld

by Property News

Accessibility

Categories

Upload Details

Usage Rights

Statistics

Flyover Beenleigh Qld Document Transcript