Writing Pluggable Software

Writing Pluggable Software Tatsuhiko Miyagawa [email_address] Six Apart, Ltd. / Shibuya Perl Mongers YAPC::Asia 2007 Tokyo

For non-JP attendees …
If you find in the code,
Replace that with backslash.
(This is MS' fault)

Plaggable
Software

Pl u ggable
Software

Agenda

#1
How to make
your app pluggable

#2
TMTOWTDP
There's More Than One Way To Deploy Plugins
Pros/Cons by examples

First-of-all:
Why pluggable?

Benefits

#1
Keep the app design
and code simple

#2
Let the app users
customize the behavior
(without hacking the internals)

#3
It's fun to write plugins
for most hackers
(see: Plagger and Kwiki)

"Can your app do XXX?"
"Yes, by plugins."

"Your app has a bug in YYY"
"No, it's the bug in plugin YYY,
Not my fault."
(Chain Of Responsibilities)

Good Enough
Reasons, huh?

#1
Make your app pluggable

Example

ack (App::Ack)

grep –r for programmers

Ack is a "full-stack"
software now.

By "full-stack" I mean:
Easy install
No configuration
No way to extend

Specifically:
These are hardcoded
Ignored directories
Filenames and types

Ignored Directories
@ignore_dirs = qw( blib CVS RCS SCCS .svn
_darcs .git );

Filenames and languages mapping
%mappings = (
asm => [qw( s S )],
binary => …,
cc => [qw( c h xs )],
cpp => [qw( cpp m h C H )],
csharp => [qw( cs )],
…
perl => [qw( pl pm pod tt ttml t )],
…
);

What if making
these pluggable?

DISCLAIMER

Don't get me wrong Andy,
I love ack the way it is…
Just thought it can be
a very good example for the tutorial.

Quickstart:
Class::Trigger Module::Pluggable
© Six Apart Ltd. Employees

Class::Trigger SYNOPSIS
package Foo;
use Class::Trigger;
sub foo {
my $self = shift;
$self->call_trigger('before_foo');
# some code ...
$self->call_trigger('after_foo');
}
package main;
Foo->add_trigger(before_foo => &sub1);
Foo->add_trigger(after_foo => &sub2);

Class::Trigger
Helps you to implement
Observer Pattern.
(Rails calls this Observer)

Module::Pluggable SYNOPSIS
package MyClass;
use Module::Pluggable;
use MyClass;
my $mc = MyClass->new();
# returns the names of all plugins installed under MyClass::Plugin::*
my @plugins = $mc->plugins();
package MyClass::Plugin::Foo;
sub new { … }
1;

Setup plugins in App::Ack
package App::Ack;
use Class::Trigger;
use Module::Pluggable require => 1;
__PACKAGE__->plugins;

Setup plugins in App::Ack
package App::Ack;
use Class::Trigger;
use Module::Pluggable require => 1 ;
__PACKAGE__->plugins; # "requires" modules

Ignored Directories (Before)
@ignore_dirs = qw( blib CVS RCS SCCS .svn
_darcs .git );

Ignored Directories (After)
# lib/App/Ack.pm
__PACKAGE__->call_trigger(' ignore_dirs.add ', @ignore_dirs);

Ignored Directories (plugins)
# lib/App/Ack/Plugin/IgnorePerlBuildDir.pm
package App::Ack::Plugin::IgnorePerlBuildDir;
App::Ack->add_trigger(
" ignore_dirs.add " => sub {
my($class, $ignore_dirs) = @_;
push @$ignore_dirs, qw( blib );
},
);
1;

Ignored Directories (plugins)
# lib/App/Ack/Plugin/IgnoreSourceControlDir.pm
package App::Ack::Plugin::IgnoreSourcdeControlDir;
" ignore_dirs.add " => sub {
my($class, $ignore_dirs) = @_;
push @$ignore_dirs, qw( CVS RCS .svn
_darcs .git );
},
);
1;

Filenames and languages (before)
%mappings = (
asm => [qw( s S )],
binary => …,
cc => [qw( c h xs )],
cpp => [qw( cpp m h C H )],
csharp => [qw( cs )],
…
perl => [qw( pl pm pod tt ttml t )],
…
);

Filenames and languages (after)
# lib/App/Ack.pm
__PACKAGE__->call_trigger('mappings.add', mappings);

Filenames and languages (plugins)
package App::Ack::Plugin::MappingCFamily;
use strict;
"mappings.add" => sub {
my($class, $mappings) = @_;
$mappings->{asm} = [qw( s S )];
$mappings->{cc} = [qw( c h xs )];
$mappings->{cpp} = [qw( cpp m h C H )];
$mappings->{csharp} = [qw( cs )];
$mappings->{css} = [qw( css )];
},
);
1;

Works great
with few lines of code!

Now it's time to add
Some useful stuff.

Example Plugin:
Content Filter

sub _search {
my $fh = shift;
my $is_binary = shift;
my $filename = shift;
my $regex = shift;
my %opt = @_;
if ($is_binary) {
my $new_fh;
App::Ack->call_trigger('filter.binary', $filename, $new_fh);
if ($new_fh) {
return _search($new_fh, 0, $filename, $regex, @_);
}
}

Example:
Search PDF content
with ack

PDF filter plugin
package App::Ack::Plugin::ExtractContentPDF;
use strict;
use CAM::PDF;
use File::Temp;
'mappings.add' => sub {
my($class, $mappings) = @_;
$mappings->{pdf} = [qw(pdf)];
},
);

PDF filter plugin (cont.)
'filter.binary' => sub {
my($class, $filename, $fh_ref) = @_;
if ($filename =~ /.pdf$/) {
my $fh = File::Temp::tempfile;
my $doc = CAM::PDF->new($file);
my $text;
for my $page (1..$doc->numPages){
$text .= $doc->getPageText($page);
}
print $fh $text;
seek $$fh, 0, 0;
$$fh_ref = $fh;
}
},
);

PDF search
> ack --type=pdf Audrey
yapcasia2007-pugs.pdf:3:Audrey Tang

Homework
Use File::Extract
To handle arbitrary media files

Homework 2:
Search non UTF-8 files
(hint: use Encode::Guess)
You'll need another hook.

Summary
Class::Trigger
+ Module::Pluggable
= Pluggable app easy

#2
TMTOWTDP
There's More Than One Way To Deploy Plugins

Module::Pluggable
+ Class::Trigger
= Simple and Nice
but has limitations

In Reality,
we need more control over how plugins behave

1)
The order of
plugin executions

2)
Per user configurations
for plugins

3)
Temporarily
Disable plugins
Should be easy

4)
How to install
& upgrade plugins

5)
Let plugins
have storage area

Etc, etc.

Examples:
Kwiki
Plagger
qpsmtpd
Movable Type

I won't talk about
Catalyst plugins
(and other framework thingy)

Because they're
NOT
"plug-ins"

Install plugins
And now you write
MORE CODE

95% of Catalyst plugins
Are NOT "plugins"
But "components"
95% of these statistics is made up by the speakers.

Kwiki 1.0

Kwiki Plugin code
package Kwiki::URLBL;
use Kwiki::Plugin -Base ;
use Kwiki::Installer -base;
const class_id => 'urlbl';
const class_title => 'URL Blacklist DNS';
const config_file => 'urlbl.yaml';
sub register {
require URI::Find;
my $registry = shift;
$registry->add(hook => 'edit:save', pre => 'urlbl_hook');
$registry->add(action => 'blacklisted_url');
}

Kwiki Plugin (cont.)
sub urlbl_hook {
my $hook = pop;
my $old_page = $self->hub->pages->new_page($self->pages->current->id);
my $this = $self->hub->urlbl;
my @old_urls = $this->get_urls($old_page->content);
my @urls = $this->get_urls($self->cgi->page_content);
my @new_urls = $this->get_new_urls(@old_urls, @urls);
if (@new_urls && $this->is_blocked(@new_urls)) {
$hook->cancel();
return $self->redirect("action=blacklisted_url");
}
}

Magic implemented
in Spoon(::Hooks)

"Install" Kwiki Plugins
# order doesn't matter here (according to Ingy)
Kwiki::Display
Kwiki::Edit
Kwiki::Theme::Basic
Kwiki::Toolbar
Kwiki::Status
Kwiki::Widgets
# Comment out (or entirely remove) to disable
# Kwiki::UnnecessaryStuff

Kwiki plugin config
# in Kwiki::URLBL plugin
__config/urlbl.yaml__
urlbl_dns: sc.surbl.org, bsb.spamlookup.net, rbl.bulkfeeds.jp
# config.yaml
urlbl_dns: myowndns.example.org

Kwiki plugins are CPAN modules

Install and Upgrade plugins
cpan> install Kwiki::SomeStuff

Using CPAN as a repository
Pros #1:
reuse most of current CPAN infrastructure.

Using CPAN as a repository
Pros #2:
Increasing # of modules
= good motivation
for Perl hackers

Cons #1:
Installing CPAN deps
could be a mess
(especially for Win32)

Cons #2:
Whenever Ingy releases
new Kwiki, lots of plugins just break.

Kwiki plugin storage
return if
grep {$page->id}
@{$self-> config->cached_display_ignore };
my $html = io->catfile(
$self-> plugin_directory ,$page->id
)->utf8;

Kwiki 2.0

Same as Kwiki 1.0

Except:
plugins are now
in SVN repository

Plagger plugin
package Plagger::Plugin::Publish::iCal;
use strict;
use base qw( Plagger::Plugin );
use Data::ICal;
use Data::ICal::Entry::Event;
use DateTime::Duration;
use DateTime::Format::ICal;
sub register {
my($self, $context) = @_;
$context-> register_hook (
$self,
' publish.feed ' => &publish_feed,
' plugin.init ' => &plugin_init,
);
}

Plagger plugin (cont)
sub plugin_init {
my($self, $context) = @_;
my $dir = $self->conf->{dir};
unless (-e $dir && -d _) {
mkdir $dir, 0755
or $context->error("Failed to mkdir $dir: $!");
}
}

Plagger plugin storage
$self->conf->{invindex} ||=
$self-> cache->path_to ('invindex');

Plagger plugin config
# The order matters in config.yaml
# if they're in the same hooks
plugins:
- module: Subscription::Config
config:
feed:
- http://www.example.com/
- module: Filter::DegradeYouTube
config:
dev_id: XYZXYZ
- module: Publish::Gmail
disable: 1

Plugins Install & Upgrade
cpan> notest install Plagger
# or …
> svn co http://…/plagger/trunk plagger
> svn update

Plagger impl.
ripped off by
many apps now

qpsmtpd

mod_perl for SMTP
Runs with tcpserver, forkserver
or Danga::Socket standalone

Plugins: Flat files
rock:/home/miyagawa/svn/qpsmtpd> ls -F plugins
async/ greylisting
auth/ hosts_allow
check_badmailfrom http_config
check_badmailfromto ident/
check_badrcptto logging/
check_badrcptto_patterns milter
check_basicheaders parse_addr_withhelo
check_earlytalker queue/
check_loop quit_fortune
check_norelay rcpt_ok
check_relay relay_only
check_spamhelo require_resolvable_fromhost
content_log rhsbl
count_unrecognized_commands sender_permitted_from
dns_whitelist_soft spamassassin
dnsbl tls
domainkeys tls_cert*
dont_require_anglebrackets virus/

qpsmtpd plugin
sub hook_mail {
my ($self, $transaction, $sender, %param) = @_;
my @badmailfrom = $self->qp->config ("badmailfrom")
or return (DECLINED);
for my $bad (@badmailfrom) {
my $reason = $bad;
$bad =~ s/^s*(S+).*/$1/;
next unless $bad;
$transaction->notes('badmailfrom', $reason) …
}
return (DECLINED);
}

Actually qpsmtpd
Plugins are "compiled"
to modules

my $eval = join(" ",
"package $package;",
'use Qpsmtpd::Constants;',
"require Qpsmtpd::Plugin;",
'use vars qw(@ISA);',
'use strict;',
'@ISA = qw(Qpsmtpd::Plugin);',
($test_mode ? 'use Test::More;' : ''),
"sub plugin_name { qq[$plugin] }",
$line,
$sub,
" ", # last line comment without newline?
);
$eval =~ m/(.*)/s;
$eval = $1;
eval $eval;
die "eval $@" if $@;

qpsmtpd plugin config
rock:/home/miyagawa/svn/qpsmtpd> ls config.sample/
config.sample:
IP logging require_resolvable_fromhost
badhelo loglevel rhsbl_zones
badrcptto_patterns plugins size_threshold
dnsbl_zones rcpthosts tls_before_auth
invalid_resolvable_fromhost relayclients tls_ciphers

config/plugins
# content filters
virus/klez_filter
# rejects mails with a SA score higher than 2
spamassassin reject_threshold 20

config/badhelo
# these domains never uses their domain when greeting us, so reject transactions
aol.com
yahoo.com

Install & Upgrade plugins
Just use subversion

MT plugins are
flat-files
(or scripts that call modules)

MT plugin code
package MT::Plugin::BanASCII;
our $Method = "deny";
use MT;
use MT::Plugin;
my $plugin = MT::Plugin->new({
name => "BanASCII v$VERSION",
description => "Deny or moderate ASCII or Latin-1 comment",
});
MT->add_plugin($plugin);
MT->add_callback('CommentFilter', 2, $plugin, &handler);

MT plugin code (cont)
sub init_app {
my $plugin = shift;
$plugin->SUPER::init_app(@_);
my($app) = @_;
return unless $app->isa('MT::App::CMS');
$app-> add_itemset_action ({
type => 'comment',
key => 'spam_submission_comment',
label => 'Report SPAM Comment(s)',
code => sub {
$plugin->submit_spams_action('MT::Comment', @_) },
}
);

MT plugin storage
require MT::PluginData;
my $data = MT::PluginData->load({
plugin => 'sidebar-manager',
key => $blog_id },
);
unless ($data) {
$data = MT::PluginData->new;
$data->plugin('sidebar-manager');
$data->key($blog_id);
}
$data->data( $modulesets );
$data->save or die $data->errstr;

Order control
MT->add_callback('CMSPostEntrySave', 9 , $rightfields, &CMSPostEntrySave);
MT->add_callback('CMSPreSave_entry', 9 , $rightfields, &CMSPreSave_entry);
MT::Entry->add_callback('pre_remove', 9 , $rightfields, &entry_pre_remove);
Defined in plugins. No Control on users end

Conclusion
Flat-files
vs.
Modules

Flat-files:
☺ Easy to install (Just grab it)
☻ Hard to upgrade
OK for simple plugins

Modules:
☺ Full-access to Perl OO goodness
☺ Avoid duplicate efforts of CPAN
☻ Might be hard to resolve deps.
Subversion to the rescue
(could be a barrier for newbies)

Nice-to-haves:
Order control
Temporarily disable plugins
Per plugin config
Per plugin storage

Resources
Class::Trigger
http:// search.cpan.org /dist/Class-Trigger/
Module::Pluggable
http:// search.cpan.org /dist/Module-Pluggable/
Ask Bjorn Hansen: Build Easily Extensible Perl Programs
http://conferences.oreillynet.com/cs/os2005/view/e_sess/6806
qpsmtpd
http:// smtpd.develooper.com /
MT plugins
http:// www.sixapart.com/pronet/plugins /
Kwiki
http:// www.kwiki.org /
Plagger
http:// plagger.org /

http://techvideos.humourbox.info	33
http://www.slideshare.net	13
http://lj-toys.com	2
http://www.linkedin.com	1

Writing Pluggable Software

by Tatsuhiko Miyagawa

Accessibility

Categories

Upload Details

Usage Rights

4 Embeds 49

Statistics

Writing Pluggable Software Presentation Transcript