Writing Pluggable Software

Writing Pluggable Software Tatsuhiko Miyagawa [email_address] Six Apart, Ltd. / Shibuya Perl Mongers YAPC::Asia 2007 Tokyo

For non-JP attendees … If you find \ in the code, Replace that with backslash. (This is MS' fault)

#1 How to make your app pluggable

#2 TMTOWTDP There's More Than One Way To Deploy Plugins Pros/Cons by examples

#1 Keep the app design and code simple

#2 Let the app users customize the behavior (without hacking the internals)

#3 It's fun to write plugins for most hackers (see: Plagger and Kwiki)

"Can your app do XXX?" "Yes, by plugins."

"Your app has a bug in YYY" "No, it's the bug in plugin YYY, Not my fault." (Chain Of Responsibilities)

Ack is a "full-stack" software now.

By "full-stack" I mean: Easy install No configuration No way to extend

Specifically: These are hardcoded Ignored directories Filenames and types

Ignored Directories @ignore_dirs = qw( blib CVS RCS SCCS .svn _darcs .git );

Filenames and languages mapping %mappings = ( asm => [qw( s S )], binary => …, cc => [qw( c h xs )], cpp => [qw( cpp m h C H )], csharp => [qw( cs )], … perl => [qw( pl pm pod tt ttml t )], … );

What if making these pluggable?

Don't get me wrong Andy, I love ack the way it is… Just thought it can be a very good example for the tutorial.

Quickstart: Class::Trigger Module::Pluggable © Six Apart Ltd. Employees

Class::Trigger SYNOPSIS package Foo; use Class::Trigger; sub foo { my $self = shift; $self->call_trigger('before_foo'); # some code ... $self->call_trigger('after_foo'); } package main; Foo->add_trigger(before_foo => \&sub1); Foo->add_trigger(after_foo => \&sub2);

Class::Trigger Helps you to implement Observer Pattern. (Rails calls this Observer)

Module::Pluggable SYNOPSIS package MyClass; use Module::Pluggable; use MyClass; my $mc = MyClass->new(); # returns the names of all plugins installed under MyClass::Plugin::* my @plugins = $mc->plugins(); package MyClass::Plugin::Foo; sub new { … } 1;

Setup plugins in App::Ack package App::Ack; use Class::Trigger; use Module::Pluggable require => 1; __PACKAGE__->plugins;

Setup plugins in App::Ack package App::Ack; use Class::Trigger; use Module::Pluggable require => 1 ; __PACKAGE__->plugins; # "requires" modules

Ignored Directories (Before) @ignore_dirs = qw( blib CVS RCS SCCS .svn _darcs .git );

Ignored Directories (After) # lib/App/Ack.pm __PACKAGE__->call_trigger(' ignore_dirs.add ', \@ignore_dirs);

Ignored Directories (plugins) # lib/App/Ack/Plugin/IgnorePerlBuildDir.pm package App::Ack::Plugin::IgnorePerlBuildDir; App::Ack->add_trigger( " ignore_dirs.add " => sub { my($class, $ignore_dirs) = @_; push @$ignore_dirs, qw( blib ); }, ); 1;

Ignored Directories (plugins) # lib/App/Ack/Plugin/IgnoreSourceControlDir.pm package App::Ack::Plugin::IgnoreSourcdeControlDir; App::Ack->add_trigger( " ignore_dirs.add " => sub { my($class, $ignore_dirs) = @_; push @$ignore_dirs, qw( CVS RCS .svn _darcs .git ); }, ); 1;

Filenames and languages (before) %mappings = ( asm => [qw( s S )], binary => …, cc => [qw( c h xs )], cpp => [qw( cpp m h C H )], csharp => [qw( cs )], … perl => [qw( pl pm pod tt ttml t )], … );

Filenames and languages (after) # lib/App/Ack.pm __PACKAGE__->call_trigger('mappings.add', \%mappings);

Filenames and languages (plugins) package App::Ack::Plugin::MappingCFamily; use strict; App::Ack->add_trigger( "mappings.add" => sub { my($class, $mappings) = @_; $mappings->{asm} = [qw( s S )]; $mappings->{cc} = [qw( c h xs )]; $mappings->{cpp} = [qw( cpp m h C H )]; $mappings->{csharp} = [qw( cs )]; $mappings->{css} = [qw( css )]; }, ); 1;

Works great with few lines of code!

Now it's time to add Some useful stuff.

Example Plugin: Content Filter

sub _search { my $fh = shift; my $is_binary = shift; my $filename = shift; my $regex = shift; my %opt = @_; if ($is_binary) { my $new_fh; App::Ack->call_trigger('filter.binary', $filename, \$new_fh); if ($new_fh) { return _search($new_fh, 0, $filename, $regex, @_); } }

Example: Search PDF content with ack

PDF filter plugin package App::Ack::Plugin::ExtractContentPDF; use strict; use CAM::PDF; use File::Temp; App::Ack->add_trigger( 'mappings.add' => sub { my($class, $mappings) = @_; $mappings->{pdf} = [qw(pdf)]; }, );

PDF filter plugin (cont.) App::Ack->add_trigger( 'filter.binary' => sub { my($class, $filename, $fh_ref) = @_; if ($filename =~ /\.pdf$/) { my $fh = File::Temp::tempfile; my $doc = CAM::PDF->new($file); my $text; for my $page (1..$doc->numPages){ $text .= $doc->getPageText($page); } print $fh $text; seek $$fh, 0, 0; $$fh_ref = $fh; } }, );

PDF search > ack --type=pdf Audrey yapcasia2007-pugs.pdf:3:Audrey Tang

Homework Use File::Extract To handle arbitrary media files

Homework 2: Search non UTF-8 files (hint: use Encode::Guess) You'll need another hook.

Summary Class::Trigger + Module::Pluggable = Pluggable app easy

#2 TMTOWTDP There's More Than One Way To Deploy Plugins

Module::Pluggable + Class::Trigger = Simple and Nice but has limitations

In Reality, we need more control over how plugins behave

1) The order of plugin executions

2) Per user configurations for plugins

3) Temporarily Disable plugins Should be easy

4) How to install & upgrade plugins

5) Let plugins have storage area

Examples: Kwiki Plagger qpsmtpd Movable Type

I won't talk about Catalyst plugins (and other framework thingy)

Because they're NOT "plug-ins"

Install plugins And now you write MORE CODE

95% of Catalyst plugins Are NOT "plugins" But "components" 95% of these statistics is made up by the speakers.

Kwiki Plugin code package Kwiki::URLBL; use Kwiki::Plugin -Base ; use Kwiki::Installer -base; const class_id => 'urlbl'; const class_title => 'URL Blacklist DNS'; const config_file => 'urlbl.yaml'; sub register { require URI::Find; my $registry = shift; $registry->add(hook => 'edit:save', pre => 'urlbl_hook'); $registry->add(action => 'blacklisted_url'); }

Kwiki Plugin (cont.) sub urlbl_hook { my $hook = pop; my $old_page = $self->hub->pages->new_page($self->pages->current->id); my $this = $self->hub->urlbl; my @old_urls = $this->get_urls($old_page->content); my @urls = $this->get_urls($self->cgi->page_content); my @new_urls = $this->get_new_urls(\@old_urls, \@urls); if (@new_urls && $this->is_blocked(\@new_urls)) { $hook->cancel(); return $self->redirect("action=blacklisted_url"); } }

Magic implemented in Spoon(::Hooks)

"Install" Kwiki Plugins # order doesn't matter here (according to Ingy) Kwiki::Display Kwiki::Edit Kwiki::Theme::Basic Kwiki::Toolbar Kwiki::Status Kwiki::Widgets # Comment out (or entirely remove) to disable # Kwiki::UnnecessaryStuff

Kwiki plugin config # in Kwiki::URLBL plugin __config/urlbl.yaml__ urlbl_dns: sc.surbl.org, bsb.spamlookup.net, rbl.bulkfeeds.jp # config.yaml urlbl_dns: myowndns.example.org

Kwiki plugins are CPAN modules

Install and Upgrade plugins cpan> install Kwiki::SomeStuff

Using CPAN as a repository Pros #1: reuse most of current CPAN infrastructure.

Using CPAN as a repository Pros #2: Increasing # of modules = good motivation for Perl hackers

Cons #1: Installing CPAN deps could be a mess (especially for Win32)

Cons #2: Whenever Ingy releases new Kwiki, lots of plugins just break.

Kwiki plugin storage return if grep {$page->id} @{$self-> config->cached_display_ignore }; my $html = io->catfile( $self-> plugin_directory ,$page->id )->utf8;

Except: plugins are now in SVN repository

Plagger plugin package Plagger::Plugin::Publish::iCal; use strict; use base qw( Plagger::Plugin ); use Data::ICal; use Data::ICal::Entry::Event; use DateTime::Duration; use DateTime::Format::ICal; sub register { my($self, $context) = @_; $context-> register_hook ( $self, ' publish.feed ' => \&publish_feed, ' plugin.init ' => \&plugin_init, ); }

Plagger plugin (cont) sub plugin_init { my($self, $context) = @_; my $dir = $self->conf->{dir}; unless (-e $dir && -d _) { mkdir $dir, 0755 or $context->error("Failed to mkdir $dir: $!"); } }

Plagger plugin storage $self->conf->{invindex} ||= $self-> cache->path_to ('invindex');

Plagger plugin config # The order matters in config.yaml # if they're in the same hooks plugins: - module: Subscription::Config config: feed: - http://www.example.com/ - module: Filter::DegradeYouTube config: dev_id: XYZXYZ - module: Publish::Gmail disable: 1

Plugins Install & Upgrade cpan> notest install Plagger # or … > svn co http://…/plagger/trunk plagger > svn update

Plagger impl. ripped off by many apps now

mod_perl for SMTP Runs with tcpserver, forkserver or Danga::Socket standalone

Plugins: Flat files rock:/home/miyagawa/svn/qpsmtpd> ls -F plugins async/ greylisting auth/ hosts_allow check_badmailfrom http_config check_badmailfromto ident/ check_badrcptto logging/ check_badrcptto_patterns milter check_basicheaders parse_addr_withhelo check_earlytalker queue/ check_loop quit_fortune check_norelay rcpt_ok check_relay relay_only check_spamhelo require_resolvable_fromhost content_log rhsbl count_unrecognized_commands sender_permitted_from dns_whitelist_soft spamassassin dnsbl tls domainkeys tls_cert* dont_require_anglebrackets virus/

qpsmtpd plugin sub hook_mail { my ($self, $transaction, $sender, %param) = @_; my @badmailfrom = $self->qp->config ("badmailfrom") or return (DECLINED); for my $bad (@badmailfrom) { my $reason = $bad; $bad =~ s/^\s*(\S+).*/$1/; next unless $bad; $transaction->notes('badmailfrom', $reason) … } return (DECLINED); }

Actually qpsmtpd Plugins are "compiled" to modules

my $eval = join("\n", "package $package;", 'use Qpsmtpd::Constants;', "require Qpsmtpd::Plugin;", 'use vars qw(@ISA);', 'use strict;', '@ISA = qw(Qpsmtpd::Plugin);', ($test_mode ? 'use Test::More;' : ''), "sub plugin_name { qq[$plugin] }", $line, $sub, "\n", # last line comment without newline? ); $eval =~ m/(.*)/s; $eval = $1; eval $eval; die "eval $@" if $@;

qpsmtpd plugin config rock:/home/miyagawa/svn/qpsmtpd> ls config.sample/ config.sample: IP logging require_resolvable_fromhost badhelo loglevel rhsbl_zones badrcptto_patterns plugins size_threshold dnsbl_zones rcpthosts tls_before_auth invalid_resolvable_fromhost relayclients tls_ciphers

config/plugins # content filters virus/klez_filter # rejects mails with a SA score higher than 2 spamassassin reject_threshold 20

config/badhelo # these domains never uses their domain when greeting us, so reject transactions aol.com yahoo.com

Install & Upgrade plugins Just use subversion

MT plugins are flat-files (or scripts that call modules)

MT plugin code package MT::Plugin::BanASCII; our $Method = "deny"; use MT; use MT::Plugin; my $plugin = MT::Plugin->new({ name => "BanASCII v$VERSION", description => "Deny or moderate ASCII or Latin-1 comment", }); MT->add_plugin($plugin); MT->add_callback('CommentFilter', 2, $plugin, \&handler);

MT plugin code (cont) sub init_app { my $plugin = shift; $plugin->SUPER::init_app(@_); my($app) = @_; return unless $app->isa('MT::App::CMS'); $app-> add_itemset_action ({ type => 'comment', key => 'spam_submission_comment', label => 'Report SPAM Comment(s)', code => sub { $plugin->submit_spams_action('MT::Comment', @_) }, } );

MT plugin storage require MT::PluginData; my $data = MT::PluginData->load({ plugin => 'sidebar-manager', key => $blog_id }, ); unless ($data) { $data = MT::PluginData->new; $data->plugin('sidebar-manager'); $data->key($blog_id); } $data->data( \$modulesets ); $data->save or die $data->errstr;

Order control MT->add_callback('CMSPostEntrySave', 9 , $rightfields, \&CMSPostEntrySave); MT->add_callback('CMSPreSave_entry', 9 , $rightfields, \&CMSPreSave_entry); MT::Entry->add_callback('pre_remove', 9 , $rightfields, \&entry_pre_remove); Defined in plugins. No Control on users end

Conclusion Flat-files vs. Modules

Flat-files: ☺ Easy to install (Just grab it) ☻ Hard to upgrade OK for simple plugins

Modules: ☺ Full-access to Perl OO goodness ☺ Avoid duplicate efforts of CPAN ☻ Might be hard to resolve deps. Subversion to the rescue (could be a barrier for newbies)

Nice-to-haves: Order control Temporarily disable plugins Per plugin config Per plugin storage

Resources Class::Trigger http:// search.cpan.org /dist/Class-Trigger/ Module::Pluggable http:// search.cpan.org /dist/Module-Pluggable/ Ask Bjorn Hansen: Build Easily Extensible Perl Programs http://conferences.oreillynet.com/cs/os2005/view/e_sess/6806 qpsmtpd http:// smtpd.develooper.com / MT plugins http:// www.sixapart.com/pronet/plugins / Kwiki http:// www.kwiki.org / Plagger http:// plagger.org /

Writing Pluggable Software

More Related Content

What's hot

Similar to Writing Pluggable Software

More from Tatsuhiko Miyagawa

Recently uploaded

Writing Pluggable Software