Api Design


Published on

Too few projects demand good API design as a critical goal. A clean and
extensible API will pay for itself many times over in fostering a community of
plugins. We certainly cannot anticipate the ways in which our users will bend
our modules, but designing an extensible system alleviates the pain. There are
many lessons to be learned from Moose, HTTP::Engine and IM::Engine,
Dist::Zilla, KiokuDB, Fey, and TAEB.

The most important lesson is to decouple the core functionality from the
"fluff" such as sugar and middleware. This forces you to have a solid API that
ordinary users can extend. This also lets users write their own sugar and
middleware. In a tightly-coupled system, there is little hope for

In this talk, you will learn how to make very productive use of Moose's roles
to form the foundation of a pluggable system. Roles provide excellent means of
code reuse and safe composition. I will also demonstrate how to use
Sub::Exporter to construct a more useful and flexible sugar layer.

Published in: Technology, Business
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Presented YAPC::Asia, 2009-09-10.
    Tokyo Institute of Technology, Tokyo, Japan.
  • There is a good Google Tech Talk on "how to design an API and why it matters". There isn't a whole lot of overlap between this talk and that one. Watch that one.

  • At YAPC::NA 2009, this guy, Hans Dieter Pearcey aka confound aka hdp, presented a talk about Dist::Zilla.

  • Dist::Zilla was written by this other guy, Ricardo Signes, aka rjbs.

  • Dieter presented this slide about Dist::Zilla's pluggable design. I loved it and I wanted to devote an entire talk to its glory.

  • I'm here to highlight really cool API designs that these projects have. In particular, they design for extensibility and pluggability. Extensibility is really important to the current and future success of these projects.
  • If you haven't noticed yet, this talk is going to be very Moose-heavy. All those modules have the Moose nature.

  • There is a poorly kept secret for designing great APIs. I hope that all of you already do this, but you probably do not do it enough.
  • Write tests.
  • Write so many tests your ears bleed. I am not joking!
  • If you remember nothing else, remember to write tests!
  • Write tests so you can tell if your API is painful to use. Which of these would you rather be stuck with?

    Make it painless for your users. Some of them might be using your module a lot. If it's tedious to use your module...
  • ... then you'll piss your users off. They'll leave and use some other module, or worse, find out where you live.
  • This is Jesse Vincent, the nicest guy in the world :)
  • Moose serves as the foundation for the rest of the talk, so I want to explain what it "got right" in terms of its API. These next few slides are difficult but it will get clearer and less heady, so wake up soon if you space out.
  • Moose is built on top of a metaobject protocol. This is Class::MOP.

    See my "Extending Moose for Applications" talk for a proper introduction to the metaobject protocol http://sartak.org/talks/yapc-na-2009/extending-moose/
  • The MOP is vital to Moose's operation. Basically, it means that every part of your class is represented by an object.
  • When you say "has" it creates an instance of the Moose::Meta::Attribute class, which holds information like the attribute's name, its type constraint, default value, etc.
  • The is => 'ro' option creates a "cache" method in your class. It also creates an object of class Moose::Meta::Method::Accessor to represent that "cache" method.
  • This is important because we can subclass Moose's class to add our own special logic, such as making the cache persist across processes. Subclassing and adding logic is ordinary object-oriented programming!
  • We can also specify roles to apply to cache's attribute object. This is slightly better because it means a single attribute can have many extensions. Just like how it's better to design with roles than subclasses in ordinary programming.
  • The metaobject protocol powers most of the MooseX modules. In my opinion, the metaobject protocol is responsible for a very large part of Moose's popularity. The other reason for Moose's popularity is it enables concise class code.
  • Moose also makes a very clean separation between its sugar layer and the rest of the system.
  • Say you wanted to get ahold of some class...
  • Then add an attribute to it. This doesn't work because "has" is not a method. Its first parameter is supposed to be the attribute name, not the class you're adding the attribute to.
  • So we have to call $class's "has" as a function. This kind of thing is ridiculous. Maybe the other class has used "no Moose" so that "has" is deleted. Or perhaps it renamed "has".
  • Not to mention how ugly this mess is.
  • If we look at the source code of Moose, we can see "has" is basically a wrapper around the "add_attribute" method of the Class::MOP::Class instance.
  • Much better. There's no messy syntax. This can be used outside of $class's namespace just fine. This also works if class has cleaned up after Moose with "no Moose" or namespace::clean.
  • Having a clean sugar layer means that other people can write better sugar. I like the idea of providing a separate Devel::Declare-powered sugar layer in a separate distribution. It forces you to cleanly separate the pieces.
  • Path::Dispatcher is a standalone-URI dispatcher. I wrote it because I wanted Jifty::Dispatcher for Prophet's command-line interface.

    This is its sugar layer. Like Moose, it has a clean, extensible API if you want the freedom to do unusual things.
  • It used to be that Path::Dispatcher::Declarative was implemented as an ordinary Sub::Exporter-using module.
  • This is not at all extensible. You can't change the meaning of "on" or "under" because these are hardcoded. Reusing this sugar would be painful as well.
  • This was fine for a few weeks, but then Robert Krimen started using Path::Dispatcher. And he wanted to extend it for a module he was writing called Getopt::Chain.
  • Path::Dispatcher::Builder makes the sugar layer creation use OOP. This let Robert subclass Path::Dispatcher::Builder and use it for his own modules. He can reuse the regular dispatcher logic, tweak it by overriding methods, and add his own behavior.
  • OO sugar is a really neat idea that I haven't seen anywhere else.
  • HTTP::Engine abstracts away the various HTTP server interfaces that Perl has accumulated since HTTP was invented. The benefit is in letting the user pick which server interface best fits their particular needs.
  • For example, you can use mod_perl if you enjoy pain.
  • Or FastCGI if you're a cool dude.
  • HTTP::Engine works well because the code you write doesn't have to worry about redirecting I/O streams, making sense of %ENV, or any of the other crap you do when writing against a particular server module.
  • HTTP::Engine boils the web server cycle to the least common denominator. You take a request...
  • … and return a response.
  • Can we please standardize on this? New server modules can implement an HTTP::Engine::Interface, then immediately every existing HTTP::Engine-based application can switch to it by changing only a single line of code.
  • Now I want to explain why this is so awesome.
  • Here's a list of plugins used by a typical Dist::Zilla-based distribution.
  • Dist::Zilla itself occasionally calls methods like this. The key bit is "plugins_with".
  • plugins_with takes a role name...
  • ...and selects the plugins that "do" the role. These plugins all do the "FileGatherer" role, which means the plugin adds files to a distribution.
  • Then, dzil calls gather_files on each of these plugins so it can actually add files to the distribution. "License", "Readme", and "MetaYAML" add the respective files, "AllFiles" adds every file the author wrote. "PodTests" adds pod testing files to the distribution.
  • Dist::Zilla uses this architecture for all of the interesting parts of building a CPAN distribution. This is "munging files", which lets plugins edit files to increase the version number, or move tests around.
  • It turns out that RT has a very similar extension mechanism.
  • This code exists in User/Prefs.html. The callback method selects all plugins that do the "User/Prefs.html" "role".
  • Then it calls the FormEnd "method" (template) on these selected plugins.
  • And you can pass arbitrary parameters to each method.
  • This works extremely well for us! We try to build most customer extensions with callbacks. It's basically the same design as Dist::Zilla's.
  • RT has had callbacks since 2002, first released in 3.0.0. This pattern has been the best mechanism for any kind of RT extension for almost seven years now.
  • This design gives the user choice over which behavior she wants. And in my experience, users really really want choice.
  • This design is also extensible for free. These are some of the modules that have been written to extend Dist::Zilla.
  • All they need to do is fulfill the requirements of the roles they "do". I'm going to talk about that more in my (Parameterized) Roles talk.

  • Extensibility is also important for code you can't share. We can't ask Ricardo to include company secrets for Dist::Zilla, and maintaining a fork really sucks.
  • So now you know!
  • IM::Engine is a project I'm working on. It's basically HTTP::Engine for IM. You can write a bot, once, that will run on any service IM::Engine can talk to, including IRC. IM::Engine smooths over the differences in the protocols.
  • I've extended Ricardo's design with a number of helper methods. plugin_collect is the one I like best.
  • For each plugin that does the ExtendsObject::User role...
  • ...call its "traits" method.
  • The return value of this call is the list of all return values of the "traits" methods.
  • This is the important part of plugin_collect's implementation. There's not much there. I like very layered APIs because they're easier to understand and reuse, especially by your users, than huge monolithic methods. Each layer does only a little bit of work.
  • Here's a piece of design I like a lot. This lets plugins participate in object construction. Each plugin can provide constructor arguments.
  • This lets plugins participate even more in object construction. Now plugins can provide roles for the object you're constructing. This lets plugins add attributes and methods to the object. I use this in a plugin to give state management methods to User objects.
  • new_with_traits comes from MooseX::Traits. It's a really nice module for designing pluggable and extensible systems. You just pass a list of roles to new_with_traits and it will arrange it so that the object does those roles.
  • Other objects of that class are not affected by new_with_traits. The way it works internally is by creating a new subclass of Class. This is vital because it maintains modularity. I don't want my extensions to screw up your extensions.
  • In Moose land, roles and traits are basically synonymous. Some people will tell you there are subtle differences, but there's no clear consensus. I just say "roles" except when I have to say "traits" for a module.
  • So that is all I have time to cover. There are plenty more nice examples in modules like KiokuDB, Fey, and the now-moosified Catalyst.
  • Moose teaches us that extensibility can lead to a great corpus of extensions. Separation of sugar keeps you and your users flexible.
  • The OO sugar layer is a new idea that I hope catches on. I'll have to dedicate more time to it.
  • If you omit inconsequential details, then your application remains flexible and concise.
  • Pluggability does not have to be implicit, as in subclassing. Explicitly controlling pluggability lets you do more interesting things.
  • … such as the things IM::Engine does, by letting plugins manipulate system objects. It also provides methods for common plugin operations so you don't have to repeat them everywhere.
  • I almost forgot...
  • I almost forgot...
  • Thank you to Ishigaki-san for translating my slides!
  • Api Design

    1. 1. API Design Shawn M Moore Best Practical Solutions http://sartak.org
    2. 2. Google API
    3. 3. CC-BY-SA Yuval Kogman, 2006
    4. 4. CC-BY-SA Yuval Kogman, 2006
    5. 5. CC-BY-SA Hans Dieter Pearcey, 2009
    6. 6. Moose Path::Dispatcher HTTP::Engine Dist::Zilla IM::Engine API
    7. 7. CC-BY-SA-NC Will Spaetzel, 2005
    8. 8. THE SECRET
    12. 12. Test! use base 'Class::Accessor::Fast'; __PACKAGE__->mk_ro_accessors('birthday'); use Moose; has birthday => (is => 'ro'); API
    13. 13. Test!
    14. 14. Test!
    15. 15. Test!
    16. 16. Test!
    17. 17. Moose package Point; use Moose; has x => ( is => 'ro', isa => 'Num', ); Moose
    18. 18. Metaobject Protocol Class::MOP Moose Class::MOP
    19. 19. Metaobject Protocol has cache => ( is => 'ro', );
    20. 20. Metaobject Protocol has cache => ( is => 'ro', ); Moose::Meta::Attribute has Moose::Meta::Attribute
    21. 21. Metaobject Protocol has cache => ( is => 'ro', ); Moose::Meta::Method::Accessor cache
    22. 22. Metaobject Protocol class PersistentAttr extends Moose::Meta::Attribute { … } has cache => ( metaclass => 'PersistentAttr', is => 'ro', ); Moose
    23. 23. Metaobject Protocol role PersistentAttr { … } has cache => ( traits => ['PersistentAttr'], is => 'ro', );
    24. 24. Moose X MooseX MOP
    25. 25. Sugar Layer Moose
    26. 26. Sugar Layer my $class = get_class();
    27. 27. Sugar Layer my $class = get_class(); $class->has( birthday => ( is => 'ro', ) ); has
    28. 28. Sugar Layer my $class = get_class(); no strict 'refs'; *{$class.'::has'}->( birthday => ( is => 'ro', ) ); has
    29. 29. Sugar Layer my $class = get_class(); no strict 'refs'; *{$class.'::has'}->( birthday => ( is => 'ro', ) );
    30. 30. Sugar Layer Class::MOP::Class ->initialize($class) ->add_attribute( $name, %options); has add_attribute
    31. 31. Sugar Layer my $class = get_class(); $class->meta->add_attribute( birthday => ( is => 'ro', ) );
    32. 32. Sugar Layer use MooseX::Declare; class Point3D extends Point { has z => (…); after clear { $self->z(0); } }
    33. 33. Path::Dispatcher use Path::Dispatcher::Declarative -base; on ['wield', qr/^w+$/] => sub { wield_weapon($2); } under display => sub { on inventory => sub { show_inventory }; on score => sub { show_score }; }; YourDispatcher->run('display score'); Jifty::Dispatcher Prophet
    34. 34. Path::Dispatcher::Declar use Sub::Exporter -setup => { exports => [ on => &build_on, under => &build_under, …, ], }; Sub::Exporter
    35. 35. Path::Dispatcher::Declar use Sub::Exporter -setup => { exports => [ on => &build_on, under => &build_under, …, ], };
    36. 36. Path::Dispatcher::Builder Robert Krimen "grink" Robert Krimen
    37. 37. Path::Dispatcher::Builder return { on => sub { $builder->on(@_) }, under => sub { $builder->under(@_) }, …, };
    38. 38. grink++ OO
    39. 39. HTTP::Engine HTTP::Engine->new( interface => { module => 'ServerSimple', args => { … }, request_handler => sub { … }, }, )->run; HTTP::Engine
    40. 40. HTTP::Engine HTTP::Engine->new( interface => { module => 'ModPerl', args => { … }, request_handler => sub { … }, }, )->run; mod_perl
    41. 41. HTTP::Engine HTTP::Engine->new( interface => { module => 'FCGI', args => { … }, request_handler => sub { … }, }, )->run; FastCGI
    42. 42. HTTP::Engine request_handler => sub { my $request = shift; return $response; } HTTP::Engine IO
    43. 43. HTTP::Engine request_handler => sub { my $request = shift; return $response; } HTTP::Engine OK
    44. 44. HTTP::Engine request_handler => sub { my $request = shift; return $response; }
    45. 45. HTTP::Engine++ 1
    46. 46. CC-BY-SA Hans Dieter Pearcey, 2009
    47. 47. Dist::Zilla AllFiles ExtraTests InstallDirs License MakeMaker Manifest ManifestSkip MetaYAML PkgVersion PodTests PodVersion PruneCruft Readme UploadToCPAN
    48. 48. Dist::Zilla AllFiles ExtraTests InstallDirs License $_->gather_files MakeMaker for Manifest $self->plugins_with( ManifestSkip -FileGatherer MetaYAML PkgVersion ); PodTests PodVersion PruneCruft Readme UploadToCPAN Dist::Zilla
    49. 49. Dist::Zilla AllFiles ExtraTests InstallDirs License $_->gather_files MakeMaker for Manifest $self->plugins_with( ManifestSkip -FileGatherer MetaYAML PkgVersion ); PodTests PodVersion PruneCruft Readme UploadToCPAN plugins_with
    50. 50. Dist::Zilla AllFiles ExtraTests InstallDirs License $_->gather_files MakeMaker for Manifest $self->plugins_with( ManifestSkip -FileGatherer MetaYAML PkgVersion ); PodTests PodVersion PruneCruft Readme UploadToCPAN
    51. 51. Dist::Zilla AllFiles ExtraTests InstallDirs License $_->gather_files MakeMaker for Manifest $self->plugins_with( ManifestSkip -FileGatherer MetaYAML PkgVersion ); PodTests PodVersion PruneCruft Readme UploadToCPAN gather_files
    52. 52. Dist::Zilla AllFiles ExtraTests InstallDirs License $_->munge_files MakeMaker for Manifest $self->plugins_with( ManifestSkip -FileMunger MetaYAML PkgVersion ); PodTests PodVersion PruneCruft Readme UploadToCPAN Dist::Zilla
    53. 53. Request Tracker User/Prefs.html $m->callback( CallbackName => 'FormEnd', UserObj => $UserObj, …, ); RT
    54. 54. Request Tracker User/Prefs.html $m->callback( CallbackName => 'FormEnd', UserObj => $UserObj, …, );
    55. 55. Request Tracker User/Prefs.html $m->callback( CallbackName => 'FormEnd', UserObj => $UserObj, …, ); FormEnd
    56. 56. Request Tracker User/Prefs.html $m->callback( CallbackName => 'FormEnd', UserObj => $UserObj, …, );
    57. 57. Request Tracker User/Prefs.html $m->callback( CallbackName => 'FormEnd', UserObj => $UserObj, …, );
    58. 58. Request Tracker commit 4c05a6835eef112701ac58dfd1b133e220059d4f Author: Jesse Vincent <jesse@bestpractical.com> Date: Fri Dec 27 18:50:06 2002 -0500 Attempting mason callouts Ticket/Update.html <& /Elements/Callback, Name => 'BeforeTextarea', %ARGS &> 7
    59. 59. Dist::Zilla Choice ModuleBuild or MakeMaker MetaYAML or MetaJSON
    60. 60. Dist::Zilla Extensibility Dist::Zilla::Plugin::CriticTests Dist::Zilla::Plugin::Repository Dist::Zilla::Plugin::PerlTidy
    61. 61. Dist::Zilla Extensibility Dist::Zilla::Plugin::CriticTests InlineFiles Dist::Zilla::Plugin::Repository MetaProvider Dist::Zilla::Plugin::PerlTidy FileMunger
    62. 62. Dist::Zilla Extensibility Dist::Zilla::Plugin::BPS::Secret
    63. 63. CC-BY-SA Hans Dieter Pearcey, 2009
    64. 64. IM::Engine incoming_callback => sub { my $incoming = shift; my $message = $incoming->plaintext; $message =~ tr[a-zA-Z][n-za-mN-ZA-M]; return $incoming->reply($message); } IM::Engine
    65. 65. IM::Engine $self->plugin_collect( role => 'ExtendsObject::User', method => 'traits', ); plugin_collect
    66. 66. IM::Engine $self->plugin_collect( role => 'ExtendsObject::User', method => 'traits', );
    67. 67. IM::Engine $self->plugin_collect( role => 'ExtendsObject::User', method => 'traits', ); traits
    68. 68. IM::Engine my @all_traits = $self->plugin_collect( role => 'ExtendsObject::User', method => 'traits', );
    69. 69. method plugin_collect { my @items; $self->each_plugin( callback => sub { push @items, shift->$method }, ); return @items; } plugin_collect
    70. 70. IM::Engine method new_with_plugins { my %args = ( $self->plugin_collect( role => …, method => 'ctor_args', ), @_, ); $self->new(%args); }
    71. 71. IM::Engine push @{ $args{traits} }, $self->plugin_collect( role => …, method => 'traits', ); $self->new_with_traits(%args);
    72. 72. MooseX::Traits $object = Class->new_with_traits( traits => ['Counter'], ); $other = Class->new; $object->counter; # 0 $other->counter; # Can't locate... new_with_traits MooseX::Traits
    73. 73. MooseX::Traits $object = Class->new_with_traits( traits => ['Counter'], ); $other = Class->new; $object->counter; # 0 $other->counter; # Can't locate...
    74. 74. Role = Trait Moose Role Trait
    75. 75. Moose Path::Dispatcher HTTP::Engine Dist::Zilla IM::Engine
    76. 76. Moose Extensibility Separation of sugar Dist::Zilla IM::Engine Moose
    77. 77. Moose Path::Dispatcher OO sugar layer Dist::Zilla IM::Engine OO
    78. 78. Moose Path::Dispatcher HTTP::Engine Omit inconsequential details IM::Engine
    79. 79. Moose Path::Dispatcher HTTP::Engine Dist::Zilla Explicit pluggability
    80. 80. Moose Path::Dispatcher HTTP::Engine Dist::Zilla IM::Engine Extreme pluggability DRY IM::Engine DRY
    81. 81. WRITEMoose Path::Dispatcher HTTP::Engine TESTS!!! Dist::Zilla IM::Engine
    82. 82. Moose Path::Dispatcher HTTP::Engine Dist::Zilla IM::Engine
    83. 83. Thanks to my Kenichi Ishigaki 83