typemap in Perl/XS
Upcoming SlideShare
Loading in...5
×
 

typemap in Perl/XS

on

  • 931 views

for OSDC.TW 2013

for OSDC.TW 2013

Statistics

Views

Total Views
931
Views on SlideShare
920
Embed Views
11

Actions

Likes
3
Downloads
8
Comments
0

1 Embed 11

https://twitter.com 11

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • # and of course, it usually doesn't.
  • # As you see, it's almost the same as declared in the .h files.
  • # so that a C compiler can compile.
  • # and should be.
  • # information on defines, includes, functions, typedefs, structs etc is stored in $converter's stash.

typemap in Perl/XS  typemap in Perl/XS Presentation Transcript

  • Wine label from Perl(located in Germany)Thanks , Wendy and Liz!
  • ... and MST
  • typemap in Perl/XSKenichi Ishigaki(charsbar)@OSDC.TW 2013April 19, 2013 View slide
  • Let me ask youfirst. View slide
  • Have you ever writtenan extension thatbridges C and Perl?
  • Part I: XS is ...?
  • Perl has a feature to loadan external C library.use DynaLoader;my @paths = dl_findfile(...);my $lib = dl_load_file($path);
  • This wont work correctlyunless the library conformswith Perls convention.use DynaLoader;my @paths = dl_findfile(...);my $lib = dl_load_file($path);
  • So we need somethingin-between.Perl XS C library
  • Writing the one-in-between is fairlyeasy.
  • (as long as you dontadd anything extra)
  • The simplest form of the one-in-between starts like this:#include "EXTERN.h"#include "perl.h"#include "XSUB.h"#include "ppport.h"#include <foo.h>MODULE = Foo PACKAGE = Foointfunc(const char* str)
  • These three lines are to usePerl API.#include "EXTERN.h"#include "perl.h"#include "XSUB.h"#include "ppport.h"#include <foo.h>MODULE = Foo PACKAGE = Foointfunc(const char* str)
  • This is for portabilitybetween perls.#include "EXTERN.h"#include "perl.h"#include "XSUB.h"#include "ppport.h"#include <foo.h>MODULE = Foo PACKAGE = Foointfunc(const char* str)
  • This is to import C functionsfrom the library.#include "EXTERN.h"#include "perl.h"#include "XSUB.h"#include "ppport.h"#include <foo.h>MODULE = Foo PACKAGE = Foointfunc(const char* str)
  • Module and packagedeclarations.#include "EXTERN.h"#include "perl.h"#include "XSUB.h"#include "ppport.h"#include <foo.h>MODULE = Foo PACKAGE = Foointfunc(const char* str)
  • The function declarations toexport (XSUBs) follow.#include "EXTERN.h"#include "perl.h"#include "XSUB.h"#include "ppport.h"#include <foo.h>MODULE = Foo PACKAGE = Foointfunc(const char* str)
  • As with .h files, there shouldbe only declarations.#include "EXTERN.h"#include "perl.h"#include "XSUB.h"#include "ppport.h"#include <foo.h>MODULE = Foo PACKAGE = Foointfunc(const char* str)
  • Nothing difficult(in principle).
  • The library you usemay not always bethat simple.
  • People may addsomething written in Cwith a variety of PerlAPI.
  • If someones XScode scares you,please remember.
  • Whats difficult isnot the XSinterface itself.
  • Its something elsethat makes themattercomplicated.
  • Dont hesitate writingXS interface if youfind a useful Clibrary.
  • Part II: How .xs file is compiled
  • To build an interfacelibrary, we need toturn its XS declarationinto pure C code.
  • If you already haveMakefile.PL, run it, andthen, make.$ perl Makefile.PL && make
  • If you have Build.PL, dosomething like this (ormaybe ./Build build).$ perl Build.PL && ./Build
  • If you dont have either,Milla or Minilla will helpyou (ask miyagawa-san).
  • The most important XSUB partwill be translated like this:XS_EUPXS(XS_Foo_func){dVAR; dXSARGS;if (items != 1)croak_xs_usage(cv, "str");{int RETVAL;dXSTARG;const char* str = (const char *)SvPV_nolen(ST(0));RETVAL = func(str);XSprePUSH; PUSHi((IV)RETVAL);}XSRETURN(1);}
  • A function exposed to the Perl worldtakes a Perl variable as its argument.XS_EUPXS(XS_Foo_func){dVAR; dXSARGS;if (items != 1)croak_xs_usage(cv, "str");{int RETVAL;dXSTARG;const char* str = (const char *)SvPV_nolen(ST(0));RETVAL = func(str);XSprePUSH; PUSHi((IV)RETVAL);}XSRETURN(1);}
  • However, a Perl variable isactually a structure in C.XS_EUPXS(XS_Foo_func){dVAR; dXSARGS;if (items != 1)croak_xs_usage(cv, "str");{int RETVAL;dXSTARG;const char* str = (const char *)SvPV_nolen(ST(0));RETVAL = func(str);XSprePUSH; PUSHi((IV)RETVAL);}XSRETURN(1);}
  • You cant pass it directly to a Cfunction, and vice versa.XS_EUPXS(XS_Foo_func){dVAR; dXSARGS;if (items != 1)croak_xs_usage(cv, "str");{int RETVAL;dXSTARG;const char* str = (const char *)SvPV_nolen(ST(0));RETVAL = func(str);XSprePUSH; PUSHi((IV)RETVAL);}XSRETURN(1);}
  • The Perl variable is convertedand cast into a C value here.XS_EUPXS(XS_Foo_func){dVAR; dXSARGS;if (items != 1)croak_xs_usage(cv, "str");{int RETVAL;dXSTARG;const char* str = (const char *)SvPV_nolen(ST(0));RETVAL = func(str);XSprePUSH; PUSHi((IV)RETVAL);}XSRETURN(1);}
  • Then, the function importedfrom the library is called.XS_EUPXS(XS_Foo_func){dVAR; dXSARGS;if (items != 1)croak_xs_usage(cv, "str");{int RETVAL;dXSTARG;const char* str = (const char *)SvPV_nolen(ST(0));RETVAL = func(str);XSprePUSH; PUSHi((IV)RETVAL);}XSRETURN(1);}
  • And the return value is convertedinto a Perl value again.XS_EUPXS(XS_Foo_func){dVAR; dXSARGS;if (items != 1)croak_xs_usage(cv, "str");{int RETVAL;dXSTARG;const char* str = (const char *)SvPV_nolen(ST(0));RETVAL = func(str);XSprePUSH; PUSHi((IV)RETVAL);}XSRETURN(1);}
  • typemap is thekey to thisconversion.
  • Basic type mappingsare defined inExtUtils::typemap.$ perldoc -m ExtUtils::typemap
  • # basic C typesint T_IVunsigned T_UVunsigned int T_UVlong T_IVunsigned long T_UVshort T_IVunsigned short T_UVchar T_CHARunsigned char T_U_CHARchar * T_PVunsigned char * T_PVconst char * T_PVcaddr_t T_PVwchar_t * T_PVwchar_t T_IV
  • # bool_t is defined in <rpc/rpc.h>bool_t T_IVsize_t T_UVssize_t T_IVtime_t T_NVunsigned long * T_OPAQUEPTRchar ** T_PACKEDARRAYvoid * T_PTRTime_t * T_PVSV * T_SV
  • Any type not listed thereshould be added locally,in a file named"typemap".
  • Part III: On type mapping
  • A typical"typemap" file hasthree sections.
  • "TYPEMAP" section containsa pair of C type and XStype.TYPEMAPchar * T_PV
  • How to map should be describedin the "INPUT" and "OUTPUT"sections, if necessary.INPUTT_PV$var = ($type)SvPV_nolen($arg)OUTPUTT_PVsv_setpv((SV*)$arg, $var);
  • These code fragments will beinterpolated while processingan XS file.INPUTT_PV$var = ($type)SvPV_nolen($arg)OUTPUTT_PVsv_setpv((SV*)$arg, $var);
  • Adding orphan C typesis easy.TYPEMAPmy_id T_IVmy_str T_PV
  • Modifying thebehavior of anexisting type needssome consideration.
  • If you are confident, justadd a new code fragment inyour local typemap.INPUTT_PVif (!SvOK($arg)) {$var = NULL;} else {$var = ($type)SvPV_nolen($arg);}
  • Or, you might want to add anothertypedef in the XS file, beforeMODULE and PACKAGE declarations.typedef char * my_nullable_str;MODULE Foo PACKAGE FOOintfunc(my_nullable_str str)
  • And then, add a behavior of thisnew type in your local typemap.TYPEMAPmy_nullable_str T_PV_OR_NULLINPUTT_PV_OR_NULLif (!SvOK($arg)) {$var = NULL;} else {$var = ($type)SvPV_nolen($arg);}
  • There is also a muchtrickier way to do it.MODULE Foo PACKAGE FOOintfunc(char * str_or_null)
  • If you dont get why thisworks, see perlref.INPUTT_PV$var = ${$var =~ /_or_null$/? qq{NULL}: qq{($type)SvPV_nolen($arg)}}
  • Remove a star and prepend"OUT" (or "IN_OUT") for anargument called by reference.MODULE Foo PACKAGE FOOintfunc(char * str, OUT int len)
  • Further Reading• perlxs• perlxstut• perlxstypemap• perlapi• perlcall• ... and others XS code :p
  • Part IV: From .h filesto XS files
  • Weve written XSfiles by hand so far.
  • Of course this canbe done with a tool.
  • The best-knowntool is h2xs.
  • Its mainly used toexpose constantsfrom a C library.
  • With the help ofC::Scan, you can use itto expose C functions aswell.$ h2xs -Axan Foo /path/to/header.h
  • Defaulting to backwards compatibility with perl 5.xx.xIf you intend this module to be compatible withearlier perl versions, please specify a minimum perlversion with the -b option.Writing Foo/ppport.hScanning typemaps...Scanning /home/xxxx/perl5/perlbrew/perls/perl-5.xx.x/lib/5.xx.x/ExtUtils/typemapScanning /path/to/header.h for functions...Scanning /path/to/header.h for typedefs...Writing Foo/lib/Foo.pmWriting Foo/Foo.xsWriting Foo/typemapWriting Foo/Makefile.PLWriting Foo/READMEWriting Foo/t/Foo.tWriting Foo/ChangesWriting Foo/MANIFEST
  • If the library is stable, this maygive you a good starting point.
  • However.
  • The files h2xs generatesare rather old-fashioned.
  • Its not good at updatingonly a part of adistribution, either.
  • I started writingConvert::H::XS.https://github.com/charsbar/convert_h_xs
  • Convert::H::XS takes a C header file,and looks for the minimuminformation to write XS components.use Convert::H::XS;my $converter = Convert::H::XS->new;$converter->process($h_file);
  • Convert::H::XS also providesmethods to write XS componentsfor convenience.$converter->write_constants("xs/constants.inc",);
  • You can tweak thosepieces of informationwith a callback.$converter->write_functions("xs/functions.inc",sub {my ($type, $name, $args) = @_;...return ($type, $name, $args);});
  • Grateful if you helpme improve this atthe hackathon.
  • Part V: CPANTSupdate
  • I talked aboutCPANTS last year.
  • CPANTS tests what moduleauthors often forget to testby themselves.
  • Perl QA Hackathon2013 in Lancaster
  • Several metrics aregoing to be removed.
  • Several metrics aregoing to be added.
  • To add new metrics,we need to findissues.
  • Issues are often raised byother QA/toolchain people.• Do not ship modules with Module::Install 1.04http://weblog.bulknews.net/post/33907905561/do-not-ship-modules-with-module-install-1-04• stop shipping MYMETA to CPANhttp://weblog.bulknews.net/post/44251476706/stop-shipping-mymeta-to-cpan
  • We need to confirmthey are measurable,and widespread.
  • What can I use toconfirm?
  • • CPANTS databases• metacpan• CPAN grep
  • I just wantedsomething else.
  • Groonga• a fulltext search engine• a column store• actively developed (monthly release)• client/server (http/gqpt)• C API• Web interface
  • Groonga::APIhttps://github.com/charsbar/groonga-api
  • Also grateful if youhelp me improve thisat the hackathon.
  • Thank you!