typemap in Perl/XS

  • 584 views
Uploaded on

for OSDC.TW 2013

for OSDC.TW 2013

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
584
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
9
Comments
0
Likes
3

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • # and of course, it usually doesn't.
  • # As you see, it's almost the same as declared in the .h files.
  • # so that a C compiler can compile.
  • # and should be.
  • # information on defines, includes, functions, typedefs, structs etc is stored in $converter's stash.

Transcript

  • 1. Wine label from Perl(located in Germany)Thanks , Wendy and Liz!
  • 2. ... and MST
  • 3. typemap in Perl/XSKenichi Ishigaki(charsbar)@OSDC.TW 2013April 19, 2013
  • 4. Let me ask youfirst.
  • 5. Have you ever writtenan extension thatbridges C and Perl?
  • 6. Part I: XS is ...?
  • 7. Perl has a feature to loadan external C library.use DynaLoader;my @paths = dl_findfile(...);my $lib = dl_load_file($path);
  • 8. This wont work correctlyunless the library conformswith Perls convention.use DynaLoader;my @paths = dl_findfile(...);my $lib = dl_load_file($path);
  • 9. So we need somethingin-between.Perl XS C library
  • 10. Writing the one-in-between is fairlyeasy.
  • 11. (as long as you dontadd anything extra)
  • 12. The simplest form of the one-in-between starts like this:#include "EXTERN.h"#include "perl.h"#include "XSUB.h"#include "ppport.h"#include <foo.h>MODULE = Foo PACKAGE = Foointfunc(const char* str)
  • 13. These three lines are to usePerl API.#include "EXTERN.h"#include "perl.h"#include "XSUB.h"#include "ppport.h"#include <foo.h>MODULE = Foo PACKAGE = Foointfunc(const char* str)
  • 14. This is for portabilitybetween perls.#include "EXTERN.h"#include "perl.h"#include "XSUB.h"#include "ppport.h"#include <foo.h>MODULE = Foo PACKAGE = Foointfunc(const char* str)
  • 15. This is to import C functionsfrom the library.#include "EXTERN.h"#include "perl.h"#include "XSUB.h"#include "ppport.h"#include <foo.h>MODULE = Foo PACKAGE = Foointfunc(const char* str)
  • 16. Module and packagedeclarations.#include "EXTERN.h"#include "perl.h"#include "XSUB.h"#include "ppport.h"#include <foo.h>MODULE = Foo PACKAGE = Foointfunc(const char* str)
  • 17. The function declarations toexport (XSUBs) follow.#include "EXTERN.h"#include "perl.h"#include "XSUB.h"#include "ppport.h"#include <foo.h>MODULE = Foo PACKAGE = Foointfunc(const char* str)
  • 18. As with .h files, there shouldbe only declarations.#include "EXTERN.h"#include "perl.h"#include "XSUB.h"#include "ppport.h"#include <foo.h>MODULE = Foo PACKAGE = Foointfunc(const char* str)
  • 19. Nothing difficult(in principle).
  • 20. The library you usemay not always bethat simple.
  • 21. People may addsomething written in Cwith a variety of PerlAPI.
  • 22. If someones XScode scares you,please remember.
  • 23. Whats difficult isnot the XSinterface itself.
  • 24. Its something elsethat makes themattercomplicated.
  • 25. Dont hesitate writingXS interface if youfind a useful Clibrary.
  • 26. Part II: How .xs file is compiled
  • 27. To build an interfacelibrary, we need toturn its XS declarationinto pure C code.
  • 28. If you already haveMakefile.PL, run it, andthen, make.$ perl Makefile.PL && make
  • 29. If you have Build.PL, dosomething like this (ormaybe ./Build build).$ perl Build.PL && ./Build
  • 30. If you dont have either,Milla or Minilla will helpyou (ask miyagawa-san).
  • 31. The most important XSUB partwill be translated like this:XS_EUPXS(XS_Foo_func){dVAR; dXSARGS;if (items != 1)croak_xs_usage(cv, "str");{int RETVAL;dXSTARG;const char* str = (const char *)SvPV_nolen(ST(0));RETVAL = func(str);XSprePUSH; PUSHi((IV)RETVAL);}XSRETURN(1);}
  • 32. A function exposed to the Perl worldtakes a Perl variable as its argument.XS_EUPXS(XS_Foo_func){dVAR; dXSARGS;if (items != 1)croak_xs_usage(cv, "str");{int RETVAL;dXSTARG;const char* str = (const char *)SvPV_nolen(ST(0));RETVAL = func(str);XSprePUSH; PUSHi((IV)RETVAL);}XSRETURN(1);}
  • 33. However, a Perl variable isactually a structure in C.XS_EUPXS(XS_Foo_func){dVAR; dXSARGS;if (items != 1)croak_xs_usage(cv, "str");{int RETVAL;dXSTARG;const char* str = (const char *)SvPV_nolen(ST(0));RETVAL = func(str);XSprePUSH; PUSHi((IV)RETVAL);}XSRETURN(1);}
  • 34. You cant pass it directly to a Cfunction, and vice versa.XS_EUPXS(XS_Foo_func){dVAR; dXSARGS;if (items != 1)croak_xs_usage(cv, "str");{int RETVAL;dXSTARG;const char* str = (const char *)SvPV_nolen(ST(0));RETVAL = func(str);XSprePUSH; PUSHi((IV)RETVAL);}XSRETURN(1);}
  • 35. The Perl variable is convertedand cast into a C value here.XS_EUPXS(XS_Foo_func){dVAR; dXSARGS;if (items != 1)croak_xs_usage(cv, "str");{int RETVAL;dXSTARG;const char* str = (const char *)SvPV_nolen(ST(0));RETVAL = func(str);XSprePUSH; PUSHi((IV)RETVAL);}XSRETURN(1);}
  • 36. Then, the function importedfrom the library is called.XS_EUPXS(XS_Foo_func){dVAR; dXSARGS;if (items != 1)croak_xs_usage(cv, "str");{int RETVAL;dXSTARG;const char* str = (const char *)SvPV_nolen(ST(0));RETVAL = func(str);XSprePUSH; PUSHi((IV)RETVAL);}XSRETURN(1);}
  • 37. And the return value is convertedinto a Perl value again.XS_EUPXS(XS_Foo_func){dVAR; dXSARGS;if (items != 1)croak_xs_usage(cv, "str");{int RETVAL;dXSTARG;const char* str = (const char *)SvPV_nolen(ST(0));RETVAL = func(str);XSprePUSH; PUSHi((IV)RETVAL);}XSRETURN(1);}
  • 38. typemap is thekey to thisconversion.
  • 39. Basic type mappingsare defined inExtUtils::typemap.$ perldoc -m ExtUtils::typemap
  • 40. # basic C typesint T_IVunsigned T_UVunsigned int T_UVlong T_IVunsigned long T_UVshort T_IVunsigned short T_UVchar T_CHARunsigned char T_U_CHARchar * T_PVunsigned char * T_PVconst char * T_PVcaddr_t T_PVwchar_t * T_PVwchar_t T_IV
  • 41. # bool_t is defined in <rpc/rpc.h>bool_t T_IVsize_t T_UVssize_t T_IVtime_t T_NVunsigned long * T_OPAQUEPTRchar ** T_PACKEDARRAYvoid * T_PTRTime_t * T_PVSV * T_SV
  • 42. Any type not listed thereshould be added locally,in a file named"typemap".
  • 43. Part III: On type mapping
  • 44. A typical"typemap" file hasthree sections.
  • 45. "TYPEMAP" section containsa pair of C type and XStype.TYPEMAPchar * T_PV
  • 46. How to map should be describedin the "INPUT" and "OUTPUT"sections, if necessary.INPUTT_PV$var = ($type)SvPV_nolen($arg)OUTPUTT_PVsv_setpv((SV*)$arg, $var);
  • 47. These code fragments will beinterpolated while processingan XS file.INPUTT_PV$var = ($type)SvPV_nolen($arg)OUTPUTT_PVsv_setpv((SV*)$arg, $var);
  • 48. Adding orphan C typesis easy.TYPEMAPmy_id T_IVmy_str T_PV
  • 49. Modifying thebehavior of anexisting type needssome consideration.
  • 50. If you are confident, justadd a new code fragment inyour local typemap.INPUTT_PVif (!SvOK($arg)) {$var = NULL;} else {$var = ($type)SvPV_nolen($arg);}
  • 51. Or, you might want to add anothertypedef in the XS file, beforeMODULE and PACKAGE declarations.typedef char * my_nullable_str;MODULE Foo PACKAGE FOOintfunc(my_nullable_str str)
  • 52. And then, add a behavior of thisnew type in your local typemap.TYPEMAPmy_nullable_str T_PV_OR_NULLINPUTT_PV_OR_NULLif (!SvOK($arg)) {$var = NULL;} else {$var = ($type)SvPV_nolen($arg);}
  • 53. There is also a muchtrickier way to do it.MODULE Foo PACKAGE FOOintfunc(char * str_or_null)
  • 54. If you dont get why thisworks, see perlref.INPUTT_PV$var = ${$var =~ /_or_null$/? qq{NULL}: qq{($type)SvPV_nolen($arg)}}
  • 55. Remove a star and prepend"OUT" (or "IN_OUT") for anargument called by reference.MODULE Foo PACKAGE FOOintfunc(char * str, OUT int len)
  • 56. Further Reading• perlxs• perlxstut• perlxstypemap• perlapi• perlcall• ... and others XS code :p
  • 57. Part IV: From .h filesto XS files
  • 58. Weve written XSfiles by hand so far.
  • 59. Of course this canbe done with a tool.
  • 60. The best-knowntool is h2xs.
  • 61. Its mainly used toexpose constantsfrom a C library.
  • 62. With the help ofC::Scan, you can use itto expose C functions aswell.$ h2xs -Axan Foo /path/to/header.h
  • 63. Defaulting to backwards compatibility with perl 5.xx.xIf you intend this module to be compatible withearlier perl versions, please specify a minimum perlversion with the -b option.Writing Foo/ppport.hScanning typemaps...Scanning /home/xxxx/perl5/perlbrew/perls/perl-5.xx.x/lib/5.xx.x/ExtUtils/typemapScanning /path/to/header.h for functions...Scanning /path/to/header.h for typedefs...Writing Foo/lib/Foo.pmWriting Foo/Foo.xsWriting Foo/typemapWriting Foo/Makefile.PLWriting Foo/READMEWriting Foo/t/Foo.tWriting Foo/ChangesWriting Foo/MANIFEST
  • 64. If the library is stable, this maygive you a good starting point.
  • 65. However.
  • 66. The files h2xs generatesare rather old-fashioned.
  • 67. Its not good at updatingonly a part of adistribution, either.
  • 68. I started writingConvert::H::XS.https://github.com/charsbar/convert_h_xs
  • 69. Convert::H::XS takes a C header file,and looks for the minimuminformation to write XS components.use Convert::H::XS;my $converter = Convert::H::XS->new;$converter->process($h_file);
  • 70. Convert::H::XS also providesmethods to write XS componentsfor convenience.$converter->write_constants("xs/constants.inc",);
  • 71. You can tweak thosepieces of informationwith a callback.$converter->write_functions("xs/functions.inc",sub {my ($type, $name, $args) = @_;...return ($type, $name, $args);});
  • 72. Grateful if you helpme improve this atthe hackathon.
  • 73. Part V: CPANTSupdate
  • 74. I talked aboutCPANTS last year.
  • 75. CPANTS tests what moduleauthors often forget to testby themselves.
  • 76. Perl QA Hackathon2013 in Lancaster
  • 77. Several metrics aregoing to be removed.
  • 78. Several metrics aregoing to be added.
  • 79. To add new metrics,we need to findissues.
  • 80. Issues are often raised byother QA/toolchain people.• Do not ship modules with Module::Install 1.04http://weblog.bulknews.net/post/33907905561/do-not-ship-modules-with-module-install-1-04• stop shipping MYMETA to CPANhttp://weblog.bulknews.net/post/44251476706/stop-shipping-mymeta-to-cpan
  • 81. We need to confirmthey are measurable,and widespread.
  • 82. What can I use toconfirm?
  • 83. • CPANTS databases• metacpan• CPAN grep
  • 84. I just wantedsomething else.
  • 85. Groonga• a fulltext search engine• a column store• actively developed (monthly release)• client/server (http/gqpt)• C API• Web interface
  • 86. Groonga::APIhttps://github.com/charsbar/groonga-api
  • 87. Also grateful if youhelp me improve thisat the hackathon.
  • 88. Thank you!