Wine label from Perl(located in Germany)Thanks , Wendy and Liz!
... and MST
typemap in Perl/XSKenichi Ishigaki(charsbar)@OSDC.TW 2013April 19, 2013
Let me ask youfirst.
Have you ever writtenan extension thatbridges C and Perl?
Part I: XS is ...?
Perl has a feature to loadan external C library.use DynaLoader;my @paths = dl_findfile(...);my $lib = dl_load_file($path);
This wont work correctlyunless the library conformswith Perls convention.use DynaLoader;my @paths = dl_findfile(...);my $l...
So we need somethingin-between.Perl XS C library
Writing the one-in-between is fairlyeasy.
(as long as you dontadd anything extra)
The simplest form of the one-in-between starts like this:#include "EXTERN.h"#include "perl.h"#include "XSUB.h"#include "pp...
These three lines are to usePerl API.#include "EXTERN.h"#include "perl.h"#include "XSUB.h"#include "ppport.h"#include <foo...
This is for portabilitybetween perls.#include "EXTERN.h"#include "perl.h"#include "XSUB.h"#include "ppport.h"#include <foo...
This is to import C functionsfrom the library.#include "EXTERN.h"#include "perl.h"#include "XSUB.h"#include "ppport.h"#inc...
Module and packagedeclarations.#include "EXTERN.h"#include "perl.h"#include "XSUB.h"#include "ppport.h"#include <foo.h>MOD...
The function declarations toexport (XSUBs) follow.#include "EXTERN.h"#include "perl.h"#include "XSUB.h"#include "ppport.h"...
As with .h files, there shouldbe only declarations.#include "EXTERN.h"#include "perl.h"#include "XSUB.h"#include "ppport.h...
Nothing difficult(in principle).
The library you usemay not always bethat simple.
People may addsomething written in Cwith a variety of PerlAPI.
If someones XScode scares you,please remember.
Whats difficult isnot the XSinterface itself.
Its something elsethat makes themattercomplicated.
Dont hesitate writingXS interface if youfind a useful Clibrary.
Part II: How .xs file is compiled
To build an interfacelibrary, we need toturn its XS declarationinto pure C code.
If you already haveMakefile.PL, run it, andthen, make.$ perl Makefile.PL && make
If you have Build.PL, dosomething like this (ormaybe ./Build build).$ perl Build.PL && ./Build
If you dont have either,Milla or Minilla will helpyou (ask miyagawa-san).
The most important XSUB partwill be translated like this:XS_EUPXS(XS_Foo_func){dVAR; dXSARGS;if (items != 1)croak_xs_usage...
A function exposed to the Perl worldtakes a Perl variable as its argument.XS_EUPXS(XS_Foo_func){dVAR; dXSARGS;if (items !=...
However, a Perl variable isactually a structure in C.XS_EUPXS(XS_Foo_func){dVAR; dXSARGS;if (items != 1)croak_xs_usage(cv,...
You cant pass it directly to a Cfunction, and vice versa.XS_EUPXS(XS_Foo_func){dVAR; dXSARGS;if (items != 1)croak_xs_usage...
The Perl variable is convertedand cast into a C value here.XS_EUPXS(XS_Foo_func){dVAR; dXSARGS;if (items != 1)croak_xs_usa...
Then, the function importedfrom the library is called.XS_EUPXS(XS_Foo_func){dVAR; dXSARGS;if (items != 1)croak_xs_usage(cv...
And the return value is convertedinto a Perl value again.XS_EUPXS(XS_Foo_func){dVAR; dXSARGS;if (items != 1)croak_xs_usage...
typemap is thekey to thisconversion.
Basic type mappingsare defined inExtUtils::typemap.$ perldoc -m ExtUtils::typemap
# basic C typesint T_IVunsigned T_UVunsigned int T_UVlong T_IVunsigned long T_UVshort T_IVunsigned short T_UVchar T_CHARun...
# bool_t is defined in <rpc/rpc.h>bool_t T_IVsize_t T_UVssize_t T_IVtime_t T_NVunsigned long * T_OPAQUEPTRchar ** T_PACKED...
Any type not listed thereshould be added locally,in a file named"typemap".
Part III: On type mapping
A typical"typemap" file hasthree sections.
"TYPEMAP" section containsa pair of C type and XStype.TYPEMAPchar * T_PV
How to map should be describedin the "INPUT" and "OUTPUT"sections, if necessary.INPUTT_PV$var = ($type)SvPV_nolen($arg)OUT...
These code fragments will beinterpolated while processingan XS file.INPUTT_PV$var = ($type)SvPV_nolen($arg)OUTPUTT_PVsv_se...
Adding orphan C typesis easy.TYPEMAPmy_id T_IVmy_str T_PV
Modifying thebehavior of anexisting type needssome consideration.
If you are confident, justadd a new code fragment inyour local typemap.INPUTT_PVif (!SvOK($arg)) {$var = NULL;} else {$var...
Or, you might want to add anothertypedef in the XS file, beforeMODULE and PACKAGE declarations.typedef char * my_nullable_...
And then, add a behavior of thisnew type in your local typemap.TYPEMAPmy_nullable_str T_PV_OR_NULLINPUTT_PV_OR_NULLif (!Sv...
There is also a muchtrickier way to do it.MODULE Foo PACKAGE FOOintfunc(char * str_or_null)
If you dont get why thisworks, see perlref.INPUTT_PV$var = ${$var =~ /_or_null$/? qq{NULL}: qq{($type)SvPV_nolen($arg)}}
Remove a star and prepend"OUT" (or "IN_OUT") for anargument called by reference.MODULE Foo PACKAGE FOOintfunc(char * str, ...
Further Reading• perlxs• perlxstut• perlxstypemap• perlapi• perlcall• ... and others XS code :p
Part IV: From .h filesto XS files
Weve written XSfiles by hand so far.
Of course this canbe done with a tool.
The best-knowntool is h2xs.
Its mainly used toexpose constantsfrom a C library.
With the help ofC::Scan, you can use itto expose C functions aswell.$ h2xs -Axan Foo /path/to/header.h
Defaulting to backwards compatibility with perl 5.xx.xIf you intend this module to be compatible withearlier perl versions...
If the library is stable, this maygive you a good starting point.
However.
The files h2xs generatesare rather old-fashioned.
Its not good at updatingonly a part of adistribution, either.
I started writingConvert::H::XS.https://github.com/charsbar/convert_h_xs
Convert::H::XS takes a C header file,and looks for the minimuminformation to write XS components.use Convert::H::XS;my $co...
Convert::H::XS also providesmethods to write XS componentsfor convenience.$converter->write_constants("xs/constants.inc",);
You can tweak thosepieces of informationwith a callback.$converter->write_functions("xs/functions.inc",sub {my ($type, $na...
Grateful if you helpme improve this atthe hackathon.
Part V: CPANTSupdate
I talked aboutCPANTS last year.
CPANTS tests what moduleauthors often forget to testby themselves.
Perl QA Hackathon2013 in Lancaster
Several metrics aregoing to be removed.
Several metrics aregoing to be added.
To add new metrics,we need to findissues.
Issues are often raised byother QA/toolchain people.• Do not ship modules with Module::Install 1.04http://weblog.bulknews....
We need to confirmthey are measurable,and widespread.
What can I use toconfirm?
• CPANTS databases• metacpan• CPAN grep
I just wantedsomething else.
Groonga• a fulltext search engine• a column store• actively developed (monthly release)• client/server (http/gqpt)• C API•...
Groonga::APIhttps://github.com/charsbar/groonga-api
Also grateful if youhelp me improve thisat the hackathon.
Thank you!
Upcoming SlideShare
Loading in...5
×

typemap in Perl/XS

701

Published on

for OSDC.TW 2013

Published in: Technology
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
701
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
11
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide
  • # and of course, it usually doesn&apos;t.
  • # As you see, it&apos;s almost the same as declared in the .h files.
  • # so that a C compiler can compile.
  • # and should be.
  • # information on defines, includes, functions, typedefs, structs etc is stored in $converter&apos;s stash.
  • typemap in Perl/XS

    1. 1. Wine label from Perl(located in Germany)Thanks , Wendy and Liz!
    2. 2. ... and MST
    3. 3. typemap in Perl/XSKenichi Ishigaki(charsbar)@OSDC.TW 2013April 19, 2013
    4. 4. Let me ask youfirst.
    5. 5. Have you ever writtenan extension thatbridges C and Perl?
    6. 6. Part I: XS is ...?
    7. 7. Perl has a feature to loadan external C library.use DynaLoader;my @paths = dl_findfile(...);my $lib = dl_load_file($path);
    8. 8. This wont work correctlyunless the library conformswith Perls convention.use DynaLoader;my @paths = dl_findfile(...);my $lib = dl_load_file($path);
    9. 9. So we need somethingin-between.Perl XS C library
    10. 10. Writing the one-in-between is fairlyeasy.
    11. 11. (as long as you dontadd anything extra)
    12. 12. The simplest form of the one-in-between starts like this:#include "EXTERN.h"#include "perl.h"#include "XSUB.h"#include "ppport.h"#include <foo.h>MODULE = Foo PACKAGE = Foointfunc(const char* str)
    13. 13. These three lines are to usePerl API.#include "EXTERN.h"#include "perl.h"#include "XSUB.h"#include "ppport.h"#include <foo.h>MODULE = Foo PACKAGE = Foointfunc(const char* str)
    14. 14. This is for portabilitybetween perls.#include "EXTERN.h"#include "perl.h"#include "XSUB.h"#include "ppport.h"#include <foo.h>MODULE = Foo PACKAGE = Foointfunc(const char* str)
    15. 15. This is to import C functionsfrom the library.#include "EXTERN.h"#include "perl.h"#include "XSUB.h"#include "ppport.h"#include <foo.h>MODULE = Foo PACKAGE = Foointfunc(const char* str)
    16. 16. Module and packagedeclarations.#include "EXTERN.h"#include "perl.h"#include "XSUB.h"#include "ppport.h"#include <foo.h>MODULE = Foo PACKAGE = Foointfunc(const char* str)
    17. 17. The function declarations toexport (XSUBs) follow.#include "EXTERN.h"#include "perl.h"#include "XSUB.h"#include "ppport.h"#include <foo.h>MODULE = Foo PACKAGE = Foointfunc(const char* str)
    18. 18. As with .h files, there shouldbe only declarations.#include "EXTERN.h"#include "perl.h"#include "XSUB.h"#include "ppport.h"#include <foo.h>MODULE = Foo PACKAGE = Foointfunc(const char* str)
    19. 19. Nothing difficult(in principle).
    20. 20. The library you usemay not always bethat simple.
    21. 21. People may addsomething written in Cwith a variety of PerlAPI.
    22. 22. If someones XScode scares you,please remember.
    23. 23. Whats difficult isnot the XSinterface itself.
    24. 24. Its something elsethat makes themattercomplicated.
    25. 25. Dont hesitate writingXS interface if youfind a useful Clibrary.
    26. 26. Part II: How .xs file is compiled
    27. 27. To build an interfacelibrary, we need toturn its XS declarationinto pure C code.
    28. 28. If you already haveMakefile.PL, run it, andthen, make.$ perl Makefile.PL && make
    29. 29. If you have Build.PL, dosomething like this (ormaybe ./Build build).$ perl Build.PL && ./Build
    30. 30. If you dont have either,Milla or Minilla will helpyou (ask miyagawa-san).
    31. 31. The most important XSUB partwill be translated like this:XS_EUPXS(XS_Foo_func){dVAR; dXSARGS;if (items != 1)croak_xs_usage(cv, "str");{int RETVAL;dXSTARG;const char* str = (const char *)SvPV_nolen(ST(0));RETVAL = func(str);XSprePUSH; PUSHi((IV)RETVAL);}XSRETURN(1);}
    32. 32. A function exposed to the Perl worldtakes a Perl variable as its argument.XS_EUPXS(XS_Foo_func){dVAR; dXSARGS;if (items != 1)croak_xs_usage(cv, "str");{int RETVAL;dXSTARG;const char* str = (const char *)SvPV_nolen(ST(0));RETVAL = func(str);XSprePUSH; PUSHi((IV)RETVAL);}XSRETURN(1);}
    33. 33. However, a Perl variable isactually a structure in C.XS_EUPXS(XS_Foo_func){dVAR; dXSARGS;if (items != 1)croak_xs_usage(cv, "str");{int RETVAL;dXSTARG;const char* str = (const char *)SvPV_nolen(ST(0));RETVAL = func(str);XSprePUSH; PUSHi((IV)RETVAL);}XSRETURN(1);}
    34. 34. You cant pass it directly to a Cfunction, and vice versa.XS_EUPXS(XS_Foo_func){dVAR; dXSARGS;if (items != 1)croak_xs_usage(cv, "str");{int RETVAL;dXSTARG;const char* str = (const char *)SvPV_nolen(ST(0));RETVAL = func(str);XSprePUSH; PUSHi((IV)RETVAL);}XSRETURN(1);}
    35. 35. The Perl variable is convertedand cast into a C value here.XS_EUPXS(XS_Foo_func){dVAR; dXSARGS;if (items != 1)croak_xs_usage(cv, "str");{int RETVAL;dXSTARG;const char* str = (const char *)SvPV_nolen(ST(0));RETVAL = func(str);XSprePUSH; PUSHi((IV)RETVAL);}XSRETURN(1);}
    36. 36. Then, the function importedfrom the library is called.XS_EUPXS(XS_Foo_func){dVAR; dXSARGS;if (items != 1)croak_xs_usage(cv, "str");{int RETVAL;dXSTARG;const char* str = (const char *)SvPV_nolen(ST(0));RETVAL = func(str);XSprePUSH; PUSHi((IV)RETVAL);}XSRETURN(1);}
    37. 37. And the return value is convertedinto a Perl value again.XS_EUPXS(XS_Foo_func){dVAR; dXSARGS;if (items != 1)croak_xs_usage(cv, "str");{int RETVAL;dXSTARG;const char* str = (const char *)SvPV_nolen(ST(0));RETVAL = func(str);XSprePUSH; PUSHi((IV)RETVAL);}XSRETURN(1);}
    38. 38. typemap is thekey to thisconversion.
    39. 39. Basic type mappingsare defined inExtUtils::typemap.$ perldoc -m ExtUtils::typemap
    40. 40. # basic C typesint T_IVunsigned T_UVunsigned int T_UVlong T_IVunsigned long T_UVshort T_IVunsigned short T_UVchar T_CHARunsigned char T_U_CHARchar * T_PVunsigned char * T_PVconst char * T_PVcaddr_t T_PVwchar_t * T_PVwchar_t T_IV
    41. 41. # bool_t is defined in <rpc/rpc.h>bool_t T_IVsize_t T_UVssize_t T_IVtime_t T_NVunsigned long * T_OPAQUEPTRchar ** T_PACKEDARRAYvoid * T_PTRTime_t * T_PVSV * T_SV
    42. 42. Any type not listed thereshould be added locally,in a file named"typemap".
    43. 43. Part III: On type mapping
    44. 44. A typical"typemap" file hasthree sections.
    45. 45. "TYPEMAP" section containsa pair of C type and XStype.TYPEMAPchar * T_PV
    46. 46. How to map should be describedin the "INPUT" and "OUTPUT"sections, if necessary.INPUTT_PV$var = ($type)SvPV_nolen($arg)OUTPUTT_PVsv_setpv((SV*)$arg, $var);
    47. 47. These code fragments will beinterpolated while processingan XS file.INPUTT_PV$var = ($type)SvPV_nolen($arg)OUTPUTT_PVsv_setpv((SV*)$arg, $var);
    48. 48. Adding orphan C typesis easy.TYPEMAPmy_id T_IVmy_str T_PV
    49. 49. Modifying thebehavior of anexisting type needssome consideration.
    50. 50. If you are confident, justadd a new code fragment inyour local typemap.INPUTT_PVif (!SvOK($arg)) {$var = NULL;} else {$var = ($type)SvPV_nolen($arg);}
    51. 51. Or, you might want to add anothertypedef in the XS file, beforeMODULE and PACKAGE declarations.typedef char * my_nullable_str;MODULE Foo PACKAGE FOOintfunc(my_nullable_str str)
    52. 52. And then, add a behavior of thisnew type in your local typemap.TYPEMAPmy_nullable_str T_PV_OR_NULLINPUTT_PV_OR_NULLif (!SvOK($arg)) {$var = NULL;} else {$var = ($type)SvPV_nolen($arg);}
    53. 53. There is also a muchtrickier way to do it.MODULE Foo PACKAGE FOOintfunc(char * str_or_null)
    54. 54. If you dont get why thisworks, see perlref.INPUTT_PV$var = ${$var =~ /_or_null$/? qq{NULL}: qq{($type)SvPV_nolen($arg)}}
    55. 55. Remove a star and prepend"OUT" (or "IN_OUT") for anargument called by reference.MODULE Foo PACKAGE FOOintfunc(char * str, OUT int len)
    56. 56. Further Reading• perlxs• perlxstut• perlxstypemap• perlapi• perlcall• ... and others XS code :p
    57. 57. Part IV: From .h filesto XS files
    58. 58. Weve written XSfiles by hand so far.
    59. 59. Of course this canbe done with a tool.
    60. 60. The best-knowntool is h2xs.
    61. 61. Its mainly used toexpose constantsfrom a C library.
    62. 62. With the help ofC::Scan, you can use itto expose C functions aswell.$ h2xs -Axan Foo /path/to/header.h
    63. 63. Defaulting to backwards compatibility with perl 5.xx.xIf you intend this module to be compatible withearlier perl versions, please specify a minimum perlversion with the -b option.Writing Foo/ppport.hScanning typemaps...Scanning /home/xxxx/perl5/perlbrew/perls/perl-5.xx.x/lib/5.xx.x/ExtUtils/typemapScanning /path/to/header.h for functions...Scanning /path/to/header.h for typedefs...Writing Foo/lib/Foo.pmWriting Foo/Foo.xsWriting Foo/typemapWriting Foo/Makefile.PLWriting Foo/READMEWriting Foo/t/Foo.tWriting Foo/ChangesWriting Foo/MANIFEST
    64. 64. If the library is stable, this maygive you a good starting point.
    65. 65. However.
    66. 66. The files h2xs generatesare rather old-fashioned.
    67. 67. Its not good at updatingonly a part of adistribution, either.
    68. 68. I started writingConvert::H::XS.https://github.com/charsbar/convert_h_xs
    69. 69. Convert::H::XS takes a C header file,and looks for the minimuminformation to write XS components.use Convert::H::XS;my $converter = Convert::H::XS->new;$converter->process($h_file);
    70. 70. Convert::H::XS also providesmethods to write XS componentsfor convenience.$converter->write_constants("xs/constants.inc",);
    71. 71. You can tweak thosepieces of informationwith a callback.$converter->write_functions("xs/functions.inc",sub {my ($type, $name, $args) = @_;...return ($type, $name, $args);});
    72. 72. Grateful if you helpme improve this atthe hackathon.
    73. 73. Part V: CPANTSupdate
    74. 74. I talked aboutCPANTS last year.
    75. 75. CPANTS tests what moduleauthors often forget to testby themselves.
    76. 76. Perl QA Hackathon2013 in Lancaster
    77. 77. Several metrics aregoing to be removed.
    78. 78. Several metrics aregoing to be added.
    79. 79. To add new metrics,we need to findissues.
    80. 80. Issues are often raised byother QA/toolchain people.• Do not ship modules with Module::Install 1.04http://weblog.bulknews.net/post/33907905561/do-not-ship-modules-with-module-install-1-04• stop shipping MYMETA to CPANhttp://weblog.bulknews.net/post/44251476706/stop-shipping-mymeta-to-cpan
    81. 81. We need to confirmthey are measurable,and widespread.
    82. 82. What can I use toconfirm?
    83. 83. • CPANTS databases• metacpan• CPAN grep
    84. 84. I just wantedsomething else.
    85. 85. Groonga• a fulltext search engine• a column store• actively developed (monthly release)• client/server (http/gqpt)• C API• Web interface
    86. 86. Groonga::APIhttps://github.com/charsbar/groonga-api
    87. 87. Also grateful if youhelp me improve thisat the hackathon.
    88. 88. Thank you!
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×