for OSDC.TW 2013

1. Wine label from Perl
(located in Germany)
Thanks , Wendy and Liz!

2. ... and MST

3. typemap in Perl/XS
Kenichi Ishigaki
(charsbar)
@OSDC.TW 2013
April 19, 2013

4. Let me ask you
first.

5. Have you ever written
an extension that
bridges C and Perl?

6. Part I: XS is ...?

7. Perl has a feature to load
an external C library.
use DynaLoader;
my @paths = dl_findfile(...);
my $lib = dl_load_file($path);

8. This won't work correctly
unless the library conforms
with Perl's convention.
use DynaLoader;
my @paths = dl_findfile(...);
my $lib = dl_load_file($path);

9. So we need something
in-between.
Perl XS C library

10. Writing the one-in-
between is fairly
easy.

11. (as long as you don't
add anything extra)

12. The simplest form of the one-
in-between starts like this:
#include "EXTERN.h"
#include "perl.h"
#include "XSUB.h"
#include "ppport.h"
#include <foo.h>
MODULE = Foo PACKAGE = Foo
int
func(const char* str)

13. These three lines are to use
Perl API.
#include "EXTERN.h"
#include "perl.h"
#include "XSUB.h"
#include "ppport.h"
#include <foo.h>
MODULE = Foo PACKAGE = Foo
int
func(const char* str)

14. This is for portability
between perls.
#include "EXTERN.h"
#include "perl.h"
#include "XSUB.h"
#include "ppport.h"
#include <foo.h>
MODULE = Foo PACKAGE = Foo
int
func(const char* str)

15. This is to import C functions
from the library.
#include "EXTERN.h"
#include "perl.h"
#include "XSUB.h"
#include "ppport.h"
#include <foo.h>
MODULE = Foo PACKAGE = Foo
int
func(const char* str)

16. Module and package
declarations.
#include "EXTERN.h"
#include "perl.h"
#include "XSUB.h"
#include "ppport.h"
#include <foo.h>
MODULE = Foo PACKAGE = Foo
int
func(const char* str)

17. The function declarations to
export (XSUBs) follow.
#include "EXTERN.h"
#include "perl.h"
#include "XSUB.h"
#include "ppport.h"
#include <foo.h>
MODULE = Foo PACKAGE = Foo
int
func(const char* str)

18. As with .h files, there should
be only declarations.
#include "EXTERN.h"
#include "perl.h"
#include "XSUB.h"
#include "ppport.h"
#include <foo.h>
MODULE = Foo PACKAGE = Foo
int
func(const char* str)

19. Nothing difficult
(in principle).

20. The library you use
may not always be
that simple.

21. People may add
something written in C
with a variety of Perl
API.

22. If someone's XS
code scares you,
please remember.

23. What's difficult is
not the XS
interface itself.

24. It's something else
that makes the
matter
complicated.

25. Don't hesitate writing
XS interface if you
find a useful C
library.

26. Part II: How .xs file is compiled

27. To build an interface
library, we need to
turn its XS declaration
into pure C code.

28. If you already have
Makefile.PL, run it, and
then, make.
$ perl Makefile.PL && make

29. If you have Build.PL, do
something like this (or
maybe ./Build build).
$ perl Build.PL && ./Build

30. If you don't have either,
Milla or Minilla will help
you (ask miyagawa-san).

31. The most important XSUB part
will be translated like this:
XS_EUPXS(XS_Foo_func)
{
dVAR; dXSARGS;
if (items != 1)
croak_xs_usage(cv, "str");
{
int RETVAL;
dXSTARG;
const char* str = (const char *)SvPV_nolen(ST(0));
RETVAL = func(str);
XSprePUSH; PUSHi((IV)RETVAL);
}
XSRETURN(1);
}

32. A function exposed to the Perl world
takes a Perl variable as its argument.
XS_EUPXS(XS_Foo_func)
{
dVAR; dXSARGS;
if (items != 1)
croak_xs_usage(cv, "str");
{
int RETVAL;
dXSTARG;
const char* str = (const char *)SvPV_nolen(ST(0));
RETVAL = func(str);
XSprePUSH; PUSHi((IV)RETVAL);
}
XSRETURN(1);
}

33. However, a Perl variable is
actually a structure in C.
XS_EUPXS(XS_Foo_func)
{
dVAR; dXSARGS;
if (items != 1)
croak_xs_usage(cv, "str");
{
int RETVAL;
dXSTARG;
const char* str = (const char *)SvPV_nolen(ST(0));
RETVAL = func(str);
XSprePUSH; PUSHi((IV)RETVAL);
}
XSRETURN(1);
}

34. You can't pass it directly to a C
function, and vice versa.
XS_EUPXS(XS_Foo_func)
{
dVAR; dXSARGS;
if (items != 1)
croak_xs_usage(cv, "str");
{
int RETVAL;
dXSTARG;
const char* str = (const char *)SvPV_nolen(ST(0));
RETVAL = func(str);
XSprePUSH; PUSHi((IV)RETVAL);
}
XSRETURN(1);
}

35. The Perl variable is converted
and cast into a C value here.
XS_EUPXS(XS_Foo_func)
{
dVAR; dXSARGS;
if (items != 1)
croak_xs_usage(cv, "str");
{
int RETVAL;
dXSTARG;
const char* str = (const char *)SvPV_nolen(ST(0));
RETVAL = func(str);
XSprePUSH; PUSHi((IV)RETVAL);
}
XSRETURN(1);
}

36. Then, the function imported
from the library is called.
XS_EUPXS(XS_Foo_func)
{
dVAR; dXSARGS;
if (items != 1)
croak_xs_usage(cv, "str");
{
int RETVAL;
dXSTARG;
const char* str = (const char *)SvPV_nolen(ST(0));
RETVAL = func(str);
XSprePUSH; PUSHi((IV)RETVAL);
}
XSRETURN(1);
}

37. And the return value is converted
into a Perl value again.
XS_EUPXS(XS_Foo_func)
{
dVAR; dXSARGS;
if (items != 1)
croak_xs_usage(cv, "str");
{
int RETVAL;
dXSTARG;
const char* str = (const char *)SvPV_nolen(ST(0));
RETVAL = func(str);
XSprePUSH; PUSHi((IV)RETVAL);
}
XSRETURN(1);
}

38. typemap is the
key to this
conversion.

39. Basic type mappings
are defined in
ExtUtils::typemap.
$ perldoc -m ExtUtils::typemap

40. # basic C types
int T_IV
unsigned T_UV
unsigned int T_UV
long T_IV
unsigned long T_UV
short T_IV
unsigned short T_UV
char T_CHAR
unsigned char T_U_CHAR
char * T_PV
unsigned char * T_PV
const char * T_PV
caddr_t T_PV
wchar_t * T_PV
wchar_t T_IV

41. # bool_t is defined in <rpc/rpc.h>
bool_t T_IV
size_t T_UV
ssize_t T_IV
time_t T_NV
unsigned long * T_OPAQUEPTR
char ** T_PACKEDARRAY
void * T_PTR
Time_t * T_PV
SV * T_SV

42. Any type not listed there
should be added locally,
in a file named
"typemap".

43. Part III: On type mapping

44. A typical
"typemap" file has
three sections.

45. "TYPEMAP" section contains
a pair of C type and XS
type.
TYPEMAP
char * T_PV

46. How to map should be described
in the "INPUT" and "OUTPUT"
sections, if necessary.
INPUT
T_PV
$var = ($type)SvPV_nolen($arg)
OUTPUT
T_PV
sv_setpv((SV*)$arg, $var);

47. These code fragments will be
interpolated while processing
an XS file.
INPUT
T_PV
$var = ($type)SvPV_nolen($arg)
OUTPUT
T_PV
sv_setpv((SV*)$arg, $var);

48. Adding orphan C types
is easy.
TYPEMAP
my_id T_IV
my_str T_PV

49. Modifying the
behavior of an
existing type needs
some consideration.

50. If you are confident, just
add a new code fragment in
your local typemap.
INPUT
T_PV
if (!SvOK($arg)) {
$var = NULL;
} else {
$var = ($type)SvPV_nolen($arg);
}

51. Or, you might want to add another
typedef in the XS file, before
MODULE and PACKAGE declarations.
typedef char * my_nullable_str;
MODULE Foo PACKAGE FOO
int
func(my_nullable_str str)

52. And then, add a behavior of this
new type in your local typemap.
TYPEMAP
my_nullable_str T_PV_OR_NULL
INPUT
T_PV_OR_NULL
if (!SvOK($arg)) {
$var = NULL;
} else {
$var = ($type)SvPV_nolen($arg);
}

53. There is also a much
trickier way to do it.
MODULE Foo PACKAGE FOO
int
func(char * str_or_null)

54. If you don't get why this
works, see perlref.
INPUT
T_PV
$var = ${
$var =~ /_or_null$/
? qq{NULL}
: qq{($type)SvPV_nolen($arg)}
}

55. Remove a star and prepend
"OUT" (or "IN_OUT") for an
argument called by reference.
MODULE Foo PACKAGE FOO
int
func(char * str, OUT int len)

56. Further Reading
• perlxs
• perlxstut
• perlxstypemap
• perlapi
• perlcall
• ... and others' XS code :p

57. Part IV: From .h files
to XS files

58. We've written XS
files by hand so far.

59. Of course this can
be done with a tool.

60. The best-known
tool is h2xs.

61. It's mainly used to
expose constants
from a C library.

62. With the help of
C::Scan, you can use it
to expose C functions as
well.
$ h2xs -Axan Foo /path/to/header.h

63. Defaulting to backwards compatibility with perl 5.xx.x
If you intend this module to be compatible with
earlier perl versions, please specify a minimum perl
version with the -b option.
Writing Foo/ppport.h
Scanning typemaps...
Scanning /home/xxxx/perl5/perlbrew/perls/perl-
5.xx.x/lib/5.xx.x/ExtUtils/typemap
Scanning /path/to/header.h for functions...
Scanning /path/to/header.h for typedefs...
Writing Foo/lib/Foo.pm
Writing Foo/Foo.xs
Writing Foo/typemap
Writing Foo/Makefile.PL
Writing Foo/README
Writing Foo/t/Foo.t
Writing Foo/Changes
Writing Foo/MANIFEST

64. If the library is stable, this may
give you a good starting point.

65. However.

66. The files h2xs generates
are rather old-
fashioned.

67. It's not good at updating
only a part of a
distribution, either.

68. I started writing
Convert::H::XS.
https://github.com/charsbar/convert_h_xs

69. Convert::H::XS takes a C header file,
and looks for the minimum
information to write XS components.
use Convert::H::XS;
my $converter = Convert::H::XS->new;
$converter->process($h_file);

70. Convert::H::XS also provides
methods to write XS components
for convenience.
$converter->write_constants(
"xs/constants.inc",
);

71. You can tweak those
pieces of information
with a callback.
$converter->write_functions(
"xs/functions.inc",
sub {
my ($type, $name, $args) = @_;
...
return ($type, $name, $args);
});

72. Grateful if you help
me improve this at
the hackathon.

73. Part V: CPANTS
update

74. I talked about
CPANTS last year.

75. CPANTS tests what module
authors often forget to test
by themselves.

76. Perl QA Hackathon
2013 in Lancaster

77. Several metrics are
going to be removed.

78. Several metrics are
going to be added.

79. To add new metrics,
we need to find
issues.

80. Issues are often raised by
other QA/toolchain people.
• Do not ship modules with Module::Install 1.04
http://weblog.bulknews.net/post/33907905561/do-not-ship-modules-with-module-install-1-04
• stop shipping MYMETA to CPAN
http://weblog.bulknews.net/post/44251476706/stop-shipping-mymeta-to-cpan

81. We need to confirm
they are measurable,
and widespread.

82. What can I use to
confirm?

83. • CPANTS databases
• metacpan
• CPAN grep

84. I just wanted
something else.

85. Groonga
• a fulltext search engine
• a column store
• actively developed (monthly release)
• client/server (http/gqpt)
• C API
• Web interface

86. Groonga::API
https://github.com/charsbar/groonga-api

87. Also grateful if you
help me improve this
at the hackathon.

88. Thank you!