A brief introduction to get you started in working with Perl's internal API. This presentation is a work in progress. Code samples: http://github.com/friedo/perl-api-terror

1. The Perl API
for the
Mortally Terrified
Mike Friedman
(friedo)
London Perl Workshop
November 30, 2013

2. Me
Mike Friedman

3. Mike Friedman
(friedo)

4. This talk
is the product of a lengthy journey.

5. This talk
is the product of a lengthy journey.
It's also a work in progress.

6. I have some limited
experience in integrating
Perl and C with XS, and
would enjoy the opportunity
to expand my skills in this
area.

7. i R dum.

9. - Learned from this book.

10. - Learned from this book.
- It’s very good

11. - Learned from this book.
- It’s very good
- The printing I got was messed up.

12. - Learned from this book.
- It’s very good
- The printing I got was messed up.
- Like every " was replaced by ©

13. The Perl API
What’s it good for?
Make things fast:

14. The Perl API
What’s it good for?
Make things fast:
- Write your slow Perl code in C

15. The Perl API
What’s it good for?
Make things fast:
- Write your slow Perl code in C
- Keep your fast Perl code in Perl

16. The Perl API
What’s it good for?
Make Perl more useful:

17. The Perl API
What’s it good for?
Make Perl more useful:
- Interface with libraries written in C

18. The Perl API
What’s it good for?
Make Perl more useful:
- Interface with libraries written in C
- Create modules exposing external APIs to Perl

19. The Perl API
What’s it good for?
Make Perl do...stuff

20. The Perl API
What’s it good for?
Make Perl do...stuff
- Change opcodes on the fly?

21. The Perl API
What’s it good for?
Make Perl do...stuff
- Change opcodes on the fly?
- Extend the parser?

22. The Perl API
What’s it good for?
Make Perl do...stuff
- Change opcodes on the fly?
- Extend the parser?
- Make exceptions work?

23. The Perl API
What’s it good for?
Make Perl do...stuff
- Change opcodes on the fly?
- Extend the parser?
- Make exceptions work?
- Why not?

24. The Perl API
What’s it good for?
Make Perl do...stuff
- Change opcodes on the fly?
- Extend the parser?
- Make exceptions work?
- Why not?

25. A Crash Course in
C

26. C Crash Course
Basic Types (there are more)

27. C Crash Course
Basic Types (there are more)
char
foo
= ‘a’;

28. C Crash Course
Basic Types (there are more)
char
int
foo
bar
= ‘a’;
= 42;

29. C Crash Course
Basic Types (there are more)
char
int
long
foo
bar
baz
= ‘a’;
= 42;
= 198547;

30. C Crash Course
Basic Types (there are more)
char
int
long
float
foo
bar
baz
quux
=
=
=
=
‘a’;
42;
198547;
3.14159;

31. C Crash Course
Basic Types (there are more)
char
int
long
float
double
foo
bar
baz
quux
splat
=
=
=
=
=
‘a’;
42;
198547;
3.14159;
3.14159265358979;

32. C Crash Course
Pointers

33. C Crash Course
Pointers
char
*foo;
/* ptr to char
*/

34. C Crash Course
Pointers
char
int
*foo;
*bar;
/* ptr to char
/* ptr to int
*/
*/

35. C Crash Course
Pointers
char
int
long
*foo;
*bar;
*baz;
/* ptr to char
/* ptr to int
/* ptr to long
*/
*/
*/

36. C Crash Course
Pointers
char
int
long
float
*foo;
*bar;
*baz;
*quux;
/*
/*
/*
/*
ptr
ptr
ptr
ptr
to
to
to
to
char
int
long
float
*/
*/
*/
*/

37. C Crash Course
Pointers
char
int
long
float
double
*foo;
*bar;
*baz;
*quux;
*splat;
/*
/*
/*
/*
/*
ptr
ptr
ptr
ptr
ptr
to
to
to
to
to
char
int
long
float
double
*/
*/
*/
*/
*/

38. C Crash Course
Pointers
char
int
long
float
double
void
*foo;
*bar;
*baz;
*quux;
*splat;
*fnarf;
/*
/*
/*
/*
/*
/*
ptr
ptr
ptr
ptr
ptr
ptr
to
to
to
to
to
to
char
int
long
float
double
something
*/
*/
*/
*/
*/
*/

39. C Crash Course
Pointers are integer variables which
hold a memory address.

40. C Crash Course
C
Perl

41. C Crash Course
C
int i
Perl
= 42;
my $i
= 42;

42. C Crash Course
C
int i
= 42;
int *i_ptr = &i;
Perl
my $i
= 42;
my $i_ref = $i;

43. C Crash Course
C
Perl
int i
= 42;
my $i
= 42;
int *i_ptr = &i;
my $i_ref = $i;
Read unary & as “address-of.”

44. C Crash Course
C
Perl
int i
= 42;
my $i
= 42;
int *i_ptr = &i;
my $i_ref = $i;
Read unary & as “address-of.”
int j;
my $j;

45. C Crash Course
C
Perl
int i
= 42;
my $i
= 42;
int *i_ptr = &i;
my $i_ref = $i;
Read unary & as “address-of.”
int j;
j = *i_ptr;
my $j;
$j = $$i_ref;

46. C Crash Course
C
Perl
int i
= 42;
my $i
= 42;
int *i_ptr = &i;
my $i_ref = $i;
Read unary & as “address-of.”
int j;
j = *i_ptr;
my $j;
$j = $$i_ref;
Read unary * as “dereference.”

47. C Crash Course
Aggregates

48. C Crash Course
Aggregates
int
foo[10];
/* array of ints */

49. C Crash Course
Aggregates
int
float
foo[10];
bar[12];
/* array of ints */
/* array of floats */

50. C Crash Course
Aggregates
int
float
char
foo[10];
bar[12];
baz[42];
/* array of ints */
/* array of floats */
/* a string! */

51. C Crash Course
Aggregates

52. C Crash Course
Aggregates
struct person {
int age;
char *name;
double acct_balance;
};

53. C Crash Course
Aggregates
struct person {
int age;
char *name;
double acct_balance;
};
age
*name
acct_balance
...

54. C Crash Course
Aggregates

55. C Crash Course
Aggregates
union thingy {
long num;
char chars[4];
foo_t *myfoo;
};

56. C Crash Course
Aggregates
union thingy {
long num;
char chars[4];
foo_t *myfoo;
};
thingy.num
thingy.chars
thingy.myfoo
f3
de
42
9a

57. C Crash Course
Using structs

58. C Crash Course
Using structs
struct person the_dude;
the_dude.age = 42;
the_dude.name = “jeffrey lebowski”;
the_dude.acct_balance = 1.79;

59. C Crash Course
Using structs
struct person the_dude;
the_dude.age = 42;
the_dude.name = “jeffrey lebowski”;
the_dude.acct_balance = 1.79;
00
2a
age
3f
39
d2
90
a4
70
3d
*name
0a
d7
a3
fc
acct_balance
j
e
f
f
r
e
y
...
3f

60. C Crash Course
Typedefs
typedef struct {
int age;
char *name;
double acct_balance;
} person;
person the_dude;

61. XS

62. What is XS?
perldoc perlxs:
XS is an interface description file format used to create an
extension interface between Perl and C code (or a C library)
which one wishes to use with Perl. The XS interface is
combined with the library to create a new library which can
then be either dynamically loaded or statically linked into
perl. The XS interface description is written in the XS
language and is the core component of the Perl extension
interface.

64. XS is a specialized templating
language for C.

65. Some XS Code

66. Some XS Code
This stuff is XS.

67. Some C Code made by xsubpp

68. XS != Perl API

69. For most things

70. For most things
You don't actually
have to use XS.

71. (OK.You don’t have to
know that you’re using it.)

72. Inline::C

73. use Inline C;
hello('Mike');
hello('Cookie Monster');
__END__
__C__
void hello(char* name) {
printf("Hello %s!n", name);
}

74. use Inline C;
hello('Mike');
hello('Grover');
__END__
__C__
void hello(char* name) {
printf("Hello %s!n", name);
}

75. use Inline C;
hello('Mike');
hello('Grover');
__END__
__C__
void hello(char* name) {
printf("Hello %s!n", name);
}

76. use Inline C;
hello('Mike');
hello('Grover');
__END__
__C__
void hello(char* name) {
printf("Hello %s!n", name);
}

77. Inline::C Code in Perl

78. Inline::C Code in Perl
Digest::MD5

79. Inline::C Code in Perl
Digest::MD5
C Code
Parse::RecDescent

80. Inline::C Code in Perl
Digest::MD5
C Code
Parse::RecDescent
Create build in a temp dir

81. Inline::C Code in Perl
Digest::MD5
C Code
Parse::RecDescent
Create build in a temp dir
Write XS

82. Inline::C Code in Perl
Digest::MD5
C Code
Parse::RecDescent
Create build in a temp dir
Write XS
Compile and install

83. Inline::C Code in Perl
Digest::MD5
C Code
Parse::RecDescent
Create build in a temp dir
Write XS
Compile and install

84. How do we make
Perl do stuff in C?

85. How do we make
Perl do stuff in C?
We need to understand Perl’s data
structures.

86. Handwaving Ahead

87. A $scalar in Perl is represented by a struct called SV.
SvNULL
SvRV
SvPV
SvIV
(These are not all of them.)
SvNV

88. What does SV look like?
sv_refcnt
sv_flags
{
sv_any
flags
type

89. What does SV look like?
sv_refcnt
sv_flags
{
sv_any
flags
type
If this scalar is undef, then sv_any is NULL

90. What does SvPV (a string) look like?
sv_any
sv_refcnt
sv_flags
pvx
current len
alloc len
SV
xpv
char* a
s
t
r
i
n
g
0

91. What do SvIV and SvNV look like?
sv_any
sv_refcnt
sv_flags
pvx
current len
alloc len
SV
xpviv
69
7a
00
00
ivx
char* 3
1
3
3
7
0

92. What do SvIV and SvNV look like?
sv_any
sv_refcnt
sv_flags
pvx
current len
alloc len
SV
xpvnv
6e
86
1b
f0
f9
ivx
char* 3
.
21
09
40
nvx
1
4
1
5
9
0

93. WRITE CODE NOW
?!??!!11!?

94. Let's make
something fast.

95. #!/usr/bin/env perl
use strict;
use warnings;
use Inline 'C';
print 'Give me a number: ';
my $num_a = <STDIN>;
print 'Give me another number: ';
my $num_b = <STDIN>;
printf "The sum is: %sn", add( $num_a, $num_b );
__END__
__C__
int add( int a, int b ) {
return a + b;
}

96. #!/usr/bin/env perl
use strict;
use warnings;
use Inline 'C';
print 'Give me a number: ';
my $num_a = <STDIN>;
print 'Give me another number: ';
my $num_b = <STDIN>;
printf "The sum is: %sn", add( $num_a, $num_b );
__END__
__C__
int add( int a, int b ) {
return a + b;
}

97. #!/usr/bin/env perl
use strict;
use warnings;
use Inline 'C';
print 'Give me a number: ';
my $num_a = <STDIN>;
print 'Give me another number: ';
my $num_b = <STDIN>;
printf "The sum is: %sn", add( $num_a, $num_b );
__END__
__C__
int add( int a, int b ) {
return a + b;
}

98. #!/usr/bin/env perl
use strict;
use warnings;
use Inline 'C';
print 'Give me a number: ';
my $num_a = <STDIN>;
print 'Give me another number: ';
my $num_b = <STDIN>;
printf "The sum is: %sn", add( $num_a, $num_b );
__END__
__C__
int add( int a, int b ) {
return a + b;
}

99. #!/usr/bin/env perl
use strict;
use warnings;
use Inline 'C';
print 'Give me a number: ';
my $num_a = <STDIN>;
print 'Give me another number: ';
my $num_b = <STDIN>;
printf "The sum is: %sn", add( $num_a, $num_b );
__END__
__C__
int add( int a, int b ) {
return a + b;
}

100. #!/usr/bin/env perl
use strict;
use warnings;
use Inline 'C';
print 'Give me a number: ';
my $num_a = <STDIN>;
print 'Give me another number: ';
my $num_b = <STDIN>;
printf "The sum is: %sn", add( $num_a, $num_b );
__END__
__C__
int add( int a, int b ) {
return a + b;
}

101. Run it!
bash-3.2$ perl cexp1.pl
Give me a number: 42
Give me another number: 11
The sum is: 53

102. Run it again!
bash-3.2$ perl cexp1.pl
Give me a number: 42
Give me another number: 31.337
The sum is: 73

103. Run it again!
bash-3.2$ perl cexp1.pl
Give me a number: 42
Give me another number: 31.337
The sum is: 73
What happened?

104. Inline::C
is
CRAZY
SMART

105. int
int
int
int
Inline::C
is
CRAZY
SMART

106. use Inline 'C';
print 'Give me a number: ';
my $num_a = <STDIN>;
print 'Give me another number: ';
my $num_b = <STDIN>;
$num_a += 0; $num_b += 0;
printf "The sum is: %sn", add( $num_a, $num_b );
__END__
__C__
SV *add( SV *a, SV *b ) {
if ( SvIOK( a ) && SvIOK( b ) ) {
return newSViv( SvIV( a ) + SvIV( b ) );
} else if ( SvNOK( a ) && SvNOK( b ) ) {
return newSVnv( SvNV( a ) + SvNV( b ) );
} else {
croak( "I don't know what to do!" );
}
}

107. use Inline 'C';
print 'Give me a number: ';
my $num_a = <STDIN>;
print 'Give me another number: ';
my $num_b = <STDIN>;
$num_a += 0; $num_b += 0;
printf "The sum is: %sn", add( $num_a, $num_b );
__END__
__C__
SV *add( SV *a, SV *b ) {
if ( SvIOK( a ) && SvIOK( b ) ) {
return newSViv( SvIV( a ) + SvIV( b ) );
} else if ( SvNOK( a ) && SvNOK( b ) ) {
return newSVnv( SvNV( a ) + SvNV( b ) );
} else {
croak( "I don't know what to do!" );
}
}

108. use Inline 'C';
print 'Give me a number: ';
my $num_a = <STDIN>;
print 'Give me another number: ';
my $num_b = <STDIN>;
$num_a += 0; $num_b += 0;
printf "The sum is: %sn", add( $num_a, $num_b );
__END__
__C__
SV *add( SV *a, SV *b ) {
if ( SvIOK( a ) && SvIOK( b ) ) {
return newSViv( SvIV( a ) + SvIV( b ) );
} else if ( SvNOK( a ) && SvNOK( b ) ) {
return newSVnv( SvNV( a ) + SvNV( b ) );
} else {
croak( "I don't know what to do!" );
}
}

109. use Inline 'C';
print 'Give me a number: ';
my $num_a = <STDIN>;
print 'Give me another number: ';
my $num_b = <STDIN>;
$num_a += 0; $num_b += 0;
printf "The sum is: %sn", add( $num_a, $num_b );
__END__
__C__
SV *add( SV *a, SV *b ) {
if ( SvIOK( a ) && SvIOK( b ) ) {
return newSViv( SvIV( a ) + SvIV( b ) );
} else if ( SvNOK( a ) && SvNOK( b ) ) {
return newSVnv( SvNV( a ) + SvNV( b ) );
} else {
croak( "I don't know what to do!" );
}
}

110. Run it!
bash-3.2$ perl cexp2.pl
Give me a number: 42
Give me another number: 11
The sum is: 53
bash-3.2$ perl cexp2.pl

111. Run it!
bash-3.2$ perl cexp2.pl
Give me a number: 42
Give me another number: 11
The sum is: 53
bash-3.2$ perl cexp2.pl
bash-3.2$ perl cexp2.pl
Give me a number: 53.4
Give me another number: 6.54
The sum is: 59.94

112. Run it!
bash-3.2$ perl cexp2.pl
Give me a number: 42
Give me another number: 11
The sum is: 53
bash-3.2$ perl cexp2.pl
bash-3.2$ perl cexp2.pl
Give me a number: 53.4
Give me another number: 6.54
The sum is: 59.94
Give me a number: 42
Give me another number: 6.54
I don't know what to do! at
cexp2.pl line 16, <STDIN>
line 2.

113. Once you get this far,
everything you need
is in perlapi.

114. Let's do something
with arrays.

115. What does an AV look like?
sv_any
sv_refcnt
sv_flags
array
fill
max
SV
xpvav
*some fields omitted
alloc
array len
SV
SV*[]
SV
...
...

116. #!/usr/bin/env perl
use strict;
use warnings;
use Inline 'C';
print "Give me some numbers: ";
my @numbers = map { $_ += 0 }
split /,/, <STDIN>;
my $result = squares( @numbers );
printf "The squares are: %sn",
join ", ", @$result;
__END__

117. #!/usr/bin/env perl
use strict;
use warnings;
use Inline 'C';
print "Give me some numbers: ";
my @numbers = map { $_ += 0 }
split /,/, <STDIN>;
my $result = squares( @numbers );
printf "The squares are: %sn",
join ", ", @$result;
__END__

118. SV *squares( SV *numbers ) {
AV *array = (AV *)SvRV( numbers );
AV *return_array = newAV();
SV **tmp;
int len = AvFILL( array );
int i, val;
for( i = 0; i <= len; i++ ) {
tmp = av_fetch( array, i, 1 );
if( !SvIOK( *tmp ) ) {
croak( "Can't handle this value!" );
}
val = SvIV( *tmp );
val = val * val;
av_push( return_array, newSViv( val ) );
}
return newRV_inc( (SV *)return_array );
}

119. SV *squares( SV *numbers ) {
AV *array = (AV *)SvRV( numbers );
AV *return_array = newAV();
SV **tmp;
int len = AvFILL( array );
int i, val;
for( i = 0; i <= len; i++ ) {
tmp = av_fetch( array, i, 1 );
if( !SvIOK( *tmp ) ) {
croak( "Can't handle this value!" );
}
val = SvIV( *tmp );
val = val * val;
av_push( return_array, newSViv( val ) );
}
return newRV_inc( (SV *)return_array );
}

120. SV *squares( SV *numbers ) {
AV *array = (AV *)SvRV( numbers );
AV *return_array = newAV();
SV **tmp;
int len = AvFILL( array );
int i, val;
for( i = 0; i <= len; i++ ) {
tmp = av_fetch( array, i, 1 );
if( !SvIOK( *tmp ) ) {
croak( "Can't handle this value!" );
}
val = SvIV( *tmp );
val = val * val;
av_push( return_array, newSViv( val ) );
}
return newRV_inc( (SV *)return_array );
}

121. SV *squares( SV *numbers ) {
AV *array = (AV *)SvRV( numbers );
AV *return_array = newAV();
SV **tmp;
int len = AvFILL( array );
int i, val;
for( i = 0; i <= len; i++ ) {
tmp = av_fetch( array, i, 1 );
if( !SvIOK( *tmp ) ) {
croak( "Can't handle this value!" );
}
val = SvIV( *tmp );
val = val * val;
av_push( return_array, newSViv( val ) );
}
return newRV_inc( (SV *)return_array );
}

122. SV *squares( SV *numbers ) {
AV *array = (AV *)SvRV( numbers );
AV *return_array = newAV();
SV **tmp;
int len = AvFILL( array );
int i, val;
for( i = 0; i <= len; i++ ) {
tmp = av_fetch( array, i, 1 );
if( !SvIOK( *tmp ) ) {
croak( "Can't handle this value!" );
}
val = SvIV( *tmp );
val = val * val;
av_push( return_array, newSViv( val ) );
}
return newRV_inc( (SV *)return_array );
}

123. SV *squares( SV *numbers ) {
AV *array = (AV *)SvRV( numbers );
AV *return_array = newAV();
SV **tmp;
int len = AvFILL( array );
int i, val;
for( i = 0; i <= len; i++ ) {
tmp = av_fetch( array, i, 1 );
if( !SvIOK( *tmp ) ) {
croak( "Can't handle this value!" );
}
val = SvIV( *tmp );
val = val * val;
av_push( return_array, newSViv( val ) );
}
return newRV_inc( (SV *)return_array );
}

124. SV *squares( SV *numbers ) {
AV *array = (AV *)SvRV( numbers );
AV *return_array = newAV();
SV **tmp;
int len = AvFILL( array );
int i, val;
for( i = 0; i <= len; i++ ) {
tmp = av_fetch( array, i, 1 );
if( !SvIOK( *tmp ) ) {
croak( "Can't handle this value!" );
}
val = SvIV( *tmp );
val = val * val;
av_push( return_array, newSViv( val ) );
}
return newRV_noinc( (SV *)return_array );
}

125. Run it!
bash-3.2$ perl cexp3.pl
Give me some numbers: 4,5,6,23
The squares are: 16, 25, 36, 529

126. Let's do something
with hashes.

127. What does an HV look like?

128. What does an HV look like?
(Diagram missing

129. What does an HV look like?
(Diagram missing
(So I stole this one))

130. What does an HV look like?
(Diagram missing
(So I stole this one))

131. #!/usr/bin/env perl
use strict;
use warnings;
use Inline 'C';
print "Give me some words: ";
my @words = split /,/, <STDIN>;
chomp @words;
my $result = uniques( @words );
printf "The uniques are: %sn",
join ", ", %$result;
__END__

132. __C__
SV *uniques( SV *words ) {
AV *array = (AV *)SvRV( words );
HV *result = newHV();
SV **tmp;
int len = AvFILL( array );
int i;
char *val;
for( i = 0; i <= len; i++ ) {
tmp = av_fetch( array, i, 1 );
if( !SvPOK( *tmp ) ) {
croak( "Can't handle this value!" );
}
val = SvPV_nolen( *tmp );
hv_store( result, val, strlen( val ), newSV(0), 0 );
}
return newRV_noinc( (SV *)result );
}

133. Run it!
bash-3.2$ perl cexp4.pl
Give me some words: foo,bar,baz,baz,foo,quux,narf,poit
The uniques are: bar, narf, baz, poit, quux, foo

134. What is
Magic?

135. Not that terrifying.

136. Not that terrifying.
(code sample missing)

137. Magic is a linked list of function pointers
attached to a *SV.

138. Magic is a linked list of function pointers
attached to a *SV.
The functions implement custom read/write
behavior.

139. Magic is a linked list of function pointers
attached to a *SV.
The functions implement custom read/write
behavior.
Special vars like %ENV are implemented via
Magic.

140. $ENV{foo} = "blah";

141. $ENV{foo} = "blah";
assignment op

142. $ENV{foo} = "blah";
assignment op
Magic on %ENV ?

143. $ENV{foo} = "blah";
assignment op
Magic on %ENV ?
Look up write function

144. $ENV{foo} = "blah";
assignment op
Magic on %ENV ?
Look up write function
call libc setenv()

145. tie() is also implemented via Magic.

146. Terror defeated!

147. Where do we go from
here?

148. Resources:
perlguts
For details on how to manipulate Perl's
data structures.

149. Resources:
perlapi
For a comprehensive reference to (almost)
everything you need to make perl do stuff.

150. Resources:
Perlguts Illustrated
For pretty diagrams and explication.

151. Resources:
Inline::C Cookbook
For common solutions to problems
interfacing between Perl and C

152. Resources:

153. Resources:
Code samples
http://github.com/friedo/perl-api-terror

154. London Perl
Workshop
Thank you.
Mike Friedman
friedo@friedo.com
http://friedo.com/
http://github.com/friedo/perl-api-terror