https://www.youtube.com/watch?v=-XITNDvqoTo Have you ever wanted to learn the secrets of the dark masters of Perl? Ever been curious about what an *SV is? Maybe you just want to make a subroutine really, really fast, but you're terrified of this thing called XS that stands in your way. This talk presents the Perl API for beginners from the ground up, explaining everything you need to know to get started hacking Perl in C. XS is not required, as the examples use the much smarter Inline module to get right at the core API with nothing in the way.

1. The Perl API
for the
Mortally Terrified
Mike Friedman
YAPC::NA 2014
Orlando, FL
@friedo

2. This talk
is the product of a lengthy journey.

3. The Perl API
What’s it good for?

4. The Perl API
What’s it good for?
Make things fast:

5. The Perl API
What’s it good for?
Make things fast:
- Write your slow Perl code in C

6. The Perl API
What’s it good for?
Make things fast:
- Write your slow Perl code in C
- Keep your fast Perl code in Perl

7. The Perl API
What’s it good for?
Make Perl more useful:

8. The Perl API
What’s it good for?
Make Perl more useful:
- Interface with libraries written in C

9. The Perl API
What’s it good for?
Make Perl more useful:
- Interface with libraries written in C
- Create modules exposing external APIs to Perl

10. The Perl API
What’s it good for?
Make Perl do...stuff

11. The Perl API
What’s it good for?
Make Perl do...stuff
- Change opcodes on the fly?

12. The Perl API
What’s it good for?
Make Perl do...stuff
- Change opcodes on the fly?
- Extend the parser?

13. The Perl API
What’s it good for?
Make Perl do...stuff
- Change opcodes on the fly?
- Extend the parser?
- Make exceptions work?

14. The Perl API
What’s it good for?
Make Perl do...stuff
- Change opcodes on the fly?
- Extend the parser?
- Make exceptions work?
- Why not?

15. The Perl API
What’s it good for?
Make Perl do...stuff
- Change opcodes on the fly?
- Extend the parser?
- Make exceptions work?
- Why not?

16. XS

17. What is XS?
perldoc perlxs:
XS is an interface description file format used to create an
extension interface between Perl and C code (or a C library)
which one wishes to use with Perl. The XS interface is
combined with the library to create a new library which can
then be either dynamically loaded or statically linked into
perl. The XS interface description is written in the XS
language and is the core component of the Perl extension
interface.

19. XS is a specialized templating
language for C.

20. Some XS Code

21. Some XS Code
XS
Directives

22. Some C Code made by xsubpp

23. XS != Perl API

24. For most things

25. For most things
You don't actually
have to use XS.

26. (OK.You don’t have to
know that you’re using it.)

27. Inline::C

28. use Inline C;
hello('Mike');
hello('Cookie Monster');
__END__
__C__
void hello(char* name) {
printf("Hello %s!n", name);
}

29. use Inline C;
hello('Mike');
hello('Grover');
__END__
__C__
void hello(char* name) {
printf("Hello %s!n", name);
}

30. use Inline C;
hello('Mike');
hello('Grover');
__END__
__C__
void hello(char* name) {
printf("Hello %s!n", name);
}

31. use Inline C;
hello('Mike');
hello('Grover');
__END__
__C__
void hello(char* name) {
printf("Hello %s!n", name);
}

32. Inline::C Code in Perl

33. Inline::C Code in Perl
Digest::MD5

34. Inline::C Code in Perl
Digest::MD5
C Code Parse::RecDescent

35. Inline::C Code in Perl
Digest::MD5
Create build in a temp dir
C Code Parse::RecDescent

36. Inline::C Code in Perl
Digest::MD5
Create build in a temp dir
C Code Parse::RecDescent
Write XS

37. Inline::C Code in Perl
Digest::MD5
Create build in a temp dir
C Code Parse::RecDescent
Write XS
Compile and install

38. Inline::C Code in Perl
Digest::MD5
Create build in a temp dir
C Code Parse::RecDescent
Write XS
Compile and install

39. How do we make
Perl do stuff in C?

40. How do we make
Perl do stuff in C?
We need to understand Perl’s data
structures.

41. Handwaving Ahead

42. A $scalar in Perl is represented by a struct called SV.
SvNULL
SvRV SvNVSvPV SvIV
(These are not all of them.)

43. What does SV look like?
ANY

44. What does SV look like?
Points to a struct with the scalar's metadata.
ANY

45. What does SV look like?

46. What does SV look like?
REFCNT

47. What does SV look like?
Reference count for garbage collection
REFCNT

48. What does SV look like?

49. What does SV look like?
FLAGS TYPE

50. What does SV look like?
Flags and type information
FLAGS TYPE

51. What does SV look like?

52. What does SV look like?
SV_U

53. What does SV look like?
Points to a union with the actual scalar's data
SV_U

54. What does SV look like?

55. What does SV look like?

56. What does SV look like?
The simplest scalar:
undef

57. What does SV look like?
The simplest scalar:
undef
The "ANY" pointer is NULL, and TYPE is zero.

58. undef is, like, totes boring.

59. What about strings?

60. What does SV look like?

61. What does SV look like?
Strings

62. What does SV look like?
Strings
"ANY" points to a "xpv", and TYPE is 4.*
* in Perl 5.20.

63. What does SV look like?
Strings
"ANY" points to a "xpv", and TYPE is 4.*
* in Perl 5.20.
SV becomes SvPV

64. What does SV look like?

65. What does SV look like?
Integers

66. What does SV look like?
Integers
"ANY" points to a "xpviv", and TYPE is 1.*
* in Perl 5.20.

67. What does SV look like?
Integers
"ANY" points to a "xpviv", and TYPE is 1.*
* in Perl 5.20.
SV becomes SvIV

68. What does SV look like?

69. What does SV look like?
Floats

70. What does SV look like?
Floats
"ANY" points to a "xpvnv", and TYPE is 2.*
* in Perl 5.20.

71. What does SV look like?
Floats
"ANY" points to a "xpvnv", and TYPE is 2.*
* in Perl 5.20.
SV becomes SvNV

72. WRITE CODE NOW
?!??!!11!?

73. Let's make
something fast.

74. #!/usr/bin/env perl
use strict;
use warnings;
use Inline 'C';
print 'Give me a number: ';
my $num_a = <STDIN>;
print 'Give me another number: ';
my $num_b = <STDIN>;
printf "The sum is: %sn", add( $num_a, $num_b );
__END__
__C__
int add( int a, int b ) {
return a + b;
}

75. #!/usr/bin/env perl
use strict;
use warnings;
use Inline 'C';
print 'Give me a number: ';
my $num_a = <STDIN>;
print 'Give me another number: ';
my $num_b = <STDIN>;
printf "The sum is: %sn", add( $num_a, $num_b );
__END__
__C__
int add( int a, int b ) {
return a + b;
}

76. #!/usr/bin/env perl
use strict;
use warnings;
use Inline 'C';
print 'Give me a number: ';
my $num_a = <STDIN>;
print 'Give me another number: ';
my $num_b = <STDIN>;
printf "The sum is: %sn", add( $num_a, $num_b );
__END__
__C__
int add( int a, int b ) {
return a + b;
}

77. #!/usr/bin/env perl
use strict;
use warnings;
use Inline 'C';
print 'Give me a number: ';
my $num_a = <STDIN>;
print 'Give me another number: ';
my $num_b = <STDIN>;
printf "The sum is: %sn", add( $num_a, $num_b );
__END__
__C__
int add( int a, int b ) {
return a + b;
}

78. #!/usr/bin/env perl
use strict;
use warnings;
use Inline 'C';
print 'Give me a number: ';
my $num_a = <STDIN>;
print 'Give me another number: ';
my $num_b = <STDIN>;
printf "The sum is: %sn", add( $num_a, $num_b );
__END__
__C__
int add( int a, int b ) {
return a + b;
}

79. #!/usr/bin/env perl
use strict;
use warnings;
use Inline 'C';
print 'Give me a number: ';
my $num_a = <STDIN>;
print 'Give me another number: ';
my $num_b = <STDIN>;
printf "The sum is: %sn", add( $num_a, $num_b );
__END__
__C__
int add( int a, int b ) {
return a + b;
}

80. bash-3.2$ perl cexp1.pl
Give me a number: 42
Give me another number: 11
The sum is: 53
Run it!

81. bash-3.2$ perl cexp1.pl
Give me a number: 42
Give me another number: 31.337
The sum is: 73
Run it again!

82. bash-3.2$ perl cexp1.pl
Give me a number: 42
Give me another number: 31.337
The sum is: 73
Run it again!
What happened?

83. Inline::C
is
CRAZY
SMART

84. int
int
int
int
Inline::C
is
CRAZY
SMART

85. Adding numbers:
What will we need?

87. •SvIOK Is it a SvIV?

88. •SvIOK Is it a SvIV?
•SvNOK Is it a SvNV?

89. •SvIOK Is it a SvIV?
•SvNOK Is it a SvNV?
•SvIV Get the C int value from SV

90. •SvIOK Is it a SvIV?
•SvNOK Is it a SvNV?
•SvIV Get the C int value from SV
•SvNV Get the C float value from SV

91. •SvIOK Is it a SvIV?
•SvNOK Is it a SvNV?
•SvIV Get the C int value from SV
•SvNV Get the C float value from SV
•newSViv Make a new integer SV

92. •SvIOK Is it a SvIV?
•SvNOK Is it a SvNV?
•SvIV Get the C int value from SV
•SvNV Get the C float value from SV
•newSViv Make a new integer SV
•newSVnv Make a new float SV

93. use Inline 'C';
print 'Give me a number: ';
my $num_a = <STDIN>;
print 'Give me another number: ';
my $num_b = <STDIN>;
$num_a += 0; $num_b += 0;
printf "The sum is: %sn", add( $num_a, $num_b );
__END__
__C__
SV *add( SV *a, SV *b ) {
if ( SvIOK( a ) && SvIOK( b ) ) {
return newSViv( SvIV( a ) + SvIV( b ) );
} else if ( SvNOK( a ) && SvNOK( b ) ) {
return newSVnv( SvNV( a ) + SvNV( b ) );
} else {
croak( "I don't know what to do!" );
}
}

94. use Inline 'C';
print 'Give me a number: ';
my $num_a = <STDIN>;
print 'Give me another number: ';
my $num_b = <STDIN>;
$num_a += 0; $num_b += 0; # coerce
printf "The sum is: %sn", add( $num_a, $num_b );
__END__
__C__
SV *add( SV *a, SV *b ) {
if ( SvIOK( a ) && SvIOK( b ) ) {
return newSViv( SvIV( a ) + SvIV( b ) );
} else if ( SvNOK( a ) && SvNOK( b ) ) {
return newSVnv( SvNV( a ) + SvNV( b ) );
} else {
croak( "I don't know what to do!" );
}
}

95. use Inline 'C';
print 'Give me a number: ';
my $num_a = <STDIN>;
print 'Give me another number: ';
my $num_b = <STDIN>;
$num_a += 0; $num_b += 0;
printf "The sum is: %sn", add( $num_a, $num_b );
__END__
__C__
SV *add( SV *a, SV *b ) {
if ( SvIOK( a ) && SvIOK( b ) ) {
return newSViv( SvIV( a ) + SvIV( b ) );
} else if ( SvNOK( a ) && SvNOK( b ) ) {
return newSVnv( SvNV( a ) + SvNV( b ) );
} else {
croak( "I don't know what to do!" );
}
}

96. use Inline 'C';
print 'Give me a number: ';
my $num_a = <STDIN>;
print 'Give me another number: ';
my $num_b = <STDIN>;
$num_a += 0; $num_b += 0;
printf "The sum is: %sn", add( $num_a, $num_b );
__END__
__C__
SV *add( SV *a, SV *b ) {
if ( SvIOK( a ) && SvIOK( b ) ) {
return newSViv( SvIV( a ) + SvIV( b ) );
} else if ( SvNOK( a ) && SvNOK( b ) ) {
return newSVnv( SvNV( a ) + SvNV( b ) );
} else {
croak( "I don't know what to do!" );
}
}

97. bash-3.2$ perl cexp2.pl
Give me a number: 42
Give me another number: 11
The sum is: 53
bash-3.2$ perl cexp2.pl
Run it!

98. bash-3.2$ perl cexp2.pl
Give me a number: 42
Give me another number: 11
The sum is: 53
bash-3.2$ perl cexp2.pl
Run it!
bash-3.2$ perl cexp2.pl
Give me a number: 53.4
Give me another number: 6.54
The sum is: 59.94

99. bash-3.2$ perl cexp2.pl
Give me a number: 42
Give me another number: 11
The sum is: 53
bash-3.2$ perl cexp2.pl
Run it!
bash-3.2$ perl cexp2.pl
Give me a number: 53.4
Give me another number: 6.54
The sum is: 59.94
Give me a number: 42
Give me another number: 6.54
I don't know what to do! at
cexp2.pl line 16, <STDIN>
line 2.

100. Once you get this far,
everything you need
is in perlapi.

101. Let's do something
with arrays.

102. What does SV look like?

103. What does SV look like?
Arrays

104. What does SV look like?
Arrays
"ANY" points to a "xpvav"

105. What does SV look like?
Arrays
"ANY" points to a "xpvav"
SV becomes AV

106. Squaring numbers:
What will we need?

107. •SvRV Get SV from reference

108. •SvRV Get SV from reference
•newAV Make a new array AV

109. •SvRV Get SV from reference
•newAV Make a new array AV
•newRV Make a new reference

110. •SvRV Get SV from reference
•newAV Make a new array AV
•newRV Make a new reference
•AvFILL Get size of array

111. •SvRV Get SV from reference
•newAV Make a new array AV
•newRV Make a new reference
•AvFILL Get size of array
•av_fetch Get an element from array

112. •SvRV Get SV from reference
•newAV Make a new array AV
•newRV Make a new reference
•AvFILL Get size of array
•av_fetch Get an element from array
•SvIOK Is it an int?

113. •SvRV Get SV from reference
•newAV Make a new array AV
•newRV Make a new reference
•AvFILL Get size of array
•av_fetch Get an element from array
•SvIOK Is it an int?
•SvIV Get C int from SV

114. •SvRV Get SV from reference
•newAV Make a new array AV
•newRV Make a new reference
•AvFILL Get size of array
•av_fetch Get an element from array
•SvIOK Is it an int?
•SvIV Get C int from SV
•av_push Obvs.

115. #!/usr/bin/env perl
use strict;
use warnings;
use Inline 'C';
print "Give me some numbers: ";
my @numbers = map { $_ += 0 }
split /,/, <STDIN>;
my $result = squares( @numbers );
printf "The squares are: %sn",
join ", ", @$result;
__END__

116. #!/usr/bin/env perl
use strict;
use warnings;
use Inline 'C';
print "Give me some numbers: ";
my @numbers = map { $_ += 0 }
split /,/, <STDIN>;
my $result = squares( @numbers );
printf "The squares are: %sn",
join ", ", @$result;
__END__

117. SV *squares( SV *numbers ) {
AV *array = (AV *)SvRV( numbers );
AV *return_array = newAV();
SV **tmp;
int len = AvFILL( array );
int i, val;
for( i = 0; i <= len; i++ ) {
tmp = av_fetch( array, i, 1 );
if( !SvIOK( *tmp ) ) {
croak( "Can't handle this value!" );
}
val = SvIV( *tmp );
val = val * val;
av_push( return_array, newSViv( val ) );
}
return newRV_inc( (SV *)return_array );
}

118. SV *squares( SV *numbers ) {
AV *array = (AV *)SvRV( numbers );
AV *return_array = newAV();
SV **tmp;
int len = AvFILL( array );
int i, val;
for( i = 0; i <= len; i++ ) {
tmp = av_fetch( array, i, 1 );
if( !SvIOK( *tmp ) ) {
croak( "Can't handle this value!" );
}
val = SvIV( *tmp );
val = val * val;
av_push( return_array, newSViv( val ) );
}
return newRV_inc( (SV *)return_array );
}

119. SV *squares( SV *numbers ) {
AV *array = (AV *)SvRV( numbers );
AV *return_array = newAV();
SV **tmp;
int len = AvFILL( array );
int i, val;
for( i = 0; i <= len; i++ ) {
tmp = av_fetch( array, i, 1 );
if( !SvIOK( *tmp ) ) {
croak( "Can't handle this value!" );
}
val = SvIV( *tmp );
val = val * val;
av_push( return_array, newSViv( val ) );
}
return newRV_inc( (SV *)return_array );
}

120. SV *squares( SV *numbers ) {
AV *array = (AV *)SvRV( numbers );
AV *return_array = newAV();
SV **tmp;
int len = AvFILL( array );
int i, val;
for( i = 0; i <= len; i++ ) {
tmp = av_fetch( array, i, 1 );
if( !SvIOK( *tmp ) ) {
croak( "Can't handle this value!" );
}
val = SvIV( *tmp );
val = val * val;
av_push( return_array, newSViv( val ) );
}
return newRV_inc( (SV *)return_array );
}

121. SV *squares( SV *numbers ) {
AV *array = (AV *)SvRV( numbers );
AV *return_array = newAV();
SV **tmp;
int len = AvFILL( array );
int i, val;
for( i = 0; i <= len; i++ ) {
tmp = av_fetch( array, i, 1 );
if( !SvIOK( *tmp ) ) {
croak( "Can't handle this value!" );
}
val = SvIV( *tmp );
val = val * val;
av_push( return_array, newSViv( val ) );
}
return newRV_inc( (SV *)return_array );
}

122. SV *squares( SV *numbers ) {
AV *array = (AV *)SvRV( numbers );
AV *return_array = newAV();
SV **tmp;
int len = AvFILL( array );
int i, val;
for( i = 0; i <= len; i++ ) {
tmp = av_fetch( array, i, 1 );
if( !SvIOK( *tmp ) ) {
croak( "Can't handle this value!" );
}
val = SvIV( *tmp );
val = val * val;
av_push( return_array, newSViv( val ) );
}
return newRV_inc( (SV *)return_array );
}

123. SV *squares( SV *numbers ) {
AV *array = (AV *)SvRV( numbers );
AV *return_array = newAV();
SV **tmp;
int len = AvFILL( array );
int i, val;
for( i = 0; i <= len; i++ ) {
tmp = av_fetch( array, i, 1 );
if( !SvIOK( *tmp ) ) {
croak( "Can't handle this value!" );
}
val = SvIV( *tmp );
val = val * val;
av_push( return_array, newSViv( val ) );
}
return newRV( (SV *)return_array );
}

124. bash-3.2$ perl cexp3.pl
Give me some numbers: 4,5,6,23
The squares are: 16, 25, 36, 529
Run it!

125. Let's do something
with hashes.

126. What does SV look like?

127. What does SV look like?
Hashes

128. What does SV look like?
Hashes
"ANY" points to a "xpvhv"

129. What does SV look like?
Hashes
"ANY" points to a "xpvhv"
SV becomes HV

130. Finding uniques:
What will we need?

131. •SvRV Get SV from reference

132. •SvRV Get SV from reference
•newHV Make a new hash HV

133. •SvRV Get SV from reference
•newHV Make a new hash HV
•newRV Make a new reference

134. •SvRV Get SV from reference
•newHV Make a new hash HV
•newRV Make a new reference
•AvFILL Get size of array

135. •SvRV Get SV from reference
•newHV Make a new hash HV
•newRV Make a new reference
•AvFILL Get size of array
•av_fetch Get an element from array

136. •SvRV Get SV from reference
•newHV Make a new hash HV
•newRV Make a new reference
•AvFILL Get size of array
•av_fetch Get an element from array
•SvPOK Is it a string?

137. •SvRV Get SV from reference
•newHV Make a new hash HV
•newRV Make a new reference
•AvFILL Get size of array
•av_fetch Get an element from array
•SvPOK Is it a string?
•SvPV Get C string from SV

138. •SvRV Get SV from reference
•newHV Make a new hash HV
•newRV Make a new reference
•AvFILL Get size of array
•av_fetch Get an element from array
•SvPOK Is it a string?
•SvPV Get C string from SV
•hv_store Obvs.

139. #!/usr/bin/env perl
use strict;
use warnings;
use Inline 'C';
print "Give me some words: ";
my @words = split /,/, <STDIN>;
chomp @words;
my $result = uniques( @words );
printf "The uniques are: %sn",
join ", ", %$result;
__END__

140. __C__
SV *uniques( SV *words ) {
AV *array = (AV *)SvRV( words );
HV *result = newHV();
SV **tmp;
int len = AvFILL( array );
int i;
char *val;
for( i = 0; i <= len; i++ ) {
tmp = av_fetch( array, i, 1 );
if( !SvPOK( *tmp ) ) {
croak( "Can't handle this value!" );
}
val = SvPV_nolen( *tmp );
hv_store( result, val, strlen( val ), newSV(0), 0 );
}
return newRV( (SV *)result );
}

141. bash-3.2$ perl cexp4.pl
Give me some words: foo,bar,baz,baz,foo,quux,narf,poit
The uniques are: bar, narf, baz, poit, quux, foo
Run it!

142. What is
Magic?

143. Not that terrifying.

144. Magic is a linked list of function pointers
attached to a *SV.

145. Magic is a linked list of function pointers
attached to a *SV.
The functions implement custom read/write
behavior.

146. Magic is a linked list of function pointers
attached to a *SV.
The functions implement custom read/write
behavior.
Special vars like %ENV are implemented via
Magic.

147. $ENV{foo} = "blah";

148. $ENV{foo} = "blah";
assignment op

149. $ENV{foo} = "blah";
Magic on %ENV ?
assignment op

150. $ENV{foo} = "blah";
Magic on %ENV ?
assignment op
Look up write function

151. $ENV{foo} = "blah";
Magic on %ENV ?
assignment op
Look up write function
call libc setenv()

152. tie() is also implemented via Magic.

153. Terror defeated!

154. Where do we go from
here?

155. Resources:
perlguts
For details on how to manipulate Perl's
data structures.

156. Resources:
perlapi
For a comprehensive reference to (almost)
everything you need to make perl do stuff.

157. Resources:
Perlguts Illustrated
For pretty diagrams and explication.

158. Resources:
Inline::C Cookbook
For common solutions to problems
interfacing between Perl and C

159. Resources:

160. Resources:
Code samples
http://github.com/friedo/perl-api-terror

161. YAPC::NA
Thank you.
Mike Friedman
friedo@friedo.com
http://friedo.com/
http://github.com/friedo/perl-api-terror