The document provides a brief history of testing on CPAN from 1987 to the present. It discusses the development of the Test Anything Protocol (TAP) and CPAN Testers for testing Perl modules. It then proposes the idea of CPANci, a continuous integration system for all of CPAN that would test each distribution in isolation on virtualized environments to avoid issues with CPAN Testers. The document outlines an approach using perlbrew and cpanminus to test each distribution on fresh Perl installations of different versions.

1. Testing CPAN
in the
21st Century
Mike Friedman
(friedo)
YAPC::NA 2013
Austin,TX
Tuesday, June 4, 13

2. A lengthy series of bad
ideas and stupid
questions.
Tuesday, June 4, 13

3. Tuesday, June 4, 13

4. Stupid Question No. 1
Tuesday, June 4, 13

5. Tuesday, June 4, 13

6. Stupid Question No. 2
Tuesday, June 4, 13

7. What’s this about,
anyway?
Tuesday, June 4, 13

8. What’s this about,
anyway?
CPANci
Tuesday, June 4, 13

9. A Brief History
Tuesday, June 4, 13

10. A Brief History
•December 18, 1987
Tuesday, June 4, 13

11. A Brief History
•December 18, 1987
•Perl 1.000 released.
Tuesday, June 4, 13

12. A Brief History
•December 18, 1987
•Perl 1.000 released.
•TAP invented.
Tuesday, June 4, 13

13. The Test Anything Protocol
Tuesday, June 4, 13

14. The Test Anything Protocol
1..42
ok 1 the thing looks good!
ok 2
ok 3 $beer isa $drink
not ok 4 too much $beer
not ok 5 $me->vomit( 'now' )
...
Tuesday, June 4, 13

15. A Brief History
Tuesday, June 4, 13

16. A Brief History
•October 17, 1994
Tuesday, June 4, 13

17. A Brief History
•October 17, 1994
•Perl 5.000 released.
Tuesday, June 4, 13

18. A Brief History
•October 17, 1994
•Perl 5.000 released.
•Perl has a module system.
Tuesday, June 4, 13

19. Tuesday, June 4, 13

20. # from this
require "funcs.pl";
Tuesday, June 4, 13

21. # from this
require "funcs.pl";
# to this
use My::Module;
Tuesday, June 4, 13

22. Tuesday, June 4, 13

23. # but under the hood
BEGIN {
require My::Module;
My::Module->import;
};
Tuesday, June 4, 13

24. A Brief History
Tuesday, June 4, 13

25. A Brief History
•October 26, 1995
Tuesday, June 4, 13

26. A Brief History
•October 26, 1995
•CPAN established.
Tuesday, June 4, 13

27. A Brief History
•October 26, 1995
•CPAN established.
•Perl modules are available.
Tuesday, June 4, 13

28. A Brief History
Tuesday, June 4, 13

29. A Brief History
•May 15, 1997
Tuesday, June 4, 13

30. A Brief History
•May 15, 1997
•Perl 5.004 released.
Tuesday, June 4, 13

31. A Brief History
•May 15, 1997
•Perl 5.004 released.
•CPAN.pm is in the core.
Tuesday, June 4, 13

32. Tuesday, June 4, 13

33. # the dark art
$ perl -MCPAN -e 'install Foo'
Tuesday, June 4, 13

34. A Brief History
Tuesday, June 4, 13

35. A Brief History
•May, 1998
Tuesday, June 4, 13

36. A Brief History
•May, 1998
•CPAN Testers conceived
Tuesday, June 4, 13

37. A Brief History
•May, 1998
•CPAN Testers conceived
•Automated feedback for authors
Tuesday, June 4, 13

38. Tuesday, June 4, 13

39. Tuesday, June 4, 13

40. Tuesday, June 4, 13

41. Tuesday, June 4, 13

42. A Brief History
Tuesday, June 4, 13

43. A Brief History
•November 15, 2003
Tuesday, June 4, 13

44. A Brief History
•November 15, 2003
•Perl 5.6.2 released.
Tuesday, June 4, 13

45. A Brief History
•November 15, 2003
•Perl 5.6.2 released.
•Test::More is in the core.
Tuesday, June 4, 13

46. Tuesday, June 4, 13

47. use Test::More tests => 3;
ok( 42 );
is( $foo, 'my value' );
isnt( 'foo', 'bar' );
Tuesday, June 4, 13

48. A Brief History
Tuesday, June 4, 13

49. A Brief History
•August 6, 2012
Tuesday, June 4, 13

50. A Brief History
•August 6, 2012
•Mike goes to work for 10gen
Tuesday, June 4, 13

51. Tuesday, June 4, 13

52. Tuesday, June 4, 13

53. WEB SCALE!!!!11
Tuesday, June 4, 13

54. WEB SCALE!!!!11
LOL
Tuesday, June 4, 13

55. Tuesday, June 4, 13

56. Bad Idea No. 1
Tuesday, June 4, 13

57. Bad Idea No. 1
Come up with a cool Perl MongoDB project
to show off atYAPC!
Tuesday, June 4, 13

58. Bad Idea No. 1
Come up with a cool Perl MongoDB project
to show off atYAPC!
It'll be fun!
Tuesday, June 4, 13

59. Bad Idea No. 1
Come up with a cool Perl MongoDB project
to show off atYAPC!
It'll be fun!
promise!
Tuesday, June 4, 13

60. CPAN Testers is Wonderful and
Amazing
Tuesday, June 4, 13

61. Tuesday, June 4, 13

62. Disadvantages:
Tuesday, June 4, 13

63. Disadvantages:
Not real time
Tuesday, June 4, 13

64. Disadvantages:
Not real time
Not consistent
Tuesday, June 4, 13

65. Disadvantages:
Not real time
Not consistent
Polluted / Inconsistent environments
Tuesday, June 4, 13

66. Disadvantages:
Not real time
Not consistent
Polluted / Inconsistent environments
Not all versions on all platforms
Tuesday, June 4, 13

67. I want Continuous Integration
for the entire CPAN.
Tuesday, June 4, 13

68. Tuesday, June 4, 13

69. Bad Idea No. 2
Tuesday, June 4, 13

70. CPANci.org
Bad Idea No. 2
Tuesday, June 4, 13

71. Tuesday, June 4, 13

72. Stupid Question No. 3
Tuesday, June 4, 13

73. Stupid Question No. 3
How can we test CPAN without the
disadvantages of CPAN Testers?
Tuesday, June 4, 13

74. Postulate:
Every CPAN distribution must be tested in isolation, on a
virgin Perl installation untouched by human hands.
Tuesday, June 4, 13

75. Postulate:
Every CPAN distribution must be tested in isolation, on a
virgin Perl installation untouched by human hands.
So how do we do that?
Tuesday, June 4, 13

76. perlbrew
Tuesday, June 4, 13

77. Virtualization
Tuesday, June 4, 13

78. Virtualization
Whoa!
Tuesday, June 4, 13

79. Tuesday, June 4, 13

80. Bad Idea No. 3
Tuesday, June 4, 13

81. Tuesday, June 4, 13

82. •Create an EC2 image
Tuesday, June 4, 13

83. •Create an EC2 image
•Put perlbrew on it
Tuesday, June 4, 13

84. •Create an EC2 image
•Put perlbrew on it
•Install every Perl locally
Tuesday, June 4, 13

85. •Create an EC2 image
•Put perlbrew on it
•Install every Perl locally
•Boot an instance for every uploaded
distribution
Tuesday, June 4, 13

86. •Create an EC2 image
•Put perlbrew on it
•Install every Perl locally
•Boot an instance for every uploaded
distribution
•Install needed dependencies for the
distribution
Tuesday, June 4, 13

87. •Create an EC2 image
•Put perlbrew on it
•Install every Perl locally
•Boot an instance for every uploaded
distribution
•Install needed dependencies for the
distribution
•Run the tests and report the results
Tuesday, June 4, 13

88. •Create an EC2 image
•Put perlbrew on it
•Install every Perl locally
•Boot an instance for every uploaded
distribution
•Install needed dependencies for the
distribution
•Run the tests and report the results
•Shut down the instance
Tuesday, June 4, 13

89. Tuesday, June 4, 13

90. Tuesday, June 4, 13

91. Stupid Question No. 4
Tuesday, June 4, 13

92. Stupid Question No. 4
How can we do this on one instance?
Tuesday, June 4, 13

93. local::lib
cpanminus
Tuesday, June 4, 13

94. App::cpanminus
Tuesday, June 4, 13

95. App::cpanminus
•Self-contained
Tuesday, June 4, 13

96. App::cpanminus
•Self-contained
•That is, the cpanm script is self-contained
Tuesday, June 4, 13

97. App::cpanminus
•Self-contained
•That is, the cpanm script is self-contained
•via App::FatPacker
Tuesday, June 4, 13

98. App::cpanminus
Tuesday, June 4, 13

99. App::cpanminus
That means the same cpanm can be run by any perl
Tuesday, June 4, 13

100. App::cpanminus
That means the same cpanm can be run by any perl
perlbrew switch master
curl -L http://cpanmin.us/ | perl - App::cpanminus
ln -s ~/perl5/perlbrew/perls/master/bin/cpanm ./cpanm
~/perl5/perlbrew/perls/perl-5.8.9/bin/perl cpanm
~/perl5/perlbrew/perls/perl-5.10.1/bin/perl cpanm
~/perl5/perlbrew/perls/perl-5.12.5/bin/perl cpanm
~/perl5/perlbrew/perls/perl-5.14.4/bin/perl cpanm
~/perl5/perlbrew/perls/perl-5.16.3/bin/perl cpanm
~/perl5/perlbrew/perls/perl-5.18.0/bin/perl cpanm
~/perl5/perlbrew/perls/perl-5.19.0/bin/perl cpanm
Tuesday, June 4, 13

101. Tuesday, June 4, 13

102. •Use perlbrew to install a "master" perl
Tuesday, June 4, 13

103. •Use perlbrew to install a "master" perl
•Use it again to install "virgin" perls of every
major version
Tuesday, June 4, 13

104. •Use perlbrew to install a "master" perl
•Use it again to install "virgin" perls of every
major version
•Use the master perl to install everything
from CPAN that makes CPANci work
Tuesday, June 4, 13

105. •Use perlbrew to install a "master" perl
•Use it again to install "virgin" perls of every
major version
•Use the master perl to install everything
from CPAN that makes CPANci work
•For each distribution, create a temp
directory
Tuesday, June 4, 13

106. •Use perlbrew to install a "master" perl
•Use it again to install "virgin" perls of every
major version
•Use the master perl to install everything
from CPAN that makes CPANci work
•For each distribution, create a temp
directory
•Tell cpanminus to install dependencies
there, as if for local::lib
Tuesday, June 4, 13

107. •Use perlbrew to install a "master" perl
•Use it again to install "virgin" perls of every
major version
•Use the master perl to install everything
from CPAN that makes CPANci work
•For each distribution, create a temp
directory
•Tell cpanminus to install dependencies
there, as if for local::lib
•Run tests and report results
Tuesday, June 4, 13

108. •Use perlbrew to install a "master" perl
•Use it again to install "virgin" perls of every
major version
•Use the master perl to install everything
from CPAN that makes CPANci work
•For each distribution, create a temp
directory
•Tell cpanminus to install dependencies
there, as if for local::lib
•Run tests and report results
•Delete temp directory, leaving each perl
untouched!
Tuesday, June 4, 13

109. What does that look like?
The "fetcher" grabs the latest distribution URLs from
MetaCPAN's RSS feed.
Tuesday, June 4, 13

110. What does that look like?
We retrieve the distribution metadata from
the MetaCPAN JSON API and saved it to
MongoDB.
Then we start the Installer.
Tuesday, June 4, 13

111. What does that look like?
We download the distribution tarball to a temp file.
Then the fun stuff starts to happen.
Tuesday, June 4, 13

112. What does that look like?
We extract the archive in a specific "work" directory
for each perl.
Then create a temp directory for building and installing
dependencies.
Tuesday, June 4, 13

113. What does that look like?
Use a specific perl binary to run cpanm and
install dependencies, with no tests, to the temp
directory.
Then we parse the cpanm log on stderr
Tuesday, June 4, 13

114. What does that look like?
"deps" : {
"log" : [
{
"indent" : 0,
"type" : "working-on",
"line" : "--> Working on .n"
},
{
"line" : "Configuring Lingua-EN-NamedEntity-1.92 ... OKn",
"type" : "config",
"indent" : 1
},
{
"type" : "found-deps",
"indent" : 1,
"line" : "==> Found dependencies: Lingua::Stem::En, DB_File, LWP::Simplen"
},
{
"indent" : 1,
"type" : "working-on",
"line" : "--> Working on Lingua::Stem::Enn"
},
{
"indent" : 2,
"type" : "fetch",
"line" : "Fetching http://www.cpan.org/authors/id/S/SN/SNOWHARE/Lingua-
Stem-0.84.tar.gz ... OKn"
},
Tuesday, June 4, 13

115. What does that look like?
Use a specific perl to run each test file,
save the TAP output and any errors, and
use the exit status to determine if it passed.
Tuesday, June 4, 13

116. What does that look like?
Parse the TAP output of each test into a
structure which can be saved in MongoDB
Tuesday, June 4, 13

117. What does that look like?
"name" : "t/05depth.t",
"lines" : [
{
"ok" : true,
"text" : "1..12",
"type" : "plan"
},
{
"number" : 1,
"ok" : true,
"type" : "test",
"desc" : "new object",
"text" : "ok 1 - new object"
},
{
"ok" : true,
"number" : 2,
"desc" : "depth",
"type" : "test",
"text" : "ok 2 - depth"
},
{
"desc" : "depth",
"text" : "ok 3 - depth",
"type" : "test",
"number" : 3,
"ok" : true
},
{
"type" : "test",
"desc" : "depth",
"text" : "ok 4 - depth",
"ok" : true,
"number" : 4
},
{
"number" : 5,
"ok" : true,
"text" : "ok 5 - depth",
"desc" : "depth",
"type" : "test"
},
{
"number" : 6,
"ok" : true,
"type" : "test",
"desc" : "depth",
"text" : "ok 6 - depth"
},
Tuesday, June 4, 13

118. What does that look like?
"name" : "t/05depth.t",
"lines" : [
{
"ok" : true,
"text" : "1..12",
"type" : "plan"
},
{
"number" : 1,
"ok" : true,
"type" : "test",
"desc" : "new object",
"text" : "ok 1 - new object"
},
{
"ok" : true,
"number" : 2,
"desc" : "depth",
"type" : "test",
"text" : "ok 2 - depth"
},
{
"desc" : "depth",
"text" : "ok 3 - depth",
"type" : "test",
"number" : 3,
"ok" : true
},
{
"type" : "test",
"desc" : "depth",
"text" : "ok 4 - depth",
"ok" : true,
"number" : 4
},
{
"number" : 5,
"ok" : true,
"text" : "ok 5 - depth",
"desc" : "depth",
"type" : "test"
},
{
"number" : 6,
"ok" : true,
"type" : "test",
"desc" : "depth",
"text" : "ok 6 - depth"
},
•This is JSON
Tuesday, June 4, 13

119. What does that look like?
"name" : "t/05depth.t",
"lines" : [
{
"ok" : true,
"text" : "1..12",
"type" : "plan"
},
{
"number" : 1,
"ok" : true,
"type" : "test",
"desc" : "new object",
"text" : "ok 1 - new object"
},
{
"ok" : true,
"number" : 2,
"desc" : "depth",
"type" : "test",
"text" : "ok 2 - depth"
},
{
"desc" : "depth",
"text" : "ok 3 - depth",
"type" : "test",
"number" : 3,
"ok" : true
},
{
"type" : "test",
"desc" : "depth",
"text" : "ok 4 - depth",
"ok" : true,
"number" : 4
},
{
"number" : 5,
"ok" : true,
"text" : "ok 5 - depth",
"desc" : "depth",
"type" : "test"
},
{
"number" : 6,
"ok" : true,
"type" : "test",
"desc" : "depth",
"text" : "ok 6 - depth"
},
•This is JSON
•Stored in MongoDB
Tuesday, June 4, 13

120. What does that look like?
"name" : "t/05depth.t",
"lines" : [
{
"ok" : true,
"text" : "1..12",
"type" : "plan"
},
{
"number" : 1,
"ok" : true,
"type" : "test",
"desc" : "new object",
"text" : "ok 1 - new object"
},
{
"ok" : true,
"number" : 2,
"desc" : "depth",
"type" : "test",
"text" : "ok 2 - depth"
},
{
"desc" : "depth",
"text" : "ok 3 - depth",
"type" : "test",
"number" : 3,
"ok" : true
},
{
"type" : "test",
"desc" : "depth",
"text" : "ok 4 - depth",
"ok" : true,
"number" : 4
},
{
"number" : 5,
"ok" : true,
"text" : "ok 5 - depth",
"desc" : "depth",
"type" : "test"
},
{
"number" : 6,
"ok" : true,
"type" : "test",
"desc" : "depth",
"text" : "ok 6 - depth"
},
•This is JSON
•Stored in MongoDB
•But it's also TAP!
Tuesday, June 4, 13

121. Tuesday, June 4, 13

122. Bad Idea No. 4
Tuesday, June 4, 13

123. Bad Idea No. 4
LIVE DEMO!
Tuesday, June 4, 13

124. The future?
Tuesday, June 4, 13

125. The future?
•Integration with Pinto
Tuesday, June 4, 13

126. The future?
•Integration with Pinto
•Integration with Stratopan
Tuesday, June 4, 13

127. The future?
•Integration with Pinto
•Integration with Stratopan
•All kinds of cool statistics on the website
Tuesday, June 4, 13

128. The future?
•Integration with Pinto
•Integration with Stratopan
•All kinds of cool statistics on the website
•Organizations using CPANci for their
internal DarkPANs
Tuesday, June 4, 13

129. Final
Thoughts
Tuesday, June 4, 13

130. Final
Thoughts
•Play with new toys.
Tuesday, June 4, 13

131. Final
Thoughts
•Play with new toys.
•Think before you code.
Tuesday, June 4, 13

132. Final
Thoughts
•Play with new toys.
•Think before you code.
•Have bad ideas.
Tuesday, June 4, 13

133. Final
Thoughts
•Play with new toys.
•Think before you code.
•Have bad ideas.
•Ask stupid questions.
Tuesday, June 4, 13

134. Final
Thoughts
•Play with new toys.
•Think before you code.
•Have bad ideas.
•Ask stupid questions.
•Have fun.
Tuesday, June 4, 13

135. Stupid Questions?
Mike Friedman
(friedo)
friedo@friedo.com
https://github.com/friedo/cpanci
Tuesday, June 4, 13