The document discusses containerizing Perl testing using tools like Docker and LXC, emphasizing the importance of isolation to prevent issues from interfering tests. It outlines different methods for managing Perl environments, including using dedicated sandboxes, process isolation, and Docker's layered images to maintain clean testing environments. The document also highlights how to utilize volumes for managing dependencies and building Perl, as well as strategies for handling potential problems during testing.

1. Smoke & Mirrors:
Containerizing Perl Testing
Steven Lembark
Workhorse Computing
lembark@wrkhors.com

2. What is “Smoking”?
● High-level sanity check, not detailed analysis.
Does it catch fire when we power it on?
● CPAN runs tests as part of installation.
We are unusual that way.
● CPAN::Reporter reports results of tests.
● CPAN::Reporter::Smoker builds with reporting.

3. Why do you smoke?
● Verify personal modules.
● Validate project platform.
● Addict (CPAN smoker).

4. What do you smoke?
● Whatever is in your sandbox.
● A project and all of its supporting code.
● All of CPAN, or the RECENTS file.

5. How do you smoke?
● “make -wk -C myproject all test” in your sandbox.
● Nightly build, git hooks, Travis CI.
● CPAN or CPANM smoke testers.

6. Risks of Smoking
● Pull in arbitrary code.
Maybe broken.
Possibly malicious.
More likely just stupid
system( rm -rf / *.foobar );

7. Second-hand Smoke
● Permanent side effects.
● Left over test files.
● Wasted disk space.
● Bandwidth.
● Still possible with personal tests:
Dependent modules.

8. Protect Yourself
● Install into a dedicated sandbox.
● Disposable environment.

9. One approach: Virtual Machines
● Heavy footprint.
● Fully configure a running machine to run a test.
● Inefficient use of hardware.

10. Lightweigit: “chroot”
● Filesystem isolation only.
● Tests can interfere with network, processes.
● Files pollute permenant storage.
● No resource limits.

11. Another approach: Process isolation.
● Keep processes from interfering with one another.
● Lightweight: Share logger, daemons, filesytems...
● More efficient use of CPU.
● Less work to set up.
● Not always easy to do right.
Jails & Zones

12. Better way: lxc
● Approach via IBM.
Mainframe LPAR ported to linux.
● Uses process groups.
● Excellent control.
● One problem: People used them like VM's.
● Two problem: lxc has hundreds of switches.

13. lxc for the Masses
● Docker: 80/20 of lxc.
Medium isolation.
● Generate an “image”.
Read-only stack of AUFS mounts.
● Run a “container”.
Mountpoint + resources + isolation.
● Decent manglement tools.

14. AUFS with lxc
● Images are layered.
Read-only.
● "Container layer"
updated.
discarded.
Good and bad:
No permenant updates.

15. Not so nice
● Images look good for smoking:
Read-only underneath: minimal harm.
Re-cycle O/S layer with Perly build.
Allows mix'n match testing.
● Catch:
No fixes or upgrades to smokebox.
Changes require complete rebuild.

16. One fix: Volumes
● Described as “data volumes”.
● Can store anything.
● Like Perl, build directory, CPAN mirror.
● Mountpoints and symlinks.
Dir's or files from O/S into container.
● Mount RO or RW.
Maintain Perl & Smoker.

17. Hybrid approach
● O/S is an image.
● Perl, CPAN on a volume.
O/S tools build and maintain perl.
perl maintains /var/lib/CPAN

18. perly volumes
● Tempting: perlbrew dir's as volumes.
Problems with storage in /home.
Hard to share across users.
Storage depends on user account.
Multiple accounts duplicate space.
● Alternate: Let Docker manage the volumes
Simpler sharing.
/var/lib/docker [hopefully] has more space.
Manage via docker.

19. Perl Testers Wiki
● Good instructions for CPAN::Reporter::Smoker.
● Start with an O/S image and get a working smoker.
● CPANM has its own version.
"minismokebox"

20. Grab a distro
● Docker Hub has hundreds of distro images.
● I'll use Gentoo here: it includes build environment.
No need to install gcc, make...
“stage3” is the starting point.

21. Installing Stage3
●
docker import $tarball into gentoo-stage3:YYYYMMDD.
● Layer custom files into smoker-gentoo-stage3:lYYYYMMDD.
root's .exrc .bashrc
● Used to bulid perl, run smoke.

22. Building perl, smoker
● Volumes for /opt, /var/tmp & script files.
Standard paths simplify build scripts.
● “docker run” with command of build script.
build perl.
install & run minicpan.
configure reporting & smoker.
CPAN update to test reporting.

23. Aside: --tmpfs
● docker run –tmpfs has a problem:
The volume has execution turned off.
Fine for data.
Useless for build.
● Avoid the issue with a tmpfs volume.
Volume must be private to build.

24. Creating a tmpfs
● Private tmp
volume via PID.
● tmpfs usually
cleaned up
automatically on
the way out.
● Use volume rm
to be sure.
tmp=”var-tmp-$BASHPID”;
$docker volume create
--driver=local
--opt type=tmpfs
--opt device=tmpfs
--opt o=size=320M
$tmp
&&
docker run
...
-v $tmp:/var/tmp
...
;
# may fail due to non-existant volume.
docker volume rm $tmp;

25. Volumes of Volumes
● Temp volume -v $tmp:/var/tmp
● Perl tarball -v $PWD/perl 5.24.1.tar.gz:/var/tmp/perl.tar.gz‑
● Build script -v $PWD/build-perl:/var/tmp/build-perl
● Perl install -v opt-perl-5.24.1:/opt
● CPAN cache -v /var/lib/CPAN:/var/lib/CPAN

26. Volumes of Volumes
● Temp volume -v $tmp:/var/tmp
● Perl tarball -v $PWD/perl 5.24.1.tar.gz‑ :/var/tmp/perl.tar.gz
● Build script -v $PWD/build-perl:/var/tmp/build-perl
● Perl install -v opt-perl-5.24.1:/opt
● CPAN cache -v /var/lib/CPAN:/var/lib/CPAN

27. Volumes of Volumes
● Temp volume -v $tmp:/var/tmp
● Perl tarball -v $PWD/perl 5.24.1.tar.gz‑ :/var/tmp/perl.tar.gz
● Build script -v $PWD/build-perl:/var/tmp/build-perl
● Perl install -v opt-perl-5.24.1:/opt
● CPAN cache -v /var/lib/CPAN:/var/lib/CPAN

28. Volumes of Volumes
● Temp volume -v $tmp:/var/tmp
● Perl tarball -v $PWD/perl 5.24.1.tar.gz‑ :/var/tmp/perl.tar.gz
● Build script -v $PWD/build-perl:/var/tmp/build-perl
● Perl install -v opt-perl-5.24.1:/opt
● CPAN cache -v /var/lib/CPAN:/var/lib/CPAN

29. Start the ball rolling
● Assemble a container.
● Dispatch the build script.
● “-t” avoids issues with
core test that uses
column/row counts.
● “-a” gets output from
container into log.
$docker volume create
--driver=local
--opt type=tmpfs
--opt device=tmpfs
--opt o=size=320M
$tmp &&
$docker volume create
--driver=local
$perl &&
$docker run --rm
-t
--name=”build-$perl”
-a stdout
-a stderr
"${vols[@]}"
$image
/var/tmp/build-perl
2>&1 | tee build.out;
docker volume rm $tmp;

30. Building Perl
● The fun starts in /var/tmp/build-perl
● Mostly here-scripts.
● Install & configure smoketest engine.
● Includes common non-smoke modules.

31. Building Perl
HOME='/var/lib/CPAN';
(
cd /var/tmp;
gzip -dc < perl.tar.gz | tar xf -;
cd perl*;
jobs=$(grep '^processor' /proc/cpuinfo | wc -l);
export TEST_JOBS=$jobs;
./Configure
-de
-Dprefix=/opt/perl5
-Dman1dir='none'
-Dman3dir='none'
-Doptimize='-O2 -pipe -march=native'
&&
/usr/bin/make -j$jobs all test_harness install;
) ||
exit -1;
● Generic
“perl.tar.gz” via
docker volume
mapping.
● /opt/perl5 is in
external
volume.
● Don't knead
man pages.

32. Building Perl
HOME='/var/lib/CPAN';
● /home is in the
O/S image!
Changes in
container layer
are discarded!!
● Persistent files
go in
/var/lib/CPAN or
/opt.

33. Building Perl
TEST_JOBS=$jobs ... /usr/bin/make -j$jobs all test_harness
install;
Files=2410, Tests=851044, 223 wallclock secs
(172.97 usr 22.15 sys + 862.57 cusr 57.18 csys = 1114.87 CPU)

34. Aside: Testing perl builds
● Use tmpfs for /opt instead of disk volume.
Fast, disposable storage for validating your build scripts.
● git clone <whatever> /scratch;
cd /scratch/whatever;
● docker run … -v /scratch/whatever:/var/tmp/whatever … ;
● build script does cd /var/tmp/whatever; make all test install;

35. Testing bleeding edge perl
● Replace tar xvf with volume:
-v /gitclone/perl-5.25.X:/var/tmp/perl
● Rest of it stays the same.
Configure and make run quickly.
Minimal overhead if core modules don't change.
Lots “up to date” messages from CPAN.

36. Minor setups
● “h2ph”
validates
$PATH.
● Pick your
remote
server.
export PATH=/opt/perl5/bin:$PATH;
h2ph -r -l /usr/include || exit -1;
# at this point install seems usable
lib=$(ls -d /opt/perl5/lib/5* | tail -1);
site=$(ls -d /opt/perl5/lib/site_perl/5* | tail -1);
cpan='/var/lib/CPAN';
build='/var/tmp/CPAN';
remote='http://mirror.uic.edu/CPAN/';
local="$cpan/sources/";

37. CPAN uses /var/lib, /var/tmp
● Push CPAN config onto
stable storage.
● CPAN config is specific
to perl version.
perl -MCPAN -e 'shell'<<CPAN;
yes
o conf cpan_home $cpan
o conf histfile $cpan/histfile
o conf prefs_dir $cpan/prefs
o conf build_dir $build
o conf keep_source_where $local
o conf auto_commit yes
o conf build_dir_reuse yes
o conf check_sigs yes
o conf inhibit_startup_message yes
o conf halt_on_failure no
o conf build_cache 1
o conf index_expire 1
o conf commit
CPAN

38. TLC
● Update CPAN.
● Deal with modules
that require user
input, special
choices.
(
CPAN_RUN_SHELL_TEST_WITHOUT_EXPECT=1;
perl -MCPAN -e 'install CPAN';
perl -MCPAN -e shell <<CPAN;
install Module::Signature
2
CPAN
AUTOMATED_TESTING=1; # avoid interactive prompt
perl -MCPAN -e 'shell <<CPAN;
install Term::Readline::Perl
CPAN
)

39. ● Common.
● Smoke modules.
● Iterate them with
${modules[@]}.
modules=(
YAML::XS
JSON::XS
CPAN::SQLite
IO::Socket::SSL
Net::SSL
Net::SSLeay
HTTP::Date
Test::Most
Metabase::Resource
Module::Version
Module::Version Bundle::CPAN
Log::Log4perl
Bundle::CPAN
CPAN::Mini
CPAN::Reporter
CPAN::Testers
Test::RequiresInternet
WWW::RobotRules
LWP::UserAgent
WWW::Mechanize
CPAN::Reporter::Smoker
Bundle::CPAN::Reporter::Smoker::Tests
Bundle::CPANReporter2
App::SmokeBox::Mini
App::cpanminus::reporter
);
Other modules

40. Generate ID file for Reporter
● ~ is /var/lib/CPAN
Not /home!
(
mkdir ~/.cpantesters;
cd ~/.cpantesters;
metabase-profile <<END;
Steven Lembark
lembark@wrkhors.com
password/secret
END
chmod 0400 metabase_id.json;
)

41. Configure CPAN::Reporter
# Generate test reports if CPAN::Reporter is installed (yes/no)? [no] yes
# Would you like me configure CPAN::Reporter now? [yes] <enter>
# email_from? [] example@example.com
# edit_report? [default:ask/no pass/na:no] <enter>
# send_report? [default:ask/yes pass/na:yes] <enter>
# transport? [Metabase uri https://metabase.cpantesters.org/api/v1/ id_file metabase_id.json] <enter>
# Would you like to run 'metabase-profile' now to create '/root/.cpanreporter/metabase_id.json'? [y] <enter>
perl -MCPAN -e shell <<CPAN;
o conf init test_report
yes
yes
lembark@wrkhors.com
no
yes
CPAN

42. Voyeurism
● Fixing the build uses “docker exec”.
● Attach to a running container.
● Does not require ssh.
● Allows fixing, re-running the build.

43. Voyeurism
● Fixing the build uses “docker exec”.
● Attach to a running container.
● Does not require ssh.
● Allows fixing, re-running the build.
● Or just watching it:
docker exec -it build-perl-5.24.1 'top';

44. Access a running container
● Say you get something like:
t/07_plugins.t ........... ok
t/08_since_epoch.t ....... Can't locate YAML/Syck.pm in @INC (you may need to install the
YAML::Syck module) (@INC contains: /var/tmp/CPAN/minismokebox-0.66-0/blib/lib
/var/tmp/CPAN/minismokebox-0.66-0/blib/arch /opt/perl5/lib/site_perl/5.24.1/x86_64-
linux /opt/perl5/lib/site_perl/5.24.1 /opt/perl5/lib/5.24.1/x86_64-linux /opt/perl5/lib/5.24.1 .)
at t/08_since_epoch.t line 12.
BEGIN failed--compilation aborted at t/08_since_epoch.t line 12.
# Looks like your test exited with 2 before it could output anything.
t/08_since_epoch.t ....... Dubious, test returned 2 (wstat 512, 0x200)
Failed 11/11 subtests

45. Access a running container
● Say you get something like:
● Q: How do you fix it?
t/07_plugins.t ........... ok
t/08_since_epoch.t ....... Can't locate YAML/Syck.pm in @INC (you may need to install the
YAML::Syck module) (@INC contains: /var/tmp/CPAN/minismokebox-0.66-0/blib/lib
/var/tmp/CPAN/minismokebox-0.66-0/blib/arch /opt/perl5/lib/site_perl/5.24.1/x86_64-
linux /opt/perl5/lib/site_perl/5.24.1 /opt/perl5/lib/5.24.1/x86_64-linux /opt/perl5/lib/5.24.1 .)
at t/08_since_epoch.t line 12.
BEGIN failed--compilation aborted at t/08_since_epoch.t line 12.
# Looks like your test exited with 2 before it could output anything.
t/08_since_epoch.t ....... Dubious, test returned 2 (wstat 512, 0x200)
Failed 11/11 subtests

46. Access a running container
● Exec into the container.
● Use CPAN to install the module.
● Which leaves you with:
docker exec -it build-perl-5.24.1 bash –-login;

47. Aside: Container ID
●
docker ps shows what is running.
Includes the container id & name:
$ docker ps;
CONTAINER ID IMAGE COMMAND
CREATED STATUS PORTS NAMES
493d9f285faf jekyll:5000/workhorse/gentoo-stage3-amd64 "/bin/bash -c /var..."
28 minutes ago Up 28 minutes build-perl-5.24.1
# same result:
$ docker exec -it 493d9f285faf 'bash login';‑‑
root@493d9f285faf ~ #
$ docker exec -it build-perl-5.4.21 'bash login';‑‑
root@493d9f285faf ~ #

48. Through the looking glass
● Inside the container you are root.
Kernel's proc struct has unchanged EUID.
● Limited view of the system.
Only container proc's in “ps” or “top”.
Nice on a crowded server.
Runlevel restrictions are ineffective!
● Run “perl -MCPAN” or “make install”.
Helpful if a non-essential test fails.

49. Through the Looking Glass
● Outside docker you have a specific PID & EUID.
● Inside docker the initial process has EUID 0 and PID 1.
● EUID 0 is manageable.
● PID 1 requires that the smoketest proc reap children.
Otherwise you end up with defunct proc's.
Will eventually eat all proc slots with long-running container.
One fix is Docker::Reaper.

50. Remember:
● Changes to the image go to a separate AUFS layer.
● They are discarded.
● Fixing files in /etc/ will rescue this build, not the next one.
● Changes to /opt & /var/lib/CPAN are persistent.

51. TMTOWTDI: minicpan
● Outside of docker.
Maintain CPAN cache in /var/lib.
● Inside docker.
Use /var/lib.

52. Running repeated jobs
● Docker can automatically restart running jobs.
● docker run -d restart=always ...‑‑
Downside: Lots of proc's started up at once.
Issues with tmpfs on system restarts.
● --restart=unless-stopped
Allows stopping volumes on the way down.
Start singly on way back up.

53. Smoketest lifecyle
● Run minicpan once on /var/lib/CPAN.
Global cache of modules.
● Run smoker with O/S images & perly vol's.
● This will only test *one* kernel!
● Use VM's running docker for multi-kernel smoke tests.

54. Zombie apocolypse
● Container startup command is process #1.
● When it dies the container exits.
● It has to reap all children.
Not a big issue for small, single-purpose containers.
Big problem for smoketests.

55. Fork & Reap
● Parent propagates signals.
● Exits with $? from child process.
● Loop reaps zombies.
Without this process slots will be
exhausted.
If that happens: log on as SU
and stop, rm the container.
for(;;)
{
my $pid = wait;
$pid != -1
or die "Lost child pid ($child)";
$pid != $child
and next;
return $?
}

56. Smoking
● Volumes for
perl
CPAN
/var/tmp
● Run smoker.
Never exits.
vols=(
"-v $perl:/opt:ro"
"-v $cpan:$cpan"
"-v $tmp:/var/tmp"
);
(
$docker volume create
...
$tmp
&&
exec $run
-t
--name="$cname"
${vols[@]}
$image
"/var/lib/CPAN/.cpantesters/docker-smoker"
) 2>&1 |
tee $log;

57. Propsed change
● Smoker processes recents and exit.
Simplifies iterating multiple perl verisons.
● Goal:
Product of images, perl vol's.
Automate bleed test.

58. Summary
● Smoking can be hazardous.
● LXC provides lightweight solution.
● Mix O/S image, perl volume.
O/S is fixed.
Volume updated to maintain smoker.
● git@github.com:lembark/docker-smoker.git
Mostly shell kwikhaks.
Working to make it cleaner, perly.