The document outlines Paulo Castro's experience and methodology for packaging Perl applications and dependencies at Net-a-Porter. It discusses the challenges encountered with dependency management, including various operating systems, library linking issues, and the need for an unattended installation process using RPM packaging. The document highlights the tools used, problems faced, and innovative solutions implemented to streamline the packaging and deployment of Perl modules.

1. Packaging Perl...
Paulo Castro
London Perl Workshop 2010 Perl Developer
Packaging Perl and dependencies Net-A-Porter

2. Introduction
• Paulo Edgar Castro aka (G@SP@R)
• Portuguese
• System Administrator
• Perl Developer ( writing baby perl since 1999 )
• Arrived in London in August 2007
• Came to learn and to be closer to technology
• No perl jobs in Porto, a lot of proprietary tech
• MobileStreams, RedBeeMedia, Net-A-Porter

3. Brief Summary
• Where we were / Who we were
• The problems of such approach
• The decision
• Investigation and Initial steps
• Questions
• Tools (part I)
• Problems and pitfalls
• Tools (part II)
• Getting closer
• More problems

4. Brief Summary
• Evil Hacks
• Wrapping it up
• Pros/Cons
• Final thoughts

5. Where we were / Who we were
• 64 bit perl 5.8.8 compiled for CentOS 5.2
• GIT repository of perl binaries and CPAN modules
checked out into standard dir (prefix=/opt/zt)
• export PATH=/opt/zt/zt-perl/bin:$PATH
• Developers using different OS'es Arch's ( Ubuntu 32,
Ubuntu 64, Fedora, OSX, Gentoo, Windows ?!?!?
QUACK !!! )
• cpan My::Evil::Module; git commit -am "Install/Update
My::Evil::Module"
• Deployment consisted of App tarball + GIT checkout tarball

6. The problems of such approach
• Very tricky for developers to update modules which build
.so libs
• .so linked against developer libs which were more recent
than the libs present in CentOS 5.2
• Only a rough idea of the modules we were currently using
• Production/Staging/Testing boxes had to be manually pre-
installed with all the dependencies
(libjpg,libpng,libgd,libdb,...)
• Trial and error deployment into new boxes ( "Oh, we forgot
to install libpng, that's why the bar-codes aren't being
generated!" )
• Only ran tests for modules being installed/updated

7. The decision
• We needed a way to describe all these related
dependencies
• I suggested RPM since I was very familiar with it ( My first
job in London consisted of packaging up their perl app in a
nice RPM package )
• Our current production system was RedHat
• We didn't seem to really have any other viable options to
accomplish this
• Our ultimate goal was, to be able to deploy our apps into a
minimal installed box and have the whole process figure
out things for us rather than the other way around.
LET'S GO!

8. Investigation and Initial steps
• We knew more or less what we wanted
• Not entirely sure what the end result was going to be
• We needed to know what we had (CPAN modules
inventory)
• 2 Apps, 2 Makefile.PL with deps, merged together yielded
235 CPAN packages
• Grand total of 618 CPAN modules to satisfy the deps of the
235.
• Dug through CPAN, PerlMonks, Google for answers ....
• cpanspec ?
• cpan2rpm ?

9. Questions
• 1 perl bin rpm package + n perl modules packages ?
• (this is overkill, who's going to maintain 618 rpm builds)
• 1 perl bin rpm package + 1 perl modules package ?
• 1 perl bin + modules package ?
• How will we handle our 618 sources ?
• CPAN mirror ?
• Customized CPAN repo of the modules we use ?
• How are we going to manage the unattended install of 618
packages ?!?!? 'Oh dear... :( Frack, let me check
jobs.perl.org'

10. Tools (part I)
• pmtools, to generate us an inventory of what we had
Text::CSV_XS (0.76) - comma-separated values manipulation routines
Compress::Bzip2 (2.09) - Interface to Bzip2 compression library
PPI (1.213) - Parse, Analyze and Manipulate Perl (without perl)
XML::Parser::Expat (2.19) - Lowlevel access to James Clark's expat XML parser
XML::Parser (2.19) - A perl module for parsing XML documents
• A couple of bash scripts to do some simple greping and
parsing of the results ( I know this is a perl workshop, but
bash one-liners seemed good enough for the job )
• Mark Overmeer's CPAN::Site, quote "The cpansite script is
used to create your own CPAN server." - GREAT STUFF!
• How the frack are we going to create an unattended install
of the CPAN deps ? 618 of them ?

11. Tools (part I)
• I remembered how we were in some cases installing new
modules... ( a way I'd used before I confess ... )
• Add the modules to the project Makefile.PL which is based in
Module::Install
• perl Makefile.PL; make; echo “Module installed”
• What about we assume perl is a project with a Makefile.PL full of
dependencies ?

12. Tools (part I)
Typical Makefile.PL layout
use strict; use warnings; use inc::Module::Install 0.98;
use Module::Install::AuthorRequires; use Module::Install::ExtraTests; use 5.008003;
name 'Moose';
perl_version '5.008003';
all_from 'lib/Moose.pm';
license 'perl';
requires 'Carp';
requires 'Class::MOP' => '1.11';
requires 'Data::OptList' => '0';
requires 'List::MoreUtils' => '0.12';
requires 'Package::DeprecationManager' => '0.10';
requires 'Params::Util' => '1.00';
requires 'Scalar::Util' => '1.19';
requires 'Sub::Exporter' => '0.980';
requires 'Sub::Name' => '0';
requires 'Task::Weaken' => '0';
requires 'Try::Tiny' => '0.02';
test_requires 'Test::More' => '0.88';
test_requires 'Test::Fatal' => '0.001';
test_requires 'Test::Requires' => '0.05';
......
.....

13. Problems and pitfalls
• Getting the sources for the modules
• wget ${Module}-${Version}.tar.gz from cpan
• if (!$found) { wget ${Module}-${Version}.tar.gz from backpan }
else { echo "Damn it, GRRRR, $#$%#$%/#$%#$ " }
• Some module versions just couldn't be found anywhere.
• Solution: Install the closest higher version and hope that
everything will be OK
• Other modules had been tweaked in house and their version
bumped manually to say (0.78_02) not matching anything
found in the wild
• This meant, the module was actually a couple versions below
(0.78) and it had a in-house patch applied.

14. Problems and pitfalls
• Some perl modules had deps on third-party libraries like
mysql-devel, db4-devel, openssl-devel, postgresql-devel,
libxml2-devel, etc...
• Others, Math::Pari, depended on specific tarballs to be
installed
• Some modules didn't pass some tests
• Others didn't pass any test at all....
• How the frack did they got installed in the first place ???
“Oh Lordy!!! Is it time to
check jobserve this time ?”

15. Tools (part II)
• RPM spec files
• They're just a clever wrapper around a tarball of stuff built in
a temporary directory
• With this wrapper we can
• Specify build requirements
• Specify lib dependencies
• Provide information about our package
• Describe a recipe for the build/installation process
• Changelog of the actions
• Just before creating the final package, foreach existing file,
rpmbuild lists:
• What the file is providing/requiring

16. Tools (part II)
Provides Example:
Byte.so()(64bit) ByteLoader.so()(64bit) Bzip2.so()(64bit) C.so()(64bit)
Opcode.so()(64bit) POSIX.so()(64bit) mysql.so()(64bit) perl(Algorithm::C3) =
0.07 perl(Algorithm::Diff) = 1.19 perl(Algorithm::Diff::_impl) perl(Catalyst::Log)
perl(Catalyst::Log::Log4perl) = 1.00 perl(Catalyst::Model)
perl(Catalyst::Model::ActiveMQ) perl(Catalyst::Model::DBIC::Schema) = 0.29
perl(Catalyst::Model::DBIC::Schema::Types) perl(Catalyst::Model::File) = 0.08
perl(DBD::mysql) = 4.007 perl(DBD::mysql::GetInfo) perl(DBD::mysql::db)
perl(Moose) >= 0.82 perl(PPI::Token::Pod) perl(PPI::Token::Prototype) .....
Requires Example:
libc.so.6(GLIBC_2.4)(64bit) libcrypt.so.1()(64bit) libgd.so.2()(64bit) libgmp.so.3()
(64bit) libjpeg.so.62()(64bit) libm.so.6()(64bit) libm.so.6(GLIBC_2.2.5)(64bit)
libmysqlclient.so.16()(64bit) libnsl.so.1()(64bit) libpng12.so.0()(64bit) .....

17. Tools (part II)
Example spec file
Summary: A searchable xml/html based mailinglist %install
archiver %makeinstall
Name: lurker install -d -m0755 $RPM_BUILD_ROOT/%
Version: 1.1 {_sysconfdir}
Release: 1.evo.2 install -m0755 lurker.conf $RPM_BUILD_ROOT/%
Group: Applications/Internet {_sysconfdir}/lurker.conf
License: GPL install -d -m2775 $RPM_BUILD_ROOT/%
URL: http://lurker.sourceforge.net/ {_localstatedir}/www/lurkdb
Packager: Jim Perrin <perrin@ohio.edu> %clean
Source0: lurker-%{version}.tar.gz rm -rf $RPM_BUILD_ROOT
Source1: mimelib-3.1.1.tar.gz %files
BuildRequires: zlib-devel %defattr(-,root,root)
Requires: libxslt %doc AUTHORS ChangeLog COPYING FAQ NEWS
BuildRoot: %{_tmppath}/%{name}-%{version}-root README INSTALL
%description %{_mandir}/man1/*
Lurker is not just another mailing list archiver. Blá blá %{_bindir}/*
blá %{_libdir}/*
%prep %config(noreplace) %{_sysconfdir}/lurker.conf
%setup -a 1 %defattr(-,apache,apache)
%build %{_localstatedir}/www/lurker/
./configure --with-mimelib-local %{_localstatedir}/www/lurkdb/
--prefix=$RPM_BUILD_ROOT %changelog
--localstatedir=$RPM_BUILD_ROOT/make %{? * Fri Jan 2 2004 Jim Perrin <perrin@ohio.edu>
_smp_mflags} - Changed the spec file to better use macros

18. Tools (part II)
• CPAN Distroprefs (Configuration for individual distributions)
• A distropref file per badly behaved package
• Define which tests to run, excluding the ones that fail and
with which we're ok
• Pass additional arguments to one of the four commands
• Set environment variables
• Instantiate an Expect object that reads from the console,
waits for some regular expressions and enters some answers
• Temporarily override assorted CPAN.pm configuration
variables
• Specify dependencies the original maintainer forgot
• Disable the installation of an object altogether
• Examples:

19. Tools (part II)
DB_File.yml
---
match:
distribution: "/DB_File-d"
pl:
env:
DB_FILE_INCLUDE: "/usr/include"
DB_FILE_LIB: "/usr/lib64"
Convert-PEM-0.07.yml
---
comment: "skip test t/01-readwrite.t, i got fed up of it failing randomly
sometimes without any noticeable explanation."
match:
distribution: "/Convert-PEM-d"
test:
args:
- TEST_FILES="t/00-compile.t t/02-encode.t t/03-ede3.t"

20. Tools (part II)
Alien-ActiveMQ-0.000003.yml
---
comment: Module doesn't explicitly require the dependency below, so it fails
miserably when it tried to require it.
match:
distribution: "/Alien-ActiveMQ-0.00003"
depends:
requires:
File::Copy::Recursive: 0
patches:
- "MYCOMPANY/patches/Alien-ActiveMQ.patch"
Term-ReadLine-Gnu.yml
---
comment: "Skip the entire test suite - hudson doesn't run with a real /dev/tty"
match:
distribution: "/Term-ReadLine-Gnu-1"
test:
args:
- TEST_FILES=""
expect:
- "Skip the test suite?"
- "Yesn"

21. Getting Closer
• Merge projects Makefile.PL creating a single Makefile.PL
with everything
• Create our CPAN site with our packages, patches, and our
distroprefs files
• Create a rigged CPAN/Config.pm based on sane defaults
• 'urllist' => [q[file:///home/pecastro/build/CPANSITE]],
• 'make_arg' => q[-j 3]
• 'connect_to_internet_ok' => q[0]
Info for the next slides....
• %new_perl is an RPM variable that means
• %{new_perl_flags} $RPM_BUILD_ROOT/%{perl_bin_dir}/perl
• %define new_perl_flags LD_LIBRARY_PATH=%{archlib}/CORE
• %perl_bin_dir = %perl_base_dir/bin; %perl_base_dir=/opt/zt/zt-perl

22. Getting Closer
• Main recipe so far:
• Extract perl sources
• %perl_base_dir = /opt/zt/zt-perl
• Configure -Dprefix=%perl_base_dir; make;
• make install DESTDIR=~/rpmbuild/opt/zt/zt-perl ( this will
install us new perl )
• export PERL5LIB,PERLLIB,PATH to match temporary
installation dir "~/rpmbuild/opt/zt/zt-perl/{bin,lib}"
• Copy our rigged CPAN/Config.pm into the right place
• %new_perl -MCPAN -e 'install Bundle::CPAN' ( so we have a
sane CPAN client that can do our bidding )
• %new_perl -MCPAN -e 'install Module::Install' ( so we can
use our auto magical unattended installation process
outsourcing dependency sorting to CPAN )

23. Getting Closer
• export PERL_MM_USE_DEFAULT=1 ( This prevents
automated processes from blocking on user input. )
• copy Makefile.PL into cwd
• %new_perl Makefile.PL --defaultdeps
• make
• Installs all the modules defined in Makefile.PL and their
dependencies as well auto-magically....

24. Getting Closer
This is excellent.
All we need now is to wrap up all of this in the SPEC
file and presto....
perl RPM with everything we need ! ROCK ON...

25. More problems
• When deployed, our custom perl, will live in /opt/zt/zt-
perl/bin
• What happens when we're building a perl binary & friends in
a system which already has a custom perl installed ?
• Kaboom !!! But Why Kaboom ?
• our ~/rpmbuild/opt/zt/zt-perl/bin/perl is being built to live
in /opt/zt/zt-perl/bin/perl and even though it's being run from
a temporary location, has it's INC paths pointing to it's final
location /opt/zt/zt-perl/lib ... hardcoded in the perl binary.
• When we say:
• %new_perl Makefile.PL --defaultdeps && make ;
• It replies: Oh, I can see that I already have everything
installed. Nothing else to do..... :(

26. Evil Hacks
• After having built perl and before starting auto magical
modules installation
• Temporarily hack Config.pm and Config_heavy.pl
%define archlib $RPM_BUILD_ROOT/%{perl_lib_dir}/%version/%
{perl_archname}
%new_perl -pi -e "
s|%perl_base_dir|$RPM_BUILD_ROOT%perl_base_dir|g"
$RPM_BUILD_ROOT/%archlib/Config.pm
$RPM_BUILD_ROOT/%archlib/Config_heavy.pl
And now...........The most EVIL HACK of all hacks......

27. Evil Hacks
%perl_base_dir = /opt/zt/zt-perl
garbled_base_dir=`
%new_perl -e '$_=”%perl_base_dir”;s|/|%|g;print'
`
%new_perl -pi -e "
s|%perl_base_dir|$mess_base_dir|g"
$RPM_BUILD_ROOT%{perl_bin_dir}/perl
REMEMBER.....
●%new_perl is an RPM variable that means
● %{new_perl_flags} $RPM_BUILD_ROOT%{perl_bin_dir}/perl
● %define new_perl_flags LD_LIBRARY_PATH=%{archlib}/CORE
● %perl_bin_dir = %perl_base_dir/bin; %perl_base_dir=/opt/zt/zt-perl

28. Evil Hacks
Characteristics of this binary (from libperl):
Compile-time options: PERL_MALLOC_WRAP USE_64_BIT_ALL USE_64_BIT_INT
USE_LARGE_FILES USE_PERLIO
Built under linux
Compiled at Dec 1 2010 20:52:24
%ENV:
PERL5LIB="/home/pecastro/build/rpmbuild/tmp/perl-zap-5.8.8-00-build/opt/zt/zt-
perl/lib/5.8.8:/home/pecastro/build/rpmbuild/tmp/perl-zap-5.8.8-00-build/opt/zt/zt-
perl/lib/site_perl/5.8.8"
PERLLIB="/home/pecastro/build/rpmbuild/tmp/perl-zap-5.8.8-00-build/opt/zt/zt-
perl/lib/5.8.8"
@INC:
/home/pecastro/build/rpmbuild/tmp/perl-zap-5.8.8-00-build/opt/zt/zt-
perl/lib/5.8.8/x86_64-linux
/home/pecastro/build/rpmbuild/tmp/perl-zap-5.8.8-00-build/opt/zt/zt-perl/lib/5.8.8
/home/pecastro/build/rpmbuild/tmp/perl-zap-5.8.8-00-build/opt/zt/zt-
perl/lib/site_perl/5.8.8/x86_64-linux
/home/pecastro/build/rpmbuild/tmp/perl-zap-5.8.8-00-build/opt/zt/zt-
perl/lib/site_perl/5.8.8
%opt%zt%zt-perl/lib/5.8.8/x86_64-linux
%opt%zt%zt-perl/lib/5.8.8
%opt%zt%zt-perl/lib/site_perl/5.8.8/x86_64-linux
%opt%zt%zt-perl/lib/site_perl/5.8.8
%opt%zt%zt-perl/lib/site_perl
.
[pecastro@brutalix perl-zap-5.8.8]$ new_perl -VT
Can't locate Config.pm in @INC (@INC contains: %opt%zt%zt-perl/lib/5.8.8/x86_64-linux %opt
%zt%zt-perl/lib/5.8.8 %opt%zt%zt-perl/lib/site_perl/5.8.8/x86_64-linux %opt%zt%zt-
perl/lib/site_perl/5.8.8 %opt%zt%zt-perl/lib/site_perl).
BEGIN failed--compilation aborted

29. Wrapping it up
• RPM specifics
• BuildRequires: postgresql-devel expat-devel openssl-devel gd-
devel graphviz jdk , etc .... to specify all the third-party
dependencies
• %prep stage used to do custom validation and basic unpacking of
sources
• Example: mysql -utest -ptest -e 'use test' || echo
"echo "GRANT ALL PRIVILEGES ON test.* to test@localhost
identified by 'test'" |mysql -uroot" && exit 1
• %build stage to configure and build perl
• %install stage to install it and do all the magical Makefile.PL bits
• Make sure you undo your Evil (Config.pm, Config_heavy.pl, perl)
• Amend our rigged CPAN/Config.pm
• o conf build_dir /tmp/.cpan/build ;
• o conf make_install_make_command 'sudo make'
• o conf urllist http://mirrorservice.org/sites/ftp.cpan.org/pub/CPAN

30. Pros/Cons
• Pros
• Easily deployed package with dependency management
• Easy to spot missing packages ( perspective of dependent
package )
• Fast spotting of badly behaved modules
• Module install/test failure means no build
• Enabled us to replicate environments like if they were
mushrooms
• Cons
• Module updates not as straightforward as before
• Tricky CPAN module dependency
• Build time tied to CPU (84 min @ Virtual Double-Core Xeon
2.33GHz/4GB mem)

31. Final Thoughts
• I'd like to thank my colleagues for their input and feedback
• Recommended experiment
• Most likely to be possible to replicate the same approach in
other Linux/Unix flavors (Debian,Gentoo,...)
• The initial process is tedious but worth going through
• Once everything is streamlined there's no thinking just
building on top of...
• I've heard of other strategies to do deployment but
honestly.....
• This is currently the one that makes more sense to me

32. END {
say “QUESTIONS ?”;
say “Paulo Castro”;
say “<pauloedgarcastro@gmail.com>”;
close $SLIDE_FILE;
}