Packaging perl (LPW2010)

Packaging Perl... Paulo CastroLondon Perl Workshop 2010 Perl DeveloperPackaging Perl and dependencies Net-A-Porter

Introduction• Paulo Edgar Castro aka (G@SP@R)• Portuguese• System Administrator• Perl Developer ( writing baby perl since 1999 )• Arrived in London in August 2007 • Came to learn and to be closer to technology • No perl jobs in Porto, a lot of proprietary tech • MobileStreams, RedBeeMedia, Net-A-Porter

Brief Summary• Where we were / Who we were• The problems of such approach• The decision• Investigation and Initial steps• Questions• Tools (part I)• Problems and pitfalls• Tools (part II)• Getting closer• More problems

Brief Summary• Evil Hacks• Wrapping it up• Pros/Cons• Final thoughts

Where we were / Who we were• 64 bit perl 5.8.8 compiled for CentOS 5.2• GIT repository of perl binaries and CPAN modules checked out into standard dir (prefix=/opt/zt)• export PATH=/opt/zt/zt-perl/bin:$PATH• Developers using different OSes Archs ( Ubuntu 32, Ubuntu 64, Fedora, OSX, Gentoo, Windows ?!?!? QUACK !!! )• cpan My::Evil::Module; git commit -am "Install/Update My::Evil::Module"• Deployment consisted of App tarball + GIT checkout tarball

The problems of such approach• Very tricky for developers to update modules which build .so libs• .so linked against developer libs which were more recent than the libs present in CentOS 5.2• Only a rough idea of the modules we were currently using• Production/Staging/Testing boxes had to be manually pre- installed with all the dependencies (libjpg,libpng,libgd,libdb,...)• Trial and error deployment into new boxes ( "Oh, we forgot to install libpng, thats why the bar-codes arent being generated!" )• Only ran tests for modules being installed/updated

The decision• We needed a way to describe all these related dependencies• I suggested RPM since I was very familiar with it ( My first job in London consisted of packaging up their perl app in a nice RPM package )• Our current production system was RedHat• We didnt seem to really have any other viable options to accomplish this• Our ultimate goal was, to be able to deploy our apps into a minimal installed box and have the whole process figure out things for us rather than the other way around. LETS GO!

Investigation and Initial steps• We knew more or less what we wanted• Not entirely sure what the end result was going to be• We needed to know what we had (CPAN modules inventory) • 2 Apps, 2 Makefile.PL with deps, merged together yielded 235 CPAN packages • Grand total of 618 CPAN modules to satisfy the deps of the 235.• Dug through CPAN, PerlMonks, Google for answers ....• cpanspec ?• cpan2rpm ?

Questions• 1 perl bin rpm package + n perl modules packages ? • (this is overkill, whos going to maintain 618 rpm builds)• 1 perl bin rpm package + 1 perl modules package ?• 1 perl bin + modules package ?• How will we handle our 618 sources ? • CPAN mirror ? • Customized CPAN repo of the modules we use ?• How are we going to manage the unattended install of 618 packages ?!?!? Oh dear... :( Frack, let me check jobs.perl.org

Tools (part I)• pmtools, to generate us an inventory of what we had Text::CSV_XS (0.76) - comma-separated values manipulation routines Compress::Bzip2 (2.09) - Interface to Bzip2 compression library PPI (1.213) - Parse, Analyze and Manipulate Perl (without perl) XML::Parser::Expat (2.19) - Lowlevel access to James Clarks expat XML parser XML::Parser (2.19) - A perl module for parsing XML documents• A couple of bash scripts to do some simple greping and parsing of the results ( I know this is a perl workshop, but bash one-liners seemed good enough for the job )• Mark Overmeers CPAN::Site, quote "The cpansite script is used to create your own CPAN server." - GREAT STUFF!• How the frack are we going to create an unattended install of the CPAN deps ? 618 of them ?

Tools (part I)• I remembered how we were in some cases installing new modules... ( a way Id used before I confess ... )• Add the modules to the project Makefile.PL which is based in Module::Install • perl Makefile.PL; make; echo “Module installed” • What about we assume perl is a project with a Makefile.PL full of dependencies ?

Tools (part I) Typical Makefile.PL layoutuse strict; use warnings; use inc::Module::Install 0.98;use Module::Install::AuthorRequires; use Module::Install::ExtraTests; use 5.008003;name Moose;perl_version 5.008003;all_from lib/Moose.pm;license perl;requires Carp;requires Class::MOP => 1.11;requires Data::OptList => 0;requires List::MoreUtils => 0.12;requires Package::DeprecationManager => 0.10;requires Params::Util => 1.00;requires Scalar::Util => 1.19;requires Sub::Exporter => 0.980;requires Sub::Name => 0;requires Task::Weaken => 0;requires Try::Tiny => 0.02;test_requires Test::More => 0.88;test_requires Test::Fatal => 0.001;test_requires Test::Requires => 0.05;...........

Problems and pitfalls• Getting the sources for the modules • wget ${Module}-${Version}.tar.gz from cpan • if (!$found) { wget ${Module}-${Version}.tar.gz from backpan } else { echo "Damn it, GRRRR, $#$%#$%/#$%#$ " }• Some module versions just couldnt be found anywhere. • Solution: Install the closest higher version and hope that everything will be OK • Other modules had been tweaked in house and their version bumped manually to say (0.78_02) not matching anything found in the wild • This meant, the module was actually a couple versions below (0.78) and it had a in-house patch applied.

Problems and pitfalls• Some perl modules had deps on third-party libraries like mysql-devel, db4-devel, openssl-devel, postgresql-devel, libxml2-devel, etc...• Others, Math::Pari, depended on specific tarballs to be installed• Some modules didnt pass some tests• Others didnt pass any test at all.... • How the frack did they got installed in the first place ??? “Oh Lordy!!! Is it time to check jobserve this time ?”

Tools (part II)• RPM spec files • Theyre just a clever wrapper around a tarball of stuff built in a temporary directory• With this wrapper we can • Specify build requirements • Specify lib dependencies • Provide information about our package • Describe a recipe for the build/installation process • Changelog of the actions• Just before creating the final package, foreach existing file, rpmbuild lists: • What the file is providing/requiring

Tools (part II) Provides Example:Byte.so()(64bit) ByteLoader.so()(64bit) Bzip2.so()(64bit) C.so()(64bit)Opcode.so()(64bit) POSIX.so()(64bit) mysql.so()(64bit) perl(Algorithm::C3) =0.07 perl(Algorithm::Diff) = 1.19 perl(Algorithm::Diff::_impl) perl(Catalyst::Log)perl(Catalyst::Log::Log4perl) = 1.00 perl(Catalyst::Model)perl(Catalyst::Model::ActiveMQ) perl(Catalyst::Model::DBIC::Schema) = 0.29perl(Catalyst::Model::DBIC::Schema::Types) perl(Catalyst::Model::File) = 0.08perl(DBD::mysql) = 4.007 perl(DBD::mysql::GetInfo) perl(DBD::mysql::db)perl(Moose) >= 0.82 perl(PPI::Token::Pod) perl(PPI::Token::Prototype) ..... Requires Example:libc.so.6(GLIBC_2.4)(64bit) libcrypt.so.1()(64bit) libgd.so.2()(64bit) libgmp.so.3()(64bit) libjpeg.so.62()(64bit) libm.so.6()(64bit) libm.so.6(GLIBC_2.2.5)(64bit)libmysqlclient.so.16()(64bit) libnsl.so.1()(64bit) libpng12.so.0()(64bit) .....

Tools (part II)Example spec fileSummary: A searchable xml/html based mailinglist %installarchiver %makeinstallName: lurker install -d -m0755 $RPM_BUILD_ROOT/%Version: 1.1 {_sysconfdir}Release: 1.evo.2 install -m0755 lurker.conf $RPM_BUILD_ROOT/%Group: Applications/Internet {_sysconfdir}/lurker.confLicense: GPL install -d -m2775 $RPM_BUILD_ROOT/%URL: http://lurker.sourceforge.net/ {_localstatedir}/www/lurkdbPackager: Jim Perrin <perrin@ohio.edu> %cleanSource0: lurker-%{version}.tar.gz rm -rf $RPM_BUILD_ROOTSource1: mimelib-3.1.1.tar.gz %filesBuildRequires: zlib-devel %defattr(-,root,root)Requires: libxslt %doc AUTHORS ChangeLog COPYING FAQ NEWSBuildRoot: %{_tmppath}/%{name}-%{version}-root README INSTALL%description %{_mandir}/man1/*Lurker is not just another mailing list archiver. Blá blá %{_bindir}/*blá %{_libdir}/*%prep %config(noreplace) %{_sysconfdir}/lurker.conf%setup -a 1 %defattr(-,apache,apache)%build %{_localstatedir}/www/lurker/./configure --with-mimelib-local %{_localstatedir}/www/lurkdb/--prefix=$RPM_BUILD_ROOT %changelog--localstatedir=$RPM_BUILD_ROOT/make %{? * Fri Jan 2 2004 Jim Perrin <perrin@ohio.edu>_smp_mflags} - Changed the spec file to better use macros

Tools (part II)• CPAN Distroprefs (Configuration for individual distributions) • A distropref file per badly behaved package • Define which tests to run, excluding the ones that fail and with which were ok • Pass additional arguments to one of the four commands • Set environment variables • Instantiate an Expect object that reads from the console, waits for some regular expressions and enters some answers • Temporarily override assorted CPAN.pm configuration variables • Specify dependencies the original maintainer forgot • Disable the installation of an object altogether• Examples:

Tools (part II)DB_File.yml---match: distribution: "/DB_File-d"pl: env: DB_FILE_INCLUDE: "/usr/include" DB_FILE_LIB: "/usr/lib64"Convert-PEM-0.07.yml---comment: "skip test t/01-readwrite.t, i got fed up of it failing randomlysometimes without any noticeable explanation."match: distribution: "/Convert-PEM-d"test: args: - TEST_FILES="t/00-compile.t t/02-encode.t t/03-ede3.t"

Tools (part II)Alien-ActiveMQ-0.000003.yml ---comment: Module doesnt explicitly require the dependency below, so it failsmiserably when it tried to require it.match: distribution: "/Alien-ActiveMQ-0.00003"depends: requires: File::Copy::Recursive: 0patches: - "MYCOMPANY/patches/Alien-ActiveMQ.patch"Term-ReadLine-Gnu.yml---comment: "Skip the entire test suite - hudson doesnt run with a real /dev/tty"match: distribution: "/Term-ReadLine-Gnu-1"test: args: - TEST_FILES="" expect: - "Skip the test suite?" - "Yesn"

Getting Closer• Merge projects Makefile.PL creating a single Makefile.PL with everything• Create our CPAN site with our packages, patches, and our distroprefs files• Create a rigged CPAN/Config.pm based on sane defaults • urllist => [q[file:///home/pecastro/build/CPANSITE]], • make_arg => q[-j 3] • connect_to_internet_ok => q[0]Info for the next slides....• %new_perl is an RPM variable that means • %{new_perl_flags} $RPM_BUILD_ROOT/%{perl_bin_dir}/perl • %define new_perl_flags LD_LIBRARY_PATH=%{archlib}/CORE • %perl_bin_dir = %perl_base_dir/bin; %perl_base_dir=/opt/zt/zt-perl

Getting Closer• Main recipe so far: • Extract perl sources • %perl_base_dir = /opt/zt/zt-perl • Configure -Dprefix=%perl_base_dir; make; • make install DESTDIR=~/rpmbuild/opt/zt/zt-perl ( this will install us new perl ) • export PERL5LIB,PERLLIB,PATH to match temporary installation dir "~/rpmbuild/opt/zt/zt-perl/{bin,lib}" • Copy our rigged CPAN/Config.pm into the right place • %new_perl -MCPAN -e install Bundle::CPAN ( so we have a sane CPAN client that can do our bidding ) • %new_perl -MCPAN -e install Module::Install ( so we can use our auto magical unattended installation process outsourcing dependency sorting to CPAN )

Getting Closer • export PERL_MM_USE_DEFAULT=1 ( This prevents automated processes from blocking on user input. ) • copy Makefile.PL into cwd • %new_perl Makefile.PL --defaultdeps • make• Installs all the modules defined in Makefile.PL and their dependencies as well auto-magically....

Getting Closer This is excellent.All we need now is to wrap up all of this in the SPEC file and presto.... perl RPM with everything we need ! ROCK ON...

More problems• When deployed, our custom perl, will live in /opt/zt/zt- perl/bin• What happens when were building a perl binary & friends in a system which already has a custom perl installed ?• Kaboom !!! But Why Kaboom ?• our ~/rpmbuild/opt/zt/zt-perl/bin/perl is being built to live in /opt/zt/zt-perl/bin/perl and even though its being run from a temporary location, has its INC paths pointing to its final location /opt/zt/zt-perl/lib ... hardcoded in the perl binary.• When we say:• %new_perl Makefile.PL --defaultdeps && make ; • It replies: Oh, I can see that I already have everything installed. Nothing else to do..... :(

Evil Hacks• After having built perl and before starting auto magical modules installation• Temporarily hack Config.pm and Config_heavy.pl%define archlib $RPM_BUILD_ROOT/%{perl_lib_dir}/%version/%{perl_archname}%new_perl -pi -e "s|%perl_base_dir|$RPM_BUILD_ROOT%perl_base_dir|g" $RPM_BUILD_ROOT/%archlib/Config.pm $RPM_BUILD_ROOT/%archlib/Config_heavy.pl And now...........The most EVIL HACK of all hacks......

Evil Hacks%perl_base_dir = /opt/zt/zt-perlgarbled_base_dir=`%new_perl -e $_=”%perl_base_dir”;s|/|%|g;print`%new_perl -pi -e "s|%perl_base_dir|$mess_base_dir|g" $RPM_BUILD_ROOT%{perl_bin_dir}/perlREMEMBER.....●%new_perl is an RPM variable that means ● %{new_perl_flags} $RPM_BUILD_ROOT%{perl_bin_dir}/perl ● %define new_perl_flags LD_LIBRARY_PATH=%{archlib}/CORE ● %perl_bin_dir = %perl_base_dir/bin; %perl_base_dir=/opt/zt/zt-perl

Evil HacksCharacteristics of this binary (from libperl): Compile-time options: PERL_MALLOC_WRAP USE_64_BIT_ALL USE_64_BIT_INT USE_LARGE_FILES USE_PERLIO Built under linux Compiled at Dec 1 2010 20:52:24 %ENV: PERL5LIB="/home/pecastro/build/rpmbuild/tmp/perl-zap-5.8.8-00-build/opt/zt/zt-perl/lib/5.8.8:/home/pecastro/build/rpmbuild/tmp/perl-zap-5.8.8-00-build/opt/zt/zt-perl/lib/site_perl/5.8.8" PERLLIB="/home/pecastro/build/rpmbuild/tmp/perl-zap-5.8.8-00-build/opt/zt/zt-perl/lib/5.8.8" @INC: /home/pecastro/build/rpmbuild/tmp/perl-zap-5.8.8-00-build/opt/zt/zt-perl/lib/5.8.8/x86_64-linux /home/pecastro/build/rpmbuild/tmp/perl-zap-5.8.8-00-build/opt/zt/zt-perl/lib/5.8.8 /home/pecastro/build/rpmbuild/tmp/perl-zap-5.8.8-00-build/opt/zt/zt-perl/lib/site_perl/5.8.8/x86_64-linux /home/pecastro/build/rpmbuild/tmp/perl-zap-5.8.8-00-build/opt/zt/zt-perl/lib/site_perl/5.8.8 %opt%zt%zt-perl/lib/5.8.8/x86_64-linux %opt%zt%zt-perl/lib/5.8.8 %opt%zt%zt-perl/lib/site_perl/5.8.8/x86_64-linux %opt%zt%zt-perl/lib/site_perl/5.8.8 %opt%zt%zt-perl/lib/site_perl .[pecastro@brutalix perl-zap-5.8.8]$ new_perl -VTCant locate Config.pm in @INC (@INC contains: %opt%zt%zt-perl/lib/5.8.8/x86_64-linux %opt%zt%zt-perl/lib/5.8.8 %opt%zt%zt-perl/lib/site_perl/5.8.8/x86_64-linux %opt%zt%zt-perl/lib/site_perl/5.8.8 %opt%zt%zt-perl/lib/site_perl).BEGIN failed--compilation aborted

Wrapping it up• RPM specifics • BuildRequires: postgresql-devel expat-devel openssl-devel gd- devel graphviz jdk , etc .... to specify all the third-party dependencies • %prep stage used to do custom validation and basic unpacking of sources • Example: mysql -utest -ptest -e use test || echo "echo "GRANT ALL PRIVILEGES ON test.* to test@localhost identified by test" |mysql -uroot" && exit 1 • %build stage to configure and build perl • %install stage to install it and do all the magical Makefile.PL bits • Make sure you undo your Evil (Config.pm, Config_heavy.pl, perl) • Amend our rigged CPAN/Config.pm • o conf build_dir /tmp/.cpan/build ; • o conf make_install_make_command sudo make • o conf urllist http://mirrorservice.org/sites/ftp.cpan.org/pub/CPAN

Pros/Cons• Pros • Easily deployed package with dependency management • Easy to spot missing packages ( perspective of dependent package ) • Fast spotting of badly behaved modules • Module install/test failure means no build • Enabled us to replicate environments like if they were mushrooms• Cons • Module updates not as straightforward as before • Tricky CPAN module dependency • Build time tied to CPU (84 min @ Virtual Double-Core Xeon 2.33GHz/4GB mem)

Final Thoughts• Id like to thank my colleagues for their input and feedback• Recommended experiment• Most likely to be possible to replicate the same approach in other Linux/Unix flavors (Debian,Gentoo,...)• The initial process is tedious but worth going through• Once everything is streamlined theres no thinking just building on top of...• Ive heard of other strategies to do deployment but honestly.....• This is currently the one that makes more sense to me

END { say “QUESTIONS ?”; say “Paulo Castro”; say “<pauloedgarcastro@gmail.com>”; close $SLIDE_FILE;}

Packaging perl (LPW2010)

by p3castro

Accessibility

Categories

Upload Details

Usage Rights

Statistics

Packaging perl (LPW2010) Presentation Transcript