Using virtualisation and pkgsrc in heterogeneous networks


Published on

In an environment where heterogeneity cannot be avoided, pkgsrc's portability allows the system administrator to easily deploy a uniform set of packages across all architectures and systems. Virtualisation helps cutting down the costs of building and maintaining the packages.

Published in: Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Using virtualisation and pkgsrc in heterogeneous networks

  1. 1. Using virtualisation and pkgsrc Quentin Garnier pkgsrcCon 2007 April 27th, 2007
  2. 2. Who, Why, Where, When? <ul><li>I manage all IT at an EDA company named EVE </li></ul><ul><li>2 different needs </li></ul><ul><ul><li>packages for my NetBSD servers </li></ul></ul><ul><ul><li>work environment for R&D on Linux </li></ul></ul><ul><ul><li>Goal is the same: save space and CPU </li></ul></ul><ul><li>Bulk building under Xen for my NetBSD servers for the past 3 years </li></ul><ul><li>The Linux part is recent, and still work in progress </li></ul>
  3. 3. Why Xen? <ul><li>For the NetBSD part, because there isn't really a choice </li></ul><ul><li>Experience with it on Linux, although VMWare is probably a good alternative </li></ul><ul><li>Intense cool free software factor </li></ul>
  4. 4. Xen terminology <ul><li>Paravirtualisation (PV) </li></ul><ul><ul><li>Requires a special kernel in the virtual machine </li></ul></ul><ul><li>Full virtualisation </li></ul><ul><ul><li>FV machines are often called HVM </li></ul></ul><ul><ul><li>Requires special support from the CPU </li></ul></ul><ul><ul><li>Any OS can run, the hardware of the HVM is emulated by QEMU </li></ul></ul><ul><li>Dom0 vs. DomU </li></ul>
  5. 5. Packages for servers: why <ul><li>Not to waste a real machine on an infrequent need </li></ul><ul><li>PV can be used for NetBSD packages </li></ul><ul><ul><li>NetBSD supports Xen 2 since release 3.0 </li></ul></ul><ul><ul><li>Userland is no different to NetBSD/i386 </li></ul></ul>
  6. 6. Packages for servers: how <ul><li>Details of the management of the packages once built is irrelevant </li></ul><ul><ul><li>Meta-packages for each server </li></ul></ul><ul><ul><li>Options are common for all servers </li></ul></ul><ul><ul><li>Management is easy but not very incremental </li></ul></ul><ul><ul><li>Stoned described a much better solution last year to manage packages </li></ul></ul>
  7. 7. Packages for servers: caveat <ul><li>My servers run NetBSD 2.0.3_STABLE </li></ul><ul><ul><li>Kernel version inside the PV domain must be faked </li></ul></ul><ul><li>I have upgraded my workstation to the post newlock2 era </li></ul><ul><ul><li>Feel the binary compatibility breakage pain </li></ul></ul><ul><ul><li>Considering a SA compatibility stub </li></ul></ul>
  8. 8. Performance <ul><li>Host is a Pentium 4 2.6 GHz </li></ul><ul><ul><li>VMs get 384MB of memory </li></ul></ul><ul><li>Rarely takes more than a few hours </li></ul><ul><ul><li>Lots of packages don't change often </li></ul></ul><ul><ul><li>Doing nothing takes more than 30 minutes </li></ul></ul>
  9. 9. Linux at EVE (1) <ul><li>Theoretically, only support for RedHat 8 and RedHat Enterprise Linux 3 </li></ul><ul><li>Practically, what our beloved clients use </li></ul><ul><ul><li>E.g., Very Important Client (VIC) uses SuSE Linux Enterprise Server 9 </li></ul></ul><ul><ul><li>They have their own toolchain, incidentally needing support for </li></ul></ul><ul><ul><li>Lots of requests for RHEL 4, SuSE 10 </li></ul></ul><ul><li>Heterogeneous network </li></ul>
  10. 10. Linux at EVE (2) <ul><li>Plan to support only and x86_64 in HEAD </li></ul><ul><ul><li>But we still have to support older versions of our software for some time, notably for VIC </li></ul></ul><ul><li>Developers need a consistent environment </li></ul><ul><li>QA needs specific tools in all environments </li></ul>
  11. 11. First attempt <ul><li>I tried the “greatest common divisor” approach </li></ul><ul><ul><li>Packages compiled on rh8-i386 </li></ul></ul><ul><ul><li>PREFER_PKGSRC set to yes, with a few exceptions </li></ul></ul><ul><li>pam-ldap is teh shit </li></ul><ul><ul><li>the compiled sudo doesn't work on SLES9 </li></ul></ul>
  12. 12. Current attempt <ul><li>Virtualisation helps consolidating the resources </li></ul><ul><li>The machines wouldn't have to be up all the time anyway </li></ul><ul><li>8 platforms to support initially: </li></ul><ul><ul><li>rh8-i386, rhel{3,4}-{i386,amd64} </li></ul></ul><ul><ul><li>sles9-i386, sles10-{i386,amd64} </li></ul></ul><ul><ul><li>More to come (e.g., rhel5-amd64), some to go away </li></ul></ul>
  13. 13. Virtualisation as the obvious solution <ul><li>I don't have much resources to affect to that task </li></ul><ul><li>Performance is never an issue </li></ul><ul><ul><li>Security of packages such as firefox are not a top priority </li></ul></ul><ul><li>Developers can access a pristine environment for each platform, for tests with our software </li></ul>
  14. 14. Hardware configuration <ul><li>4-core machine seems enough </li></ul><ul><ul><li>cheap enough for a bi-Xeon 51xx </li></ul></ul><ul><ul><li>4GB of memory, reserving ~1GB for dom0 </li></ul></ul><ul><ul><li>3 or 4 running HVMs </li></ul></ul><ul><li>Dom0 gets its own CPU, in order to help (slightly) with I/Os </li></ul><ul><li>Each VM gets a 8GB LVM volume as its disk </li></ul><ul><ul><li>The rest is served by dom0 through NFS </li></ul></ul><ul><ul><li>Not enough for rhel4-amd64! </li></ul></ul>
  15. 15. Installations of Linux in the HVMs <ul><li>RedHat 8 was a real pain </li></ul><ul><ul><li>Installation kernel crashed </li></ul></ul><ul><ul><li>Latest RedHat 8 kernel would freeze during installation, and would have a lot of trouble keeping time afterwards </li></ul></ul><ul><li>Others went more or less easy </li></ul><ul><ul><li>Time keeping under Xen/QEMU is hard, NTP is required </li></ul></ul><ul><ul><li>2.6 kernels are much better </li></ul></ul>
  16. 16. Configuration of the Linux HVMs <ul><li>Network </li></ul><ul><li>Mount points </li></ul><ul><ul><li>/auto/pkg </li></ul></ul><ul><ul><li>/pkgbuild/src </li></ul></ul><ul><ul><li>/pkgbuild/obj </li></ul></ul><ul><li>NTP </li></ul><ul><li>pkg_install </li></ul><ul><li>poweroff </li></ul>
  17. 17. Configuration of pkgsrc <ul><ul><li>_EVE_HOST!= hostname -s </li></ul></ul><ul><ul><li>PKG_DBDIR= /auto/pkg/${_EVE_HOST}/db </li></ul></ul><ul><ul><li>LOCALBASE= /auto/pkg/${_EVE_HOST}/base </li></ul></ul><ul><ul><li>VARBASE= /auto/pkg/${_EVE_HOST}/var </li></ul></ul><ul><ul><li>PKG_TOOLS_BIN= /auto/pkg/${_EVE_HOST}/base/sbin </li></ul></ul><ul><ul><li>PKGMANDIR= man </li></ul></ul><ul><ul><li>FETCH_CMD= /auto/pkg/${_EVE_HOST}/base/bin/ftp </li></ul></ul><ul><ul><li>TOOLS_PLATFORM.pax?= /auto/pkg/${_EVE_HOST}/base/bin/pax </li></ul></ul><ul><ul><li>TOOLS_PLATFORM.tar?= /bin/tar </li></ul></ul><ul><ul><li>WRKOBJDIR= /pkgbuild/obj/obj </li></ul></ul><ul><ul><li>OBJHOSTNAME= yes </li></ul></ul><ul><ul><li>DISTDIR= /pkgbuild/obj/dist/${_EVE_HOST} </li></ul></ul><ul><ul><li>PACKAGES= /pkgbuild/obj/packages/${_EVE_HOST} </li></ul></ul><ul><ul><li>PKG_OPTIONS.sudo= ldap pam </li></ul></ul><ul><ul><li>PKG_OPTIONS.libgnomeprint= cups </li></ul></ul><ul><ul><li>PKG_OPTIONS.evolution= ssl </li></ul></ul><ul><ul><li>BULKFILESDIR= /pkgbuild/obj/bulk/files/${_EVE_HOST} </li></ul></ul><ul><ul><li>ALLOW_VULNERABLE_PACKAGES= yes </li></ul></ul><ul><ul><li>_ACCEPTABLE= yes </li></ul></ul><ul><ul><li>CFLAGS+= -pipe </li></ul></ul><ul><ul><li>CXXFLAGS+= -pipe </li></ul></ul>
  18. 18. Installation of packages <ul><li>/auto/pkg is what end clients see </li></ul><ul><li>The mount point the build VMs see is only for building </li></ul><ul><li>The actual /auto/pkg is populated by rsync on the dom0 </li></ul>
  19. 19. Built packages <ul><li>seamonkey, evolution, thunderbird, firefox </li></ul><ul><ul><li>Those takes a lot of time </li></ul></ul><ul><li>gcc4 </li></ul><ul><ul><li>Not so hard to compile, but takes ages to extract </li></ul></ul><ul><li>bvi, gtkwave, some Perl packages </li></ul>
  20. 20. pkgsrc issues (1) <ul><li>Linux/x86_64 support is poor </li></ul><ul><ul><li>buildlink3 lacks knowledge of ${ABI_SUFFIX} </li></ul></ul><ul><ul><li>x11-links, too </li></ul></ul><ul><li>devel/autoconf213 errors out on SLES9 </li></ul><ul><ul><li>I have to rebuild autoconf for all the Gecko packages </li></ul></ul><ul><li>openoffice2-bin doesn't support x86_64 </li></ul><ul><ul><li>But I make it install 32bits binaries </li></ul></ul>
  21. 21. pkgsrc issues (2) <ul><li>Trying a release branch, 2007Q1 </li></ul><ul><ul><li>Immediately stumbled upon an issue fixed in HEAD but for which the committer didn't request a pull-up </li></ul></ul><ul><li>RedHat wants you to use openssl.pc </li></ul><ul><li>Firefox plug-ins packages can't use a native firefox </li></ul>
  22. 22. pkgsrc issues (3) <ul><li>Build of Gecko packages break at some point on Linux/x86_64 </li></ul><ul><ul><li>Doing “make build” again works, but it's not possible during a bulk build </li></ul></ul><ul><li>The rest in related to packages behaving in a special way on Linux </li></ul><ul><ul><li>fake Xfixes.pc expected </li></ul></ul><ul><ul><li>gnutls and ld version script </li></ul></ul><ul><ul><li>gnome-vfs2 and krb5 </li></ul></ul><ul><ul><li>acroread and pax-as-tar </li></ul></ul>
  23. 23. Xen-related issues <ul><li>Passed the installation, time-keeping and poweroff issues, everything is just fine </li></ul><ul><li>Controlling domains go through a mix of two different sets of C APIs and a Python API </li></ul><ul><ul><li>No documentation whatsoever, of course </li></ul></ul><ul><ul><li>Thus that part is still in early stages </li></ul></ul>
  24. 24. Performance <ul><li>Takes slightly more than 24 hours on rhel4-i386 for the build of all my packages </li></ul><ul><li>rhel3-i386 takes a lot longer </li></ul><ul><li>rh8-i386 takes very, very, very long </li></ul><ul><ul><li>although it uses the same kernel as rhel3-i386 </li></ul></ul><ul><ul><li>build launched on Monday still isn't finished </li></ul></ul>
  25. 25. Conclusion <ul><li>Still a lot of work is needed to automate things </li></ul><ul><ul><li>But first I'd like all packages to compile on all platforms... </li></ul></ul><ul><ul><li>Squid is still not correctly configured </li></ul></ul><ul><li>pkgsrc is the ideal solution to the problem at stake, and it's not just zealotism </li></ul><ul><li>Virtualisation answers a very specific need nicely, but it's not perfect </li></ul>
  26. 26. Questions and comments <ul><li>There are probably many ways of improving that system... </li></ul>