Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime. presentation


Published on is a tool to help upstream projects build their own packages for various Operating Systems directly from their VCS/CMS on a regular basis. It provides a continuous packaging approch

Published in: Technology
  • Be the first to comment presentation

  1. 1. Project Builder:A GPL continuous packaging solutionBruno Cornec HP EMEA Open Source Profession LeadV2.0 April 2013
  2. 2. Introducing Myself● Software engineering and Unices since 1988 ● Mostly Configuration Management Systems (CMS), Build systems, quality tools, on multiple commercial Unix systems ● Discover Open Source & Linux (OSL) & first contributions in 1993 ● Full time on OSL since 1995, first as HP reseller then @HP● Currently: ● Master Technology Architect on OSL for the HP/Intel Solution Center, Grenoble ● OSL HP Advocate ● EMEA OSL HP Profession Lead ● Solutions Linux Conference and OWF board member ● MondoRescue,, project lead ● LinuxCOE, mrepo, tellico, rinse, fossology, collectl contributor ● FOSSBazaar and OSL Governance enthusiast ● Mandriva, Mageia, Fedora packager 2 HP / OSSI / Bruno Cornec
  3. 3. What is the problem ? Users / Sysadmins environment: >= 90% < 10%Users: easyness (GUI, CLI)Admins: distribution compliance, < 1%smooth integration – reduce admin load 3 HP / OSSI / Bruno Cornec
  4. 4. What is the problem ? Developers environment: Sourceforge Flat Files ftp, web Project CMS/VCS autotools ? CVS Install proc ? Soft. Eng. ? SVN Fear of distro ? GIT, ...4 HP / OSSI / Bruno Cornec
  5. 5. What is the problem ? Packagers environment: Distro ftp, web Flat Files Package CMS/VCSDoes s/he only exists ? CVSOn time ? Lag behind ? rpmlint ?Distro coverage (ver, #) SVN lintian ?Beta avail ? GIT, ... 5 HP / OSSI / Bruno Cornec
  6. 6. My answer with my various hats Im a sysadmin so I want to deal with packages only (deployment server integration) Im a developer so I want to ease my tests without screwing up my system and testing as a std user/admin Im a software engineer and want no duplication of code nor information (metadata) Im a a packager so I want to streamline that work for new projects Im interested in packaging Im the marketing department of my own project and use packages as a “competitive advantage” Founding no tool existing to help me doing all that (2005), so I started writing one to scratch my own itch6 HP / OSSI / Bruno Cornec
  7. 7. What is the solution ? Packaging should be a project concern as well as coding, testing, installing, .... especially for smaller projects New mantra: “Package early, package always” Use packaging as your only way of delivery (not a dream) Overhead minimal (not much longer than providing a tgz), maximum benefit (consistancy, reproduceability, team work improvement, distribution integration, improved deployment) Packaging is also a marketing activity based on a technical content. Its an easy way to extend your user base, and improve your community. Makes tests easy. THE SOLUTION IS CONTINUOUS PACKAGING7 HP / OSSI / Bruno Cornec
  8. 8. Cloud based Continuous Packaging Local Build Server (may host VMs and/or VEs)Packagers VM or VE Build Repository Build + Server metadata Local build Project RM Build Farm (may host VMs) Remote buildDevelopers 9 HP / OSSI / Bruno Cornec
  9. 9. Provisioning the Continuous Packaging Cloud Local Build Server (may host VMs and/or VEs) VM or VE Build Build + metadata Local build Repository Server Project RM Build Farm (may host VMs) Remote build10 HP / OSSI / Bruno Cornec
  10. 10. Goals Project-builder main goal is to help you package continuously being agnostic: CMS agnostic: no CMS but guys its 21st century now, SVN, CVS, Mercurial, GIT, SVK.... OS agnostic: Linux: RPM, deb, ebuild, slack based, ... 120+ distro tuples at the moment – repositories for yum, urpmi, apt. Solaris pkg. HP-UX sd in progress Build environment agnostic: local, VM (QEMU, KVM...), VE (rpmbootstrap, rinse, mock, debootstrap...), RM (external build farm) No project impact: preserves the md5sum of the delivered upstream sources. Can be completely external to the upstream project.11 HP / OSSI / Bruno Cornec
  11. 11. Goodies Project-builder provides additional goodies: Easy VMs/VEs/RMs management Macro system with perl variables to avoid duplication Skeleton generation to help starting Manages delivery up to your repository (ssh based) Manages announces on mailing lists Integration of tests in the process Manages patches/additional sources when not upstream Checks validity of packages built (lintian, rpmlint) Easy creation of new versions for upstream management13 HP / OSSI / Bruno Cornec
  12. 12. VM, VE & RM Management VE Creation: debootstrap for .deb based ones. rpmbootstrap (modeled after debootstrap): create chroot for multiple RPM based distributions (Mageia, Fedora, Mandriva, CentOS, OpenSuSE) VM|RM Creation: KVM or QEMU iso install possibilities. LinuxCOE/OpenStack as provisioning tool for your VM|RM VM|VE|RM setup: adds pb tools and build account VM|VE Snapshot feature Building in VM|VE|RM == building locally (build2vm|ve|rm vs build2pkg). ssh communication for VM/RM14 HP / OSSI / Bruno Cornec
  13. 13. Metadata Management Packaging process independent of the project and external Metadata isolated - No duplication15 HP / OSSI / Bruno Cornec
  14. 14. Configuration file usage Macro example:./rpmbootstrap/pbfilter/all.pbf:filter PBSUMMARY = rpmbootstrap is a tool similarto debootstrap for RPM based distributions./rpmbootstrap/rpm/rpmbootstrap.spec:Summary: PBSUMMARY./rpmbootstrap/deb/control:Description: PBSUMMARY./rpmbootstrap/pkg/pkginfo:NAME="PBSUMMARY" Instantiation possible from distro-ver-arch, distro-ver, distro, distro- family, distro-type, os Similar approach for all other parameters (Cf: man pb.conf)16 HP / OSSI / Bruno Cornec
  15. 15. Commands[cms|sbx]2build: Create tar files for the project under your CMS. Parameters arepackages to build if not using default listbuild2pkg: Create packages for your running distributionbuild2ssh: Send the tar files to a SSH hostpkg2ssh: Send the packages built to a SSH hostbuild2vm|ve|rm: Create packages in VMs/VEs/RMs, launching them if needed andsend those packages to a SSH host once builtlaunchvm: Launch one virtual machinescript2vm|ve|rm: Launch a VM/VE/RM if needed and executes a script on ittest2vm|ve|rm: Test inside a VM/VE/RMnewvm|ve: Create a new VM/VE/RMsetupvm|ve|rm: Setup a VM/VE/RM for pb usagenewver: Create a new version of the project derived from the current onenewproj: Create a new project and a template set of configuration files underpbconf17 HP / OSSI / Bruno Cornec
  16. 16. The Big picturePackagers newver / newproj newvm|ve / setupvm|ve|rm Packaging build2vm|ve|rm cms|sbx2build Project Building Packages build2pkg pkg2ssh build2ssh Developers 18 HP / OSSI / Bruno Cornec
  17. 17. Multi platform life cycleUser Requirements Validation Functional Integration tests Specification Design Unit tests Specification 19 Coding HP / OSSI / Bruno Cornec
  18. 18. HP+Intel Solution Brief21 HP / OSSI / Bruno Cornec
  19. 19. Multi platform packaging cycle VMs/VEs/RMsDevelopers UUWL Packages VM/VE/RM 23 Packagers HP / OSSI / Bruno Cornec
  20. 20. Q & A up to now ?24 HP / OSSI / Bruno Cornec
  21. 21. DEMONSTRATION25 HP / OSSI / Bruno Cornec
  22. 22. To be done LSB chroot support. (or not) LXC support Support for libvirt, virsh, openstack API Other CMS (Bazaar, …) only when/if needed Other VMs (VMWare, Xen, ...) only when/if needed Multiple delivery means Look at interactions with Buildbot Add non-interactive signature support for .deb Config-Model for configuration file management REST API and Web Interface26 HP / OSSI / Bruno Cornec
  23. 23. Learn Start with the Lab (63 pages) Use man (pb, pb.conf + 8 ProjectBuilder::* man pages) Use the mailing-list pb-announce and pb-devel at Use examples from HP / OSSI / Bruno Cornec
  24. 24. Web ResourcesProject-Builder Web site / Trac / Wiki:●● http://trac.project-builder.orgProjects using ● (of course :-) ● ● Http:// ● ● 28 HP / OSSI / Bruno Cornec
  25. 25. Related toolsProject-Builder is mostly suited for upstream projects wanting to package theirapplicationsDistributions provide each their build tools ● SuSE: Open Build Service (Multi distro, Web based, BaaS) ● Fedora (Koji) ● Mandriva/Mageia (Youri, mdvsys, mgarepo...)Other complementary tools: ● Buildbot ● KVM/QEMU ● rpmbootstrap, rinse, mock ● Parallel::ForkManager, DBI, DB::SQLite, File::MimeInfo, is one of the tool of the initiative. 29 HP / OSSI / Bruno Cornec
  26. 26. Contact (Linux Solution Consultant in the HP/Intel Solution Center) ”Changes are never easy to make. There is comfort and safety in tradition, but change must come, no matter how painful or expensive it may be.” Bill Hewlett Linus Torvalds, Richard Stallman, Eric Raymond, Nat Makarevitch, René Cougnenc, Eric Dumas, Rémy Card, Bdale Thanks Garbee, Bryan Gartner, Gallig Renaud, Joachim Langenbach among others, for their work and devotion to the Open Source Software cause... and my family Open Source at HP for his patience :-) http://opensource.hp.com30 HP / OSSI / Bruno Cornec