LCA 2014 project-builder.org presentation
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

LCA 2014 project-builder.org presentation

  • 653 views
Uploaded on

Presentation of the GPL Continuous Packaging tool helping producing packages mostly for upstream Open Source projects made during the Linux.conf.au 2014 Continuous Integration Mini-Conf in Perth.

Presentation of the GPL Continuous Packaging tool helping producing packages mostly for upstream Open Source projects made during the Linux.conf.au 2014 Continuous Integration Mini-Conf in Perth.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
653
On Slideshare
651
From Embeds
2
Number of Embeds
1

Actions

Shares
Downloads
2
Comments
0
Likes
0

Embeds 2

https://www.linkedin.com 2

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Project Builder: A GPL continuous packaging solution 1 Bruno Cornec V2.1 6th of January 2014 © Copyright 2012 Hewlett-Packard Development Company, L.P. - Licensed under CC-by-SA 3.0 HP EMEA Open Source Profession Lead
  • 2. Introducing Myself Software engineering and Unices since 1988 ● – Mostly Configuration Management Systems, Build systems, quality tools, on multiple commercial Unix systems – Discover Open Source & Linux (OSL) & first contributions in 1993 – Full time on OSL since 1995, first as HP reseller then @HP Currently: ● – Master Technology Architect on OSL for the HP/Intel Solution Center, Grenoble – HP OSL Advocate and Converged Infrastructure Ambassador – HP EMEA OSL Profession Lead – Solutions Linux Conference and AFUL board member. Conferences at WW level in LinuxCon, Linux.conf.au – MondoRescue, Project-Builder.org, UUWL and PUSK Project Lead – LinuxCOE, mrepo, tellico, rinse, fossology, collectl contributor – FOSSBazaar and OSL Governance enthusiast – Mandriva, Mageia, Fedora packager 2 ● And also: – Amateur singer (Alto / Tenor) and recorder player since 1976 and Choir director since 1987 – CD collector since 1981 (5000+ and counting) – Concert attendance since 1976 – Amateur photograph since 1976 © Copyright 2012 Hewlett-Packard Development Company, L.P. - Licensed under CC-by-SA 3.0
  • 3. What is project-builder.org’s goal ? Make upstream projects life easier with regards to packaging their software 3 © Copyright 2012 Hewlett-Packard Development Company, L.P. - Licensed under CC-by-SA 3.0
  • 4. What is project-builder.org’s itch ? Make *my* life easier with regards to packaging *my* software 4 © Copyright 2012 Hewlett-Packard Development Company, L.P. - Licensed under CC-by-SA 3.0
  • 5. Users / Sysadmins want packaged software ● Ease of use: GUI, CLI >= 90% ●Distribution compliance ●Smooth Integration with deployment tools U 5 < 10% < 1% © Copyright 2012 Hewlett-Packard Development Company, L.P. - Licensed under CC-by-SA 3.0
  • 6. Users / Sysadmins want packaged software ● Ease of use: GUI, CLI >= 90% ●Distribution compliance ●Smooth Integration with deployment tools U < 10% < 1% 6 D © Copyright 2012 Hewlett-Packard Development Company, L.P. - Licensed under CC-by-SA 3.0 Lag between SW and Pkg availability ●Test of alpha/beta/VCS SW ●Too many: ● distributions, ● versions, ● package formats, ● tools to generate packages ● tools to manage repositories ●
  • 7. Upstream benefits from Continuous packaging Packaging should be a project concern as well as coding, testing, installing, .... especially for smaller projects Use packaging as your only way of delivery (not a dream) Minial overhead, slightly longer than providing a tgz, maximum benefit: Consistancy and reproduceability for devs and users Distribution & deployment server integration, Improved deployment without risk of screwing up the system 7 Packaging is also a marketing activity for the upstream project based on a technical content. It's an easy way to extend your user base, and improve your community relationship and is a “competitive advantage”. New mantra: “Package early, package always” THE SOLUTION IS INDEED CONTINUOUS PACKAGING (whatever the tool) © Copyright 2012 Hewlett-Packard Development Company, L.P. - Licensed under CC-by-SA 3.0
  • 8. Continuous Packaging Architecture Local Build Server (may host VMs and/or VEs) Packagers VM or VE Build Repository Server Build + metadata Local build 8 Project RM Build Farm Developers (may host VMs) © Copyright 2012 Hewlett-Packard Development Company, L.P. - Licensed under CC-by-SA 3.0 Remote build
  • 9. Provisioning the Continuous Packaging Cloud Local Build Server (may host VMs and/or VEs) VM or VE Build Build + 9 metadata Local build Project RM Build Farm (may host VMs) Remote build © Copyright 2012 Hewlett-Packard Development Company, L.P. - Licensed under CC-by-SA 3.0 Repository Server
  • 10. Goals Project-builder main goal is to help you package continuously being agnostic: VCS agnostic: no VCS but guys it's 21st century now, SVN, CVS, Mercurial, GIT and GIT/SVN, SVK.... OS agnostic: Linux: RPM, deb, ebuild, slack based, ... 150+ distro tuples made and counting – repositories for yum, urpmi, apt. Solaris pkg. HP-UX sd in roadmap 10 Build environment agnostic: local, VM (QEMU, KVM...), VE (rpmbootstrap, rinse, mock, debootstrap...), RM (external build farm) No project impact: preserves the md5sum of the delivered upstream sources. Can be completely external to the upstream project. Avoids duplication of code and metadata THE SOLUTION IS INDEED CONTINUOUS PACKAGING (with projectbuilder.org !) © Copyright 2012 Hewlett-Packard Development Company, L.P. - Licensed under CC-by-SA 3.0
  • 11. Goodies Project-builder provides additional goodies: Easy VMs/VEs/RMs installation/setup. More on that later Macro system with perl variables to avoid duplication Skeleton generation to help starting Manages package delivery up to your repository (ssh based) with repository management (yum, apt, urpmi) Manages announces on mailing lists 11 Integration of tests in the process Manages patches/additional sources when not upstream Checks validity of packages built (lintian, rpmlint) Easy creation of new versions for upstream management Manages website delivery © Copyright 2012 Hewlett-Packard Development Company, L.P. - Licensed under CC-by-SA 3.0
  • 12. VM, VE & RM Management VE Creation: debootstrap for .deb based ones. rpmbootstrap (modeled after debootstrap): create chroot for multiple RPM based distributions (Mageia, Fedora, Mandriva, CentOS, OpenSuSE) VM|RM Creation: KVM or QEMU iso install possibilities. 12 LinuxCOE/OpenStack as provisioning tool for your VM|RM VM|VE|RM setup: adds pb tools and build account VM|VE Snapshot feature – Update of VM|VE|M optional Building in VM|VE|RM == building locally (build2vm|ve|rm vs build2pkg). ssh communication for VM/RM © Copyright 2012 Hewlett-Packard Development Company, L.P. - Licensed under CC-by-SA 3.0
  • 13. Metadata Management Packaging process independent of the project and external Metadata isolated - No duplication 13 © Copyright 2012 Hewlett-Packard Development Company, L.P. - Licensed under CC-by-SA 3.0
  • 14. Configuration file usage Macro example: ./rpmbootstrap/pbfilter/all.pbf:filter PBSUMMARY = rpmbootstrap is a tool similar to debootstrap for RPM based distributions ./rpmbootstrap/rpm/rpmbootstrap.spec:Summary: PBSUMMARY ./rpmbootstrap/deb/control:Description: PBSUMMARY ./rpmbootstrap/pkg/pkginfo:NAME="PBSUMMARY" 14 Instantiation possible from distro-ver-arch, distro-ver, distro, distro-family, distro-type, os Similar approach for all other parameters (Cf: man pb.conf) © Copyright 2012 Hewlett-Packard Development Company, L.P. - Licensed under CC-by-SA 3.0
  • 15. Commands [cms|sbx]2build: Create tar files for the project under your CMS. Parameters are packages to build if not using default list build2pkg: Create packages for your running distribution build2ssh: Send the tar files to a SSH host pkg2ssh: Send the packages built to a SSH host build2vm|ve|rm: Create packages in VMs/VEs/RMs, launching them if needed and send those packages to a SSH host once built launchvm: Launch one virtual machine script2vm|ve|rm: Launch a VM/VE/RM if needed and executes a script on it test2vm|ve|rm: Test inside a VM/VE/RM newvm|ve: Create a new VM/VE/RM setupvm|ve|rm: Setup a VM/VE/RM for pb usage newver: Create a new version of the project derived from the current one newproj: Create a new project and a template set of configuration files under pbconf 15 © Copyright 2012 Hewlett-Packard Development Company, L.P. - Licensed under CC-by-SA 3.0
  • 16. The Big picture Packagers newver / newproj Packaging newvm|ve / setupvm|ve|rm build2vm|ve|rm cms|sbx2build Project 16 Developers © Copyright 2012 Hewlett-Packard Development Company, L.P. - Licensed under CC-by-SA 3.0 Building build2pkg build2ssh Packages pkg2ssh
  • 17. Multi platform packaging cycle VMs/VEs/RMs Developers UUWL Packages 17 VM/VE/RM Packagers © Copyright 2012 Hewlett-Packard Development Company, L.P. - Licensed under CC-by-SA 3.0
  • 18. Q & A up to now ? 18 © Copyright 2012 Hewlett-Packard Development Company, L.P. - Licensed under CC-by-SA 3.0
  • 19. DEMONSTRATION 19 © Copyright 2012 Hewlett-Packard Development Company, L.P. - Licensed under CC-by-SA 3.0
  • 20. Special announce for LCA 2014 ! 20 © Copyright 2012 Hewlett-Packard Development Company, L.P. - Licensed under CC-by-SA 3.0
  • 21. More Special announce for LCA 2014 ! I hate time release delivery Of course, 0.12.3 has bugs ;-) Lack of time to test thoroughly So I hate time release delivery So expect a 0.12.4 later this week, when more tests have been done ! So back to feature set release in fact 21 Did I say I hate time release delivery ? © Copyright 2012 Hewlett-Packard Development Company, L.P. - Licensed under CC-by-SA 3.0
  • 22. To be done LSB chroot support. (or not) LXC support (which one ?) Support for libvirt, virsh, openstack API Other CMS (Bazaar, …) only when/if needed Other VMs (VMWare, Xen, ...) only when/if needed Multiple delivery means 22 Look at interactions with Buildbot Add non-interactive signature support for .deb Config-Model for configuration file management REST API and Web Interface would be nice. Any contributor ? © Copyright 2012 Hewlett-Packard Development Company, L.P. - Licensed under CC-by-SA 3.0
  • 23. Learn Project-Builder.org Start with the Lab (63 pages) Use man (pb, pb.conf + 8 ProjectBuilder::* man pages) Use the mailing-list pb-announce and pb-devel at http://www.mondorescue.org/sympa Use examples from 23 http://trac.project-builder.org/browser/projects/ © Copyright 2012 Hewlett-Packard Development Company, L.P. - Licensed under CC-by-SA 3.0
  • 24. Web Resources Project-Builder Web site / Trac / Wiki: ● http://www.project-builder.org ● http://trac.project-builder.org Projects using project-builder.org: ● http://www.project-builder.org ( of course :-) ● http://www.mondorescue.org 24 ● Http://www.linuxcoe.org ● http://www.fossology.org © Copyright 2012 Hewlett-Packard Development Company, L.P. - Licensed under CC-by-SA 3.0
  • 25. Related tools Project-Builder is mostly suited for upstream projects wanting to package their applications Distributions provide each their build tools ● SuSE: Open Build Service (Multi distro, Web based, BaaS) ● Fedora (Koji) ● Mandriva/Mageia (Youri, mdvsys, mgarepo...) Other complementary tools: ● Buildbot ● KVM/QEMU ● rpmbootstrap, rinse, mock ● Parallel::ForkManager, DBI, DB::SQLite, File::MimeInfo, Mail::Sendmail 25 Project-Builder.org is one of the tool of the vcs-pkg.org initiative. © Copyright 2012 Hewlett-Packard Development Company, L.P. - Licensed under CC-by-SA 3.0
  • 26. Contact - Thanks Bruno.Cornec@hp.com (Open Source and Linux Technology Architect at the HP/Intel Solution Center) http://www.hp.com/linux http://opensource.hp.com Thanks goes to: 26 Linus Torvalds, Richard Stallman, Eric Raymond, Nat Makarevitch, René Cougnenc, Eric Dumas, Rémy Card, Bdale Garbee, Bryan Gartner, Craig Lamparter, Lee Mayes, Gallig Renaud, Andree Leidenfrost, Phil Robb, Bob Gobeille, Martin Michlmayr among others, for their work and devotion to the Open Source Software cause... and my family for their patience :-) © Copyright 2012 Hewlett-Packard Development Company, L.P. - Licensed under CC-by-SA 3.0 ”Changes are never easy to make. There is comfort and safety in tradition, but change must come, no matter how painful or expensive it may be.” Bill Hewlett