The Gory Details of Debian packages


Published on

This is the presentation I gave at FSCons in 2008

Published in: Technology

The Gory Details of Debian packages

  1. 1. Debian Packages: the gory details <ul><ul><li>An operetta, of sorts. </li></ul></ul>
  2. 2. Debian GNU/Linux <ul><li>Free, community developed operating system </li></ul><ul><li>Always will be free because of its Social Contract and legally shielded by Software in the Public Interest </li></ul><ul><li>Developed by a group of about 1,000 volunteers, who receive no financial compensation </li></ul><ul><li>Stable </li></ul><ul><ul><li>More info in debian-history.deb </li></ul></ul>
  3. 3. Development reference <ul><li>Tens of thousands of packages (~25,000) </li></ul><ul><li>Security, package and bug tracking </li></ul><ul><li>Gigantic infrastructure </li></ul><ul><li>Serves as sort of a reference platform due to its adherence to Linux Standards Base, File Hierarchy Standard, etc. </li></ul>
  4. 4. Different versions <ul><li>Names taken from Toy Story </li></ul><ul><li>Bruce Perens worked for Pixar and was Debian Project Leader at the time </li></ul><ul><li>Current version: etch </li></ul><ul><li>Next version: lenny (after that its squeeze) </li></ul><ul><li>Scheduled for release in 2008, but could be 2009 (currently ~100 RC bugs.) </li></ul>
  5. 5. What about sid? <ul><li>The kid in Toy Story who broke toys </li></ul><ul><li>This is the unstable version of debian, and will always be unstable </li></ul><ul><li>Periodically testing gets frozen (like it is right now), release critical bugs get fixed, and then released as stable </li></ul><ul><li>So - </li></ul><ul><ul><li>Stable -> 4.0 & 4.1 code name etch </li></ul></ul><ul><ul><li>Testing -> 5.0 code name lenny </li></ul></ul><ul><ul><li>Unstable -> sid (Will never have a version) </li></ul></ul>
  6. 6. How software gets into debian <ul><li>Debian uses “packages” to create a modular system </li></ul><ul><li>The package itself is a binary format called a “deb” </li></ul><ul><li>This means it has to be built (compiled) on a packager's machine before it is uploaded to debian </li></ul><ul><li>A developer finds some software, not already in debian, and files an ITP (Intend To Package) bug, then closes the bug with a new deb </li></ul>
  7. 7. Mentoring / Sponsorship <ul><li>If you have a package that you want to get into debian you can submit it to </li></ul><ul><li>Lots of helpful and experienced debian people to get your package in good shape for debian </li></ul>
  8. 8. First stop, unstable <ul><li>A package gets built from source code </li></ul><ul><li>The packager uploads the resulting binary deb and the pristine source code into unstable (This is a bit of a gloss) </li></ul><ul><li>It sits in unstable for 10 days then moves to testing the package gets built on a variety of architectures </li></ul><ul><li>Once in testing it gets frozen and tested again before release for “release critical” bugs </li></ul>
  9. 9. How do you create a deb? <ul><li>TIMTOWTDI, but debian has excellent tools: </li></ul><ul><ul><li>BTS (Bug Tracking System) </li></ul></ul><ul><ul><li>PTS (Package Tracking System) </li></ul></ul><ul><ul><ul><li>Recently moved to a new server, much more responsive. </li></ul></ul></ul><ul><ul><li>dh-make-perl </li></ul></ul><ul><ul><li>dpkg </li></ul></ul><ul><ul><li>Clean room building with pbuilder or cowbuilder </li></ul></ul>
  10. 10. Closer look at the tools <ul><li>BTS is widely used as a development tool, half-way to a million bugs </li></ul><ul><li>Bugs very often move upstream to perl, ruby, python for example </li></ul><ul><li>Specific types of bugs; start with RFP (Request For Packaging) to get a module into debian </li></ul><ul><li>File an ITP (Intend To Package) bug if you are going to package a module for debian </li></ul><ul><li>Filing packaging bugs is important because it prevents the duplication of work </li></ul>
  11. 11. Package tracking <ul><li>The PTS gives you an overview of every package in debian </li></ul><ul><li>You can look through every debian version (i.e. stable, testing, unstable) </li></ul><ul><li>You can follow bug reports, and package status changes via email from the PTS and on the web </li></ul>
  12. 12. More PTS <ul><li>PTS system has recently been updated </li></ul><ul><li>Hosted in Europe by 1&1 </li></ul><ul><li>Significant traffic: 5.7 Million unique visitors (excluding robots) </li></ul><ul><li>122 Million hits monthly, again excluding robots </li></ul><ul><li>By filing an ITP bug you prevent duplicate work and confusion. </li></ul>
  13. 13. dh_make_perl <ul><li>Powerful perl version of dh_make </li></ul><ul><li>Gobbles up a module from cpan (using the cpan shell) </li></ul><ul><li>Then builds the deb with the --build option </li></ul><ul><li>Lots of options and potential for customization </li></ul><ul><li>Calculates perl dependencies AND operating system dependencies </li></ul>
  14. 14. dh-makin' bacon <ul><li>dh-make-perl will create the files required to build a debian source package out of a perl package. </li></ul><ul><li>This works for most simple packages and is also useful for getting started with packaging perl modules. </li></ul><ul><li>You can build and install the debian package using the --build and --install command line switches. </li></ul><ul><li>Using this program is no excuse for not reading the debian developer documentation, including the Debian policy, the perl policy, the packaging manual and so on. </li></ul>
  15. 15. Flow Source code dh-make Debian directory dpkg Deb
  16. 16. dpkg-buildpackage <ul><li>dpkg is also an extremely powerful build tool. It is a set of tools really </li></ul><ul><li>Responsible for installing and removing packages, gets called by high-level tools like apt-get and aptitude </li></ul><ul><li>dpkg-buildpackage builds your deb as well as the .dsc file you need for uploading to debian </li></ul>
  17. 17. How does dpkg know how to build the package? <ul><li>Because dh-make (alternatively dh-make-perl) built, and populated with boilerplate, the debian directory </li></ul><ul><li>This directory holds files and one executable, the debian/rules script </li></ul><ul><li>debian/rules is a Makefile which calls lots of other small debian scripts called debhelper scripts </li></ul><ul><li>This creates a highly modular build system with lots of opportunity for specific, small changes </li></ul>
  18. 18. Pbuilder and cowbuilder <ul><li>Pbuilder stands for personal builder </li></ul><ul><li>Automatically builds debian packages in a “clean-room” environment </li></ul><ul><li>Helps to verify that a package can be built on the average debian system </li></ul><ul><li>It creates a base chroot image, downloads and installs a complete, but minimal, debian system </li></ul><ul><li>Cowbuilder is pbuilder sped up by using cowdancer, a copy-on-write system in userland </li></ul>
  19. 19. lintian <ul><li>Lintian is the debian policy enforcer </li></ul><ul><li>Chatty, and pretty complete </li></ul><ul><li>Being “lintian clean” is pretty much a requirement for getting your deb into debian and definitely if you want to be a debian developer </li></ul>
  20. 20. What else is in the debian dir? <ul><li>The debian directory also contains files which help to define the package and help in packaging. </li></ul><ul><ul><li>Copyright </li></ul></ul><ul><ul><li>Control </li></ul></ul><ul><ul><li>Changelog </li></ul></ul><ul><ul><li>Watch </li></ul></ul><ul><ul><li>Compat </li></ul></ul>
  21. 21. Changelog <ul><li>This required file lists all the changes </li></ul><ul><li>libyaml-perl (0.39-1) unstable; urgency=low </li></ul><ul><li>* New upstream release. </li></ul><ul><li>* New Maintainer with acknowledgment from Chip. </li></ul><ul><li>* Added the copyright statement to debian/copyright. </li></ul><ul><li>* Updated Standards-Version to 3.6.2. </li></ul><ul><li>* Moved the perl build-dep to build-dep-indep (Closes: #305525). </li></ul><ul><li>* Updated compat version to 4 and upgraded the debhelper dependency </li></ul><ul><li>accordingly. </li></ul><ul><li>-- Florian Ragwitz <> Thu, 1 Dec 2005 15:33:42 +0100 </li></ul>
  22. 22. More changelog <ul><li>The recorded changes are mostly relative to the debian packaging, not the upstream source </li></ul><ul><li>Crucially, this is the canonical location of the package's version information </li></ul><ul><li>Specifies which package fixed which bug (specific syntax allows for automated bug closing from the changelog) </li></ul>
  23. 23. Control <ul><li>Source: libpod-strip-perl </li></ul><ul><li>Section: perl </li></ul><ul><li>Priority: optional </li></ul><ul><li>Build-Depends: debhelper (>= 7), libmodule-build-perl </li></ul><ul><li>Build-Depends-Indep: perl (>= 5.6.10-12) </li></ul><ul><li>Maintainer: Debian Perl Group <> </li></ul><ul><li>Uploaders: Damyan Ivanov <> </li></ul><ul><li>Standards-Version: 3.8.0 </li></ul><ul><li>Homepage: </li></ul><ul><li>Vcs-Svn: svn:// </li></ul><ul><li>Vcs-Browser: </li></ul><ul><li>Package: libpod-strip-perl </li></ul><ul><li>Architecture: all </li></ul><ul><li>Depends: ${perl:Depends}, ${misc:Depends} </li></ul><ul><li>Description: remove POD documentation from Perl code </li></ul><ul><li>Pod::Strip is a subclass of Pod::Simple that removes the POD (plain old </li></ul><ul><li>documentation) from Perl code. The POD may optionally be replaced with </li></ul><ul><li>comments so that line numbers of the code stay the same. </li></ul><ul><li>. </li></ul><ul><li>Pod::Strip is useful in Perl code parsers that don't want to bother about POD. </li></ul>
  24. 24. Copyright <ul><li>Please, please, please, included copyright and license information in your modules on CPAN. </li></ul><ul><li>It may not matter that much to you, it matters a lot to those who package your software for linux (at least for the Free Software distros) </li></ul><ul><li>It may matter a lot to you one day </li></ul><ul><li>It strengthens the license to have a copyright and it appears clear that the licenses DO have legal weight (the artistic license just won a big legal victory recently.) </li></ul>
  25. 25. Debian's copyright file <ul><li>Format-Specification: </li></ul><ul><li>Upstream-Maintainer: Thomas Klausner <> </li></ul><ul><li>Upstream-Source: </li></ul><ul><li>Upstream-Name: Pod-Strip </li></ul><ul><li>Files: * </li></ul><ul><li>Copyright: Copyright 2004, 2005, 2006 Thomas Klausner, All Rights Reserved. </li></ul><ul><li>License-Alias: Perl </li></ul><ul><li>License: Artistic | GPL-1+ </li></ul><ul><li>This program is free software; you can redistribute it and/or modify it under </li></ul><ul><li>the same terms as Perl itself. </li></ul>
  26. 26. More copyright fun <ul><li>Files: debian/* </li></ul><ul><li>Copyright: © 2008, Damyan Ivanov <> </li></ul><ul><li>License: Artistic | GPL-1+ </li></ul><ul><li>Packaging is free; you can redistribute it and/or modify it inder the same terms as Perl itself. </li></ul>
  27. 27. Licensing <ul><li>License: Artistic </li></ul><ul><li>This program is free software; you can redistribute it and/or modify </li></ul><ul><li>it under the terms of the Artistic License, which comes with Perl. </li></ul><ul><li>On Debian GNU/Linux systems, the complete text of the Artistic License </li></ul><ul><li>can be found in /usr/share/common-licenses/Artistic </li></ul><ul><li>License: GPL-1+ </li></ul><ul><li>This program is free software; you can redistribute it and/or modify </li></ul><ul><li>it under the terms of the GNU General Public License as published by </li></ul><ul><li>the Free Software Foundation; either version 1, or (at your option) </li></ul><ul><li>any later version. </li></ul><ul><li>On Debian GNU/Linux systems, the complete text of the GNU General </li></ul><ul><li>Public License can be found in `/usr/share/common-licenses/GPL' </li></ul>
  28. 28. Watch file <ul><li># format version number, currently 3; this line is compulsory! </li></ul><ul><li>version=3 </li></ul><ul><li># URL to the package page followed by a regex to search </li></ul><ul><li> .*/Pod-Strip-v?(d[d_.-]+).(?:tar(?:.gz|.bz2)?|tgz|zip)$ </li></ul><ul><li>Allows for an automated update of source and package when there is a new version on CPAN, via the uscan tool </li></ul>
  29. 29. Uscan <ul><li>This allows us to automatically update the package contents without having to rebuild the entire package scaffolding. $ uscan libspreadsheet-read-perl: Newer version (0.29) available on remote site: (local version is 0.28) libspreadsheet-read-perl: Successfully downloaded updated package Spreadsheet-Read-0.29.tgz and symlinked libspreadsheet-read-perl_0.29.orig.tar.gz to it </li></ul>
  30. 30. How do we know there is a new version on CPAN? <ul><li>Our QA reporting tool: </li></ul><ul><li> </li></ul><ul><li>Part of the debian-perl team's workflow </li></ul><ul><li>Some of this functionality is moving towards CPANTS, maybe other automated debian-centric tests suites perhaps? </li></ul>
  31. 31. Managing packages <ul><li>sudo aptitude install courier-filter-perl </li></ul><ul><li>Reading package lists... Done </li></ul><ul><li>Building dependency tree </li></ul><ul><li>Reading state information... Done </li></ul><ul><li>Reading extended state information </li></ul><ul><li>Initializing package states... Done </li></ul><ul><li>Reading task descriptions... Done </li></ul><ul><li>The following NEW packages will be installed: </li></ul><ul><li>courier-authdaemon{a} courier-authlib{a} courier-authlib-userdb{a} courier-base{a} courier-filter-perl courier-mta{a} expect{a} fam{a} libarchive-zip-perl{a} libclamav-client-perl{a} libfam0{a} libltdl3{a} libmail-spf-perl{a} libnet-rblclient-perl{a} libnetaddr-ip-perl{a} portmap{a} tcl8.4{a} </li></ul><ul><li>The following packages will be REMOVED: </li></ul><ul><li>postfix{a} </li></ul><ul><li>0 packages upgraded, 17 newly installed, 1 to remove and 0 not upgraded. </li></ul><ul><li>Need to get 4012kB of archives. After unpacking 8303kB will be used. </li></ul><ul><li>Do you want to continue? [Y/n/?] </li></ul>
  32. 32. Implications of automated packaging systems <ul><li>Obsoletes debian-perl(?) </li></ul><ul><li>Most automated tools currently come with some form of caveat stating that there is no support, i.e. Security tracking. </li></ul><ul><li>Most likely quite useful if you are SURE no one else will ever use your deb </li></ul><ul><li>Creation of a potentially vast invisible, distributed package repository </li></ul><ul><li>Probably inevitable :-( </li></ul>
  33. 33. How to become a debian developer <ul><li>Maintain packages and understand debian's Social Contract </li></ul><ul><li>Contribute to debian </li></ul><ul><li>Patience </li></ul><ul><li>Sponsorship </li></ul><ul><li>You don't need to be a DD to contribute, only tangible difference is you get a cool [email_address] and you get to vote </li></ul><ul><li>If you don't become a DD, you don't go MIA </li></ul><ul><li>The entire process is improving and becoming faster – there are changes afoot (DM, DME, etc.) Potentially contreversial. </li></ul>
  34. 34. Resources and Thank yous <ul><li> </li></ul><ul><li>Integrating perl in a wider distribution: The debian-perl group, by Gunnar Wolf </li></ul><ul><li>The Debian System, by Martin Krafft </li></ul><ul><li>dam, gregoa, gwolf, tincho, nomeata, sukria, djpig, dom, Florian R., Gabor, Jos, Nadim, joeyh, eloy, ntyni, rra, hertzog </li></ul>