Debian Quality Assurance      Lucas Nussbaum     lucas@debian.org    lucas@debian.org Debian Quality Assurance   1 / 31
Me     Assistant professor (Maître de conférences) at Univ. de Lorraine         Teach Free Software in a system administra...
DebianWell known and respected for:                       lucas@debian.org Debian Quality Assurance   3 / 31
DebianWell known and respected for:    Its long and bumpy release cycles                       lucas@debian.org Debian Qua...
DebianWell known and respected for:    Its long and bumpy release cycles    Its qualityBut it was not a given:    > 1000 D...
Quality factors                  lucas@debian.org Debian Quality Assurance   4 / 31
Quality factors   Culture: "We release when it’s ready "       Many DDs won’t compromise on this                     lucas...
Quality factors   Culture: "We release when it’s ready "       Many DDs won’t compromise on this   Packages ownership: ind...
Quality factors   Culture: "We release when it’s ready "       Many DDs won’t compromise on this   Packages ownership: ind...
Quality factors   Culture: "We release when it’s ready "       Many DDs won’t compromise on this   Packages ownership: ind...
Debian QA team  Goal: Improve the quality of Debian as a whole  Not really a team (no strict membership), more like a cent...
Debian QA team: Tasks  Run archive-wide checks and mass bug filings  Maintain QA infrastructure  Take care of the dirty are...
Archive-wide checksand Mass bug filings  lucas@debian.org Debian Quality Assurance   7 / 31
Archive-wide checks and MBF  Developers are volunteers:     Focus on interesting things     Manual testing is boring → not...
Archive rebuilds  Rebuild all packages from source  7 hours on 40 build nodes (using Amazon EC2)  Detect packages that Fai...
Debcheck  Check (statically) that dependencies can be satisfied  Not trivial: versioned dependencies, alternate depends, co...
Piuparts  Testing package installation, upgrade and removal  Find bugs in maintainer scripts  (Shell) scripts executed dur...
Lintian   Static analysis of packages   Usually run locally by the developer, before uploading   Archive-wide setup for co...
DEHS - Debian External Health Status  Monitor upstream version of packages  Detect when the Debian package is outdated  HT...
And more . . .   Scan build logs for dangerous warnings   Track security issues   Track popularity of packages   ...      ...
Maintaining infrastructure    lucas@debian.org Debian Quality Assurance   15 / 31
Infrastructure: mitigating the data hell   Debian: aggregation of loosely-connected services       No Launchpad!       Eve...
Package Tracking System (PTS)              Source package-centric view         http://packages.qa.debian.org/package      ...
Developer Packages Overview (DDPO)                Developer-centric view          http://qa.debian.org/developer.php      ...
Ultimate Debian Database (UDD)     Gather everything into a single SQL DB     Perfect tool for data-mining DebianCurrently...
Some UDD queries   Packages with the highest number of different lintian errors or   warningsselect package, count(distinc...
Some UDD queries   Packages with the highest number of different lintian errors or   warningsselect package, count(distinc...
Some UDD queries    Who uploaded wheezy’s packages?select changed_by, count(*) from sources s, upload_history uhwhere s.so...
Some UDD queries    Who uploaded wheezy’s packages?select changed_by, count(*) from sources s, upload_history uhwhere s.so...
Taking care of the dirty areas of Debian           lucas@debian.org Debian Quality Assurance   22 / 31
Taking care of the dirty areas of DebianTwo complementary approaches:   Focus on packages neglected by their maintainers  ...
Neglected packages  Already orphaned:      Do minimalistic maintenance      Problem: > 700 orphaned packages  Not (yet) or...
Bapase: finding neglected packages  Multi-criteria search for neglected packages  Based on Ultimate Debian Database        ...
Inactive maintainers  Missing In Action (MIA) team  Find them, track them, orphan their packages  Mostly based on reportin...
Open challenges:following Debian evolutions     lucas@debian.org Debian Quality Assurance   27 / 31
Co-maintenance                 lucas@debian.org Debian Quality Assurance   28 / 31
Use of VCSes               lucas@debian.org Debian Quality Assurance   29 / 31
Following Debian evolutionsNeed to improve QA infrastructure    Team-aware    VCS-aware       Abstraction layer?        If...
ConclusionsDebian:    A large-scale volunteer-based distribution    A continuous effort to organize development and improv...
Upcoming SlideShare
Loading in...5
×

Distro Recipes 2013 : Debian and quality assurance

2,647

Published on

https://distro-recipes.org

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,647
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
8
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Distro Recipes 2013 : Debian and quality assurance

  1. 1. Debian Quality Assurance Lucas Nussbaum lucas@debian.org lucas@debian.org Debian Quality Assurance 1 / 31
  2. 2. Me Assistant professor (Maître de conférences) at Univ. de Lorraine Teach Free Software in a system administration curriculum (Licence Professionnelle Administration Système, Réseaux et Applications à base de Logiciel Libre) Debian involvement: Collaboration between Debian and Ubuntu Ruby packaging Archive rebuilds Ultimate Debian Database lucas@debian.org Debian Quality Assurance 2 / 31
  3. 3. DebianWell known and respected for: lucas@debian.org Debian Quality Assurance 3 / 31
  4. 4. DebianWell known and respected for: Its long and bumpy release cycles lucas@debian.org Debian Quality Assurance 3 / 31
  5. 5. DebianWell known and respected for: Its long and bumpy release cycles Its qualityBut it was not a given: > 1000 Debian Developers Mostly volunteers, with their own agendas > 2000 Package maintainers 18117 software packages Some very popular packages A lot of niche packages lucas@debian.org Debian Quality Assurance 3 / 31
  6. 6. Quality factors lucas@debian.org Debian Quality Assurance 4 / 31
  7. 7. Quality factors Culture: "We release when it’s ready " Many DDs won’t compromise on this lucas@debian.org Debian Quality Assurance 4 / 31
  8. 8. Quality factors Culture: "We release when it’s ready " Many DDs won’t compromise on this Packages ownership: individuals or small teams Many of the maintainers are experts of their packages lucas@debian.org Debian Quality Assurance 4 / 31
  9. 9. Quality factors Culture: "We release when it’s ready " Many DDs won’t compromise on this Packages ownership: individuals or small teams Many of the maintainers are experts of their packages Release Critical (RC) bugs Prevent packages from being part of a release Everybody is welcomed to fix them Non-Maintainer Uploads lucas@debian.org Debian Quality Assurance 4 / 31
  10. 10. Quality factors Culture: "We release when it’s ready " Many DDs won’t compromise on this Packages ownership: individuals or small teams Many of the maintainers are experts of their packages Release Critical (RC) bugs Prevent packages from being part of a release Everybody is welcomed to fix them Non-Maintainer Uploads Debian Quality Assurance team lucas@debian.org Debian Quality Assurance 4 / 31
  11. 11. Debian QA team Goal: Improve the quality of Debian as a whole Not really a team (no strict membership), more like a central place to discuss and work on QA IRC: #debian-qa@irc.debian.org Mailing list: debian-qa@lists.debian.org lucas@debian.org Debian Quality Assurance 5 / 31
  12. 12. Debian QA team: Tasks Run archive-wide checks and mass bug filings Maintain QA infrastructure Take care of the dirty areas of Debian lucas@debian.org Debian Quality Assurance 6 / 31
  13. 13. Archive-wide checksand Mass bug filings lucas@debian.org Debian Quality Assurance 7 / 31
  14. 14. Archive-wide checks and MBF Developers are volunteers: Focus on interesting things Manual testing is boring → not done Need automated tests Other advantage: treats all packages equally Niche packages are very important to some users lucas@debian.org Debian Quality Assurance 8 / 31
  15. 15. Archive rebuilds Rebuild all packages from source 7 hours on 40 build nodes (using Amazon EC2) Detect packages that Fail To Build From Source (FTBFS) Bugs in packages Toolchain bugs Failures caused by changes in other packages Ran every 2-3 weeks Scripts to analyze failures and file bugs semi-automatically (about 200 bugs filed per hour) Also used for test rebuilds with new compilers, linkers, . . . → Valuable information for upstream lucas@debian.org Debian Quality Assurance 9 / 31
  16. 16. Debcheck Check (statically) that dependencies can be satisfied Not trivial: versioned dependencies, alternate depends, conflicts 2 versions: "Simple one", in PHP http://qa.debian.org/debcheck.php Better one: edos-debcheck (EDOS project) Can also work with RPMs http://edos.debian.net/ lucas@debian.org Debian Quality Assurance 10 / 31
  17. 17. Piuparts Testing package installation, upgrade and removal Find bugs in maintainer scripts (Shell) scripts executed during package installation/removal Developers don’t run it on their systems → Archive-wide setup http://piuparts.debian.org lucas@debian.org Debian Quality Assurance 11 / 31
  18. 18. Lintian Static analysis of packages Usually run locally by the developer, before uploading Archive-wide setup for continuous testing Packages now rejected at upload time for some errors Some example findings: Use of deprecated packaging features Common packaging mistakes Embedded copies of libraries Spelling errors in binaries usefull visiters treshold targetted superseeded dependancies splitted maintainance explicitely equiped informations mispelled http://lintian.debian.org lucas@debian.org Debian Quality Assurance 12 / 31
  19. 19. DEHS - Debian External Health Status Monitor upstream version of packages Detect when the Debian package is outdated HTML scraping + regular expressions Configured in debian/watch lucas@debian.org Debian Quality Assurance 13 / 31
  20. 20. And more . . . Scan build logs for dangerous warnings Track security issues Track popularity of packages ... lucas@debian.org Debian Quality Assurance 14 / 31
  21. 21. Maintaining infrastructure lucas@debian.org Debian Quality Assurance 15 / 31
  22. 22. Infrastructure: mitigating the data hell Debian: aggregation of loosely-connected services No Launchpad! Everyone can start their own service Data everywhere A dozen of places to get interesting data Tools required to gather data in a central place and expose it to the maintainers lucas@debian.org Debian Quality Assurance 16 / 31
  23. 23. Package Tracking System (PTS) Source package-centric view http://packages.qa.debian.org/package lucas@debian.org Debian Quality Assurance 17 / 31
  24. 24. Developer Packages Overview (DDPO) Developer-centric view http://qa.debian.org/developer.php lucas@debian.org Debian Quality Assurance 18 / 31
  25. 25. Ultimate Debian Database (UDD) Gather everything into a single SQL DB Perfect tool for data-mining DebianCurrently imported in UDD:Debian Sources and Packages, bugs, popcon, testing migrations, upload history,orphaned packages, carnivore, lintian, debtags, translations, NEW queue,screenshots, DEHS, ldap, wanna-build, removals, Ubuntu Sources and Packages,Ubuntu bugs, Ubuntu popconPossible questions you can answer with UDD: RC buggy packages in testing, sorted by popcon? Packages for which the last 4 uploads were NMUs? Packages not maintained by official DDs? http://udd.debian.org/ lucas@debian.org Debian Quality Assurance 19 / 31
  26. 26. Some UDD queries Packages with the highest number of different lintian errors or warningsselect package, count(distinct tag) as cntfrom lintianwhere tag_type in(’error’,’warning’)group by packageorder by cnt desc limit 15; lucas@debian.org Debian Quality Assurance 20 / 31
  27. 27. Some UDD queries Packages with the highest number of different lintian errors or warningsselect package, count(distinct tag) as cntfrom lintianwhere tag_type in(’error’,’warning’)group by packageorder by cnt desc limit 15; package count tsocks 18 pygopherd 18 cyclades-serial-client 17 muddleftpd 17 elib 17 lucas@debian.org Debian Quality Assurance 20 / 31
  28. 28. Some UDD queries Who uploaded wheezy’s packages?select changed_by, count(*) from sources s, upload_history uhwhere s.source = uh.source and s.version = uh.versionand s.distribution=’debian’ and s.release = ’wheezy’group by changed_by order by count desc limit 8; lucas@debian.org Debian Quality Assurance 21 / 31
  29. 29. Some UDD queries Who uploaded wheezy’s packages?select changed_by, count(*) from sources s, upload_history uhwhere s.source = uh.source and s.version = uh.versionand s.distribution=’debian’ and s.release = ’wheezy’group by changed_by order by count desc limit 8; changed_by count gregor herrmann <gregoa@debian.org> 790 Joachim Breitner <nomeata@debian.org> 237 Christian Perrier <bubulle@debian.org> 229 Alessio Treglia <alessio@debian.org> 222 Salvatore Bonaccorso <carnil@debian.org> 207 Clint Adams <clint@debian.org> 206 Jonas Smedegaard <dr@jones.dk> 195 lucas@debian.org Debian Quality Assurance 21 / 31
  30. 30. Taking care of the dirty areas of Debian lucas@debian.org Debian Quality Assurance 22 / 31
  31. 31. Taking care of the dirty areas of DebianTwo complementary approaches: Focus on packages neglected by their maintainers Focus on maintainers neglecting their packages lucas@debian.org Debian Quality Assurance 23 / 31
  32. 32. Neglected packages Already orphaned: Do minimalistic maintenance Problem: > 700 orphaned packages Not (yet) orphaned: Find them, orphan or remove them Goal: Put maintenance in the hands of people with time lucas@debian.org Debian Quality Assurance 24 / 31
  33. 33. Bapase: finding neglected packages Multi-criteria search for neglected packages Based on Ultimate Debian Database http://udd.debian.org/bapase.cgi lucas@debian.org Debian Quality Assurance 25 / 31
  34. 34. Inactive maintainers Missing In Action (MIA) team Find them, track them, orphan their packages Mostly based on reporting by users or fellow developers Echelon: records activity of DDs ; Carnivore: tracks identities Collaboration with the Account Managers: possible removal of their account (Unused accounts → possible security problems) lucas@debian.org Debian Quality Assurance 26 / 31
  35. 35. Open challenges:following Debian evolutions lucas@debian.org Debian Quality Assurance 27 / 31
  36. 36. Co-maintenance lucas@debian.org Debian Quality Assurance 28 / 31
  37. 37. Use of VCSes lucas@debian.org Debian Quality Assurance 29 / 31
  38. 38. Following Debian evolutionsNeed to improve QA infrastructure Team-aware VCS-aware Abstraction layer? If I’ve learned anything, it’s that OSS projects can’t make a choice. And when you can’t make a choice, what do you do? You support all options! Yay! How do we support all options? An abstraction layer! Yay! Abstraction layers are fun to write! Yay! – – Linux Hater’s Blog Standard workflows? Keep track of sponsorship requests http://mentors.debian.org + sponsorship-requests Track state of packages in VCS Partial solution: PET (http://pet.debian.net/) lucas@debian.org Debian Quality Assurance 30 / 31
  39. 39. ConclusionsDebian: A large-scale volunteer-based distribution A continuous effort to organize development and improve quality lucas@debian.org Debian Quality Assurance 31 / 31
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×