Leveraging Solver Preferences to Tame your Package Manager

624 views

Published on

In this presentation, you will learn how to use solver preferences and external solvers to guide your package managers and find the best installation or upgrade for you.
With work from Mancoosi, Irill, and OCamlPro

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
624
On SlideShare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
7
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Leveraging Solver Preferences to Tame your Package Manager

  1. 1. Leveraging solver preferences to tame your package managers Roberto Di Cosmo http://www.dicosmo.org Universit´e Paris Diderot and INRIA, Irill http://www.irill.org July 8th, 2014 OUPS Paris Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 1 / 28
  2. 2. Package managers are all around us Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 2 / 28
  3. 3. Package managers : the basics Definition of Package Manager (Wikipedia) ... tools to automate ... installing, upgrading, configuring, and removing software packages ... in a consistent manner. It typically maintains a database of software dependencies and version information ... Some package managers Binary distributions apt, aptitude, yum,... Source distributions portage, *BSD ports, homebrew, opam... Language specific PyPI, Eclipse P2, (opam), ... Application specific steam, ... Functional approach nixOS, disnixOS,... ... you name it Do they perform well? Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 3 / 28
  4. 4. Installations may really go wrong Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 4 / 28
  5. 5. Installation woes: no solution # sudo apt-get install baobab Reading package lists... Done Building dependency tree Reading state information... Done Some packages could not be installed. This may mean that you have requested an impossible situation or if you are using the unstable distribution that some required packages have not yet been created or been moved out of Incoming. The following information may help to resolve the situation: The following packages have unmet dependencies: gnome-settings-daemon: Breaks: gnome-screensaver (< 2.28.0) but 2.26.1-1 is to be installed E: Broken packages (whereas a solution did exist) Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 5 / 28
  6. 6. Installation woes: solver timeout https://github.com/ocaml/opam/issues/685 (whereas a solution could be found in less than 1/10th of a second) Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 6 / 28
  7. 7. Managing packages is hard Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 7 / 28
  8. 8. How hard are package installation problems? Theorem The following problems are NP-complete: installability of a single package co-installability of a set of packages Proof idea Bi-directional mapping between dependency resolution and boolean satisfiability. Details: Di Cosmo, Leroy, Treinen, Vouillon et al. Managing the complexity of large free and open source package-based software distributions. ASE 2006: Automated Software Engineering. Alternative proof of NP-hardness (Daniel Burrows, 2008) Encode Sudoku solving as a package installation problem, left as exercise for the public. Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 8 / 28
  9. 9. Finding a needle in a haystack Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 9 / 28
  10. 10. An exponential number of solutions Too many upgrade candidates Suppose we have components qi , for 1 ≤ i ≤ n, available in versions 1 and 2, all of which are installed in version 1 on the system. We want to install a component p in version 1 that depends on all of q1, . . . , qn, any version. Any of the 2n configurations {(p, 1)} ∪ {(qi , i)|i ∈ 1 . . . n, 1 ≤ i ≤ 2} is a solution. Which one do we choose? paranoid only install p trendy install p and step up all qi ’s to version 2 ... and exponentially many in between! Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 10 / 28
  11. 11. An exponential number of solutions??? The sidestep approach centralize and group : patch tuesday, service pack, ... isolate on a monolith : AppStore(s), ... coexist : NixOS, backward compatibility policy, no conflicts policy, ... The delegate to others approach identify “suites” of components (e.g. stable, testing, unstable), let the user order them, and then try to stick to this order Advantage tries to align the system with a chosen suite Disadvantage depends on the quality of the most recent state: not assured for development versions, or when mixing repositories difficult and cumbersome to obtain a different behaviour when things go wrong, we may really get lost Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 11 / 28
  12. 12. Installation woes: debatable solution # sudo apt-get install debhelper Reading Package Lists... Done Building Dependency Tree... Done The following extra packages will be installed: armagetron armagetron-common autoconf bonobo-activation codebreaker debconf debconf-i18n debconf-utils dialog esound-common fb-music-high fontconfig frozen-bubble-data grepmail gv intltool-debian libaiksaurus-data libaiksaurus0c102 libatk1.0-0 libatk1.0-dev libbonobo-activation4 libbonobo2-0 libbonobo2-common libdb3 libdbd-mysql-perl libdbi-perl libeel2-data libesd0 ... The following packages will be REMOVED: autoconf2.13 frozen-bubble frozen-bubble-lib gconf2 gnomemeeting itk3.1-dev libbonoboui2-0 libbonoboui2-common libdigest-md5-perl libforms0.89 libgconf2-4 libgnome2-0 libgnome2-common libgnomeui-0 libgnomevfs2-0 libgnomevfs2-common libgtk1.2-dev libgtk2.0-0png3 libgtk2.0-dev libmime-base64-perl libpango1.0-dev libsdl-mixer1.2-dev libsdl-perl libsdl-ttf1.2-dev libsdl1.2-dev libsmpeg-dev libstorable-perl nautilus tk8.3-dev tktable-dev x-window-system x-window-system-core xaw3dg-dev xlib6g xlib6g-dev xlibmesa-dev xlibmesa3 xlibosmesa3 xlibs-dev xlibs-pic xpdf xpdf-reader The following NEW packages will be installed: armagetron-common debconf-i18n fb-music-high fontconfig intltool-debian libaiksaurus-data libaiksaurus0c102 libeel2-data libfilehandle-unget-perl libfontconfig1 libforms1 libgdbm3 libgnutls7 libgsf-1 libice-dev libice6 libidl0 liblzo1 libmagick5.5.7 libmail-mbox-messageparser-perl libmysqlclient12 libncursesw5 libnet-daemon-perl libnewt0.51 libpaper1 libplrpc-perl libsdl-console ... 75 packages upgraded, 80 newly installed, 42 to remove and 858 not upgraded. Need to get 67.1MB of archives. After unpacking 26.9MB will be used. Do you want to continue? [Y/n] Abort. Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 12 / 28
  13. 13. Separation of concerns is the key Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 13 / 28
  14. 14. Package management reloaded There are several distinct tasks that need to be performed in a package management system: Server side Maintain a coherent set of packages when we add, remove, update packages that are made available for the users... left for another talk Client side fetch and autenticate metadata and packages user preferences ⇒specify the kind of solution we want dependency solver ⇒find a solution deploy the chosen solution Factoring out common work User preferences and dependency solving are really common to all. Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 14 / 28
  15. 15. Separation of concerns: solver Dependency solving is NP-hard: stop coding a petty solver for every new component based system, and adopt a modular approach! Common Upgrade Description Format Modular component manager Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 15 / 28
  16. 16. An excerpt from a CUDF file preamble: property: opam-version: string, opam-name: string package: herelib version: 3 depends: ocamlfind conflicts: herelib opam-name: herelib opam-version: 109.12.00 ... package: lwt version: 6 depends: base-threads , base-unix , camlp4 , ocamlfind conflicts: react >= 3 , lwt opam-name: lwt opam-version: 2.4.4 ... request: opam install: tyxml Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 16 / 28
  17. 17. Available external solvers Three external CUDF solvers packaged in Debian $apt-cache search cudf aspcud - CUDF solver based on Answer Set Programming mccs - multi-critera CUDF solver packup - CUDF solver based on pseudo-Boolean constraints There is also a nice solver for Java addicts p2cudf, based on the Eclipse P2 plugin dependency resolver, available from http://wiki.eclipse.org/Equinox/p2/CUDFResolver Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 17 / 28
  18. 18. Talking the external solver into doing what you want Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 18 / 28
  19. 19. Separation of concerns: user preferences Combinators from the latest Mancoosi preference language1 count(X) : number of packages in X sum(X,f) : sum of values of key f over packages in X aligned(X,g1,g2) : number of packages aligned according to given criteria notuptodate(X) : number of packages in X not current where X can be solution, down, up, removed, new, changed, request,installrequest,upgraderequest Combining preferences We can ask for a solution that maximizes (+) or minimizes (-) each of these criteria, and combine them in lexicographic order, e.g. -count(removed),-count(changed) 1 For details, see see http://www.mancoosi.org/misc-2012/criteria/ and http://opam.ocaml.org/doc/Specifying_Solver_Preferences.html Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 19 / 28
  20. 20. Examples preferences when installing your packages Simple paranoid -count(removed),-count(changed) trendy -count(removed),-notuptodate(solution),-count(new) Gourmet paranoid -count(removed),-notuptodate(request),-count(down),-count(changed) trendy -notuptodate(request),-count(removed),-count(down),-count(changed) Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 20 / 28
  21. 21. More exotic examples Building systems Minimal system size -sum(solution,installedsize),-count(solution) Noah’s ark +count(solution) Noah’s ark, fresh -notuptodate(solution),+count(solution) Performing upgrades upgrade -count(down),-count(removed),-notuptodate(solution),-count(new) priority -count(down),-count(removed),-notuptodate(solution), +sum(solution,priority),-count(new) Repairing a broken system configuration Use an empty request with fixup simple -count(changed) fixup trendy -count(changed),-count(down),-notuptodate(solution) Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 21 / 28
  22. 22. Ok, how can I use all this in opam? Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 22 / 28
  23. 23. External solvers in opam Thanks to the work done by the Mancoosi team at Irill and OCamlPro, in the DORM project, aspcud is supported out of the box in opam since 1.0,and all others in trunk: typing opam --help shows ... OPTIONS --criteria=CRITERIA Specify user preferences for dependency solving for this run. Overrides both $OPAMCRITERIA and $OPAMUPGRADECRITERIA. For details on the supported language, see http://opam.ocaml.org/doc/Specifying_Solver_Preferences.html. The default value is -removed,-notuptodate,-changed for upgrades, and -removed,-changed,-notuptodate otherwise. --cudf=FILENAME Debug option: Save the CUDF requests sent to the solver to FILENAME-<n>.cudf. --solver=CMD Specify the name of the external dependency solver. The default value is aspcud ... Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 23 / 28
  24. 24. Examples (when aspcud is present) Opam built-in defaults opam install ... --criteria="-count(removed),-count(changed),-notuptodate(solution)" opam ugrade --criteria="-count(removed),-noutuptodate(solution),-count(changed)" More advanced preferences opam install ... --criteria="-count(removed),-count(down), -notuptodate(request),-notuptodate(changed),-count(changed)" opam upgrade --criteria="-count(removed),-count(down),-notuptodate(solution), -count(new),+count(up)" Repair (now in trunk as opam upgrade --fixup) opam upgrade --criteria="-count(changed)" Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 24 / 28
  25. 25. Getting your external solver Debian/Ubuntu user? Lucky guy! Just apt-get install aspcud, and you are done No aspcud for you? Low-power arch (arduino, raspberri-pi)? You are still safe! go to http://cudf-solvers.irill.org follow the instructions access the Irill CUDF solver farm get your solving done in the cloud Big thanks to OCamlPro folks (Benjamin,Fabrice,Gregoire,Pierre), for help setting this up. Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 25 / 28
  26. 26. Credits The Mancoosi team in Paris Diderot and Irill Pietro Abate Ralf Treinen J´erˆome Vouillon Stefano Zacchiroli libcudf CUDF manipulation library: opam show cudf dose Mancoosi toolbox: opam show dose Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 26 / 28
  27. 27. Conclusions Package managers are complex: a very hard part is dependency solving! Modern package managers must share common components, in particular dependency solvers user preference language You can should use external solvers and preferences in opam today! Learn more at www.mancoosi.org and www.irill.org Di Cosmo, Leroy, Treinen, Vouillon et al Managing the complexity of large FOSS package-based software distributions. ASE 2006 Abate, Treinen Mancoosi Deliverable D5.4: Report on the international competition http://hal.inria.fr/hal-00698967/PDF/d5.4.pdf, 2011 Abate, Di Cosmo, Treinen, Zacchiroli Dependency solving: a separate concern in component evolution management. Journal of Systems and Software, 2012. Abate, Di Cosmo, Treinen, Zacchiroli A modular package manager architecture. Information and Software Technology, 2013. Full details at http://www.dicosmo.org/Articles/usercriteria.pdf Questions? Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 27 / 28
  28. 28. Abbreviations/old combinators The combinators from the old preference language are still recognised by aspcud: Old language New language removed count(removed) new count(new) changed count(changed) nutuptodate notuptodate(solution) Di Cosmo (Upd/Inria/Irill) Solver preferences for package managers 08/07/2014 28 / 28

×