Your SlideShare is downloading. ×
Pkg slides from BSDCan conference
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Pkg slides from BSDCan conference

63
views

Published on



In the context of package management, the solver is an algorithm (or set of algorithms) to resolve dependencies and conflicts. The solver must handle options, upgrades, multiple repos, locally installed software, as well as other factors. The upcoming 1.3 release of pkg will have the new solver that has some important consequences.

This talk is dedicated to the design concepts of the new solver in pkg management system (pkg-ng initially). In this talk, I describe the basic architecture of the solver, ideas used and the consequences of using this algorithm. Moreover, this talk describes the proposed pkg and ports architecture to simplify binary packages and ports using for all FreeBSD users.

Published in: Software

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
63
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. FreeBSD package management system Vsevolod Stakhov vsevolod@FreeBSD.org BSDCan May 17, 2014
  • 2. What is pkg Pkg (previously pkgng) is the binary package management system written for FreeBSD. Binary packages management Replaces old pkg_* tools Uses central sqlite3 based storage Provides the comprehensive toolset for binary packages management Vsevolod Stakhov vsevolod@FreeBSD.org 2 of 26
  • 3. Pkg development goals The main goal of pkg is to simplify system management tasks. Easy install, remove and upgrade of binary packages Integration with the ports Automatic resolving of dependencies and conflicts Provide secure package management tool Encourage users to install software from binary packages Vsevolod Stakhov vsevolod@FreeBSD.org 3 of 26
  • 4. Pkg development goals The main goal of pkg is to simplify system management tasks. Easy install, remove and upgrade of binary packages Integration with the ports Automatic resolving of dependencies and conflicts Provide secure package management tool Encourage users to install software from binary packages . . . but do not prevent users from building custom packages using the ports Vsevolod Stakhov vsevolod@FreeBSD.org 3 of 26
  • 5. Planned ports and pkg interaction Build STAGE Packages Pkg Instal Build Depends Run Depends Vsevolod Stakhov vsevolod@FreeBSD.org 4 of 26
  • 6. What is new in pkg 1.3 New solver that can automatically resolve complex upgrade or install scenarios Vsevolod Stakhov vsevolod@FreeBSD.org 5 of 26
  • 7. What is new in pkg 1.3 New solver that can automatically resolve complex upgrade or install scenarios Improved security by sandboxing untrusted operations: Pkg environment Sandbox Unsafe operation Result Sandboxing: archives extracting vulnxml parsing repositories signatures checking and public keys extracting Vsevolod Stakhov vsevolod@FreeBSD.org 5 of 26
  • 8. What is new in pkg 1.3 New solver that can automatically resolve complex upgrade or install scenarios Improved security by sandboxing untrusted operations: Pkg environment Sandbox Unsafe operation Result Sandboxing: archives extracting vulnxml parsing repositories signatures checking and public keys extracting Concurrent locking system Vsevolod Stakhov vsevolod@FreeBSD.org 5 of 26
  • 9. Pkg architecture Mirrors Jobs fetch Solver Checksum + Manifest Files... Package Install FS PkgDB User Request Remote Repos Vsevolod Stakhov vsevolod@FreeBSD.org 6 of 26
  • 10. The problems of the old solver in pkg Absence of conflicts resolving No alternatives support (plain dependencies only) Can perform merely a single task: either install or upgrade or remove Vsevolod Stakhov vsevolod@FreeBSD.org 7 of 26
  • 11. Tasks to solve Ports renaming: simple: racket-textual → racket-minimal splitting/merging: A A B C A B C A Ports reorganising: files moving dependencies change adding or removing new conflicts Vsevolod Stakhov vsevolod@FreeBSD.org 8 of 26
  • 12. Tasks to solve There are another issues to be resolved: Find conflicts using files list Set jobs priorities using the following rules: install dependencies first check for reverse dependencies and increase priority deal with conflicts using the same priority packages removing reverses the priority order Vsevolod Stakhov vsevolod@FreeBSD.org 9 of 26
  • 13. Existing systems There are many examples of solvers used in different package management systems, for example: Zypper/SUSE - uses libsolv as the base Yum/RedHat - migrating to libsolv OpenBSD/pkg_add - uses internal solver Apt/Debian - uses internal solver Pacman/Archlinux - uses internal solver Vsevolod Stakhov vsevolod@FreeBSD.org 10 of 26
  • 14. External solvers To interact with an external solver we have chosen the CUDF format used in the Mancoosi research project http://mancoosi.org: package: devel/libblah version: 1 depends: x11/libfoo package: security/blah version: 2 depends: devel/libblah conflicts: security/blah-devel Vsevolod Stakhov vsevolod@FreeBSD.org 11 of 26
  • 15. Interaction with external solver There are some limitations and incompatibilities with CUDF. CUDF supports plain integers as versions and we need to convert versions twice There is no support of options in CUDF packages formulas External solvers are often too complicated and large CUDF transformation is expensive in terms of performance Vsevolod Stakhov vsevolod@FreeBSD.org 12 of 26
  • 16. We need an internal solver! Alternatives: Write own logic of dependencies and conflicts resolution? Vsevolod Stakhov vsevolod@FreeBSD.org 13 of 26
  • 17. We need an internal solver! Alternatives: Write own logic of dependencies and conflicts resolution? Use some existing solution? Vsevolod Stakhov vsevolod@FreeBSD.org 13 of 26
  • 18. We need an internal solver! Alternatives: Write own logic of dependencies and conflicts resolution? Use some existing solution? Use some known algorithm? Vsevolod Stakhov vsevolod@FreeBSD.org 13 of 26
  • 19. We need an internal solver! Alternatives: Write own logic of dependencies and conflicts resolution? Use some existing solution? Use some known algorithm? Use SAT solver for packages management SAT expression (x1 ¬x2 x3) Clause &(x3 ¬x1)&(x2) Vsevolod Stakhov vsevolod@FreeBSD.org 13 of 26
  • 20. Packages universe We convert all packages involved to a packages universe of the following structure: Name Version Conflicts Chain Depend Conflict N1 N2 N3 Nn V1 V1 V1 V1 V2 V2 V2 V3 Vsevolod Stakhov vsevolod@FreeBSD.org 14 of 26
  • 21. Making a SAT problem Assign a variable to each package: package A → a1, package B → b1 Interpret a request as a set of unary clauses: Install/Upgrade package A → (a1) Delete package B → (¬b1) Convert dependencies and conflicts to disjuncted clauses Vsevolod Stakhov vsevolod@FreeBSD.org 15 of 26
  • 22. Converting dependencies and conflicts If package A depends on package B (versions B1 and B2), then we can either have package A not installed or any of B installed: (¬A B1 B2) Vsevolod Stakhov vsevolod@FreeBSD.org 16 of 26
  • 23. Converting dependencies and conflicts If package A depends on package B (versions B1 and B2), then we can either have package A not installed or any of B installed: (¬A B1 B2) If we have a conflict between versions of B (B1, B2 and B3) then we ensure that merely one version is installed: (¬B1 ¬B2)&(¬B1 ¬B3)&(¬B2 ¬B3) Conflicts chain Vsevolod Stakhov vsevolod@FreeBSD.org 16 of 26
  • 24. The solving of SAT problem Some rules to follow to speed up SAT problem solving. Trivial propagation - solve unary clauses Unit propagation - solve clauses with only a single unsolved variable DPLL algorithm backtracking. Package specific assumptions. Vsevolod Stakhov vsevolod@FreeBSD.org 17 of 26
  • 25. SAT problem propagation Trivial propagation - direct install or delete rules (¬A B)& (A) true & (¬C) false &(¬A ¬D) Vsevolod Stakhov vsevolod@FreeBSD.org 18 of 26
  • 26. SAT problem propagation Trivial propagation - direct install or delete rules (¬A B)& (A) true & (¬C) false &(¬A ¬D) Unit propagation - simple depends and conflicts Dependency (¬A B) B→true & true (A) & false (¬C) & Conflict (¬A ¬D) D→false Vsevolod Stakhov vsevolod@FreeBSD.org 18 of 26
  • 27. DPLL algorithm DPLL is proved to be one of the efficient algorithms to solve SAT problem (not the fastest but more simple than alternatives). x1 x2 x3 1 0 1 Vsevolod Stakhov vsevolod@FreeBSD.org 19 of 26
  • 28. DPLL algorithm DPLL is proved to be one of the efficient algorithms to solve SAT problem (not the fastest but more simple than alternatives). x1 x2 x3 1 0 1 0 Vsevolod Stakhov vsevolod@FreeBSD.org 19 of 26
  • 29. DPLL algorithm DPLL is proved to be one of the efficient algorithms to solve SAT problem (not the fastest but more simple than alternatives). x1 x2 x3 1 0 1 0 1 0 0 1 Vsevolod Stakhov vsevolod@FreeBSD.org 19 of 26
  • 30. Package specific assumptions Pure SAT solvers cannot deal with package management as they do not consider several packages peculiarities: try to keep installed packages (if no direct conflicts) do not install packages if they are not needed (but try to upgrade if a user has requested upgrade) These options also improve SAT performance providing a good initial assignment. Vsevolod Stakhov vsevolod@FreeBSD.org 20 of 26
  • 31. Solvers and Pkg Pkg may pass the formed universe to an external CUDF solver: convert versions format request parse output Alternatively the internal SAT solver may be used: convert the universe to SAT problem formulate request ??? PROFIT Vsevolod Stakhov vsevolod@FreeBSD.org 21 of 26
  • 32. Perspectives Using pkg solver for ports management Better support of multiple repositories Test different solvers algorithms using CUDF New dependencies and conflicts format Provides and alternatives Vsevolod Stakhov vsevolod@FreeBSD.org 22 of 26
  • 33. New dependencies format libblah >= 1.0 + option1, +option2 libfoo! = 1.1 Can depend on normal packages and virtual packages (provides) Easy to define the concrete dependency versions Alternative dependencies P1 Depends P2 Vx > Vy Vz ! P3 Or Conflict Vsevolod Stakhov vsevolod@FreeBSD.org 23 of 26
  • 34. Alternatives Used to organize packages with the same functionality (e.g. web-browser) May be used to implement virtual dependencies (provides/requires) Alternatives Text Editor Depends Package Vsevolod Stakhov vsevolod@FreeBSD.org 24 of 26
  • 35. Existing issues to be solved prior to 1.3 A solver cannot find install candidates for non-automatic top level packages (those without reverse depends) Package upgrade is performed improperly (need to rename, install and unlink) Minor issues and crashes Vsevolod Stakhov vsevolod@FreeBSD.org 25 of 26
  • 36. Questions? vsevolod@FreeBSD.org Vsevolod Stakhov vsevolod@FreeBSD.org 26 of 26