FreeBSD package management system
Vsevolod Stakhov
vsevolod@FreeBSD.org

ruBSD conference December 14, 2013
Ports and packages

Ports is the comprehensive system of source packages.
Mature
Clear and well defined
Simple (sometimes n...
Ports before pkg
Packages
Makefile

Distfiles

Ports

Installed
Files

Patches

http://highsecure.ru/pkg

Vsevolod Stakhov...
Disadvantages of the old architecture

Make cannot handle complex packages relationships
Complicated upgrade procedure (ha...
Planned ports and pkg interaction

Build Depends

Build

STAGE

Packages

Pkg

Instal

Run Depends

http://highsecure.ru/p...
Ports and packages

Ports are used to build packages
Dependencies are resolved by pkg, not make
Stable branch of ports has...
Repositories creation

Repo
Stable
ports

Build

Mirrors
Checksum
Checksum
Signature

Package
Checksum

Package
Checksum

...
Pkg architecture

User

Request

Remote
Repos

Solver
FS

Mirrors

Jobs
fetch

http://highsecure.ru/pkg

Package

Install
...
The current problems with pkg

Legacy ports support (with no staging, for example)
Plain dependencies style
Naive solver

...
The problems of the solver in pkg

Absence of conflicts resolving/handling
No alternatives support
Can perform merely a sin...
Existing systems

There are many examples of solvers used in different package
management systems, for example:
Zypper/SUSE...
External solvers

To interact with an external solver we have chosen CUDF format
used in the Mancoosi research project htt...
We need an internal solver!
Alternatives:
Write own logic of dependencies and conflicts resolution?

http://highsecure.ru/p...
We need an internal solver!
Alternatives:
Write own logic of dependencies and conflicts resolution?
Use some existing solut...
We need an internal solver!
Alternatives:
Write own logic of dependencies and conflicts resolution?
Use some existing solut...
We need an internal solver!
Alternatives:
Write own logic of dependencies and conflicts resolution?
Use some existing solut...
Making a SAT problem

Assign a variable to each package: package A → a1 , package
B → b1
Interpret a request as a set of u...
Converting dependencies and conflicts
If package A depends on package B (versions B1 and B2 ),
then we can either have pack...
Converting dependencies and conflicts
If package A depends on package B (versions B1 and B2 ),
then we can either have pack...
The solving of SAT problem

Some rules to follow to speed up SAT problem solving.
Trivial propagation - solve unary clause...
SAT problem propagation

Trivial propagation - direct install or delete rules
(¬A B)& (A) & (¬C ) &(¬A ¬D)
true

http://hi...
SAT problem propagation

Trivial propagation - direct install or delete rules
(¬A B)& (A) & (¬C ) &(¬A ¬D)
true

false

Un...
Conflicts driven learning

To handle alternatives it is required to test all variables unassigned:
1. full depth-first enume...
Package specific assumptions

Pure SAT solvers cannot deal with package management as they
do not consider several packages...
Packages universe
We convert all packages involved to a packages universe of the
following structure:
Conflict
Depend

Nam...
Package management task

A request is splitted to install/upgrade and delete requests
which could be passed simultaneously...
Solvers and Pkg

Pkg may pass the formed universe to an external CUDF
solver:
convert versions
format request
parse output...
Perspectives

Using pkg solver for ports management
Better support of multiple repositories
Test different solvers algorith...
New dependencies format
libblah >= 1.0 + option1 , +option2 libfoo! = 1.1
Can depend on normal packages and virtual packag...
Alternatives
Used to organize packages with the same functionality (e.g.
web-browser)
May be used to implement virtual dep...
Thank you for your attention!
Questions?
vsevolod@FreeBSD.org

http://highsecure.ru/pkg

Vsevolod Stakhov vsevolod@FreeBSD...
Upcoming SlideShare
Loading in …5
×

Всеволод Стахов: Новое в Pkg и перспективы развития системы пакетов в FreeBSD

2,930 views
2,781 views

Published on

Доклад посвящён новому солверу, разработанному для Pkg. Мы рассмотрим его архитектуру, предпосылки создания, существующие аналоги и открывающиеся перспективы.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
2,930
On SlideShare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Всеволод Стахов: Новое в Pkg и перспективы развития системы пакетов в FreeBSD

  1. 1. FreeBSD package management system Vsevolod Stakhov vsevolod@FreeBSD.org ruBSD conference December 14, 2013
  2. 2. Ports and packages Ports is the comprehensive system of source packages. Mature Clear and well defined Simple (sometimes not) Configurable http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 2 of 26
  3. 3. Ports before pkg Packages Makefile Distfiles Ports Installed Files Patches http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 3 of 26
  4. 4. Disadvantages of the old architecture Make cannot handle complex packages relationships Complicated upgrade procedure (hard to keep up-to-date) Hard to migrate between releases Long build time http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 4 of 26
  5. 5. Planned ports and pkg interaction Build Depends Build STAGE Packages Pkg Instal Run Depends http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 5 of 26
  6. 6. Ports and packages Ports are used to build packages Dependencies are resolved by pkg, not make Stable branch of ports has an appropriate stable branch of packages Encourage users to install software from binary packages But do not prevent them from building custom packages from the ports http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 6 of 26
  7. 7. Repositories creation Repo Stable ports Build Mirrors Checksum Checksum Signature Package Checksum Package Checksum http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 7 of 26
  8. 8. Pkg architecture User Request Remote Repos Solver FS Mirrors Jobs fetch http://highsecure.ru/pkg Package Install + Manifest Checksum Files... Vsevolod Stakhov vsevolod@FreeBSD.org PkgDB 8 of 26
  9. 9. The current problems with pkg Legacy ports support (with no staging, for example) Plain dependencies style Naive solver http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 9 of 26
  10. 10. The problems of the solver in pkg Absence of conflicts resolving/handling No alternatives support Can perform merely a single task: install, upgrade or remove, so install task cannot remove packages for example http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 10 of 26
  11. 11. Existing systems There are many examples of solvers used in different package management systems, for example: Zypper/SUSE - uses libsolv as the base Yum/RedHat - migrating to libsolv Apt/Debian - uses internal solver Pacman/Archlinux - uses naive internal solver http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 11 of 26
  12. 12. External solvers To interact with an external solver we have chosen CUDF format used in the Mancoosi research project http://mancoosi.org: package: devel/libblah version: 1 depends: x11/libfoo package: security/blah version: 2 depends: devel/libblah conflicts: security/blah-devel http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 12 of 26
  13. 13. We need an internal solver! Alternatives: Write own logic of dependencies and conflicts resolution? http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 13 of 26
  14. 14. We need an internal solver! Alternatives: Write own logic of dependencies and conflicts resolution? Use some existing solution? http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 13 of 26
  15. 15. We need an internal solver! Alternatives: Write own logic of dependencies and conflicts resolution? Use some existing solution? Use some known algorithm? http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 13 of 26
  16. 16. We need an internal solver! Alternatives: Write own logic of dependencies and conflicts resolution? Use some existing solution? Use some known algorithm? Use SAT solver for packages management SAT expression (x1 ¬x2 x3 ) &(x3 ¬x1 )&(x2 ) Clause http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 13 of 26
  17. 17. Making a SAT problem Assign a variable to each package: package A → a1 , package B → b1 Interpret a request as a set of unary clauses: Install/Upgrade package A → (a1 ) Delete package B → (¬b1 ) Convert dependencies and conflicts to disjuncted clauses http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 14 of 26
  18. 18. Converting dependencies and conflicts If package A depends on package B (versions B1 and B2 ), then we can either have package A not installed or any of B installed: (¬A B1 B2 ) http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 15 of 26
  19. 19. Converting dependencies and conflicts If package A depends on package B (versions B1 and B2 ), then we can either have package A not installed or any of B installed: (¬A B1 B2 ) If we have a conflict between versions of B (B1 , B2 and B3 ) then we ensure that merely one version is installed: (¬B1 ¬B2 )&(¬B1 ¬B3 )&(¬B2 ¬B3 ) Conflicts chain http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 15 of 26
  20. 20. The solving of SAT problem Some rules to follow to speed up SAT problem solving. Trivial propagation - solve unary clauses Unit propagation - solve clauses with only a single unsolved variable Conflicts learning - if we assign some free variable and detect a conflict during unit propagation, we can fallback and learn that this variable must be negated Package specific assumptions. http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 16 of 26
  21. 21. SAT problem propagation Trivial propagation - direct install or delete rules (¬A B)& (A) & (¬C ) &(¬A ¬D) true http://highsecure.ru/pkg false Vsevolod Stakhov vsevolod@FreeBSD.org 17 of 26
  22. 22. SAT problem propagation Trivial propagation - direct install or delete rules (¬A B)& (A) & (¬C ) &(¬A ¬D) true false Unit propagation - simple depends and conflicts Dependency true false Conflict (¬A B) & (A) & (¬C ) & (¬A ¬D) B→true http://highsecure.ru/pkg D→false Vsevolod Stakhov vsevolod@FreeBSD.org 17 of 26
  23. 23. Conflicts driven learning To handle alternatives it is required to test all variables unassigned: 1. full depth-first enumeration of possible values 2. fallback if a conflict found 3. remember which assignment caused conflict 4. make negative assignment for the learned variable and go to the first step http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 18 of 26
  24. 24. Package specific assumptions Pure SAT solvers cannot deal with package management as they do not consider several packages peculiarities: try to keep installed packages (if no direct conflicts) do not install packages if they are not needed prefer high priority packages and repositories over low priority ones These options also improve SAT performance providing a good initial assignment. http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 19 of 26
  25. 25. Packages universe We convert all packages involved to a packages universe of the following structure: Conflict Depend Name N1 Version N2 N3 Nn V1 V1 V1 V1 V2 V2 V2 Conflicts Chain V3 http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 20 of 26
  26. 26. Package management task A request is splitted to install/upgrade and delete requests which could be passed simultaneously to the solver A conflicts between packages are detected with a repository creation All depends, reverse and conflicts of the requested packages are analyzed and the package universe is created Each package is defined by its name and the digest of significant fields (version, options and so on) http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 21 of 26
  27. 27. Solvers and Pkg Pkg may pass the formed universe to an external CUDF solver: convert versions format request parse output Alternatively the internal SAT solver may be used: convert the universe to SAT problem formulate request ??? PROFIT http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 22 of 26
  28. 28. Perspectives Using pkg solver for ports management Better support of multiple repositories Test different solvers algorithms using CUDF New dependencies and conflicts format Provides and alternatives http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 23 of 26
  29. 29. New dependencies format libblah >= 1.0 + option1 , +option2 libfoo! = 1.1 Can depend on normal packages and virtual packages (provides) Easy to define the concrete dependency versions Alternative dependencies P3 Or P1 Depends P2 > Vx Vy Vz ! Conflict http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 24 of 26
  30. 30. Alternatives Used to organize packages with the same functionality (e.g. web-browser) May be used to implement virtual dependencies (provides/requires) Alternatives Text Editor http://highsecure.ru/pkg Depends Package Vsevolod Stakhov vsevolod@FreeBSD.org 25 of 26
  31. 31. Thank you for your attention! Questions? vsevolod@FreeBSD.org http://highsecure.ru/pkg Vsevolod Stakhov vsevolod@FreeBSD.org 26 of 26

×