IPS: Image Packaging System
Upcoming SlideShare
Loading in...5
×
 

IPS: Image Packaging System

on

  • 1,708 views

An introduction to the Image Package System, found in illumos distributions such as OmniOS and OpenIndiana, as well as Oracle Solaris 11.

An introduction to the Image Package System, found in illumos distributions such as OmniOS and OpenIndiana, as well as Oracle Solaris 11.

Statistics

Views

Total Views
1,708
Views on SlideShare
1,685
Embed Views
23

Actions

Likes
1
Downloads
8
Comments
0

1 Embed 23

https://twitter.com 23

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    IPS: Image Packaging System IPS: Image Packaging System Presentation Transcript

    • IPS: It Sucks Less Than You Think Eric Sproul OmniTI
    • What Is IPS? Image Packaging System, aka "pkg(5)" Created by Sun for OpenSolaris Now used by OmniOS, OpenIndiana, Oracle Solaris 11 Transactional, metadata-driven and integrated with ZFS Network-based, extensive search grammar Changes-only updates
    • Motivations Unify packaging and OS patching Be smf(5)- and ZFS-aware Verify correct installation Optimize for the update case Ease developer burden Add dependency-based network retrieval
    • IPS: The Good Every package 100% described by metadata Updating requires fetching only changed assets Get a new BE automatically, when needed Automatic fetching of dependencies
    • IPS: The Not-So-Good No single-file on-disk format (except archives) Latency-sensitive No pre- or post-install scripting* * This is actually a good thing! Tasks usually scripted are now first-class actions
    • A Few IPS Commands pkg(1) :: installation and information client pkgsend(1) :: publication client pkgrecv(1) :: raw contents transfer utility pkg.depotd(1M) :: repository server pkgsign(1) :: cryptographic signing utility
    • IPS Concepts FMRI :: Fault Managed Resource Identifier Manifest :: describes a specific version of a package Publisher :: entity that provides one or more packages Repository :: location for publishing and retrieving pkgs Image :: location where packages may be installed Boot Environment :: (BE) bootable instance of an image
    • FMRIs in IPS pkg://omnios/web/curl@7.31.0,5.11-0.151006:20130703T175442Z omnios Publisher curl Name 7.31.0,5.11-0.151006:20130703T175442Z Version pkg Scheme web Category* * Category can be arbitrarily deep; Name is the basename (last element)
    • FMRIs in IPS Publisher name is optional: pkg://omnios/web/curl Must be preceded by '//' if present Scheme is also optional: /web/curl Leading '/' anchors to any publisher root pkg:/web/curl Note the use of only one '/' after the scheme web/curl Anything ending in '/web/curl' curl Anything named 'curl' or ending in '/curl' //omnios/web/curl Publisher included
    • FMRIs: Version Strictly numeric comparison, split on punctuation Comparison is left to right 7.31.0,5.11-0.151006:20130703T175442Z 7.31.0 Component Version ("the software's version") 5.11 Build Version (OS version, aka `uname -r`) 0.151006 Branch Version (distro-specific meaning) 20130703T175442Z Timestamp (ISO 8601)
    • FMRIs: Version Versions may be included when specifying names: curl@7.31 curl@*-0.151006 curl@7 Anything 7.x 7.31.x Any version for branch 0.151006
    • FMRIs: Version pkg://omnios/web/curl@7.31.0,5.11-0.151006:20130703T175442Z But, that's hard to read!? Version strings are for machines, not people! Rarely do you need to worry about anything but the component version
    • IPS Concepts FMRI :: Fault Managed Resource Identifier Manifest :: describes a specific version of a package Publisher :: entity that provides one or more packages Repository :: location for publishing and retrieving pkgs Image :: location where packages may be installed Boot Environment :: (BE) bootable instance of an image
    • Package Manifest Describes a specific version of a package Collection of actions that deliver files, dirs, links, dependencies, etc. via attributes Attributes are key-value pairs Viewable with `pkg contents -m <name>`
    • Package Manifest set name=pkg.fmri value=pkg://omnios/web/curl@7.31.0,5.11-0.151006:20130703T175442Z set name=pkg.summary value="curl - command line tool for transferring data with URL syntax" set name=pkg.descr value="curl - command line tool for transferring data with URL syntax" set name=publisher value=sa@omniti.com dir group=bin mode=0755 owner=root path=usr/bin/amd64 file 3a8938b01cf732fc0b4838218d94508fca75e54c chash=d923dfc752598ed149a64c873065fc71cbbf83fb elfarch=i386 elfbits=64 elfhash=aabff399422fb0e74df8ffb4356d7bee97db89a5 group=bin mode=0755 owner=root path=usr/bin/amd64/curl pkg.csize=100864 pkg.size=174672 ... link path=usr/lib/amd64/libcurl.so target=libcurl.so.4.3.0 ... depend fmri=library/security/openssl@1.0.1 type=require depend fmri=library/zlib type=require depend fmri=web/ca-bundle type=require
    • Manifests: Dependencies Require :: the referenced package provides essential functionality; including a version sets a "floor" Optional :: non-essential, but if installed, must meet version constraint, if any (same as require) Exclude :: conflicts; may not be installed with this package (these are evil, avoid them) Incorporate :: like optional, but sets "ceiling" as well as "floor" to the given degree of precision
    • Manifests: Dependencies require optional exclude # any version of foo library/foo ! # foo >= 2 library/foo@2 ! # foo >= 2.1 library/foo@2.1
    • Manifests: Dependencies incorporate # foo 2.x, not 1.x or 3.x library/foo@2 ! # foo 2.1.x, not 2.0 or 2.2 library/foo@2.1 ! # foo 2.1.2 only library/foo@2.1.2
    • Manifests: Dependencies Packages containing only incorporate dependencies are called "incorporations" ! Used to ensure a compatible set of installed software ! Used carefully, they can be very handy: ! omniti/incorporation/perl-516-incorporation
    • Manifests: Dependencies $ pkg contents -mr perl-516-incorporation set name=pkg.fmri value=pkg://perl.omniti.com/omniti/incorporation/perl-516-incorporation@5.16... set name=pkg.summary value="Constrains omniti/runtime/perl to version 5.16.x" set name=pkg.descr value="Constrains omniti/runtime/perl to version 5.16.x" set name=pkg.human-version value=5.16 set name=publisher value=sa@omniti.com depend fmri=omniti/runtime/perl@5.16 type=incorporate Version of omniti/runtime/perl must be 5.16.x Module dist pkgs have their own versions, but require the incorporation matching the perl they were built with
    • IPS Concepts FMRI :: Fault Managed Resource Identifier Manifest :: describes a specific version of a package Publisher :: entity that provides one or more packages Repository :: location for publishing and retrieving pkgs Image :: location where packages may be installed Boot Environment :: (BE) bootable instance of an image
    • Publisher An entity that provides packages Named for products ("omnios") 
 or domain style ("ms.omniti.com") One publisher can have multiple URLs
    • Publisher $ pkg publisher PUBLISHER TYPE STATUS URI omnios origin online http://pkg.omniti.com/omnios/release/ ms.omniti.com origin online http://pkg.omniti.com/omniti-ms/ perl.omniti.com origin online http://pkg.omniti.com/omniti-perl/ Publishers are searched in the listed order List publishers
    • Publisher # pkg set-publisher -g http://pkg.omniti.com/omniti-ms/ ms.omniti.com Add a publisher Change publisher URL # pkg set-publisher -G <old_url> —g <new_url> <publisher> # pkg unset-publisher ms.omniti.com Remove a publisher
    • IPS Concepts FMRI :: Fault Managed Resource Identifier Manifest :: describes a specific version of a package Publisher :: entity that provides one or more packages Repository :: location for publishing and retrieving pkgs Image :: location where packages may be installed Boot Environment :: (BE) bootable instance of an image
    • Repository Location to which packages are published Can be used locally (file://) 
 or remotely (http://) via pkg.depotd(1M) Created and managed by pkgrepo(1)
    • Repository # pkgrepo get -s /repo/omniti-ms/ SECTION PROPERTY VALUE publisher prefix ms.omniti.com repository version 4 ! # pkgrepo info -s /repo/omniti-ms/ PUBLISHER PACKAGES STATUS UPDATED ms.omniti.com 602 online 2014-03-23T20:50:49.146202Z Get repo information
    • Repository # pkgrepo get -s /repo/omniti-ms/publisher/ms.omniti.com/ SECTION PROPERTY VALUE feed description "" feed icon web/_themes/pkg-block-icon.png feed id "" feed logo web/_themes/pkg-block-logo.png feed name package repository feed feed window 24 publisher alias "" publisher prefix "" repository collection_type core repository description "" repository detailed_url "" repository legal_uris () repository maintainer "" repository maintainer_url "" repository mirrors () repository name package repository repository origins () repository refresh_seconds 14400 repository registration_uri "" repository related_uris () repository version 3 Get per-publisher information
    • Repository pkg5.repository publisher/ publisher/ms.omniti.com/ publisher/ms.omniti.com/catalog publisher/ms.omniti.com/catalog/catalog.attrs publisher/ms.omniti.com/catalog/catalog.base.C publisher/ms.omniti.com/catalog/catalog.dependency.C publisher/ms.omniti.com/catalog/catalog.summary.C publisher/ms.omniti.com/catalog/update.20140403T20Z.C Repository layout
    • Repository publisher/ms.omniti.com/file publisher/ms.omniti.com/file/23 publisher/ms.omniti.com/file/23/2394829fbd7dfffdcccf3108492fd439b5b39235 publisher/ms.omniti.com/file/5e publisher/ms.omniti.com/file/5e/5edd8900fe90576085fe13062568c633c9ad6b8f publisher/ms.omniti.com/file/86 publisher/ms.omniti.com/file/86/8624bcdae55baeef00cd11d5dfcfa60f68710a02 publisher/ms.omniti.com/file/ff publisher/ms.omniti.com/file/ff/ff26358690ff2fbe7d1b6171ab680eb40bc2ee64 Repository layout
    • Repository publisher/ms.omniti.com/index publisher/ms.omniti.com/pkg publisher/ms.omniti.com/pkg/omniti%2Fsystem%2Fmbuffer publisher/ms.omniti.com/pkg/omniti%2Fsystem%2Fmbuffer/ 20130220%2C5.11-0.151006%3A20130619T143708Z publisher/ms.omniti.com/tmp publisher/ms.omniti.com/tmp/lock publisher/ms.omniti.com/trans Repository layout
    • IPS Concepts FMRI :: Fault Managed Resource Identifier Manifest :: describes a specific version of a package Publisher :: entity that provides one or more packages Repository :: location for publishing and retrieving pkgs Image :: location where packages may be installed Boot Environment :: (BE) bootable instance of an image
    • Image Location where packages can be installed May be rooted at arbitrary point in the filesystem tree Default image rooted at '/' Has properties that govern policy; see pkg(1)
    • Image $ pkg property PROPERTY VALUE be-policy default ca-path /etc/ssl/certs check-certificate-revocation False flush-content-cache-on-success True mirror-discovery False preferred-authority publisher-search-order ['omnios', 'ms.omniti.com', 'circonus'] send-uuid True signature-policy verify signature-required-names [] trust-anchor-directory etc/ssl/certs use-system-repo False Image properties
    • IPS Concepts FMRI :: Fault Managed Resource Identifier Manifest :: describes a specific version of a package Publisher :: entity that provides one or more packages Repository :: location for publishing and retrieving pkgs Image :: location where packages may be installed Boot Environment :: (BE) bootable instance of an image
    • Boot Environment Bootable instance of an image Integrated with ZFS Can be auto-created according to image policy Can be manually created Created and managed by beadm(1M)
    • Boot Environment $ beadm list BE Active Mountpoint Space Policy Created omnios - - 6.76M static 2012-08-13 21:02 omnios-backup-1 - - 6.33M static 2012-09-11 17:00 omnios-backup-2 - - 210K static 2012-10-29 18:01 omnios-r151004 - - 7.06M static 2012-11-02 18:36 omnios-r151004-1 - - 63.0K static 2012-12-18 15:17 omnios-r151004-backup-1 - - 62.0K static 2012-12-18 15:44 omnios-r151004-backup-2 - - 87.0K static 2013-03-21 18:39 omnios-r151006 NR / 4.96G static 2014-04-03 14:12
    • IPS Concepts OK got it. Now, how do I actually do stuff?
    • Use Cases Install Update List/Info Inventory Search Audit
    • Use Cases: Install # dry run, verbose pkg install -nv foo ! # latest pkg install foo ! # latest available 2.x pkg install foo@2 ! # exact version pkg install foo@2.1.2 When "foo" is not installed
    • Use Cases: Update # dry run, verbose pkg update -nv foo ! # latest available pkg update foo ! # stay within 2.x line pkg update foo@2 ! # downgrade pkg update foo@1.9 Assuming "foo 2.1" is installed
    • Use Cases: List/Info # all installed packages pkg list ! # list installed packages matching "foo" pkg list foo ! # list all known versions of foo, installed or not pkg list -fav foo ! # detailed information pkg info foo ! # same, but remote pkg info -r foo
    • Use Cases: Inventory # file/directory paths only pkg contents foo ! # raw manifest pkg contents -m foo ! # same, but remote pkg contents -mr foo ! # list deps pkg contents -t depend -o fmri
    • Use Cases: Search Powerful due to package metadata Local or remote Expressive grammar Results sometimes non-obvious
    • Use Cases: Search pkg_name : action_type : key : token pkg_name :: the value of pkg.fmri action_type :: file, dir, link, depend, set, etc. key :: attribute name within the selected action token :: attribute value, i.e., "what you're searching for"
    • Use Cases: Search pkg_name : action_type : key : token Blank fields implicitly wild-carded Simple globbing permitted for pkg_name, token Leading colons optional `pkg search tmux` is effectively: `pkg search ':::tmux'` To have success, understand what you're looking for
    • Use Cases: Search # 'tmux' as any value pkg search tmux INDEX ACTION VALUE PACKAGE basename file usr/bin/tmux pkg:/terminal/tmux@1.6-0.151004 basename file usr/bin/tmux pkg:/terminal/tmux@1.6-0.151002 basename file usr/bin/tmux pkg:/terminal/tmux@1.7-0.151006 pkg.fmri set omnios/terminal/tmux pkg:/terminal/tmux@1.6-0.151004 pkg.fmri set omnios/terminal/tmux pkg:/terminal/tmux@1.6-0.151002 pkg.fmri set omnios/terminal/tmux pkg:/terminal/tmux@1.7-0.151006
    • Use Cases: Search # same as before, but show only pkg name pkg search -p tmux PACKAGE PUBLISHER pkg:/terminal/tmux@1.6-0.151002 omnios pkg:/terminal/tmux@1.6-0.151004 omnios pkg:/terminal/tmux@1.7-0.151006 omnios
    • Use Cases: Search $ pkg search 'dir::pgsql*' INDEX ACTION VALUE PACKAGE ... basename dir opt/pgsql925 pkg:/omniti/database/postgresql-925/ltree@9.2.5-0.151006 ... results from this manifest entry: dir group=bin mode=0755 owner=root path=opt/pgsql925 This answer:
    • Use Cases: Search $ pkg search -o pkg.name 'file:path:*perl*.so' PKG.NAME omniti/perl/db_file omniti/perl/b-callchecker omniti/perl/bsd-resource omniti/perl/clone ... Packages that deliver perl .so files
    • Use Cases: Search $ pkg search -H -o pkg.name 'depend::web/curl' developer/versioning/git developer/versioning/mercurial entire incorporation/jeos/omnios-userland Reverse dependencies $ pkg search -o pkg.fmri,fmri '*-0.151006:depend:incorporate:web/curl' PKG.FMRI FMRI pkg:/incorporation/jeos/omnios-userland@11,5.11-0.151006:20130506T214442Z web/curl@7,5.11-0.151006 pkg:/incorporation/jeos/omnios-userland@11,5.11-0.151006:20130716T202721Z web/curl@7,5.11-0.151006 pkg:/incorporation/jeos/omnios-userland@11,5.11-0.151006:20131030T205312Z web/curl@7,5.11-0.151006 What r151006 packages incorporate on curl, and at what version?
    • Use Cases: Audit # check installed state of all pkgs pkg verify ! # check state of a single package pkg verify <pkg> ! # repair installed state of a package pkg fix <pkg>
    • Use Cases: Audit # pkg verify -v curl PACKAGE STATUS pkg://omnios/web/curl OK ! # rm /usr/share/man/man3/libcurl.3 ! # pkg verify -v curl PACKAGE STATUS pkg://omnios/web/curl ERROR file: usr/share/man/man3/libcurl.3 Missing: regular file does not exist
    • Use Cases: Audit # pkg fix curl Verifying: pkg://omnios/web/curl ERROR file: usr/share/man/man3/libcurl.3 Missing: regular file does not exist Created ZFS snapshot: 2013-10-16-02:07:42 Repairing: pkg://omnios/web/curl ! DOWNLOAD PKGS FILES XFER (MB) Completed 1/1 1/1 0.0/0.0 ! PHASE ACTIONS Update Phase 1/1 ! PHASE ITEMS Image State Update Phase 2/2
    • Creating IPS Packages Build software however you wish Place build product in a proto area Create manifest Publish to a repo IPS does not impose a build framework (think rpmbuild, debuild)
    • Creating IPS Packages 1. `pkgsend generate /path/to/proto > /tmp/manifest.p5m` 2. Add FMRI, any other 'set' actions to manifest.p5m 3. `pkgsend publish -s <repo_url> -d /path/to/proto 
 /tmp/manifest.p5m` pkgsend(1) creates manifests and publishes packages
    • Creating IPS Packages Adding the 'set' stuff is tedious May want to make other changes/additions to manifest This needs to be automated! Use pkgmogrify(1)
    • Creating IPS Packages pkgmogrify(1) Programmatic transformations of manifest contents Macro replacements Include other manifests or manifest fragments Transformation of actions By convention, we store these directives in a .mog file beside our build scripts
    • Creating IPS Packages group gid=90 groupname=postgres user ftpuser=false gcos-field="PostgreSQL Reserved UID" group=postgres login-shell=/usr/bin/pfksh password=NP uid=90 username=postgres home-dir=/home/postgres license COPYING license=GPLv2 pkgmogrify: Add actions
    • Creating IPS Packages <transform dir path=opt/riak/data.* -> set owner riak> ! <transform dir path=opt/riak/data.* -> set group riak> ! <transform file path=opt/riak/etc/.*.args -> set mode 0644> ! <transform file path=opt/apache22/libexec/amd64/libphp5.so -> edit path libphp5.so libphp5.53.so> ! <transform file path=opt/elasticsearch/config/elasticsearch.yml -> set preserve true> ! <transform file path=opt/omni/lib/ruby/gems/1.9/cache.* -> drop> ! <transform file path=(var|lib)/svc/manifest/.*.xml -> add restart_fmri svc:/system/manifest-import:default> pkgmogrify: Transform actions
    • Creating IPS Packages Tangent: renaming pkg:/network/iftop pkg:/omniti/network/iftop Forgot to follow naming convention Also useful if upstream name changes Users may have installed it, can't just abandon it
    • Tangent: renaming Solution: publish a "rename package" Transitional package that allows update to new name Creating IPS Packages set name=pkg.fmri value=pkg://ms.omniti.com/network/iftop@1.0.2,5.11-0.151006:20130816T191418Z set name=pkg.renamed value=true set name=variant.opensolaris.zone value=global value=nonglobal depend fmri=pkg://ms.omniti.com/omniti/network/iftop type=require
    • Creating IPS Packages # pkgrepo create /data/myrepo # pkgrepo set -s /data/myrepo publisher/prefix=myrepo.example.com Create a repo with pkgrepo(1) May now use file:///data/myrepo to publish packages publisher/prefix sets the default publisher name
    • Creating IPS Packages $ pkgrecv -s http://pkg.omniti.com/omnios/release/ -d web_curl.p5a -a web/curl Retrieving packages for publisher omnios ... Retrieving and evaluating 1 package(s)... DOWNLOAD PKGS FILES XFER (MB) Completed 1/1 88/88 1.3/1.3 ! ! ARCHIVE FILES STORE (MB) web_curl.p5a 158/158 1.5/1.5 ! $ scp web_curl.p5a me@my-other-box: Create an archive with pkgrecv(1) # pkg install -g web_curl.p5a web/curl
    • Signing IPS Packages pkgsign(1) updates the manifest in place on the repo Adds the 'signature' action Validates the manifest, which in turn validates its content Signed package retains original timestamp
    • Signing IPS Packages signature <hash of certificate> algorithm=<signature algorithm> value=<signature value> chain="<hashes of certs needed to validate primary certificate>" version=<pkg version of signature> Payload & chain :: hashes of certs downloadable from originating repo Value :: signed hash of manifest's message text Algorithm :: hash algorithm used, default is rsa-sha256 Version :: pkg(5) version of the signature action
    • Signing IPS Packages First, publish the unsigned package(s); then: # pkgsign -c /path/to/signing.crt -k /path/to/signing.key -s <repo_url> <fmri_list> Multiple signatures (even from different entities) will not interfere with each other Allows different entities to indicate acceptance during publication process (e.g., dev/QA/release)
    • Questions? Further reading Man pages: pkg(5), pkg(1), pkgsend(1), pkgrecv(1), pkgmogrify(1), pkgrepo(1) http://omnios.omniti.com/wiki.php/ GeneralAdministration#PackageManagement http://omnios.omniti.com/media/ipsdevguide.pdf http://web.archive.org/web/20100105071515/http:// blogs.sun.com/sch/entry/pkg_1_a_no_scripting