inside-BigData.com, profile picture
Uploaded byinside-BigData.com
PDF, PPTX882 views

Binary Packaging for HPC with Spack

This document describes Spack, an open source package management system for HPC environments. Spack allows building and installing software and its dependencies across different compiler versions and build configurations. It uses a Python DSL for package definitions and handles dependency resolution and installation. Recent updates include support for binary packages to improve performance through pre-built binaries while retaining flexibility of source installations.

Embed presentation

Download as PDF, PPTX
LLNL-PRES-745747 This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under contract DE-AC52-07NA27344. Lawrence Livermore National Security, LLC Binary packaging for HPC with Spack HPC, Big Data, and Data Science Devroom at FOSDEM 2018 Brussels, Belgium Todd Gamblin Center for Applied Scientific Computing LLNLFeburary 4, 2018
LLNL-PRES-745747 2@spackpmgithub.com/spack Spack is a general purpose, from-source package manager § Inspired somewhat by homebrew and nix § Targets HPC and scientific computing — Community is growing! § Goals: — Facilitate experimenting with performance options — Flexibility. Make these things easy: • Build packages with many different: – compilers/versions/build options • Change compilers and flags in builds (keep provenance) • Swap implementations of ABI-incompatible libraries – MPI, BLAS, LAPACK, others like jpeg/jpeg-turbo, etc. — Build software stacks for scientific simulation and analysis — Run on laptops, Linux clusters, and some of the largest supercomputers in the world Spack https://spack.io
LLNL-PRES-745747 3@spackpmgithub.com/spack Spec CLI syntax makes it easy to install different ways $ spack install mpileaks unconstrained $ spack install mpileaks@3.3 @ custom version $ spack install mpileaks@3.3 %gcc@4.7.3 % custom compiler $ spack install mpileaks@3.3 %gcc@4.7.3 +threads +/- build option $ spack install mpileaks@3.3 cflags="-O3 –g3" setting compiler flags $ spack install mpileaks@3.3 ^mpich@3.2 %gcc@4.9.3 ^ dependency constraints § Each expression is a spec for a particular configuration — Each clause adds a constraint to the spec — Constraints are optional – specify only what you need. — Customize install on the command line! § Spec syntax is recursive — ^ (caret) adds constraints on dependencies
LLNL-PRES-745747 4@spackpmgithub.com/spack Spack packages are templates: they define how to build a spec from spack import * class Dyninst(Package): """API for dynamic binary instrumentation.""” homepage = "https://paradyn.org" url = "http://www.paradyn.org/release8.1.2/DyninstAPI-8.1.2.tgz" version('8.2.1', 'abf60b7faabe7a2e’) version('8.1.2', 'bf03b33375afa66f’) version('8.1.1', 'd1a04e995b7aa709’) depends_on("cmake", type="build") depends_on("libelf", type="link") depends_on("libdwarf", type="link") depends_on("boost @1.42: +multithreaded") def install(self, spec, prefix): with working_dir('spack-build', create=True): cmake('-DBoost_INCLUDE_DIR=‘ + spec['boost'].prefix.include, '-DBoost_LIBRARY_DIR=‘ + spec['boost'].prefix.lib, '-DBoost_NO_SYSTEM_PATHS=TRUE’ '..') make() make("install") Metadata at the class level Versions Install logic in instance methods Dependencies Patches, variants, resources, conflicts, etc. (not shown) Simple Python DSL — Packages are classes (ala homebrew) — Directives use the same spec syntax
LLNL-PRES-745747 5@spackpmgithub.com/spack § mpi is a virtual dependency § Install the same package built with two different MPI implementations: § Virtual deps are replaced with a valid implementation at resolution time. — If the user didn’t pick something and there are multiple options, Spack picks. Depend on interfaces (not implementations) with virtual dependencies $ spack install mpileaks ^mvapich $ spack install mpileaks ^openmpi@1.4: mpileaks mpi callpath dyninst libdwarf libelf class Mpileaks(Package): depends_on("mpi@2:") class Mvapich(Package): provides("mpi@1” when="@:1.8") provides("mpi@2” when="@1.9:") class Openmpi(Package): provides("mpi@:2.2" when="@1.6.5:") Virtual dependencies can be versioned: dependent provider provider
LLNL-PRES-745747 6@spackpmgithub.com/spack Spack builds packages with compiler wrappers Spack Process Set up environment CC = spack/env/intel/icc SPACK_CC = /opt/ic-15.1/bin/icc CXX = spack/env/intel/icpc SPACK_CXX = /opt/ic-15.1/bin/icpc F77 = spack/env/intel/ifort SPACK_F77 = /opt/ic-15.1/bin/ifort FC = spack/env/intel/ifort SPACK_FC = /opt/ic-15.1/bin/ifort PKG_CONFIG_PATH = ... PATH = spack/env:$PATH CMAKE_PREFIX_PATH = ... LIBRARY_PATH = ... do_install() Install dep1 Install dep2 Install package… Build Process Fork install() configure make make install -I /dep1-prefix/include -L /dep1-prefix/lib -Wl,-rpath=/dep1-prefix/lib Compiler wrappers (spack-cc, spack-c++, spack-f77, spack-f90) icc icpc ifort ▪ Similar to homebrew “shims” ▪ Forked build process isolates environment for each build ▪ Use compiler wrappers to add include, lib, and RPATH flags ▪ RPATHs ensure that the correct dependencies are found automatically at runtime.
LLNL-PRES-745747 7@spackpmgithub.com/spack § Each unique dependency graph is a unique configuration. § Each configuration installed in a unique directory. — Configurations of the same package can coexist. § Hash of directed acyclic graph (DAG) metadata is appended to each prefix — Note: we hash the metadata, not the artifact. § Installed packages automatically find dependencies — Spack embeds RPATHs in binaries. — No need to set LD_LIBRARY_PATH — Things work the way you built them Hashes handle combinatorial software complexity. spack/opt/ linux-rhel7-x86_64/ gcc-4.7.2/ mpileaks-1.1-0f54bf34cadk/ intel-14.1/ hdf5-1.8.15-lkf14aq3nqiz/ bgq/ xl-12.1/ hdf5-1-8.16-fqb3a15abrwx/ ... mpileaks mpi callpath dyninst libdwarf libelf Installation Layout Dependency DAG Hash
LLNL-PRES-745747 8@spackpmgithub.com/spack Spack’s dependency model centers around “concretization” mpileaks ^callpath@1.0+debug ^libelf@0.8.11 User input: abstract spec Concrete spec is fully constrained and can be built. mpileaks@2.3 %gcc@4.7.3 =linux-ppc64 mpich@3.0.4 %gcc@4.7.3 =linux-ppc64 callpath@1.0 %gcc@a4.7.3+debug =linux-ppc64 dyninst@8.1.2 %gcc@4.7.3 =linux-ppc64 libelf@0.8.11 %gcc@4.7.3 =linux-ppc64 libdwarf@20130729 %gcc@4.7.3 =linux-ppc64 Abstract, normalized spec with some dependencies. mpileaks mpi callpath@1.0 +debug dyninst libelf@0.8.11 libdwarf Concretize § Similar to other dependency resolvers, but solves for more than just package and version. § Full spec is stored in a file in the installation directory — Can reinstall same build with: spack install –f spec.yaml Store spec: - mpileaks: arch: linux-x86_64 compiler: name: gcc version: 4.9.2 dependencies: adept-utils: kszrtkpbzac3ss2ixcjkcorlaybnptp4 callpath: bah5f4h4d2n47mgycej2mtrnrivvxy77 mpich: aa4ar6ifj23yijqmdabeakpejcli72t3 hash: 33hjjhxi7p6gyzn5ptgyes7sghyprujh variants: {} version: '1.0' - adept-utils: arch: linux-x86_64 compiler: name: gcc version: 4.9.2 dependencies: boost: teesjv7ehpe5ksspjim5dk43a7qnowlq mpich: aa4ar6ifj23yijqmdabeakpejcli72t3 hash: kszrtkpbzac3ss2ixcjkcorlaybnptp4 variants: {} version: 1.0.1 ... spec.yaml Detailed provenance is stored with the installed package
LLNL-PRES-745747 9@spackpmgithub.com/spack Source installs are great, but they’re slow § Most people prefer using a binary package manager — Binary packages typically use portable code — Binary installs are typically a lot slower than what you get from building from source § We’d like to have the best of both worlds: — Optimized buids for specific machine models (skylake, haswell, ivy bridge, etc.) — Binary packages available without having to build from source § What’s needed? 1. Binary packaging capability 2. Metadata describing architecture-specific builds 3. Good dependency resolution to select optimized or generic versions of packages
LLNL-PRES-745747 10@spackpmgithub.com/spack We recently released Spack v0.11 § 2,178 packages (up from 1,114 a year ago) § Big features for users: — Relocatable binary packages (spack buildcache) — Full support for Python 3 — Improved module support; custom module templates using jinja2 § Many improvements for packagers: — Multi-valued variants — Test dependency type — Packages can patch their dependencies (not just themselves) § Many speed improvements (to Spack itself) https://github.com/spack/spack/releases/
LLNL-PRES-745747 11@spackpmgithub.com/spack Binary packaging in Spack v0.11 § Spack v0.11 has a new spack buildcache command: spack buildcache create <spec> # create a new binary package spack buildcache list # list available binaries spack buildcache install # install a binary package (specifically) § Typically, install is not needed; you can just do: spack install --use-cache # prefer binaries if available § We don’t enable binaries by default yet — We’ll make –use-cache default when we start hosting stable binaries § Thanks to our collaboration with Fermilab, CERN, and Kitware for this feature!
LLNL-PRES-745747 12@spackpmgithub.com/spack How to make a binary 1. Set up GNU PG for binary signing — Unsigned binaries can be created but are discouraged spack gpg create "Todd Gamblin" tgamblin@gmail.com # create a new signing keypair spack gpg init # trust initial keypair 2. Install something spack install m4 # install m4 ... spack find ==> 2 installed packages. -- darwin-elcapitan-x86_64 / clang@8.0.0-apple ------------------ libsigsegv@2.11 m4@1.4.18 3. Run spack buildcache create on that thing spack buildcache create -d /path/to/mirror m4 # create a binary package in mirror § Binaries and metadata for the package and its dependencies will be in: /path/to/mirror/build_cache/
LLNL-PRES-745747 13@spackpmgithub.com/spack Binary mirror structure § Binaries go in build_cache/<platform>/<compiler>/<pkg-version> § Metadata for all packages is (currently) kept at the top level — We’ll need to index these files eventually § build_cache subdirectory sits inside of a Spack mirror directory — Makes it easy to add binaries to an existing source mirror § This structure is very easy to host in something like S3, a web server, or a shared filesystem mirror/ !""build_cache #""darwin-elcapitan-x86_64 $ !""clang-8.0.0-apple $ #""libsigsegv-2.11 $ $ !"" darwin-elcapitan-x86_64-clang-8.0.0-apple-libsigsegv-2.11-5rjv56uui3crfmsgsqgqab52yfhr6t4b.spack $ !""m4-1.4.18 $ !"" darwin-elcapitan-x86_64-clang-8.0.0-apple-m4-1.4.18-eqbzieloqiwxfe4drksmekvfqia7mqbu.spack #"" darwin-elcapitan-x86_64-clang-8.0.0-apple-libsigsegv-2.11-5rjv56uui3crfmsgsqgqab52yfhr6t4b.spec.yaml #"" darwin-elcapitan-x86_64-clang-8.0.0-apple-m4-1.4.18-eqbzieloqiwxfe4drksmekvfqia7mqbu.spec.yaml !"" index.html
LLNL-PRES-745747 14@spackpmgithub.com/spack Pointing Spack to a mirror § Spack v0.11 has a new spack buildcache command: $ spack mirror add mypkgs https://example.spack-mirror.com/mirror § You can verify that it worked by looking at what mirrors are configured: $ spack mirror list mypkgs https://example.spack-mirror.com/mirror § Mirrors can contain source tarballs and binaries — Detailed info in docs on mirrors.yaml
LLNL-PRES-745747 15@spackpmgithub.com/spack How fetching works in spack Concretize fetch source code build load mpileaks/package.py from repo fetch from user mirrors from var/spack/cache (local mirror) fetch from package URL install install relocate configureBinary available? Yes No Install from binary Build from source verify signature verify checksum ... spack install mpileaks
LLNL-PRES-745747 16@spackpmgithub.com/spack What’s in a Spack binary package? § The binary is just a tarball § Contains: 1. Another tarball of the installed prefix 2. The spec.yaml: • describes the build (Spack metadata) • Contains a special entry with the checksum of the source tarball (maps spack hash to SHA256) 3. A signature • tells us we can trust the spec.yaml $ tar tzf darwin-elcapitan-x86_64-clang-8.0.0-apple-m4-1.4.18-eqbzieloqiwxfe4drksmekvfqia7mqbu.spack darwin-elcapitan-x86_64-clang-8.0.0-apple-m4-1.4.18-eqbzieloqiwxfe4drksmekvfqia7mqbu.tar.gz darwin-elcapitan-x86_64-clang-8.0.0-apple-m4-1.4.18-eqbzieloqiwxfe4drksmekvfqia7mqbu.spec.yaml darwin-elcapitan-x86_64-clang-8.0.0-apple-m4-1.4.18-eqbzieloqiwxfe4drksmekvfqia7mqbu.spec.yaml.asc
LLNL-PRES-745747 17@spackpmgithub.com/spack Why do we checksum source but sign binaries? § Other systems provide checksums for sources and binaries in their package files — e.g., homebrew “bottles” § In Spack, the number of binaries associated with a source tarball can be very large! — We could have thousands of binaries for the same source: • Different flags, different build options etc. — Each of these would have a different Spack § Putting checksums for all of these in the package files: — Would add a lot of extra bytes to a package repository — Is unmaintainable — Means that we have to update package.py files whenever we update a mirror § With signing, the client can trust one or several keys and verify a large number of packages with a small number of public keys
LLNL-PRES-745747 18@spackpmgithub.com/spack What’s relocation? § When Spack creates a binary package, it traverses the installation directory and examines the files — Uses the file command § It records the files that need to be relocated after installation: — Libraries with RPATHs — Shell scripts with #! Lines § After installation, Spack: — rewrites RPATHs with patchelf (Linux) or install_name_tool (macOS) — Rewrites #! lines to point to the Spack installation on the installing machine § Install is faster b/c we record the needed relocations at package creation time spack/opt/ linux-rhel7-x86_64/ gcc-4.7.2/ mpileaks-1.1-0f54bf34cadk/ bin/ mpileaks-run lib/ libmpileaks.so Installation Layout § We try to make root-relative RPATHs when possible, but don’t always get everything. § We also rewrite RPATHs to directories within the spack root fi the install machine uses a different layout. § We are not currently relocating compiler runtime paths — We should. This is work in progress
LLNL-PRES-745747 19@spackpmgithub.com/spack How do we decide which binaries to fetch? § We currently only fetch binaries if they satisfy the exact hash result of concretization § This doesn’t leave a lot of room for change in the system — Small changes in Spack mean having to build from source again — Works best on a stable release § We are working on a new concretizer that will consider available binary specs — Doing this better requires a backtracking SAT solve mpileaks@2.3 %gcc@4.7.3 =linux-ppc64 mpich@3.0.4 %gcc@4.7.3 =linux-ppc64 callpath@1.0 %gcc@a4.7.3+debug =linux-ppc64 dyninst@8.1.2 %gcc@4.7.3 =linux-ppc64 libelf@0.8.11 %gcc@4.7.3 =linux-ppc64 libdwarf@20130729 %gcc@4.7.3 =linux-ppc64 mpileaks mpi callpath@1.0 +debug dyninst libelf@0.8.11 libdwarf Concretize Current In progress Fetch exact hashes, if available mpileaks@2.3 %gcc@4.7.3 =linux-ppc64 mpich@3.0.4 %gcc@4.7.3 =linux-ppc64 callpath@1.0 %gcc@a4.7.3+debug =linux-ppc64 dyninst@8.1.2 %gcc@4.7.3 =linux-ppc64 libelf@0.8.11 %gcc@4.7.3 =linux-ppc64 libdwarf@20130729 %gcc@4.7.3 =linux-ppc64 mpileaks mpi callpath@1.0 +debug dyninst libelf@0.8.11 libdwarf Concretize Prefer available binaries in concretization Download available specs from mirror
LLNL-PRES-745747 20@spackpmgithub.com/spack Spack can ship optimized binaries § The Spack architecture descriptor currently includes: — Platform: cray, mac, linux, bgq • Meant to represent a family of machines with potentially many OS/target combinations — OS: rhel6, rhel7, ubuntu14, elcapitan, sierra, centos6, centos7, etc. — Target: • Generic: x86_64, ppc64le, etc. • Specific: haswell, ivybridge, knl, power8, power9, etc. § Some triples: — darwin-sierra-x86_64 — darwin-elcapitan-x86_64 — cray-cnl6-knl — cray-cnl6-haswell § These architecture descriptors are part of the binary metadata — If we can fetch an index of available packages first, we can be picky about what binaries we want — This is a core spec parameter in Spack, not just a naming convention for packages
LLNL-PRES-745747 21@spackpmgithub.com/spack Detecting optimized architectures § Currently Spack will only use an optimized architecture descriptor on Cray — We get the architecture name from the Cray Programming Environment — We can know whether we’re building for Haswell, Broadwell, KNL, etc. § We have work in progress that detects these names for Intel, AMD, Power, and ARM hardware (looking at available info in /proc/cpuinfo, etc.) — We’re planning to shift to a model where we use the specific descriptor by default — We would still allow a user to set preferences to build generic if they want — Important for CERN and Fermi collaborators who run heterogeneous clusters § Once this is done, we do plan to make arch-specific binaries available.
LLNL-PRES-745747 22@spackpmgithub.com/spack Some issues with optimized binaries § Architectures like ARM don’t lend themselves to concise descriptors. — We may need to be more fine-grained here § We may need to add a more fine-grained architecture descriptor that just exposes the instruction sets available on the machine — E.g., instead of “haswell”, put “sse4.2, avx, avx2, etc.” — These attributes may actually be easier for packagers and maintainers to use § We need a setting on the user side (e.g. in packages.yaml) that lets them choose what the minimum architecture is for compatibility — This seems easier to do with well known system names
LLNL-PRES-745747 23@spackpmgithub.com/spack Tuning for optimized binaries may be tricky § Most compilers give you two knobs to control architecture-specific tuning: -march=[generic|native] -mtune=[generic|native] — If you tune generic for an older architecture, it will run fine on that architecture and on newer architectures. — If you tune native, you’ll get code that only runs fast on the specific architecture, and may not perform as well on future chips. § In Spack, we might want policies like this: — “Generic tuning, with code no later than sandy bridge”, e.g. if sandy bridge is the older architecture on a heterogeneous cluster — “Native tuning, just for this machine”, e.g., if we know we want to optimize for a long-lived, homogeneous cluster.
LLNL-PRES-745747 24@spackpmgithub.com/spack Building a primary binary mirror for Spack § We’re currently setting up build automation to create binaries for: — All default package configurations in releases (result of spack install <name>) — Selected other slices of those configuration spaces, e.g.: • x MPI versions • x Compilers • x OS’s • x large-scale DOE machines (Cori, Theta, Titan, Summit, etc.) § Once this is done, we’ll also continuously build packages for the develop branch as PRs come in — We’ll need to determine when to purge old builds and binaries — Depends on analytics § We are currently planning to host binaries in S3 — but Jfrog/bintray sounds interesting. Maybe we should talk to them.
LLNL-PRES-745747 25@spackpmgithub.com/spack Summary We built relocatable binary packaging into Spack Current projects: 1. Binary build infrastructure 2. Better concretization to support optimized binaries 3. Compiler library relocation Shooting for September to have all of this done. Spack Come and get Spack stickers! https://spack.io
Binary Packaging for HPC with Spack

More Related Content

PPT
2 Linux Container and Docker
byFabio Fumarola
 
PDF
Docker在豆瓣的实践 刘天伟-20160709
byTianwei Liu
 
PDF
Docker meetup - PaaS interoperability
byLudovic Piot
 
PDF
[네이버오픈소스세미나] Maglev Hashing Scheduler in IPVS, Linux Kernel - 송인주
byNAVER Engineering
 
PDF
Continuous Integration: SaaS vs Jenkins in Cloud
byIdeato
 
PDF
Effizientere WordPress-Plugin-Entwicklung mit Softwaretests
byDECK36
 
ODP
Docker for Developers - php[tek] 2017
byChris Tankersley
 
PDF
Puppet evolutions
byAlessandro Franceschi
 
2 Linux Container and Docker
byFabio Fumarola
 
Docker在豆瓣的实践 刘天伟-20160709
byTianwei Liu
 
Docker meetup - PaaS interoperability
byLudovic Piot
 
[네이버오픈소스세미나] Maglev Hashing Scheduler in IPVS, Linux Kernel - 송인주
byNAVER Engineering
 
Continuous Integration: SaaS vs Jenkins in Cloud
byIdeato
 
Effizientere WordPress-Plugin-Entwicklung mit Softwaretests
byDECK36
 
Docker for Developers - php[tek] 2017
byChris Tankersley
 
Puppet evolutions
byAlessandro Franceschi
 

What's hot

PDF
Hands on Docker - Launch your own LEMP or LAMP stack - SunshinePHP
byDana Luther
 
PDF
How to deploy PHP projects with docker
byRuoshi Ling
 
PPTX
Getting Started with Docker
byGeeta Vinnakota
 
ODP
Docker for PHP Developers - Madison PHP 2017
byChris Tankersley
 
PDF
開放運算&GPU技術研究班
byPaul Chao
 
PDF
NAVER Ceph Storage on ssd for Container
byJangseon Ryu
 
PDF
Virtual CD4PE Workshop
byPuppet
 
PDF
The Docker "Gauntlet" - Introduction, Ecosystem, Deployment, Orchestration
byErica Windisch
 
PDF
2012 09-08-josug-jeff
byZheng (Jeff) Xu
 
PPT
Develop with linux containers and docker
byFabio Fumarola
 
PDF
Chef & OpenStack: OSCON 2014
byMatt Ray
 
PPTX
Drupal in Libraries
byCary Gordon
 
PDF
On demand-block-storage-for-docker
byJangseon Ryu
 
PDF
Ansible 實戰：top down 觀點
byWilliam Yeh
 
PDF
SCALE12X: Chef for OpenStack
byMatt Ray
 
PDF
douban happyday docker for daeqaci
byTianwei Liu
 
PDF
Salt conf 2014-installing-openstack-using-saltstack-v02
byYazz Atlas
 
PPTX
Docker for PHP Developers - ZendCon 2016
byChris Tankersley
 
PDF
Supercharging Content Delivery with Varnish
bySamantha Quiñones
 
PDF
Configuring Django projects for multiple environments
byApptension
 
Hands on Docker - Launch your own LEMP or LAMP stack - SunshinePHP
byDana Luther
 
How to deploy PHP projects with docker
byRuoshi Ling
 
Getting Started with Docker
byGeeta Vinnakota
 
Docker for PHP Developers - Madison PHP 2017
byChris Tankersley
 
開放運算&GPU技術研究班
byPaul Chao
 
NAVER Ceph Storage on ssd for Container
byJangseon Ryu
 
Virtual CD4PE Workshop
byPuppet
 
The Docker "Gauntlet" - Introduction, Ecosystem, Deployment, Orchestration
byErica Windisch
 
2012 09-08-josug-jeff
byZheng (Jeff) Xu
 
Develop with linux containers and docker
byFabio Fumarola
 
Chef & OpenStack: OSCON 2014
byMatt Ray
 
Drupal in Libraries
byCary Gordon
 
On demand-block-storage-for-docker
byJangseon Ryu
 
Ansible 實戰：top down 觀點
byWilliam Yeh
 
SCALE12X: Chef for OpenStack
byMatt Ray
 
douban happyday docker for daeqaci
byTianwei Liu
 
Salt conf 2014-installing-openstack-using-saltstack-v02
byYazz Atlas
 
Docker for PHP Developers - ZendCon 2016
byChris Tankersley
 
Supercharging Content Delivery with Varnish
bySamantha Quiñones
 
Configuring Django projects for multiple environments
byApptension
 

Similar to Binary Packaging for HPC with Spack

PPTX
Spack - A Package Manager for HPC
byinside-BigData.com
 
PDF
SPACK: A Package Manager for Supercomputers, Linux, and MacOS
byinside-BigData.com
 
PPTX
spack_hpc.pptx
byObject Automation
 
PDF
Princeton RSE: Building Python Packages (+binary)
byHenry Schreiner
 
PDF
Package management and creation in Gentoo Linux
byDonnie Berkholz
 
PDF
Debian Packaging
byJavier Carranza
 
PDF
Python packaging and dependency resolution
byTatiana Al-Chueyr
 
PDF
Course 102: Lecture 22: Package Management
byAhmed El-Arabawy
 
PDF
Dependencies Managers in C/C++. Using stdcpp 2014
bybiicode
 
PDF
Package Management via Spack on SJTU π Supercomputer
byJianwen Wei
 
PDF
Packaging perl (LPW2010)
byp3castro
 
PDF
Reliable from-source builds (Qshare 28 Nov 2023).pdf
byRalf Gommers
 
PPTX
HPC_MPI_CICID_OA.pptx
byObjectAutomation2
 
PDF
$ make install
byMarios Isaakidis
 
ODP
Python-specific packaging
bydwvisser
 
PDF
Build and deploy scientific Python Applications
byRamakrishna Reddy
 
PDF
Effectively using Open Source with conda
byTravis Oliphant
 
ODP
Debian packaging talk, Pysheff sept 2012
bytakluyver
 
PPT
10 11-hart installing pythonsoftware
byWilliam Hart
 
PDF
Using Python Packages - An Overview
byDaniel Hepper
 
Spack - A Package Manager for HPC
byinside-BigData.com
 
SPACK: A Package Manager for Supercomputers, Linux, and MacOS
byinside-BigData.com
 
spack_hpc.pptx
byObject Automation
 
Princeton RSE: Building Python Packages (+binary)
byHenry Schreiner
 
Package management and creation in Gentoo Linux
byDonnie Berkholz
 
Debian Packaging
byJavier Carranza
 
Python packaging and dependency resolution
byTatiana Al-Chueyr
 
Course 102: Lecture 22: Package Management
byAhmed El-Arabawy
 
Dependencies Managers in C/C++. Using stdcpp 2014
bybiicode
 
Package Management via Spack on SJTU π Supercomputer
byJianwen Wei
 
Packaging perl (LPW2010)
byp3castro
 
Reliable from-source builds (Qshare 28 Nov 2023).pdf
byRalf Gommers
 
HPC_MPI_CICID_OA.pptx
byObjectAutomation2
 
$ make install
byMarios Isaakidis
 
Python-specific packaging
bydwvisser
 
Build and deploy scientific Python Applications
byRamakrishna Reddy
 
Effectively using Open Source with conda
byTravis Oliphant
 
Debian packaging talk, Pysheff sept 2012
bytakluyver
 
10 11-hart installing pythonsoftware
byWilliam Hart
 
Using Python Packages - An Overview
byDaniel Hepper
 

More from inside-BigData.com

PDF
CUDA-Python and RAPIDS for blazing fast scientific computing
byinside-BigData.com
 
PDF
How to Achieve High-Performance, Scalable and Distributed DNN Training on Mod...
byinside-BigData.com
 
PDF
State of ARM-based HPC
byinside-BigData.com
 
PDF
Introducing HPC with a Raspberry Pi Cluster
byinside-BigData.com
 
PDF
Versal Premium ACAP for Network and Cloud Acceleration
byinside-BigData.com
 
PDF
Overview of HPC Interconnects
byinside-BigData.com
 
PDF
HPC at Scale Enabled by DDN A3i and NVIDIA SuperPOD
byinside-BigData.com
 
PPTX
Transforming Private 5G Networks
byinside-BigData.com
 
PDF
Energy Efficient Computing using Dynamic Tuning
byinside-BigData.com
 
PDF
The Incorporation of Machine Learning into Scientific Simulations at Lawrence...
byinside-BigData.com
 
PDF
Machine Learning for Weather Forecasts
byinside-BigData.com
 
PDF
Major Market Shifts in IT
byinside-BigData.com
 
PPTX
HPC AI Advisory Council Update
byinside-BigData.com
 
PDF
Biohybrid Robotic Jellyfish for Future Applications in Ocean Monitoring
byinside-BigData.com
 
PDF
Fugaku Supercomputer joins fight against COVID-19
byinside-BigData.com
 
PDF
Preparing to program Aurora at Exascale - Early experiences and future direct...
byinside-BigData.com
 
PDF
HPC Impact: EDA Telemetry Neural Networks
byinside-BigData.com
 
PDF
Evolving Cyberinfrastructure, Democratizing Data, and Scaling AI to Catalyze ...
byinside-BigData.com
 
PDF
Zettar: Moving Massive Amounts of Data across Any Distance Efficiently
byinside-BigData.com
 
PDF
Scaling TCO in a Post Moore's Era
byinside-BigData.com
 
CUDA-Python and RAPIDS for blazing fast scientific computing
byinside-BigData.com
 
How to Achieve High-Performance, Scalable and Distributed DNN Training on Mod...
byinside-BigData.com
 
State of ARM-based HPC
byinside-BigData.com
 
Introducing HPC with a Raspberry Pi Cluster
byinside-BigData.com
 
Versal Premium ACAP for Network and Cloud Acceleration
byinside-BigData.com
 
Overview of HPC Interconnects
byinside-BigData.com
 
HPC at Scale Enabled by DDN A3i and NVIDIA SuperPOD
byinside-BigData.com
 
Transforming Private 5G Networks
byinside-BigData.com
 
Energy Efficient Computing using Dynamic Tuning
byinside-BigData.com
 
The Incorporation of Machine Learning into Scientific Simulations at Lawrence...
byinside-BigData.com
 
Machine Learning for Weather Forecasts
byinside-BigData.com
 
Major Market Shifts in IT
byinside-BigData.com
 
HPC AI Advisory Council Update
byinside-BigData.com
 
Biohybrid Robotic Jellyfish for Future Applications in Ocean Monitoring
byinside-BigData.com
 
Fugaku Supercomputer joins fight against COVID-19
byinside-BigData.com
 
Preparing to program Aurora at Exascale - Early experiences and future direct...
byinside-BigData.com
 
HPC Impact: EDA Telemetry Neural Networks
byinside-BigData.com
 
Evolving Cyberinfrastructure, Democratizing Data, and Scaling AI to Catalyze ...
byinside-BigData.com
 
Zettar: Moving Massive Amounts of Data across Any Distance Efficiently
byinside-BigData.com
 
Scaling TCO in a Post Moore's Era
byinside-BigData.com
 

Recently uploaded

PDF
One String, Many Prompts: AI Code Generation
byGeoffrey Goetz
 
PPTX
UNIT 1- Introduction to Basic of Computer Fundamentals
bypawarbhaktiit
 
PDF
UiPath Coded Agents - A Developer Driven Agentic Automation Era
byUiPathCommunity
 
PDF
Adding Copilot+ PCs with Snapdragon to your HP fleet won’t require IT deploym...
byPrincipled Technologies
 
PDF
Pervasive Encryption - Keeping Data Secure.pdf
byPrecisely
 
PDF
Mercury Computer Systems & The Cell Broadband Engine
bySlide_N
 
PDF
Implementing A Data Lakehouse Using Iceberg & Spark
byAnass Nabil
 
PDF
6 Insights from the Patron Saint of AI Failure
byScott M. Graffius
 
PDF
2006 IEEE International Solid-State Circuits Conference
bySlide_N
 
PPTX
Spacewalk 2026 - Presented at the IMAX in Raleigh, NC
byAll Things Open
 
PPTX
WIFI Enabled 8 by 32 Matrix.pptx for final year project
bymdsabbirhossain524856
 
PDF
TrustArc Webinar - Unpacking India’s DPDP Act: What You Should Know
byTrustArc
 
PDF
Special Reeling Cable Specification _ Anhui Feichun Special Cable Co., Ltd_.pdf
byAnhui Feichun Special Cable Co., Ltd.
 
PDF
Zniber Partners. Audience Foresight for Confident Decisions
bySamuel Zniber
 
PDF
TopMate ES11 3-Wheel Electric Scooter: Comfort, Stability, and Independence f...
byTopmate
 
PDF
Effective Ai powered mock interview .pdf
byamazingnishusingh766
 
PDF
Philippine Agricultural Engineering Standard_Hand Tractor .pdf
by6xfrnmdp4b
 
PDF
Intro to CrafterQ - AI Agent Platform for the Enterprise.pdf
byCrafterQ
 
PPTX
The Abomination of AI – keynote at ICoSCI 2026
byAlan Dix
 
PPT
connectives-linking words in English.ppt
byAmrMohammed82
 
One String, Many Prompts: AI Code Generation
byGeoffrey Goetz
 
UNIT 1- Introduction to Basic of Computer Fundamentals
bypawarbhaktiit
 
UiPath Coded Agents - A Developer Driven Agentic Automation Era
byUiPathCommunity
 
Adding Copilot+ PCs with Snapdragon to your HP fleet won’t require IT deploym...
byPrincipled Technologies
 
Pervasive Encryption - Keeping Data Secure.pdf
byPrecisely
 
Mercury Computer Systems & The Cell Broadband Engine
bySlide_N
 
Implementing A Data Lakehouse Using Iceberg & Spark
byAnass Nabil
 
6 Insights from the Patron Saint of AI Failure
byScott M. Graffius
 
2006 IEEE International Solid-State Circuits Conference
bySlide_N
 
Spacewalk 2026 - Presented at the IMAX in Raleigh, NC
byAll Things Open
 
WIFI Enabled 8 by 32 Matrix.pptx for final year project
bymdsabbirhossain524856
 
TrustArc Webinar - Unpacking India’s DPDP Act: What You Should Know
byTrustArc
 
Special Reeling Cable Specification _ Anhui Feichun Special Cable Co., Ltd_.pdf
byAnhui Feichun Special Cable Co., Ltd.
 
Zniber Partners. Audience Foresight for Confident Decisions
bySamuel Zniber
 
TopMate ES11 3-Wheel Electric Scooter: Comfort, Stability, and Independence f...
byTopmate
 
Effective Ai powered mock interview .pdf
byamazingnishusingh766
 
Philippine Agricultural Engineering Standard_Hand Tractor .pdf
by6xfrnmdp4b
 
Intro to CrafterQ - AI Agent Platform for the Enterprise.pdf
byCrafterQ
 
The Abomination of AI – keynote at ICoSCI 2026
byAlan Dix
 
connectives-linking words in English.ppt
byAmrMohammed82
 

Binary Packaging for HPC with Spack

  • 1.
    LLNL-PRES-745747 This work wasperformed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under contract DE-AC52-07NA27344. Lawrence Livermore National Security, LLC Binary packaging for HPC with Spack HPC, Big Data, and Data Science Devroom at FOSDEM 2018 Brussels, Belgium Todd Gamblin Center for Applied Scientific Computing LLNLFeburary 4, 2018
  • 2.
    LLNL-PRES-745747 2@spackpmgithub.com/spack Spack is ageneral purpose, from-source package manager § Inspired somewhat by homebrew and nix § Targets HPC and scientific computing — Community is growing! § Goals: — Facilitate experimenting with performance options — Flexibility. Make these things easy: • Build packages with many different: – compilers/versions/build options • Change compilers and flags in builds (keep provenance) • Swap implementations of ABI-incompatible libraries – MPI, BLAS, LAPACK, others like jpeg/jpeg-turbo, etc. — Build software stacks for scientific simulation and analysis — Run on laptops, Linux clusters, and some of the largest supercomputers in the world Spack https://spack.io
  • 3.
    LLNL-PRES-745747 3@spackpmgithub.com/spack Spec CLI syntaxmakes it easy to install different ways $ spack install mpileaks unconstrained $ spack install mpileaks@3.3 @ custom version $ spack install mpileaks@3.3 %gcc@4.7.3 % custom compiler $ spack install mpileaks@3.3 %gcc@4.7.3 +threads +/- build option $ spack install mpileaks@3.3 cflags="-O3 –g3" setting compiler flags $ spack install mpileaks@3.3 ^mpich@3.2 %gcc@4.9.3 ^ dependency constraints § Each expression is a spec for a particular configuration — Each clause adds a constraint to the spec — Constraints are optional – specify only what you need. — Customize install on the command line! § Spec syntax is recursive — ^ (caret) adds constraints on dependencies
  • 4.
    LLNL-PRES-745747 4@spackpmgithub.com/spack Spack packages aretemplates: they define how to build a spec from spack import * class Dyninst(Package): """API for dynamic binary instrumentation.""” homepage = "https://paradyn.org" url = "http://www.paradyn.org/release8.1.2/DyninstAPI-8.1.2.tgz" version('8.2.1', 'abf60b7faabe7a2e’) version('8.1.2', 'bf03b33375afa66f’) version('8.1.1', 'd1a04e995b7aa709’) depends_on("cmake", type="build") depends_on("libelf", type="link") depends_on("libdwarf", type="link") depends_on("boost @1.42: +multithreaded") def install(self, spec, prefix): with working_dir('spack-build', create=True): cmake('-DBoost_INCLUDE_DIR=‘ + spec['boost'].prefix.include, '-DBoost_LIBRARY_DIR=‘ + spec['boost'].prefix.lib, '-DBoost_NO_SYSTEM_PATHS=TRUE’ '..') make() make("install") Metadata at the class level Versions Install logic in instance methods Dependencies Patches, variants, resources, conflicts, etc. (not shown) Simple Python DSL — Packages are classes (ala homebrew) — Directives use the same spec syntax
  • 5.
    LLNL-PRES-745747 5@spackpmgithub.com/spack § mpi isa virtual dependency § Install the same package built with two different MPI implementations: § Virtual deps are replaced with a valid implementation at resolution time. — If the user didn’t pick something and there are multiple options, Spack picks. Depend on interfaces (not implementations) with virtual dependencies $ spack install mpileaks ^mvapich $ spack install mpileaks ^openmpi@1.4: mpileaks mpi callpath dyninst libdwarf libelf class Mpileaks(Package): depends_on("mpi@2:") class Mvapich(Package): provides("mpi@1” when="@:1.8") provides("mpi@2” when="@1.9:") class Openmpi(Package): provides("mpi@:2.2" when="@1.6.5:") Virtual dependencies can be versioned: dependent provider provider
  • 6.
    LLNL-PRES-745747 6@spackpmgithub.com/spack Spack builds packageswith compiler wrappers Spack Process Set up environment CC = spack/env/intel/icc SPACK_CC = /opt/ic-15.1/bin/icc CXX = spack/env/intel/icpc SPACK_CXX = /opt/ic-15.1/bin/icpc F77 = spack/env/intel/ifort SPACK_F77 = /opt/ic-15.1/bin/ifort FC = spack/env/intel/ifort SPACK_FC = /opt/ic-15.1/bin/ifort PKG_CONFIG_PATH = ... PATH = spack/env:$PATH CMAKE_PREFIX_PATH = ... LIBRARY_PATH = ... do_install() Install dep1 Install dep2 Install package… Build Process Fork install() configure make make install -I /dep1-prefix/include -L /dep1-prefix/lib -Wl,-rpath=/dep1-prefix/lib Compiler wrappers (spack-cc, spack-c++, spack-f77, spack-f90) icc icpc ifort ▪ Similar to homebrew “shims” ▪ Forked build process isolates environment for each build ▪ Use compiler wrappers to add include, lib, and RPATH flags ▪ RPATHs ensure that the correct dependencies are found automatically at runtime.
  • 7.
    LLNL-PRES-745747 7@spackpmgithub.com/spack § Each uniquedependency graph is a unique configuration. § Each configuration installed in a unique directory. — Configurations of the same package can coexist. § Hash of directed acyclic graph (DAG) metadata is appended to each prefix — Note: we hash the metadata, not the artifact. § Installed packages automatically find dependencies — Spack embeds RPATHs in binaries. — No need to set LD_LIBRARY_PATH — Things work the way you built them Hashes handle combinatorial software complexity. spack/opt/ linux-rhel7-x86_64/ gcc-4.7.2/ mpileaks-1.1-0f54bf34cadk/ intel-14.1/ hdf5-1.8.15-lkf14aq3nqiz/ bgq/ xl-12.1/ hdf5-1-8.16-fqb3a15abrwx/ ... mpileaks mpi callpath dyninst libdwarf libelf Installation Layout Dependency DAG Hash
  • 8.
    LLNL-PRES-745747 8@spackpmgithub.com/spack Spack’s dependency modelcenters around “concretization” mpileaks ^callpath@1.0+debug ^libelf@0.8.11 User input: abstract spec Concrete spec is fully constrained and can be built. mpileaks@2.3 %gcc@4.7.3 =linux-ppc64 mpich@3.0.4 %gcc@4.7.3 =linux-ppc64 callpath@1.0 %gcc@a4.7.3+debug =linux-ppc64 dyninst@8.1.2 %gcc@4.7.3 =linux-ppc64 libelf@0.8.11 %gcc@4.7.3 =linux-ppc64 libdwarf@20130729 %gcc@4.7.3 =linux-ppc64 Abstract, normalized spec with some dependencies. mpileaks mpi callpath@1.0 +debug dyninst libelf@0.8.11 libdwarf Concretize § Similar to other dependency resolvers, but solves for more than just package and version. § Full spec is stored in a file in the installation directory — Can reinstall same build with: spack install –f spec.yaml Store spec: - mpileaks: arch: linux-x86_64 compiler: name: gcc version: 4.9.2 dependencies: adept-utils: kszrtkpbzac3ss2ixcjkcorlaybnptp4 callpath: bah5f4h4d2n47mgycej2mtrnrivvxy77 mpich: aa4ar6ifj23yijqmdabeakpejcli72t3 hash: 33hjjhxi7p6gyzn5ptgyes7sghyprujh variants: {} version: '1.0' - adept-utils: arch: linux-x86_64 compiler: name: gcc version: 4.9.2 dependencies: boost: teesjv7ehpe5ksspjim5dk43a7qnowlq mpich: aa4ar6ifj23yijqmdabeakpejcli72t3 hash: kszrtkpbzac3ss2ixcjkcorlaybnptp4 variants: {} version: 1.0.1 ... spec.yaml Detailed provenance is stored with the installed package
  • 9.
    LLNL-PRES-745747 9@spackpmgithub.com/spack Source installs aregreat, but they’re slow § Most people prefer using a binary package manager — Binary packages typically use portable code — Binary installs are typically a lot slower than what you get from building from source § We’d like to have the best of both worlds: — Optimized buids for specific machine models (skylake, haswell, ivy bridge, etc.) — Binary packages available without having to build from source § What’s needed? 1. Binary packaging capability 2. Metadata describing architecture-specific builds 3. Good dependency resolution to select optimized or generic versions of packages
  • 10.
    LLNL-PRES-745747 10@spackpmgithub.com/spack We recently releasedSpack v0.11 § 2,178 packages (up from 1,114 a year ago) § Big features for users: — Relocatable binary packages (spack buildcache) — Full support for Python 3 — Improved module support; custom module templates using jinja2 § Many improvements for packagers: — Multi-valued variants — Test dependency type — Packages can patch their dependencies (not just themselves) § Many speed improvements (to Spack itself) https://github.com/spack/spack/releases/
  • 11.
    LLNL-PRES-745747 11@spackpmgithub.com/spack Binary packaging inSpack v0.11 § Spack v0.11 has a new spack buildcache command: spack buildcache create <spec> # create a new binary package spack buildcache list # list available binaries spack buildcache install # install a binary package (specifically) § Typically, install is not needed; you can just do: spack install --use-cache # prefer binaries if available § We don’t enable binaries by default yet — We’ll make –use-cache default when we start hosting stable binaries § Thanks to our collaboration with Fermilab, CERN, and Kitware for this feature!
  • 12.
    LLNL-PRES-745747 12@spackpmgithub.com/spack How to makea binary 1. Set up GNU PG for binary signing — Unsigned binaries can be created but are discouraged spack gpg create "Todd Gamblin" tgamblin@gmail.com # create a new signing keypair spack gpg init # trust initial keypair 2. Install something spack install m4 # install m4 ... spack find ==> 2 installed packages. -- darwin-elcapitan-x86_64 / clang@8.0.0-apple ------------------ libsigsegv@2.11 m4@1.4.18 3. Run spack buildcache create on that thing spack buildcache create -d /path/to/mirror m4 # create a binary package in mirror § Binaries and metadata for the package and its dependencies will be in: /path/to/mirror/build_cache/
  • 13.
    LLNL-PRES-745747 13@spackpmgithub.com/spack Binary mirror structure §Binaries go in build_cache/<platform>/<compiler>/<pkg-version> § Metadata for all packages is (currently) kept at the top level — We’ll need to index these files eventually § build_cache subdirectory sits inside of a Spack mirror directory — Makes it easy to add binaries to an existing source mirror § This structure is very easy to host in something like S3, a web server, or a shared filesystem mirror/ !""build_cache #""darwin-elcapitan-x86_64 $ !""clang-8.0.0-apple $ #""libsigsegv-2.11 $ $ !"" darwin-elcapitan-x86_64-clang-8.0.0-apple-libsigsegv-2.11-5rjv56uui3crfmsgsqgqab52yfhr6t4b.spack $ !""m4-1.4.18 $ !"" darwin-elcapitan-x86_64-clang-8.0.0-apple-m4-1.4.18-eqbzieloqiwxfe4drksmekvfqia7mqbu.spack #"" darwin-elcapitan-x86_64-clang-8.0.0-apple-libsigsegv-2.11-5rjv56uui3crfmsgsqgqab52yfhr6t4b.spec.yaml #"" darwin-elcapitan-x86_64-clang-8.0.0-apple-m4-1.4.18-eqbzieloqiwxfe4drksmekvfqia7mqbu.spec.yaml !"" index.html
  • 14.
    LLNL-PRES-745747 14@spackpmgithub.com/spack Pointing Spack toa mirror § Spack v0.11 has a new spack buildcache command: $ spack mirror add mypkgs https://example.spack-mirror.com/mirror § You can verify that it worked by looking at what mirrors are configured: $ spack mirror list mypkgs https://example.spack-mirror.com/mirror § Mirrors can contain source tarballs and binaries — Detailed info in docs on mirrors.yaml
  • 15.
    LLNL-PRES-745747 15@spackpmgithub.com/spack How fetching worksin spack Concretize fetch source code build load mpileaks/package.py from repo fetch from user mirrors from var/spack/cache (local mirror) fetch from package URL install install relocate configureBinary available? Yes No Install from binary Build from source verify signature verify checksum ... spack install mpileaks
  • 16.
    LLNL-PRES-745747 16@spackpmgithub.com/spack What’s in aSpack binary package? § The binary is just a tarball § Contains: 1. Another tarball of the installed prefix 2. The spec.yaml: • describes the build (Spack metadata) • Contains a special entry with the checksum of the source tarball (maps spack hash to SHA256) 3. A signature • tells us we can trust the spec.yaml $ tar tzf darwin-elcapitan-x86_64-clang-8.0.0-apple-m4-1.4.18-eqbzieloqiwxfe4drksmekvfqia7mqbu.spack darwin-elcapitan-x86_64-clang-8.0.0-apple-m4-1.4.18-eqbzieloqiwxfe4drksmekvfqia7mqbu.tar.gz darwin-elcapitan-x86_64-clang-8.0.0-apple-m4-1.4.18-eqbzieloqiwxfe4drksmekvfqia7mqbu.spec.yaml darwin-elcapitan-x86_64-clang-8.0.0-apple-m4-1.4.18-eqbzieloqiwxfe4drksmekvfqia7mqbu.spec.yaml.asc
  • 17.
    LLNL-PRES-745747 17@spackpmgithub.com/spack Why do wechecksum source but sign binaries? § Other systems provide checksums for sources and binaries in their package files — e.g., homebrew “bottles” § In Spack, the number of binaries associated with a source tarball can be very large! — We could have thousands of binaries for the same source: • Different flags, different build options etc. — Each of these would have a different Spack § Putting checksums for all of these in the package files: — Would add a lot of extra bytes to a package repository — Is unmaintainable — Means that we have to update package.py files whenever we update a mirror § With signing, the client can trust one or several keys and verify a large number of packages with a small number of public keys
  • 18.
    LLNL-PRES-745747 18@spackpmgithub.com/spack What’s relocation? § WhenSpack creates a binary package, it traverses the installation directory and examines the files — Uses the file command § It records the files that need to be relocated after installation: — Libraries with RPATHs — Shell scripts with #! Lines § After installation, Spack: — rewrites RPATHs with patchelf (Linux) or install_name_tool (macOS) — Rewrites #! lines to point to the Spack installation on the installing machine § Install is faster b/c we record the needed relocations at package creation time spack/opt/ linux-rhel7-x86_64/ gcc-4.7.2/ mpileaks-1.1-0f54bf34cadk/ bin/ mpileaks-run lib/ libmpileaks.so Installation Layout § We try to make root-relative RPATHs when possible, but don’t always get everything. § We also rewrite RPATHs to directories within the spack root fi the install machine uses a different layout. § We are not currently relocating compiler runtime paths — We should. This is work in progress
  • 19.
    LLNL-PRES-745747 19@spackpmgithub.com/spack How do wedecide which binaries to fetch? § We currently only fetch binaries if they satisfy the exact hash result of concretization § This doesn’t leave a lot of room for change in the system — Small changes in Spack mean having to build from source again — Works best on a stable release § We are working on a new concretizer that will consider available binary specs — Doing this better requires a backtracking SAT solve mpileaks@2.3 %gcc@4.7.3 =linux-ppc64 mpich@3.0.4 %gcc@4.7.3 =linux-ppc64 callpath@1.0 %gcc@a4.7.3+debug =linux-ppc64 dyninst@8.1.2 %gcc@4.7.3 =linux-ppc64 libelf@0.8.11 %gcc@4.7.3 =linux-ppc64 libdwarf@20130729 %gcc@4.7.3 =linux-ppc64 mpileaks mpi callpath@1.0 +debug dyninst libelf@0.8.11 libdwarf Concretize Current In progress Fetch exact hashes, if available mpileaks@2.3 %gcc@4.7.3 =linux-ppc64 mpich@3.0.4 %gcc@4.7.3 =linux-ppc64 callpath@1.0 %gcc@a4.7.3+debug =linux-ppc64 dyninst@8.1.2 %gcc@4.7.3 =linux-ppc64 libelf@0.8.11 %gcc@4.7.3 =linux-ppc64 libdwarf@20130729 %gcc@4.7.3 =linux-ppc64 mpileaks mpi callpath@1.0 +debug dyninst libelf@0.8.11 libdwarf Concretize Prefer available binaries in concretization Download available specs from mirror
  • 20.
    LLNL-PRES-745747 20@spackpmgithub.com/spack Spack can shipoptimized binaries § The Spack architecture descriptor currently includes: — Platform: cray, mac, linux, bgq • Meant to represent a family of machines with potentially many OS/target combinations — OS: rhel6, rhel7, ubuntu14, elcapitan, sierra, centos6, centos7, etc. — Target: • Generic: x86_64, ppc64le, etc. • Specific: haswell, ivybridge, knl, power8, power9, etc. § Some triples: — darwin-sierra-x86_64 — darwin-elcapitan-x86_64 — cray-cnl6-knl — cray-cnl6-haswell § These architecture descriptors are part of the binary metadata — If we can fetch an index of available packages first, we can be picky about what binaries we want — This is a core spec parameter in Spack, not just a naming convention for packages
  • 21.
    LLNL-PRES-745747 21@spackpmgithub.com/spack Detecting optimized architectures §Currently Spack will only use an optimized architecture descriptor on Cray — We get the architecture name from the Cray Programming Environment — We can know whether we’re building for Haswell, Broadwell, KNL, etc. § We have work in progress that detects these names for Intel, AMD, Power, and ARM hardware (looking at available info in /proc/cpuinfo, etc.) — We’re planning to shift to a model where we use the specific descriptor by default — We would still allow a user to set preferences to build generic if they want — Important for CERN and Fermi collaborators who run heterogeneous clusters § Once this is done, we do plan to make arch-specific binaries available.
  • 22.
    LLNL-PRES-745747 22@spackpmgithub.com/spack Some issues withoptimized binaries § Architectures like ARM don’t lend themselves to concise descriptors. — We may need to be more fine-grained here § We may need to add a more fine-grained architecture descriptor that just exposes the instruction sets available on the machine — E.g., instead of “haswell”, put “sse4.2, avx, avx2, etc.” — These attributes may actually be easier for packagers and maintainers to use § We need a setting on the user side (e.g. in packages.yaml) that lets them choose what the minimum architecture is for compatibility — This seems easier to do with well known system names
  • 23.
    LLNL-PRES-745747 23@spackpmgithub.com/spack Tuning for optimizedbinaries may be tricky § Most compilers give you two knobs to control architecture-specific tuning: -march=[generic|native] -mtune=[generic|native] — If you tune generic for an older architecture, it will run fine on that architecture and on newer architectures. — If you tune native, you’ll get code that only runs fast on the specific architecture, and may not perform as well on future chips. § In Spack, we might want policies like this: — “Generic tuning, with code no later than sandy bridge”, e.g. if sandy bridge is the older architecture on a heterogeneous cluster — “Native tuning, just for this machine”, e.g., if we know we want to optimize for a long-lived, homogeneous cluster.
  • 24.
    LLNL-PRES-745747 24@spackpmgithub.com/spack Building a primarybinary mirror for Spack § We’re currently setting up build automation to create binaries for: — All default package configurations in releases (result of spack install <name>) — Selected other slices of those configuration spaces, e.g.: • x MPI versions • x Compilers • x OS’s • x large-scale DOE machines (Cori, Theta, Titan, Summit, etc.) § Once this is done, we’ll also continuously build packages for the develop branch as PRs come in — We’ll need to determine when to purge old builds and binaries — Depends on analytics § We are currently planning to host binaries in S3 — but Jfrog/bintray sounds interesting. Maybe we should talk to them.
  • 25.
    LLNL-PRES-745747 25@spackpmgithub.com/spack Summary We built relocatablebinary packaging into Spack Current projects: 1. Binary build infrastructure 2. Better concretization to support optimized binaries 3. Compiler library relocation Shooting for September to have all of this done. Spack Come and get Spack stickers! https://spack.io