Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Deploying NPM packages with the Nix package manager

2,284 views

Published on

Distributions devroom presentation given @ FOSDEM 2017. Describes how an NPM workflow can be integrated into the vision of Nix deployment.

Published in: Software
  • Be the first to comment

Deploying NPM packages with the Nix package manager

  1. 1. Deploying NPM packages with the Nix package manager Sander van der Burg Feb 4, 2017 Sander van der Burg Deploying NPM packages with the Nix package manager
  2. 2. The Nix project Family of declarative software deployment tools: Nix. A purely functional package manager NixOS. Nix based GNU/Linux distribution Hydra. Nix based continuous build and integration server Disnix. Nix based distributed service deployment NixOps. NixOS-based multi-cloud deployment tool Sander van der Burg Deploying NPM packages with the Nix package manager
  3. 3. The Nix project Non-functional properties: Generic. Can be used with many programming languages, component technologies, and operating systems. Reproducible. (Almost) no impurities – if inputs are the same, result should be the same. Reliable. Dependency completeness, (almost) atomic upgrades and rollbacks. Efficient. Only the required deployment activities are executed. Sander van der Burg Deploying NPM packages with the Nix package manager
  4. 4. NixOS configuration /etc/nixos/configuration.nix {pkgs, ...}: { boot.loader.grub.device = "/dev/sda"; fileSystems = [ { mountPoint = "/"; device = "/dev/sda2"; } ]; swapDevices = [ { device = "/dev/sda1"; } ]; services = { openssh.enable = true; xserver = { enable = true; desktopManager.kde4.enable = true; }; }; environment.systemPackages = [ pkgs.mc pkgs.firefox ]; } Sander van der Burg Deploying NPM packages with the Nix package manager
  5. 5. NixOS configuration nixos-rebuild switch Nix package manager builds a complete system configuration Includes all packages and generates all configuration files, e.g. OpenSSH configuration Upgrades are (almost) atomic Components are stored safely next to each other, due to hashes No files are automatically removed or overwritten Users can switch to older generations of system configurations not garbage collected yet Sander van der Burg Deploying NPM packages with the Nix package manager
  6. 6. NixOS bootloader Sander van der Burg Deploying NPM packages with the Nix package manager
  7. 7. The Nix project Basis of all tools: The Nix package manager Sander van der Burg Deploying NPM packages with the Nix package manager
  8. 8. Nix store Main idea: store all packages in isolation from each other: /nix/store/rpdqxnilb0cg... -firefox-3.5.4 Paths contain a 160-bit cryptographic hash of all inputs used to build the package: Sources Libraries Compilers Build scripts . . . /nix/store l9w6773m1msy...-openssh-4.6p1 bin ssh sbin sshd smkabrbibqv7...-openssl-0.9.8e lib libssl.so.0.9.8 c6jbqm2mc0a7...-zlib-1.2.3 lib libz.so.1.2.3 im276akmsrhv...-glibc-2.5 lib libc.so.6 Sander van der Burg Deploying NPM packages with the Nix package manager
  9. 9. Nix expressions openssh.nix { stdenv, fetchurl, openssl, zlib }: stdenv.mkDerivation { name = "openssh-4.6p1"; src = fetchurl { url = http://.../openssh-4.6p1.tar.gz; sha256 = "0fpjlr3bfind0y94bk442x2p..."; }; buildCommand = ’’ tar xjf $src ./configure --prefix=$out --with-openssl=${openssl} make; make install ’’; } Sander van der Burg Deploying NPM packages with the Nix package manager
  10. 10. Nix expressions all-packages.nix openssh = import ../tools/networking/openssh { inherit fetchurl stdenv openssl zlib; }; openssl = import ../development/libraries/openssl { inherit fetchurl stdenv perl; }; stdenv = ...; openssl = ...; zlib = ...; perl = ...; nix-env -f all-packages.nix -iA openssh Produces a /nix/store/l9w6773m1msy...-openssh-4.6p1 package in the Nix store. Sander van der Burg Deploying NPM packages with the Nix package manager
  11. 11. User environments Users can have different sets of installed applications. PATH /nix/.../profiles current 42 /nix/store pp56i0a01si5...-user-env bin firefox ssh l9w6773m1msy...-openssh-4.6p1 bin ssh rpdqxnilb0cg...-firefox-3.5.4 bin firefox Sander van der Burg Deploying NPM packages with the Nix package manager
  12. 12. User environments Users can have different sets of installed applications. nix-env operations create new user environments in the store. PATH /nix/.../profiles current 42 /nix/store pp56i0a01si5...-user-env bin firefox ssh l9w6773m1msy...-openssh-4.6p1 bin ssh rpdqxnilb0cg...-firefox-3.5.4 bin firefox aqn3wygq9jzk...-openssh-5.2p1 bin ssh (nix-env -u openssh) Sander van der Burg Deploying NPM packages with the Nix package manager
  13. 13. User environments Users can have different sets of installed applications. nix-env operations create new user environments in the store. PATH /nix/.../profiles current 42 /nix/store pp56i0a01si5...-user-env bin firefox ssh l9w6773m1msy...-openssh-4.6p1 bin ssh rpdqxnilb0cg...-firefox-3.5.4 bin firefox aqn3wygq9jzk...-openssh-5.2p1 bin ssh i3d9vh6d8ip1...-user-env bin ssh firefox (nix-env -u openssh) Sander van der Burg Deploying NPM packages with the Nix package manager
  14. 14. User environments Users can have different sets of installed applications. nix-env operations create new user environments in the store. PATH /nix/.../profiles current 42 43 /nix/store pp56i0a01si5...-user-env bin firefox ssh l9w6773m1msy...-openssh-4.6p1 bin ssh rpdqxnilb0cg...-firefox-3.5.4 bin firefox aqn3wygq9jzk...-openssh-5.2p1 bin ssh i3d9vh6d8ip1...-user-env bin ssh firefox (nix-env -u openssh) Sander van der Burg Deploying NPM packages with the Nix package manager
  15. 15. User environments Users can have different sets of installed applications. nix-env operations create new user environments in the store. We can atomically switch between them. PATH /nix/.../profiles current 42 43 /nix/store pp56i0a01si5...-user-env bin firefox ssh l9w6773m1msy...-openssh-4.6p1 bin ssh rpdqxnilb0cg...-firefox-3.5.4 bin firefox aqn3wygq9jzk...-openssh-5.2p1 bin ssh i3d9vh6d8ip1...-user-env bin ssh firefox (nix-env -u openssh) Sander van der Burg Deploying NPM packages with the Nix package manager
  16. 16. User environments Users can have different sets of installed applications. nix-env operations create new user environments in the store. We can atomically switch between them. These are roots of the garbage collector. PATH /nix/.../profiles current 43 /nix/store pp56i0a01si5...-user-env bin firefox ssh l9w6773m1msy...-openssh-4.6p1 bin ssh rpdqxnilb0cg...-firefox-3.5.4 bin firefox aqn3wygq9jzk...-openssh-5.2p1 bin ssh i3d9vh6d8ip1...-user-env bin ssh firefox (nix-env --remove-generations old) Sander van der Burg Deploying NPM packages with the Nix package manager
  17. 17. User environments Users can have different sets of installed applications. nix-env operations create new user environments in the store. We can atomically switch between them. These are roots of the garbage collector. PATH /nix/.../profiles current 43 /nix/store rpdqxnilb0cg...-firefox-3.5.4 bin firefox aqn3wygq9jzk...-openssh-5.2p1 bin ssh i3d9vh6d8ip1...-user-env bin ssh firefox (nix-collect-garbage) Sander van der Burg Deploying NPM packages with the Nix package manager
  18. 18. Nix expressions openssh.nix { stdenv, fetchurl, openssl, zlib }: stdenv.mkDerivation { name = "openssh-4.6p1"; src = fetchurl { url = http://.../openssh-4.6p1.tar.gz; sha256 = "0fpjlr3bfind0y94bk442x2p..."; }; buildCommand = ’’ tar xjf $src ./configure --prefix=$out --with-openssl=${openssl} make; make install ’’; } Nix complements existing build tools with a pure environment You can also run other build tools in an environment, e.g. cmake, ant, scons Sander van der Burg Deploying NPM packages with the Nix package manager
  19. 19. Pure environments Precautions in service of making build results bit-identical (regardless on what machine the package has been built): Clearing (most) environment variables or setting them to dummy values Modifying search environment variables to contain Nix store paths to only specified dependencies (e.g. PATH, PYTHONPATH, CLASSPATH, PERL5LIB) Using designated temp folders and output folders Making packages immutable by making their files read-only Resetting timestamps to 1 Chroot environments/namespaces Restricting network access Sander van der Burg Deploying NPM packages with the Nix package manager
  20. 20. What about Node.js/NPM projects? NPM is Node.js’ ubiquitous deployment tool. NPM also does dependency management in addition to build management NPM’s dependency management conflicts with Nix’s dependency management Sander van der Burg Deploying NPM packages with the Nix package manager
  21. 21. package.json configuration { "name" :"nijs", "version" :"0.0.23", "description" :"An internal DSL for the Nix package manager in JavaScript", "repository" :{ "type" :"git", "url" :"https://github.com/svanderburg/nijs.git" }, "author" :"Sander van der Burg", "license" :"MIT", "main" :"./lib/nijs", "dependencies" :{ "optparse" :">= 1.0.3", "slasp": "0.0.4" } } $ npm install $ ls node modules/ optparse slasp Sander van der Burg Deploying NPM packages with the Nix package manager
  22. 22. Dealing with a conflicting dependency manager Solution: substitute the conflicting dependency manager by providing the dependencies with Nix first! Sander van der Burg Deploying NPM packages with the Nix package manager
  23. 23. Solution: substitute NPM’s dependency management { stdenv, fetchgit, nodejs, optparse, slasp }: let nodeSources = ... in stdenv.mkDerivation { name = "nijs-0.0.23"; src = fetchgit { src = https://github.com/svanderburg/nijs.git; rev = "..."; sha256 = "..."; }; buildInputs = [ nodejs ]; buildCommand = ’’ # Compose node_modules/ folder from the dependencies ... # Perform the build by running npm install # Since a node_modules/ folder with the dependencies already exists, # NPM will not obtain them npm --registry http://www.example.com --nodedir=${nodeSources} install ’’; } Sander van der Burg Deploying NPM packages with the Nix package manager
  24. 24. Solution: substitute NPM’s dependency management { stdenv, fetchgit, nodejs, optparse, slasp }: let nodeSources = ... in stdenv.mkDerivation { name = "nijs-0.0.23"; src = fetchgit { src = https://github.com/svanderburg/nijs.git; rev = "..."; sha256 = "..."; }; buildInputs = [ nodejs ]; buildCommand = ’’ # Compose node_modules/ folder from the dependencies ... # Perform the build by running npm install # Since a node_modules/ folder with the dependencies already exists, # NPM will not obtain them npm --registry http://www.example.com --nodedir=${nodeSources} install ’’; } Sander van der Burg Deploying NPM packages with the Nix package manager Substituting dependencies We may be able to automatically generate a Nix expression from a package.json file since they capture similar proper- ties!
  25. 25. Solution: substitute NPM’s dependency management { stdenv, fetchgit, nodejs, optparse, slasp }: let nodeSources = ... in stdenv.mkDerivation { name = "nijs-0.0.23"; src = fetchgit { src = https://github.com/svanderburg/nijs.git; rev = "..."; sha256 = "..."; };; buildInputs = [ nodejs ]; buildCommand = ’’ # Compose node_modules/ folder from the dependencies ... # Perform the build by running npm install # Since a node_modules/ folder with the dependencies already exists, # NPM will not obtain them npm --registry http://www.example.com --nodedir=${nodeSources} install ’’; } Sander van der Burg Deploying NPM packages with the Nix package manager Substituting dependencies Appears to be straight forward! Is it really that straight forward?
  26. 26. Generating Nix expressions It is actually much more complicated! Sander van der Burg Deploying NPM packages with the Nix package manager
  27. 27. NPM dependency classes Dependencies: Run-time dependencies of a package Development dependencies: Build-time dependencies, e.g. transpilers Should only be installed if they have been requested Peer dependencies: Check whether a shared dependency does not conflict. In old versions of NPM, missing peer dependencies were also installed. Bundled dependencies: Statically bundled with a package. No need to install. Optional dependencies: Dependencies that are allowed to break. Typically used to bundle platform-specific code. Permitting errors (especially non-deterministic ones) incompatible with Nix deployment model Sander van der Burg Deploying NPM packages with the Nix package manager
  28. 28. NPM dependency classes Dependencies: Run-time dependencies of a package Development dependencies: Build-time dependencies, e.g. transpilers Should only be installed if they have been requested Peer dependencies: Check whether a shared dependency does not conflict. In old versions of NPM, missing peer dependencies were also installed. Bundled dependencies: Statically bundled with a package. No need to install. Optional dependencies: Dependencies that are allowed to break. Typically used to bundle platform-specific code. Permitting errors (especially non-deterministic ones) incompatible with Nix deployment model Sander van der Burg Deploying NPM packages with the Nix package manager Interpreting dependency classes We only need to consider dependencies and development de- pendencies (if they are requested) and substract the bundled dependencies
  29. 29. Version specifiers: NPM registry NPM version specifiers are nominal and semantically versioned: Precise version numbers: e.g. 1.0.0, 2.1.1 Version ranges and wildcards: e.g. 1.0.x, *, >= 1.0.1 Tags: latest, beta The above version specifiers refer to packages obtained from the NPM registry. Sander van der Burg Deploying NPM packages with the Nix package manager
  30. 30. Version specifiers: external packages Local filesystem paths, e.g. /home/sander/nijs External URLs, e.g. http://example.com/packages/nijs-0.0.23.tgz Git URLs, e.g. https://github.com/svanderburg/nijs.git GitHub, BitBucket, GitLab, e.g. github:svanderburg/nijs Above specifiers obtain packages from other sources than the registry. They indirectly resolve to a package with a name and version number. Sander van der Burg Deploying NPM packages with the Nix package manager
  31. 31. Version specifiers: translation to Nix Nix version specifiers are exact (any point of variation is reflected in the hash prefix): /nix/store/rpdqxnilb0cg...-firefox-3.5.4 Sander van der Burg Deploying NPM packages with the Nix package manager
  32. 32. Version specifiers: translation to Nix Nix version specifiers are exact (any point of variation is reflected in the hash prefix): /nix/store/rpdqxnilb0cg...-firefox-3.5.4 To make a translation from NPM version specifiers to exact version specifiers: We must snapshot a version range and pinpoint the resolved version (version ranges are unsupported in Nix) We must supply the output hash to make it deterministic: src = fetchurl { url = "https://registry.npmjs.org/nijs/-/nijs-0.0.23.tgz"; sha1 = "dbf8f4a0acafbe3b8d9b71c24cbd1d851de6c31a"; }; Sander van der Burg Deploying NPM packages with the Nix package manager
  33. 33. Private, shared and cyclic dependencies When including a package as a CommonJS module: ./node2nix/node modules/nijs/lib/execute/index.js var slasp = require(’slasp’); The module loader searches recursively upwards in node modules/ sub folders for the corresponding package: ./node2nix/nijs/node_modules/slasp ./node2nix/node_modules/slasp ./node_modules/slasp Sander van der Burg Deploying NPM packages with the Nix package manager
  34. 34. Private, shared and cyclic dependencies Private dependency: a package residing in a package’s node modules/ sub folder. Shared dependency: a package residing in a parent directory’s node modules/ sub folder. Sander van der Burg Deploying NPM packages with the Nix package manager
  35. 35. Private, shared and cyclic dependencies When installing dependencies with NPM, NPM will not install a dependency privately if a conforming shared dependency exists. Some undesired implications: A version range does not always yield the latest version that fits in a version range. Cyclic dependencies are permitted. Bad practice, as packages are supposed to be units of reuse. Sander van der Burg Deploying NPM packages with the Nix package manager
  36. 36. Flat-module installations npm < 3.x installs dependencies privately by default (unless a conforming dependency exists in any parent node modules/ folder): webapp/ ... package.json node_modules/ express/ ... package.json node_modules/ accepts/ array-flatten/ content-disposition/ ... ejs/ ... package.json Sander van der Burg Deploying NPM packages with the Nix package manager
  37. 37. Flat-module installations npm ≥ 3.x implements flat-module installations (moving packages as high as possible in the node modules/ hierarchy): webapp/ ... package.json node_modules/ accepts/ array-flatten/ content-disposition/ express/ ... package.json ejs/ ... package.json Because no packages conflict, they all appear in the top level node modules/ folder. Implications: Shorter path names, some degree of deduplication The order in which packages are installed matters Sander van der Burg Deploying NPM packages with the Nix package manager
  38. 38. Simulating flat-module installations We cannot build each NPM dependency as a Nix package, with a separate store path: CommonJS module resolves its own symlink location. Some relative paths may not work: var slasp = require(’../slasp’); Flattening the module hierarchy is imperative. Nix packages are made immutable after they have been built. Solution: compose the entire dependency tree statically ahead of time in one Nix package Sander van der Burg Deploying NPM packages with the Nix package manager
  39. 39. Replacing impure version specifiers Some version specifiers (e.g. tags and Git URLs), trigger NPM to always check the remote locations for changes in each npm install run: Replace these version specifiers by: ’*’. Downside: it sometimes confuses NPM, in particular flat module installations. Better solution: create fake processes (e.g git) giving a deterministic result. This is still an open issue. Sander van der Burg Deploying NPM packages with the Nix package manager
  40. 40. The solution1 : node2nix 1 Sort of, but not entirely :-) Sander van der Burg Deploying NPM packages with the Nix package manager
  41. 41. Example: using node2nix on project-level Generating Nix expressions from a package.json file: $ node2nix Building the project as a Nix package: $ nix-build -A package ./result/bin/node2nix --help Opening a development shell: $ nix-shell -A shell $ node bin/node2nix.js --help Sander van der Burg Deploying NPM packages with the Nix package manager
  42. 42. Example: using node2nix on a package set custom-packages.json [ "node2nix" ,"bower" ,{"nijs": "https://github.com/svanderburg/nijs.git" } ,{"grunt-cli": "1.2.0" } ] Generating Nix expressions: $ node2nix -i custom-packages.json Installing an NPM package with Nix: $ nix-env -f default.nix -iA node2nix $ node2nix --help Sander van der Burg Deploying NPM packages with the Nix package manager
  43. 43. Example: simulating global dependencies Global packages do not exist in Nix build environments. This is problematic for certain kinds of projects, such as Grunt projects, requiring the grunt-cli to be installed globally: package.json { "name": "grunt-test", "version": "0.0.1", "private": "true", "devDependencies": { "grunt": "*", "grunt-contrib-jshint": "*", "grunt-contrib-watch": "*" } } $ npm install $ grunt build Sander van der Burg Deploying NPM packages with the Nix package manager
  44. 44. Example: simulating global dependencies We can supply global packages as extra packages: supplement.json [ "grunt-cli" ] And generate the expressions as follows: $ node2nix -d -i package.json --supplement-input supplement.json Sander van der Burg Deploying NPM packages with the Nix package manager
  45. 45. Example: simulating global dependencies override.nix { pkgs ? import <nixpkgs> {} , system ? builtins.currentSystem }: let nodePackages = import ./default.nix { inherit pkgs system; }; in nodePackages // { package = nodePackages.package.override { postInstall = "grunt"; }; } $ nix-build override.nix Sander van der Burg Deploying NPM packages with the Nix package manager
  46. 46. Conclusion I have explained Nix and NPM’s deployment concepts I have described node2nix generating Nix expressions from package.json configurations, making it possible to deploy NPM packages with the Nix package manager. Besides the package manager, it also becomes possible to deploy entire systems using Node.js with NixOps and Disnix. Sander van der Burg Deploying NPM packages with the Nix package manager
  47. 47. Lessons Naming packages: a name and version number typically does not suffice! Nix uses hash codes derived from all build inputs .NET global assembly cache: strong names Organization: isolate dependencies Cyclic dependencies: disallow them – packages are supposed to be units of reuse Sander van der Burg Deploying NPM packages with the Nix package manager
  48. 48. Related work: other generators npm2nix: Original generation attempt done by Shea Levy. Composes Nix store paths for each dependency and symlinks them. Faster, but less accurate when dealing when shared dependencies Does not support flat module installations or cyclic dependencies nixfromnpm: Implementation of npm2nix’s Nix concepts in Haskell. Can partially regenerate expressions. Sander van der Burg Deploying NPM packages with the Nix package manager
  49. 49. Related work: NPM alternatives yarn: parallelization for speed improvements, version locking ied: parallelization, content-addressable store, atomicity, maximal sharing of dependencies (some of its concepts are heavily inspired by Nix according to its author :-) ) Sander van der Burg Deploying NPM packages with the Nix package manager
  50. 50. References Nix project: http://nixos.org. Nix package manager: http://nixos.org/nix. node2nix: https://github.com/svanderburg/node2nix. Sander van der Burg Deploying NPM packages with the Nix package manager
  51. 51. Questions Sander van der Burg Deploying NPM packages with the Nix package manager

×