Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Nix for Python developers

3,776 views

Published on

What do you do when your python project has dependencies beyond Python? Nix is a "purely functional" package manager, which can also be used like "virtualenv", but for everything. I would cover single user installation of Nix package manager on Linux or Mac, defining Nix based development environments, generating Nix expressions for large Python projects using buildout, and finally, building minimal Docker images from scratch by using those generated expressions.

Published in: Technology
  • Be the first to comment

Nix for Python developers

  1. 1. PyConFinland19.10.2015 Nix Python fordevelopers devops shell virtualenv Docker buildout packaging testing building deployment developing nixpkgs DisNix Mesos NixOS NixOps development distributing ldap linux lxml Aurora NodeLATEX darwin deploying nix-build nix-copy-closure nix-shell python3 setuptoolsErlang Haskell RabbitMQ containers dependencies libxml libxslt nix-collect-garbage openldap pillow pyramid texlive zope
  2. 2. Nix for Python developers by Asko Soukka • github.com/datakurre • twitter.com/datakurre
  3. 3. Dependencies?
  4. 4. Installing a Python package may require external libraries – Pillow (libjpeg, libtiff, libwebp, zlib) – lxml (libxml2, libxslt) – numpy (gfortran) – python-ldap (curys-sasl, openldap, openssl) – ...
  5. 5. Using a Python package may require external software – Graphviz – ImageMagick – LATEX – NodeJS – ...
  6. 6. Running a Python application may require external services – PostgreSQL – memcached – Redis – RabbitMQ – ...
  7. 7. ONE DOES NOT SIMPLYONE DOES NOT SIMPLY pip installpip install
  8. 8. What if you could simply define any dependencies in a hashbang? #! /usr/bin/env nix-shell #! nix-shell -i python -p pythonPackages.pillow
  9. 9. . . . or define all required dependencies just when calling? $ nix-shell -p python -p gnuplot -p git --run "python gitstats input output"→
  10. 10. . . . build an environment with any dependencies? $ nix-build filename.nix -o env $ ls env/bin 2to3 dvilj4 makeindex pdflatex afm2pl dvilj4l makejvf pdfmex afm2tfm dvilj6 mendex pdftex ... ctangle gftopk otfinfo python ...
  11. 11. . . . enter into a shell with any dependencies? $ nix-shell filename.nix $ which latex /nix/store/.../bin/latex $ which python /nix/store/.../bin/python $ exit
  12. 12. . . . easily build an app with any dependencies into a single container? FROM scratch ADD myapp.nix.tar.gz / EXPOSE 8080 ENTRYPOINT ["/app/bin/python3"]
  13. 13. Nix – the purely functional package manager – Functional DSL for building stuff – Builds named by their expression hash – Builds isolated and reproducible – Build results immutable and shareable – Single or multi-user installation – Garbage collection on request – Tools for development and distribution
  14. 14. $ nix-build filename.nix -o env $ tree -L 2 `realpath env` /nix/store/n8rydlmdqjc4c6yik1dxyhl869jl5fbr-env bin/python -> /nix/store/64856yqx1rk2f3qi3qs1dmxmsk83yfyw -python-2.7.10-env/bin/python→ ... bin/redis-server -> /nix/store/p76nwndjnd8mm00k5di8plmj7n8kcnm0 -redis-3.0.2/bin/redis-server→ /nix/store/ + <hash> + <name> + <version>
  15. 15. nixpkgs – the Nix packages collection – 70 530 commits – 697 authors
  16. 16. $ bash <(curl https://nixos.org/nix/install)
  17. 17. ”Friends sometimes let friends curl to shell” –Domen Kožar
  18. 18. I DON’T ALWAYSI DON’T ALWAYS USE CODE EXAMPLESUSE CODE EXAMPLES BUT WHEN I DO . . .BUT WHEN I DO . . .
  19. 19. Nix ”virtualenv” with import <nixpkgs> {}; buildEnv { name = "env"; paths = [ redis (python.buildEnv.override { extraLibs = [ pythonPackages.redis ]; }) ]; } $ nix-build filename.nix -o env
  20. 20. Nix build shell with import <nixpkgs> {}; stdenv.mkDerivation { name = "env"; buildInputs = [ redis (python.buildEnv.override { extraLibs = [ pythonPackages.redis ]; }) ]; shellHook = '' export PYTHONPATH=`pwd` ''; }
  21. 21. Nix build shell usage $ nix-shell filename.nix --run "..." $ nix-shell filename.nix ... $ exit shellHook = '' export SSL_CERT_FILE=${cacert}/etc/ssl/certs/ca-bundle.crt '';
  22. 22. Nix build shell helper function nix() { if [ ! -e shell.drv ]; then nix-instantiate --indirect --add-root $PWD/shell.drv fi echo nix-shell $PWD/shell.drv --run "$@" | sh; } $ nix ... $ rm shell.drv
  23. 23. $ nix-env -qaP|grep -i needle
  24. 24. Nix python package with import <nixpkgs> {}; buildPythonPackage { name = "myapp-1.0"; src = ./.; buildInputs = []; propagatedBuildInputs = [ pythonPackages.redis ]; doCheck = false; } $ nix-env -i -f filename.nix
  25. 25. With custom python dependencies with import <nixpkgs> {}; let dependencies = rec { _future = buildPythonPackage { name = "future-0.15.2"; src = fetchurl { url = ".../future-0.15.2.tar.gz"; md5 = "...b76714c5ceb8c09ea3a06"; }; doCheck = false; }; }; in with dependencies; buildPythonPackage { ... [ _future ]; };
  26. 26. Reproducible with import (fetchTarball https://github.com/NixOS/nixpkgs/archive/ 2766a4b44ee6eafae03a042801270c7f6b8ed32a.tar.gz) {}; → → ...
  27. 27. $ nix-collect-garbage -d
  28. 28. Generating Nix expressions. . . [mercurial] recipe = collective.recipe.nix eggs = mercurial propagated-build-inputs = keyring=hgtools keyring=setuptools-scm mercurial=mercurial-keyring mercurial=pyopenssl build-inputs = mercurial=gettext nixpkgs = cryptography=pythonPackages.cryptography pyopenssl=pyopenssl
  29. 29. . . . with collective.recipe.nix with import <nixpkgs> {}; stdenv.mkDerivation { name = "env"; buildInputs = [ pythonPackages.zc_buildout_nix ]; shellHook = '' export SSL_CERT_FILE=... ''; } $ nix-shell filename.nix --run "buildout-nix filename.cfg" $ nix-env -i -f mercurial-mercurial.nix
  30. 30. github.com/datakurre/collective.recipe.nix
  31. 31. DEPLOY?DEPLOY?
  32. 32. Nix is not a container technology, but Nix builds are good to go with any container technology!
  33. 33. Some deployment options – tar cvz `nix-store -qR env` – Docker – nix-build – nix-serve -p 8080 – nix-copy-closure – NixOps – DisNix
  34. 34. Runtime isolation with Docker $ tar cv --files-from /dev/null | docker import - empty $ docker run --rm -v /nix/store:/nix/store -p 8080:8080 -v `pwd`:/app empty /app/result/bin/python /app/hello_world.py→
  35. 35. github.com/datakurre/nix-build-pack-docker
  36. 36. linux-headers-3.12.32 acl-2.2.52 openssl-1.0.1p attr-2.4.47 python-2.7.10 bash-4.3-p42 bzip2-1.0.6 glibc-2.21 zlib-1.2.8 coreutils-8.24
  37. 37. Official docs – nixos.org/nix – nixos.org/nixops – nixos.org/disnix Community resources – nixos.org/wiki – planet.nixos.org – conf.nixos.org From the author – datakurre.pandala.org/search/label/nix – gist.github.com/datakurre Freenode (IRC) channels – #nixos – ##nix-darwin
  38. 38. Stop worrying.

×