Successfully reported this slideshow.
Your SlideShare is downloading. ×

The NixOS project and deploying systems declaratively

Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Ad
Upcoming SlideShare
The Nix project
The Nix project
Loading in …3
×

Check these out next

1 of 48 Ad
Advertisement

More Related Content

Slideshows for you (20)

Similar to The NixOS project and deploying systems declaratively (20)

Advertisement

More from Sander van der Burg (17)

Recently uploaded (20)

Advertisement

The NixOS project and deploying systems declaratively

  1. 1. The NixOS project and deploying systems declaratively Sander van der Burg March 12, 2016 Sander van der Burg The NixOS project and deploying systems declaratively
  2. 2. On being declarative A declarative sentence makes a statement. It is punctuated by a period: The dog in the neighbor’s yard is barking. (Source: http://www.slideshare.net/luigi a97/parts-of-a-sentence-8862361) Sander van der Burg The NixOS project and deploying systems declaratively
  3. 3. On being imperative An imperative sentence is a command or polite request: (Source: https://xkcd.com/149/) Sander van der Burg The NixOS project and deploying systems declaratively
  4. 4. On being declarative in programming A style of building the structure and elements of computer programs – that expresses the logic of a computation without describing its control flow (Source: https://en.wikipedia.org/wiki/Declarative programming) Sander van der Burg The NixOS project and deploying systems declaratively
  5. 5. On being declarative in programming Declarative: describing ”what” is to be computed rather than ”how” to compute the result/behavior Imperative: a description of a computation that involves implicit effects, usually mutable state and input/output. (Source: http://wcook.blogspot.com/2013/05/declarative-versus-imperative.html) Sander van der Burg The NixOS project and deploying systems declaratively
  6. 6. On being declarative in programming Declarative: describing ”what” is to be computed rather than ”how” to compute the result/behavior Imperative: a description of a computation that involves implicit effects, usually mutable state and input/output. (Source: http://wcook.blogspot.com/2013/05/declarative-versus-imperative.html) Sander van der Burg The NixOS project and deploying systems declaratively Declarative “declarative” is a spectrum – hard to draw a hard line between “what” and “how”. Imperative is not necessarily the opposite of declarative.
  7. 7. Example: HTML and CSS <!DOCTYPE html> <html> <head> <title>Test</title> <link rel=”stylesheet” href=”style.css” type=”text/css”> </head> <body> <div id=”outer”> <div id=”inner”> <p>HTML and CSS are declarative and so cool!</p> </div> </div> </body> </html> #outer { margin−left: auto; margin−right: auto; width: 20%; border−style: solid; } #inner { width: 500px; } Sander van der Burg The NixOS project and deploying systems declaratively
  8. 8. Example: HTML and CSS Sander van der Burg The NixOS project and deploying systems declaratively
  9. 9. Deployment: What do we want? Sander van der Burg The NixOS project and deploying systems declaratively
  10. 10. Deployment: Activities Building Packaging Transferring packages from producer to consumer site Activating Deactivating Modifying configuration files Upgrading Sander van der Burg The NixOS project and deploying systems declaratively
  11. 11. Deployment complexity Diverse technology imposes many kinds of deployment procedures: Different operating systems, different dependencies, many variants Sander van der Burg The NixOS project and deploying systems declaratively
  12. 12. Deployment complexity Deployment may need to be done on a large scale: Sander van der Burg The NixOS project and deploying systems declaratively
  13. 13. Deployment complexity How to update the deployment frequently? How not to break the system while upgrading? How to minimize downtimes? How to roll back in case of a failure? Sander van der Burg The NixOS project and deploying systems declaratively
  14. 14. Deployment automation To deal with deployment complexities automation is needed! Sander van der Burg The NixOS project and deploying systems declaratively
  15. 15. Deployment automation To deal with deployment complexities automation is needed! Many automated deployment solutions available Automation is typically driven by a specification Some solutions have been developed for specific kinds of technology: Apache Felix (for OSGi components) Some solutions are general: Chef Puppet CFEngine Nix Some solutions use declarative deployment specifications Sander van der Burg The NixOS project and deploying systems declaratively
  16. 16. On being declarative in deployment Declare what system you want to run in the consumer environment, not the activities that need to be executed to accomplish it! Sander van der Burg The NixOS project and deploying systems declaratively
  17. 17. Chef: convergent declarative deployment wordpress_latest = Chef::Config[:file_cache_path] + "/wordpress-latest.tar.gz" remote_file wordpress_latest do source "http://wordpress.org/latest.tar.gz" mode "0644" end directory node["phpapp"]["path"] do owner "root" group "root" mode "0755" action :create recursive true end execute "untar-wordpress" do cwd node[’phpapp’][’path’] command "tar --strip-components 1 -xzf " + wordpress_latest creates node[’phpapp’][’path’] + "/wp-settings.php" end (Source: http://gettingstartedwithchef.com/first-steps-with-chef.html) Sander van der Burg The NixOS project and deploying systems declaratively
  18. 18. Chef: convergent declarative deployment wordpress_latest = Chef::Config[:file_cache_path] + "/wordpress-latest.tar.gz" remote_file wordpress_latest do source "http://wordpress.org/latest.tar.gz" mode "0644" end directory node["phpapp"]["path"] do owner "root" group "root" mode "0755" action :create recursive true end execute "untar-wordpress" do cwd node[’phpapp’][’path’] command "tar --strip-components 1 -xzf " + wordpress_latest creates node[’phpapp’][’path’] + "/wp-settings.php" end (Source: http://gettingstartedwithchef.com/first-steps-with-chef.html) Sander van der Burg The NixOS project and deploying systems declaratively Declarative The specification captures the outcome of a set of changes as a fixpoint. Chef converges to the outcome. Specification applies to set of machines – but does not guarantee that an entire machine’s configuration can be reproduced elsewhere How to roll back to a previous configuration? How to mimimize downtime?
  19. 19. NixOS NixOS: A GNU/Linux distribution using the Nix package manager Sander van der Burg The NixOS project and deploying systems declaratively
  20. 20. NixOS configuration /etc/nixos/configuration.nix {pkgs, ...}: { boot.loader.grub.device = "/dev/sda"; fileSystems = [ { mountPoint = "/"; device = "/dev/sda2"; } ]; swapDevices = [ { device = "/dev/sda1"; } ]; services = { openssh.enable = true; xserver = { enable = true; desktopManager.kde4.enable = true; }; }; environment.systemPackages = [ pkgs.mc pkgs.firefox ]; } Sander van der Burg The NixOS project and deploying systems declaratively
  21. 21. NixOS configuration nixos-rebuild switch Nix package manager builds a complete system configuration Includes all packages and generates all configuration files, e.g. OpenSSH configuration Upgrades are (almost) atomic Components are stored safely next to each other, due to hashes No files are automatically removed or overwritten Users can switch to older generations of system configurations not garbage collected yet Sander van der Burg The NixOS project and deploying systems declaratively
  22. 22. NixOS bootloader Sander van der Burg The NixOS project and deploying systems declaratively
  23. 23. Nix store Main idea: store all packages in isolation from each other: /nix/store/rpdqxnilb0cg... -firefox-3.5.4 Paths contain a 160-bit cryptographic hash of all inputs used to build the package: Sources Libraries Compilers Build scripts . . . /nix/store l9w6773m1msy...-openssh-4.6p1 bin ssh sbin sshd smkabrbibqv7...-openssl-0.9.8e lib libssl.so.0.9.8 c6jbqm2mc0a7...-zlib-1.2.3 lib libz.so.1.2.3 im276akmsrhv...-glibc-2.5 lib libc.so.6 Sander van der Burg The NixOS project and deploying systems declaratively
  24. 24. Nix expressions openssh.nix { stdenv, fetchurl, openssl, zlib }: stdenv.mkDerivation { name = "openssh-4.6p1"; src = fetchurl { url = http://.../openssh-4.6p1.tar.gz; sha256 = "0fpjlr3bfind0y94bk442x2p..."; }; buildCommand = ’’ tar xjf $src ./configure --prefix=$out --with-openssl=${openssl} make; make install ’’; } Sander van der Burg The NixOS project and deploying systems declaratively
  25. 25. Nix expressions all-packages.nix openssh = import ../tools/networking/openssh { inherit fetchurl stdenv openssl zlib; }; openssl = import ../development/libraries/openssl { inherit fetchurl stdenv perl; }; stdenv = ...; openssl = ...; zlib = ...; perl = ...; nix-env -f all-packages.nix -iA openssh Produces a /nix/store/l9w6773m1msy...-openssh-4.6p1 package in the Nix store. Sander van der Burg The NixOS project and deploying systems declaratively
  26. 26. User environments Users can have different sets of installed applications. PATH /nix/.../profiles current 42 /nix/store pp56i0a01si5...-user-env bin firefox ssh l9w6773m1msy...-openssh-4.6p1 bin ssh rpdqxnilb0cg...-firefox-3.5.4 bin firefox Sander van der Burg The NixOS project and deploying systems declaratively
  27. 27. User environments Users can have different sets of installed applications. nix-env operations create new user environments in the store. PATH /nix/.../profiles current 42 /nix/store pp56i0a01si5...-user-env bin firefox ssh l9w6773m1msy...-openssh-4.6p1 bin ssh rpdqxnilb0cg...-firefox-3.5.4 bin firefox aqn3wygq9jzk...-openssh-5.2p1 bin ssh (nix-env -u openssh) Sander van der Burg The NixOS project and deploying systems declaratively
  28. 28. User environments Users can have different sets of installed applications. nix-env operations create new user environments in the store. PATH /nix/.../profiles current 42 /nix/store pp56i0a01si5...-user-env bin firefox ssh l9w6773m1msy...-openssh-4.6p1 bin ssh rpdqxnilb0cg...-firefox-3.5.4 bin firefox aqn3wygq9jzk...-openssh-5.2p1 bin ssh i3d9vh6d8ip1...-user-env bin ssh firefox (nix-env -u openssh) Sander van der Burg The NixOS project and deploying systems declaratively
  29. 29. User environments Users can have different sets of installed applications. nix-env operations create new user environments in the store. PATH /nix/.../profiles current 42 43 /nix/store pp56i0a01si5...-user-env bin firefox ssh l9w6773m1msy...-openssh-4.6p1 bin ssh rpdqxnilb0cg...-firefox-3.5.4 bin firefox aqn3wygq9jzk...-openssh-5.2p1 bin ssh i3d9vh6d8ip1...-user-env bin ssh firefox (nix-env -u openssh) Sander van der Burg The NixOS project and deploying systems declaratively
  30. 30. User environments Users can have different sets of installed applications. nix-env operations create new user environments in the store. We can atomically switch between them. PATH /nix/.../profiles current 42 43 /nix/store pp56i0a01si5...-user-env bin firefox ssh l9w6773m1msy...-openssh-4.6p1 bin ssh rpdqxnilb0cg...-firefox-3.5.4 bin firefox aqn3wygq9jzk...-openssh-5.2p1 bin ssh i3d9vh6d8ip1...-user-env bin ssh firefox (nix-env -u openssh) Sander van der Burg The NixOS project and deploying systems declaratively
  31. 31. User environments Users can have different sets of installed applications. nix-env operations create new user environments in the store. We can atomically switch between them. These are roots of the garbage collector. PATH /nix/.../profiles current 43 /nix/store pp56i0a01si5...-user-env bin firefox ssh l9w6773m1msy...-openssh-4.6p1 bin ssh rpdqxnilb0cg...-firefox-3.5.4 bin firefox aqn3wygq9jzk...-openssh-5.2p1 bin ssh i3d9vh6d8ip1...-user-env bin ssh firefox (nix-env --remove-generations old) Sander van der Burg The NixOS project and deploying systems declaratively
  32. 32. User environments Users can have different sets of installed applications. nix-env operations create new user environments in the store. We can atomically switch between them. These are roots of the garbage collector. PATH /nix/.../profiles current 43 /nix/store rpdqxnilb0cg...-firefox-3.5.4 bin firefox aqn3wygq9jzk...-openssh-5.2p1 bin ssh i3d9vh6d8ip1...-user-env bin ssh firefox (nix-collect-garbage) Sander van der Burg The NixOS project and deploying systems declaratively
  33. 33. NixOS In NixOS, all packages including the Linux kernel and configuration files are managed by Nix. NixOS does not have directories such as: /lib and /usr NixOS has a minimal /bin and /etc Sander van der Burg The NixOS project and deploying systems declaratively
  34. 34. Distributed deployment NixOS has good properties for deployment of a single system Can we extend these properties to distributed systems? Sander van der Burg The NixOS project and deploying systems declaratively
  35. 35. Motivating example: Trac Sander van der Burg The NixOS project and deploying systems declaratively
  36. 36. Motivating example: Trac Trac can be deployed in a distributed environment: Subversion server Database server Web server Sander van der Burg The NixOS project and deploying systems declaratively
  37. 37. Distributed NixOS configuration network.nix { storage = {pkgs, ...}: { services.nfsKernel.server.enable = true; ... }; postgresql = {pkgs, ...}: { services.postgresql.enable = true; ... }; webserver = {pkgs, ...}: { fileSystems = [ { mountPoint = "/repos"; device = "storage:/repos"; } ]; services.httpd.enable = true; services.httpd.extraSubservices = [ { serviceType = "trac"; } ]; ... }; ... } Sander van der Burg The NixOS project and deploying systems declaratively
  38. 38. Distributed deployment $ nixops create network.nix -d production $ nixops deploy -d production Build system configurations by the Nix package manager Transfer complete system and all dependencies to target machines in the network Efficient: only missing store paths must be transferred Safe: Existing configuration is not affected, because no files are overwritten or removed Activate new system configuration In case of a failure, roll back all configurations Relatively cheap operation, because old configuration is stored next to new configuration Sander van der Burg The NixOS project and deploying systems declaratively
  39. 39. The Nix project Tools part of the Nix-project: http://nixos.org: Nix. A purely functional package manager NixOS. Nix based GNU/Linux distribution Hydra. Nix based continuous build and integration server Disnix. Nix based distributed service deployment NixOps. NixOS-based multi-cloud deployment tool Sander van der Burg The NixOS project and deploying systems declaratively
  40. 40. The Nix project Automated deployment using declarative specifications with the following properties: Generic. Can be used with many programming languages, component technologies, and operating systems. Reproducible. (Almost) no impurities – if inputs are the same, result should be the same regardless of its location Reliable. Dependency completeness, (almost) atomic upgrades and rollbacks. Efficient. Only the required deployment activities are executed. Sander van der Burg The NixOS project and deploying systems declaratively
  41. 41. Nix-related tools: how declarative are they? Nix-related tools solve problems in a technical domain: e.g. deployment of packages, machines, services, ... What about your domain? Sander van der Burg The NixOS project and deploying systems declaratively
  42. 42. A real world example: Conference Compass Conference Compass provides a service to improve the way people experience events Most visible part of the service: apps for conference attendees Each customer basically gets “their own” app. Sander van der Burg The NixOS project and deploying systems declaratively
  43. 43. A real world example: Conference Compass We have a product-line using a Nix-based build infrastructure, including Hydra, driven by simple app specific configurations: { name = "wroclove.rb 2016"; homepage = "http://www.wrocloverb.com"; iconSet = ./icons; backgroundImage" = ./background.png; ... } Sander van der Burg The NixOS project and deploying systems declaratively
  44. 44. A real world example: Conference Compass The app’s contents is customizable with a configurator service allowing organizers to create and update their content Apps connect to a configurator to retrieve the data to be displayed and other configuration settings Integration with third party information systems is also possible Sander van der Burg The NixOS project and deploying systems declaratively
  45. 45. A real world example: Conference Compass { wrocloverb = { eventName = "wroclove.rb 2016"; domain = "http://www.wrocloverb.com"; channels = [ "wrocloverb" ]; }; otherevent = ...; yetanotherevent = ...; ... } We have developed a formalism to concisely model such configurations and to automatically deploy them Tool figures out which machines to configure, what services to deploy etc. If underlying implementation and technology evolves, specifications (probably) remains the same. Sander van der Burg The NixOS project and deploying systems declaratively
  46. 46. Conclusions I have illustated a declarative deployment vision I have demonstrated NixOS and the Nix package manager I have explained that domain specific deployment tools can be built on top of tools from the Nix project Sander van der Burg The NixOS project and deploying systems declaratively
  47. 47. References NixOS project homepage: http://nixos.org Software available under free and open-source licenses (LGPL/X11) Nix package manager can be used on any Linux system, Mac OS X, and (in some extent) Cygwin and FreeBSD. Sander van der Burg The NixOS project and deploying systems declaratively
  48. 48. Questions Sander van der Burg The NixOS project and deploying systems declaratively

×