Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
The NixOS project and deploying systems
declaratively
Sander van der Burg
March 12, 2016
Sander van der Burg The NixOS pro...
On being declarative
A declarative sentence makes a statement. It is punctuated by a
period:
The dog in the neighbor’s yar...
On being imperative
An imperative sentence is a command or polite request:
(Source: https://xkcd.com/149/)
Sander van der ...
On being declarative in programming
A style of building the structure and elements of computer
programs – that expresses t...
On being declarative in programming
Declarative: describing ”what” is to be computed rather than
”how” to compute the resu...
On being declarative in programming
Declarative: describing ”what” is to be computed rather than
”how” to compute the resu...
Example: HTML and CSS
<!DOCTYPE html>
<html>
<head>
<title>Test</title>
<link rel=”stylesheet” href=”style.css” type=”text...
Example: HTML and CSS
Sander van der Burg The NixOS project and deploying systems declaratively
Deployment: What do we want?
Sander van der Burg The NixOS project and deploying systems declaratively
Deployment: Activities
Building
Packaging
Transferring packages from producer to consumer site
Activating
Deactivating
Mod...
Deployment complexity
Diverse technology imposes many kinds of deployment procedures:
Different operating systems, different...
Deployment complexity
Deployment may need to be done on a large scale:
Sander van der Burg The NixOS project and deploying...
Deployment complexity
How to update the deployment frequently?
How not to break the system while upgrading?
How to minimiz...
Deployment automation
To deal with deployment complexities automation is needed!
Sander van der Burg The NixOS project and...
Deployment automation
To deal with deployment complexities automation is needed!
Many automated deployment solutions avail...
On being declarative in deployment
Declare what system you want to run in the consumer environment,
not the activities tha...
Chef: convergent declarative deployment
wordpress_latest = Chef::Config[:file_cache_path] + "/wordpress-latest.tar.gz"
rem...
Chef: convergent declarative deployment
wordpress_latest = Chef::Config[:file_cache_path] + "/wordpress-latest.tar.gz"
rem...
NixOS
NixOS: A GNU/Linux distribution using the Nix package manager
Sander van der Burg The NixOS project and deploying sy...
NixOS configuration
/etc/nixos/configuration.nix
{pkgs, ...}:
{
boot.loader.grub.device = "/dev/sda";
fileSystems = [ { mou...
NixOS configuration
nixos-rebuild switch
Nix package manager builds a complete system configuration
Includes all packages an...
NixOS bootloader
Sander van der Burg The NixOS project and deploying systems declaratively
Nix store
Main idea: store all packages
in isolation from each other:
/nix/store/rpdqxnilb0cg...
-firefox-3.5.4
Paths cont...
Nix expressions
openssh.nix
{ stdenv, fetchurl, openssl, zlib }:
stdenv.mkDerivation {
name = "openssh-4.6p1";
src = fetch...
Nix expressions
all-packages.nix
openssh = import ../tools/networking/openssh {
inherit fetchurl stdenv openssl zlib;
};
o...
User environments
Users can have
different sets of
installed applications.
PATH
/nix/.../profiles
current
42
/nix/store
pp56...
User environments
Users can have
different sets of
installed applications.
nix-env operations
create new user
environments ...
User environments
Users can have
different sets of
installed applications.
nix-env operations
create new user
environments ...
User environments
Users can have
different sets of
installed applications.
nix-env operations
create new user
environments ...
User environments
Users can have
different sets of
installed applications.
nix-env operations
create new user
environments ...
User environments
Users can have
different sets of
installed applications.
nix-env operations
create new user
environments ...
User environments
Users can have
different sets of
installed applications.
nix-env operations
create new user
environments ...
NixOS
In NixOS, all packages including the Linux kernel and
configuration files are managed by Nix.
NixOS does not have dire...
Distributed deployment
NixOS has good properties for deployment of a single system
Can we extend these properties to distr...
Motivating example: Trac
Sander van der Burg The NixOS project and deploying systems declaratively
Motivating example: Trac
Trac can be deployed in a distributed environment:
Subversion server
Database server
Web server
S...
Distributed NixOS configuration
network.nix
{ storage = {pkgs, ...}:
{
services.nfsKernel.server.enable = true; ...
};
post...
Distributed deployment
$ nixops create network.nix -d production
$ nixops deploy -d production
Build system configurations ...
The Nix project
Tools part of the Nix-project: http://nixos.org:
Nix. A purely functional package manager
NixOS. Nix based...
The Nix project
Automated deployment using declarative specifications with the
following properties:
Generic. Can be used w...
Nix-related tools: how declarative are they?
Nix-related tools solve problems in a technical domain:
e.g. deployment of pa...
A real world example: Conference Compass
Conference Compass provides a service to improve the way
people experience events...
A real world example: Conference Compass
We have a product-line using a Nix-based build infrastructure,
including Hydra, d...
A real world example: Conference Compass
The app’s contents is customizable with a configurator service
allowing organizers...
A real world example: Conference Compass
{
wrocloverb = {
eventName = "wroclove.rb 2016";
domain = "http://www.wrocloverb....
Conclusions
I have illustated a declarative deployment vision
I have demonstrated NixOS and the Nix package manager
I have...
References
NixOS project homepage: http://nixos.org
Software available under free and open-source licenses
(LGPL/X11)
Nix ...
Questions
Sander van der Burg The NixOS project and deploying systems declaratively
Upcoming SlideShare
Loading in …5
×

The NixOS project and deploying systems declaratively

34,566 views

Published on

Presentation given @ wroclove.rb 2016

Published in: Software
  • Be the first to comment

The NixOS project and deploying systems declaratively

  1. 1. The NixOS project and deploying systems declaratively Sander van der Burg March 12, 2016 Sander van der Burg The NixOS project and deploying systems declaratively
  2. 2. On being declarative A declarative sentence makes a statement. It is punctuated by a period: The dog in the neighbor’s yard is barking. (Source: http://www.slideshare.net/luigi a97/parts-of-a-sentence-8862361) Sander van der Burg The NixOS project and deploying systems declaratively
  3. 3. On being imperative An imperative sentence is a command or polite request: (Source: https://xkcd.com/149/) Sander van der Burg The NixOS project and deploying systems declaratively
  4. 4. On being declarative in programming A style of building the structure and elements of computer programs – that expresses the logic of a computation without describing its control flow (Source: https://en.wikipedia.org/wiki/Declarative programming) Sander van der Burg The NixOS project and deploying systems declaratively
  5. 5. On being declarative in programming Declarative: describing ”what” is to be computed rather than ”how” to compute the result/behavior Imperative: a description of a computation that involves implicit effects, usually mutable state and input/output. (Source: http://wcook.blogspot.com/2013/05/declarative-versus-imperative.html) Sander van der Burg The NixOS project and deploying systems declaratively
  6. 6. On being declarative in programming Declarative: describing ”what” is to be computed rather than ”how” to compute the result/behavior Imperative: a description of a computation that involves implicit effects, usually mutable state and input/output. (Source: http://wcook.blogspot.com/2013/05/declarative-versus-imperative.html) Sander van der Burg The NixOS project and deploying systems declaratively Declarative “declarative” is a spectrum – hard to draw a hard line between “what” and “how”. Imperative is not necessarily the opposite of declarative.
  7. 7. Example: HTML and CSS <!DOCTYPE html> <html> <head> <title>Test</title> <link rel=”stylesheet” href=”style.css” type=”text/css”> </head> <body> <div id=”outer”> <div id=”inner”> <p>HTML and CSS are declarative and so cool!</p> </div> </div> </body> </html> #outer { margin−left: auto; margin−right: auto; width: 20%; border−style: solid; } #inner { width: 500px; } Sander van der Burg The NixOS project and deploying systems declaratively
  8. 8. Example: HTML and CSS Sander van der Burg The NixOS project and deploying systems declaratively
  9. 9. Deployment: What do we want? Sander van der Burg The NixOS project and deploying systems declaratively
  10. 10. Deployment: Activities Building Packaging Transferring packages from producer to consumer site Activating Deactivating Modifying configuration files Upgrading Sander van der Burg The NixOS project and deploying systems declaratively
  11. 11. Deployment complexity Diverse technology imposes many kinds of deployment procedures: Different operating systems, different dependencies, many variants Sander van der Burg The NixOS project and deploying systems declaratively
  12. 12. Deployment complexity Deployment may need to be done on a large scale: Sander van der Burg The NixOS project and deploying systems declaratively
  13. 13. Deployment complexity How to update the deployment frequently? How not to break the system while upgrading? How to minimize downtimes? How to roll back in case of a failure? Sander van der Burg The NixOS project and deploying systems declaratively
  14. 14. Deployment automation To deal with deployment complexities automation is needed! Sander van der Burg The NixOS project and deploying systems declaratively
  15. 15. Deployment automation To deal with deployment complexities automation is needed! Many automated deployment solutions available Automation is typically driven by a specification Some solutions have been developed for specific kinds of technology: Apache Felix (for OSGi components) Some solutions are general: Chef Puppet CFEngine Nix Some solutions use declarative deployment specifications Sander van der Burg The NixOS project and deploying systems declaratively
  16. 16. On being declarative in deployment Declare what system you want to run in the consumer environment, not the activities that need to be executed to accomplish it! Sander van der Burg The NixOS project and deploying systems declaratively
  17. 17. Chef: convergent declarative deployment wordpress_latest = Chef::Config[:file_cache_path] + "/wordpress-latest.tar.gz" remote_file wordpress_latest do source "http://wordpress.org/latest.tar.gz" mode "0644" end directory node["phpapp"]["path"] do owner "root" group "root" mode "0755" action :create recursive true end execute "untar-wordpress" do cwd node[’phpapp’][’path’] command "tar --strip-components 1 -xzf " + wordpress_latest creates node[’phpapp’][’path’] + "/wp-settings.php" end (Source: http://gettingstartedwithchef.com/first-steps-with-chef.html) Sander van der Burg The NixOS project and deploying systems declaratively
  18. 18. Chef: convergent declarative deployment wordpress_latest = Chef::Config[:file_cache_path] + "/wordpress-latest.tar.gz" remote_file wordpress_latest do source "http://wordpress.org/latest.tar.gz" mode "0644" end directory node["phpapp"]["path"] do owner "root" group "root" mode "0755" action :create recursive true end execute "untar-wordpress" do cwd node[’phpapp’][’path’] command "tar --strip-components 1 -xzf " + wordpress_latest creates node[’phpapp’][’path’] + "/wp-settings.php" end (Source: http://gettingstartedwithchef.com/first-steps-with-chef.html) Sander van der Burg The NixOS project and deploying systems declaratively Declarative The specification captures the outcome of a set of changes as a fixpoint. Chef converges to the outcome. Specification applies to set of machines – but does not guarantee that an entire machine’s configuration can be reproduced elsewhere How to roll back to a previous configuration? How to mimimize downtime?
  19. 19. NixOS NixOS: A GNU/Linux distribution using the Nix package manager Sander van der Burg The NixOS project and deploying systems declaratively
  20. 20. NixOS configuration /etc/nixos/configuration.nix {pkgs, ...}: { boot.loader.grub.device = "/dev/sda"; fileSystems = [ { mountPoint = "/"; device = "/dev/sda2"; } ]; swapDevices = [ { device = "/dev/sda1"; } ]; services = { openssh.enable = true; xserver = { enable = true; desktopManager.kde4.enable = true; }; }; environment.systemPackages = [ pkgs.mc pkgs.firefox ]; } Sander van der Burg The NixOS project and deploying systems declaratively
  21. 21. NixOS configuration nixos-rebuild switch Nix package manager builds a complete system configuration Includes all packages and generates all configuration files, e.g. OpenSSH configuration Upgrades are (almost) atomic Components are stored safely next to each other, due to hashes No files are automatically removed or overwritten Users can switch to older generations of system configurations not garbage collected yet Sander van der Burg The NixOS project and deploying systems declaratively
  22. 22. NixOS bootloader Sander van der Burg The NixOS project and deploying systems declaratively
  23. 23. Nix store Main idea: store all packages in isolation from each other: /nix/store/rpdqxnilb0cg... -firefox-3.5.4 Paths contain a 160-bit cryptographic hash of all inputs used to build the package: Sources Libraries Compilers Build scripts . . . /nix/store l9w6773m1msy...-openssh-4.6p1 bin ssh sbin sshd smkabrbibqv7...-openssl-0.9.8e lib libssl.so.0.9.8 c6jbqm2mc0a7...-zlib-1.2.3 lib libz.so.1.2.3 im276akmsrhv...-glibc-2.5 lib libc.so.6 Sander van der Burg The NixOS project and deploying systems declaratively
  24. 24. Nix expressions openssh.nix { stdenv, fetchurl, openssl, zlib }: stdenv.mkDerivation { name = "openssh-4.6p1"; src = fetchurl { url = http://.../openssh-4.6p1.tar.gz; sha256 = "0fpjlr3bfind0y94bk442x2p..."; }; buildCommand = ’’ tar xjf $src ./configure --prefix=$out --with-openssl=${openssl} make; make install ’’; } Sander van der Burg The NixOS project and deploying systems declaratively
  25. 25. Nix expressions all-packages.nix openssh = import ../tools/networking/openssh { inherit fetchurl stdenv openssl zlib; }; openssl = import ../development/libraries/openssl { inherit fetchurl stdenv perl; }; stdenv = ...; openssl = ...; zlib = ...; perl = ...; nix-env -f all-packages.nix -iA openssh Produces a /nix/store/l9w6773m1msy...-openssh-4.6p1 package in the Nix store. Sander van der Burg The NixOS project and deploying systems declaratively
  26. 26. User environments Users can have different sets of installed applications. PATH /nix/.../profiles current 42 /nix/store pp56i0a01si5...-user-env bin firefox ssh l9w6773m1msy...-openssh-4.6p1 bin ssh rpdqxnilb0cg...-firefox-3.5.4 bin firefox Sander van der Burg The NixOS project and deploying systems declaratively
  27. 27. User environments Users can have different sets of installed applications. nix-env operations create new user environments in the store. PATH /nix/.../profiles current 42 /nix/store pp56i0a01si5...-user-env bin firefox ssh l9w6773m1msy...-openssh-4.6p1 bin ssh rpdqxnilb0cg...-firefox-3.5.4 bin firefox aqn3wygq9jzk...-openssh-5.2p1 bin ssh (nix-env -u openssh) Sander van der Burg The NixOS project and deploying systems declaratively
  28. 28. User environments Users can have different sets of installed applications. nix-env operations create new user environments in the store. PATH /nix/.../profiles current 42 /nix/store pp56i0a01si5...-user-env bin firefox ssh l9w6773m1msy...-openssh-4.6p1 bin ssh rpdqxnilb0cg...-firefox-3.5.4 bin firefox aqn3wygq9jzk...-openssh-5.2p1 bin ssh i3d9vh6d8ip1...-user-env bin ssh firefox (nix-env -u openssh) Sander van der Burg The NixOS project and deploying systems declaratively
  29. 29. User environments Users can have different sets of installed applications. nix-env operations create new user environments in the store. PATH /nix/.../profiles current 42 43 /nix/store pp56i0a01si5...-user-env bin firefox ssh l9w6773m1msy...-openssh-4.6p1 bin ssh rpdqxnilb0cg...-firefox-3.5.4 bin firefox aqn3wygq9jzk...-openssh-5.2p1 bin ssh i3d9vh6d8ip1...-user-env bin ssh firefox (nix-env -u openssh) Sander van der Burg The NixOS project and deploying systems declaratively
  30. 30. User environments Users can have different sets of installed applications. nix-env operations create new user environments in the store. We can atomically switch between them. PATH /nix/.../profiles current 42 43 /nix/store pp56i0a01si5...-user-env bin firefox ssh l9w6773m1msy...-openssh-4.6p1 bin ssh rpdqxnilb0cg...-firefox-3.5.4 bin firefox aqn3wygq9jzk...-openssh-5.2p1 bin ssh i3d9vh6d8ip1...-user-env bin ssh firefox (nix-env -u openssh) Sander van der Burg The NixOS project and deploying systems declaratively
  31. 31. User environments Users can have different sets of installed applications. nix-env operations create new user environments in the store. We can atomically switch between them. These are roots of the garbage collector. PATH /nix/.../profiles current 43 /nix/store pp56i0a01si5...-user-env bin firefox ssh l9w6773m1msy...-openssh-4.6p1 bin ssh rpdqxnilb0cg...-firefox-3.5.4 bin firefox aqn3wygq9jzk...-openssh-5.2p1 bin ssh i3d9vh6d8ip1...-user-env bin ssh firefox (nix-env --remove-generations old) Sander van der Burg The NixOS project and deploying systems declaratively
  32. 32. User environments Users can have different sets of installed applications. nix-env operations create new user environments in the store. We can atomically switch between them. These are roots of the garbage collector. PATH /nix/.../profiles current 43 /nix/store rpdqxnilb0cg...-firefox-3.5.4 bin firefox aqn3wygq9jzk...-openssh-5.2p1 bin ssh i3d9vh6d8ip1...-user-env bin ssh firefox (nix-collect-garbage) Sander van der Burg The NixOS project and deploying systems declaratively
  33. 33. NixOS In NixOS, all packages including the Linux kernel and configuration files are managed by Nix. NixOS does not have directories such as: /lib and /usr NixOS has a minimal /bin and /etc Sander van der Burg The NixOS project and deploying systems declaratively
  34. 34. Distributed deployment NixOS has good properties for deployment of a single system Can we extend these properties to distributed systems? Sander van der Burg The NixOS project and deploying systems declaratively
  35. 35. Motivating example: Trac Sander van der Burg The NixOS project and deploying systems declaratively
  36. 36. Motivating example: Trac Trac can be deployed in a distributed environment: Subversion server Database server Web server Sander van der Burg The NixOS project and deploying systems declaratively
  37. 37. Distributed NixOS configuration network.nix { storage = {pkgs, ...}: { services.nfsKernel.server.enable = true; ... }; postgresql = {pkgs, ...}: { services.postgresql.enable = true; ... }; webserver = {pkgs, ...}: { fileSystems = [ { mountPoint = "/repos"; device = "storage:/repos"; } ]; services.httpd.enable = true; services.httpd.extraSubservices = [ { serviceType = "trac"; } ]; ... }; ... } Sander van der Burg The NixOS project and deploying systems declaratively
  38. 38. Distributed deployment $ nixops create network.nix -d production $ nixops deploy -d production Build system configurations by the Nix package manager Transfer complete system and all dependencies to target machines in the network Efficient: only missing store paths must be transferred Safe: Existing configuration is not affected, because no files are overwritten or removed Activate new system configuration In case of a failure, roll back all configurations Relatively cheap operation, because old configuration is stored next to new configuration Sander van der Burg The NixOS project and deploying systems declaratively
  39. 39. The Nix project Tools part of the Nix-project: http://nixos.org: Nix. A purely functional package manager NixOS. Nix based GNU/Linux distribution Hydra. Nix based continuous build and integration server Disnix. Nix based distributed service deployment NixOps. NixOS-based multi-cloud deployment tool Sander van der Burg The NixOS project and deploying systems declaratively
  40. 40. The Nix project Automated deployment using declarative specifications with the following properties: Generic. Can be used with many programming languages, component technologies, and operating systems. Reproducible. (Almost) no impurities – if inputs are the same, result should be the same regardless of its location Reliable. Dependency completeness, (almost) atomic upgrades and rollbacks. Efficient. Only the required deployment activities are executed. Sander van der Burg The NixOS project and deploying systems declaratively
  41. 41. Nix-related tools: how declarative are they? Nix-related tools solve problems in a technical domain: e.g. deployment of packages, machines, services, ... What about your domain? Sander van der Burg The NixOS project and deploying systems declaratively
  42. 42. A real world example: Conference Compass Conference Compass provides a service to improve the way people experience events Most visible part of the service: apps for conference attendees Each customer basically gets “their own” app. Sander van der Burg The NixOS project and deploying systems declaratively
  43. 43. A real world example: Conference Compass We have a product-line using a Nix-based build infrastructure, including Hydra, driven by simple app specific configurations: { name = "wroclove.rb 2016"; homepage = "http://www.wrocloverb.com"; iconSet = ./icons; backgroundImage" = ./background.png; ... } Sander van der Burg The NixOS project and deploying systems declaratively
  44. 44. A real world example: Conference Compass The app’s contents is customizable with a configurator service allowing organizers to create and update their content Apps connect to a configurator to retrieve the data to be displayed and other configuration settings Integration with third party information systems is also possible Sander van der Burg The NixOS project and deploying systems declaratively
  45. 45. A real world example: Conference Compass { wrocloverb = { eventName = "wroclove.rb 2016"; domain = "http://www.wrocloverb.com"; channels = [ "wrocloverb" ]; }; otherevent = ...; yetanotherevent = ...; ... } We have developed a formalism to concisely model such configurations and to automatically deploy them Tool figures out which machines to configure, what services to deploy etc. If underlying implementation and technology evolves, specifications (probably) remains the same. Sander van der Burg The NixOS project and deploying systems declaratively
  46. 46. Conclusions I have illustated a declarative deployment vision I have demonstrated NixOS and the Nix package manager I have explained that domain specific deployment tools can be built on top of tools from the Nix project Sander van der Burg The NixOS project and deploying systems declaratively
  47. 47. References NixOS project homepage: http://nixos.org Software available under free and open-source licenses (LGPL/X11) Nix package manager can be used on any Linux system, Mac OS X, and (in some extent) Cygwin and FreeBSD. Sander van der Burg The NixOS project and deploying systems declaratively
  48. 48. Questions Sander van der Burg The NixOS project and deploying systems declaratively

×