Functional Operations: Packaging, system/configuration building, and testing infrastructure with [Nix] lambda Maintaining configurations for different kinds of nodes and cloud resources in a [micro]service architecture can be an operational nightmare, especially if not managed with the application codebase. CI and CD job environments diverge from production configuration yielding their results unpredictable at best or produce false positives in the worst case. Code pushes to staging and production can have unintended consequences that can't be reasoned about before deploy and often can’t be inspected thoroughly on a dry run. Leading to unhappy users when problems do arise. This session will demonstrate the use of the Nix and NixOS ecosystem to define and build packages in a referentially transparent way which can be leveraged as a solid foundation to configure systems and test multiple [virtual] machines with coordinated scenarios. We also look at how reliable packaging allows us to build a consistent CI/CD pipeline where upgrading your version of the JVM doesn't break your CI build servers for days.

1. Functional Operations
Functional Programming @ Comcast Labs Connect
Susan Potter (@SusanPotter)
2018-03-09 Fri

3. Un-Fun Ops
0

4. Core InfraEng Problems
1

5. Core InfraEng Problems
• non-repeatable environments
• unpredictable machine rollback
• out-of-band updating of CI dependencies
• lack of muti-node integration testing
1

6. Causes
2

7. Causes
• no single source of truth
• partial definitions yield inconsistencies (over time)
• shared mutable state (file manipulation most Linux/UNIX distros)
2

8. Nix: Reliable Packaging

9. Nix: What is it?
3

10. Nix: What is it?
• "The Purely Functional Package Manager" (https://nixos.org/nix)
3

11. Nix: What is it?
• "The Purely Functional Package Manager" (https://nixos.org/nix)
• (And lazily evaluated expression language)
3

12. Nix: Source Example
This Nix lambda denotes the hello package
# Named keyword arguments: all non -optional
{ stdenv , fetchurl , perl }: let
version = "2.10";
in stdenv.mkDerivation {
inherit perl;
name = "hello -${version}";
builder = ./builder.sh;
src = fetchurl {
url = "mirror://gnu/hello/${name}.tar.gz";
sha256 = "0ssi1wpaf7plaswqqjw...l7c9lng89ndq1i";
};
}
4

13. Nix: Derivation drvPath
Inspecting hello package drvPath
$ nix -repl ’<nixpkgs >’
nix -repl > hello.drvPath
"/nix/store /3mdx...-hello -2.10. drv"
nix -repl > hello.outPath
"/nix/store/kmwd ...-hello -2.10"
nix -repl > :q
$ stat -t /nix/store /3mdx...-hello -2.10. drv
/nix/store /3mdx...-hello -2.10. drv 1219 8 8124 0 0 fe00 13940452 ...
5

14. Nix: Package outPath
Lazy evaluation means we only build the packages we need
$ stat -t /nix/store/kmwd ...-hello -2.10
stat: cannot stat ’/nix/store/kmwd ...-hello -2.10 ’: No such file or ...
To evaluate or realize the package we nix build
$ nix build nixpkgs.hello
[1 copied (0.2 MiB), 0.0 MiB DL]
Now we can see the hello package outPath in the Nix store
$ stat -t /nix/store/kmwd ...-hello -2.10
/nix/store/kmwd ...-hello -2.10 4096 8 416d 0 0 fe00 14115111 4 0 0 ...
6

15. Nix: Package outPath
The high-level layout of the hello.outPath
$ tree --du -hugFDL 2 /nix/store/kmwd ...-hello -2.10
/nix/store/ kmwd1hq55akdb9sc7l3finr175dajlby -hello -2.10
|-- [root root 33K Dec 31 1969] bin/
| |-- [root root 29K Dec 31 1969] hello*
|-- [root root 16K Dec 31 1969] share/
|-- [root root 4.0K Dec 31 1969] info/
|-- [root root 4.0K Dec 31 1969] locale/
|-- [root root 4.0K Dec 31 1969] man/
53K used in 5 directories , 1 file
7

16. Nix: Package in user profile
Since it is built, surely we can run the binary? (No, not yet)
$ hello
The program ’hello ’ is currently not installed. It is provided by ...
To link into the user profile we install ("Look Ma, no hands/sudo")
$ nix -env -iA nixos.hello
installing ’hello -2.10 ’
building ’/nix/store /6 s28kd3hdnv5cpd ...1yi5 -user -environment.drv ’...
created 6852 symlinks in user environment
$ hello -v
hello (GNU Hello) 2.10
8

17. Nix: Properties
9

18. Nix: Properties
• Lazily evaluated
• Nix store is immutable!
• nixpkgs is an expression: snapshot of community pkgs plus overlayed packages
9

19. Nix: Properties
• Lazily evaluated
• Nix store is immutable!
• nixpkgs is an expression: snapshot of community pkgs plus overlayed packages
• When packages sandboxed:
• chroot -ed
• private /proc
• disabled network/IPC/etc
• no env var inheritance
9

20. Nix: Properties
• Lazily evaluated
• Nix store is immutable!
• nixpkgs is an expression: snapshot of community pkgs plus overlayed packages
• When packages sandboxed:
• chroot -ed
• private /proc
• disabled network/IPC/etc
• no env var inheritance
• Same inputs and src and definition yields same result (outPath and content)
• Different inputs or src or definition yields different result (outPath and/or
content)
• Equational reasoning allows binary substitution
9

21. Nix Shell: Consistent Envs

22. nix-shell: Getting started
• CI scripts can use nix-shell as shebang
#!/usr/bin/env nix -shell
#!nix -shell -p python27Full python27Packages .boto3
#!nix -shell -I /path/to/pinned/nixpkgs
#!nix -shell -i python
import boto3
ec2 = boto3.client(’ec2’)
response = ec2. describe_instances ()
print(response)
Note: Approach requires Nix/NixOS installed on builders
10

23. nix-shell: Running (first time)
$ chmod u+x boto.py
$ ./ boto.py
these paths will be fetched (7.70 MiB download , 45.59 MiB unpacked ):
/nix/store/mj8vmgxff9m ...-python -2.7.14
/nix/store/q6js3yms8xd ...-tix -8.4.3
/nix/store/yqgj43ancc0 ...- python2 .7-boto3 -1.4.8
copying path ’/nix/store/yqg...- python2 .7-boto3 -1.4.8 ’ from ’https :// cache.n
copying path ’/nix/store/q6j...-tix -8.4.3 ’ from ’https :// cache.nixos.org ’...
copying path ’/nix/store/mj8...-python -2.7.14 ’ from ’https :// cache.nixos.org
OUTPUT HERE
11

24. nix-shell: Running (subsequent times)
$ ./ boto.py
OUTPUT HERE
12

25. Nix Shell: Recap
• Claim: we can get consistent environment, assuming:
• our core Nix expression pins version of nixpkgs
• devs reload shell at the git revisions to pick up dependency changes
• CI server runs tests using shell.nix referenced by nix-shell at that revision
• infrastructure for all environments is built using pinned Nix expression
• devenv, CI, production-like system builds share the same core expression
13

26. NixOS

27. What is NixOS?
• "The Purely Functional Linux Distribution" (https://nixos.org/)
• Provides congruent system-level configuration
• System-level rollbacks
14

28. NixOS: Example (nginx)
# ./machines/nginx.nix
{ pkgs , ... }:
{ boot.kernelPackages = pkgs.linuxPackages_4_15;
time.timeZone = "UTC";
services.nginx = {
enable = true;
virtualHosts."my.domain.tld" = {
forceSSL = true;
enableACME = true;
locations."/assets".root = "/var/www/assets";
locations."/".proxyPass = "http://backend -lb:3000";
};
};
} 15

29. NixOS: Example (Scala microservice)
# ./machines/scala -services.nix
{ pkgs , ... }: let
inherit (builtins) listToAttrs map;
inherit (pkgs.myorg) scalaServices mulib;
in {
# Interal custom myorg module; overlayed into pkgs
myorg.defaults.enable = true;
users.extraUsers = map mulib.mkSystemUser scalaServices;
systemd.services = map mulib.mkServiceUnit scalaServices;
}
16

30. NixOS: Example (memcached)
# ./machines/memcached.nix
{ config , pkgs , ... }:
{
services.memcached.enable = true;
services.memcached.maxConnections = 2048;
services.memcached.maxMemory = 256;
services.memcached.listen =
config.networking.eth1.ipAddress;
}
17

31. NixOS: Containers (composition)
{ config , pkgs , ... }:
{
containers.cache.config = import ./memcached.nix;
containers.proxy.config = import ./nginx.nix;
containers.services = {
config = import ./scala -services.nix;
bindMounts."/webapp" = {
hostPath = pkgs.myorg.scalaServices.webapp.outPath;
isReadOnly = true;
};
};
}
18

32. NixOS: Multi-node testing (node setup)
import ./make -test.nix ({ pkgs , lib , ... }:
let
kafkaPackage = pkgs.apacheKafka_1_0;
in {
name = "kafka -1.0";
nodes = {
zookeeper = import ./machines/zookeeper.nix;
kafka = import ./machines/kafka.nix;
};
testScript = import ./tests/kafka.nix {
package = kafkaPackage;
util = pkgs.myorg.testing.util;
};
}) 19

33. NixOS: Multi-node testing (test script)
{ pkg , kPort ? 9092 , zkPort ? 2181 , topic ? "test" , util }:
let
inherit (util) createTopicCmd producerPipe consumerPipe;
in ’’
startAll;
$zookeeper ->waitForUnit("zookeeper");
$zookeeper ->waitForOpenPort(${zkPort});
$kafka ->waitForUnit("apache -kafka");
$kafka ->waitForOpenPort(${kPort});
$kafka ->waitUntilSucceeds("${createTopicCmd}");
$kafka ->mustSucceed("echo ’test 1’ | ${producerPipe}");
$kafka ->mustSucceed("${consumerPipe} | grep ’test 1 ’");
’’
20

34. NixOS & nixpkgs: Properties
• modular (See nixpkgs overlays)
• extensible (see nixos modules)
• everything (almost) is a package in the Nix store
• system rollbacks (with system profile snapshots saved as generations)
• builds on purity of Nix packaging
• NO FHS mutation-in-place (links into FHS to immutable Nix store)
21

35. Reliability

36. Reliable Software
"Those who want really reliable software will discover that they must find means
of avoiding the majority of bugs to start with, and as a result the programming
process will become cheaper." – EWD
22

37. Why do properties/laws matter?
23

38. Why do properties/laws matter?
• Because it allows us humble programmers to:
23

39. Why do properties/laws matter?
• Because it allows us humble programmers to:
• compose our reasoning
• know what we cannot reason about
• break down big problems into smaller parts
• accomplish the same result with less effort
23

40. Questions?
@SusanPotter (heckle me, ask me questions)
24