While probably the most prominent, Docker is not the only tool for building and managing containers. Originally meant to be a "chroot on steroids" to help debug systemd, systemd-nspawn provides a fairly uncomplicated approach to work with containers. Being part of systemd, it is available on most recent distributions out-of-the-box and requires no additional dependencies. This deck will introduce a few concepts involved in containers and will guide you through the steps of building a container from scratch. The payload will be a simple service, which will be automatically activated by systemd when the first request arrives.

1. Containers with
systemd-nspawn
Gábor Nyers
Consultant & Trainer @Trebut
gnyers@trebut.com
@gabornyers

2. Agenda
● An example systemd-nspawn container
● What is systemd-nspawn and systemd
● Related Concept: Kernel CGroups
● Bootable containters
● Containers as Service
● Advanced topic: Socket Activation

3. 3
An example
systemd-nspawn
container

4. 4
A Simple Application Container
• Start up container
• List of processes
• Try to install package
‣ Limited footprint and exposure!
• On host OS: list kernel control
groups:
# systemd-nspawn -jD /srv/containers/opensuse132/
-M opensuse132c0
/bin/bash
# opensuse132c0:~ # ps -ef
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 04:16 ? 00:00:00 -bash
root 43 1 0 04:18 ? 00:00:00 ps -ef
#
opensuse132c0:~ # zypper install wget
-bash: zypper: command not found
#
# machinectl
MACHINE CONTAINER SERVICE
opensuse132c0 container nspawn
1 machines listed.
physnode1:~ #
# systemd-cgls
├─1 /usr/lib/systemd/systemd --switched-root --system --deserialize
21
├─machine.slice
│ └─machine-opensuse132c0.scope
│ └─18329 -bash
[…]
# ps -ef -o pid,ppid,machine,cmd

5. 5
Create application container
• Bootstrap directory
• Install a few packages
# zypper --root /srv/containers/opensuse132/ addrepo
http://download.opensuse.org/distribution/13.2/repo/oss/
repo-oss
# zypper --root /srv/containers/opensuse132/ addrepo
http://download.opensuse.org/distribution/13.2/repo/non-oss/
repo-non-oss
# zypper --root /srv/containers/opensuse132/ install
openSUSE-release-13.2 install bash procps coreutils vim

6. 6
systemd-nspawn • What is systemd?
• What is systemd-nspawn?
• Adoption

7. 7
What is systemd? 1/3
• a system- and session manager for Linux,
• provides aggressive parallelization capabilities,
(no shell during boot!)
• uses socket and D-Bus activation for starting services,
• offers on-demand starting of services,
• keeps track of processes using Linux cgroups,

8. 8
What is systemd? 2/3
• supports restoring the system's state to a predefined state,
• maintains mount and auto-mount points,
• provides dependency based service control logic,
• provides replacement for a nr. of well-known tools, e.g.:
udev, automount, inetd, consolekit and syslog,
• a drop-in replacement for sysvinit

9. 9
What is systemd? 3/3
There is a lot of criticism and opinions as well...
• “It's not the UNIX way”
referring to the “do one thing and do it well” maxim
• “It's monolithic”
• “It introduces too many dependencies”
• (and worse)
... but we won't be addressing these today :-)

10. 10
An aside: People and Innovation...
“If I had asked people
what they wanted, they
would have said faster
horses”
Henry Ford

11. 11
What is systemd-nspawn?
• “chroot on steroids...”
• Invented for debug and test of systemd development
• Turns out to be a great container manager
• systemd-nspawn vs. docker
‣ Management container vs. container+images
‣ Inherited networking vs. Need to set up networking

12. 12
systemd adoption
Distribution Added to repositories Enabled by default? Released as default
SUSE Linux
Enterprise
v12 Yes Yes
openSUSE v11.4 Yes v12.2 (2012)
Fedora v15 (2011) Yes v15 (2011)
Red Hat Linux
Enterprise
v7 (2014) Yes v7 (2014)
Debian in 2012 Yes v8 (2015)
Arch Linux in 2012 Yes 2012
Ubuntu v13.04 (2013) Yes v15.04 (2015)
see also: http://en.wikipedia.org/wiki/Systemd#Adoption_and_reception

13. 13
Related Concept • Kernel cgroups
(independent of systemd)

14. 14
Kernel Cgroups (Control Groups)
• Linux Kernel facility allowing
the grouping of processes
(and their “children”) into a
tree-structure hierarchy
• Each group can be assigned a
quota for these system
resources:
‣ CPU
‣ RAM
‣ Disk I/O
‣ Network I/O
Control groups hierarchy created by systemd
├─machine.slice
│ └─machine-qemux2dsles1201.scope
│ └─20958 /usr/bin/qemu-system-x86_64 -m...
├─user.slice
│ ├─user-0.slice
│ │ └─user@0.service
│ │ ├─4322 /usr/lib/systemd/systemd --us...
│ │ └─4323 (sd-pam)
│ ├─user-1000.slice
│ │ ├─session-560.scope
│ │ │ ├─ 2810 /usr/bin/claws-mail
│ │ │ ├─ 3035 /usr/lib64/firefox/firefox
│ │ │ ├─ 3086 /usr/lib/mozilla/kmozillahel...
│ │ │ ├─ 5459 /bin/bash
│ │ │ ├─ 7854 /usr/bin/kwalletmanager --kw...
│ │ ├─session-1.scope
│ │ │ ├─4179 /bin/bash ./bridge start
│ │ │ └─4182 dnsmasq --conf-file=mydnsmasq...
│ │ └─user@1000.service
│ │ ├─1891 /usr/lib/systemd/systemd --us...
│ │ └─1892 (sd-pam)
│ └─user-489.slice
│ └─user@489.service
│ ├─1703 /usr/lib/systemd/systemd --us...
│ └─1704 (sd-pam)
└─system.slice
├─libvirtd.service
│ └─4008 /usr/sbin/libvirtd --listen
├─rsyslog.service
│ └─985 /usr/sbin/rsyslogd -n
├─apache2.service
│ ├─1254 /usr/sbin/httpd2-prefork -f /et...
│ └─1840 /usr/sbin/httpd2-prefork -f /et...

15. 15
Bootable
containers

16. 16
Bootable OS container [1/4]
Bootstrap
• Host properties
• Install YUM
• Bootstrap RPM DB
• Install CentOS 7 release
package
• Install a few package and their
dependencies
# hostnamectl
Static hostname: physnode1.trebut.com
Icon name: computer-laptop
Chassis: laptop
Machine ID: b4ea4eb15ab7c29b6cc20a47544e5eb7
Boot ID: 3c4e7b5067d247939b89d7e7b57c1132
Operating System: openSUSE 13.2 (Harlequin) (x86_64)
CPE OS Name: cpe:/o:opensuse:opensuse:13.2
Kernel: Linux 3.16.7-7-desktop
Architecture: x86-64
# zypper install yum
# rpm --root /srv/containers/centos/ --initdb
# rpm --root /srv/containers/centos/
-ihv
http://mirror.centos.org/centos/7.1.1503/os/x86_64/Packages/centos-
release-7-1.1503.el7.centos.2.8.x86_64.rpm
# yum -y --nogpg --releasever=7
--installroot=/srv/containers/centos/
install systemd passwd yum vim-minimal

17. 17
Bootable OS container [2/4]
Boot container
• Boot container
‣ systemd-nspawn
-bD /srv/containers/centos/
# systemd-nspawn -bD /srv/containers/centos/
systemd 208 running in system mode. (+PAM +LIBWRAP +AUDIT +SELINUX
+IMA +SYSVINIT +LIBCRYPTSETUP +GCRYPT +ACL +XZ)
Detected virtualization 'systemd-nspawn'.
Welcome to CentOS Linux 7 (Core)!
Set hostname to <centos7c0>.
[ OK ] Reached target Remote File Systems.
[ OK ] Created slice Root Slice.
[ OK ] Created slice User and Session Slice.
[ OK ] Created slice System Slice.
[ OK ] Created slice system-getty.slice.
[ OK ] Reached target Slices.
[ OK ] Listening on Delayed Shutdown Socket.
[ OK ] Listening on /dev/initctl Compatibility Named Pipe.
[ OK ] Listening on Journal Socket.
Starting Journal Service...
[ OK ] Started Journal Service.
[ OK ] Reached target Paths.
Mounting Debug File System...
Mounting FUSE Control File System...
Starting Create static device nodes in /dev...
Mounting POSIX Message Queue File System...
[...]
[ OK ] Started Login Service.
[ OK ] Started Permit User Sessions.
Starting Console Getty...
[ OK ] Started Console Getty.
[ OK ] Reached target Login Prompts.
[ OK ] Reached target Multi-User System.
CentOS Linux 7 (Core)
Kernel 3.16.7-7-desktop on an x86_64
centos7c0 login:

18. 18
Bootable OS container [3/4]
Instance properties
OS Properties from inside the
container
CentOS Linux 7 (Core)
Kernel 3.16.7-7-desktop on an x86_64
centos7c0 login: root
Password:
Last login: Sat Apr 11 23:22:04 on console
-bash-4.2#
-bash-4.2# hostnamectl
Static hostname: centos7c0
Icon name: computer-container
Chassis: container
Machine ID: afb4a0719ad842c99dd7cc704919a2fe
Boot ID: 7c03b147c9114632b96bbeb2a462cf5a
Virtualization: systemd-nspawn
Operating System: CentOS Linux 7 (Core)
CPE OS Name: cpe:/o:centos:centos:7
Kernel: Linux 3.16.7-7-desktop
Architecture: x86_64
-bash-4.2#
Container properties
# machinectl
MACHINE CONTAINER SERVICE
centos container nspawn
1 machines listed.
physnode1:~
# systemd-cgls
├─1 /usr/lib/systemd/systemd --switched-root --system
--deserialize 21
├─machine.slice
│ └─machine-centos.scope
│ ├─10159 /usr/lib/systemd/systemd
│ └─system.slice
│ ├─dbus.service
│ │ └─10184 /bin/dbus-daemon --system --address=systemd:
--nofork --nopidfile --systemd-activation
│ ├─systemd-journald.service
│ │ └─10167 /usr/lib/systemd/systemd-journald
│ ├─systemd-logind.service
│ │ └─10183 /usr/lib/systemd/systemd-logind
│ └─console-getty.service
│ └─10189 /sbin/agetty --noclear --keep-baud console
115200 38400 9600
├─system.slice

19. 19
Bootable OS container [4/4]
Shutdown container
• Shutdown container from the
inside:
‣ Type: `init 0` or `poweroff`
Note: will require running init in
container
‣ Type: ^]^]^] ( 3x CTRL+[ )
• Shutdown container from the
host
‣ machinectl terminate $CONT
-bash-4.2# init 0
[ OK ] Removed slice user-0.slice.
[ OK ] Removed slice system-getty.slice.
Stopping Hostname Service...
[ OK ] Stopped target Graphical Interface.
[ OK ] Stopped target Multi-User System.
[ OK ] Stopped target Login Prompts.
Stopping Console Getty...
Stopping Login Service...
Stopping D-Bus System Message Bus...
[ OK ] Stopped Login Service.
[ OK ] Stopped D-Bus System Message Bus.
[ OK ] Stopped Console Getty.
Stopping Permit User Sessions...
[ OK ] Stopped Permit User Sessions.
[ OK ] Stopped target Remote File Systems.
[ OK ] Stopped Hostname Service.
[ OK ] Stopped target Basic System.
[ OK ] Stopped target Slices.
[ OK ] Removed slice User and Session Slice.
[ OK ] Stopped target Paths.
[ OK ] Stopped target Timers.
[ OK ] Stopped target Sockets.
[ OK ] Closed D-Bus System Message Bus Socket.
[ OK ] Stopped target System Initialization.
[ OK ] Stopped target Encrypted Volumes.
Stopping Load/Save Random Seed...
Stopping Update UTMP about System Reboot/Shutdown...
[ OK ] Stopped target Swap.
[ OK ] Stopped Update UTMP about System Reboot/Shutdown.
[ OK ] Stopped Load/Save Random Seed.
Stopping Create Volatile Files and Directories...
[ OK ] Stopped Create Volatile Files and Directories.
[ OK ] Reached target Shutdown.
physnode1:/srv/containers #

20. 20
Networking and systemd-nspawn containers
Networking in container
-bash-4.2# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state
UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: wlp12s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
mq state UP qlen 1000
link/ether 00:11:22:33:44:55 brd ff:ff:ff:ff:ff:ff
inet 10.1.2.27/21 brd 10.1.7.255 scope global dynamic
wlp12s0
valid_lft 14611sec preferred_lft 14611sec
inet6 fe80::224:d6ff:fe89:521e/64 scope link
valid_lft forever preferred_lft forever
3: enp0s25: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500
qdisc pfifo_fast state DOWN qlen 1000
link/ether 00:aa:bb:cc:dd:ee brd ff:ff:ff:ff:ff:ff
-bash-4.2# md5sum /etc/resolv.conf
a92a6e440cd677ad17748aa29c5a7333 /etc/resolv.conf
‣ By default the nspawn container will inherit the network settings
‣ /etc/resolv.conf will be copied into container
Networking at Host OS
physnode1:~ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state
UNKNOWN group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: wlp12s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
mq state UP group default qlen 1000
link/ether 00:11:22:33:44:55 brd ff:ff:ff:ff:ff:ff
inet 10.1.2.27/21 brd 10.1.7.255 scope global dynamic
wlp12s0
valid_lft 14433sec preferred_lft 14433sec
inet6 fe80::224:d6ff:fe89:521e/64 scope link
valid_lft forever preferred_lft forever
3: enp0s25: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500
qdisc pfifo_fast state DOWN group default qlen 1000
link/ether 00:aa:bb:cc:dd:ee brd ff:ff:ff:ff:ff:ff
physnode1:~ # md5sum /etc/resolv.conf
a92a6e440cd677ad17748aa29c5a7333 /etc/resolv.conf

21. 21
More advanced networking
‣ Create a virtual ethernet device, with name “vb-$machinename”
‣ Connect veth device to bridge “virbr0”
systemd-nspawn -bD /srv/containers/opensuse132/
--network-bridge=virbr0 --network-veth
virbr0
veth
(host0)
veth
(vb-opensuse132c0)
opensuse132
physnode1
opensuse132c0:~ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group [...]
2: host0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen
1000
link/ether 36:e3:35:8d:8e:95 brd ff:ff:ff:ff:ff:ff
opensuse132c0:~ #
physnode1:~ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group
[...]
29: vb-opensuse132c0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc
pfifo_fast master virbr0 state DOWN group default qlen 1000
link/ether 0a:62:90:a4:b5:72 brd ff:ff:ff:ff:ff:ff
physnode1:~ #

22. 22
journald and systemd-nspawn containers
• Integrating the journal of the
host and the container
# systemd-nspawn
-bD /srv/containers/centos
--link-journal=host

23. 23
Containers as
Service

24. 24
Container as service
• Install Apache and a few other
packages
• Create a machine-id for the
container
• Create systemd unit file
#install Apache
zypper --root /srv/containers/opensuse132/ install
openSUSE-release-13.2 apache2 timezone iproute2 rsyslog
# set up machine-id
systemd-nspawn -D /srv/containers/opensuse132/
systemd-machine-id-setup
# unit file:
cat <<EOF > /etc/systemd/system/opensuse132c0.service
[Unit]
Description=Start an openSUSE 13.2 container
Wants=network.target nss-lookup.target
After=network.target nss-lookup.target
[Service]
Type=notify
PrivateTmp=true
ExecStart=/usr/bin/systemd-nspawn -M opensuse132c0
-jD /srv/containers/opensuse132/
ExecStop=/usr/bin/machinectl terminate opensuse132c0
[Install]
WantedBy=machines.target
EOF

25. 25
Managing containers
nsenter
• nsenter - run program with
namespaces of other
processes
# machinectl
MACHINE CONTAINER SERVICE
opensuse132c0 container nspawn
1 machines listed.
# machinectl status opensuse132c0
opensuse132c0
Since: Sun 2015-04-12 03:54:18 CEST; 37s ago
Leader: 17717 (systemd)
Service: nspawn; class container
Root: /srv/containers/opensuse132
Unit: machine-opensuse132c0.scope
├─17717 /usr/lib/systemd/systemd
└─system.slice
├─dbus.service
[…]
# nsenter --target 17717 --mount --uts --ipc --net –pid
opensuse132c0:/ #
opensuse132c0:/ # systemctl disable rsyslog
rm '/etc/systemd/system/multi-user.target.wants/rsyslog.service'
rm '/etc/systemd/system/syslog.service'
opensuse132c0:/

26. 26
Summary
systemd-nspawn
• Makes containers easy
• Everyone familiar with “chroot” instantly “gets” systemd-nspawn
• Does not have special dependencies, like e.g. docker
• It is available on all modern Linux distro's

27. Thank you.
27
Questions?