LinuxKit, a toolkit for building custom minimal, immutable Linux distributions. Secure defaults without compromising usability Everything is replaceable and customisable Immutable infrastructure applied to building Linux distributions Completely stateless, but persistent storage can be attached Easy tooling, with easy iteration Built with containers, for running containers Designed for building and running clustered applications, including but not limited to container orchestration such as Docker or Kubernetes Designed from the experience of building Docker Editions, but redesigned as a general-purpose toolkit Designed to be managed by external tooling, such as Infrakit or similar tools Includes a set of longer-term collaborative projects in various stages of development to innovate on kernel and userspace changes, particularly around security

1. Declare Your Infrastructure pt 2:
LinuxKit Swarm Nodes
Dave Freitag, IBM

2. Overview and Comparison
Ubuntu

2GB Image size

3-5 min provision

5 minutes setup time

Runtime Package
Updates

Base security
LinuxKit

190MB Image size

1.5-3 min provision

1 minute setup time

Build-time Package
Updates

Built-in Security
•Swarm nodes on Ubuntu
•LinuxKit Investigation and Future

3. LinuxKit Swarm Node Architecture
Linux Kernel 4.9.x
init runc containerd ca-certs
Kernel
Init
sysctlOnboot
Services
rngd
docker ssh
syslog
sysfs mount metadata
ntpd
getty
nfs
logrotate
iptables
dns

4. Scenarios
•Synchronization

Need access to NFS to store/retrieve infrakit files
prior to starting docker
•Serviceability

Need to keep system and audit logs for diagnostics

Administrator needs command line access

5. Provision Flow - Initial Configuration
resource "ibm_compute_vm_instance" "manager" {
hostname = "${var.name}-mgr1"
image_id = "${var.image_id}"
datacenter = "${var.datacenter}"
...
user_metadata = <<EOD
cat << EOF > ${var.working_dir}/d4ic-vars.json
{
"worker_size":${var.worker_count},
"manager_size":${var.manager_count},
"nfs_id":${nfs_manager.id},
"nfs_mountpoint":"${nfs_manager.mountpoint}",
...
EOD
Terraform
•Initial configuration supplied via user metadata

6. Provision Flow - Boot
on-boot metadata
onboot: - name: metadata
image: ibm-metadata:latest
binds:
# Cloud drive
- /dev:/dev
# DNS
- /etc/resolv.conf:/etc/resolv.conf
# Logs
- /var/log:/var/log
# SSH Keys
- /var/ibm/.ssh:/var/.ssh
# Userdata
- /var/ibm/metadata:/var/ibm/metadata
# Networking
- /var/ibm/network:/var/ibm/network
# For NFS
- /var/ibm/etc/nfs:/etc/nfs

Cloud metadata container configures network

User metadata copied to filesystem for execution
later
/var/*Network
Cloud drive
/dev/xvdh

7. onboot:
- name: metadata
image: ibm-metadata:latest
...
binds:
- /var/ibm/metadata:/var/ibm/metadata
- /var/ibm/etc/nfs:/etc/nfs
services:
- name: docker
image: ibm-docker:latest
...
binds:
- /var:/var:rshared,rbind
- /var/ibm/etc/nfs:/etc/nfs
- name: nfs
image: ibm-nfs:latest
...
binds:
- /var:/var:rshared,rbind
- /var/ibm/etc/nfs:/etc/nfs
Data Flow - Filesystem Access
/var
metadata
nfs
docker

Onboot and service
containers share access to
common storage volume

8. Data Flow - Initial Configuration
docker
metadata
nfs
Cloud-init Drive
/dev/xvdh
SCOPE=docker sh userdata.sh
...
if [ "$SCOPE" == "docker" ]; then
/configure-docker.sh
fi
SCOPE=nfs sh userdata.sh
...
if [ "$SCOPE" == "nfs" ]; then
/nfs.sh $NFS_MOUNTPOINT $LOCAL_MOUNTPOINT
fi

Each service container runs their own portion of
the configuration script

9. Data Flow - Synchronization
docker
metadata
nfs
if [ -f /etc/nfs/nfslock ]; then
echo "NFS locked, waiting for NFS."
while true; do
grep "0" /etc/nfs/nfslock
if [ $? -ne 0 ]; then
echo "NFS ready, continuing."
break
fi
echo "NFS not ready, waiting."
sleep 2
done
fi
if [ "$NFS" == "1" ]; then
echo "0" > /etc/nfs/nfslock
fi
/sbin/rpcbind -d
if [ -f $USERDATA_FILE ]; then
"SCOPE=nfs sh $USERDATA_FILE
fi
./nfs.sh $MOUNT_POINT $LOCAL_MOUNTPOINT
# Finished mounting NFS.
echo "Unlocking NFS."
echo "1" > /etc/nfs/nfslock

Metadata initializes lock for NFS

Docker waits on NFS lock before continuing

10. 25GB
</dev/xvda1>
/var
mount
syslog
logrotate
Serviceability - Logging and Auditing

Persistent storage volume mounted at boot

All on-boot and service containers log to /var/log

Container logs to /var/lib/docker/...

Syslog captures kernel output, boot logs

Logrotate keeps file sizes in check
docker

11. Serviceability - Administration

SSH container with public-key
access

Console (Getty) container
available for private network
access only
ssh
Console
Internet
Private 10.x.x.x
Network

12. Serviceability - Network Overrides

Provide ability to respond to external network
failures

dnsmasq service with hosts file
InternetSwarm
DNS
HTTP Service

13. Demo, Q/A? Thanks!