Talk from Docker meetup Jakarta. Presented and demoed various Linux kernel features that enable container runtime, i.e. chroot, namespaces, cgroups, capabilities.

1. Containers from Scratch:
what are they made from?
GIRI KUNCORO

2. Giri Kuncoro
Senior System Engineer
@GO-PAY Core Infrastructure
github.com/girikuncoro

3. Build Your Own Smartphone
http://www.instructables.com/id/Build-Your-Own-Smartphone

4. Raspberry Pi A+ 256 MB
Adafruit FONA - Mini GSM Breakout
GSM Antenna
Electret Microphone
1200 mAh Lithium Ion Battery
Ingredients

5. Today
Build a container without a container runtime,
i.e. Docker, lxc, rkt

6. Ingredient #1: Container Image
TL;DR: nothing but tarball
● Application metadata
● Filesystem

7. Ingredient #1: Container Image
Container filesystem:
looks like an OS; but no kernel, no init system

8. Ingredient #1: Container Image
Build root: http://www.buildroot.org/
Debootstrap: https://wiki.debian.org/Debootstrap
YUM / DNF
Gentoo: https://www.gentoo.org/downloads/
Buildah: https://github.com/projectatomic/buildah

9. $ mkdir rootfs
$ sudo dnf -y
--installroot=$PWD/rootfs
--releasever=24 install
@development-tools
procps-ng
python3
which
iproute
net-tools
$ ls rootfs

10. Ingredient #2: chroot
Execute a process in our container filesystem
chroot(2): http://man7.org/linux/man-pages/man2/chroot.2.html

11. $ sudo chroot rootfs /bin/bash

12. Ingredient #3: namespaces
Limit the “view” of a container:
Process namespace (pid)
Network namespace (net)
Mount namespace (mnt)
https://en.wikipedia.org/wiki/Linux_namespaces

13. Ingredient #3: namespaces
chroot of other systems:
clone(2): http://man7.org/linux/man-pages/man2/clone.2.html
unshare(2): http://man7.org/linux/man-pages/man2/unshare.2.html
Process trees
Network interfaces
Mount volumes

14. $ sudo unshare -p -f
--mount-proc=$PWD/rootfs/proc
chroot rootfs /bin/bash

15. Ingredient #4: enter namespaces
Namespaces are composable
Example: Kubernetes pod
setns(2): http://man7.org/linux/man-pages/man2/setns.2.html
k8s pod
di r p o s ,
di r c o t
sa t o k,
sa un

16. # PID=321
# ls /proc/$PID/ns
cgroup ipc mnt net pid user uts
# nsenter
--pid=/proc/$PID/ns/pid
--mnt=/proc/$PID/ns/mnt
chroot $PWD/rootfs /bin/bash

17. Ingredient #5: volume mounts
Inject files into our chroot
$ docker run -d
--name=nginxtest
-v nginx-vol:/usr/share/nginx/html
nginx:latest

18. apiVersion: v1
kind: Pod
metadata:
name: test-pd
spec:
containers:
- image: k8s.gcr.io/test-webserver
name: test-container
volumeMounts:
- mountPath: /test-pd
name: test-volume
volumes:
- name: test-volume
hostPath:
path: /data

19. # nsenter --mount=/proc/$PID/ns/mnt
mount --bind -o ro
$PWD/readonlyfiles
$PWD/rootfs/var/readonlyfiles

20. Ingredient #6: cgroups
Restrict resources for processes

21. # ls /sys/fs/cgroup
# mkdir /sys/fs/cgroup/memory/demo
# echo $$ > /sys/fs/cgroup/memory/demo/tasks
# cat /proc/self/cgroup

22. # CGROUP=/sys/fs/cgroup/memory/demo
# echo “100000000” >
$CGROUP/memory.limit_in_bytes
# echo “0” > $CGROUP/memory.swappiness
# python3 hungry.py

23. Ingredient #7: cgroup namespace
Q: How do you restrict a process from reassigning
cgroup?
A: More namespaces!

24. # unshare -C
# cat /proc/self/cgroup

25. # (how to remove cgroups: reassign)
# echo $$ > /sys/fs/cgroup/memory/tasks
# rmdir /sys/fs/cgroup/memory/demo

26. Ingredient #8: capabilities
“Docker is about running random code downloaded
from Internet and running it as root” - Dan Walsh (Red
Hat)

27. Ingredient #8: capabilities
SELinux, seccomp, AppArmor should’ve been covered
Show Linux capabilities instead
http://man7.org/linux/man-pages/man7/capabilities.7.html

28. $ go build -o /tmp/listen listen.go
$ sudo setcap cap_net_bind_service=+ep
/tmp/listen
$ getcap /tmp/listen

29. $ sudo capsh --print
$ sudo capsh --drop=cap_chown --

30. Ingredient #9: network namespace
Huge topic, will do simple demo for now
For the impatient, probably next talk:
https://github.com/girikuncoro/netns-demo

31. $ sudo unshare -n chroot rootfs
# ip addr
# ip link set dev lo up

32. $ sudo ip link add veth0 type veth peer name
veth1
$ sudo ip link set veth1 netns $PID
$ sudo ip address add 10.1.1.2/24 dev veth0
$ sudo ip link set dev veth0 up
# (inside namespace)
# ip address add 10.1.1.3/24 dev veth1
# ip link set dev veth1 up

33. Conclusion
Containers are a combination between Linux kernel
features
Docker, rkt, lxc (container runtime) are just opinionated
wrapper around these

34. References
Containers from scratch, Eric Chiang
https://ericchiang.github.io/post/containers-from-scratch/
Building minimal containers, Brian Redbeard
https://github.com/brianredbeard/minimal_containers
Namespaces in operation, Michael Kerrisk
https://lwn.net/Articles/531114/
cgroups v1, Paul Menage
https://www.kernel.org/doc/Documentation/cgroup-v1/cgroups.txt
Bocker, Docker implemented in 100 lines of bash
https://github.com/p8952/bocker

35. Thanks!
giri.kuncoro@go-jek.com
twitter.com/girikuncoro
github.com/girikuncoro
Our team is hiring, come talk to us or open www.go-jek.com/careers