There are multiple reasons one might want to run VMs inside containers. One scenario could be running VMs as part of a CI pipeline which executes all build steps inside containers. This can be useful, for example, for emulating bare-metal deployments. In this talk, I will demonstrate how to create a virtual machine inside a Docker container using KVM as the hypervisor driver, explain how the container needs to be configured in terms of privileges and discuss some of the challenges involved.

1. OSDC Berlin | May 14 - 15 2019

2. Kosisochukwu Anyanwu
Software Engineer, Kinvolk
Github: kosyfrances
Twitter: kosyfrances
Email: kosy@kinvolk.io

3. The Deep-stack Kubernetes
Experts
Engineering services and products for
Kubernetes, containers, process
management and Linux user-space +
kernel
Blog: kinvolk.io/blog
Github: kinvolk
Twitter: kinvolkio
Email: hello@kinvolk.io

4. Containers and VMs

8. Also known as Virtual Machine Monitor.
A process that creates and runs virtual machines.
Shares host machine’s resources.
Examples: VirtualBox, VMware, KVM.

9. Kernel-based Virtual Machine
Lets you turn Linux into a hypervisor

10. root@591532:/# egrep -c '(vmx|svm)' /proc/cpuinfo
vmx – Intel VT-x, virtualisation support enabled in BIOS
svm – AMD SVM,virtualisation enabled in BIOS.
If the count is 0, then your CPU does not support hardware virtualisation.

11. Source: Docker

12. VM inside a Container?

13. Examples:
You want a VM in a CI that runs everything in containers.
Your OS runs everything in containers and you want a
VM

14. How can this be done?

16. FROM ubuntu:18.04
RUN apt-get update -y &&
DEBIAN_FRONTEND=noninteractive apt-get
install -y qemu-kvm libvirt-daemon-system
libvirt-clients bridge-utils vagrant &&
apt-get autoclean &&
apt-get autoremove &&
vagrant plugin install vagrant-libvirt
COPY startup.sh /
ENTRYPOINT ["/startup.sh" ]

17. FROM ubuntu:18.04
RUN apt-get update -y &&
DEBIAN_FRONTEND=noninteractive apt-get
install -y qemu-kvm libvirt-daemon-system
libvirt-clients bridge-utils vagrant &&
apt-get autoclean &&
apt-get autoremove &&
vagrant plugin install vagrant-libvirt
COPY startup.sh /
ENTRYPOINT ["/startup.sh" ]

18. FROM ubuntu:18.04
RUN apt-get update -y &&
DEBIAN_FRONTEND=noninteractive apt-get
install -y qemu-kvm libvirt-daemon-system
libvirt-clients bridge-utils vagrant &&
apt-get autoclean &&
apt-get autoremove &&
vagrant plugin install vagrant-libvirt
COPY startup.sh /
ENTRYPOINT ["/startup.sh" ]

19. FROM ubuntu:18.04
RUN apt-get update -y &&
DEBIAN_FRONTEND=noninteractive apt-get
install -y qemu-kvm libvirt-daemon-system
libvirt-clients bridge-utils vagrant &&
apt-get autoclean &&
apt-get autoremove &&
vagrant plugin install vagrant-libvirt
COPY startup.sh /
ENTRYPOINT ["/startup.sh" ]

20. FROM ubuntu:18.04
RUN apt-get update -y &&
DEBIAN_FRONTEND=noninteractive apt-get
install -y qemu-kvm libvirt-daemon-system
libvirt-clients bridge-utils vagrant &&
apt-get autoclean &&
apt-get autoremove &&
vagrant plugin install vagrant-libvirt
COPY startup.sh /
ENTRYPOINT ["/startup.sh" ]

21. FROM ubuntu:18.04
RUN apt-get update -y &&
DEBIAN_FRONTEND=noninteractive apt-get
install -y qemu-kvm libvirt-daemon-system
libvirt-clients bridge-utils vagrant &&
apt-get autoclean &&
apt-get autoremove &&
vagrant plugin install vagrant-libvirt
COPY startup.sh /
ENTRYPOINT ["/startup.sh" ]

22. # startup.sh script
set -eou pipefail
chown root:kvm /dev/kvm
service libvirtd start
service virtlogd start
exec "$@"

23. # startup.sh script
set -eou pipefail
chown root:kvm /dev/kvm
service libvirtd start
service virtlogd start
exec "$@"

24. Remember to
chmod +x startup.sh
Build the container
docker build -t vmindocker:latest -f Dockerfile .

25. Let us run the container.

26. docker run --privileged -it
vmindocker bash
"privileged" is pretty much an alias to
"all the security features disabled"

27. docker run -it
--device=/dev/kvm
--device=/dev/net/tun
-v /sys/fs/cgroup:/sys/fs/cgroup:rw
--cap-add=NET_ADMIN
--cap-add=SYS_ADMIN vmindocker bash

28. docker run -it
--device=/dev/kvm
--device=/dev/net/tun
-v /sys/fs/cgroup:/sys/fs/cgroup:rw
--cap-add=NET_ADMIN
--cap-add=SYS_ADMIN vmindocker bash

29. docker run -it
--device=/dev/kvm
--device=/dev/net/tun
-v /sys/fs/cgroup:/sys/fs/cgroup:rw
--cap-add=NET_ADMIN
--cap-add=SYS_ADMIN vmindocker bash

30. docker run -it
--device=/dev/kvm
--device=/dev/net/tun
-v /sys/fs/cgroup:/sys/fs/cgroup:rw
--cap-add=NET_ADMIN
--cap-add=SYS_ADMIN vmindocker bash

31. docker run -it
--device=/dev/kvm
--device=/dev/net/tun
-v /sys/fs/cgroup:/sys/fs/cgroup:rw
--cap-add=NET_ADMIN
--cap-add=SYS_ADMIN vmindocker bash

32. docker run -it
--device=/dev/kvm
--device=/dev/net/tun
-v /sys/fs/cgroup:/sys/fs/cgroup:rw
--cap-add=NET_ADMIN
--cap-add=SYS_ADMIN vmindocker bash
You’d also need to:
mount -o remount,rw /sys
mount -o remount,rw /proc/sys

33. Spin up a VM in the container.
vagrant init generic/alpine37
vagrant up
vagrant ssh
# Check that it supports virtualisation
root@59157fd1ed32:/# egrep -c '(vmx|svm)' /proc/cpuinfo
# Spin up another VM inside of this VM :)

34. Short demo
https://youtu.be/1quxgKLnEy0

36. Kosisochukwu Anyanwu
Github: kosyfrances
Twitter: kosyfrances
Email: kosy@kinvolk.io
Kinvolk
Blog: kinvolk.io/blog
Github: kinvolk
Twitter: kinvolkio
Email: hello@kinvolk.io