This document discusses the use of Docker containers to deploy an OpenStack cloud (Corona). It summarizes the different node types used, including controllers, hypervisors, ONE servers, NFS, and Sunstone. It describes challenges with configuring containers that require host privileges or access to resources like cgroups and devices. Systemd and Supervisord are compared for managing processes. Configuration and managing the Oneadmin token/SSH keys across dynamic nodes is challenging. Overall the document evaluates approaches to deploying OpenStack components in Docker containers for scalability, automation, and manageability.

1. CORONA
Dan Kelleher | Scientific Systems Administrator | Harvard FAS Research Computing

2. Overview
• Node Types
• Docker Containers
– Base, Libvirt, nfs, ONE server, ONE hypervisor, Sunstone
– Docker network, privilege and PID
– Systemd vs Supervisord
– Compose
• Challenges
– Configuration
– Oneadmin token and SSH keys

3. Motivation
• Binary roll out and roll back
– Simple deployment
– Automated testing
– Upgrades
• Elasticity?
– Deploy a new node using cluster scheduler
– Tear it down when it’s not longer needed
• Kolla
– https://github.com/openstack/kolla

4. Node Types
Controller
Hybrid
ONE Server
• Host Net
• Host PID
Hybrid
Hybrid
Hypervisor
NFS
• Host Net
• Host PID
• Privileged
Sunstone
• Host Net
• Host PID
• Privileged
Libvirt
• Host Net
• Host PID
• Privileged
• Cgroups
• /dev
• /run
• /lib/
modules
NFS
• Host Net
• Host PID
• Privileged
ONE Hyper
• Host Net
• Host PID
• Privileged
• Cgroups
• /dev
• /run
ONE Server
• Host Net
• Host PID
NFS
• Host Net
• Host PID
• Privileged
Libvirt
• Host Net
• Host PID
• Privileged
• Cgroups
• /dev
• /run
• /lib/
modules
ONE Hyper
• Host Net
• Host PID
• Privileged
• Cgroups
• /dev
• /run
Sunstone
• Host Net
• Host PID
• Privileged

5. Overview
• Node Types
• Docker Containers
– Base, Libvirt, nfs, ONE server, ONE hypervisor, Sunstone
– Docker network, privilege and PID
– Systemd vs Supervisord
– Compose
• Challenges
– Configuration
– Oneadmin token and SSH keys

6. Corona-Base

7. Libvirt
The Libvirt container is straight forward build but:
– In order to setup the virtualization limitations the libvirt container
needs access to cgroups
– It also needs access to the device directory and kernel modules
directory to control the hardware
Libvirt is also a self contained daemon which makes
process management easier

8. Libvirt

9. NFS

10. NFS supervisor

11. NFS
• NFS on the other hand is easy to build and doesn’t
require access to a lot of resources but is complex to
configure
• NFS requires 5 intertwined daemons which each need to
be configured
• In addition a reload condition needs to be defined
• This is one of the cases where systemd would have
excelled. (More on that in a bit)

12. ONE Server

13. ONE Hypervisor

14. ONE
• By comparison the OpenNebula software was both easy
to build and simple to configure in supervisord
• The challenge in the ONE software lies in configuring the
software it self without jumping into the container.

15. Sunstone
This container needs a lot more work

16. CORONA Un-Contained
• Host PID is mostly for convenience of management
• Host Networking was the best solution to create the
bridges for the trunked vlans
• Privilege is required by anything which needs to bind to
the host networking stack in the protected port range
– Some of this could likely be reduced using docker port binding
– Libvirt needs to be able to inject kernel modules
– ONE hypervisor needs to create the bridges

17. MySQL
• There’s a MySQL container straight from docker hub
– https://hub.docker.com/_/mysql/
• Also there is a MariaDB official container on docker hub
– https://hub.docker.com/_/mariadb/
• In the future docker should allow the controller container
to pick up the database name, password, and username
if they are on the same host

18. How to Supervise
Systemd
• Pros:
– Dependencies are handled
– Reloads are simple
– Configured by the
packages
• Cons:
– Running multiple instances
proved challenging
– Require a lot of packages
Supervisord
• Pros:
– Running multiple instances
is simple
– Doesn’t run as root
• Cons:
– Has to be hand configured

19. Docker Compose

20. Overview
• Node Types
• Docker Containers
– Base, Libvirt, nfs, ONE server, ONE hypervisor, Sunstone
– Docker network, privilege and PID
– Systemd vs Supervisord
– Compose
• Challenges
– Configuration
– Oneadmin token and SSH keys

21. Configuration
• The Docker standard is to use environment variable
– At the moment we are using flat files mounted as read only
docker volumes.
• A Script needs to be created to parse environment
variables and put them in the correct config files
– Building this script has been challenging
– A lot of configuration variables add to complexity

22. Token and SSH
• In our current setup these are laid down by puppet
• Mounting these out in production is not very secure
• Hypervisors can’t auto check-in without these
– This is critical for elasticity

23. Demo?

24. Questions?