Orchestrating Docker with OpenStack Nov 3rd, 2014
Compute 
MAGNUM 
Containers as a Service
Project 
SOLUM 
FROM CODE TO MANAGED APP 
“Convert code into a managed application running on 
an OpenStack cloud at the push of a button.”
Key element of the Solum data plane 
Docker Docker
Applying Heat 
Orchestration for Docker API
Docker 
Heat Resource 
Heat API 
VM 
Docker 
Nova resource Nova 
Docker resource 
Container1 
Container2 
Container3 
HOT
Installing the plugin 
git clone https://github.com/openstack/heat 
ln -sf $PWD/heat/heat/contrib/docker/plugin;  
/usr/lib/heat/docker" 
echo “plugin_dirs=$PWD/heat/heat/contrib/docker/plugin” 
>> /etc/heat/heat.conf
Docker 
Heat Resource 
Heat API 
VM 
Docker 
Nova resource Nova 
Docker resource 
Container1 
Container2 
Container3 
HOT
Docker 
Heat Resource 
Heat API 
VM 
Docker 
Nova resource Nova 
Docker resource 
Container1 
Container2 
Container3 
HOT 
1. Heat provides a Docker resource 
2. Docker resource communicates 
directly to Docker 
3. Templates may glue Nova and 
Docker resources 
4. Can deploy containers on top of 
VMs or bare-metal instances.
Heat: Cirros 
heat_template_version: 2013-05-23 
description: Single compute instance running cirros in a Docker 
container. 
resources: 
my_instance: 
type: OS::Nova::Server 
properties: 
key_name: ewindisch_key 
image: ubuntu-precise 
flavor: m1.large 
user_data: #include https://get.docker.io 
my_docker_container: 
type: DockerInc::Docker::Container 
docker_endpoint: { get_attr: [my_instance, first_address] } 
image: cirros
Applying Heat 
Heat API 
VM 
Docker 
Nova resource Nova 
Docker resource 
Container1 
Container2 
Container3 
HOT
$ cat template.yml 
heat_template_version: 2013-05-23 
description: Single compute instance running cirros in a Docker 
container. 
resources: 
my_instance: 
type: OS::Nova::Server 
properties: 
key_name: ewindisch_key 
image: ubuntu-precise 
flavor: m1.large 
user_data: #include https://get.docker.io 
my_docker_container: 
type: DockerInc::Docker::Container 
docker_endpoint: { get_attr: [my_instance, first_address] } 
image: cirros 
$ heat stack-create -f template.yml docker 
Heat API 
VM 
Docker 
Nova resource Nova 
Docker resource 
Container1 
Container2 
Container3 
HOT 
Applying Heat
Heat: Dockenstack 
heat_template_version: 2013-05-23 
description: Single compute instance running Tempest 
resources: 
my_instance: 
type: OS::Nova::Server 
properties: 
key_name: ewindisch_key 
image: ubuntu-precise 
flavor: m1.large 
user_data: #include https://get.docker.io 
my_docker_container: 
type: DockerInc::Docker::Container 
properties: 
docker_endpoint: { get_attr: [my_instance, first_address] } 
image: dockenstack 
privileged: true 
cmd: /opt/dockenstack/bin/tempest
heat_template_version: 2013-05-23 
description: Two containers, one host with shared volumes 
resources: 
my_instance: 
type: OS::Nova::Server 
properties: 
key_name: ewindisch_key 
image: ubuntu-precise 
flavor: m1.large 
user_data: #include https://get.docker.io 
ftp_container: 
type: DockerInc::Docker::Container 
properties: 
docker_endpoint: { get_attr: [my_instance, first_address] } 
image: mikz/vsftpd 
ports: [ “21:21” ] 
volumes: [ “/ftp” ] 
name: “FTP” 
apache_container: 
type: DockerInc::Docker::Container 
properties: 
docker_endpoint: { get_attr: [my_instance, first_address] } 
image: fedora/apache 
ports: [ “80:80” ] 
volumes-from: “FTP” 
cmd: “rm -rf /var/www; ln -s /ftp /var/www; /run-apache.sh”
Resources: Heat 
• http://blog.oddbit.com/2014/08/30/docker-plugin-for- 
openstack-he/ 
• http://techs.enovance.com/7104/multi-tenant-docker- 
with-openstack-heat
MAGNUM 
Containers as a Service 
a new service of the OpenStack Compute program
The Containers Team 
Working Group of the Compute Program
The Containers Team 
Working Group of the Compute Program 
• Operating underneath Compute program 
• Outlined a proposal for Magnum (Nova Mid-cycle) 
• Magnum would directly orchestrate containers 
• Would leverage all benefits and features unique to 
containers. 
• It would be the “nova of containers” 
• It could use Nova to spawn instances to hold 
containers. 
• Those instances may be VMs, Baremetal, or 
Containers.
See Adrian Otto’s presentation: 
Containers for Multi-cloud Apps 
Tomorrow: 17:20
Nova Integration 
Docker plugin for Nova
Awesome People 
Ian Main (Red Hat) 
Chris Alfonso (Red Hat) 
Davanum ‘dims’ (IBM) 
ChangBo Guo 
Julien Vey (Numergy) 
Aaron Rosen (Nicera) 
Derek Higgins (Red Hat) 
Paul Czarkowski (Rackspace) 
Daniel Kuffner 
Pedro R Marques (Juniper) 
Lars Kellogg-Stedman 
(Red_Hat) 
Sam Alba (Docker) 
& more…
What? 
Enables control of 
Docker via OpenStack: 
• Nova API 
• Horizon UI 
Supports: 
• launch 
• terminate 
• reboot 
• serial console 
• snapshot 
• Glance 
• Neutron 
• Pause/unpause 
https://wiki.openstack.org/wiki/ 
HypervisorSupportMatrix
Identity Crisis
Nova doesn’t… 
Link container networks 
Pass environment variables 
Specify working directories 
Create docker-volumes 
Share docker-volumes between containers 
Arbitrary commands 
Arbitrary command-arguments 
Pass devices 
Nova is a machine abstraction, not a process one.
Docker doesn’t… 
• Support mounting devices (unprivileged) 
• Live-migration is future-speak 
• Boot from block devices (natively - it’s possible…) 
• Support Glance natively 
• PCI pass-through
Havana & Icehouse 
Image Management 
(at-release)
Havana & Icehouse 
Image Management 
(at-release) 
• docker-registry worked as a proxy 
• Users had to upload through 
docker-registry. 
• docker pulls images through the 
docker-registry proxy
Havana & Icehouse 
Image Management 
(at-release)
Havana & Icehouse 
Image Management 
(at-release) 
• Glance was only used to provide 
visibility of Docker images for Nova.
Havana & Icehouse 
Image Management 
(at-release) 
• Glance was only used to provide 
visibility of Docker images for Nova. 
• Users could not upload through 
Glance directly
Havana & Icehouse 
Image Management 
(at-release) 
• Glance was only used to provide 
visibility of Docker images for Nova. 
• Users could not upload through 
Glance directly 
• Making that work would require a 
special procedure for glance uploads.
so… we took out the 
docker-registry instead.
Just Enough Docker
Just Enough Docker
Just Enough Docker 
• A subset of Nova features…
Just Enough Docker 
• A subset of Nova features… 
• A subset of Docker features…
Just Enough Docker 
• A subset of Nova features… 
• A subset of Docker features… 
• Enough for Nova to allow running 
Docker-in-Docker.
Just Enough Docker 
• A subset of Nova features… 
• A subset of Docker features… 
• Enough for Nova to allow running 
Docker-in-Docker. 
• DinD retains most performance 
benefits of Docker.
Just Enough Docker 
• A subset of Nova features… 
• A subset of Docker features… 
• Enough for Nova to allow running 
Docker-in-Docker. 
• DinD retains most performance 
benefits of Docker. 
• DinD is Docker and everything 
you love about Docker.
OpenStack Docker 
Nova 
nova 
docker
OpenStack Docker 
Nova 
nova 
docker 
Docker
OpenStack Docker 
Nova 
nova 
docker 
Docker 
OpenStack API 
Docker API
OpenStack Docker 
Nova 
nova 
docker 
Docker 
OpenStack API 
Docker API Docker API
Docker
Kubernetes Heat 
Docker 
Solum OpenShift 
Mesos CloudFoundry 
Magnum
neutron nova-api 
nova-compute 
VM 
VM 
docker 
docker 
Hypervisor 
container 
container
neutron nova-api
neutron nova-api
neutron nova-api 
nova-compute 
Docker 
container 
container
neutron nova-api 
nova-compute 
Docker 
container 
container 
nova-compute 
VM 
VM 
docker 
docker 
Hypervisor 
container 
container
Hybrid Nova configuration 
neutron nova-api 
nova-compute 
Docker 
container 
container 
nova-compute 
VM 
VM 
docker 
docker 
Hypervisor 
container 
container
neutron nova-api 
nova-compute 
Docker 
container 
container 
nova-compute 
VM 
VM 
docker 
docker 
Hypervisor 
container 
container 
nova-compute 
Ironic 
Machine docker 
container 
container
Hybrid Nova configuration + Ironic 
neutron nova-api 
nova-compute 
Docker 
container 
container 
nova-compute 
VM 
VM 
docker 
docker 
Hypervisor 
container 
container 
nova-compute 
Ironic 
Machine docker 
container 
container
nova-api 
nova-compute 
Docker 
container docker 
container 
container
nova-api 
nova-compute 
Docker 
container docker 
container 
container
nova-api 
nova-compute 
Docker 
container docker 
container 
container 
Kubernetes 
Heat 
Mesos 
CloudFoundry 
Magnum
Install the plugin 
mkdir git-co; cd git-co" 
git clone https://github.com/stackforge/nova-docker" 
cd nova-driver" 
python setup.py install
Configure Nova 
Set in nova.conf:" 
compute_driver=novadocker.virt.docker.DockerDriver"
Putting an image into your repository 
docker pull cirros" 
docker save cirros | glance image-create  
--is-public=True  
--container-format=docker  
--disk-format=raw  
--name cirros
‘nova boot’
Networking 
Nova 
Network
Please welcome: 
Ian Main
Testing - Running & Passing
Testing - Running & Passing 
- Get as many tests passing as possible.! 
- Now running 1726 tests, 0 failures.! 
- Turned off: 
volumes 
resizing & suspending 
rescue! 
! ! migrations.
Working Upstream
Working Upstream 
• Added pause and unpause support for docker containers. 
• Well accepted into the Docker project. 
• Dynamic device support needed for Cinder volumes. 
• First API that modifies running containers. 
• Docker community wants the user experience to be right. 
• It will land, just need to get it right
Cinder Volumes 
Use cases:! 
• Direct access to block device – not common.! 
• Mounting file systems.! 
- Possible security issues.! 
- Different from VMs.! 
- Privileged containers.! 
- FUSE filesystem support through user namespaces.! 
• PoC of boot from volume.
KILO Nova-Docker
KILO
KILO 
- Cinder support
KILO 
- Cinder support 
- Security groups (merged)
KILO 
- Cinder support 
- Security groups (merged) 
- docker-py (merged)
KILO 
- Cinder support 
- Security groups (merged) 
- docker-py (merged) 
- privileged containers
KILO 
- Cinder support 
- Security groups (merged) 
- docker-py (merged) 
- privileged containers 
- more +2 contributors
use our code… 
Fix our Bugs!
Q & A 
Eric Windisch 
<erw>@freenode 
@ewindisch 
Ian Main 
<slower>@freenode

Orchestrating Docker with OpenStack

  • 1.
    Orchestrating Docker withOpenStack Nov 3rd, 2014
  • 3.
  • 4.
    Project SOLUM FROMCODE TO MANAGED APP “Convert code into a managed application running on an OpenStack cloud at the push of a button.”
  • 5.
    Key element ofthe Solum data plane Docker Docker
  • 6.
  • 7.
    Docker Heat Resource Heat API VM Docker Nova resource Nova Docker resource Container1 Container2 Container3 HOT
  • 8.
    Installing the plugin git clone https://github.com/openstack/heat ln -sf $PWD/heat/heat/contrib/docker/plugin; /usr/lib/heat/docker" echo “plugin_dirs=$PWD/heat/heat/contrib/docker/plugin” >> /etc/heat/heat.conf
  • 9.
    Docker Heat Resource Heat API VM Docker Nova resource Nova Docker resource Container1 Container2 Container3 HOT
  • 10.
    Docker Heat Resource Heat API VM Docker Nova resource Nova Docker resource Container1 Container2 Container3 HOT 1. Heat provides a Docker resource 2. Docker resource communicates directly to Docker 3. Templates may glue Nova and Docker resources 4. Can deploy containers on top of VMs or bare-metal instances.
  • 11.
    Heat: Cirros heat_template_version:2013-05-23 description: Single compute instance running cirros in a Docker container. resources: my_instance: type: OS::Nova::Server properties: key_name: ewindisch_key image: ubuntu-precise flavor: m1.large user_data: #include https://get.docker.io my_docker_container: type: DockerInc::Docker::Container docker_endpoint: { get_attr: [my_instance, first_address] } image: cirros
  • 12.
    Applying Heat HeatAPI VM Docker Nova resource Nova Docker resource Container1 Container2 Container3 HOT
  • 13.
    $ cat template.yml heat_template_version: 2013-05-23 description: Single compute instance running cirros in a Docker container. resources: my_instance: type: OS::Nova::Server properties: key_name: ewindisch_key image: ubuntu-precise flavor: m1.large user_data: #include https://get.docker.io my_docker_container: type: DockerInc::Docker::Container docker_endpoint: { get_attr: [my_instance, first_address] } image: cirros $ heat stack-create -f template.yml docker Heat API VM Docker Nova resource Nova Docker resource Container1 Container2 Container3 HOT Applying Heat
  • 14.
    Heat: Dockenstack heat_template_version:2013-05-23 description: Single compute instance running Tempest resources: my_instance: type: OS::Nova::Server properties: key_name: ewindisch_key image: ubuntu-precise flavor: m1.large user_data: #include https://get.docker.io my_docker_container: type: DockerInc::Docker::Container properties: docker_endpoint: { get_attr: [my_instance, first_address] } image: dockenstack privileged: true cmd: /opt/dockenstack/bin/tempest
  • 15.
    heat_template_version: 2013-05-23 description:Two containers, one host with shared volumes resources: my_instance: type: OS::Nova::Server properties: key_name: ewindisch_key image: ubuntu-precise flavor: m1.large user_data: #include https://get.docker.io ftp_container: type: DockerInc::Docker::Container properties: docker_endpoint: { get_attr: [my_instance, first_address] } image: mikz/vsftpd ports: [ “21:21” ] volumes: [ “/ftp” ] name: “FTP” apache_container: type: DockerInc::Docker::Container properties: docker_endpoint: { get_attr: [my_instance, first_address] } image: fedora/apache ports: [ “80:80” ] volumes-from: “FTP” cmd: “rm -rf /var/www; ln -s /ftp /var/www; /run-apache.sh”
  • 16.
    Resources: Heat •http://blog.oddbit.com/2014/08/30/docker-plugin-for- openstack-he/ • http://techs.enovance.com/7104/multi-tenant-docker- with-openstack-heat
  • 17.
    MAGNUM Containers asa Service a new service of the OpenStack Compute program
  • 18.
    The Containers Team Working Group of the Compute Program
  • 19.
    The Containers Team Working Group of the Compute Program • Operating underneath Compute program • Outlined a proposal for Magnum (Nova Mid-cycle) • Magnum would directly orchestrate containers • Would leverage all benefits and features unique to containers. • It would be the “nova of containers” • It could use Nova to spawn instances to hold containers. • Those instances may be VMs, Baremetal, or Containers.
  • 20.
    See Adrian Otto’spresentation: Containers for Multi-cloud Apps Tomorrow: 17:20
  • 21.
    Nova Integration Dockerplugin for Nova
  • 22.
    Awesome People IanMain (Red Hat) Chris Alfonso (Red Hat) Davanum ‘dims’ (IBM) ChangBo Guo Julien Vey (Numergy) Aaron Rosen (Nicera) Derek Higgins (Red Hat) Paul Czarkowski (Rackspace) Daniel Kuffner Pedro R Marques (Juniper) Lars Kellogg-Stedman (Red_Hat) Sam Alba (Docker) & more…
  • 23.
    What? Enables controlof Docker via OpenStack: • Nova API • Horizon UI Supports: • launch • terminate • reboot • serial console • snapshot • Glance • Neutron • Pause/unpause https://wiki.openstack.org/wiki/ HypervisorSupportMatrix
  • 24.
  • 25.
    Nova doesn’t… Linkcontainer networks Pass environment variables Specify working directories Create docker-volumes Share docker-volumes between containers Arbitrary commands Arbitrary command-arguments Pass devices Nova is a machine abstraction, not a process one.
  • 26.
    Docker doesn’t… •Support mounting devices (unprivileged) • Live-migration is future-speak • Boot from block devices (natively - it’s possible…) • Support Glance natively • PCI pass-through
  • 27.
    Havana & Icehouse Image Management (at-release)
  • 28.
    Havana & Icehouse Image Management (at-release) • docker-registry worked as a proxy • Users had to upload through docker-registry. • docker pulls images through the docker-registry proxy
  • 29.
    Havana & Icehouse Image Management (at-release)
  • 30.
    Havana & Icehouse Image Management (at-release) • Glance was only used to provide visibility of Docker images for Nova.
  • 31.
    Havana & Icehouse Image Management (at-release) • Glance was only used to provide visibility of Docker images for Nova. • Users could not upload through Glance directly
  • 32.
    Havana & Icehouse Image Management (at-release) • Glance was only used to provide visibility of Docker images for Nova. • Users could not upload through Glance directly • Making that work would require a special procedure for glance uploads.
  • 33.
    so… we tookout the docker-registry instead.
  • 34.
  • 35.
  • 36.
    Just Enough Docker • A subset of Nova features…
  • 37.
    Just Enough Docker • A subset of Nova features… • A subset of Docker features…
  • 38.
    Just Enough Docker • A subset of Nova features… • A subset of Docker features… • Enough for Nova to allow running Docker-in-Docker.
  • 39.
    Just Enough Docker • A subset of Nova features… • A subset of Docker features… • Enough for Nova to allow running Docker-in-Docker. • DinD retains most performance benefits of Docker.
  • 40.
    Just Enough Docker • A subset of Nova features… • A subset of Docker features… • Enough for Nova to allow running Docker-in-Docker. • DinD retains most performance benefits of Docker. • DinD is Docker and everything you love about Docker.
  • 41.
  • 42.
    OpenStack Docker Nova nova docker Docker
  • 43.
    OpenStack Docker Nova nova docker Docker OpenStack API Docker API
  • 44.
    OpenStack Docker Nova nova docker Docker OpenStack API Docker API Docker API
  • 45.
  • 46.
    Kubernetes Heat Docker Solum OpenShift Mesos CloudFoundry Magnum
  • 47.
    neutron nova-api nova-compute VM VM docker docker Hypervisor container container
  • 48.
  • 49.
  • 50.
    neutron nova-api nova-compute Docker container container
  • 51.
    neutron nova-api nova-compute Docker container container nova-compute VM VM docker docker Hypervisor container container
  • 52.
    Hybrid Nova configuration neutron nova-api nova-compute Docker container container nova-compute VM VM docker docker Hypervisor container container
  • 53.
    neutron nova-api nova-compute Docker container container nova-compute VM VM docker docker Hypervisor container container nova-compute Ironic Machine docker container container
  • 54.
    Hybrid Nova configuration+ Ironic neutron nova-api nova-compute Docker container container nova-compute VM VM docker docker Hypervisor container container nova-compute Ironic Machine docker container container
  • 55.
    nova-api nova-compute Docker container docker container container
  • 56.
    nova-api nova-compute Docker container docker container container
  • 57.
    nova-api nova-compute Docker container docker container container Kubernetes Heat Mesos CloudFoundry Magnum
  • 58.
    Install the plugin mkdir git-co; cd git-co" git clone https://github.com/stackforge/nova-docker" cd nova-driver" python setup.py install
  • 59.
    Configure Nova Setin nova.conf:" compute_driver=novadocker.virt.docker.DockerDriver"
  • 60.
    Putting an imageinto your repository docker pull cirros" docker save cirros | glance image-create --is-public=True --container-format=docker --disk-format=raw --name cirros
  • 61.
  • 62.
  • 63.
  • 64.
  • 65.
    Testing - Running& Passing - Get as many tests passing as possible.! - Now running 1726 tests, 0 failures.! - Turned off: volumes resizing & suspending rescue! ! ! migrations.
  • 66.
  • 67.
    Working Upstream •Added pause and unpause support for docker containers. • Well accepted into the Docker project. • Dynamic device support needed for Cinder volumes. • First API that modifies running containers. • Docker community wants the user experience to be right. • It will land, just need to get it right
  • 68.
    Cinder Volumes Usecases:! • Direct access to block device – not common.! • Mounting file systems.! - Possible security issues.! - Different from VMs.! - Privileged containers.! - FUSE filesystem support through user namespaces.! • PoC of boot from volume.
  • 69.
  • 70.
  • 71.
  • 72.
    KILO - Cindersupport - Security groups (merged)
  • 73.
    KILO - Cindersupport - Security groups (merged) - docker-py (merged)
  • 74.
    KILO - Cindersupport - Security groups (merged) - docker-py (merged) - privileged containers
  • 75.
    KILO - Cindersupport - Security groups (merged) - docker-py (merged) - privileged containers - more +2 contributors
  • 76.
    use our code… Fix our Bugs!
  • 77.
    Q & A Eric Windisch <erw>@freenode @ewindisch Ian Main <slower>@freenode