This document discusses using Docker containers for continuous integration testing of Neutron plugins. It notes that testing all 38 Neutron vendor plugins is impossible, and outlines efforts to establish third-party CI rules. It then describes problems with using virtual machines for testing, such as slow startup times and performance issues. As an alternative, it proposes running OpenStack on Docker containers to allow running tests in isolated, disposable environments. It details fixes needed to run OpenStack with Docker and use Neutron ML2 plugins. Jenkins would launch test scripts within containers, binding host volumes to easily access logs. This allows Neutron API tests to be run in parallel across container instances.

1. Copyright©2014 NTT corp. All Rights Reserved.
Neutron CI Run on Docker
Hirofumi Ichihara
NTT

2. 2Copyright©2014 NTT corp. All Rights Reserved.
• Neutron has many vendor plugins
• 38 vendor plugins and drivers in Neutron(include
proposing)
• Impossible to test all vendor plugins
Neutron plugins
http://stackalytics.com/report/driverlog?project_id=openstack%2Fneutron
A10 Networks LBaaS Driver
Arista Neutron ML2 Driver
Big Switch Neutron ML2 Driver
Big Switch Controller Plugin
Brocade Neutron plugin
Brocade Neutron ML2 Mechanism driver for VDX/VCS
Switches
Brocade Vyatta
CSR1kv router service plugin
Neutron ML2 Driver For Cisco Nexus Devices
Cisco APIC ML2 Driver
Cisco APIC L3 Service
Cisco VPNaaS
NetScaler LBaas driver
Cloudbase Hyper-V Plugin
Embrane Neutron Plugin
Embrane LBaaS Driver
Extreme Networks Plugin
Freescale SDN ML2 Mechanism Driver
IBM SDN-VE Plugin
Juniper Networks plug-in for OpenStack Neutron
Juniper Opencontrail plugin for Neutron resources
Mellanox Neutron Plugin
Mellanox Neutron ML2 Driver
Mellanox Neutron ML2 SR-IOV Driver
MidoNet Plugin
NEC OpenFlow Plugin
Nuage Networks Neutron Plugin
Nuage Networks ML2 driver
One Convergence NVSD Controller
OpenDaylight ML2 MechanismDriver
Meta Plugin
Ryu OpenFlow Controller Plugin
OpenFlow Agent
PLUMgrid OpenStack Networking Suite Neutron
Plugin
Radware LBaaS Driver for Alteon.
Tail-f NCS ML2 Mechanism Driver
VMware NSX Network Virtualization Platform
Plugin
vArmour-Firewall

3. 3Copyright©2014 NTT corp. All Rights Reserved.
• Establish third party CI rules for Neutron
• https://wiki.openstack.org/wiki/NeutronThirdPartyTesting
Neutron Third Party CI System
https://review.openstack.org/#/c/131905/

4. 4Copyright©2014 NTT corp. All Rights Reserved.
• Metaplugin
• provided since Folsom version
• allows you to use multiple plugins at the same time
• Metaplugin CI System
Neutron CI System for Metaplugin
1000 tests/month
Max 3 parallels

5. 5Copyright©2014 NTT corp. All Rights Reserved.
• Test must be run in clean environment
Neutron CI System Requirement
Test Node
start
openstack
OVS setting
Package
ENV setting
install openstack
run tests
finish tests
stop openstack
Clean Node
Clean
Clean
Clean
OVS setting
Package
ENV setting
start
openstack
install openstack
fail to install or test
Jenkins: http://jenkins-ci.org/

6. 6Copyright©2014 NTT corp. All Rights Reserved.
• Metaplugin CI System uses VM
Neutron CI System with VM
create VM
start test
VM
install openstack
OpenStack
VM
OpenStack
VM
OpenStack
finish tests
delete VM
certain Public Cloud Service
VM VM VM
Jenkins: http://jenkins-ci.org/

7. 7Copyright©2014 NTT corp. All Rights Reserved.
• VM startup is very slow
• VM performance is not good
VM Problems
But a VM is very cheap:)

8. 8Copyright©2014 NTT corp. All Rights Reserved.
• VM startup is very slow
• VM performance is not good
VM Problems
But a VM is very cheap:)
Really......?

9. 9Copyright©2014 NTT corp. All Rights Reserved.
Not cheap!

10. 10Copyright©2014 NTT corp. All Rights Reserved.
Let’s use container

11. 11Copyright©2014 NTT corp. All Rights Reserved.
Docker
docker: https://www.docker.com

12. 12Copyright©2014 NTT corp. All Rights Reserved.
• Dockenstack
• Run OpenStack on Docker
• Use nova-docker project to run nova instances
• Written by Paul Czarkowski, Eric Windisch
• Provide to https://github.com/ewindisch/dockenstack
OpenStack on Docker
Container
OpenStack
instance instance

13. 13Copyright©2014 NTT corp. All Rights Reserved.
OpenStack on Docker Problems
• Well known
• Not support OpenvSwitch
• Not support Cinder Volume
• Devstack failed to
• Reserve keystone port
• install nova-docker
• Restart docker
• Nova-docker didn’t support Neutron ML2
Linuxbridge driver
• Neutron failed to ip6tables

14. 14Copyright©2014 NTT corp. All Rights Reserved.
• OpenvSwitch
• Select ML2 Linuxbridge as Neutron plugin
• Cinder Volume
• Don’t use volume
• Reserve Keystone port ⇒ Bugfix #125740
• Install nova-docker ⇒ Bugfix #128746
• Restart docker
• Replace command (Workaround)
• start-stop-daemon --stop --pidfile "/var/run/docker.pid”
• ML2 Linuxbridge driver ⇒ Bugfix #130520
• Don’t use ip6tables
• Disable ipv6 in container(Workaround)
• echo 1 > /proc/sys/net/ipv6/conf/default/disable_ipv6
OpenStack on Docker Run

15. 15Copyright©2014 NTT corp. All Rights Reserved.
• Jenkins runs shell script
Jenkins Run
#!/usr/bin/env bash
set -e
mkdir -p $BUILD_NUMBER/logs
docker run -v
$WORKSPACE/$BUILD_NUMBER/logs:/opt/stack/logs --
privileged $DOCKER_IMAGE $GERRIT_REFSPEC
• Docker can bind mount a
volume from the host
• Storing log became very
easy

16. 16Copyright©2014 NTT corp. All Rights Reserved.
• Jenkins detects test success inside container
Jenkins Success

17. 17Copyright©2014 NTT corp. All Rights Reserved.
• Tempest Neutron API test is successful
Neutron Test Result

18. 18Copyright©2014 NTT corp. All Rights Reserved.
• Jenkins detects test failure inside container
Jenkins Failure

19. 19Copyright©2014 NTT corp. All Rights Reserved.
$ git clone -b neutron-ci
https://github.com/hichihara/dockenstack.git
$ cd dockenstack/dockenstack
$ docker build .
TRY

20. 20Copyright©2014 NTT corp. All Rights Reserved.
Contact
ichihara.hirofumi@lab.ntt.co.jp