Magnum, Kuryr, Neutron Integration
Networking for Nested Containers
Fawad Khaliq - @fawadkhaliq
Antoni Segura – @celebdor
Gal Sagie - @GalSagie
Copyright © PLUMgrid, Inc. 2011-2016
Introduction
Speakers
Sr. Software Engineer
PLUMgrid
Khaliq
Fawad
2
Senior Engineer
Midokura
Segura
Antoni
Architect
Huawei
Sagie
Gal
Copyright © PLUMgrid, Inc. 2011-2016
• Magnum, Neutron
• Kuryr
• Nested Containers and Networking Problem
• Nested Containers Networking – Solution/Design
• Capabilities and considerations
• Current Status
• Next Steps
• Q&A
Agenda
3
Magnum
Container-as-a-service in OpenStack
4
Copyright © PLUMgrid, Inc. 2011-2016
Magnum
Container-as-a-service in OpenStack
5
Docker Swarm (Bay)
Nova Instance
Container
Container
Container
Nova Instance
Container
Container
Container
Copyright © PLUMgrid, Inc. 2011-2016
Magnum
Container-as-a-service in OpenStack
6
Kubernetes (Bay)
Nova Instance
Pod
Container
Container
Nova Instance
Pod
Container
Container
Copyright © PLUMgrid, Inc. 2011-20167
Neutron
Networking in OpenStack
8
Copyright © PLUMgrid, Inc. 2011-2016
• Provides “network as a service”
• Provides rich network topologies
• Technology agnostic; pluggable networking backends
• Extensible
• Offers advanced services like LBaas, VPNaas, FWaas etc
Neutron
9
Kuryr
Container Networking in OpenStack
10
Copyright © PLUMgrid, Inc. 2011-2016
Kuryr
11
Neutron as the 

production-ready 

networking
abstraction

containers need
Copyright © PLUMgrid, Inc. 2011-2016
VM/Container Networking: Similar Concepts
12
Docker C1 Docker C2 Docker C3
		libNetwork
Endpoint Endpoint EndpointEndpoint
Frontend
Network
Backend
Network
Network Sandbox Network Sandbox Network Sandbox
VM2
192.168.1.7
192.168.5.2
VM1
Tenant A Net1
192.168.1.0/0
Tenant A Net2
192.168.5.0/0
192.168.1.5
Neutron
Copyright © PLUMgrid, Inc. 2011-2016
• Open source
• Part of OpenStack Big-Tent
• Brings the Neutron networking model to containers
• Aims to support different Container Runtimes (docker, rkt, etc)
• E.g. Kubernetes, Mesos, Docker Swarm
• Weekly IRC meetings
• Working together with OpenStack community
• Neutron, Magnum, Kolla
Kuryr Project Overview
13
Copyright © PLUMgrid, Inc. 2011-2016
Kuryr Components
14
Configuration
ManagementKuryr libNetwork
Network Plugin
K8S CNI Driver
Keystone
Authentication
& Neutron
Client
Interface
Generic
VIF Binding
Kuryr libNetwork
IPAM Plugin
Problems with current Nested Containers
Why do we need to consider this as a special scenario?
15
Copyright © PLUMgrid, Inc. 2011-2016
• Two Separate networking infrastructures
• Hard to enforce network policy (N-tier applications)
• Security and Isolation
• Performance and unneeded overhead
Problems with Current Nested Containers Networking
16
Copyright © PLUMgrid, Inc. 2011-2016
Problems with Current Nested Containers Networking
17
Docker 0
OVS
VXLAN Overlay
VM
Docker 0
Neutron Plugin
VXLAN Overlay
VM
SDN Overlay
Neutron
Overlay
Copyright © PLUMgrid, Inc. 2011-2016
Problems with Current Nested Containers Networking
18
Neutron Networks
VMVM VM
Tenant A Net1
192.168.1.0/0
Copyright © PLUMgrid, Inc. 2011-2016
Problems with Current Nested Containers Networking
19
Container Networks
VMVM VM
Backend
Network
10.2.0.0/24
Endpoint Endpoint Endpoint Endpoint Endpoint Endpoint Endpoint
Frontend Network
10.1.0.0/24
Nested Container Networking Solution
Design for the nested container networking in OpenStack
20
Copyright © PLUMgrid, Inc. 2011-2016
• Nested/baremetal container to nested/baremetal container same/
different hosts
• Nested/baremetal container to virtual machine communication
• Nested/baremetal container to baremetal communication
• Container networking as a first class entity in Neutron
• Consistent policy enforcement across containers, VMs, bare metal
• Enable advanced networking services like FWaas, LBaas, VPNaas
etc
Nested Container Networking Use Cases
21
Copyright © PLUMgrid, Inc. 2011-2016
Nested Container Networking Design
Magnum, Kuryr, Neutron Integration
22
VLAN:100 VLAN:200 VLAN:400 VLAN:100
Copyright © PLUMgrid, Inc. 2011-2016
Neutron Trunk Ports
23
Nova Instance
port-1
port-0
port-2
network-1
network-0
network-2
Port combined into one vif
by turning port-0 into trunk
and other ports into supports
of the trunk
Copyright © PLUMgrid, Inc. 2011-201624
Capabilities and Considerations
25
Copyright © PLUMgrid, Inc. 2011-2016
• Neutron resources spec approved and patches under review
• Trunk
• Subport
• Subports bring isolation to container-in-VM use cases
• Port forwarding can take us further
• Vendors can implement new segmentation types
• Tagged traffic that does not match a sub port, is considered of the
trunk port
Capabilities and Considerations
26
Copyright © PLUMgrid, Inc. 2011-2016
• Limitations
• Policy is applied at the Host level
• Initial only VLAN tags for segmentation type
• Tags are unique per “trunk port” scope
• VM users can alter subport traffic
• Logging of VM actions is dependent on integration
• Can't work with current OVS
Capabilities and Considerations
27
Current Status
28
Copyright © PLUMgrid, Inc. 2011-2016
• Trunk Port Extension spec approved and code in progress
• Binding profile workaround to proceed in parallel
• Nested Container networking spec approved in Kuryr
• Docker Swarm Integration completed
• Kubernetes in progress
• Mesos in design stages
Current Status
29
Next Steps
30
Copyright © PLUMgrid, Inc. 2011-2016
• Follow up on the Neutron Trunk port implementation
• Finish COE baremetal integration
• Policy translation
• Make Neutron resources available through native APIs
• Magnum deployment prototype of worker VM with Kuryr agent
• Magnum administrator VM that communicates with Neutron
Next Steps
31
Questions
32
Join us at #openstack-kuryr
THANK YOU!
irc: #openstack-kuryr @ freenode

Networking For Nested Containers: Magnum, Kuryr, Neutron Integration

  • 1.
    Magnum, Kuryr, NeutronIntegration Networking for Nested Containers Fawad Khaliq - @fawadkhaliq Antoni Segura – @celebdor Gal Sagie - @GalSagie
  • 2.
    Copyright © PLUMgrid,Inc. 2011-2016 Introduction Speakers Sr. Software Engineer PLUMgrid Khaliq Fawad 2 Senior Engineer Midokura Segura Antoni Architect Huawei Sagie Gal
  • 3.
    Copyright © PLUMgrid,Inc. 2011-2016 • Magnum, Neutron • Kuryr • Nested Containers and Networking Problem • Nested Containers Networking – Solution/Design • Capabilities and considerations • Current Status • Next Steps • Q&A Agenda 3
  • 4.
  • 5.
    Copyright © PLUMgrid,Inc. 2011-2016 Magnum Container-as-a-service in OpenStack 5 Docker Swarm (Bay) Nova Instance Container Container Container Nova Instance Container Container Container
  • 6.
    Copyright © PLUMgrid,Inc. 2011-2016 Magnum Container-as-a-service in OpenStack 6 Kubernetes (Bay) Nova Instance Pod Container Container Nova Instance Pod Container Container
  • 7.
    Copyright © PLUMgrid,Inc. 2011-20167
  • 8.
  • 9.
    Copyright © PLUMgrid,Inc. 2011-2016 • Provides “network as a service” • Provides rich network topologies • Technology agnostic; pluggable networking backends • Extensible • Offers advanced services like LBaas, VPNaas, FWaas etc Neutron 9
  • 10.
  • 11.
    Copyright © PLUMgrid,Inc. 2011-2016 Kuryr 11 Neutron as the 
 production-ready 
 networking abstraction
 containers need
  • 12.
    Copyright © PLUMgrid,Inc. 2011-2016 VM/Container Networking: Similar Concepts 12 Docker C1 Docker C2 Docker C3 libNetwork Endpoint Endpoint EndpointEndpoint Frontend Network Backend Network Network Sandbox Network Sandbox Network Sandbox VM2 192.168.1.7 192.168.5.2 VM1 Tenant A Net1 192.168.1.0/0 Tenant A Net2 192.168.5.0/0 192.168.1.5 Neutron
  • 13.
    Copyright © PLUMgrid,Inc. 2011-2016 • Open source • Part of OpenStack Big-Tent • Brings the Neutron networking model to containers • Aims to support different Container Runtimes (docker, rkt, etc) • E.g. Kubernetes, Mesos, Docker Swarm • Weekly IRC meetings • Working together with OpenStack community • Neutron, Magnum, Kolla Kuryr Project Overview 13
  • 14.
    Copyright © PLUMgrid,Inc. 2011-2016 Kuryr Components 14 Configuration ManagementKuryr libNetwork Network Plugin K8S CNI Driver Keystone Authentication & Neutron Client Interface Generic VIF Binding Kuryr libNetwork IPAM Plugin
  • 15.
    Problems with currentNested Containers Why do we need to consider this as a special scenario? 15
  • 16.
    Copyright © PLUMgrid,Inc. 2011-2016 • Two Separate networking infrastructures • Hard to enforce network policy (N-tier applications) • Security and Isolation • Performance and unneeded overhead Problems with Current Nested Containers Networking 16
  • 17.
    Copyright © PLUMgrid,Inc. 2011-2016 Problems with Current Nested Containers Networking 17 Docker 0 OVS VXLAN Overlay VM Docker 0 Neutron Plugin VXLAN Overlay VM SDN Overlay Neutron Overlay
  • 18.
    Copyright © PLUMgrid,Inc. 2011-2016 Problems with Current Nested Containers Networking 18 Neutron Networks VMVM VM Tenant A Net1 192.168.1.0/0
  • 19.
    Copyright © PLUMgrid,Inc. 2011-2016 Problems with Current Nested Containers Networking 19 Container Networks VMVM VM Backend Network 10.2.0.0/24 Endpoint Endpoint Endpoint Endpoint Endpoint Endpoint Endpoint Frontend Network 10.1.0.0/24
  • 20.
    Nested Container NetworkingSolution Design for the nested container networking in OpenStack 20
  • 21.
    Copyright © PLUMgrid,Inc. 2011-2016 • Nested/baremetal container to nested/baremetal container same/ different hosts • Nested/baremetal container to virtual machine communication • Nested/baremetal container to baremetal communication • Container networking as a first class entity in Neutron • Consistent policy enforcement across containers, VMs, bare metal • Enable advanced networking services like FWaas, LBaas, VPNaas etc Nested Container Networking Use Cases 21
  • 22.
    Copyright © PLUMgrid,Inc. 2011-2016 Nested Container Networking Design Magnum, Kuryr, Neutron Integration 22 VLAN:100 VLAN:200 VLAN:400 VLAN:100
  • 23.
    Copyright © PLUMgrid,Inc. 2011-2016 Neutron Trunk Ports 23 Nova Instance port-1 port-0 port-2 network-1 network-0 network-2 Port combined into one vif by turning port-0 into trunk and other ports into supports of the trunk
  • 24.
    Copyright © PLUMgrid,Inc. 2011-201624
  • 25.
  • 26.
    Copyright © PLUMgrid,Inc. 2011-2016 • Neutron resources spec approved and patches under review • Trunk • Subport • Subports bring isolation to container-in-VM use cases • Port forwarding can take us further • Vendors can implement new segmentation types • Tagged traffic that does not match a sub port, is considered of the trunk port Capabilities and Considerations 26
  • 27.
    Copyright © PLUMgrid,Inc. 2011-2016 • Limitations • Policy is applied at the Host level • Initial only VLAN tags for segmentation type • Tags are unique per “trunk port” scope • VM users can alter subport traffic • Logging of VM actions is dependent on integration • Can't work with current OVS Capabilities and Considerations 27
  • 28.
  • 29.
    Copyright © PLUMgrid,Inc. 2011-2016 • Trunk Port Extension spec approved and code in progress • Binding profile workaround to proceed in parallel • Nested Container networking spec approved in Kuryr • Docker Swarm Integration completed • Kubernetes in progress • Mesos in design stages Current Status 29
  • 30.
  • 31.
    Copyright © PLUMgrid,Inc. 2011-2016 • Follow up on the Neutron Trunk port implementation • Finish COE baremetal integration • Policy translation • Make Neutron resources available through native APIs • Magnum deployment prototype of worker VM with Kuryr agent • Magnum administrator VM that communicates with Neutron Next Steps 31
  • 32.
  • 33.
    Join us at#openstack-kuryr THANK YOU! irc: #openstack-kuryr @ freenode