Quantum Modular L2Plugin and Agent Bob Kukura Red HatGrizzly Design Summit 10/15/2012
History● Prior to Folsom, Quantum assumed “green-field” cloud data center – Single networking technology – Uniform connectivity – No access to existing networks – Exception: Cisco plugins sub-plugins for UCS, Nexus● Red Hat Quantum/oVirt meeting March 2012 – Discussed Quantum “gaps” preventing use by oVirt for enterprise virtualization – Many apply to private OpenStack clouds● Presented “Quantum in the Data Center” at Folsom design summit – Existing networks – Heterogeneous technology – Non-uniform connectivity – Deployability issues● Folsom features – Provider networks – Metaplugin – RPC support
Terminology● Network – An abstract Quantum isolated L2 network whose ports can be attached to VMs, agents, etc.● Tenant Network – A “normal” Quantum network created by a tenant.● Provider Network – A Quantum network administratively created to map to a specific existing network in the data center.● Network Type – The specification by which a Quantum network segment is realized “on the wire” (i.e. VLAN, GRE tunnel, flat).● Physical Network – A specific network “wire” supporting a set of Quantum networks.● Segmentation ID – An identifier distinguishing Quantum networks of the same network type from each other on the same physical network (i.e. VLAN tag or tunnel ID).● Network Mechanism – A host networking facility that can provide access to networks of one or more network type (i.e. OVS or Linux bridging).● Network Segment – A portion of a network implemented with a particular network type and associated details.
Current Plugin CapabilitiesPlugin Tenant Network Provider Network Network Types Types Mechanismsopenvswitch VLAN, GRE, local VLAN, GRE, flat, Open vSwitch local contolled locally via agentlinuxbridge VLAN, local VLAN, flat, local Linux bridging controlled via agentnec Trema, PFC Not implemented Trema, PFC controllers with agent to discover portsryu Ryu Not implemented Ryu controllernvp NVP Not implemented Nicira NVP controllercisco VLAN Not implemented UCS, Nexus switches, sub- pluginsmetaplugin sub-plugins sub-plugins sub-plugins
Problem Statement● Current Quantum plugins each support a single L2 networking technology – Typically a specific OpenFlow controller – Focus is on isolating L2 tenant networks● Also need to access provider networks – External networks – Existing data center networks● May have a mixture of networking technologies and mechanisms supporting them – Different systems accessing the same VLAN trunks via Linux bridging, OVS, and Cisco UCS – Combination of legacy networking and SDN in the data center – Physical appliances (LB, firewall, routing, VPN, etc.)● Quantum needs to support multiple L2 networking technologies simultaneously – Multiple types of networks – Multiple mechanism to access a network type – Networks made up of multiple segments, possibly of different types
Options● Monolithic plugin – Pick a plugin that supports everything you need – Add provider network capabilities to controller-based plugins ● Via controller? ● Via parallel mechanism (L2 agent, bridging, OVS)?● Meta-plugin – Support multiple Quantum plugins simultaneously – Several possible semantics: ● Each network belongs to exactly one sub-plugin ● Each network created in all sub-plugins● Modular Plugin – Separate the network type from the mechanism a system uses to access that network type – Drivers for network types – Drivers for mechanisms
Meta-plugin Overview● Wraps multiple real plugins – Gary coined name “rosetta-plugin”● Flavor extension – “flavor:network” attribute on network identifies implementing plugin – Similar “flavor:router” attribute on router● DB table – Maps network and router IDs to implementing plugin● Plugin operation – Network/router create – dispatch to plugin named by flavor or use default, record flavor mapping – Other network/router operations – read flavor mapping, dispatch to implementing plugin● Agent wrappers?● MetaInterfaceDriver for VIFs
Meta-Plugin Limitations● Considered “experimental” in Folsom● Tightly coupled to plugin implementations – Requires that they all inherit QuantumDBPluginV2, use same inherited DB tables – Cant have conflicting DB tables – L3 must be compatible● Cannot make same network accessible via multiple plugins – Common with data center provider networks – Could be required for VLAN tenant networks● flavor:router extended attribute not visible because router is an extension
Cisco Plugin(s)● Readme at http://wiki.openstack.org/cisco-quantum: – “A reference implementation for a Quantum Plugin Framework” – “Supports use of multiple L2 technologies”● Main network_plugin.PluginV2 + sub-plugins – Supports Cisco UCS, Nexus switch, openvswitch – Seems to delegate most calls to all sub-plugins, not just one “owning” network like in meta-plugin● Several Quantum API extensions● Tightly coupled with openvswitch plugin
Is there a better approach?● Existing linuxbridge and openvswitch plugins are almost clones – DB schemata for VLAN, flat, and local networks could be identical – Support for GRE networks in openvswitch only meaningful difference in plugin – Very little work to make linuxbridge agent work with openvswitch plugin – Agents also have much similar code● Most plugin functionality is specific to the supported network types, not to the mechanism used to access the network type – DB schema – Tenant network pooling and allocation – Provider attribute validation● So far, all provider network types can be represented by network_type, physical_network, segmentation_id tuple● Should be able to add new technologies/mechanisms be supported without having to write a whole new plugin
Modular L2 Plugin & Agent● Make multiple networking technologies work together – Single plugin and L2 agent for all network types and network mechanisms where an external controller doesnt “own the world”● Pluggable network type drivers in server – Each supports a single network type (i.e. VLAN, GRE, flat, local, VXLAN, ...) – Provides any needed DB schema – Validates and manages provider attributes – Optionally pools/allocates tenant networks● Pluggable mechanism drivers in L2 agent – Execute network administration commands to realize supported network types using a specific networking mechanism (OVS, Linux bridging, ...)● Pluggable mechanism drivers in server – Interface with external network controllers – Remotely manage OVS on nodes
Modular L2 Proposed Plan● Start simple – Merge openvswitch and linuxbridge functionality – Refactor into modular drivers for network types and agent-based mechanisms● Avoid short-term risk – Maintain existing plugins throughout Grizzly cycle – Possibly deprecate non-modular openvswitch and linuxbridge plugins at Grizzly release● Evolve – Add support for multiple-segment networks – Add server-based drivers ● OpenFlow controller drivers ● UCS, Nexus drivers ● Remote OVS driver? – Improve Nova integration – Add provisioning API extension – Add scheduler support for non-uniform connectivity● Meta-plugin future?● Monolithic plugins future?
Plugin/Agent/Driver Interactions● Define driver APIs via abstract classes implemented by drivers● Provider network creation – Dispatch to network type driver based on specified network type – Driver validates parameters, manages DB● Tenant network creation – Network type drivers optionally support allocation – Can use pools if needed – Possibly support allocation of specific network types● Port plugging – Current VIF driver approach would support single mechanism per node ● Triggered by tap device discovery, etc. – Proposed Nova->Quantum call to select/configure VIF driver could support modular plugin picking “best” of multiple mechanisms for node to connect to segment of network
Network Segments● Allow single L2 network to span multiple technologies – e.g. Switch connects VLAN 20 on one physical network to VLAN 30 on a different physical network● Introduce segment abstraction underneath network – Network has one or more segments – network_type/physical_network/segmentation_id tuple belongs to segment rather than network● Not visible in core API – Usually single segment per network – Create 1st segment automatically when creating network – Ability to add additional segments to network● No immediate plan to manage bridging between segments, but could
Non-uniform Connectivity● Several scenarios – Existing linuxbridge and openvswitch agents might not have connections to all physical networks – Existing openvswitch agents may support GRE on some nodes but not others – Modular plugin means some nodes may not have mechanisms supporting all network types in use● Possible solutions – Avoid non-uniformity – Align Quantum connectivity with Nova construct (cell, zone, flavor, ...?) – Modular plugin answer queries from Nova Scheduler filter plug-in● More complex topology – Segment concept so “same” L2 network can be realized in different places by different technologies – Scheduler taking topology, latency, throughput into account
Provisioning API● Current openvswitch and linuxbridge plugins use per-node config to map physical networks to bridges/interfaces● Modular plugin needs to understand connectivity – pick segment and mechanism to use when port plugged – support Nova scheduling● Move mappings to server? – Provisioning/management API – Store in DB – Use profile to avoid duplication across identical nodes
Discussion● Does this make sense at least for openvswitch and linuxbridge?● Is there interest in drivers for OpenFlow controllers in this framework?● Can/should Cisco UCS and Nexus support be refactored as drivers in this framework?● Is metaplugin still needed for some scenatios?