Overview of Distributed Virtual Router (DVR) in Openstack/Neutron

Vivekanandan Narasimhan
Paris OpenStack® Summit

Agenda
 Introduction
 High level architecture and DVR Configuration
 East-West Routing
 OVS Rules
 North-South Routing
 API Changes and DB Extensions
 Scheduling
 Services
 Plans for Kilo
chitectural Overview of Distributed Virtual Routers in OpenStack Neutron Paris OpenStack® Summit

Legacy Routing in Neutron
Network node provides:
• IP forwarding
– Inter-subnet (east-west) traffic between VMs
– Floating IP (north-south) traffic between
external and VM
– Default SNAT (north-south) traffic from VM to
external
• Metadata Agent
– access to Nova metadata service
Issues:
• Performance bottleneck
• Scalability limitations
• Single Point of Failure
chitectural Overview of Distributed Virtual Routers in OpenStack Neutron
Compute
node
Network
node
Compute
node
VM1 VM2
VM
1
VM
2
Virtual
Physical
Router
A
Rtr A

Distributed Routing in
Neutron
Compute nodes provide:
• IP forwarding for local VMs
– Inter-subnet (east-west) traffic between VMs
– Floating IP (north-south) traffic between external
and VM
• Metadata Agent for local VMs
– access to Nova metadata service
Advantages:
• Bypass network node improves performance
• Scales with size of compute farm
• Limited failure domain (per compute node)
Limitations:
• Default SNAT function is still centralized
Compute
node
Network
node
Compute
node
VM1 VM2
VM
1
VM
2
Virtual
Physical
Router
A
Rtr A
Default
SNAT
Rtr
A’’
Rtr A’

High level requirements for
DVR
• Provider feature
– Tenants should not have to care
• Default router type set by global config knob
– Distributed or Legacy
• Distributed and Legacy routers coexist in same cloud
• Minimize overhead use of public IP addresses
– one public IP per compute node
• Leverage existing code base
• Ability to migrate from Legacy to Distributed

DVR High Level Architecture
and Configuration

Network/Service Node(s)
Compute Node(s)
VM
VM
VM
l3-agentovs-
agent
Controller Node
Neutron controller/plugin L3 Plugin
Network/Service Node(s)
l3-agentovs-agent
dhcp-agent qr
qrqr
agent_mode=dvr
router_distributed=True
enable_distributed_routing=True
agent_mode=dvr_snat
enable_distributed_routing=True
• One FIP namespace supports multiple routers
• Legacy OpenStack Routers (non-distributed) can exist on the network/service node
• Enhanced l3-agent works in different modes “dvr_snat”, “legacy” and “dvr”.
FIP
SNAT
metadata
metadata
qrqr
DVR High level Architecture

Configuring DVR

Plugin Configuration ( Default Router Option)
“neutron.conf”
router_distributed= True
L3 Agent Configuration and Modes
“l3_agent.ini”
agent_mode=dvr_snat ( Network/Service Node)
agent_mode=dvr (Compute Node only)
agent_mode=legacy ( Network/Service Node)
router_namespaces=True ( Enable namespace cleanup)(Optional)
L2/OVS Agent Configuration
“ml2_conf.ini”
[ovs]
enable_tunneling = True
local_ip = <Local Data Network IP (or) TEP IP>
[agents]
tunnel_types = vxlan
l2_population=True
enable_distributed_routing = True
hitectural Overview of Distributed Virtual Routers in OpenStack Neutron
DVR Configuration Overview

br-int br-intbr-int
br-tun br-tun br-tunbr-ex
qr
VM1 VM2 VM3 VM4DHCP
Agent
L3 Agent
OVS
Agent
Q_DVR_MODE=legacy
External Network
Internal Network
Network Node Compute Node Compute Node
OVS
Agent
Legacy Deployment

br-int br-intbr-int
br-tun br-tun br-tunbr-ex
qr
VM1 VM2 VM3 VM4DHCP
Agent
L3 Agent
OVS
Agent
Q_DVR_MODE=dvr_snat Q_DVR_MODE=dvr Q_DVR_MODE=dvr
L3
Agent
L3
Agent
qr qrqr
snat
Network/Service Node Compute Node Compute Node
DVR Deployment without FIP
br-ex br-ex

br-int br-int
br-tun br-tun
VM1 VM2 VM3 VM4
OVS
Agent
L3
Agent
L3
Agent
qr qr
fip fip
br-ex br-ex
DVR Deployment with FIP
br-int
br-tunbr-ex
qr
DHCP
Agent
L3 Agent
qr
snat
Network/Service Node Compute Node Compute Node

East-West Routing

L3 Agent
Rules
------
------
Rules
------
------
vm1 vm2
br-int
br-tun
qr
LMAC
East-West routing is accomplished
through a combination of the following
in each CN:
•qr : a namespace that forwards traffic
among routed subnets using route table
entries.
•LMAC: a MAC address unique to each
CN for use as source address in
forwarding routed frames from one CN to
another.
•A set of rules in the OVS bridges that
prevent frames with src=gateway MAC
from egressed tunnel bridge or the IR.
These rules also cause peer bridges to
substitute/restore gateway MACs with
LMACs while routed frames traverse
through br-tuns and underlay network.
br-ex
Compute Node entities for enabling the
DVR East West Routing
L3 Agent

vm1
br-int-cn1
br-tun-cn1
qr
vm2
br-int-cn2
br-tun-cn2
qr
Tenant has two VMs - vm1 that is in RED Net & vm2 that is in GREEN Net
CN1 and CN2 are two compute nodes. qr represents a dvr owned by Tenant 1
Data Network
CN1 CN2
PING REQUEST from vm1 to vm2
vm1 ipqr red macvm1 mac vm2 ip
vm1 ip
qr red
mac
vm2 ip
vm1 ipvm2 macqr grn mac vm2 ip
vm1 ipvm2 macdvr cn1 mac vm2 ip
vm1 ipvm2 macdvr cn1 mac vm2 ip
1
2
3
4
5
grn vni
grn 2
vlan
vm1 ipvm2 macqr grn mac vm2 ip6
vm1 mac
Flow of DVR routed packets (example)
Architectural Overview of Distributed Virtual Routers in OpenStack Neutron

Table 0: Dispatch from in_port
from patch-int:
goto table 1
default:
drop
Table 1: DVR process Table
If packet is from dvr-router-intf-
mac, replace SRCMAC with DVR-
node-unique-mac and forward to
table 2
Any other packet, forward as is to
table 2
Table 2: Local VLAN to TUN Table
If unicast frame, forward to table
20
If broadcast/multicast frame,
forward to table 21
Table 20: Unicast to Tunnel Table
Pre-populated L2-pop rules that
bind dst-mac in frame to a single
output tunnel
Frame out to
single tunnel
Table 0: Local Switching Table
from patch-tun:
Is src-mac == DVR-Unique-
MAC - goto table 1
default:
forward packet to ports -
NORMAL action
1
2
to dvr
router
Integration Bridge (br-int)
Tunnel Bridge (br-tun)4
5
6
3
from dvr router
7
8
Egress to Cloud

Table 0: Dispatch from in_port
from vxlan-port:
forward to table 4
default:
drop
Table 4: VXLAN Tun to Local
Vlan Table
match tun_id:
Set local vlan-id, forward to
table 9
default:
drop
Table 9: DVR Learning blocker
If src_mac == DVR-Unique-MAC
frame, don’t learn, just output
to patch-int
default:
forward to table 10
4
Integration Bridge (br-int)
Tunnel Bridge (br-tun)
1
2
3
Ingress from Cloud
Table 0: Local Switching Table
from patch-tun:
Is dvr_routed - goto table 1
default:
forward packet to ports -
NORMAL action
Table 1: DVR to LMAC Table
match DVR-Unique-MAC:
strip off dvr-unique-mac,
add local dvr-router-intf-mac,
output direct to vm-port
default:
drop
5
vm1
Architectural Overview of Distributed Virtual Routers in OpenStack Neutron Paris OpenStack® Summit
6

North-South Routing

Overview of North-South Routing
External Network
vm1 vm3
br-int
br-tun
vm5
br-ex
• IP Rules
•Routing Entries
• Iptable based NATing rules
• /32 addresses for the floating IPs
• One Floating IP Namespace per
CN
• Interface on external network
• Proxy ARPs enabled for hosted
floating IPs
qr
FIP
L3 Agent
Data Network

Default SNAT Traffic flow

API Changes and DB
Extensions

API Extensions
Adds ‘distributed’ attributed to ‘router’ object
Router Create:
neutron router-create –distributed=True/False
Router Update:
neutron router-update –distributed=True/False
Can be set by admin user through the above API
Global default is set as “router_distributed” in neutron.conf
The attribute is only visible to admin tenant in GET.
NOTE: Migrating or Converting a Legacy Router to Distributed is Work in
Progress.

DB Changes for DVR
ml2_dvr_port_bindings

Scheduling

br-int
br-tun
VM3 VM4
OVS
Agent
Q_DVR_MODE=dvr_snat
Q_DVR_MODE=dvr
L3
Agent
qr
br-ex
Scheduling a “qr/snat” in Service/Compute Node
br-int
br-tunbr-ex
qr
DHCP
Agent
L3 Agent
qr
snat
Schedule Events
 Create a Router
 Add one or more subnets
with VMs
 Set a default Gateway for
the Router
Compute NodeService/Network Node

br-int
br-tun
VM3 VM4
OVS
Agent
Q_DVR_MODE=dvr_snat Q_DVR_MODE=dvr
L3
Agent
qr
fip
br-ex
br-int
br-tunbr-ex
qr
DHCP
Agent
L3 Agent
qr
snat
Network/Service Node Compute Node
Scheduling a “fip” in Compute Node
Schedule Events
 Make sure you have a router
configured with External
Network or gateway.
 Create a Floating IP
 Associate a Floating IP to a
VM

Un-Scheduling Routers and FIP
Un Schedule Events
FIP Namespace cleanup
 Last VM holding the FIP determines the FIP namespace cleanup
 Delete a FloatingIP
 Disassociate a Floating IP
Router Namespace cleanup
 When no more DVR related ports ( Includes Compute, VIP, DHCP etc.,) are
serviced by the routed subnet, the router namespaces are cleaned up.
 Both l2 agent and l3 agent takes part in the clean up.
SNAT Namespace cleanup
 When a gateway is removed from a router the snat namespace will be
cleaned up.
NOTE: Namespace cleanup should be enabled “router_namespaces=True” in
l3_agent.ini

Services

Services support with DVR
Services Support in DVR as of Juno
 LBaaS
 FWaaS
• North-South only ( External Traffic)
• East-West ( Internal Traffic)
 MetadataService
 VPNaaS is still supported with Centralized Routers.
 VPNaaS support for DVR is Work in Progress.

br-int br-int
br-tun br-tun
VM1 VM2 VM3 VM4
OVS
Agent
L3
Agent
L3
Agent
qr qr
fip fip
br-ex br-ex
Firewall Firewall
LBaaS
AgentVIP
br-int
br-tunbr-ex
qr
DHCP
Agent
L3 Agent
qr
snat
Network/Service Node
Firewall
Compute Node Compute Node
Firewall
Services deployment with DVR
Meta
data
Meta
data

DVR Enhancements in kilo
 VPNaaS support for DVR
 Full migration support for DVR routers.
 HA for Service Node
 VLAN Support

Liberty Plans for DVR
 L3 Agent Refactor
 Distributed DHCP
 Performance tuning.
 Distributed SNAT

Questions?

Overview of Distributed Virtual Router (DVR) in Openstack/Neutron

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Overview of Distributed Virtual Router (DVR) in Openstack/Neutron

Similar to Overview of Distributed Virtual Router (DVR) in Openstack/Neutron (20)

Recently uploaded

Recently uploaded (20)

Overview of Distributed Virtual Router (DVR) in Openstack/Neutron