Neutron kilo

Neutron Networking / OpenStack Kilo
Lukáš Korous

Lukáš Korous
Outline
0. OpenStack review, training environment setup
a. User & Administrator dashboard use
b. Training environment setup & walkthrough
1. Related networking technologies
a. Linux bridges and namespaces(,kernel settings – sysctl)
b. Network segmentation, encapsulation
i. vLAN
ii. GRE
iii. VXLAN
c. SDN Overview
2. Open vSwitch
a. Description, comparison with Linux bridge
b. Configuration
3. Neutron concept and relationship to other OpenStack components
a. Overview
b. Relationship to other components & projects
4. OpenStack commands for Neutron checking
5. Neutron configuration files
a. Controller
i. neutron.conf
ii. nova.conf
iii. plugins/*/*.ini
iv. dnsmasq-neutron.conf
b. Nodes
i. l3_agent.ini
ii. dhcp_agent.ini
iii. metadata_agent.ini
iv. plugins/*/*.ini
6. OpenStack network architecture
a. Public network
b. External (flat) network
c. Management network(, SSH network)
d. Tunnel (internal) network
7. Network traffic schemes
a. VM <-> VM
b. VM -> Outside world
c. Outside world -> VM (Floating IP)
d. Outside world -> VM (directly on Public network)
8. Network traffic inspection
a. Host level, guest level

Lukáš Korous
OpenStack review, training environment setup
OpenStack user dashboard
Testing dashboard address
http://test.ultimum-cloud.com/
Login
Please contact Ultimum Technologies at info@ultimum.io for access to the testing environment.
Keypairs for SSH
In the left menu, select "Access & Security", and after opening in the right panel, select the tab "Keypairs".
Now there are two ways how to add an SSH key - either create a new one, or import an existing one.
Create a key for SSH
In the top buttons row (see screenshot) select "Create Keypair". In the dialog, insert a name, e.g.
"myFirstKeypair".

Lukáš Korous
Confirm by selecting "Create Keypair". Keypair is created and you can see it on the list.
Import a key for SSH
In the top buttons row (see screenshot) select "Import Keypair".

Lukáš Korous
In the dialog, fill in a name, e.g. "myImportedKeypair", and copy your public key to the "Public Key" field. If
you do not know where to look for a public key, it is usually in your home folder (typically "/home/'user'" on
Linux, "C:Users'user'" on Windows), in subdirectory (caution, may be hidden) ".ssh", in a file id_rsa.pub
(id_dsa.pub).
The whole key, e.g. the entire content of the file id_rsa.pub, should then be copied into this field.
Confirm by selecting "Import Keypair". Keypair is created and you can see it on the list.
Security Groups
In the left menu, select "Access & Security", which opens in the right panel. Select the "Security Groups" tab.
In the default settings, there is one security group created - with the name "default":

Lukáš Korous
If you want to check the default security group rules, click on it and the detail will open:

Lukáš Korous
For the purpose of this tutorial though, we will create a new security group that will later allow us to
connect to our instances using SSH. On the list of security groups, press the button "Create Security Group"
and create a group for SSH connections by filling the information in the creation dialog:
The security group appears in the list:

Lukáš Korous
Adjustment of the security group rules
For adjustments of the SSH security group rules, select "Edit Rules":

Lukáš Korous
Then select "Add Rule" and add the SSH pre-defined rule (or you may fill the custom rule dialog and add it
manually). Note that the default CIDR is 0.0.0.0/0 which stands for any (=all) IP address.
The rule is created:

Lukáš Korous
Set up networking
Now we have to set up networking in order to plug the instance we are about to create into a proper
network.
Navigate to "Network Topology" in the left menu. The basic setup looks like in the following figure.

Lukáš Korous
Creating a private network for your servers
Select "Create Network" and network creation dialog appears.

Lukáš Korous
Creating a subnet
Now we have to create a subnet in the network being created. Navigate to the second tab of the dialog -
Subnet.
Subnet represents a logical set of connected devices, addressed in a common address space.
After you confirm, the network is created and it appears in the topology diagram.

Lukáš Korous
.
Creating a router
Now we have to create a router in order to be able to connect the new network to the outside world.
Select "Create Router" and name your new router. Router is created and we can see it in the router list:

Lukáš Korous
Also the router appears in the Network Topology schema:

Lukáš Korous
If you navigate to the router's detail, it opens and looks similarly to the following figure.
Now we need to create an interface, select "Add Interface" and add an interface to the private network.
When you confirm, the interface is created and we can see it on the router's detail:

Lukáš Korous
Adding a gateway to a router
Now we have to connect to the outside world using a gateway. For this, open the router list from the left
menu ("Routers") and select "Set Gateway" next to the appropriate router. The following dialog appears:

Lukáš Korous
Select the public network there and confirm.
Now the network topology reflects that the router is connected to the outside world:
First run of a cloud server
Let us now go through running a cloud server from a collection of public images (according to the operating
systems) and connect to it via the remote desktop. In the left menu, choose "Images & Snapshots". In the
right panel, a list of all available public images appears.
List of available ready-to-use images
Select "Images & Snapshots" in the left menu. In the right panel, the list of publicly available images will
appear.

Lukáš Korous
Instantiation of a public server image
Next to the selected image (for the purpose of this manual we will select "Ubuntu 13.04 cloudimg x86_64"),
select "Launch".
A dialog window appears, where on the "Details" tab we fill in the instance name, e.g. "myFirstInstance" and
we select the type of the virtual hardware (flavor).

Lukáš Korous
Before we confirm by pressing "Launch", we need to check the security settings on the "Access & Security"
tab:

Lukáš Korous
Here we should add the security group for SSH connection that we created earlier and the keypair we either
created or imported earlier as well.
We also have to address the networking. Switch to the "Networking" tab.
Here you have to assign the private network to the instance being launched.
Now we confirm by pressing "Launch". We are automatically redirected to the instance overview (available
from the left menu by selecting "Instances"), where we can see the newly created instance.

Lukáš Korous
We can also see the newly launched instance in the "Network Topology" schema:

Lukáš Korous
Public IP address assignment
For a successful connection to the newly created instance, it is important to add a public IP address. On the
instance overview we select "More" by the appropriate instance and in the context menu we select
"Associate Public IP":
A dialog appears:

Lukáš Korous
We need to select the "+" button next to the IP addresses drop-down list to add allocate a new IP address.
The allocation dialog appears.

Lukáš Korous
Select the public pool and confirm. The address is allocated and we can select it now in the previous
assignment dialog.
After confirmation, we can select "Access & Security" in the left menu and then "Public IPs" in the right
window to see the list of allocated or assigned IP addresses:

Lukáš Korous
SSH connection to your newly created instance
Now we are able to connect to this new instance.
Username: debian
Password: debian / SSH key
Here is what a terminal looks like after a successful login.

Lukáš Korous
Training environment setup & walkthrough
Networks
(details of OpenStack networks will follow in one of subsequent sections)
 Public (not part of the training environment)
o physical network in the datacenter
 Openstack-external-flat
o external (public) network from the testing environment viewpoint
 Openstack-ssh
o network dedicated for SSH, might be merged with management network, except for
deployment
 Openstack-management
o management network, access to controllers, and nodes
 Openstack-tunnel
o tunneled communication between nodes (VXLAN tunneling is used in the training
environment)
Machines
Name Public IP address Purpose OS
Controller1 XXX.YYY.ZZZ.239 Controller debian
Compute1 XXX.YYY.ZZZ.236 Compute & Network Node debian
Compute2 XXX.YYY.ZZZ.237 Compute & Network Node debian
Client1 XXX.YYY.ZZZ.238 Client machine debian
Services
List all installed OpenStack services on the controller
debian@controller1:~ $ source admin-openrc.sh
debian@controller1:~ $ openstack service list
+----------------------------------+----------+----------------+
| ID | Name | Type |
+----------------------------------+----------+----------------+
| 01dd76e265c34c98b30a80a95f4f744a | heat | orchestration |
| 0e313bbb8cc44d66b4691a65152450d3 | neutron | network |
| 41ed6e5e1ad84b5783ed52840b11bf2b | keystone | identity |
| a8761537893e490799af9cccd114df27 | cinderv2 | volumev2 |
| aa5dc322091549028b7c1b755a061dd1 | glance | image |
| c31f2edc4f14411d9efc7f3b6f9db5a6 | nova | compute |
| d5c2a54ee66648849607a8feea028662 | heat-cfn | cloudformation |
| f4cb7d150eb04920a9cfab8f24e29062 | cinder | volume |
+----------------------------------+----------+----------------+
debian@controller1:~ $

Lukáš Korous
View all services in the dashboard
Admin -> System -> System Information
-> Services
-> Compute Services
-> Network Agents

Lukáš Korous
Related networking technologies
Linux bridges
A piece of software used to unite two or more network segments. A bridge behaves like a virtual network
switch, working transparently (the other machines do not need to know or care about its existence). Any
real devices (e.g. eth0) and virtual devices (e.g. tap0) can be connected to it.
Layer 2 device.
Part of Linux kernel since 2.2 (stable as of 2.6).
$ brctl addbr mybridge
$ brctl addif mybridge eth0
$ brctl addif mybridge eth1
$ ifconfig mybridge up
$ brctl show
$ brctl showmacs mybridge
$ brctl delbr mybridge
Network namespaces
Isolated containers which can hold a network configuration and are not seen from outside of the
namespace.
Used to encapsulate specific network functionality or provide a network service in isolation as well as simply
help to organize a complicated network setup.
Used for many purposes and heavily used in OpenStack networking.
$ ip netns add ns1
$ ip netns add ns2
$ ip netns list
BUT: Cannot contain physical interfaces. Only virtual Ethernet (veth) interfaces.
Virtual Ethernet (veth) interfaces
Always come in pairs.
Connected like a tube—whatever comes in one veth interface will come out the other peer veth interface.
Used to connect a network namespace to the outside world via the “default” or “global” namespace
where physical interfaces exist.
Simple connection is achieved by the following:

Lukáš Korous
# create the veth pair
$ ip link add tap1 type veth peer name tap2
$ ip link list
# move the interfaces to the namespaces
$ ip link set tap1 netns ns1
$ ip link list # no veth here
$ ip netns exec ns1 ip link list
$ ip netns exec ns2 ip link list
# bring up the links
$ ip netns exec ns1 ifconfig tap1 10.0.0.0/24 up
Connecting namespaces by a Linux bridge
$ ip netns exec ns1 ip link delete tap1
$ ip netns exec ns2 ip link delete tap2
$ brctl addbr mybridge
$ ip link set dev mybridge up
#### PORT 1
# create a port pair
$ ip link add tap1 type veth peer name br-tap1
# attach one side to linuxbridge
$ brctl addif mybridge br-tap1
# attach the other side to namespace
# set the ports to up
$ ip link set dev br-tap1 up
#
#### PORT 2
A veth pair connecting namespaces

Lukáš Korous
$ ip link add tap2 type veth peer name br-tap2
# attach one side to linuxbridge
$ brctl addif mybridge br-tap2
$ ip link set dev br-tap2 up
Linux bridge between two tap devices
Relevant kernel settings
/etc/sysctl.conf
$ vim /etc/sysctl.conf
net.ipv4.ip_forward=1
net.ipv4.conf.all.rp_filter=0
net.ipv4.conf.default.rp_filter=0
net.bridge.bridge-nf-call-arptables=1
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1

Lukáš Korous
$ sysctl –p # save changes without reboot

Lukáš Korous
Network segmentation, encapsulation
vLAN tagging
 Layer 2
 One vLAN (One vLAN ID) = One broadcast domain
 No impact on MTU (MTU does not include ethernet header size)
Ethernet packet with VLAN Tag
[length in bytes]
Destination MAC
address
Source MAC
address
Type (VLAN: 0x8100) VLAN Tag Payload
6 6 2 4 1500
VLAN Tag
[length in bits]
Priority CFI ID Ethernet Type/Length
3 1 12 16
Example packet
Frame 53 (70 bytes on wire, 70 bytes captured)
Ethernet II, Src: 00:40:05:40:ef:24, Dst: 00:60:08:9f:b1:f3
802.1q Virtual LAN
000. .... .... .... = Priority: 0
...0 .... .... .... = CFI: 0
.... 0000 0010 0000 = ID: 32
Type: IP (0x0800)
Internet Protocol, Src Addr: 131.151.32.129 (131.151.32.129), Dst
Addr: 131.151.32.21 (131.151.32.21)
Transmission Control Protocol, Src Port: 1173 (1173), Dst Port:
6000 (6000), Seq: 0, Ack: 128, Len: 0
Disadvantages
 Only 12bits for VLAN ID => 4094 Available Ids
 STP imposes existence of a single active path => resiliency issues
GRE tunneling
 Layer 3
 Overlay network / Tunneling – IP packet is wrapped in another IP packet

Lukáš Korous
Ethernet packet tunneled through GRE
[length in bytes]
MTU considerations
 overhead is 24 bytes
 MTU by default 1476 bytes, but if DF bit set, packets dropped
 GRE clears the DF bit, unless 'tunnel path-mtu-discovery' set
Example packet
Frame 1: 138 bytes on wire (1104 bits), 138 bytes captured (1104
bits)
Ethernet II, Src: c2:00:57:75:00:00 (c2:00:57:75:00:00), Dst:
c2:01:57:75:00:00 (c2:01:57:75:00:00)
Internet Protocol Version 4, Src: 10.0.0.1 (10.0.0.1), Dst:
10.0.0.2 (10.0.0.2)
Generic Routing Encapsulation (IP)
Flags and Version: 0x0000
Protocol Type: IP (0x0800)
Internet Protocol Version 4, Src: 1.1.1.1 (1.1.1.1), Dst: 2.2.2.2
(2.2.2.2)
6000 (6000), Seq: 0, Ack: 128, Len: 0
Transport Ethernet
header
Transport IP header
[Protocol = GRE(47)]
GRE header Original IP header Payload
18 20 4 20 1500 (1476)
GRE schema

Lukáš Korous
VXLAN Tagging
 Layer 4, default port: 4789
 Overlay network / Tunneling – Ethernet packet is wrapped in a UDP packet
VTEP - VXLAN Tunnel End Point
 VTEP is located within the hypervisor
 VTEP is assigned an IP address and acts as an IP host to the IP network
Ethernet packet tunneled through VXLAN
[length in bytes]
Recommended to increase the MTU to 1550:
VxLAN schema

Lukáš Korous
- 1500 byte payload which includes the original IP header/s.
- 14 byte inner Ethernet header
- 8 byte VXLAN header
- 8 byte UDP header
- 20 byte IP header
Transport Ethernet
header
Transport IP
header
Transport UDP
header
VXLAN header
Original
Ethernet header
Payload
18 20 8 8 18 1500
Example packet
Frame 1: 148 bytes on wire (1184 bits), 148 bytes captured (1184
bits)
Ethernet II, Src: c2:00:57:75:00:00 (c2:00:57:75:00:00), Dst:
c2:01:57:75:00:00 (c2:01:57:75:00:00)
Internet Protocol Version 4, Src: 10.0.0.1 (10.0.0.1), Dst:
10.0.0.2 (10.0.0.2)
Protocol: UDP (17)
User Datagram Protocol, Src Port: 46219 (46219), Dst Port: 4789
(4789)
Virtual eXtensible Local Area Network
...
.... 1... = VXLAN Network ID(VNI): Present
...
VXLAN Network Identifier (VNI): 1234
Ethernet II, Src: a6:00:57:75:00:00 (a6:00:57:75:00:00), Dst:
b3:01:57:75:00:00 (b3:01:57:75:00:00)
Internet Protocol Version 4, Src: 1.1.1.1 (1.1.1.1), Dst: 2.2.2.2
(2.2.2.2)
6000 (6000), Seq: 0, Ack: 128, Len: 0

Lukáš Korous
SDN Overview
SDN – source: opennetworking.org
Control Plane
Planning, setup, internal traffic, configuration, management, routers ‘talking’.
Routers exchange topology information and construct routing tables based on a routing protocol, for
example Routing Information Protocol (RIP) or Open Shortest Path First (OSPF).
Control plane packets are processed by the router to update the routing table information.
Analogy with telephony – network signaling.
Data Plane
Also known as Forwarding Plane.
Analogy with telephony – network media.
Handles ‘user’ or ‘data’ traffic according to the setup from the Control Plane.

Lukáš Korous
SDN with OpenStack
Source: docs.openstack.org:
If an SDN implementation requires layer-2 access because it directly manipulates switches, we do not
recommend running an overlay network or a layer-3 agent. If the controller resides within an OpenStack
installation, it may be necessary to build an ML2 plug-in and schedule the controller instances to connect to
tenant VLANs that then talk directly to the switch hardware. Alternatively, depending on the external device
support, use a tunnel that terminates at the switch hardware itself.
OpenStack hosted SDN controller
OpenStack participating in an SDN controller network

Lukáš Korous
Open vSwitch (OVS)
Description, setup
Not a full-stack SDN / SDDC platform: focused on providing L2/L3 virtual networking.
Advantage of using (OVS) over Linux bridge as a L2 switching backend for Neutron is mainly performance.
Replacing Linux bridge
In the section about Linux bridges we used Linux bridges to connect virtual Ethernet interfaces tap1, tap2
(These will serve as vNICs for OpenStack VMs).
Commands
Here is how we can achieve the same with OVS:
# clean the namespaces if they exist
root@server:~ $ ip netns delete ns1
root@server:~ $ ip netns delete ns2
# add the namespaces
root@server:~ $ ip netns add ns1
root@server:~ $ ip netns add ns2
# create the switch
root@server:~ $ ovs-vsctl add-br myovs
#### PORT 1
root@server:~ $ ip link add tap1 type veth peer name ovs-tap1
# attach one side to ovs
root@server:~ $ ovs-vsctl add-port myovs ovs-tap1
root@server:~ $ ip link set tap1 netns ns1
root@server:~ $ ip netns exec ns1 ip link set dev tap1 up
root@server:~ $ ip link set dev ovs-tap1 up
#### PORT 2
root@server:~ $ ip link add tap2 type veth peer name ovs-tap2
# attach one side to ovs
root@server:~ $ ovs-vsctl add-port myovs ovs-tap2
root@server:~ $ ip link set tap2 netns ns2

Lukáš Korous
root@server:~ $ ip netns exec ns2 ip link set dev tap2 up
root@server:~ $ ip link set dev ovs-tap2 up
Schema
This is what we will end up with:
Problem (OpenStack - related) – We cannot (so far) impose firewall (security group) rules on OVS
Currently the implementation employs a standard Linux Bridge „in the middle“ that does no bridging,
merely filtering using iptables.
Replacing Linux bridge with Open vSwitch for namespaces connection

Lukáš Korous
(New info (20th October 2015): „Changed in neutron: importance: Undecided → Wishlist“)
Open vSwitch ports
Veth pairs can be replaced by openvswitch ports for better performance results:
Open vSwitch ports
# add the namespaces
ip netns add ns1

Lukáš Korous
ip netns add ns2
# create the switch
ovs-vsctl add-br ovstest
# PORT 1
# create an internal ovs port
ovs-vsctl add-port ovstest tap1 -- set Interface tap1 type=internal
# attach it to namespace
ip link set tap1 netns ns1
ip netns exec ns1 ip link set dev tap1 up
# PORT 2
# create an internal ovs port
ovs-vsctl add-port ovstest tap2 -- set Interface tap2 type=internal
# attach it to namespace
ip link set tap2 netns ns2
ip netns exec ns2 ip link set dev tap2 up
Performance comparison
Switch and Connection type
# of iperf threads
1 2 4 8 16
linuxbridge with two veth pairs 3.9 8.5 8.8 9.5 9.1
openvswitch with two veth pairs 4.5 9.7 11 11 11
openvswitch with two internal ovs ports 42 69 76 67 74
Performance comparison Linux Bridge vs. OVS [GBit/s] on a single machine between two
namespaces – source: opencloudblog.com

Lukáš Korous
Full picture
OVS Overview

Lukáš Korous
Management
Sources: openvswitch.org, yet.org
Two kinds of flow
 OpenFlows - User Space based
 Datapath - kernel based, a kind of cached version of the OpenFlow ones.
CLI parts
 ovs-vsctl - high level interface for Open vSwitch Database
 ovs-ofctl - speak to OpenFlow module
 ovs-dpctl - speak to Kernel module
ovs-vsctl
 ovs-vsctl - V version of openvswitch
 ovs-vsctl show - print a brief overview of database configuration
 ovs-vsctl list-br - list of configured bridges
 ovs-vsctl list-ports <bridge> - list of ports on a specific bridge
 ovs-vsctl list interface - list of interfaces
ovs-ofctl
 ovs-ofctl dump-flows <br> - examine OpenFlow tables
 ovs-ofctl show <br> - port number to port name mapping
 ovs-ofctl dump-ports <br> - port statistics by port number
ovs-dpctl
low level datapath manipulation
 ovs-dpctl show - basic info
 ovs-dpctl dump-flows - dump datapath (kernel cached) flows
Data Plane Development Kit (DPDK) with OVS
Development state analysis
Component Development Documentation Testing
Production
readiness
DPDK library
OK (http://dpdk.org/dev)
Last commit: today
OK OK
Intel - OK
Other –
Scarse use
(80%)
OVS 2.4
(needed for
DPDK 2.0)
OK
Last commit: today
OK OK 100%
OVS
compiled
with DPDK
Poor (github.com/01org/dpdk-ovs)
Last commit: 8 months ago
Very poor (e.g.
makefiles
wrong)
DPDK Version 1.7
– Poor
10% – 50%

Lukáš Korous
DPDK Version 2.0
– Non-existent
OVS + DPDK
in Neutron
In development
(https://github.com/openstack/netw
orking-ovs-dpdk)
Last commit: a week ago
Non-existent Non-existent (?) 0% - 20%
Mechanism needed for passing (very) custom parameters to qemu when running instances
(to use DPDK features). Such mechanism is a part of nova, although this is not its intended
use and such use may not be stable.

Lukáš Korous
OpenContrail and comparison Neutron+OVS vs. OpenContrail+vRouter
Source: opencontrail.org
Configuration node
Management layer
North-bound APIs:
 ReST
 GUI
Integration

Lukáš Korous
Transformation
Horizontally scalable
Failover-safe (including active-active HA setup)
Control node
Control layer
South-bound APIs
 vRouters
 physical routers
Analytics node
North-bound analytics APIs
Triggers, monitoring
vRouter on an OpenStack network/compute node
vRouter agent
The vRouter agent is a user space process running inside Linux. It acts as the local, light-weight control plane
and is responsible for the following functions:

Lukáš Korous
 Exchanging control state such as routes with the Control nodes using XMPP.
 Receiving low-level configuration state such as routing instances and forwarding policy from the
Control nodes using XMPP.
 Reporting analytics state such as logs, statistics, and events to the analytics nodes.
 Installing forwarding state into the forwarding plane.
 Discovering the existence and attributes of VMs in cooperation with the Nova agent.
 Applying forwarding policy for the first packet of each new flow and installing a flow entry in the
flow table of the forwarding plane.
 Proxying DHCP, ARP, DNS, and MDNS. Additional proxies may be added in the future.
Each vRouter agent is connected to at least two control nodes for redundancy in an active-active
redundancy model.
vRouter forwarding plane
The vRouter forwarding plane runs as a kernel loadable module in Linux and is responsible for the following
functions:
 Encapsulating packets sent to the overlay network and decapsulating packets received from the
overlay network.
 Assigning packets to a routing instance:
o Packets received from the overlay network are assigned to a routing instance based on the
MPLS label or Virtual Network Identifier (VNI).
o Virtual interfaces to local virtual machines are bound to routing instances.
o Doing a lookup of the destination address of the packet in the Forwarding Information
Base (FIB) and forwarding the packet to the correct destination. The routes may be layer-
3 IP prefixes or layer-2 MAC addresses.
o Optionally, applying forwarding policy using a flow table:
 Match packets against the flow table and apply the flow actions.
 Optionally, punt the packets for which no flow rule is found (i.e. the first packet
of every flow) to the vRouter agent which then installs a rule in the flow table.

Lukáš Korous
 Punting certain packets such as DHCP, ARP, MDNS to the vRouter agent for
proxying.
Comparision Neutron+OVS vs. OpenContrail+vRouter
Production readiness
OVS
 Initial commit: before 8.7.2009
 https://bugs.launchpad.net/neutron/+bugs?field.tag=ovs
vRouter
 Initial commit: 24.8.2013
 https://bugs.launchpad.net/juniperopenstack/+bugs?field.tag=vrouter
Management + functional level (SDN)
Neutron + OVS
 OpenFlow
 API – native
o Sec. groups, FwaaS, LbaaS, Vpnaas
o Will support QoS in Liberty
 GUI – e.g. Floodlight
 DPDK
 Neutron initial commit: 11.5.2011
OpenContrail + vRouter
 Complete platform, GUI & API included

Lukáš Korous
 Dynamic routing
 Service chaining
- Initial commit: 24.8.2013
Performance comparison - vRouter vs. Linux Bridge
Source: opencontrail.org
Description
On this setup, the unidirectional throughput measured with vRouter using MPLS over GRE as the
encapsulation is 9.18Gbps. The CPU consumption on the sender is 128% (1.28 CPU cores) and it is 166% on
the receiver. The CPU consumption includes the processing to push packets to and from the guest and does
not include the CPU consumed by the guest itself. With bidirectional traffic (one TCP stream in each
direction), the aggregate throughput is 13.1 Gbps and the CPU consumption is 188% on both ends.
Results
The table below compares the throughput and CPU consumption of vRouter with the Linux bridge numbers
for a unidirectional TCP streaming test.
Throughput Sender CPU Receiver CPU
Linux bridge 9.41 Gbps 85% 125%
vRouter 9.18 Gbps 128% 166%
Table 1: TCP unidirectional streaming test
The table below compares the numbers for a bidirectional TCP streaming test. The throughput below is the
aggregate of the measured throughput at each end. The CPU consumption is the same on both servers as
the traffic is bidirectional
Throughput CPU consumption
Linux bridge 13.9 Gbps 128%
vRouter 13.1 Gbps 188%
Table 2: TCP bidirectional streaming test
In order to measure the latency of communication, a TCP request-response test was run between the 2
servers. The table below compares the number of request-response transactions seen with Linux bridge and
vRouter.
Request-response transactions
Linux bridge 11050
VRouter 10800
Table 3: TCP request-response test
Data center networks often enable jumbo frames for better performance. The following table compares the
performance of vRouter with Linux bridge with a jumbo MTU on the 10G interface. The guest application
was modified to use sendfile() instead of send() in order to avoid a copy from user space to kernel.

Lukáš Korous
Otherwise, the single-threaded guest application couldn’t achieve a bidirectional throughput higher than 14
Gbps.
Throughput CPU consumption
Linux bridge 18 Gbps 125%
vRouter 17.4 Gbps 120%
Table 4: TCP bidirectional streaming test (jumbo MTU)
SR-IOV (Single-root I/O Virtualization)
Alternative to virtual switches (OVS, Linux bridge).
Has to be supported by the physical NICs: https://access.redhat.com/articles/1390483
Protocol specification in 2007.
In OpenStack from 2014 (Juno).
Virtual Switch without (left) and with (right) SR-IOV enabled NIC – source: Glenn K. Lockwood, edited
Comparison to virtual switches:
 performance (still a large gap between native – non-virtualized)

Lukáš Korous
Comparison of virtualized w/o SR-IOV, w/ SR-IOV and native 10G switches using Amazon EC2 – source: Glenn K. Lockwood
Comparison of virtualized w/o SR-IOV, w/ SR-IOV and native 10G switches using Amazon EC2 – source: Glenn K. Lockwood
- features (iptables, migration, HA)
- limited number of Virtual Functions per NIC

Lukáš Korous
Neutron concept and relationship to other OpenStack components
Basic OpenStack services interaction scheme
(source: openstack.org)

Lukáš Korous
Objective - To provide Network-as-a-Service (NaaS)
Sources: openstack.org, Hewlett-Packard Development Company, L.P.
 To allow tenants to control their own private networks
o Ability to create “multi-tier” networks
o Control IP addressing (IP address overlapping)
 Neutron API for operating logical networks
o Separate logical operations and backend provisioning
o Backend technologies are provisioned/configured by Neutron plugins/drivers.
 Neutron provides common abstract network API independent from specific technologies/vendors.
 Abstractions - what abstractions do we need to provide?
o L2 (Switches)
o L3 (Routers)
o L4-7 (Firewalls, Load Balancers)
Basic network model
 Tenant can create multiple (L2) networks
o Subnet defines L3 subnet information (CIDR, gateway etc)
 Multiple subnets can be associated to a network
o E.g., IPv4 subnet + IPv6 subnet. Multiple IPv4 address pools
 Support IP address overlapping among tenants
o Network have multiple ports (similar to physical L2SW ports)
 Virtual NICs of VMs or router interfaces are associated with neutron ports.
Basic Neutron network model

Lukáš Korous
Components
API server - controller
Handles incoming API requests (including dashboard).
Handles DB connections & requests from agents.
Manages the agents.
OpenStack Networking plug-in and agents - nodes
Plugs and unplugs ports, creates networks or subnets, and provides IP addressing. The chosen plug-in and
agents differ depending on the vendor and technologies used in the particular cloud. It is important to
mention that only one plug-in can be used at a time.
Messaging queue – AMQP server & agents
Accepts and routes RPC requests between agents to complete API operations. Message queue is used in the
ML2 plug-in for RPC between the neutron server and neutron agents that run on each hypervisor, in the
ML2 mechanism drivers for Open vSwitch and Linux bridge.

Lukáš Korous
Neutron networks
Tenant networks
Users create tenant networks for connectivity within projects. By default, they are fully isolated and are not
shared with other projects. OpenStack Networking supports the following types of network isolation and
overlay technologies.
Flat
(useless for multi-tenant environments)
All instances reside on the same network, which can also be shared with the hosts. No VLAN tagging or
other network segregation takes place.
VLAN
Described in the section “Network segmentation, encapsulation”. Networking allows users to create
multiple provider or tenant networks using VLAN IDs (802.1Q tagged) that correspond to VLANs present in
the physical network. This allows instances to communicate with each other across the environment. They
can also communicate with dedicated servers, firewalls, load balancers, and other networking infrastructure
on the same layer 2 VLAN (including hosts).
GRE and VXLAN
Described in the section “Network segmentation, encapsulation”. A Networking router is required to allow
traffic to flow outside of the GRE or VXLAN tenant network. A router is also required to connect directly-
connected tenant networks with external networks, including the Internet. The router provides the ability to
connect to instances directly from an external network using floating IP addresses.
Provider networks
Networks that map to existing (physical) provider’s networks. Can be created only by the administrator. Can
be:
 External
 Shared
Subnets
A block of IP addresses and associated configuration state. This is also known as the native IPAM (IP Address
Management) provided by the networking service for both tenant and provider networks. Subnets are used
to allocate IP addresses when new ports are created on a network.
Ports
A port is a connection point for attaching a single device, such as the NIC of a virtual server, to a virtual
network. Also describes the associated network configuration, such as the MAC and IP addresses to be used
on that port.

Lukáš Korous
Routers
This is a logical component that forwards data packets between networks. It also provides L3 and NAT
(SNAT for connections from VMs to outside world, DNAT for floating IP addresses connections to VMs)
forwarding to provide external network access for VMs on tenant networks. Required by certain plug-ins
only.
Security groups
A security group acts as a virtual firewall for your compute instances to control inbound and outbound
traffic. Security groups act at the port level, not the subnet level. Therefore, each port (and therefore each
VM) in a subnet could be assigned to a different set of security groups.
Security groups and security group rules give administrators and tenants the ability to specify the type of
traffic and direction (ingress/egress) that is allowed to pass through a port. A security group is a container
for security group rules. When a port is created, it is associated with a security group.
Extensions
Applications can programmatically list available extensions by performing a GET on the /extensions URI.

Lukáš Korous
Relationship to other components & projects
Projects – Nova
Nova metadata being accessed from neutron and vice-versa.
Nova passing bootfile DHCP parameter to Neutron for PXE boot.
Nova asks neutron for default networks when launching an instance.
# launching an instance
$ nova boot [--flavor <flavor>] [--image <image>]
[--image-with <key=value>]
[--boot-volume <volume_id>]
[--snapshot <snapshot_id>] [--min-count <number>]
[--max-count <number>] [--meta <key=value>]
[--file <dst-path=src-path>]
[--key-name <key-name>]
[--user-data <user-data>]
[--availability-zone <availability-zone>]
[--security-groups <security-groups>]
[--block-device-mapping <dev-name=mapping>]
[--block-device key1=value1[,key2=value2...]]
[--swap <swap_size>]
[--ephemeral size=<size>[,format=<format>]]
[--hint <key=value>]
[--nic <net-id=net-uuid,v4-fixed-ip=ip-addr,v6-
fixed-ip=ip-addr,port-id=port-uuid>]
[--config-drive <value>] [--poll]
[--admin-pass <value>]
<name>
# DHCP
root@compute1: $ ps –A | grep dnsmasq # just to get process id
980 ? 00:00:00 dnsmasq
root@compute1: $ cat /proc/980(dnsmasq-proc-id)/cmdline
dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --
interface=tap6cb4855a-ce --except-interface=lo --pid-
file=/var/lib/neutron/dhcp/e58741d0-60c7-457e-8780-366ecd3200aa/pid
--dhcp-hostsfile=/var/lib/neutron/dhcp/e58741d0-60c7-457e-8780-
366ecd3200aa/host
--addn-hosts=/var/lib/neutron/dhcp/e58741d0-60c7-457e-8780-
366ecd3200aa/addn_hosts --dhcp-
optsfile=/var/lib/neutron/dhcp/e58741d0-60c7-457e-8780-
366ecd3200aa/opts --dhcp-leasefile=/var/lib/neutron/dhcp/e58741d0-
60c7-457e-8780-366ecd3200aa/leases --dhcp-
range=set:tag0,192.168.150.0,static,86400s --dhcp-lease-max=256 --
conf-file=/etc/neutron/dnsmasq-neutron.conf --server=8.8.8.8 --
server=8.8.4.4 --domain=openstacklocal
root@compute1: $ cat /var/lib/neutron/dhcp/e58741d0-60c7-457e-8780-
366ecd3200aa/host

Lukáš Korous
fa:16:3e:03:4e:65,host-192-168-150-
233.openstacklocal,192.168.150.233
fa:16:3e:9f:ee:3d,host-192-168-150-
fa:16:3e:01:53:94,host-192-168-150-
fa:16:3e:b1:d1:e4,host-192-168-150-
DHCP traffic follows the same principles discussed in section Network traffic schemes / VM <-> VM.
Projects – Keystone
Standard authentication mechanisms when using Neutron API.
Components – AMQP (RabbitMQ)
Internal communication among Neutron services performed via RPC.
tcpdump -f "port 5672" -i eth1 # management network
Components – Database
Neutron has its dedicated database (on the controller / dedicated db server).
Neutron architecture

Lukáš Korous

Lukáš Korous
OpenStack commands for Neutron checking
Sources: openstack.org, http://docs.openstack.org/cli-reference/content/neutronclient_commands.html
Common parameters
*-list
 [-c, -F] specify columns to be listed
 [-f] format output (html, json, ...)
 [-D] details
 [--sort-key] sort according to
 [--sort-dir] sort direction
*-create
 [-c] specify columns to be listed
 [-f] format output (html, json, ...)
 [--tenant-id]
*-show
 name or ID
*-update
 name or ID
*-delete
 name or ID
Basic commands
openstack network
(NOT external network)
 list
 create
o name
o [--shared]
o [--provider:network_type]
o [--provider:physical_network]
o [--provider:segmentation_id]
o [--vlan-transparent]
o [--qos-policy]
 delete
 update
 show
external network has to be created using the ‘old way’:

Lukáš Korous
$ neutron net-create ext-net --router:external True
--provider:physical_network external --provider:network_type flat
Created a new network:
+---------------------------+--------------------------------------|
Field | Value |
+---------------------------+--------------------------------------+
| admin_state_up | True |
| id | 893aebb9-1c1e-48be-8908-6b947f3237b3 |
name | ext-net |
provider:network_type | flat |
provider:physical_network | external |
provider:segmentation_id | |
router:external | True |
shared | False |
status | ACTIVE |
subnets | |
tenant_id | 54cd044c64d5408b83f843d63624e0d8 +--
-------------------------+--------------------------------------
$ neutron net-external-list
+--------------------------------------+--------+--------------------------
-----------------------------+
| id | name | subnets
|
+--------------------------------------+--------+--------------------------
-----------------------------+
| e58741d0-60c7-457e-8780-366ecd3200aa | public | 31a30f35-54c9-4767-83bc-
8df76b909cdc 192.168.150.0/24 |
+--------------------------------------+--------+--------------------------
-----------------------------+
subnet
(NOT external network)
 list
 create
o network
o cidr
o [--allocation-pool]
o [--host-route]
o [--gateway / --no-gateway]
o [--dns-nameserver]
o [--enable-dhcp/--disable-dhcp]
o [--subnetpol]
o [--prefixlen]
 delete
 update
o [--allocation-pool]
o [--host-route]
o [--gateway / --no-gateway]
o [--dns-nameserver]
o [--enable-dhcp/--disable-dhcp]

Lukáš Korous
 show
external subnet:
$ neutron subnet-create ext-net --name ext-subnet
--allocation-pool start=FLOATING_IP_START,end=FLOATING_IP_END
--disable-dhcp --gateway EXTERNAL_NETWORK_GATEWAY
EXTERNAL_NETWORK_CIDR
Notes from docs.openstack.org:
Replace FLOATING_IP_START and FLOATING_IP_END with the first and last IP addresses of the
range that you want to allocate for floating IP addresses. Replace EXTERNAL_NETWORK_CIDR with the
subnet associated with the physical network. Replace EXTERNAL_NETWORK_GATEWAY with the
gateway associated with the physical network, typically the ".1" IP address.
 OK
You should disable DHCP on this subnet because instances do not connect directly to the external
network and floating IP addresses require manual assignment.
 This is for consideration.
floatingip
 list
 create
o public network (from which the floating ip is allocated)
o [--port-id]
o [--fixed-ip-address]
o [--floating-ip-address]
o
 delete
 show
port
 list
 create
o network
o [--fixed-ip]
o [--device-id]
o [--device-owner]
o [--mac-address]
o [--security-group/--no-security-groups]
o [--qos-policy]
 delete (!!! does not delete the vm’s interface, or its ip address)
 update
o [--fixed-ip]
o [--device-id]
o [--device-owner]
o [--security-group/--no-security-groups]

Lukáš Korous
o [--qos-policy]
 show
 nova interface-attach –port-id=(name or id of port) name-or-id-of-instance
router
 list
 create
o name
o [--ha]
o [--distributed]
o [--admin-state-down]
 delete
 update
o [--name]
o [--distributed]
o [--admin-state-down]
 show
security-group
 list
 create
o name
o [--description]
 delete
 update
o [--name]
o [--description]
 show
security-group-rule
 list
 create
o security_group
o [--direction]
o [--ethertype]
o [--protocol]
o [--port-range-min / port-range-max]
o [--remote-ip-prefix]
 delete
 show
QoS
Blueprints
https://blueprints.launchpad.net/neutron?searchtext=qos

Lukáš Korous
Kilo
no support
Liberty
first official support (=> (prediction) not practically usable)
neutron qos-policy
policy that is later possible to assign to a vm, or a router
 list
 create
o name
o [--shared]
 delete
 update
o [--name]
o [--shared]
 show
neutron qos-available-rule-types
list available rule types (in Liberty – 7.0.0 – only bandwidth limit rules are available)
neutron qos-bandwith-limit-rule
 create
o [--max-kbps]
o [--max-burst-kbps]
o qos_policy
 show
 update
o [--max-kbps]
o [--max-burst-kbps]
o qos_policy
 delete
 list
neutron queue
QoS queues
 create
o [--min]
o [--max]
o [--qos-marking]
o [--default]
o [--dscp]
o name
 show
 delete
 list

Lukáš Korous
Neutron configuration files
Controller
Log file
/var/log/neutron-server.log
Main configuration file
/etc/neutron/neutron.conf
[DEFAULT]
# Print more verbose output (set logging level to INFO instead of
default WARNING level).
# Handles logging to /var/log/neutron-server.log on the controller
verbose = True
# ===Start Global Config Option for Distributed L3 Router=======
# Setting the "router_distributed" flag to "True" will default to
the creation
# of distributed tenant routers. The admin can override this flag by
specifying
# the type of the router on the create request (admin-only
attribute). Default
# value is "False" to support legacy mode (centralized) routers.
#
# Legacy mode (no DVR)
router_distributed = False
#
# ==End Global Config Option for Distributed L3 Router==========
# Print debugging output (set logging level to DEBUG instead of
# Handles logging to /var/log/neutron-server.log on the controller
# debug = False
# Where to store Neutron state files.
# This directory must be writable by the user executing the agent.
# Temporary files
# state_path = /var/lib/neutron
# log_format = %(asctime)s %(levelname)8s [%(name)s] %(message)s
# log_date_format = %Y-%m-%d %H:%M:%S
# Start - logging to syslog, stderr
# use_syslog -> syslog
# ... (logging to syslog, stderr)
# publish_errors = False
# End - logging to syslog, stderr

Lukáš Korous
# API endpoint specification
# Address to bind the API server to
bind_host = controller1
# Port the bind the API server to
bind_port = 9696
# Path to the API extensions
# ...
# api_extensions_path =
# (StrOpt) Neutron core plugin entrypoint to be loaded from the
# neutron.core_plugins namespace. See setup.cfg for the entrypoint
names of the
# plugins included in the neutron source distribution. For
compatibility with
# previous versions, the class name of a plugin can be specified
instead of its
# entrypoint name.
#
# Example: core_plugin = ml2
# https://github.com/openstack/neutron/blob/master/setup.cfg#L103
core_plugin = neutron.plugins.ml2.plugin.Ml2Plugin
# (ListOpt) List of service plugin entrypoints to be loaded from the
# neutron.service_plugins namespace. See setup.cfg for the
entrypoint names of
# the plugins included in the neutron source distribution. For
compatibility
# with previous versions, the class name of a plugin can be
specified instead
# of its entrypoint name.
#
# Example: service_plugins = router,firewall,lbaas,vpnaas,metering
# https://github.com/openstack/neutron/blob/master/setup.cfg#L108
service_plugins = router,lbaas,vpnaas
# api-paste.ini – API & authentication settings
# Paste configuration file
# api_paste_config = api-paste.ini
# default = hostname
# (StrOpt) Hostname to be used by the neutron server, agents and
services
# running on this machine. All the agents and services running on
this machine
# must use the same host value.
# The default value is hostname of the machine.
#
# host =

Lukáš Korous
# Neutron authentication
# The strategy to be used for auth.
# Supported values are 'keystone'(default), 'noauth'.
auth_strategy = keystone
# Random MAC generation (defaults apply)
# Base MAC address. The first 3 octets will remain unchanged. If the
# 4h octet is not 00, it will also be used. The others will be
# randomly generated.
# 3 octet
# base_mac = fa:16:3e:00:00:00
# 4 octet
# base_mac = fa:16:3e:4f:00:00
# DVR Base MAC address. The first 3 octets will remain unchanged. If
the
# 4th octet is not 00, it will also be used. The others will be
randomly
# generated. The 'dvr_base_mac' *must* be different from 'base_mac'
to
# avoid mixing them up with MAC's allocated for tenant ports.
# A 4 octet example would be dvr_base_mac = fa:16:3f:4f:00:00
# The default is 3 octet
# dvr_base_mac = fa:16:3f:00:00:00
# Maximum amount of retries to generate a unique MAC address
# mac_generation_retries = 16
# DNSMASQ settings
# DHCP Lease duration (in seconds). Use -1 to
# tell dnsmasq to use infinite lease times.
# dhcp_lease_duration = 86400
# Allow sending resource operation notification to DHCP agent
# dhcp_agent_notification = True
# Enable or disable bulk create/update/delete operations
# allow_bulk = True
# Enable or disable pagination
# allow_pagination = False
# Enable or disable sorting
# allow_sorting = False
# Enable or disable overlapping IPs for subnets
# Attention: the following parameter MUST be set to False if Neutron
is
# being used in conjunction with nova security groups
allow_overlapping_ips = True
# Ensure that configured gateway is on subnet. For IPv6, validate
only if
# gateway is not a link local address. Deprecated, to be removed
during the

Lukáš Korous
# K release, at which point the check will be mandatory.
# force_gateway_on_subnet = True
# Default maximum number of items returned in a single response,
# value == infinite and value < 0 means no max limit, and value must
# be greater than 0. If the number of items requested is greater
than
# pagination_max_limit, server will just return pagination_max_limit
# of number of items.
# pagination_max_limit = -1
# Maximum number of DNS nameservers per subnet
# max_dns_nameservers = 5
# Maximum number of host routes per subnet
# max_subnet_host_routes = 20
# Maximum number of fixed ips per port
# max_fixed_ips_per_port = 5
# Maximum number of routes per router
# max_routes = 30
# Default Subnet Pool to be used for IPv4 subnet-allocation.
# Specifies by UUID the pool to be used in case of subnet-create
# being called without a subnet-pool ID. The default of None means
# that no pool will be used unless passed explicitly to subnet
# If no pool is used, then a CIDR must be passed to create a subnet
# and that subnet will not be allocated from any pool; it will be
# considered part of the tenant's private address space.
# default_ipv4_subnet_pool =
# Default Subnet Pool to be used for IPv6 subnet-allocation.
# Specifies by UUID the pool to be used in case of subnet-create
# being called without a subnet-pool ID.
# default_ipv6_subnet_pool =
# New in Kilo, MTU for instances done in a different config file.
# ==== items for MTU selection and advertisement =============
# Advertise MTU. If True, effort is made to advertise MTU
# settings to VMs via network methods (ie. DHCP and RA MTU options)
# when the network's preferred MTU is known.
# advertise_mtu = False
# ==== end of items for MTU selection and advertisement =========
# =========== items for agent management extension =============
# Seconds to regard the agent as down; should be at least twice
# report_interval, to be sure the agent is down for good
# agent_down_time = 75
# =========== end of items for agent management extension =====
# =========== items for agent scheduler extension =============
# Driver to use for scheduling network to DHCP agent

Lukáš Korous
#
https://github.com/openstack/neutron/blob/master/neutron/scheduler/d
hcp_agent_scheduler.py
# network_scheduler_driver =
# neutron.scheduler.dhcp_agent_scheduler.ChanceScheduler
# Driver to use for scheduling router to a default L3 agent
#
https://github.com/openstack/neutron/blob/master/neutron/scheduler/l
3_agent_scheduler.py
# router_scheduler_driver =
neutron.scheduler.l3_agent_scheduler.ChanceScheduler
# Driver to use for scheduling a loadbalancer pool to an lbaas agent
# loadbalancer_pool_scheduler_driver =
neutron.services.loadbalancer.agent_scheduler.ChanceScheduler
# (StrOpt) Representing the resource type whose load is being
# reported by the agent.
# This can be 'networks','subnets' or 'ports'. When specified
# (Default is networks), the server will extract particular load
# sent as part of its agent configuration object from the agent
# report state, which is the number of resources being consumed, at
# every report_interval.
# dhcp_load_type can be used in combination with
# network_scheduler_driver =
# neutron.scheduler.dhcp_agent_scheduler.WeightScheduler
# When the network_scheduler_driver is WeightScheduler,
# dhcp_load_type can be configured to represent the choice for the
# resource being balanced.
# Example: dhcp_load_type = networks
# Values:
# networks - number of networks hosted on the agent
# subnets - number of subnets associated with the networks
# hosted on the agent
# ports - number of ports associated with the networks hosted
# on the agent
# dhcp_load_type = networks
# Allow auto scheduling networks to DHCP agent. It will schedule
# non-hosted networks to first DHCP agent which sends
# get_active_networks message to neutron server
# network_auto_schedule = True
# Allow auto scheduling routers to L3 agent. It will schedule non-
# hosted routers to first L3 agent which sends sync_routers message
# to neutron server
# router_auto_schedule = True
# Allow automatic rescheduling of routers from dead L3 agents with
# admin_state_up set to True to alive agents.
# allow_automatic_l3agent_failover = False
# Allow automatic removal of networks from dead DHCP agents with
# admin_state_up set to True.

Lukáš Korous
# Networks could then be rescheduled if network_auto_schedule is
# True
# allow_automatic_dhcp_failover = True
# Number of DHCP agents scheduled to host a network. This enables
# redundant DHCP agents for configured networks.
# dhcp_agents_per_network = 1
# Enable services on agents with admin_state_up False.
# If this option is False, when admin_state_up of an agent is
# turned to False, services on it will be disabled. If this option
# is True, services on agents with admin_state_up False keep
# available and manual scheduling to such agents is available.
# Agents with admin_state_up False are not selected for automatic
# scheduling regardless of this option.
# enable_services_on_agents_with_admin_state_down = False
# =========== end of items for agent scheduler extension =====
# =========== items for l3 extension ==============
# Enable high availability for virtual routers.
# l3_ha = False
#
# Maximum number of l3 agents which a HA router will be scheduled
# on.If it is set to 0 the router will be scheduled on every agent.
# max_l3_agents_per_router = 3
#
# Minimum number of l3 agents which a HA router will be scheduled
# on. The default value is 2.
# min_l3_agents_per_router = 2
#
# CIDR of the administrative network if HA mode is enabled
# l3_ha_net_cidr = 169.254.192.0/18
# =========== end of items for l3 extension =======
# =========== items for metadata proxy configuration ==============
# User (uid or name) running metadata proxy after its
# initialization (if empty: agent effective user)
# metadata_proxy_user =
# Group (gid or name) running metadata proxy after its
# initialization (if empty: agent effective group)
# metadata_proxy_group =
# Enable/Disable log watch by metadata proxy, it should be disabled
# when metadata_proxy_user/group is not allowed to read/write its
# log file and
# 'copytruncate' logrotate option must be used if logrotate is
# enabled on metadata proxy log files. Option default value is
# deduced from metadata_proxy_user: watch log is enabled if
# metadata_proxy_user is agent effective user id/name.
# metadata_proxy_watch_log =

Lukáš Korous
# Location of Metadata Proxy UNIX domain socket
# metadata_proxy_socket = $state_path/metadata_proxy
# ===== end of items for metadata proxy configuration ============
# ========== items for VLAN trunking networks ==========
# Setting this flag to True will allow plugins that support it to
# create VLAN transparent networks. This flag has no effect for
# plugins that do not support VLAN transparent networks.
# vlan_transparent = False
# ========== end of items for VLAN trunking networks ==========
# == WSGI (Web Server Gateway Interface) API server parameters ==
# Number of separate worker processes to spawn. The default, 0,
# runs the worker thread in the current process. Greater than 0
# launches that number of child processes as workers. The parent
# process manages them.
# api_workers = 0
# Number of separate RPC worker processes to spawn. The default,
# 0, runs the worker thread in the current process. Greater than 0
# launches that number of child processes as RPC workers. The
# parent process manages them. This feature is experimental until
# issues are addressed and testing has been enabled for various
# plugins for compatibility.
# rpc_workers = 0
# Timeout for client connections socket operations. If an
# incoming connection is idle for this number of seconds it
# will be closed. A value of '0' means wait forever. (integer
# value)
# client_socket_timeout = 900
# wsgi keepalive option. Determines if connections are allowed to
# be held open by clients after a request is fulfilled. A value of #
False will ensure that the socket connection will be explicitly
# closed once a response has been sent to the client.
# wsgi_keep_alive = True
# Sets the value of TCP_KEEPIDLE in seconds to use for each server
# socket when starting API server. Not supported on OS X.
# tcp_keepidle = 600
# Number of seconds to keep retrying to listen
# retry_until_window = 30
# Number of backlog requests to configure the socket with.
# backlog = 4096
# Max header line to accommodate large tokens
# max_header_line = 16384
# Enable SSL on the API server
# use_ssl = False

Lukáš Korous
# Certificate file to use when starting API server securely
# ssl_cert_file = /path/to/certfile
# Private key file to use when starting API server securely
# ssl_key_file = /path/to/keyfile
# CA certificate file to use when starting API server securely to
# verify connecting clients. This is an optional parameter only
required if
# API clients need to authenticate to the API server using SSL
# certificates signed by a trusted CA
# ssl_ca_file = /path/to/cafile
# ===== end of WSGI parameters related to the API server =========
# ======== neutron nova interactions ==========
# Send notification to nova when port status is active.
notify_nova_on_port_status_changes = True
# Send notifications to nova when port data (fixed_ips/floatingips)
# change so nova can update it's cache.
notify_nova_on_port_data_changes = True
# URL for connection to nova (Only supports one nova region
currently).
nova_url = http://controller1:8774/v2
# Name of nova region to use. Useful if keystone manages more than
one region
# nova_region_name =
# Username for connection to nova in admin context
# nova_admin_username =
# The uuid of the admin nova tenant
# nova_admin_tenant_id =
# The name of the admin nova tenant. If the uuid of the admin nova
# tenant is set, this is optional. Useful for cases where the uuid
# of the admin nova tenant is not available when configuration is
# being done.
# nova_admin_tenant_name =
# Password for connection to nova in admin context.
# nova_admin_password =
# Authorization URL for connection to nova in admin context.
# nova_admin_auth_url =
# CA file for novaclient to verify server certificates
# nova_ca_certificates_file =
# Boolean to control ignoring SSL errors on the nova url

Lukáš Korous
# nova_api_insecure = False
# Number of seconds between sending events to nova if there are any
events to send
# send_events_interval = 2
# ======== end of neutron nova interactions ==========
# ================= AMQP ======================
# Use durable queues in amqp. (boolean value)
# Deprecated group/name - [DEFAULT]/rabbit_durable_queues
# amqp_durable_queues=false
# Auto-delete queues in amqp. (boolean value)
# amqp_auto_delete=false
# Size of RPC connection pool. (integer value)
# rpc_conn_pool_size=30
# The messaging driver to use, defaults to rabbit. Other
# drivers include qpid and zmq. (string value)
rpc_backend = rabbit
# The default exchange under which topics are scoped. May be
# overridden by an exchange name specified in the
# transport_url option. (string value)
# control_exchange=openstack
[matchmaker_redis]
#
# Options defined in oslo.messaging
#
# Host to locate redis. (string value)
# host=127.0.0.1
# Use this port to connect to redis host. (integer value)
# port=6379
# Password for Redis server (optional). (string value)
# password=
[matchmaker_ring]
#
# Options defined in oslo.messaging
#
# Matchmaker ring file (JSON). (string value)
# Deprecated group/name - [DEFAULT]/matchmaker_ringfile
# ringfile=/etc/oslo/matchmaker_ring.json

Lukáš Korous
# ================= end of AMQP ======================
# ================= quotas ======================
# Default driver to use for quota checks
# quota_driver = neutron.db.quota_db.DbQuotaDriver
# Resource name(s) that are supported in quota features
# quota_items = network,subnet,port
# Default number of resource allowed per tenant. A negative value
means
# unlimited.
# default_quota = -1
# Number of networks allowed per tenant. A negative value means
unlimited.
# quota_network = 10
# Number of subnets allowed per tenant. A negative value means
unlimited.
# quota_subnet = 10
# Number of ports allowed per tenant. A negative value means
unlimited.
# quota_port = 50
# Number of security groups allowed per tenant. A negative value
means
# unlimited.
# quota_security_group = 10
# Number of security group rules allowed per tenant. A negative
value means
# unlimited.
# quota_security_group_rule = 100
# Number of vips allowed per tenant. A negative value means
unlimited.
# quota_vip = 10
# Number of pools allowed per tenant. A negative value means
unlimited.
# quota_pool = 10
# Number of pool members allowed per tenant. A negative value means
unlimited.
# The default is unlimited because a member is not a real resource
consumer
# on Openstack. However, on back-end, a member is a resource
consumer
# and that is the reason why quota is possible.
# quota_member = -1

Lukáš Korous
# Number of health monitors allowed per tenant. A negative value
means
# unlimited.
# The default is unlimited because a health monitor is not a real
resource
# consumer on Openstack. However, on back-end, a member is a
resource consumer
# and that is the reason why quota is possible.
# quota_health_monitor = -1
# Number of loadbalancers allowed per tenant. A negative value means
unlimited.
# quota_loadbalancer = 10
# Number of listeners allowed per tenant. A negative value means
unlimited.
# quota_listener = -1
# Number of v2 health monitors allowed per tenant. A negative value
means
# unlimited. These health monitors exist under the lbaas v2 API
# quota_healthmonitor = -1
# Number of routers allowed per tenant. A negative value means
unlimited.
# quota_router = 10
# Number of floating IPs allowed per tenant. A negative value means
unlimited.
# quota_floatingip = 50
# Number of firewalls allowed per tenant. A negative value means
unlimited.
# quota_firewall = 1
# Number of firewall policies allowed per tenant. A negative value
means
# unlimited.
# quota_firewall_policy = 1
# Number of firewall rules allowed per tenant. A negative value
means
# unlimited.
# quota_firewall_rule = 100
# ================= end of quotas ======================
# ================= agent ======================
# Use "sudo neutron-rootwrap /etc/neutron/rootwrap.conf" to use the
# real root filter facility. Change to "sudo" to skip the filtering
# and just run the command directly
root_helper = sudo neutron-rootwrap /etc/neutron/rootwrap.conf

Lukáš Korous
# Set to true to add comments to generated iptables rules that
# describe each rule's purpose. (System must support the iptables
# comments module.)
# comment_iptables_rules = True
# Root helper daemon application to use when possible.
# root_helper_daemon =
# Use the root helper when listing the namespaces on a system. This
# may not be required depending on the security configuration. If
# the root helper is not required, set this to False for a
# performance improvement.
# use_helper_for_ns_read = True
# The interval to check external processes for failure in seconds
(0=disabled)
# check_child_processes_interval = 60
# Action to take when an external process spawned by an agent dies
# Values:
# respawn - Respawns the external process
# exit - Exits the agent
# check_child_processes_action = respawn
# ================= end of agent ======================
# =========== items for agent management extension =============
# seconds between nodes reporting state to server; should be less
# than agent_down_time, best if it is half or less than
# agent_down_time
# report_interval = 30
# =========== end of items for agent management extension =====
# ============== keystone_authtoken ==========================
auth_uri = http://controller1:5000
auth_url = http://controller1:35357
auth_plugin = password
project_domain_id = default
user_domain_id = default
project_name = service
username = neutron
password = asd08f756ae0ftya
# ============== end of keystone_authtoken ========================
# ====================== database ==========================
# This line MUST be changed to actually run the plugin.
# Example: connection = mysql://root:pass@127.0.0.1:3306/neutron
# Replace 127.0.0.1 above with the IP address of the database used
# by the main neutron server. (Leave it as is if the database runs
# on this host.)
# NOTE: In deployment the [database] section and its connection

Lukáš Korous
# attribute may be set in the corresponding core plugin '.ini'
# file. However, it is suggested to put the [database] section and
# its connection attribute in this configuration file.
connection = mysql://neutron:k9xBUqUbsjf8BY6wRmVz@localhost/neutron
# Database engine for which script will be generated when using
# offline migration
# engine =
# The SQLAlchemy connection string used to connect to the slave
database
# slave_connection =
# Database reconnection retry times - in event connectivity is lost
# set to -1 implies an infinite retry count
# max_retries = 10
# Database reconnection interval in seconds - if the initial
# connection to the database fails
# retry_interval = 10
# Minimum number of SQL connections to keep open in a pool
# min_pool_size = 1
# Maximum number of SQL connections to keep open in a pool
# max_pool_size = 10
# Timeout in seconds before idle sql connections are reaped
# idle_timeout = 3600
# If set, use this value for max_overflow with sqlalchemy
# max_overflow = 20
# Verbosity of SQL debugging information. 0=None, 100=Everything
# connection_debug = 0
# Add python stack traces to SQL as comment strings
# connection_trace = False
# If set, use this value for pool_timeout with sqlalchemy
# pool_timeout = 10
# ====================== end of database ==========================
# =========================== nova ==============================
# Name of the plugin to load
# auth_plugin =
# Config Section from which to load plugin specific options
# auth_section =
# PEM encoded Certificate Authority to use when verifying HTTPs
connections.
# cafile =

Lukáš Korous
# PEM encoded client certificate cert file
# certfile =
# Verify HTTPS connections.
# insecure = False
# PEM encoded client certificate key file
# keyfile =
# Timeout value for http requests
# timeout =
region_name = regionOne
username = nova
# ========================= end of nova ===========================
# ====================== oslo_concurrency =========================
# Directory to use for lock files. For security, the specified
# directory should only be writable by the user running the
# processes that need locking. Defaults to environment variable
# OSLO_LOCK_PATH. If external locks are used, a lock path must be
# set.
lock_path = $state_path/lock
# Enables or disables inter-process locks.
# disable_process_locking = False
# ========================= oslo_policy ==========================
# The JSON file that defines policies.
# policy_file = policy.json
# Default rule. Enforced when a requested rule is not found.
# policy_default_rule = default
# Directories where policy configuration files are stored.
# They can be relative to any directory in the search path defined
# by the config_dir option, or absolute paths. The file defined by
# policy_file must exist for these directories to be searched.
# Missing or empty directories are ignored.
# policy_dirs = policy.d
# ===================== oslo_messaging_amqp =======================
# Address prefix used when sending to a specific server (string
value)
# Deprecated group/name - [amqp1]/server_request_prefix

Lukáš Korous
# server_request_prefix = exclusive
# Address prefix used when broadcasting to all servers (string
value)
# Deprecated group/name - [amqp1]/broadcast_prefix
# broadcast_prefix = broadcast
# Address prefix when sending to any server in group (string value)
# Deprecated group/name - [amqp1]/group_request_prefix
# group_request_prefix = unicast
# Name for the AMQP container (string value)
# Deprecated group/name - [amqp1]/container_name
# container_name =
# Timeout for inactive connections (in seconds) (integer value)
# Deprecated group/name - [amqp1]/idle_timeout
# idle_timeout = 0
# Debug: dump AMQP frames to stdout (boolean value)
# Deprecated group/name - [amqp1]/trace
# trace = false
# CA certificate PEM file for verifing server certificate (string
value)
# Deprecated group/name - [amqp1]/ssl_ca_file
# ssl_ca_file =
# Identifying certificate PEM file to present to clients (string
value)
# Deprecated group/name - [amqp1]/ssl_cert_file
# ssl_cert_file =
# Private key PEM file used to sign cert_file certificate (string
value)
# Deprecated group/name - [amqp1]/ssl_key_file
# ssl_key_file =
# Password for decrypting ssl_key_file (if encrypted) (string value)
# Deprecated group/name - [amqp1]/ssl_key_password
# ssl_key_password =
# Accept clients using either SSL or plain TCP (boolean value)
# Deprecated group/name - [amqp1]/allow_insecure_clients
# allow_insecure_clients = false
# ==================== oslo_messaging_rabbit ======================
# Use durable queues in AMQP. (boolean value)
# Deprecated group/name - [DEFAULT]/rabbit_durable_queues
# amqp_durable_queues=False
# Auto-delete queues in AMQP. (boolean value)
# Deprecated group/name - [DEFAULT]/amqp_auto_delete
# amqp_auto_delete = false

Lukáš Korous
# Size of RPC connection pool. (integer value)
# Deprecated group/name - [DEFAULT]/rpc_conn_pool_size
# rpc_conn_pool_size = 30
# SSL version to use (valid only if SSL enabled). Valid values are
TLSv1 and
# SSLv23. SSLv2, SSLv3, TLSv1_1, and TLSv1_2 may be available on
some
# distributions. (string value)
# Deprecated group/name - [DEFAULT]/kombu_ssl_version
# kombu_ssl_version =
# SSL key file (valid only if SSL enabled). (string value)
# Deprecated group/name - [DEFAULT]/kombu_ssl_keyfile
# kombu_ssl_keyfile =
# SSL cert file (valid only if SSL enabled). (string value)
# Deprecated group/name - [DEFAULT]/kombu_ssl_certfile
# kombu_ssl_certfile =
# SSL certification authority file (valid only if SSL enabled).
(string value)
# Deprecated group/name - [DEFAULT]/kombu_ssl_ca_certs
# kombu_ssl_ca_certs =
# How long to wait before reconnecting in response to an AMQP
consumer cancel
# notification. (floating point value)
# Deprecated group/name - [DEFAULT]/kombu_reconnect_delay
# kombu_reconnect_delay = 1.0
# The RabbitMQ broker address where a single node is used. (string
value)
# Deprecated group/name - [DEFAULT]/rabbit_host
rabbit_host = rabbitmq1
# The RabbitMQ broker port where a single node is used. (integer
value)
# Deprecated group/name - [DEFAULT]/rabbit_port
rabbit_port = 5672
# RabbitMQ HA cluster host:port pairs. (list value)
# Deprecated group/name - [DEFAULT]/rabbit_hosts
# rabbit_hosts = $rabbit_host:$rabbit_port
# Connect over SSL for RabbitMQ. (boolean value)
# Deprecated group/name - [DEFAULT]/rabbit_use_ssl
rabbit_use_ssl = False
# The RabbitMQ userid. (string value)
# Deprecated group/name - [DEFAULT]/rabbit_userid
rabbit_userid = guest

Lukáš Korous
# The RabbitMQ password. (string value)
# Deprecated group/name - [DEFAULT]/rabbit_password
rabbit_password = ousthf67thb9R8876RBASD
# The RabbitMQ login method. (string value)
# Deprecated group/name - [DEFAULT]/rabbit_login_method
# rabbit_login_method = AMQPLAIN
# The RabbitMQ virtual host. (string value)
# Deprecated group/name - [DEFAULT]/rabbit_virtual_host
rabbit_virtual_host = /
# How frequently to retry connecting with RabbitMQ. (integer value)
# rabbit_retry_interval=1
# How long to backoff for between retries when connecting to
RabbitMQ. (integer
# value)
# Deprecated group/name - [DEFAULT]/rabbit_retry_backoff
# rabbit_retry_backoff=2
# Maximum number of RabbitMQ connection retries. Default is 0
(infinite retry
# count). (integer value)
# Deprecated group/name - [DEFAULT]/rabbit_max_retries
# rabbit_max_retries=0
# Use HA queues in RabbitMQ (x-ha-policy: all). If you change this
# option, you must wipe the RabbitMQ database. (boolean value)
# Deprecated group/name - [DEFAULT]/rabbit_ha_queues
# rabbit_ha_queues=False
# Deprecated, use rpc_backend=kombu+memory or rpc_backend=fake
(boolean value)
# Deprecated group/name - [DEFAULT]/fake_rabbit
fake_rabbit = False
# ===================== service providers ====================
# Specify service providers (drivers) for advanced services like
# loadbalancer, VPN, Firewall. Must be in form:
# service_provider=<service_type>:<name>:<driver>[:default]
# List of allowed service types includes LOADBALANCER, FIREWALL, VPN
# Combination of <service type> and <name> must be unique; <driver>
must also be unique
# This is multiline option, example for default provider:
#
service_provider=LOADBALANCER:name:lbaas_plugin_driver_path:default
# example of non-default provider:
# service_provider=FIREWALL:name2:firewall_driver_path
service_provider =
LOADBALANCER:Haproxy:neutron_lbaas.services.loadbalancer.drivers.hap
roxy.plugin_driver.HaproxyOnHostPluginDriver:default

Lukáš Korous
service_provider =
VPN:openswan:neutron_vpnaas.services.vpn.service_drivers.ipsec.IPsec
VPNDriver:default

Lukáš Korous
Nova configuration file
/etc/nova/nova.conf
# Control for checking for default networks (string value)
#use_neutron_default_nets=False
# Default tenant id when creating neutron networks (string
# value)
#neutron_default_tenant_id=default
# Number of private networks allowed per project (integer
# value)
#quota_networks=3
# The full class name of the network API class to use (string
# value)
network_api_class = nova.network.neutronv2.api.API
# === Options defined in nova.network.driver ===
# Driver to use for network creation (string value)
#network_driver=nova.network.linux_net
# === Options defined in nova.network.floating_ips ===
# Default pool for floating IPs (string value)
default_floating_pool = public
# Autoassigning floating IP to VM (boolean value)
#auto_assign_floating_ip=false
====== Options defined in nova.network.neutronv2.api ======
# URL for connecting to neutron (string value)
url = http://controller1:9696
# User id for connecting to neutron in admin context.
# DEPRECATED: specify an auth_plugin and appropriate
# credentials instead. (string value)
#admin_user_id=<None>
# Username for connecting to neutron in admin context
admin_username = neutron
# Password for connecting to neutron in admin context
admin_password = asd08f756ae0ftya
# Tenant id for connecting to neutron in admin context
#admin_tenant_id=<None>

Lukáš Korous
# Tenant name for connecting to neutron in admin context. This
# option will be ignored if neutron_admin_tenant_id is set.
# Note that with Keystone V3 tenant names are only unique
# within a domain. DEPRECATED: specify an auth_plugin and
# appropriate credentials instead. (string value)
admin_tenant_name = service
# Region name for connecting to neutron in admin context
# (string value)
#region_name=<None>
# Authorization URL for connecting to neutron in admin
# context. DEPRECATED: specify an auth_plugin and appropriate
admin_auth_url = http://controller1:35357/v2.0
# Authorization strategy for connecting to neutron in admin
# context. DEPRECATED: specify an auth_plugin and appropriate
# credentials instead. If an auth_plugin is specified strategy
# will be ignored. (string value)
auth_strategy = keystone
# Name of Integration Bridge used by Open vSwitch (string
# value)
#ovs_bridge=br-int
# Number of seconds before querying neutron for extensions
# (integer value)
#extension_sync_interval=600

Lukáš Korous
ML2 plugin configuration file
/etc/neutron/plugins/ml2_conf.ini
# Drivers to be subsequently used in tenant_network_types
# (ListOpt) List of network type driver entrypoints to be loaded
# from the neutron.ml2.type_drivers namespace.
# Example: type_drivers = local,flat,vlan,gre,vxlan
type_drivers = flat,vxlan
# (ListOpt) Ordered list of network_types to allocate as tenant
# networks. The default value 'local' is useful for single-box
# testing but provides no connectivity between hosts.
#
# Example: tenant_network_types = vlan,gre,vxlan
tenant_network_types = vxlan
# (ListOpt) Ordered list of networking mechanism driver entrypoints
# to be loaded from the neutron.ml2.mechanism_drivers namespace.
# Example: mechanism_drivers = openvswitch,mlnx
# Example: mechanism_drivers = arista
# Example: mechanism_drivers = cisco,logger
# Example: mechanism_drivers = openvswitch,brocade
# Example: mechanism_drivers = linuxbridge,brocade
mechanism_drivers = openvswitch,l2population
# (ListOpt) Ordered list of extension driver entrypoints
# to be loaded from the neutron.ml2.extension_drivers namespace.
# extension_drivers =
# Example: extension_drivers = anewextensiondriver
# =========== items for MTU selection and advertisement =========
# MTU is possible to be set in different places, here is one
# option, in DHCP response is another
# (IntOpt) Path MTU. The maximum permissible size of an
# unfragmented packet travelling from and to addresses where
# encapsulated Neutron traffic is sent. Drivers calculate
# maximum viable MTU for validating tenant requests based on this
# value (typically, path_mtu - max encap header size). If <=0,
# the path MTU is indeterminate and no calculation takes place.
# path_mtu = 0
# (IntOpt) Segment MTU. The maximum permissible size of an
# unfragmented packet travelling a L2 network segment. If <=0,
# the segment MTU is indeterminate and no calculation takes place.
# segment_mtu = 0
# (ListOpt) Physical network MTUs. List of mappings of physical
# network to MTU value. The format of the mapping is
# <physnet>:<mtu val>. This mapping allows specifying a
# physical network MTU value that differs from the default
# segment_mtu value.
# physical_network_mtus =
# Example: physical_network_mtus = physnet1:1550, physnet2:1500

Lukáš Korous
# ====================== flat networks ======================
# (ListOpt) List of physical_network names with which flat
# networks
# can be created. Use * to allow flat networks with arbitrary
# physical_network names.
#
# Example:flat_networks = physnet1,physnet2
# Example:flat_networks = *
flat_networks = external
# ====================== vlan networks ======================
# (ListOpt) List of <physical_network>[:<vlan_min>:<vlan_max>]
tuples
# specifying physical_network names usable for VLAN provider and
# tenant networks, as well as ranges of VLAN tags on each
# physical_network available for allocation as tenant networks.
#
# network_vlan_ranges =
# Example: network_vlan_ranges = physnet1:1000:2999,physnet2
# ====================== gre networks ======================
# (ListOpt) Comma-separated list of <tun_min>:<tun_max> tuples
enumerating ranges of GRE tunnel IDs that are available for tenant
network allocation
# tunnel_id_ranges =
# ====================== vxlan networks ======================
# (ListOpt) Comma-separated list of <vni_min>:<vni_max> tuples
enumerating
# ranges of VXLAN VNI IDs that are available for tenant network
allocation.
#
vni_ranges = 65537:69999
# (StrOpt) Multicast group for the VXLAN interface. When configured,
will
# enable sending all broadcast traffic to this multicast group. When
left
# unconfigured, will disable multicast VXLAN mode.
#
# vxlan_group =
# Example: vxlan_group = 239.1.1.1
# ====================== security groups ======================
# Controls if neutron security group is enabled or not.
# It should be false when you use nova security group.
enable_security_group = True
# Use ipset to speed-up the iptables security groups. Enabling ipset
support
# requires that ipset is installed on L2 agent node.

Lukáš Korous
enable_ipset = True

Lukáš Korous
OVS plugin configuration file
/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini
[ovs]
# (BoolOpt) Set to True in the server and the agents to enable
support
# for GRE or VXLAN networks. Requires kernel support for OVS patch
ports and
# GRE or VXLAN tunneling.
enable_tunneling = True
# Do not change this parameter unless you have a good reason to.
# This is the name of the OVS integration bridge. There is one per
hypervisor.
# The integration bridge acts as a virtual "patch bay". All VM VIFs
are
# attached to this bridge and then "patched" according to their
network
# connectivity.
#
integration_bridge = br-int
# Do not change this parameter unless you have a good reason to.
# Only used for the agent if tunnel_id_ranges is not empty for
# the server. In most cases, the default value should be fine.
#
# tunnel_bridge = br-tun
# Peer patch port in integration bridge for tunnel bridge
# int_peer_patch_port = patch-tun
# Peer patch port in tunnel bridge for integration bridge
# tun_peer_patch_port = patch-int
# Uncomment this line for the agent if tunnel_id_ranges is not
# empty for the server. Set local-ip to be the local IP address
# of this hypervisor.
#
# local_ip =
# (ListOpt) Comma-separated list of <physical_network>:<bridge>
tuples
# mapping physical network names to the agent's node-specific OVS
# bridge names to be used for flat and VLAN networks. The length of
# bridge names should be no more than 11. Each bridge must
# exist, and should have a physical network interface configured as
a
# port. All physical networks configured on the server should have
# mappings to appropriate bridges on each agent.
#
# Example: bridge_mappings = physnet1:br-eth1
bridge_mappings = external:br-ex

Lukáš Korous
# (BoolOpt) Use veths instead of patch ports to interconnect the
integration
# bridge to physical networks. Support kernel without ovs patch port
support
# so long as it is set to True.
# use_veth_interconnection = False
# (StrOpt) Which OVSDB backend to use, defaults to 'vsctl'
# vsctl - The backend based on executing ovs-vsctl
# native - The backend based on using native OVSDB
# ovsdb_interface = vsctl
# (StrOpt) The connection string for the native OVSDB backend
# To enable ovsdb-server to listen on port 6640:
# ovs-vsctl set-manager ptcp:6640:127.0.0.1
# ovsdb_connection = tcp:127.0.0.1:6640
[agent]
# Agent's polling interval in seconds
polling_interval = 15
# Minimize polling by monitoring ovsdb for interface changes
# minimize_polling = True
# When minimize_polling = True, the number of seconds to wait before
# respawning the ovsdb monitor after losing communication with it
# ovsdb_monitor_respawn_interval = 30
# (ListOpt) The types of tenant network tunnels supported by the
agent.
# Setting this will enable tunneling support in the agent. This can
be set to
# either 'gre' or 'vxlan'. If this is unset, it will default to []
and
# disable tunneling support in the agent.
# You can specify as many values here as your compute hosts
supports.
#
# Example: tunnel_types = gre
# Example: tunnel_types = vxlan
# Example: tunnel_types = vxlan, gre
tunnel_types = vxlan
# (IntOpt) The port number to utilize if tunnel_types includes
'vxlan'. By
# default, this will make use of the Open vSwitch default value of
'4789' if
# not specified.
#
# vxlan_udp_port =
# Example: vxlan_udp_port = 8472
# (IntOpt) This is the MTU size of veth interfaces.

Lukáš Korous
# Do not change unless you have a good reason to.
# The default MTU size of veth interfaces is 1500.
# This option has no effect if use_veth_interconnection is False
# veth_mtu =
# Example: veth_mtu = 1504
# L2 population – speed up of the ARP requests
# (BoolOpt) Flag to enable l2-population extension. This option
# should only be used in conjunction with ml2 plugin and
# l2population mechanism driver. It'll enable plugin to populate
# remote ports macs and IPs (using fdb_add/remove RPC calbbacks
# instead of tunnel_sync/update) on OVS agents in order to
# optimize tunnel management.
#
l2_population = True
# Enable local ARP responder. Requires OVS 2.1. This is only used
# by the l2 population ML2 MechanismDriver.
#
arp_responder = False
# Enable suppression of ARP responses that don't match an IP address
that
# belongs to the port from which they originate.
# Note: This prevents the VMs attached to this agent from spoofing,
# it doesn't protect them from other devices which have the
capability to spoof
# (e.g. bare metal or VMs attached to agents without this flag set
to True).
# Requires a version of OVS that can match ARP headers.
#
# prevent_arp_spoofing = False
# (BoolOpt) Set or un-set the don't fragment (DF) bit on outgoing IP
packet
# carrying GRE/VXLAN tunnel. The default value is True.
#
# dont_fragment = True
# (BoolOpt) Set to True on L2 agents to enable support
# for distributed virtual routing.
#
enable_distributed_routing = False
# (IntOpt) Set new timeout in seconds for new rpc calls after agent
receives
# SIGTERM. If value is set to 0, rpc timeout won't be changed"
#
# quitting_rpc_timeout = 10
[securitygroup]
# Firewall driver for realizing neutron security group function.

Lukáš Korous
# Example: firewall_driver =
neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDrive
r
firewall_driver =
r
# Controls if neutron security group is enabled or not.
# It should be false when you use nova security group.
# enable_security_group = True
dnsmasq config file – MTU settings
dhcp-option-force=26,1380

Lukáš Korous
Nodes
L3 agent configuration file
/etc/neutron/l3_agent.ini
[DEFAULT]
verbose = True
# Show debugging output in log (sets DEBUG log level output)
# debug = False
# L3 requires that an interface driver be set. Choose the one that
best
# matches your plugin.
# Example of interface_driver option for OVS based plugins (OVS,
# Ryu, NEC) that supports L3 agent
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
# Use veth for an OVS interface or not.
# Support kernels with limited namespace support
# (e.g. RHEL 6.5) so long as ovs_use_veth is set to True.
# ovs_use_veth = False
# Example of interface_driver option for LinuxBridge
# interface_driver =
neutron.agent.linux.interface.BridgeInterfaceDriver
# Allow overlapping IP (Must have kernel build with
# CONFIG_NET_NS=y and iproute2 package that supports namespaces).
# This option is deprecated and will be removed in a future
# release, at which point the old behavior
# of use_namespaces = True will be enforced.
use_namespaces = True
# If use_namespaces is set as False then the agent can only
configure one router.
# This is done by setting the specific router_id.
# router_id =
# When external_network_bridge is set, each L3 agent can be
# associated with no more than one external network. This value
# should be set to the UUID of that external network. To allow L3
# agent support multiple external networks, both the
# external_network_bridge and gateway_external_network_id must be
# left empty.
# gateway_external_network_id =
# With IPv6, the network used for the external gateway does not
# need to have an associated subnet, since the automatically

Lukáš Korous
# assigned link-local address (LLA) can be used. However, an IPv6
# gateway address is needed for use as the next-hop for the
# default route. If no IPv6 gateway address is configured here,
# (and only then) the neutron router will be configured to get
# its default route from router advertisements (RAs) from the
# upstream router; in which case the upstream router must also be
# configured to send these RAs.
# The ipv6_gateway, when configured, should be the LLA of the
# interface on the upstream router. If a next-hop using a global
# unique address (GUA) is desired, it needs to be done via a
# subnet allocated to the network and not through this parameter.
# ipv6_gateway =
# Indicates that this L3 agent should also handle routers that do
# not have an external network gateway configured. This option
# should be True only for a single agent in a Neutron deployment,
# and may be False for all agents if all routers must have an
#external network gateway
handle_internal_only_routers = True
# Name of bridge used for external network traffic. This should
# be set to empty value for the linux bridge. when this parameter
# is set, each L3 agent can be associated with no more than one
# external network.
external_network_bridge = br-ex
# TCP Port used by Neutron metadata server
# metadata_port = 9697
# Send this many gratuitous ARPs for HA setup. Set it below or equal
to 0
# to disable this feature.
# send_arp_for_ha = 3
# seconds between re-sync routers' data if needed
# periodic_interval = 40
# seconds to start to sync routers' data after
# starting agent
# periodic_fuzzy_delay = 5
# enable_metadata_proxy, which is true by default, can be set to
False
# if the Nova metadata server is not available
enable_metadata_proxy = True
# Iptables mangle mark used to mark metadata valid requests
# metadata_access_mark = 0x1
# Iptables mangle mark used to mark ingress from external network
# external_ingress_mark = 0x2

Lukáš Korous
# router_delete_namespaces, which is false by default, can be set to
True if
# namespaces can be deleted cleanly on the host running the L3
agent.
# Do not enable this until you understand the problem with the Linux
iproute
# utility mentioned in
https://bugs.launchpad.net/neutron/+bug/1052535 and
# you are sure that your version of iproute does not suffer from the
problem.
# If True, namespaces will be deleted when a router is destroyed.
router_delete_namespaces = False
# Timeout for ovs-vsctl commands.
# If the timeout expires, ovs commands will fail with ALARMCLOCK
error.
# ovs_vsctl_timeout = 10
# The working mode for the agent. Allowed values are:
# - legacy: this preserves the existing behavior where the L3 agent
is
# deployed on a centralized networking node to provide L3 services
# like DNAT, and SNAT. Use this mode if you do not want to adopt
DVR.
# - dvr: this mode enables DVR functionality, and must be used for
an L3
# agent that runs on a compute host.
# - dvr_snat: this enables centralized SNAT support in conjunction
with
# DVR. This mode must be used for an L3 agent running on a
centralized
# node (or in single-host deployments, e.g. devstack).
agent_mode = legacy
# Location to store keepalived and all HA configurations
# ha_confs_path = $state_path/ha_confs
# VRRP authentication type AH/PASS
# ha_vrrp_auth_type = PASS
# VRRP authentication password
# ha_vrrp_auth_password =
# The advertisement interval in seconds
# ha_vrrp_advert_int = 2
allow_automatic_l3agent_failover = False

Lukáš Korous
DHCP agent configuration file
/etc/neutron/dhcp_agent.ini
[DEFAULT]
verbose = True
# debug = False
# The DHCP agent will resync its state with Neutron to recover from
any
# transient notification or rpc errors. The interval is number of
# seconds between attempts.
# resync_interval = 5
# The DHCP agent requires an interface driver be set. Choose the one
that best
# matches your plugin.
# Example of interface_driver option for OVS based plugins(OVS, Ryu,
NEC, NVP,
# BigSwitch/Floodlight)
interface_driver = neutron.agent.linux.interface.OVSInterfaceDriver
# Name of Open vSwitch bridge to use
ovs_integration_bridge = br-int
# Use veth for an OVS interface or not.
# Support kernels with limited namespace support
# (e.g. RHEL 6.5) so long as ovs_use_veth is set to True.
# ovs_use_veth = False
# Example of interface_driver option for LinuxBridge
# interface_driver =
neutron.agent.linux.interface.BridgeInterfaceDriver
# The agent can use other DHCP drivers. Dnsmasq is the simplest and
requires
# no additional setup of the DHCP server.
dhcp_driver = neutron.agent.linux.dhcp.Dnsmasq
# Allow overlapping IP (Must have kernel build with CONFIG_NET_NS=y
and
# iproute2 package that supports namespaces). This option is
# deprecated and will be removed in a future release, at which
# point the old behavior of use_namespaces = True will be
# enforced.
use_namespaces = True
# The DHCP server can assist with providing metadata support on
# isolated networks. Setting this value to True will cause the
# DHCP server to append specific host routes to the DHCP request.

Lukáš Korous
# The metadata service will only be activated when the subnet
# does not contain any router port. The guest instance must be
# configured to request host routes via DHCP (Option 121).
enable_isolated_metadata = True
# Allows for serving metadata requests coming from a dedicated
# metadata access network whose cidr is 169.254.169.254/16 (or
# larger prefix), and is connected to a Neutron router from which
# the VMs send metadata request. In this case DHCP Option 121
# will not be injected in VMs, as they will be able to reach
# 169.254.169.254 through a router.
# This option requires enable_isolated_metadata = True
enable_metadata_network = False
# Number of threads to use during sync process. Should not exceed
# connection pool size configured on server.
# num_sync_threads = 4
# Location to store DHCP server config files
# dhcp_confs = $state_path/dhcp
# Domain to use for building the hostnames
# dhcp_domain = openstacklocal
# Override the default dnsmasq settings with this file
dnsmasq_config_file = /etc/neutron/dnsmasq-neutron.conf
# Comma-separated list of DNS servers which will be used by
# dnsmasq as forwarders.
dnsmasq_dns_servers = 8.8.8.8,8.8.4.4
# Limit number of leases to prevent a denial-of-service.
# dnsmasq_lease_max = 16777216
# Location to DHCP lease relay UNIX domain socket
# dhcp_lease_relay_socket = $state_path/dhcp/lease_relay
# Use broadcast in DHCP replies
# dhcp_broadcast_reply = False
# dhcp_delete_namespaces, which is false by default, can be set
# to True if namespaces can be deleted cleanly on the host
# running the dhcp agent.
# Do not enable this until you understand the problem with the
# Linux iproute utility mentioned in
# https://bugs.launchpad.net/neutron/+bug/1052535 and
# you are sure that your version of iproute does not suffer from
# the problem.
# If True, namespaces will be deleted when a dhcp server is
disabled.
dhcp_delete_namespaces = False
# Timeout for ovs-vsctl commands.

Lukáš Korous
# If the timeout expires, ovs commands will fail with ALARMCLOCK
error.
# ovs_vsctl_timeout = 10

Lukáš Korous
Metadata agent configuration file
/etc/neutron/metadata_agent.ini
[DEFAULT]
verbose = True
# debug = True
# The Neutron user information for accessing the Neutron API.
auth_uri = http://controller1:5000
auth_region = regionOne
username = neutron
# Turn off verification of the certificate for ssl
# auth_insecure = False
# Certificate Authority public key (CA cert) file for ssl
# auth_ca_cert =
# Network service endpoint type to pull from the keystone catalog
# endpoint_type = adminURL
# IP address used by Nova metadata server
nova_metadata_ip = controller1
# TCP Port used by Nova metadata server
nova_metadata_port = 8775
# Which protocol to use for requests to Nova metadata server, http
or https
nova_metadata_protocol = http
# Whether insecure SSL connection should be accepted for Nova
metadata server
# requests
# nova_metadata_insecure = False
# Client certificate for nova api, needed when nova api requires
client
# certificates
# nova_client_cert =
# Private key for nova client certificate
# nova_client_priv_key =

Lukáš Korous
# When proxying metadata requests, Neutron signs the Instance-ID
header with a
# shared secret to prevent spoofing. You may select any string for
a secret,
# but it must match here and in the configuration used by the Nova
Metadata
# Server. NOTE: Nova uses the same config key, but in [neutron]
section.
metadata_proxy_shared_secret = spCRNxYrm4sLvfJb
# Location of Metadata Proxy UNIX domain socket
# metadata_proxy_socket = $state_path/metadata_proxy
# Metadata Proxy UNIX domain socket mode, 3 values allowed:
# 'deduce': deduce mode from metadata_proxy_user/group values,
# 'user': set metadata proxy socket mode to 0o644, to use when
# metadata_proxy_user is agent effective user or root,
# 'group': set metadata proxy socket mode to 0o664, to use when
# metadata_proxy_group is agent effective group,
# 'all': set metadata proxy socket mode to 0o666, to use otherwise.
# metadata_proxy_socket_mode = deduce
# Number of separate worker processes for metadata server. Defaults
to
# half the number of CPU cores
# metadata_workers =
# Number of backlog requests to configure the metadata server socket
with
metadata_backlog = 4096
# URL to connect to the cache backend.
# default_ttl=0 parameter will cause cache entries to never expire.
# Otherwise default_ttl specifies time in seconds a cache entry is
valid for.
# No cache is used in case no value is passed.
# cache_url = memory://?default_ttl=5

Lukáš Korous
ML2 plugin configuration file
/etc/neutron/plugins/ml2_conf.ini
### Settings identical to the controller equivalent, with the
### following additions:
firewall_driver =
r
[ovs]
local_ip = 10.0.1.11
enable_tunneling = True
[agent]
tunnel_types = vxlan
OVS plugin configuration file
/etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini
### Identical to the controller equivalent

Lukáš Korous
OpenStack network architecture
Sources: openstack.org
‘Global’ network architecture
OpenStack ’global’ network architecture – source:openstack.org
 API network – not connected to Nodes (not interesting for our topic)
 Management network – not part of the OpenStack itself, not available to VMs (not interesting for our
topic)
External / public flat network – L3 routing, connection to the internet
Guest network – using any segmentation (VLAN, GRE, VXLAN)
In the following, only Guest (VLAN or Tunnel) and External networks are described.

Lukáš Korous
‘Operational’ network architecture
OpenStack network architecture – source:openstack.org
 Instances = VMs
 Firewall = iptables
 Switch = open vSwitch
 Router SNAT / DNAT = Neutron router

Lukáš Korous
Network Node
________________________________________________________________________________________

Lukáš Korous
Compute Node
________________________________________________________________________________________

Lukáš Korous
Network and Compute Node combined into a single machine – source: openstack.org, edited

Lukáš Korous
External network
 Create
 Manage (update) – what can admin do in dashboard & in console
 Allocation – routing on the host level
Guest network
Handling described in section OpenStack commands for Neutron checking.

Lukáš Korous
Network traffic schemes
VM <-> VM
Setup
 Project network ‘private1’
o Network e.g. 192.168.1.0/24
o Gateway PRIVATE_1_GATEWAY
 e.g. 192.168.1.1 with MAC address PRIVATE_1_GATEWAY_MAC
o Network e.g. 192.168.2.0/24
o Gateway PRIVATE_2_GATEWAY
 e.g. 192.168.2.1 with MAC address PRIVATE_2_GATEWAY_MAC
 Compute node ‘compute1’
o Instance ‘vm1’
 private address: VM_1_PRIVATE_IP (private1 network)
 e.g. 192.168.1.3
 MAC address: VM_1_MAC
 private address: VM_2_PRIVATE_IP (private2 network)
 e.g. 192.168.2.3
 MAC address: VM_2_MAC
 Router ‘router1’
o interface address on private1 network: ROUTER_PRIVATE_1_IP (==PRIVATE_1_GATEWAY)
o interface address on private2 network: ROUTER_PRIVATE_2_IP (==PRIVATE_2_GATEWAY)
 Network node ‘network1’

Lukáš Korous
Use case
The machine vm1 sends a packet to vm2 using its private address VM_2_PRIVATE_IP.
Notes:
 This setup only describes what is important for the use case, obviously, some external connectivity to
machines has to be available and this is taken care of by both of them having another interface
configured which is connected to a router connected to the external network.
 When created, instances might happen to be created on the same compute node. If they are on the
same one, no tunneling is done.
o the tunnel is not even created – this is done dynamically upon being needed.
 If the instances are on the same tenant network, there is no routing involved, but still they are
generally on different nodes, so tunneling is involved.
 Even if instances are on the same network and on the same compute node, dhcp requests for the
particular network can be handled by a dhcp-agent from a different node.
 In the following, deployment with a dedicated network node is presented. The deployment with
compute node and network node bundled together do not differ from this in the case routing (L3
routing) is involved, because the two instances and the router can truly reside on three different
nodes and only the l3-agent service is used on the network1 node.
compute1
The vm1 tap interface (1) forwards the packet to the Linux bridge qbr.

Lukáš Korous
# vm1 tap interface
$ ifconfig -a | grep -A5 tap23bc0b1f-a1 # 23bc... is port id
tap23bc0b1f-a1 Link encap:Ethernet HWaddr fe:16:3e:0d:ea:b0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:4551 errors:0 dropped:0 overruns:0 frame:0
TX packets:29688 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:418474 (408.6 KiB) TX bytes:37616758 (35.8 MiB)
# linux bridge qbr in interface list
$ ifconfig -a | grep -A5 qbr23bc0b1f-a1 # 23bc... is port id
qbr23bc0b1f-a1 Link encap:Ethernet HWaddr 5a:00:22:a4:95:e1
RX bytes:3360 (3.2 KiB) TX bytes:0 (0.0 B)
# the bridge shown in brctl show output
$ brctl show | grep qbr23bc0b1f-a1
bridge name|bridge id|STP enabled|interfaces
qbr23bc0b1f-a1|8000.5a0022a495e1|no|qvb23bc0b1f-a1 tap23bc0b1f-a1
# qvb is the ‘Linux Bridge part’ of the Linux Bridge <-> OVS veth
pair qvb <-> qvo:
$ ifconfig -a | grep -A5 qv[bo]23bc0b1f-a1
qvb23bc0b1f-a1 Link encap:Ethernet HWaddr 5a:00:22:a4:95:e1
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX bytes:37621347 (35.8 MiB) TX bytes:421205 (411.3 KiB)
--
qvo23bc0b1f-a1 Link encap:Ethernet HWaddr 6e:e4:7d:0c:6b:a3
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX bytes:421205 (411.3 KiB) TX bytes:37621347 (35.8 MiB)
The packet contains destination MAC address PRIVATE_1_GATEWAY_MAC because the destination resides
on another network.
Security group rules (2) on the Linux bridge qbr handle state tracking for the packet.
(security group rules implementation shown in forthcoming sections).
The Linux bridge qbr forwards the packet to the Open vSwitch integration bridge br-int.
The Open vSwitch integration bridge br-int adds the internal tag for private1.
For VLAN project networks:

Lukáš Korous
 The Open vSwitch integration bridge br-int forwards the packet to the Open vSwitch VLAN bridge
br-vlan.
 The Open vSwitch VLAN bridge br-vlan replaces the internal tag with the actual VLAN tag of
private1.
 The Open vSwitch VLAN bridge br-vlan forwards the packet to network1 via the VLAN interface.
For VXLAN and GRE project networks:
 The Open vSwitch integration bridge br-int forwards the packet to the Open vSwitch tunnel bridge
br-tun.
$ ovs-vsctl show
Bridge br-tun
fail_mode: secure
Port patch-int
Interface patch-int
type: patch
options: {peer=patch-tun}
Port br-tun
Interface br-tun
type: internal
Port "vxlan-0a00010c"
Interface "vxlan-0a00010c"
type: vxlan
options: {df_default="true", in_key=flow,
local_ip="10.0.1.11", out_key=flow, remote_ip="10.0.1.12"}
 The Open vSwitch tunnel bridge br-tun wraps the packet in a VXLAN or GRE tunnel and adds a tag
to identify private1.
 The Open vSwitch tunnel bridge br-tun forwards the packet to network1 via the tunnel interface.
$ ovs-ofctl show br-tun
$ ovs-ofctl dump-flows br-tun
network1
 The VLAN interface forwards the packet to the Open vSwitch VLAN bridge br-vlan.
 The Open vSwitch VLAN bridge br-vlan forwards the packet to the Open vSwitch integration bridge
br-int.
 The Open vSwitch integration bridge br-int replaces the actual VLAN tag of private1 with the
internal tag.
 The tunnel interface forwards the packet to the Open vSwitch tunnel bridge br-tun.
 The Open vSwitch tunnel bridge br-tun unwraps the packet and adds the internal tag for private1.
 The Open vSwitch tunnel bridge br-tun forwards the packet to the Open vSwitch integration bridge
br-int.

Lukáš Korous
# The same rules, only for different OVS port
$ ovs-ofctl show br-tun
The Open vSwitch integration bridge br-int forwards the packet to the qr interface (3) in the router
namespace qrouter on private1.
$ ovs-vsctl show
...
Bridge br-int
...
# We can identify the correct router by its interfaces
# which are shown in neutron (API / dashboard)
Port "qr-08c54a3d-d7"
tag: 12
Interface "qr-08c54a3d-d7"
type: internal
Port "qr-9e3fc2ff-c7"
tag: 11
Interface "qr-9e3fc2ff-c7"
type: internal
Port "qr-e5c327d2-0d"
tag: 11
Interface "qr-e5c327d2-0d"
type: internal
$ ip netns list
qrouter-2b881a48-6a45-4395-92a8-b7060aea2f67
qrouter-7c5068d8-7d23-46f3-89d6-64c3345609c2
# The ids here are ids of the router in neutron
# The namespace contains both interfaces to both virtual networks.
$ ip netns exec qrouter-2b881a48-6a45-4395-92a8-b7060aea2f67
ifconfig –a
# This is the one on private2
qr-08c54a3d-d7 Link encap:Ethernet HWaddr fa:16:3e:30:52:b9
inet addr:192.168.2.1 Bcast:192.168.2.255
Mask:255.255.255.0
inet6 addr: fe80::f816:3eff:fe30:52b9/64 Scope:Link
RX bytes:505820 (493.9 KiB) TX bytes:57269 (55.9 KiB)
# This is the one on private1
qr-9e3fc2ff-c7 Link encap:Ethernet HWaddr fa:16:3e:66:5f:71
inet addr:192.168.1.254 Bcast:192.168.1.255
Mask:255.255.255.0
inet6 addr: fe80::f816:3eff:fe66:5f71/64 Scope:Link

Lukáš Korous
RX bytes:45279 (44.2 KiB) TX bytes:9168 (8.9 KiB) This qr
interface contains ROUTER_PRIVATE_1_IP.
This qr interface contains the private1 gateway IP address ROUTER_PRIVATE_1_IP.
This qr interface contains the private2 gateway IP address ROUTER_PRIVATE_2_IP.
The router namespace qrouter routes the packet to the qr interface on private2 (4).
$ ip netns exec qrouter-2b881a48-6a45-4395-92a8-b7060aea2f67 route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref
Use Iface
192.168.1.0 * 255.255.255.0 U 0 0
0 qr-9e3fc2ff-c7
192.168.2.0 * 255.255.255.0 U 0 0
0 qr-08c54a3d-d7
The router namespace qrouter then forwards the packet to the Open vSwitch integration bridge br-int.
br-vlan.
private2.
 The Open vSwitch VLAN bridge br-vlan forwards the packet to compute2 via the VLAN interface.
br-tun.
 The Open vSwitch tunnel bridge br-tun forwards the packet to compute2 via the tunnel interface.
compute2
br-int.
internal tag.

Lukáš Korous
br-int.
The Open vSwitch integration bridge br-int forwards the packet to the Linux bridge qbr.
Security group rules (5) on the Linux bridge qbr handle firewalling and state tracking for the packet.
The Linux bridge qbr forwards the packet to the tap interface (6) on vm2.

Lukáš Korous
VM <-> VM – source: openstack.org, edited

Lukáš Korous
VM -> Outside world (North->South)
Setup
 External network ‘public’
o Network e.g. 203.0.113.0/24
o IP address allocation pool PUBLIC_ALLOCATION_POOL
 e.g. from 203.0.113.101 to 203.0.113.200
o Network e.g. 192.168.1.0/24
o Gateway PRIVATE_GATEWAY
 e.g. 192.168.1.1 with MAC address PRIVATE_GATEWAY_MAC
 private address: VM_PRIVATE_IP (private1 network)
 MAC address: VM_MAC
o interface address on public network: ROUTER_PUBLIC_IP (from PUBLIC_ALLOCATION_POOL)
 e.g. 203.0.113.101
o interface address on private1 network: ROUTER_PRIVATE_IP (== PRIVATE_GATEWAY)
Use case
The machine vm1 sends a packet to a host on public network.
Notes:
 The instance and the router can reside on the same machine (when bundled nodes are used), in this
case, no tunneling is involved.
compute1
The vm1 tap interface (1) forwards the packet to the Linux bridge qbr. The packet contains destination MAC
address PRIVATE_GATEWAY_MAC because the destination resides on another network.
Security group rules (2) on the Linux bridge qbr handle state tracking for the packet.
$ iptables –L –v
# Egress rules
Chain neutron-openvswi-o23bc0b1f-a (2 references)
pkts bytes target prot opt in out source
destination
2 656 RETURN udp -- any any anywhere
anywhere udp spt:bootpc dpt:bootps /* Allow DHCP client
traffic. */
3040 231K neutron-openvswi-s23bc0b1f-a all -- any any
anywhere anywhere
0 0 DROP udp -- any any anywhere
anywhere udp spt:bootps dpt:bootpc /* Prevent DHCP
Spoofing by VM. */

Lukáš Korous
2746 208K RETURN all -- any any anywhere
anywhere state RELATED,ESTABLISHED /* Direct packets
associated with a known session to the RETURN chain. */
294 22536 RETURN all -- any any anywhere
anywhere
0 0 DROP all -- any any anywhere
anywhere state INVALID /* Drop packets that appear
related to an existing connection (e.g. TCP ACK/FIN) but do not have
an entry in conntrack. */
0 0 neutron-openvswi-sg-fallback all -- any any
anywhere anywhere /* Send unmatched traffic
to the fallback chain. */
The Linux bridge qbr forwards the packet to the Open vSwitch integration bridge br-int.
 The Open vSwitch integration bridge br-int forwards the packet to the Open vSwitch VLAN bridge br-
vlan.
 The Open vSwitch VLAN bridge br-vlan replaces the internal tag with the actual VLAN tag of private1.
 The Open vSwitch VLAN bridge br-vlan forwards the packet to network1 via the VLAN interface.
br-tun.
 The Open vSwitch tunnel bridge br-tun forwards the packet to network1 via the tunnel interface.
network1
br-int.
internal tag.
br-int.

Lukáš Korous
The Open vSwitch integration bridge br-int forwards the packet to the qr interface (3) in the router
namespace qrouter. The qr interface contains ROUTER_PRIVATE_IP.
$ ip netns exec qrouter-7c5068d8-7d23-46f3-89d6-64c3345609c2
ifconfig qr-e5c327d2-0d
qr-e5c327d2-0d Link encap:Ethernet HWaddr fa:16:3e:3f:18:b4
inet addr:192.168.1.1 Bcast:192.168.1.255
Mask:255.255.255.0
inet6 addr: fe80::f816:3eff:fe3f:18b4/64 Scope:Link
RX bytes:244400 (238.6 KiB) TX bytes:37434692 (35.7
MiB)The router namespace qrouter forwards the packet to the Open
vSwitch integration bridge br-int.
The iptables service (4) performs SNAT on the packet using the qg interface (5) as the source IP address.
iptables -t nat –L
Chain neutron-l3-agent-float-snat (1 references)
target prot opt source destination
SNAT all -- host-192-168-1-3.openstacklocal anywhere
to:192.168.150.243
Chain neutron-l3-agent-snat (1 references)
neutron-l3-agent-float-snat all -- anywhere anywhere
SNAT all -- anywhere anywhere
to:203.0.113.101
SNAT all -- anywhere anywhere mark
match ! 0x2 ctstate DNAT to:203.0.113.101
Chain neutron-postrouting-bottom (1 references)
neutron-l3-agent-snat all -- anywhere anywhere
/* Perform source NAT on outgoing traffic. */
neutron-vpnaas-a-snat all -- anywhere anywhere
/* Perform source NAT on outgoing traffic. */
Chain neutron-vpnaas-a-OUTPUT (1 references)
Chain neutron-vpnaas-a-POSTROUTING (1 references)
ACCEPT all -- anywhere anywhere !
ctstate DNAT
Chain neutron-vpnaas-a-PREROUTING (1 references)

Lukáš Korous
REDIRECT tcp -- anywhere 169.254.169.254 tcp
dpt:http redir ports 9697
Chain neutron-vpnaas-a-float-snat (1 references)
Chain neutron-vpnaas-a-snat (1 references)
neutron-vpnaas-a-float-snat all -- anywhere anywhere
SNAT all -- anywhere anywhere
to:203.0.113.101
SNAT all -- anywhere anywhere mark
match ! 0x2 ctstate DNAT to:203.0.113.101
The qg interface contains ROUTER_PUBLIC_IP.
$ ip netns
qrouter-7c5068d8-7d23-46f3-89d6-64c3345609c2
ifconfig qg-fe71d0cc-ea
qg-fe71d0cc-ea Link encap:Ethernet HWaddr fa:16:3e:87:4c:b2
inet addr: 203.0.113.101 Bcast: 203.0.113.255
Mask:255.255.255.0
inet6 addr: fe80::f816:3eff:fe87:4cb2/64 Scope:Link
The router namespace qrouter forwards the packet to the Open vSwitch integration bridge br-int via the qg
interface.
The Open vSwitch integration bridge br-int forwards the packet to the Open vSwitch external bridge br-ex.
The Open vSwitch external bridge br-ex forwards the packet to public via the external interface.
$ ovs-vsctl show
2b86e0d4-7022-464d-bfca-778e4c09ce4e
Bridge br-ex # external bridge
Port phy-br-ex # patch to internal bridge
Interface phy-br-ex
type: patch
options: {peer=int-br-ex}
Port "eth2" # external “physical” interface
Interface "eth2"

Lukáš Korous
VM -> Outside world – source: openstack.org, edited
Outside world -> VM (Floating IP)
Setup
 External network ‘public’
o Network e.g. 203.0.113.0/24
o IP address allocation pool PUBLIC_ALLOCATION_POOL
 e.g. from 203.0.113.101 to 203.0.113.200

Lukáš Korous
o Network e.g. 192.168.1.0/24
o Gateway PRIVATE_GATEWAY
 e.g. 192.168.1.1 with MAC address PRIVATE_GATEWAY_MAC
 private address: VM_PRIVATE_IP (private1 network)
 e.g. 192.168.1.3
 MAC address: VM_MAC
 floating IP address: VM_FLOATING_IP
 e.g. 203.0.113.200
o interface address on public network: ROUTER_PUBLIC_IP (from PUBLIC_ALLOCATION_POOL)
 e.g. 203.0.113.101
o interface address on private1 network: ROUTER_PRIVATE_IP (== PRIVATE_GATEWAY)
Use case
An external host sends a packet to VM_FLOATING_IP
network1
The external interface forwards the packet to the Open vSwitch external bridge br-ex.
The Open vSwitch external bridge br-ex forwards the packet to the Open vSwitch integration bridge br-int
via a patch:
$ ovs-vsctl show
2b86e0d4-7022-464d-bfca-778e4c09ce4e
Bridge br-ex # external bridge
Port phy-br-ex # patch to internal bridge
Interface phy-br-ex
type: patch
options: {peer=int-br-ex}
Port "eth2" # external “physical” interface
Interface "eth2"
...
Bridge br-int # internal bridge
Port br-int # internal bridge interface
Interface br-int
type: internal
Port int-br-ex # patch to external bridge
Interface int-br-ex
type: patch
options: {peer=phy-br-ex}
The Open vSwitch integration bridge forwards the packet to the qg interface (1) in the router namespace
qrouter.
The qg interface contains ROUTER_PUBLIC_IP.
$ ip netns

Lukáš Korous
qrouter-7c5068d8-7d23-46f3-89d6-64c3345609c2
ifconfig qg-fe71d0cc-ea
qg-fe71d0cc-ea Link encap:Ethernet HWaddr fa:16:3e:87:4c:b2
inet addr: 203.0.113.101 Bcast: 203.0.113.255
Mask:255.255.255.0
inet6 addr: fe80::f816:3eff:fe87:4cb2/64 Scope:Link
The iptables service (2) performs DNAT on the packet using the qr interface (3) as the source IP address.
iptables -t nat –L
Chain neutron-l3-agent-OUTPUT (1 references)
DNAT all -- anywhere 203.0.113.200
to:192.168.1.3
Chain neutron-l3-agent-POSTROUTING (1 references)
ACCEPT all -- anywhere anywhere !
ctstate DNAT
Chain neutron-l3-agent-PREROUTING (1 references)
DNAT all -- anywhere 203.0.113.200
to:192.168.1.3
The qr interface contains ROUTER_PRIVATE_IP.
ifconfig qr-e5c327d2-0d
qr-e5c327d2-0d Link encap:Ethernet HWaddr fa:16:3e:3f:18:b4
inet addr:192.168.1.1 Bcast:192.168.1.255
Mask:255.255.255.0
inet6 addr: fe80::f816:3eff:fe3f:18b4/64 Scope:Link
RX bytes:244400 (238.6 KiB) TX bytes:37434692 (35.7
MiB)The router namespace qrouter forwards the packet to the Open
vSwitch integration bridge br-int.

Lukáš Korous
br-vlan.
private1.
 The Open vSwitch VLAN bridge br-vlan forwards the packet to compute1 via the VLAN interface.
br-tun.
 The Open vSwitch tunnel bridge br-tun forwards the packet to compute1 via the tunnel interface.
compute1
br-int.
 The Open vSwitch integration bridge br-int replaces the actual VLAN tag private1 with the internal
tag.
br-int.
 The Open vSwitch integration bridge br-int forwards the packet to the Linux bridge qbr.
$ brctl show # qbr bridge
bridge name|bridge id|STP enabled|interfaces
qbr23bc0b1f-a1|8000.5a0022a495e1|no|qvb23bc0b1f-a1 tap23bc0b1f-a1
Security group rules (4) on the Linux bridge qbr handle firewalling and state tracking for the packet.
# Ingress rules
Chain neutron-openvswi-i23bc0b1f-a (1 references)
pkts bytes target prot opt in out source
destination
27878 37M RETURN all -- any any anywhere
anywhere state RELATED,ESTABLISHED /* Direct packets
associated with a known session to the RETURN chain. */
2 730 RETURN udp -- any any host-192-168-1-
2.openstacklocal anywhere udp spt:bootps dpt:bootpc
109 5245 RETURN all -- any any anywhere
anywhere
0 0 RETURN tcp -- any any 56.Red-88-12-
34.staticIP.rima-tde.net anywhere tcp dpt:ssh

Lukáš Korous
0 0 DROP all -- any any anywhere
anywhere state INVALID /* Drop packets that appear
related to an existing connection (e.g. TCP ACK/FIN) but do not have
an entry in conntrack. */
0 0 neutron-openvswi-sg-fallback all -- any any
anywhere anywhere /* Send unmatched traffic
to the fallback chain. */
The Linux bridge qbr forwards the packet to the tap interface (5) on vm1.
Outside world -> VM (Floating IP) – source: openstack.org, edited

Lukáš Korous
Network traffic inspection, troubleshooting
Traffic tools
tcpdump
$ tcpdump -envi br-int
$ tcpdump -envi br-tun
$ ip netns exec qrouter-UUID tcpdump -i qr-63ea2815-b5 icmp
$ ip netns exec qrouter-UUID tcpdump -i qg-e7110dba-a9 icmp
$ tcpdump -envi 192.168.122.163
$ tcpdump -envi br-ex
$ tcpdump -i eth0 -n arp or icmp
$ tcpdump -i br-ex -n icmp
$ tcpdump -i eth0 -n icmp
$ tcpdump -i any -n icmp
$ tcpdump -i tape7110dba-a9 -n icmp
$ tcpdump -envi qvbb71536f2-dd -n arp or icmp
$ tcpdump -i eth0 -n not port 22
$ tcpdump -i eth0 -n not port 22 and not port amqp
$ tcpdump -i eth2 -f "not arp and not icmp and not port 22 and not
port amqp"
tshark
similar to tcpdump
$ tshark -i eth2 -f "not arp and not icmp and not port 22 and not
port amqp"
ovs-vsctl
$ ovs-vsctl show
$ ovs-dpctl show
$ ovs-dpctl dump-flows
$ ovs-ofctl dump-flows br-tun table=21
$ ovs-ofctl dump-flows br-int
DHCP state files
$ ls /var/lib/neutron/dhcp

Lukáš Korous
Static tools
ip
$ ip addr
$ ip route
$ ip –d link
$ ip netns
$ in netns qrouter-UUID ip a
$ ip netns qrouter-UUID exec ip link
$ ip netns qrouter-UUID exec route -n
$ ip netns qrouter-UUID exec iptables -L -t nat
$ ip netns qdhcp-UUID ip a
$ ip netns qdhcp-UUID ip link
$ ip netns qdhcp-UUID exec route -n
ifconfig
$ ifconfig -a
iptables
$ iptables -t nat –L
$ iptables -t nat –L | grep SNAT
$ iptables -t nat –L | grep DNAT
brctl
$ brctl show
cat /proc/<proc-id>/cmdline
e.g. dnsmasq
$ cat /proc/980(dnsmasq-proc-id)/cmdline
dnsmasq --no-hosts --no-resolv --strict-order --bind-interfaces --
interface=tap6cb4855a-ce --except-interface=lo --pid-
file=/var/lib/neutron/dhcp/e58741d0-60c7-457e-8780-366ecd3200aa/pid
--dhcp-hostsfile=/var/lib/neutron/dhcp/e58741d0-60c7-457e-8780-
366ecd3200aa/host
--addn-hosts=/var/lib/neutron/dhcp/e58741d0-60c7-457e-8780-
366ecd3200aa/addn_hosts --dhcp-
optsfile=/var/lib/neutron/dhcp/e58741d0-60c7-457e-8780-
366ecd3200aa/opts --dhcp-leasefile=/var/lib/neutron/dhcp/e58741d0-
60c7-457e-8780-366ecd3200aa/leases --dhcp-

Lukáš Korous
range=set:tag0,192.168.150.0,static,86400s --dhcp-lease-max=256 --
conf-file=/etc/neutron/dnsmasq-neutron.conf --server=8.8.8.8 --
server=8.8.4.4 --domain=openstacklocal
Logging
/var/log/neutron/*
/var/log/syslog # dnsmasq
Links
Troubleshooting
http://docs.openstack.org/openstack-ops/content/network_troubleshooting.html
https://www.rdoproject.org/Networking_in_too_much_detail
http://www.yet.org/2014/09/openvswitch-troubleshooting/
General
http://docs.openstack.org/admin-guide-cloud/networking.html
http://docs.openstack.org/developer/neutron/devref/layer3.html
http://docs.openstack.org/admin-guide-cloud/networking_adv-features.html#provider-networks
https://access.redhat.com/documentation/en/red-hat-enterprise-linux-openstack-platform/version-
7/architecture-guide/
https://access.redhat.com/documentation/en/red-hat-enterprise-linux-openstack-platform/version-
7/networking-guide/
https://www.rdoproject.org/networking/neutron-with-existing-external-network/
https://access.redhat.com/documentation/en-
US/Red_Hat_Enterprise_Linux_OpenStack_Platform/5/html/Cloud_Administrator_Guide/section_networki
ng-scenarios.html
https://access.redhat.com/documentation/en-
US/Red_Hat_Enterprise_Linux/6/html/Virtualization_Host_Configuration_and_Guest_Installation_Guide/ch
ap-Virtualization_Host_Configuration_and_Guest_Installation_Guide-SR_IOV.html
http://docs.openstack.org/kilo/config-reference/content/neutron-conf-changes-kilo.html

Neutron kilo

More Related Content

Viewers also liked

Similar to Neutron kilo

Recently uploaded

Neutron kilo