A presentation given at the Manchester OpenStack Meetup, talking through some of the odd things we've hit up against in our time as a public OpenStack operator using Neuton with OpenvSwitch.

1. Weird stuff we've seen with
OpenStack Neutron
(And what to do about it)

3. OpenStack Neutron
• So$ware-defined networking component
• Users define their own virtual networks
• Manages IP address assignment
• Floa?ng IP addresses
• Supports many different back-ends - OpenvSwitch, VMware
NSX, Cisco UCS, Midokura....

4. Neutron usage1
1
Source: OpenStack User Survey, October 2015

5. Simplified logical architecture

6. Architecture, con-nued
• neutron-{server,agent}
• OpenvSwitch
• Linux bridging
• Linux network namespaces
• L2
• L3

7. Namespaces
• L2 namespace
• DHCP
• L3 namespace
• Rou4ng
• NAT
• Metadata

8. Common problems - typical user complaints
• VM can't obtain an IP address
• Can't ping / connect to my VM
• Intermi9ent connec:vity

9. Weirdness #1 - orphaned namespaces
• Default (on Ubuntu) is not to delete namespaces at all (!)
• Bug in iproute2 package
• h=ps://bugs.launchpad.net/neutron/+bug/1052535
• Misconfigured sudo rules meant that network namespaces
weren't being deleted
• Mismatch between interfaces configured in a namespace and
what Neutron expects

10. Finding out what's supposed to be where
for netnode in osnet{0..4} ; do
echo $netnode
for router in $(ssh $netnode 'ip netns list | grep qrouter | cut -d - -f 2-20') ; do
neutron router-show $router | grep -i unable
done
done
Then delete each invalid namespace and associated OVS port.
• Pro%p: Don't run neutron-ovs-cleanup!

11. Weirdness #2 - duplicate segmenta4on ID
• Customer support ,cket with instances unable to obtain an IP via
DHCP
• Some serious digging required...

12. Tracing packet flows
• tcpdump on compute node and in network namespaces
• Packets not always arriving where you'd expect
• Have to look at OpenFlow rules

13. DHCP agent
neutron dhcp-agent-list-hosting-net 4dc325ed-f141-41d9-8d0a-4f513defacad
+--------------------------------------+--------+----------------+-------+
| id | host | admin_state_up | alive |
+--------------------------------------+--------+----------------+-------+
| 1beb99ef-e6f6-4083-8fb6-661f2f61c565 | osnet1 | True | :-) |
+--------------------------------------+--------+----------------+-------+
neutron net-show -F provider:segmentation_id 4dc325ed-f141-41d9-8d0a-4f513defacad
+--------------------------+-------+
| Field | Value |
+--------------------------+-------+
| provider:segmentation_id | 11 |
+--------------------------+-------+
• 11 in hex = 0xb

14. root@osnet1:~# ovs-ofctl dump-flows br-tun table=2
NXST_FLOW reply (xid=0x4):
cookie=0x0, duration=875584.823s, table=2, n_packets=85, n_bytes=10880, idle_age=11560, hard_age=65534, priority=1,tun_id=0x14 actions=mod_vlan_vid:43,resubmit(,10)
cookie=0x0, duration=2578615.436s, table=2, n_packets=1345, n_bytes=128202, idle_age=27174, hard_age=65534, priority=1,tun_id=0x10 actions=mod_vlan_vid:2,resubmit(,10)
cookie=0x0, duration=2578611.677s, table=2, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=1,tun_id=0xd actions=mod_vlan_vid:12,resubmit(,10)
cookie=0x0, duration=1806356.959s, table=2, n_packets=5140, n_bytes=364533, idle_age=341, hard_age=65534, priority=1,tun_id=0x21 actions=mod_vlan_vid:35,resubmit(,10)
cookie=0x0, duration=2578610.661s, table=2, n_packets=1035919, n_bytes=180430025, idle_age=65534, hard_age=65534, priority=1,tun_id=0x11 actions=mod_vlan_vid:16,resubmit(,10)
cookie=0x0, duration=1465355.359s, table=2, n_packets=418252, n_bytes=81112777, idle_age=52, hard_age=65534, priority=1,tun_id=0x13 actions=mod_vlan_vid:42,resubmit(,10)
cookie=0x0, duration=1631281.273s, table=2, n_packets=445, n_bytes=52848, idle_age=65534, hard_age=65534, priority=1,tun_id=0x17 actions=mod_vlan_vid:37,resubmit(,10)
cookie=0x0, duration=2578609.671s, table=2, n_packets=1821, n_bytes=167272, idle_age=16439, hard_age=65534, priority=1,tun_id=0xc actions=mod_vlan_vid:17,resubmit(,10)
cookie=0x0, duration=2574619.932s, table=2, n_packets=490592856, n_bytes=279835052124, idle_age=65534, hard_age=65534, priority=1,tun_id=0x19 actions=mod_vlan_vid:19,resubmit(,10)
cookie=0x0, duration=2578613.06s, table=2, n_packets=18, n_bytes=756, idle_age=65534, hard_age=65534, priority=1,tun_id=0xe actions=mod_vlan_vid:8,resubmit(,10)
cookie=0x0, duration=1469974.534s, table=2, n_packets=6992536, n_bytes=1567235429, idle_age=9, hard_age=65534, priority=1,tun_id=0x7 actions=mod_vlan_vid:41,resubmit(,10)
cookie=0x0, duration=2144082.193s, table=2, n_packets=2583, n_bytes=461773, idle_age=65534, hard_age=65534, priority=1,tun_id=0x1d actions=mod_vlan_vid:32,resubmit(,10)
cookie=0x0, duration=2578611.169s, table=2, n_packets=4230304, n_bytes=917966422, idle_age=0, hard_age=65534, priority=1,tun_id=0x5 actions=mod_vlan_vid:14,resubmit(,10)
cookie=0x0, duration=85135.825s, table=2, n_packets=1739, n_bytes=130092, idle_age=65534, hard_age=65534, priority=1,tun_id=0x1f actions=mod_vlan_vid:53,resubmit(,10)
cookie=0x0, duration=979.195s, table=2, n_packets=123, n_bytes=11895, idle_age=933, priority=1,tun_id=0x22 actions=mod_vlan_vid:54,resubmit(,10)
cookie=0x0, duration=1898543.732s, table=2, n_packets=240, n_bytes=30712, idle_age=65534, hard_age=65534, priority=1,tun_id=0x16 actions=mod_vlan_vid:34,resubmit(,10)
cookie=0x0, duration=2578614.004s, table=2, n_packets=5595775, n_bytes=5465543420, idle_age=4, hard_age=65534, priority=1,tun_id=0x8 actions=mod_vlan_vid:6,resubmit(,10)
cookie=0x0, duration=1473941.345s, table=2, n_packets=4202494, n_bytes=2516931444, idle_age=9, hard_age=65534, priority=1,tun_id=0x4 actions=mod_vlan_vid:40,resubmit(,10)
cookie=0x0, duration=2578619.787s, table=2, n_packets=103506, n_bytes=13925984, idle_age=0, hard_age=65534, priority=0 actions=drop
wat.

15. OpenFlow flows2
2
h$p://assafmuller.com/2013/10/14/gre-tunnels-in-openstack-neutron/

16. Missing OpenFlow rule
root@osnet1:~# ovs-ofctl dump-flows br-tun table=2 | grep 0xb
root@osnet1:~# echo $?
1
Try to re-add that network to the responsible agent:
$ neutron dhcp-agent-network-remove 1beb99ef-e6f6-4083-8fb6-661f2f61c565
4dc325ed-f141-41d9-8d0a-4f513defacad
Removed network 4dc325ed-f141-41d9-8d0a-4f513defacad from DHCP agent
$ neutron dhcp-agent-network-add 1beb99ef-e6f6-4083-8fb6-661f2f61c565
4dc325ed-f141-41d9-8d0a-4f513defacad
Added network 4dc325ed-f141-41d9-8d0a-4f513defacad to DHCP agent
root@osnet1:~# ovs-ofctl dump-flows br-tun table=2 | grep 0xb
cookie=0x0, duration=0.945s, table=2, n_packets=14, n_bytes=588, idle_age=0,
priority=1,tun_id=0xb actions=mod_vlan_vid:55,resubmit(,10)

17. Weirdness #3 - duplicate routers
• Intermi)ent connec-vity issues groan
• No DVR or L3-HA enabled
• Routers scheduled and created twice on two network nodes
• Same network configura-on in each namespace

18. Duplicate routers
› neutron l3-agent-list-hosting-router fe79ae7e-debf-44b9-8fd7-601abd5fb928
+--------------------------------------+--------+----------------+-------+----------+
| id | host | admin_state_up | alive | ha_state |
+--------------------------------------+--------+----------------+-------+----------+
| 48132c36-b6b1-40fa-b9d9-5474f4f27c3a | osnet0 | True | :-) | |
| c821a370-b301-40c5-8b7b-25d147ffc904 | osnet1 | True | :-) | |
+--------------------------------------+--------+----------------+-------+----------+
› neutron router-show fe79ae7e-debf-44b9-8fd7-601abd5fb928
+-----------------------+----------------------------------+
| Field | Value |
+-----------------------+----------------------------------+
| admin_state_up | True |
| distributed | False |
| ha | False |
| status | ACTIVE |
| tenant_id | 7d718c99276c43d1992d64d061d98f15 |
+-----------------------+----------------------------------+

19. How to approach troubleshoo0ng
Troubleshoo*ng checklist
• UUIDs for instance, loca2on, MAC address
• UUIDs for network, subnet, router
• Network node hos2ng L2 and L3 agents

20. Useful commands - neutron
$ neutron agent-list
$ neutron l3-agent-list-hosting-router $router_uuid
$ neutron dhcp-agent-list-hosting-net $net_uuid
$ neutron router-list-on-l3-agent $agent_uuid
$ neutron net-list-on-dhcp-agent $net_uuid
$ neutron help

21. Useful commands - OpenvSwitch
$ ovs-vsctl show
$ ovs-ofctl dump-flows $bridge
$ ovs-dpctl show

22. (More) useful commands
Standard network troubleshoo1ng toolkit:
$ tcpdump -enl -i eth1 | grep -i dhcp
$ ip netns exec $netns tcpdump port 67 or port 68 -lne
$ ip route
$ ip address
$ iptables-save
$ brctl
$ mtr
Etc.

23. Thanks!
Nick Jones
DataCentred
h"p://www.datacentred.co.uk
h"p://dischord.org
@yankcrime