Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Weird things we've seen with OpenStack Neutron

528 views

Published on

A presentation given at the Manchester OpenStack Meetup, talking through some of the odd things we've hit up against in our time as a public OpenStack operator using Neuton with OpenvSwitch.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Weird things we've seen with OpenStack Neutron

  1. 1. Weird stuff we've seen with OpenStack Neutron (And what to do about it)
  2. 2. OpenStack Neutron • So$ware-defined networking component • Users define their own virtual networks • Manages IP address assignment • Floa?ng IP addresses • Supports many different back-ends - OpenvSwitch, VMware NSX, Cisco UCS, Midokura....
  3. 3. Neutron usage1 1 Source: OpenStack User Survey, October 2015
  4. 4. Simplified logical architecture
  5. 5. Architecture, con-nued • neutron-{server,agent} • OpenvSwitch • Linux bridging • Linux network namespaces • L2 • L3
  6. 6. Namespaces • L2 namespace • DHCP • L3 namespace • Rou4ng • NAT • Metadata
  7. 7. Common problems - typical user complaints • VM can't obtain an IP address • Can't ping / connect to my VM • Intermi9ent connec:vity
  8. 8. Weirdness #1 - orphaned namespaces • Default (on Ubuntu) is not to delete namespaces at all (!) • Bug in iproute2 package • h=ps://bugs.launchpad.net/neutron/+bug/1052535 • Misconfigured sudo rules meant that network namespaces weren't being deleted • Mismatch between interfaces configured in a namespace and what Neutron expects
  9. 9. Finding out what's supposed to be where for netnode in osnet{0..4} ; do echo $netnode for router in $(ssh $netnode 'ip netns list | grep qrouter | cut -d - -f 2-20') ; do neutron router-show $router | grep -i unable done done Then delete each invalid namespace and associated OVS port. • Pro%p: Don't run neutron-ovs-cleanup!
  10. 10. Weirdness #2 - duplicate segmenta4on ID • Customer support ,cket with instances unable to obtain an IP via DHCP • Some serious digging required...
  11. 11. Tracing packet flows • tcpdump on compute node and in network namespaces • Packets not always arriving where you'd expect • Have to look at OpenFlow rules
  12. 12. DHCP agent neutron dhcp-agent-list-hosting-net 4dc325ed-f141-41d9-8d0a-4f513defacad +--------------------------------------+--------+----------------+-------+ | id | host | admin_state_up | alive | +--------------------------------------+--------+----------------+-------+ | 1beb99ef-e6f6-4083-8fb6-661f2f61c565 | osnet1 | True | :-) | +--------------------------------------+--------+----------------+-------+ neutron net-show -F provider:segmentation_id 4dc325ed-f141-41d9-8d0a-4f513defacad +--------------------------+-------+ | Field | Value | +--------------------------+-------+ | provider:segmentation_id | 11 | +--------------------------+-------+ • 11 in hex = 0xb
  13. 13. root@osnet1:~# ovs-ofctl dump-flows br-tun table=2 NXST_FLOW reply (xid=0x4): cookie=0x0, duration=875584.823s, table=2, n_packets=85, n_bytes=10880, idle_age=11560, hard_age=65534, priority=1,tun_id=0x14 actions=mod_vlan_vid:43,resubmit(,10) cookie=0x0, duration=2578615.436s, table=2, n_packets=1345, n_bytes=128202, idle_age=27174, hard_age=65534, priority=1,tun_id=0x10 actions=mod_vlan_vid:2,resubmit(,10) cookie=0x0, duration=2578611.677s, table=2, n_packets=0, n_bytes=0, idle_age=65534, hard_age=65534, priority=1,tun_id=0xd actions=mod_vlan_vid:12,resubmit(,10) cookie=0x0, duration=1806356.959s, table=2, n_packets=5140, n_bytes=364533, idle_age=341, hard_age=65534, priority=1,tun_id=0x21 actions=mod_vlan_vid:35,resubmit(,10) cookie=0x0, duration=2578610.661s, table=2, n_packets=1035919, n_bytes=180430025, idle_age=65534, hard_age=65534, priority=1,tun_id=0x11 actions=mod_vlan_vid:16,resubmit(,10) cookie=0x0, duration=1465355.359s, table=2, n_packets=418252, n_bytes=81112777, idle_age=52, hard_age=65534, priority=1,tun_id=0x13 actions=mod_vlan_vid:42,resubmit(,10) cookie=0x0, duration=1631281.273s, table=2, n_packets=445, n_bytes=52848, idle_age=65534, hard_age=65534, priority=1,tun_id=0x17 actions=mod_vlan_vid:37,resubmit(,10) cookie=0x0, duration=2578609.671s, table=2, n_packets=1821, n_bytes=167272, idle_age=16439, hard_age=65534, priority=1,tun_id=0xc actions=mod_vlan_vid:17,resubmit(,10) cookie=0x0, duration=2574619.932s, table=2, n_packets=490592856, n_bytes=279835052124, idle_age=65534, hard_age=65534, priority=1,tun_id=0x19 actions=mod_vlan_vid:19,resubmit(,10) cookie=0x0, duration=2578613.06s, table=2, n_packets=18, n_bytes=756, idle_age=65534, hard_age=65534, priority=1,tun_id=0xe actions=mod_vlan_vid:8,resubmit(,10) cookie=0x0, duration=1469974.534s, table=2, n_packets=6992536, n_bytes=1567235429, idle_age=9, hard_age=65534, priority=1,tun_id=0x7 actions=mod_vlan_vid:41,resubmit(,10) cookie=0x0, duration=2144082.193s, table=2, n_packets=2583, n_bytes=461773, idle_age=65534, hard_age=65534, priority=1,tun_id=0x1d actions=mod_vlan_vid:32,resubmit(,10) cookie=0x0, duration=2578611.169s, table=2, n_packets=4230304, n_bytes=917966422, idle_age=0, hard_age=65534, priority=1,tun_id=0x5 actions=mod_vlan_vid:14,resubmit(,10) cookie=0x0, duration=85135.825s, table=2, n_packets=1739, n_bytes=130092, idle_age=65534, hard_age=65534, priority=1,tun_id=0x1f actions=mod_vlan_vid:53,resubmit(,10) cookie=0x0, duration=979.195s, table=2, n_packets=123, n_bytes=11895, idle_age=933, priority=1,tun_id=0x22 actions=mod_vlan_vid:54,resubmit(,10) cookie=0x0, duration=1898543.732s, table=2, n_packets=240, n_bytes=30712, idle_age=65534, hard_age=65534, priority=1,tun_id=0x16 actions=mod_vlan_vid:34,resubmit(,10) cookie=0x0, duration=2578614.004s, table=2, n_packets=5595775, n_bytes=5465543420, idle_age=4, hard_age=65534, priority=1,tun_id=0x8 actions=mod_vlan_vid:6,resubmit(,10) cookie=0x0, duration=1473941.345s, table=2, n_packets=4202494, n_bytes=2516931444, idle_age=9, hard_age=65534, priority=1,tun_id=0x4 actions=mod_vlan_vid:40,resubmit(,10) cookie=0x0, duration=2578619.787s, table=2, n_packets=103506, n_bytes=13925984, idle_age=0, hard_age=65534, priority=0 actions=drop wat.
  14. 14. OpenFlow flows2 2 h$p://assafmuller.com/2013/10/14/gre-tunnels-in-openstack-neutron/
  15. 15. Missing OpenFlow rule root@osnet1:~# ovs-ofctl dump-flows br-tun table=2 | grep 0xb root@osnet1:~# echo $? 1 Try to re-add that network to the responsible agent: $ neutron dhcp-agent-network-remove 1beb99ef-e6f6-4083-8fb6-661f2f61c565 4dc325ed-f141-41d9-8d0a-4f513defacad Removed network 4dc325ed-f141-41d9-8d0a-4f513defacad from DHCP agent $ neutron dhcp-agent-network-add 1beb99ef-e6f6-4083-8fb6-661f2f61c565 4dc325ed-f141-41d9-8d0a-4f513defacad Added network 4dc325ed-f141-41d9-8d0a-4f513defacad to DHCP agent root@osnet1:~# ovs-ofctl dump-flows br-tun table=2 | grep 0xb cookie=0x0, duration=0.945s, table=2, n_packets=14, n_bytes=588, idle_age=0, priority=1,tun_id=0xb actions=mod_vlan_vid:55,resubmit(,10)
  16. 16. Weirdness #3 - duplicate routers • Intermi)ent connec-vity issues groan • No DVR or L3-HA enabled • Routers scheduled and created twice on two network nodes • Same network configura-on in each namespace
  17. 17. Duplicate routers › neutron l3-agent-list-hosting-router fe79ae7e-debf-44b9-8fd7-601abd5fb928 +--------------------------------------+--------+----------------+-------+----------+ | id | host | admin_state_up | alive | ha_state | +--------------------------------------+--------+----------------+-------+----------+ | 48132c36-b6b1-40fa-b9d9-5474f4f27c3a | osnet0 | True | :-) | | | c821a370-b301-40c5-8b7b-25d147ffc904 | osnet1 | True | :-) | | +--------------------------------------+--------+----------------+-------+----------+ › neutron router-show fe79ae7e-debf-44b9-8fd7-601abd5fb928 +-----------------------+----------------------------------+ | Field | Value | +-----------------------+----------------------------------+ | admin_state_up | True | | distributed | False | | ha | False | | status | ACTIVE | | tenant_id | 7d718c99276c43d1992d64d061d98f15 | +-----------------------+----------------------------------+
  18. 18. How to approach troubleshoo0ng Troubleshoo*ng checklist • UUIDs for instance, loca2on, MAC address • UUIDs for network, subnet, router • Network node hos2ng L2 and L3 agents
  19. 19. Useful commands - neutron $ neutron agent-list $ neutron l3-agent-list-hosting-router $router_uuid $ neutron dhcp-agent-list-hosting-net $net_uuid $ neutron router-list-on-l3-agent $agent_uuid $ neutron net-list-on-dhcp-agent $net_uuid $ neutron help
  20. 20. Useful commands - OpenvSwitch $ ovs-vsctl show $ ovs-ofctl dump-flows $bridge $ ovs-dpctl show
  21. 21. (More) useful commands Standard network troubleshoo1ng toolkit: $ tcpdump -enl -i eth1 | grep -i dhcp $ ip netns exec $netns tcpdump port 67 or port 68 -lne $ ip route $ ip address $ iptables-save $ brctl $ mtr Etc.
  22. 22. Thanks! Nick Jones DataCentred h"p://www.datacentred.co.uk h"p://dischord.org @yankcrime

×