Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

How VXLAN works on L2 and across L3 networks ?


Published on

Presentation on How VXLAN works on L2 and across L3 networks ? A small exercise in the end and a hands-on lab that will give you a complete understanding of the concept in SDN.

Published in: Technology

How VXLAN works on L2 and across L3 networks ?

  1. 1. VXLAN By Anand Nande
  2. 2. AGENDA ○ What is VXLAN ? ○ Why VXLAN ? ○ How does it work ? ○ So now we can migrate VMs across subnets? ○ What about routing across VXLANs? ○ Any Performance Impact? ○ Demo
  3. 3. What is it ? Virtual eXtensible Local Area Network
  4. 4. ● Tunneling protocol co-developed by a group of companies. ● Original description as per the RFC: “ Used to address the need for overlay networks within virtualized data centers accommodating multiple tenants. The scheme and the related protocols can be used in networks for cloud service providers and enterprise data centers ” ● Encapsulation method to extend L2 traffic over a L3 network. VXLAN uses multicast in the backend. It uses traditional flood and learn for MAC learning and Address Resolution Protocol (ARP) resolution.
  5. 5. Why VXLAN? What problems does it address and how am I selling it
  6. 6. It's intended to solve the following issues ● Limitations Imposed by Spanning Tree and VLAN Ranges ○ STP blocks the use of links to avoid the replication and looping of frames. ■ This is a prob to some DC admins who pay for ports and links ■ Resiliency due to multipathing - not available. ○ 2^12 = 4096 ■ A 12-bit VLAN ID used to divide multiple broadcast domains ■ STP blocks few id’s from this domain ● Multi-tenant Environments ○ Is it possible to address need for multiple VLANs per tenant w/ 4096 limit? ○ L3 networks not a comprehensive solution. 2 tenants might use the same set of Layer 3 addresses within their networks. ● Inadequate Table Sizes at ToR Switch
  7. 7. How does it work? Let’s not draw conclusions yet..there’s a lot more to it
  8. 8. ● VXLAN encapsulates Layer 2 frames into Layer 3 packets (using UDP) ● Adds a 24-bit VXLAN Network Identifier (VNI) that allows for up to 16 million unique combinations ● VXLAN Segments are built between VXLAN Tunnel Endpoints (VTEPs)...more on this in coming slides ● Not understood by any physical networking devices (the transport that carries the encapsulated frames only needs an IP-based network).
  9. 9. VXLAN Encapsulation The Final packet that goes out on L-3. Note how inner-MAC is payload here and outer-MAC + UDP is added. The VNI is the only distinguished identifier for a VXLAN segment The L2 Ethernet Frame
  10. 10. VTEPs across a L3 network VM1-1 VM2-1 L3 Network VNI=10 VM1-2 VM2-2 VNI=20 IP= MAC=52:54:00:0e:08:b3 VNI=10 IP= MAC=52:54:00:30:de:e3 VNI=10 IP= MAC=00:0C:29:2F:32:A0 VNI=20 IP= MAC=00:0C:29:2F:23:A0 VNI=20 VTEP-1 VTEP-2 VXLAN segment-1 w/ VNI=10 VXLAN segment-2 w/ VNI=20 Hypervisor 1 Hypervisor 2 VTEPs responsible for encap/decap packets at both ends of the wire
  11. 11. VXLAN Multicast group MAC VNI ID REMOTE VTEP 52:54:00:a0:1b:bb 10 52:54:00:8a:bd:ff 10 MAC VNI ID REMOTE VTEP 52:54:00:60:18:f9 10 52:54:00:8a:bd:ff 10 MAC VNI ID REMOTE VTEP 52:54:00:a0:1b:bb 10 52:54:00:60:18:f9 10 VTEP-1s table VTEP-2s table VTEP-3s table
  12. 12. packet capture on one of the VTEP’s
  13. 13. Additional Wireshark plugin required to analyse the UDP data here
  14. 14. So now can we migrate VMs across subnets? ..the answer is ‘YES - we can’
  15. 15. Whats the most important thing when we migrate a VM ? - VM’s ip and mac addresses should not change even if the VM has been migrated to a new hypervisor. - We use the reference controller which learns the new VM-placement using ovsDB.# ps -ef | grep controller | grep -v color root 1396 1325 0 Feb17 pts/3 00:00:01 controller -v ptcp:6633 - This reference controller is located on each VTEP.
  16. 16. VM communication via VTEP post-migration VM2-1 VNI=10 VM1-1VM1-2 IP= MAC=52:54:00:0e:08:b3 VNI=10 IP= MAC=52:54:00:30:de:e3 VNI=10 IP= MAC=00:0C:29:2F:32:A0 VNI=20 VTEP-1 VTEP-2 Hypervisor 1 Hypervisor 2 * The VM's port is updated to point to the new host. * When the port is updated by Nova, Neutron executes ML2 mechanism drivers, including l2pop, which then sends RPC messages to the new host. Flows related to vm1-1 removed on VTEP-1 and added to VTEP-2
  17. 17. What about routing across VXLANs ?
  18. 18. Lets look at the 2 way communication that happens physical:virtual [VLAN-to-VXLAN] - also known as ‘vxlan-bridging’ that extends the l2-domain over a vast l3-network. - VTEP’s are responsible in this case to the check for the VNI id in the VXLAN header. If it matches the one of the VNI’s thats in its table, forward the packet to the relevant vxlan-segment to take care of it. - Switch should be capable of vxlan handling. virtual:physical [VXLAN-to-VLAN] - vlan to router - hardware VTEP(router) only does bridging - on physical router: - in_ports, out_ports, loopback_ports 1. in_ports > bridge into a vlan > lo_ports 2. router_fib > next_hop > physical_world - performance loss because of loopback
  19. 19. Any Performance Impact? cliche but true “Your loss is someone else's gain”
  20. 20. ● An overlay network is simply a computer network which is built on top of another network. ● The added flexibility comes at a cost, due to the additional processing overhead for encapsulation and de-encapsulation of packets. This consumes both CPU resources and degrades network performance, especially for high speed connections. ● By introducing hardware offloading capabilities that can be found in some of today’s modern NICs, the added overhead for packet processing can be offloaded to the NIC hardware, resulting in improved CPU utilization and higher throughput. ● Improve performance of VXLAN w/ hardware offload:
  21. 21. DemoLet’s put this into action by creating a simple topology
  22. 22. Ignore for the purpose of demo. You can do this later as an advanced exercise
  23. 23. [On your KVM host or laptop] # yum install virt-manager libvirt* qemu-kvm openvswitch # cd /var/lib/libvirt/images ## Get the mininet vmdk image from the mininet website ## write the # chmod +x # ./ <vmname> (note you need to spawn 2 mininet VMs, hence use the above script 2wice) Hands on # cat > << EOM > #!/bin/sh > export VM_NAME=$1 > export portgroup=$2 > IMAGES_BASE=/var/lib/libvirt/images > cp $IMAGES_BASE/mininet.vmdk $IMAGES_BASE/$VM_NAME.vmdk > > virt-install -r 256 > -n $VM_NAME > --vcpus=1 > --import > --autostart > --memballoon virtio > --network network=host-only > --disk $IMAGES_BASE/$VM_NAME.vmdk > EOM
  24. 24. [Inside your Mininet VM-1] Create a simple topology: mininet@mininet-vm:~$ sudo mn --topo single,1 Create vxlan interface on each side: mininet> sh ovs-vsctl add-port s1 vxlan Setup vxlan interface on each side mininet> sh ovs-vsctl set interface vxlan type=vxlan option: remote_ip= Setup IP for hosts: mininet> h1 ifconfig h1-eth0 [Inside your Mininet VM-2] mininet@mininet-vm:~$ sudo mn --topo single,1 mininet> sh ovs-vsctl add-port s1 vxlan mininet> sh ovs-vsctl set interface vxlan type=vxlan option: remote_ip= mininet> h1 ifconfig h1-eth0
  25. 25. Advanced VXLAN configuration Manually assign VXLAN Network ID (VNI) and/or OpenFlow port number e.g. VNI=20, OF_PORT=9 # ovs-vsctl set interface vxlan type=vxlan option:remote_ip= option:key=5566 ofport_request=9 Assign VNI flow by flow (Useful when implementing multi-tenant environment) # ovs-vsctl set interface vxlan type=vxlan option:remote_ip= option:key=flow ofport_request=9 and then setup your flow entries by # ovs-ofctl add-flow s1 in_port=1,actions=set_field:5566->tun_id,output:9 # ovs-ofctl add-flow s1 in_port=9,tun_id=5566,actions=output:1
  26. 26. References - IETF RFC : - Scott Lowe’s blog : - David Mahlers youtube channel : - VMware VXLAN Performance Evaluation : vSphere-VXLAN-Perf.pdf - TCP/IP over VXLAN Bandwidth Overheads: bandwidth-overheads/