SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.
SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.
Successfully reported this slideshow.
Activate your 14 day free trial to unlock unlimited reading.
A First Look At VXLAN over Infiniband Network On Linux 3.7-rc7 & iproute2
A First Look At VXLAN over Infiniband Network On Linux 3.7-rc7 & iproute2
1.
29 Nov, 2012
SAKURA Internet Research Center
Senior Researcher / Naoto MATSUMOTO
2.
VXLAN TESTING Overview
1) Linux bridge-based VXLAN is developing by Stephen Hemminger & et al.
http://linux-network-plumber.blogspot.jp/2012/09/just-published-linux-kernel.html
+config VXLAN
+ tristate "Virtual eXtensible Local Area Network (VXLAN)"
+ depends on EXPERIMENTAL
+ ---help---
+ This allows one to create vxlan virtual interfaces that provide
+ Layer 2 Networks over Layer 3 Networks. VXLAN is often used
+ to tunnel virtual network infrastructure in virtualized environments.
+ For more information see:
+ http://tools.ietf.org/html/draft-mahalingam-dutt-dcops-vxlan-02
2) Infiniband Adapter is the fastest of all for datacenter networking.
Dual-port Infiniband FDR 56Gbit/s
Host Bus Adapter
VXLAN + IP over Infiniband Network is AWESOME!.
3.
Buildup to the New Kernel & etc
1) Install OFED for Infiniband Networking
# wget https://www.openfabrics.org/downloads/OFED/ofed-1.5.4/OFED-1.5.4.1.tgz
# tar xzvf ./OFED-1.5.4.1.tgz
# ./OFED-1.5.4.1/install.pl /*Install OFED software */
# sync; sync; sync; reboot;
2) Linux Kernel Update
# cd /usr/local/src
# wget http://www.kernel.org/pub/linux/kernel/v3.0/testing/linux-3.7-rc7.tar.bz2
# tar xvfj linux-3.7-rc7.tar.bz2
# cd linux-3.7-rc7
# cp /boot/config-2.6.32-220.23.1.el6.x86_64 ./.config
# make oldconfig
# vi .config
CONFIG_VXLAN=m /* Add New Line */
# make; make modules_install ; make install
# sync; sync; sync; reboot
3) iproute2 Update
# yum install db4-devel.x86_64
# git clone git://git.kernel.org/pub/scm/linux/kernel/git/shemminger/iproute2.git
# cd iproute2; ./configure ; make ; make install
Source: SAKURA Internet Research Center. 11/2012: Project THORN
4.
VXLAN Configuration
# uname -a
Linux VXLAN-A 3.7.0-rc7 #1 SMP .... 56Gbit/s IPoIB
# ip addr add 10.1.1.1/24 dev ib1 IP over Infiniband Network
# ip route add 224.0.0.0/4 dev ib1
# ip link set up dev ib1
# ip link add vxlan99 type vxlan id 5001 group 239.0.0.99 ttl 10 dev ib1 VXLAN Network
# ip addr add 192.168.99.1/24 dev vxlan99
# ip link set up dev vxlan99
# iperf -u -s -B 239.0.0.99 & /* IGMPv3 Join TIPS */
# uname -a
Linux VXLAN-B 3.7.0-rc7 #1 SMP ....
# ip addr add 10.1.1.2/24 dev ib1
# ip route add 224.0.0.0/4 dev ib1
# ip link set up dev ib1
# ip link add vxlan99 type vxlan id 5001 group 239.0.0.99 ttl 10 dev ib1
# ip addr add 192.168.99.2/24 dev vxlan99
# ip link set up dev vxlan99
# iperf -u -s -B 239.0.0.99 & /* IGMPv3 Join TIPS */
Source: SAKURA Internet Research Center. 11/2012: Project THORN
5.
VXLAN performance check
VXLAN-A# ip link show dev vxlan99
5: vxlan99: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65470 qdisc
56Gbit/s IPoIB
link/ether b2:cc:a5:b8:74:95 brd ff:ff:ff:ff:ff:ff IP over Infiniband Network
VXLAN-A# iperf -c 192.168.99.2 -d
: VXLAN Network
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 29.7 GBytes 25.5 Gbits/sec
VXLAN-B# ip link show dev vxlan99
5: vxlan99: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65470 qdisc
link/ether 4a:a6:cb:85:fb:2b brd ff:ff:ff:ff:ff:ff
VXLAN-B# iperf -s & vnstat -l -i ib1
Monitoring ib1... (press CTRL-C to stop)
rx: 18.49 Gbit/s 53431 p/s tx: 22.82 Gbit/s 58258 p/s
Source: SAKURA Internet Research Center. 11/2012: Project THORN
6.
Traffic Analysis using Wireshark
Outer Header
Inner Header
Source: SAKURA Internet Research Center. 11/2012: Project THORN
7.
Debug TIPS for tshark
1) VXLAN Traffic Monitoring
# tshark -pni ib1
Capturing on ib1
0.000000 10.1.1.1 -> 239.0.0.99 UDP Source port: 38367 Destination port: 8472
0.001072 10.1.1.2 -> 10.1.1.1 UDP Source port: 39994 Destination port: 8472
2) IGMPv3 Join/Releave Monitoring
# tshark -i ib1 -VV
Frame 1 (56 bytes on wire, 56 bytes captured)
Internet Protocol, Src: 10.1.1.1 (10.1.1.1), Dst: 224.0.0.22 (224.0.0.22)
Internet Group Management Protocol
[IGMP Version: 3]
Type: Membership Report (0x22)
Header checksum: 0xea9a [correct]
Num Group Records: 1
Group Record : 239.0.0.99 Change To Exclude Mode
Record Type: Change To Exclude Mode (4)
Aux Data Len: 0
Num Src: 0
Multicast Address: 239.0.0.99 (239.0.0.99)
Source: SAKURA Internet Research Center. 11/2012: Project THORN
8.
Debug TIPS for iproute2
1) VXLAN Link status
# ip -d link show dev vxlan99
5: vxlan99: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65470 qdisc noqueue state UNKNOWN
link/ether 4a:a6:cb:85:fb:2b brd ff:ff:ff:ff:ff:ff
vxlan id 5001 group 239.0.0.99 dev ib1 port 32768 61000 ageing 300
2) VXLAN IP Address information
# ip -d addr show dev vxlan99
5: vxlan99: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65470 qdisc noqueue state UNKNOWN
link/ether 4a:a6:cb:85:fb:2b brd ff:ff:ff:ff:ff:ff
inet 192.168.99.1/24 scope global vxlan99
inet6 fe80::48a6:cbff:fe85:fb2b/64 scope link
valid_lft forever preferred_lft forever
3) VXLAN Network Table Lookup
# netstat -nr
Kernel IP routing table
Destination Gateway Genmask Flags MSS Window irtt Iface
10.1.1.0 0.0.0.0 255.255.255.0 U 00 0 ib1
192.168.99.0 0.0.0.0 255.255.255.0 U 00 0 vxlan99
224.0.0.0 0.0.0.0 240.0.0.0 U 00 0 ib1
Source: SAKURA Internet Research Center. 11/2012: Project THORN