A First Look At VXLAN over Infiniband Network On Linux 3.7-rc7 & iproute2

29 Nov, 2012 SAKURA Internet Research CenterSenior Researcher / Naoto MATSUMOTO

VXLAN TESTING Overview 1) Linux bridge-based VXLAN is developing by Stephen Hemminger & et al. http://linux-network-plumber.blogspot.jp/2012/09/just-published-linux-kernel.html +config VXLAN + tristate "Virtual eXtensible Local Area Network (VXLAN)" + depends on EXPERIMENTAL + ---help--- + This allows one to create vxlan virtual interfaces that provide + Layer 2 Networks over Layer 3 Networks. VXLAN is often used + to tunnel virtual network infrastructure in virtualized environments. + For more information see: + http://tools.ietf.org/html/draft-mahalingam-dutt-dcops-vxlan-02 2) Infiniband Adapter is the fastest of all for datacenter networking. Dual-port Infiniband FDR 56Gbit/s Host Bus Adapter VXLAN + IP over Infiniband Network is AWESOME!.

Buildup to the New Kernel & etc 1) Install OFED for Infiniband Networking # wget https://www.openfabrics.org/downloads/OFED/ofed-1.5.4/OFED-1.5.4.1.tgz # tar xzvf ./OFED-1.5.4.1.tgz # ./OFED-1.5.4.1/install.pl /*Install OFED software */ # sync; sync; sync; reboot; 2) Linux Kernel Update # cd /usr/local/src # wget http://www.kernel.org/pub/linux/kernel/v3.0/testing/linux-3.7-rc7.tar.bz2 # tar xvfj linux-3.7-rc7.tar.bz2 # cd linux-3.7-rc7 # cp /boot/config-2.6.32-220.23.1.el6.x86_64 ./.config # make oldconfig # vi .config CONFIG_VXLAN=m /* Add New Line */ # make; make modules_install ; make install # sync; sync; sync; reboot 3) iproute2 Update # yum install db4-devel.x86_64 # git clone git://git.kernel.org/pub/scm/linux/kernel/git/shemminger/iproute2.git # cd iproute2; ./configure ; make ; make install Source: SAKURA Internet Research Center. 11/2012: Project THORN

VXLAN Configuration# uname -aLinux VXLAN-A 3.7.0-rc7 #1 SMP .... 56Gbit/s IPoIB# ip addr add 10.1.1.1/24 dev ib1 IP over Infiniband Network# ip route add 224.0.0.0/4 dev ib1# ip link set up dev ib1# ip link add vxlan99 type vxlan id 5001 group 239.0.0.99 ttl 10 dev ib1 VXLAN Network# ip addr add 192.168.99.1/24 dev vxlan99# ip link set up dev vxlan99# iperf -u -s -B 239.0.0.99 & /* IGMPv3 Join TIPS */# uname -aLinux VXLAN-B 3.7.0-rc7 #1 SMP ....# ip addr add 10.1.1.2/24 dev ib1# ip route add 224.0.0.0/4 dev ib1# ip link set up dev ib1# ip link add vxlan99 type vxlan id 5001 group 239.0.0.99 ttl 10 dev ib1# ip addr add 192.168.99.2/24 dev vxlan99# ip link set up dev vxlan99# iperf -u -s -B 239.0.0.99 & /* IGMPv3 Join TIPS */ Source: SAKURA Internet Research Center. 11/2012: Project THORN

VXLAN performance checkVXLAN-A# ip link show dev vxlan995: vxlan99: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65470 qdisc 56Gbit/s IPoIBlink/ether b2:cc:a5:b8:74:95 brd ff:ff:ff:ff:ff:ff IP over Infiniband NetworkVXLAN-A# iperf -c 192.168.99.2 -d: VXLAN Network[ ID] Interval Transfer Bandwidth[ 3] 0.0-10.0 sec 29.7 GBytes 25.5 Gbits/secVXLAN-B# ip link show dev vxlan995: vxlan99: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65470 qdisclink/ether 4a:a6:cb:85:fb:2b brd ff:ff:ff:ff:ff:ffVXLAN-B# iperf -s & vnstat -l -i ib1Monitoring ib1... (press CTRL-C to stop)rx: 18.49 Gbit/s 53431 p/s tx: 22.82 Gbit/s 58258 p/s Source: SAKURA Internet Research Center. 11/2012: Project THORN

Traffic Analysis using Wireshark Outer Header Inner Header Source: SAKURA Internet Research Center. 11/2012: Project THORN

Debug TIPS for tshark 1) VXLAN Traffic Monitoring # tshark -pni ib1 Capturing on ib1 0.000000 10.1.1.1 -> 239.0.0.99 UDP Source port: 38367 Destination port: 8472 0.001072 10.1.1.2 -> 10.1.1.1 UDP Source port: 39994 Destination port: 8472 2) IGMPv3 Join/Releave Monitoring # tshark -i ib1 -VV Frame 1 (56 bytes on wire, 56 bytes captured) Internet Protocol, Src: 10.1.1.1 (10.1.1.1), Dst: 224.0.0.22 (224.0.0.22) Internet Group Management Protocol [IGMP Version: 3] Type: Membership Report (0x22) Header checksum: 0xea9a [correct] Num Group Records: 1 Group Record : 239.0.0.99 Change To Exclude Mode Record Type: Change To Exclude Mode (4) Aux Data Len: 0 Num Src: 0 Multicast Address: 239.0.0.99 (239.0.0.99) Source: SAKURA Internet Research Center. 11/2012: Project THORN

Debug TIPS for iproute2 1) VXLAN Link status # ip -d link show dev vxlan99 5: vxlan99: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65470 qdisc noqueue state UNKNOWN link/ether 4a:a6:cb:85:fb:2b brd ff:ff:ff:ff:ff:ff vxlan id 5001 group 239.0.0.99 dev ib1 port 32768 61000 ageing 300 2) VXLAN IP Address information # ip -d addr show dev vxlan99 5: vxlan99: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65470 qdisc noqueue state UNKNOWN link/ether 4a:a6:cb:85:fb:2b brd ff:ff:ff:ff:ff:ff inet 192.168.99.1/24 scope global vxlan99 inet6 fe80::48a6:cbff:fe85:fb2b/64 scope link valid_lft forever preferred_lft forever 3) VXLAN Network Table Lookup # netstat -nr Kernel IP routing table Destination Gateway Genmask Flags MSS Window irtt Iface 10.1.1.0 0.0.0.0 255.255.255.0 U 00 0 ib1 192.168.99.0 0.0.0.0 255.255.255.0 U 00 0 vxlan99 224.0.0.0 0.0.0.0 240.0.0.0 U 00 0 ib1 Source: SAKURA Internet Research Center. 11/2012: Project THORN

Thank you for your interest.

https://twitter.com	28
https://si0.twimg.com	4
http://twitter.com	3
http://app.brandwatch.com	2
https://abs.twimg.com	2
http://moderation.local	1
https://twimg0-a.akamaihd.net	1
http://www.slashdocs.com	1
http://www.docseek.net	1

A First Look At VXLAN over Infiniband Network On Linux 3.7-rc7 & iproute2

by Naoto MATSUMOTO on Nov 29, 2012

Accessibility

Categories

Upload Details

Usage Rights

9 Embeds 43

Statistics

A First Look At VXLAN over Infiniband Network On Linux 3.7-rc7 & iproute2 Presentation Transcript